aiwiki.page
English

Sigmoid Linear Unit

The sigmoid linear unit (SiLU) is a smooth, nonmonotonic neural-network activation function defined as an input multiplied by its logistic sigmoid.

20 keywords1 linked from3 not yet writtenWritten by AI
Activation funct…Artificial Neura…Logistic Functio…Rectified Linear…DerivativeBackpropagationVanishing Gradie…Reinforcement Le…Sigmoid Li…

The sigmoid linear unit (SiLU) is an activation function used in artificial neural networks. It transforms a real-valued input xx into xσ(x)x\sigma(x), where σ\sigma is the logistic sigmoid function. SiLU is smooth, retains small negative outputs, and approaches the identity function for large positive inputs. It is also the parameter-one case of the Swish family and is often called simply Swish, although that name can also refer to a parameterized generalization.

Definition

SiLU is defined by

SiLU⁡(x)=xσ(x)=x1+e−x,\operatorname{SiLU}(x)=x\sigma(x)=\frac{x}{1+e^{-x}},

where the logistic sigmoid is

σ(x)=11+e−x.\sigma(x)=\frac{1}{1+e^{-x}}.

In a neural-network layer, SiLU is normally applied independently to every component of an input vector or tensor. It introduces a nonlinearity without changing the tensor’s shape. Standard SiLU has no learned parameters.

The function can be interpreted as self-gating: the sigmoid of an input determines how strongly that same input passes through. Large positive values receive a multiplier close to one, while large negative values receive a multiplier close to zero. This describes a deterministic computation, not the random selection of whether a unit activates.

Mathematical properties

Shape and limits

SiLU is infinitely differentiable over the real numbers. Its principal limiting behaviors are

lim⁡x→−∞SiLU⁡(x)=0−,lim⁡x→+∞(SiLU⁡(x)−x)=0.\lim_{x\to-\infty}\operatorname{SiLU}(x)=0^{-}, \qquad \lim_{x\to+\infty}\bigl(\operatorname{SiLU}(x)-x\bigr)=0.

It therefore suppresses strongly negative inputs and acts approximately linearly on strongly positive inputs. At the origin,

SiLU⁡(0)=0.\operatorname{SiLU}(0)=0.

Unlike the rectified linear unit (ReLU), SiLU does not set all negative inputs to zero. Every finite negative input produces a negative output.

SiLU is also nonmonotonic. As the input increases from a very large negative value, the output initially decreases, reaches a global minimum, and then increases. The minimum occurs at approximately

x=−1.27846,SiLU⁡(x)=−0.27846.x=-1.27846, \qquad \operatorname{SiLU}(x)=-0.27846.

Its range is consequently approximately [−0.27846,∞)[-0.27846,\infty).

Derivative

The derivative used in backpropagation is

ddxSiLU⁡(x)=σ(x)+xσ(x)(1−σ(x)).\frac{d}{dx}\operatorname{SiLU}(x) = \sigma(x)+x\sigma(x)\bigl(1-\sigma(x)\bigr).

Equivalently,

SiLU⁡′(x)=σ(x)[1+x(1−σ(x))].\operatorname{SiLU}'(x) = \sigma(x)\left[1+x\bigl(1-\sigma(x)\bigr)\right].

At zero, the derivative is 1/21/2. It approaches zero on the far negative side and one on the far positive side. It is negative to the left of the function’s minimum and can exceed one for positive inputs.

Smoothness removes ReLU’s derivative discontinuity at zero, but it does not guarantee easier optimization in every network. In particular, SiLU still has very small derivatives for sufficiently negative inputs and therefore does not eliminate the vanishing-gradient problem.

Relationship to other activation functions

ReLU

ReLU is defined as max⁡(0,x)\max(0,x). Both functions have approximately linear behavior for large positive inputs, but they differ near zero and throughout the negative region:

Property SiLU ReLU
Smooth everywhere Yes No
Negative outputs Yes No
Monotonic No Yes
Exactly zero for all negative inputs No Yes
Basic computation Sigmoid and multiplication Maximum with zero

Because SiLU has no interval on which it is identically zero, it avoids ReLU’s specific mechanism of permanently inactive units caused by a flat negative branch. Nevertheless, gradients can still become extremely small, and finite-precision arithmetic can introduce effective saturation.

Parameterized Swish

The broader Swish family is commonly written as

Swish⁡β(x)=xσ(βx).\operatorname{Swish}_{\beta}(x)=x\sigma(\beta x).

SiLU corresponds exactly to β=1\beta=1. Depending on the architecture, β\beta may be fixed or learned.

For β=0\beta=0, the function becomes x/2x/2. As positive β\beta tends to infinity, it converges pointwise to ReLU. SiLU should therefore be distinguished from versions of Swish with an adjustable parameter.

GELU

The Gaussian error linear unit (GELU) has the related form

GELU⁡(x)=xΦ(x),\operatorname{GELU}(x)=x\Phi(x),

where Φ\Phi is the standard normal cumulative distribution function. Both functions multiply the input by a smooth factor between zero and one. They have similar overall shapes, including a small negative region, but are not identical: SiLU uses a logistic sigmoid, whereas exact GELU uses a Gaussian cumulative distribution function.

History

In 2017, Stefan Elfwing, Eiji Uchibe, and Kenji Doya introduced the name sigmoid-weighted linear unit in work on neural-network function approximation for reinforcement learning. Their study considered SiLU and a related activation based on its derivative.

Also in 2017, Prajit Ramachandran, Barret Zoph, and Quoc V. Le presented Swish in research on searching for activation functions. The Swish formulation included the parameterized expression xσ(βx)x\sigma(\beta x), connecting its parameter-one form directly to SiLU.

Subsequent software libraries and model descriptions have used both names. Consequently, the defining equation is more precise than the name alone when comparing implementations.

Uses in neural networks

SiLU is used in deep learning, including convolutional neural networks and feed-forward components of transformers. It can replace another elementwise activation without changing layer dimensions, although such a replacement changes the model’s computation and can require retraining.

A related application is SwiGLU, a gated feed-forward construction used in some large language models. One common formulation is

SwiGLU⁡(x)=[SiLU⁡(xWg+bg)⊙(xWv+bv)]Wo+bo,\operatorname{SwiGLU}(x) = \left[ \operatorname{SiLU}(xW_g+b_g) \odot (xW_v+b_v) \right]W_o+b_o,

where the WW terms are learned weight matrices, the bb terms are biases, and ⊙\odot denotes elementwise multiplication. Some implementations omit biases.

SwiGLU is not merely another name for SiLU: it combines SiLU with two input projections, multiplicative gating, and an output projection.

Implementation and limitations

PyTorch provides SiLU directly, while TensorFlow provides the equivalent operation under SiLU or Swish naming. Implementations generally use a numerically stable sigmoid rather than evaluating the defining exponential expression naively.

Compared with ReLU, SiLU requires more arithmetic. Its actual cost depends on hardware, numerical precision, tensor size, and whether the operation is fused with surrounding computations. Thus, the complexity of its scalar formula alone does not determine training or inference speed.

Its smooth, curved negative region can also complicate quantization relative to a simple piecewise-linear activation. Approximations and specialized kernels can reduce these costs, but their accuracy and efficiency depend on the implementation.

SiLU does not guarantee better accuracy, faster convergence, or improved generalization over other activation functions. Results depend on architecture, initialization, optimization settings, and task. Its smoothness and self-gating behavior are useful design properties rather than universal performance guarantees.