{
  "id": "sigmoid-linear-unit",
  "title": "Sigmoid Linear Unit",
  "lang": "en",
  "source_lang": "en",
  "summary": "The sigmoid linear unit (SiLU) is a smooth, nonmonotonic neural-network activation function defined as an input multiplied by its logistic sigmoid.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "SiLU",
    "Sigmoid-weighted linear unit",
    "Swish-1"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/sigmoid-linear-unit/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/sigmoid-linear-unit/",
      "json": "https://aiwiki.page/zh/sigmoid-linear-unit.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The sigmoid linear unit (SiLU) is an activation function used in artificial neural networks. It transforms a real-valued input \\(x\\) into \\(x\\sigma(x)\\), where \\(\\sigma\\) is the logistic sigmoid function. SiLU is smooth, retains small negative outputs, and approaches the identity function for large positive inputs. It is also the parameter-one case of the Swish family and is often called simply Swish, although that name can also refer to a parameterized generalization."
    },
    {
      "heading": "Definition",
      "text": "SiLU is defined by\n\n\\[\\operatorname{SiLU}(x)=x\\sigma(x)=\\frac{x}{1+e^{-x}},\\]\n\nwhere the logistic sigmoid is\n\n\\[\\sigma(x)=\\frac{1}{1+e^{-x}}.\\]\n\nIn a neural-network layer, SiLU is normally applied independently to every component of an input vector or tensor. It introduces a nonlinearity without changing the tensor’s shape. Standard SiLU has no learned parameters.\n\nThe function can be interpreted as self-gating: the sigmoid of an input determines how strongly that same input passes through. Large positive values receive a multiplier close to one, while large negative values receive a multiplier close to zero. This describes a deterministic computation, not the random selection of whether a unit activates."
    },
    {
      "heading": "Mathematical properties",
      "text": "Shape and limits\n\nSiLU is infinitely differentiable over the real numbers. Its principal limiting behaviors are\n\n\\[\\lim_{x\\to-\\infty}\\operatorname{SiLU}(x)=0^{-}, \\qquad \\lim_{x\\to+\\infty}\\bigl(\\operatorname{SiLU}(x)-x\\bigr)=0.\\]\n\nIt therefore suppresses strongly negative inputs and acts approximately linearly on strongly positive inputs. At the origin,\n\n\\[\\operatorname{SiLU}(0)=0.\\]\n\nUnlike the rectified linear unit (ReLU), SiLU does not set all negative inputs to zero. Every finite negative input produces a negative output.\n\nSiLU is also nonmonotonic. As the input increases from a very large negative value, the output initially decreases, reaches a global minimum, and then increases. The minimum occurs at approximately\n\n\\[x=-1.27846, \\qquad \\operatorname{SiLU}(x)=-0.27846.\\]\n\nIts range is consequently approximately \\([-0.27846,\\infty)\\).\n\nDerivative\n\nThe derivative used in backpropagation is\n\n\\[\\frac{d}{dx}\\operatorname{SiLU}(x) = \\sigma(x)+x\\sigma(x)\\bigl(1-\\sigma(x)\\bigr).\\]\n\nEquivalently,\n\n\\[\\operatorname{SiLU}'(x) = \\sigma(x)\\left[1+x\\bigl(1-\\sigma(x)\\bigr)\\right].\\]\n\nAt zero, the derivative is \\(1/2\\). It approaches zero on the far negative side and one on the far positive side. It is negative to the left of the function’s minimum and can exceed one for positive inputs.\n\nSmoothness removes ReLU’s derivative discontinuity at zero, but it does not guarantee easier optimization in every network. In particular, SiLU still has very small derivatives for sufficiently negative inputs and therefore does not eliminate the vanishing-gradient problem."
    },
    {
      "heading": "Relationship to other activation functions",
      "text": "ReLU\n\nReLU is defined as \\(\\max(0,x)\\). Both functions have approximately linear behavior for large positive inputs, but they differ near zero and throughout the negative region:\n\nProperty | SiLU | ReLU\nSmooth everywhere | Yes | No\nNegative outputs | Yes | No\nMonotonic | No | Yes\nExactly zero for all negative inputs | No | Yes\nBasic computation | Sigmoid and multiplication | Maximum with zero\n\nBecause SiLU has no interval on which it is identically zero, it avoids ReLU’s specific mechanism of permanently inactive units caused by a flat negative branch. Nevertheless, gradients can still become extremely small, and finite-precision arithmetic can introduce effective saturation.\n\nParameterized Swish\n\nThe broader Swish family is commonly written as\n\n\\[\\operatorname{Swish}_{\\beta}(x)=x\\sigma(\\beta x).\\]\n\nSiLU corresponds exactly to \\(\\beta=1\\). Depending on the architecture, \\(\\beta\\) may be fixed or learned.\n\nFor \\(\\beta=0\\), the function becomes \\(x/2\\). As positive \\(\\beta\\) tends to infinity, it converges pointwise to ReLU. SiLU should therefore be distinguished from versions of Swish with an adjustable parameter.\n\nGELU\n\nThe Gaussian error linear unit (GELU) has the related form\n\n\\[\\operatorname{GELU}(x)=x\\Phi(x),\\]\n\nwhere \\(\\Phi\\) is the standard normal cumulative distribution function. Both functions multiply the input by a smooth factor between zero and one. They have similar overall shapes, including a small negative region, but are not identical: SiLU uses a logistic sigmoid, whereas exact GELU uses a Gaussian cumulative distribution function."
    },
    {
      "heading": "History",
      "text": "In 2017, Stefan Elfwing, Eiji Uchibe, and Kenji Doya introduced the name sigmoid-weighted linear unit in work on neural-network function approximation for reinforcement learning. Their study considered SiLU and a related activation based on its derivative.\n\nAlso in 2017, Prajit Ramachandran, Barret Zoph, and Quoc V. Le presented Swish in research on searching for activation functions. The Swish formulation included the parameterized expression \\(x\\sigma(\\beta x)\\), connecting its parameter-one form directly to SiLU.\n\nSubsequent software libraries and model descriptions have used both names. Consequently, the defining equation is more precise than the name alone when comparing implementations."
    },
    {
      "heading": "Uses in neural networks",
      "text": "SiLU is used in deep learning, including convolutional neural networks and feed-forward components of transformers. It can replace another elementwise activation without changing layer dimensions, although such a replacement changes the model’s computation and can require retraining.\n\nA related application is SwiGLU, a gated feed-forward construction used in some large language models. One common formulation is\n\n\\[\\operatorname{SwiGLU}(x) = \\left[ \\operatorname{SiLU}(xW_g+b_g) \\odot (xW_v+b_v) \\right]W_o+b_o,\\]\n\nwhere the \\(W\\) terms are learned weight matrices, the \\(b\\) terms are biases, and \\(\\odot\\) denotes elementwise multiplication. Some implementations omit biases.\n\nSwiGLU is not merely another name for SiLU: it combines SiLU with two input projections, multiplicative gating, and an output projection."
    },
    {
      "heading": "Implementation and limitations",
      "text": "PyTorch provides SiLU directly, while TensorFlow provides the equivalent operation under SiLU or Swish naming. Implementations generally use a numerically stable sigmoid rather than evaluating the defining exponential expression naively.\n\nCompared with ReLU, SiLU requires more arithmetic. Its actual cost depends on hardware, numerical precision, tensor size, and whether the operation is fused with surrounding computations. Thus, the complexity of its scalar formula alone does not determine training or inference speed.\n\nIts smooth, curved negative region can also complicate quantization relative to a simple piecewise-linear activation. Approximations and specialized kernels can reduce these costs, but their accuracy and efficiency depend on the implementation.\n\nSiLU does not guarantee better accuracy, faster convergence, or improved generalization over other activation functions. Results depend on architecture, initialization, optimization settings, and task. Its smoothness and self-gating behavior are useful design properties rather than universal performance guarantees."
    }
  ],
  "links": [
    "activation-function",
    "artificial-neural-network",
    "logistic-function",
    "rectified-linear-unit",
    "derivative",
    "backpropagation",
    "vanishing-gradient-problem",
    "reinforcement-learning",
    "deep-learning",
    "convolutional-neural-network",
    "transformer-architecture",
    "large-language-model",
    "pytorch",
    "tensorflow",
    "neural-network-inference",
    "neural-network-quantization",
    "generalization"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}