{
  "id": "residual-connection",
  "title": "Residual Connection",
  "lang": "en",
  "source_lang": "en",
  "summary": "A residual connection adds a layer’s input to its transformed output, helping deep neural networks preserve information and train effectively.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Residual shortcut",
    "additive skip connection",
    "residual skip connection"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/residual-connection/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/residual-connection/",
      "json": "https://aiwiki.page/zh/residual-connection.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "A residual connection is a connection in an artificial neural network that bypasses one or more computational layers and adds the bypassed input to their output. Instead of learning an entire transformation directly, the intervening layers learn a correction, or residual, to an existing representation. Residual connections are a central component of residual neural networks and are widely used in deep learning, including convolutional models and Transformers. They make very deep networks easier to optimize by providing direct paths for information and gradients."
    },
    {
      "heading": "Mathematical definition",
      "text": "The simplest residual block computes\n\n\\[y=x+F(x;\\theta),\\]\n\nwhere \\(x\\) is the input, \\(F\\) is a learned transformation with parameters \\(\\theta\\), and \\(y\\) is the output. The transformation may contain several layers, nonlinearities, normalization operations, or other computations. The direct path carrying \\(x\\) is called the shortcut or skip path; the path computing \\(F(x;\\theta)\\) is the residual branch.\n\nIf the desired mapping is \\(H(x)\\), the residual formulation represents it as\n\n\\[H(x)=x+F(x), \\qquad F(x)=H(x)-x.\\]\n\n“Residual” therefore refers to the difference between the desired mapping and the identity mapping. It does not mean that the branch directly computes prediction errors, and it is not itself a loss function.\n\nAddition requires the two branches to produce compatible shapes. When dimensions differ, the shortcut can use a projection:\n\n\\[y=P(x)+F(x;\\theta).\\]\n\nFor a vector representation, a common choice is \\(P(x)=W_sx\\). In a convolutional neural network, a \\(1\\times1\\) convolution, sometimes with a stride greater than one, can change the channel count and spatial resolution. Such a projection shortcut is no longer an exact identity path and may introduce additional trainable parameters.\n\nSome blocks apply an activation function after addition:\n\n\\[y=\\phi\\bigl(x+F(x;\\theta)\\bigr).\\]\n\nOthers leave the addition unmodified and place activation and normalization operations within the residual branch. This distinction affects both information preservation and gradient propagation."
    },
    {
      "heading": "Why residual connections help",
      "text": "Learning changes rather than complete replacements\n\nA conventional stack repeatedly replaces its current representation:\n\n\\[x_{l+1}=F_l(x_l).\\]\n\nA residual stack instead repeatedly updates it:\n\n\\[x_{l+1}=x_l+F_l(x_l).\\]\n\nIf a useful transformation is close to the identity, a residual branch can represent it by producing a small correction. For an identity-shortcut block with no activation after addition, setting the residual output to zero makes the entire block an identity mapping.\n\nThis observation motivated residual learning as a response to the degradation problem: adding layers to a plain network can increase training error, even though the deeper architecture appears capable of representing the shallower solution. This is an optimization problem, distinct from overfitting, in which training performance improves while performance on unseen data worsens.\n\nResidual parameterization does not make every desired transformation simpler. Its advantage depends on the architecture, initialization, data, and optimization procedure.\n\nGradient propagation\n\nResidual connections create a direct route for derivatives during backpropagation. For\n\n\\[y=x+F(x),\\]\n\nthe Jacobian is\n\n\\[\\frac{\\partial y}{\\partial x}=I+J_F(x),\\]\n\nwhere \\(I\\) is the identity matrix and \\(J_F(x)\\) is the Jacobian of the residual branch. With column-vector gradients, the gradient of a scalar loss \\(L\\) is\n\n\\[\\nabla_x L = \\bigl(I+J_F(x)\\bigr)^{\\mathsf T}\\nabla_y L.\\]\n\nThe identity term supplies a contribution that does not pass through the residual branch’s internal operations. Across many layers, these shortcut routes can alleviate the vanishing gradient problem.\n\nThey do not guarantee well-behaved gradients. Products of matrices of the form \\(I+J_F\\) can still grow, shrink, or become poorly conditioned. An activation applied after addition also modifies the derivative of the shortcut route and can suppress some components.\n\nPreservation of intermediate representations\n\nFor a sequence of identity-shortcut blocks without post-addition transformations,\n\n\\[x_L=x_l+\\sum_{k=l}^{L-1}F_k(x_k).\\]\n\nThe representation at depth \\(l\\) thus appears explicitly in the expression for a later representation. Subsequent layers can add new features without necessarily reconstructing all previously useful information.\n\nThis is an algebraic property, not a guarantee that information remains recoverable: residual updates can overwrite or cancel earlier components."
    },
    {
      "heading": "Historical development",
      "text": "Connections that bypass layers existed before modern residual networks. The distinctive contribution of deep residual learning was to make additive identity shortcuts the organizing principle of a highly effective deep architecture.\n\nKaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun introduced Deep Residual Learning for Image Recognition in 2015; the paper appeared at CVPR in 2016. Their experiments showed that residual architectures could train substantially deeper models for computer vision. A 152-layer model was a prominent demonstration, and residual networks formed part of the system that won the classification task of the 2015 ImageNet Large Scale Visual Recognition Challenge.\n\nThe authors’ 2016 work Identity Mappings in Deep Residual Networks examined the importance of clean identity paths and introduced influential pre-activation designs.\n\nResidual connections subsequently became a general architectural device rather than a feature exclusive to ResNet. The Transformer introduced in 2017 used residual connections around attention and feed-forward sublayers, helping establish them as a standard component of natural language processing architectures."
    },
    {
      "heading": "Major forms",
      "text": "Identity and projection shortcuts\n\nAn identity shortcut carries its input unchanged. It introduces no trainable parameters and is appropriate when input and output shapes match.\n\nA projection shortcut transforms the input to match the residual branch. Projection is common at transitions between stages of an image model, where spatial resolution decreases and channel count increases. Projection shortcuts provide architectural flexibility but do not retain all the mathematical properties of exact identity shortcuts.\n\nPost-activation and pre-activation blocks\n\nIn the original ResNet design, addition is followed by a rectified linear unit. Convolutions, batch normalization, and intermediate activations appear within the residual branch.\n\nA pre-activation design places normalization and activation before the branch’s weight layers and avoids an activation immediately after addition. This creates a cleaner identity route between successive blocks. “Pre-activation” describes this ordering in residual networks; it is related to, but not identical with, Transformer pre-normalization.\n\nScaled and gated residual connections\n\nA scaled residual block computes\n\n\\[y=x+\\alpha F(x),\\]\n\nwhere \\(\\alpha\\) may be fixed or learned. Residual scaling controls the magnitude of updates and can help manage activation and gradient growth in deep networks.\n\nA gated variant can use\n\n\\[y=x+g(x)\\odot F(x),\\]\n\nwhere \\(g(x)\\) controls the update and \\(\\odot\\) denotes elementwise multiplication. More general gated architectures may also gate the shortcut itself. Such designs differ from an unconditional identity shortcut because information flow depends on the gate."
    },
    {
      "heading": "Residual connections in Transformers",
      "text": "Transformers use residual connections around attention and position-wise feed-forward sublayers. For self-attention, the shortcut carries the current representation while the branch computes a context-dependent update.\n\nThe placement of layer normalization is especially important. A schematic post-normalization sublayer computes\n\n\\[y=\\operatorname{LN}\\bigl(x+F(x)\\bigr).\\]\n\nA pre-normalization sublayer computes\n\n\\[y=x+F\\bigl(\\operatorname{LN}(x)\\bigr).\\]\n\nThese equations omit details such as dropout and residual scaling.\n\nPre-normalization leaves the shortcut unchanged through each sublayer, whereas post-normalization transforms the sum before passing it onward. This difference affects optimization and gradient flow. Many modern large language models use pre-normalization or related arrangements, although normalization choices vary.\n\nThe sequence of representations carried between residual additions is often called the residual stream. Attention and feed-forward branches write updates into this shared representation. The term does not imply that each component occupies a separate, permanently identifiable part of the vector."
    },
    {
      "heading": "Relationship to other skip connections",
      "text": "Skip connection is a broader term than residual connection. A skip connection can bypass layers and combine representations in several ways.\n\n- Addition: Residual connections normally merge compatible representations element by element, retaining the same output shape.\n- Concatenation: Some architectures join representations along a feature or channel dimension, increasing the width of the resulting representation.\n- Gated combination: Other architectures use learned gates to regulate the relative contributions of transformed and bypassed inputs.\n\nLong-range connections between encoder and decoder stages, including those used for image segmentation, are therefore not necessarily residual connections. The defining feature is the additive residual formulation, not merely the presence of a bypass."
    },
    {
      "heading": "Applications and limitations",
      "text": "Residual connections are used in image recognition, object detection, segmentation, language modeling, and diffusion models. They support deep stacks of transformations while allowing intermediate representations to be updated incrementally.\n\nThe addition itself is usually inexpensive relative to a large convolution or attention operation, and an identity shortcut adds no trainable weights. Nevertheless, residual architectures still incur the computational and memory costs of their branches. Projection shortcuts add work, and training implementations must account for the activations needed by the computational graph.\n\nResidual connections do not replace normalization, suitable initialization, a well-chosen learning rate, or regularization. Very deep models can still exhibit unstable gradients or excessive growth in representation magnitudes. Residual branches may also become negligible, so additional depth does not necessarily provide a corresponding increase in effective computation.\n\nFinally, improved optimization does not by itself guarantee improved generalization, robustness, or interpretability. Residual structure primarily changes how a network represents and learns transformations; those other properties depend on the broader model and training process."
    }
  ],
  "links": [
    "artificial-neural-network",
    "residual-neural-network",
    "deep-learning",
    "transformer-architecture",
    "loss-function",
    "convolutional-neural-network",
    "activation-function",
    "overfitting",
    "backpropagation",
    "matrix",
    "vanishing-gradient-problem",
    "computer-vision",
    "imagenet-large-scale-visual-recognition-challenge",
    "natural-language-processing",
    "rectified-linear-unit",
    "batch-normalization",
    "attention-mechanism",
    "self-attention",
    "layer-normalization",
    "dropout",
    "large-language-model",
    "image-segmentation",
    "object-detection",
    "diffusion-model",
    "computational-graph",
    "learning-rate",
    "regularization",
    "generalization"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}