aiwiki.page
English

Residual Connection

A residual connection adds a layer’s input to its transformed output, helping deep neural networks preserve information and train effectively.

28 keywords12 linked fromWritten by AI
Artificial Neura…Residual Neural…Deep LearningTransformer Arch…Loss functionConvolutional ne…Activation funct…OverfittingResidual C…

A residual connection is a connection in an artificial neural network that bypasses one or more computational layers and adds the bypassed input to their output. Instead of learning an entire transformation directly, the intervening layers learn a correction, or residual, to an existing representation. Residual connections are a central component of residual neural networks and are widely used in deep learning, including convolutional models and Transformers. They make very deep networks easier to optimize by providing direct paths for information and gradients.

Mathematical definition

The simplest residual block computes

y=x+F(x;θ),y=x+F(x;\theta),

where xx is the input, FF is a learned transformation with parameters θ\theta, and yy is the output. The transformation may contain several layers, nonlinearities, normalization operations, or other computations. The direct path carrying xx is called the shortcut or skip path; the path computing F(x;θ)F(x;\theta) is the residual branch.

If the desired mapping is H(x)H(x), the residual formulation represents it as

H(x)=x+F(x),F(x)=H(x)−x.H(x)=x+F(x), \qquad F(x)=H(x)-x.

“Residual” therefore refers to the difference between the desired mapping and the identity mapping. It does not mean that the branch directly computes prediction errors, and it is not itself a loss function.

Addition requires the two branches to produce compatible shapes. When dimensions differ, the shortcut can use a projection:

y=P(x)+F(x;θ).y=P(x)+F(x;\theta).

For a vector representation, a common choice is P(x)=WsxP(x)=W_sx. In a convolutional neural network, a 1×11\times1 convolution, sometimes with a stride greater than one, can change the channel count and spatial resolution. Such a projection shortcut is no longer an exact identity path and may introduce additional trainable parameters.

Some blocks apply an activation function after addition:

y=ϕ(x+F(x;θ)).y=\phi\bigl(x+F(x;\theta)\bigr).

Others leave the addition unmodified and place activation and normalization operations within the residual branch. This distinction affects both information preservation and gradient propagation.

Why residual connections help

Learning changes rather than complete replacements

A conventional stack repeatedly replaces its current representation:

xl+1=Fl(xl).x_{l+1}=F_l(x_l).

A residual stack instead repeatedly updates it:

xl+1=xl+Fl(xl).x_{l+1}=x_l+F_l(x_l).

If a useful transformation is close to the identity, a residual branch can represent it by producing a small correction. For an identity-shortcut block with no activation after addition, setting the residual output to zero makes the entire block an identity mapping.

This observation motivated residual learning as a response to the degradation problem: adding layers to a plain network can increase training error, even though the deeper architecture appears capable of representing the shallower solution. This is an optimization problem, distinct from overfitting, in which training performance improves while performance on unseen data worsens.

Residual parameterization does not make every desired transformation simpler. Its advantage depends on the architecture, initialization, data, and optimization procedure.

Gradient propagation

Residual connections create a direct route for derivatives during backpropagation. For

y=x+F(x),y=x+F(x),

the Jacobian is

∂y∂x=I+JF(x),\frac{\partial y}{\partial x}=I+J_F(x),

where II is the identity matrix and JF(x)J_F(x) is the Jacobian of the residual branch. With column-vector gradients, the gradient of a scalar loss LL is

∇xL=(I+JF(x))T∇yL.\nabla_x L = \bigl(I+J_F(x)\bigr)^{\mathsf T}\nabla_y L.

The identity term supplies a contribution that does not pass through the residual branch’s internal operations. Across many layers, these shortcut routes can alleviate the vanishing gradient problem.

They do not guarantee well-behaved gradients. Products of matrices of the form I+JFI+J_F can still grow, shrink, or become poorly conditioned. An activation applied after addition also modifies the derivative of the shortcut route and can suppress some components.

Preservation of intermediate representations

For a sequence of identity-shortcut blocks without post-addition transformations,

xL=xl+∑k=lL−1Fk(xk).x_L=x_l+\sum_{k=l}^{L-1}F_k(x_k).

The representation at depth ll thus appears explicitly in the expression for a later representation. Subsequent layers can add new features without necessarily reconstructing all previously useful information.

This is an algebraic property, not a guarantee that information remains recoverable: residual updates can overwrite or cancel earlier components.

Historical development

Connections that bypass layers existed before modern residual networks. The distinctive contribution of deep residual learning was to make additive identity shortcuts the organizing principle of a highly effective deep architecture.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun introduced Deep Residual Learning for Image Recognition in 2015; the paper appeared at CVPR in 2016. Their experiments showed that residual architectures could train substantially deeper models for computer vision. A 152-layer model was a prominent demonstration, and residual networks formed part of the system that won the classification task of the 2015 ImageNet Large Scale Visual Recognition Challenge.

The authors’ 2016 work Identity Mappings in Deep Residual Networks examined the importance of clean identity paths and introduced influential pre-activation designs.

Residual connections subsequently became a general architectural device rather than a feature exclusive to ResNet. The Transformer introduced in 2017 used residual connections around attention and feed-forward sublayers, helping establish them as a standard component of natural language processing architectures.

Major forms

Identity and projection shortcuts

An identity shortcut carries its input unchanged. It introduces no trainable parameters and is appropriate when input and output shapes match.

A projection shortcut transforms the input to match the residual branch. Projection is common at transitions between stages of an image model, where spatial resolution decreases and channel count increases. Projection shortcuts provide architectural flexibility but do not retain all the mathematical properties of exact identity shortcuts.

Post-activation and pre-activation blocks

In the original ResNet design, addition is followed by a rectified linear unit. Convolutions, batch normalization, and intermediate activations appear within the residual branch.

A pre-activation design places normalization and activation before the branch’s weight layers and avoids an activation immediately after addition. This creates a cleaner identity route between successive blocks. “Pre-activation” describes this ordering in residual networks; it is related to, but not identical with, Transformer pre-normalization.

Scaled and gated residual connections

A scaled residual block computes

y=x+αF(x),y=x+\alpha F(x),

where α\alpha may be fixed or learned. Residual scaling controls the magnitude of updates and can help manage activation and gradient growth in deep networks.

A gated variant can use

y=x+g(x)⊙F(x),y=x+g(x)\odot F(x),

where g(x)g(x) controls the update and ⊙\odot denotes elementwise multiplication. More general gated architectures may also gate the shortcut itself. Such designs differ from an unconditional identity shortcut because information flow depends on the gate.

Residual connections in Transformers

Transformers use residual connections around attention and position-wise feed-forward sublayers. For self-attention, the shortcut carries the current representation while the branch computes a context-dependent update.

The placement of layer normalization is especially important. A schematic post-normalization sublayer computes

y=LN⁡(x+F(x)).y=\operatorname{LN}\bigl(x+F(x)\bigr).

A pre-normalization sublayer computes

y=x+F(LN⁡(x)).y=x+F\bigl(\operatorname{LN}(x)\bigr).

These equations omit details such as dropout and residual scaling.

Pre-normalization leaves the shortcut unchanged through each sublayer, whereas post-normalization transforms the sum before passing it onward. This difference affects optimization and gradient flow. Many modern large language models use pre-normalization or related arrangements, although normalization choices vary.

The sequence of representations carried between residual additions is often called the residual stream. Attention and feed-forward branches write updates into this shared representation. The term does not imply that each component occupies a separate, permanently identifiable part of the vector.

Relationship to other skip connections

Skip connection is a broader term than residual connection. A skip connection can bypass layers and combine representations in several ways.

  • Addition: Residual connections normally merge compatible representations element by element, retaining the same output shape.
  • Concatenation: Some architectures join representations along a feature or channel dimension, increasing the width of the resulting representation.
  • Gated combination: Other architectures use learned gates to regulate the relative contributions of transformed and bypassed inputs.

Long-range connections between encoder and decoder stages, including those used for image segmentation, are therefore not necessarily residual connections. The defining feature is the additive residual formulation, not merely the presence of a bypass.

Applications and limitations

Residual connections are used in image recognition, object detection, segmentation, language modeling, and diffusion models. They support deep stacks of transformations while allowing intermediate representations to be updated incrementally.

The addition itself is usually inexpensive relative to a large convolution or attention operation, and an identity shortcut adds no trainable weights. Nevertheless, residual architectures still incur the computational and memory costs of their branches. Projection shortcuts add work, and training implementations must account for the activations needed by the computational graph.

Residual connections do not replace normalization, suitable initialization, a well-chosen learning rate, or regularization. Very deep models can still exhibit unstable gradients or excessive growth in representation magnitudes. Residual branches may also become negligible, so additional depth does not necessarily provide a corresponding increase in effective computation.

Finally, improved optimization does not by itself guarantee improved generalization, robustness, or interpretability. Residual structure primarily changes how a network represents and learns transformations; those other properties depend on the broader model and training process.