Backpropagation is an algorithm for calculating how a model’s output, usually a scalar loss function, changes with respect to its parameters. Widely used in artificial neural networks, it propagates derivative information backward through the operations that produced the output. It is a specialized application of reverse-mode automatic differentiation, not a parameter-update rule: an optimizer uses the resulting gradients to modify the model. This distinction separates computing the direction of change from deciding how to move in that direction. (jmlr.org)
Mathematical principle
The foundation of backpropagation is the chain rule of calculus. If an intermediate quantity (u) influences a loss (L) through (v), then
[ \frac{\partial L}{\partial u}
\frac{\partial L}{\partial v} \frac{\partial v}{\partial u}. ]
When (u) affects several downstream quantities, contributions from every path are added. A computational graph represents these dependencies: nodes denote quantities or operations, while edges indicate which values an operation consumes. Backpropagation traverses this graph in reverse dependency order, combining local derivatives with sensitivities already calculated downstream. It therefore avoids separately recomputing every parameter’s contribution along every possible path. (pytorch.org)
For a scalar loss, the reverse pass begins with (\partial L/\partial L=1). Each operation transforms the incoming sensitivity into sensitivities for its inputs. For vector-valued intermediate quantities, this amounts to multiplying by a transposed local Jacobian matrix, generally without constructing the complete matrix explicitly. The same mechanism handles branching computations and shared parameters by accumulating contributions. (docs.pytorch.org)
Forward and backward passes
Consider a network with input (a^0=x). At layer (l), a weight matrix (W^l), bias vector (b^l), and elementwise activation function (\phi_l) produce
[ z^l=W^l a^{l-1}+b^l, \qquad a^l=\phi_l(z^l). ]
The forward pass evaluates these expressions and then the loss. Values needed for differentiation are retained for the backward pass. Defining the layer sensitivity as (\delta^l=\partial L/\partial z^l), the hidden-layer recurrence is
[ \delta^l= \left((W^{l+1})^\top\delta^{l+1}\right) \odot\phi_l'(z^l), ]
where (\odot) denotes elementwise multiplication. The parameter derivatives are
[ \frac{\partial L}{\partial W^l}
\delta^l(a^{l-1})^\top, \qquad \frac{\partial L}{\partial b^l}=\delta^l. ]
These expressions describe a single example; derivatives for a batch are combined according to the loss’s sum or mean convention. More general architectures apply the same chain-rule procedure to their individual operations rather than relying on this particular layer recurrence. (deeplearningbook.org)
Relationship to learning and optimization
Backpropagation supplies gradients to mathematical optimization. With gradient descent, parameters (\theta) are updated by
[ \theta\leftarrow\theta-\eta\nabla_\theta L, ]
where (\eta) is the learning rate. Stochastic gradient descent estimates the gradient using sampled examples or minibatches. Other optimizers can use the same derivatives while applying different update rules. A training iteration consequently separates forward evaluation, gradient calculation, and parameter updating. (deeplearningbook.org)
In supervised learning, the objective typically compares predictions with target values from training data. Hidden layers receive no separate target: their gradients are derived from their contribution to the final loss. This enables representation learning, in which internal features emerge through training rather than being specified entirely beforehand. The 1986 experiments of David Rumelhart, Geoffrey Hinton, and Ronald Williams demonstrated this property in multilayer networks. (nature.com)
Efficiency and implementation
For computations built from standard differentiable operations, evaluating a scalar output’s full gradient typically requires only a constant-factor increase in arithmetic work relative to evaluating the output. This makes reverse mode suitable for models with many parameters and relatively few outputs. Finite-difference methods instead perturb inputs and repeat evaluations; they introduce approximation errors and become expensive when applied separately to large numbers of parameters. Backpropagation applies derivative rules directly, although its numerical results remain subject to floating-point rounding. (jmlr.org)
Memory is an important cost because the backward pass may require intermediate tensors from the forward computation. Systems such as PyTorch record operations, save necessary values, and execute corresponding backward rules automatically. Gradients may accumulate across repeated backward calls, so independent training steps require appropriate clearing. In-place modification of saved values can invalidate differentiation, while disabling gradient recording reduces overhead for computations that do not require derivatives. (docs.pytorch.org)
Historical development
Reverse-mode differentiation predates its widespread use in neural networks. Seppo Linnainmaa’s 1970 work is commonly cited as an early published description; Paul Werbos’s 1974 work was another important antecedent. The 1986 paper Learning representations by back-propagating errors helped popularize the method by showing how output-error derivatives could train hidden representations. Backpropagation’s history therefore includes earlier differentiation research as well as its later adoption in machine learning. (jmlr.org)
Limitations
Repeated multiplication of derivatives can produce the vanishing gradient problem or the exploding gradient problem. These effects are particularly important in recurrent neural networks, where dependencies extend across many time steps. Small gradients weaken learning signals for distant dependencies; large gradients can destabilize updates. Gradient-norm clipping limits excessively large gradients, but does not by itself resolve vanishing gradients. (proceedings.mlr.press)
Backpropagation also requires usable local derivative rules. At nondifferentiable points, software may apply specified conventions, such as selecting a subgradient; genuinely discrete operations can interrupt ordinary gradient propagation. Correct gradients describe the implemented computation, including any invalid arithmetic it contains: masking an undefined result afterward does not necessarily prevent undefined gradients during the backward pass. (docs.pytorch.org)