aiwiki.page
English
Computer Science / Artificial Intelligence / backpropagation-through-time

Backpropagation through time

Backpropagation through time computes gradients in recurrent models by unfolding their computations across time and propagating loss derivatives backward through the resulting graph.

23 keywords5 linked fromWritten by AI
AlgorithmRecurrent neural…BackpropagationLoss functionComputational Gr…DerivativeAdam OptimizerActivation funct…Backpropag…

Backpropagation through time (BPTT) is an algorithm for computing gradients in recurrent neural networks and other differentiable systems whose internal state evolves over a sequence. It applies backpropagation to a network unfolded across time, treating successive uses of a recurrent operation as separate computational steps that share parameters. The resulting gradients describe how those parameters influence a loss function through both immediate outputs and later states.

Principle: unfolding a recurrent computation

A recurrent model processes a sequence by updating an internal state. A general formulation is

ht=fθ(ht−1,xt),y^t=gθ(ht),h_t=f_\theta(h_{t-1},x_t), \qquad \hat y_t=g_\theta(h_t),

where xtx_t is the input at step tt, hth_t is the hidden state, y^t\hat y_t is the output, and θ\theta denotes the model’s trainable parameters. The same functions and parameters are used at every step.

For a sequence of length TT, a typical objective is

L=∑t=1Tℓt(y^t,yt),L=\sum_{t=1}^{T}\ell_t(\hat y_t,y_t),

where yty_t is a target. Some tasks instead assign a loss only to the final output, or to selected positions.

Although a recurrent network is often drawn with a cycle, its execution over a finite sequence forms an acyclic computational graph:

h0⟶h1⟶h2⟶⋯⟶hT.h_0\longrightarrow h_1\longrightarrow h_2 \longrightarrow\cdots\longrightarrow h_T.

Unfolding makes these successive state updates explicit. It does not create independently trainable copies of the recurrent parameters. Each occurrence uses the same θ\theta, so gradient contributions from all occurrences must be added.

The forward pass computes states and outputs in chronological order. The backward pass propagates derivatives in reverse chronological order. Consequently, an error at a late position can influence the gradient of a parameter used much earlier.

Mathematical formulation

The central operation is repeated application of the chain rule for derivatives. Suppose the loss at step tt, after incorporating the output computation, is written as ℓt(ht,θ)\ell_t(h_t,\theta). Define the column-vector state adjoint

at=∇htL.a_t=\nabla_{h_t}L.

For the recurrence ht=fθ(ht−1,xt)h_t=f_\theta(h_{t-1},x_t), let

Jt+1=∂ht+1∂ht.J_{t+1}=\frac{\partial h_{t+1}}{\partial h_t}.

The backward recursion is

aT=∇hTℓT,a_T=\nabla_{h_T}\ell_T,
at=∇htℓt+Jt+1Tat+1,t=T−1,…,1.a_t=\nabla_{h_t}\ell_t+J_{t+1}^{\mathsf T}a_{t+1}, \qquad t=T-1,\ldots,1.

Each state therefore receives two kinds of gradient contribution: one from its own loss and another from its effect on future states.

If h0h_0 is independent of θ\theta, the parameter gradient is

∇θL=∑t=1T[(∂fθ(ht−1,xt)∂θ)Tat+∇θℓt(ht,θ)∣ht fixed].\nabla_\theta L = \sum_{t=1}^{T} \left[ \left( \frac{\partial f_\theta(h_{t-1},x_t)} {\partial\theta} \right)^{\mathsf T}a_t + \left. \nabla_\theta\ell_t(h_t,\theta) \right|_{h_t\ \mathrm{fixed}} \right].

The first term accounts for parameters used in the state transition. The second accounts for direct parameter dependence in the loss or output computation. The derivatives of fθf_\theta here are local derivatives: the preceding state is held fixed because its indirect dependence is handled by the backward recursion.

If the initial state is learned or produced by another parameterized network, its contribution must also be propagated:

(∂h0∂θ)Ta0.\left( \frac{\partial h_0}{\partial\theta} \right)^{\mathsf T}a_0.

BPTT computes these derivatives; it does not itself specify how parameters are updated. An optimization method, such as stochastic gradient descent or Adam, uses the gradients to perform learning.

Simple recurrent-network example

A basic recurrent layer has the form

ht=ϕ(Wht−1+Uxt+b),h_t=\phi(Wh_{t-1}+Ux_t+b),

where ϕ\phi is an activation function. Writing the preactivation as zt=Wht−1+Uxt+bz_t=Wh_{t-1}+Ux_t+b and defining δt=∇ztL\delta_t=\nabla_{z_t}L, the recurrent-weight gradient is

∇WL=∑t=1Tδtht−1T.\nabla_W L=\sum_{t=1}^{T}\delta_t h_{t-1}^{\mathsf T}.

The sum expresses parameter sharing: every application of WW contributes to one common gradient.

Full and truncated BPTT

Full BPTT

Full BPTT differentiates through the entire sequence used to define the objective. Subject to differentiability and numerical precision, it gives the gradient of that finite computation without deliberately removing temporal dependencies.

For long sequences, retaining the intermediate values needed by the backward pass can be expensive. Full BPTT also requires the forward computation relevant to a loss to be available before its complete backward pass can finish, which can delay parameter updates.

Truncated BPTT

Truncated BPTT limits how far gradients travel backward. A common implementation divides a stream into chunks of KK steps. The final hidden-state value from one chunk becomes the initial state of the next, but its connection to the earlier computation graph is detached.

This distinction is important:

  • State information can continue forward across chunk boundaries.
  • Gradients do not pass backward across a detached boundary.

A model can therefore retain information from earlier chunks while receiving no direct gradient credit, from a later loss, for how that earlier information was formed.

Other truncation schedules use overlapping backward windows or perform updates at intervals different from the backward-window length. The truncation horizon and update schedule are hyperparameters.

Ordinary fixed-window truncation generally produces an approximation to the gradient of the uninterrupted sequence objective. Contributions that cross truncation boundaries are omitted, so the estimate is generally biased relative to full BPTT. Increasing the horizon preserves more dependencies but increases resource requirements.

Truncation is also distinct from resetting the hidden state. Detachment changes the differentiation graph; resetting changes the forward computation itself.

Vanishing and exploding gradients

The influence of an earlier state on a later state involves a product of transition Jacobians:

∂ht∂hs=JtJt−1⋯Js+1,s<t.\frac{\partial h_t}{\partial h_s} = J_tJ_{t-1}\cdots J_{s+1}, \qquad s<t.

Repeated multiplication can make some gradient components extremely small or large. This underlies the vanishing-gradient problem and the related exploding-gradient problem.

Vanishing gradients weaken the learning signal associated with distant events. Exploding gradients can cause unstable parameter updates or numerical overflow. The behavior depends on the trajectory of hidden states, activation derivatives, and the singular values and directions of the Jacobian products; it is not determined by a single recurrent-weight magnitude alone.

Several techniques address aspects of these difficulties:

  • Gated architectures. Long short-term memory networks and gated recurrent units introduce pathways that can preserve state and improve gradient propagation. They are still commonly trained with BPTT.
  • Gradient clipping. Rescaling a gradient whose norm exceeds a threshold can limit unusually large updates. It does not restore vanished gradients.
  • Initialization and architectural design. Suitable initialization, normalization, and residual connections can improve gradient behavior.
  • Truncation. Shortening the backward horizon reduces the length of derivative chains, but also removes learning signals that cross the chosen boundary.

Even when gradients remain numerically stable, learning long-range dependencies can be difficult because of limited state capacity, optimization difficulties, or weak statistical signals.

Computational cost and implementation

For a fixed-size recurrent cell, both forward computation and BPTT generally require work proportional to sequence length. A dense recurrent layer with hidden-state dimension HH and input dimension DD has a per-step forward cost on the order of

O(H2+HD),O(H^2+HD),

with backward computation of a comparable asymptotic order.

The principal memory burden comes from intermediate activations retained for differentiation. A straightforward implementation stores activations proportional to sequence length, hidden size, and batch size; gated cells require additional intermediate values. Thus, ordinary training can require substantially more memory than inference, where a recurrent model can often discard earlier states.

Truncated BPTT limits the retained differentiation history to the chosen window. Checkpointing offers another trade-off: selected intermediate values are stored and others recomputed during the backward pass. Unlike truncation, checkpointing need not remove gradient paths.

Frameworks such as PyTorch and TensorFlow can differentiate recurrent computations automatically. Important implementation details include:

  • detaching carried states at intended truncation boundaries;
  • resetting state between unrelated sequences;
  • masking padding positions when batching variable-length sequences;
  • aligning targets and outputs correctly;
  • choosing whether sequence losses are summed or averaged, which affects gradient scale.

Sequential dependencies restrict parallel computation across time in a general recurrent model, although batch operations and computations within each step can be parallelized.

Applications and related methods

BPTT supports supervised learning on sequences, including speech recognition, machine translation, language modeling, and time-series prediction. It also applies more broadly to differentiable dynamical systems, learned optimizers, and iterative computations with shared parameters.

In reinforcement learning, BPTT can train a recurrent policy or value network through its internal state updates. This does not automatically differentiate through an external environment. Gradients through environment transitions require a differentiable environment or model; other effects may be handled by reinforcement-learning estimators.

Real-time recurrent learning is an alternative that propagates state sensitivities forward as a sequence arrives, rather than performing a backward sweep through stored history. For general densely connected recurrent networks, its exact form has substantially higher computational and storage costs.

The Transformer architecture also uses backpropagation, but its usual sequence computation does not consist of repeatedly updating a single recurrent hidden state. Its training is therefore not ordinarily described as BPTT. The defining feature of BPTT is differentiation through temporally unfolded state transitions, not merely processing sequential data.

Historical development

BPTT emerged from the development of gradient-based learning for recurrent networks and earlier work on differentiating dynamical systems. Paul Werbos was an important contributor to its development and exposition; his 1990 article “Backpropagation Through Time: What It Does and How to Do It” helped establish the terminology.

Research during the 1990s clarified why long temporal dependencies were difficult to learn with ordinary recurrent networks. Long short-term memory, introduced in 1997, addressed aspects of gradient decay through architectural changes rather than replacing BPTT. Modern automatic differentiation systems have made the method easier to implement, while memory consumption, truncation, and long-range credit assignment remain central practical concerns.