{
  "id": "gradient-checkpointing",
  "title": "Gradient Checkpointing",
  "lang": "en",
  "source_lang": "en",
  "summary": "Gradient checkpointing reduces memory use during neural-network training by storing selected intermediate activations and recomputing others when needed for backpropagation.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Activation checkpointing",
    "Activation recomputation",
    "Rematerialization"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/gradient-checkpointing/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/gradient-checkpointing/",
      "json": "https://aiwiki.page/zh/gradient-checkpointing.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Gradient checkpointing is a technique that reduces the memory required to train an artificial neural network by retaining only selected intermediate results from its forward computation and recomputing discarded results during backpropagation. It exchanges additional computation for lower memory consumption, making it possible to train larger models, use larger batches, or process longer inputs within a fixed memory budget. Despite its name, the values saved as checkpoints are usually activations or other intermediate tensors, not gradients."
    },
    {
      "heading": "Why training requires intermediate values",
      "text": "Training typically involves a forward pass that computes a model’s output and a loss function, followed by a backward pass that calculates derivatives with respect to its parameters. Backpropagation is an application of reverse-mode automatic differentiation: it traverses the computation in reverse, using intermediate values produced during the forward pass.\n\nFor a sequence of layers,\n\n\\[h_i=f_i(h_{i-1};\\theta_i), \\qquad i=1,\\ldots,L,\\]\n\nthe backward computation for layer \\(i\\) may need its input \\(h_{i-1}\\), its output \\(h_i\\), or other values created inside \\(f_i\\). Ordinary training therefore retains many such values until their backward computations have finished.\n\nIn deep learning, these retained activations can consume a substantial fraction of the memory on a graphics processing unit. Their size often grows with batch size, input resolution, sequence length, and network depth. This storage is separate from memory used for model parameters, parameter gradients, optimizer state, and temporary workspaces.\n\nGradient checkpointing changes the lifetime of intermediate values rather than the mathematical learning objective."
    },
    {
      "heading": "How checkpointing works",
      "text": "A checkpointed computation is divided into regions whose inputs, or other sufficient boundary values, are retained. Internal values that would ordinarily be saved for differentiation are discarded.\n\nDuring the backward pass:\n\n- The system reaches a checkpointed region.\n- It reruns the necessary part of that region’s forward computation using its saved boundary values.\n- It uses the reconstructed intermediate values to calculate gradients.\n- It releases those values when they are no longer needed.\n\nFor example, a network with twelve sequential blocks might retain the inputs to blocks 1, 5, and 9. When differentiating the final four-block region, it reconstructs that region’s intermediate activations from its saved input. Earlier regions are handled in reverse order.\n\nThe retained boundaries must contain everything needed to reproduce the region. In a general computational graph, this can mean several tensors rather than a single layer output. Branches, shared inputs, and residual connections make checkpoint placement more complicated than dividing a simple chain into equal pieces.\n\nCheckpointing is intended to preserve the gradients of the original computation, subject to ordinary numerical effects and correct handling of randomness and program state. It is not inherently an approximate-gradient method."
    },
    {
      "heading": "Memory–computation trade-off",
      "text": "A simple sequential model\n\nConsider \\(L\\) sequential layers, each requiring approximately \\(A\\) bytes of saved activation storage. Without checkpointing, activation storage scales approximately as\n\n\\[M_{\\text{ordinary}}=O(LA).\\]\n\nSuppose a checkpoint is stored every \\(k\\) layers. There are approximately \\(L/k\\) checkpoint boundaries, and reconstructing one segment requires working storage proportional to \\(k\\) layers. Under these simplified assumptions,\n\n\\[M_{\\text{checkpointed}} = O\\!\\left(A\\left(\\frac{L}{k}+k\\right)\\right).\\]\n\nBalancing the two terms gives \\(k\\approx\\sqrt{L}\\), yielding\n\n\\[M_{\\text{checkpointed}}=O(A\\sqrt{L}).\\]\n\nThis familiar square-root result concerns activation storage in an idealized chain, not total training memory. It assumes broadly comparable layer sizes and a particular segment-by-segment recomputation schedule. Parameters, optimizer state, large individual operations, and unequal activation sizes can substantially change practical results.\n\nComputational cost\n\nA simple segmented scheme may repeat approximately one forward pass’s worth of computation over checkpointed regions. If the original forward and backward costs are \\(C_f\\) and \\(C_b\\), respectively, a rough comparison is\n\n\\[C_{\\text{ordinary}}=C_f+C_b,\\]\n\n\\[C_{\\text{checkpointed}}\\approx 2C_f+C_b.\\]\n\nThis is an illustrative model, not a universal overhead estimate. Some implementations stop recomputation once the required intermediates have been reconstructed; nested schemes may recompute certain operations several times.\n\nWall-clock overhead also depends on memory traffic, kernel execution, communication, and hardware utilization. A slower training step can nevertheless enable a more efficient overall configuration—for example, by allowing a larger batch."
    },
    {
      "heading": "Checkpointing strategies",
      "text": "Uniform and block-level checkpointing\n\nThe simplest strategy checkpoints regularly spaced layers or architectural blocks. This is common in Transformers, where repeated blocks provide natural boundaries.\n\nUniform placement is straightforward but does not account for differences in tensor size or computational cost. A segment containing an especially large activation may determine peak memory even when all segments contain the same number of layers.\n\nSelective checkpointing\n\nSelective checkpointing retains some intermediate results while recomputing others within the same region. The decision can reflect an operation’s output size, execution cost, and dependencies.\n\nFor example, retaining an expensive matrix multiplication result while recomputing inexpensive elementwise operations may provide a better trade-off than recomputing the entire region. The best choice depends on tensor shapes, hardware, operation fusion, and the surrounding graph; operation names alone do not determine an optimal policy.\n\nNested and recursive checkpointing\n\nCheckpointed regions can contain smaller checkpointed regions. Such hierarchical schedules can reduce memory further, at the expense of repeated recomputation.\n\nThese methods connect checkpointing to graph pebbling: placing a pebble represents retaining a computed value, removing it represents freeing storage, and placing it again requires recomputation. The problem becomes one of scheduling computations under a storage constraint.\n\nAutomated placement\n\nAn algorithm or compiler can choose checkpoints using graph structure and estimated memory and execution costs. Depending on the formulation, approaches include dynamic programming, graph optimization, and heuristics.\n\nAutomated placement is difficult when shapes, branches, or execution paths vary at runtime. Accurate cost modeling is also challenging because device execution and memory allocation do not always match simple per-operation estimates."
    },
    {
      "heading": "Correctness requirements",
      "text": "Checkpointing relies on recomputation being consistent with the original forward execution.\n\nRandomness\n\nOperations such as dropout use random values. If recomputation uses a different random mask, the resulting backward computation generally does not correspond to the original forward pass.\n\nFrameworks may save and restore pseudorandom number generator state around checkpointed regions. Support can depend on which generators and devices are involved. Randomness outside the framework’s tracked state may require separate treatment.\n\nMutable state and side effects\n\nA checkpointed function may execute more than once. Consequently, side effects can also occur more than once.\n\nPotential issues include:\n\n- Updating running statistics, as in some uses of batch normalization.\n- Modifying global variables or counters.\n- Mutating inputs in place.\n- Writing files or recording metrics during the function.\n- Reading external state that changes between executions.\n\nConceptually, the safest checkpointed region behaves like a pure function of explicitly available inputs and controlled state. Stateful operations require implementation-specific handling.\n\nNumerical consistency\n\nEven when recomputation is mathematically equivalent, nondeterministic device operations or different execution choices can produce small floating-point differences. Checkpointing is therefore not a general guarantee of bitwise-identical training trajectories.\n\nA more serious problem occurs when recomputation follows different control flow or uses changed state. Such a mismatch can produce incorrect gradients rather than merely small numerical variation."
    },
    {
      "heading": "Implementation in machine-learning frameworks",
      "text": "PyTorch provides activation-checkpointing utilities through torch.utils.checkpoint. Its reentrant and non-reentrant implementations differ in how they interact with automatic differentiation, which backward interfaces they support, and how much computation they rerun. These distinctions matter for nested data structures, detached tensors, and more advanced differentiation patterns.\n\nTensorFlow provides recomputation functionality through tf.recompute_grad. Other automatic-differentiation systems offer related rematerialization transformations or compiler-based mechanisms.\n\nA framework implementation must do more than rerun a function. It must preserve the dependencies needed for differentiation, manage saved tensors and randomness, and interact correctly with execution and compilation mechanisms. The actual memory savings depend on which values the framework would otherwise retain and which values remain live outside the checkpointed region."
    },
    {
      "heading": "Applications and complementary techniques",
      "text": "Gradient checkpointing is widely used in training large language models, image models, and other networks with substantial activation memory requirements. In language models, it can help accommodate longer sequences or larger batches; in image models, it can support higher-resolution inputs.\n\nIt also applies during fine-tuning. Freezing most parameters does not necessarily eliminate activation-memory requirements: gradients may still need to propagate through frozen operations to reach trainable components.\n\nCheckpointing can be combined with:\n\n- Lower-precision computation, which reduces the storage required by many tensors.\n- Parameter and optimizer-state sharding, which distributes other major memory costs across devices.\n- Activation offloading, which moves saved activations to another memory tier instead of recomputing them.\n- Gradient accumulation, which processes a larger effective batch as multiple smaller microbatches.\n- Memory-efficient attention, including FlashAttention, which avoids materializing some large attention intermediates.\n\nThese techniques address different sources of memory use. Their benefits are not necessarily additive: reducing one bottleneck can expose another, and some memory-efficient operators already perform internal recomputation.\n\nIn distributed computing, checkpoint boundaries also interact with communication. Recomputing a region may trigger additional communication or affect overlap between computation and data transfer, so a placement that is effective on one device may behave differently across many devices."
    },
    {
      "heading": "Limitations and terminology",
      "text": "Checkpointing is most useful when retained activations are a major memory bottleneck. It cannot by itself eliminate memory dominated by parameters, optimizer state, or an indivisible operation’s temporary workspace. It also adds computational work, so its value depends on the relative costs of memory capacity, execution time, and energy.\n\nThe term should be distinguished from training-state checkpointing, which saves model and optimizer state to persistent storage so that training can resume after interruption. Gradient checkpointing normally concerns temporary in-memory values within a training step; it does not provide recovery from a process or hardware failure.\n\nIt is also distinct from gradient accumulation. Accumulation combines gradients from multiple microbatches before an optimizer update, whereas checkpointing changes how intermediate values are stored and reconstructed within a computation."
    },
    {
      "heading": "Historical background",
      "text": "Recomputing intermediate values to reduce storage predates modern deep learning and is a longstanding technique in reverse-mode automatic differentiation and adjoint computation. Andreas Griewank and Andrea Walther’s Revolve, published in 2000, established influential scheduling methods for checkpointing sequential computations under a limited checkpoint budget.\n\nThe 2016 paper Training Deep Nets with Sublinear Memory Cost by Tianqi Chen and collaborators helped popularize checkpointing in deep learning, including the square-root-memory construction for sequential networks. Subsequent work extended the approach toward irregular graphs, selective recomputation, automated scheduling, and integration with large-scale training systems."
    }
  ],
  "links": [
    "artificial-neural-network",
    "backpropagation",
    "loss-function",
    "automatic-differentiation",
    "deep-learning",
    "graphics-processing-unit",
    "computational-graph",
    "residual-connection",
    "transformer-architecture",
    "matrix",
    "algorithm",
    "compiler",
    "dynamic-programming",
    "dropout",
    "pseudorandom-number-generator",
    "batch-normalization",
    "pytorch",
    "tensorflow",
    "large-language-model",
    "fine-tuning",
    "flashattention",
    "distributed-computing"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}