{
  "id": "learning-rate-schedule",
  "title": "Learning rate schedule",
  "lang": "en",
  "source_lang": "en",
  "summary": "A learning rate schedule controls how an optimizer’s step size changes during training, influencing convergence speed, stability, and final model performance.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Learning rate scheduling",
    "Learning rate scheduling policy",
    "Step-size schedule"
  ],
  "generated_at": "2026-11-06",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/learning-rate-schedule/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/learning-rate-schedule/",
      "json": "https://aiwiki.page/zh/learning-rate-schedule.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "A learning rate schedule is a rule for changing the learning rate during iterative optimization, especially when training machine learning models. Rather than using one fixed step size throughout training, a schedule varies it according to training progress, observed performance, or both. Schedules can decrease the learning rate, increase it temporarily, or alternate between larger and smaller values. They influence how quickly optimization progresses, how stable the updates are, and the quality of the resulting model."
    },
    {
      "heading": "Role in optimization",
      "text": "For stochastic gradient descent (SGD), a basic parameter update is\n\n\\[\\theta_{t+1}=\\theta_t-\\eta_t g_t,\\]\n\nwhere \\(\\theta_t\\) denotes model parameters at update \\(t\\), \\(g_t\\) is a gradient estimate computed from a batch of training data, and \\(\\eta_t\\) is the learning rate. A schedule specifies how \\(\\eta_t\\) changes.\n\nLarger learning rates permit larger parameter changes and can accelerate early progress. Excessively large rates can cause oscillation, unstable updates, or divergence. Smaller rates allow finer adjustments but may make training slow or leave it making little progress within a limited computational budget.\n\nIn deep learning, the loss function is generally nonconvex, and batch gradients introduce noise. Changing the learning rate therefore affects both movement across the loss landscape and the scale of stochastic fluctuations. These effects can influence generalization, not just the final training loss, although no schedule guarantees better generalization across all models and datasets.\n\nA schedule is usually a collection of hyperparameters rather than a single setting. Relevant choices include the initial or peak rate, minimum rate, decay timescale, warmup length, and total number of updates."
    },
    {
      "heading": "Measuring training progress",
      "text": "Schedules require a definition of progress. Common units include:\n\n- Optimizer updates: the number of actual parameter updates.\n- Epochs: passes through the training dataset.\n- Examples or tokens processed: useful when comparing runs with different batch sizes or sequence lengths.\n- Evaluation events: checkpoints at which a performance metric is measured.\n\nThese units are not interchangeable. Increasing batch size reduces the number of updates per epoch, so a schedule defined in epochs behaves differently from one defined in updates. With gradient accumulation, several batches may contribute to one optimizer update; advancing the schedule after every batch can make it progress faster than intended.\n\nA schedule may be predetermined, depending only on progress, or feedback-based, depending on observed quantities such as validation loss."
    },
    {
      "heading": "Main schedule families",
      "text": "Constant and piecewise-constant schedules\n\nA constant learning rate,\n\n\\[\\eta_t=\\eta_0,\\]\n\nprovides a baseline. It is not a changing schedule in the strict sense, but scheduling systems commonly support it, and it can form one phase of a composite schedule.\n\nStep decay holds the rate constant over intervals and reduces it at specified boundaries. For equal-length intervals,\n\n\\[\\eta_t=\\eta_0\\gamma^{\\lfloor t/K\\rfloor},\\]\n\nwhere \\(K\\) is the interval length and \\(0<\\gamma<1\\) is the decay factor.\n\nMultistep decay instead uses an explicit list of milestones. These schedules are simple and interpretable, but their abrupt changes and milestone choices can materially affect training.\n\nExponential and inverse-time decay\n\nAn exponential schedule decreases the learning rate smoothly:\n\n\\[\\eta_t=\\eta_0 e^{-kt},\\]\n\nwhere \\(k>0\\). Another common notation is \\(\\eta_t=\\eta_0\\gamma^{t/K}\\), expressing a multiplicative reduction over a chosen interval.\n\nAn inverse-time schedule has the form\n\n\\[\\eta_t=\\frac{\\eta_0}{1+kt}.\\]\n\nA broader family is\n\n\\[\\eta_t=\\frac{\\eta_0}{(1+kt)^p},\\]\n\nwhere \\(p>0\\) controls how quickly the rate decreases. Unlike fixed-horizon schedules, these expressions do not require a predetermined stopping point.\n\nLinear and polynomial decay\n\nFor a planned horizon of \\(T\\) updates, polynomial decay can be written as\n\n\\[\\eta_t=\\eta_{\\min} +(\\eta_{\\max}-\\eta_{\\min}) \\left(1-\\frac{t}{T}\\right)^p, \\qquad 0\\leq t\\leq T.\\]\n\nThe case \\(p=1\\) is linear decay. Larger values of \\(p\\) produce a faster initial reduction and a flatter approach to the minimum. Implementations must define what happens after \\(T\\), often by holding the learning rate at \\(\\eta_{\\min}\\).\n\nBecause the curve depends on the training horizon, extending a run after its learning rate has nearly vanished is not equivalent to having planned a longer run from the start.\n\nCosine annealing\n\nCosine decay smoothly transitions from a maximum to a minimum:\n\n\\[\\eta_t=\\eta_{\\min} +\\frac{\\eta_{\\max}-\\eta_{\\min}}{2} \\left(1+\\cos\\left(\\frac{\\pi t}{T}\\right)\\right), \\qquad 0\\leq t\\leq T.\\]\n\nIt changes slowly near the beginning and end of the interval, with a steeper decline in the middle.\n\nCosine annealing with warm restarts repeats such curves. At each restart, the rate rises again, often to a prescribed maximum; cycle lengths may remain fixed or increase. “Restart” generally refers to the learning rate, not to reinitializing model parameters. Whether optimizer state is also reset is a separate design choice.\n\nCyclical and one-cycle schedules\n\nA cyclical schedule repeatedly raises and lowers the learning rate between bounds. Triangular cycles use approximately linear increases and decreases; other policies use smooth curves or progressively smaller peaks.\n\nA one-cycle policy raises the rate to a maximum and then lowers it, typically ending at a rate substantially below the initial value. Some implementations also vary momentum inversely with the learning rate. Such policies can support rapid training in suitable settings, but their behavior depends on the peak rate, optimizer, batch size, and other forms of regularization."
    },
    {
      "heading": "Warmup and composite schedules",
      "text": "Learning rate warmup begins training at a small learning rate and gradually raises it to a target value. Linear warmup over \\(W\\) updates can be expressed as\n\n\\[\\eta_t=\\eta_{\\mathrm{start}} +(\\eta_{\\mathrm{peak}}-\\eta_{\\mathrm{start}}) \\frac{t}{W}, \\qquad 0\\leq t\\leq W.\\]\n\nWarmup is often followed by linear, cosine, polynomial, or inverse-square-root decay. It is therefore usually a phase within a larger schedule, not a complete policy by itself.\n\nWarmup can reduce instability while parameter scales, activations, and optimizer statistics are changing rapidly near initialization. It is common in large-batch training and in training Transformers, but it is not universally necessary, and its effects depend on the model and optimizer.\n\nA widely used Transformer schedule combines warmup with inverse-square-root decay:\n\n\\[\\eta_t=c\\min\\left(t^{-1/2},\\,tW^{-3/2}\\right), \\qquad t\\geq1,\\]\n\nwhere \\(c\\) is a scaling constant and \\(W\\) is the warmup length. The rate increases linearly until \\(t=W\\), then decreases in proportion to \\(t^{-1/2}\\).\n\nOther composite policies include warmup followed by a constant-rate phase and a final decay phase. Such designs separate initial stabilization, sustained optimization, and late-stage refinement."
    },
    {
      "heading": "Performance-based scheduling",
      "text": "A reduce-on-plateau schedule lowers the learning rate when a monitored metric stops improving sufficiently. Typical parameters include:\n\n- a multiplicative reduction factor;\n- a minimum improvement threshold;\n- a patience period;\n- a cooldown period after a reduction;\n- a lower bound on the learning rate.\n\nThe monitored quantity may be training loss or a metric measured on a validation set. Validation-based scheduling makes the training procedure responsive to held-out performance, but it also makes the validation data part of model selection. A test set used for final assessment should not control the schedule.\n\nPlateau detection is sensitive to measurement noise and evaluation frequency. A temporary fluctuation can trigger a reduction prematurely, while an overly long patience period can delay useful changes. Learning rate reduction is distinct from early stopping: the former changes subsequent updates, whereas the latter ends training."
    },
    {
      "heading": "Relationship to adaptive optimizers",
      "text": "Learning rate scheduling and adaptive optimization operate at different levels.\n\nAn adaptive optimizer such as Adam changes parameter-wise update scales using gradient statistics. A scheduler changes the base learning rate supplied to the optimizer. Adam can therefore use warmup, cosine decay, or another schedule even though its individual parameter updates are already adaptive.\n\nThe actual update also depends on momentum, gradient normalization, numerical stabilizers, gradient clipping, and weight decay. Consequently, identical nominal schedules can produce very different parameter trajectories under different optimizers.\n\nFor optimizers with decoupled weight decay, the shrinkage applied per update commonly depends on the learning rate. Changing a schedule can therefore change cumulative regularization as well as optimization progress."
    },
    {
      "heading": "Theoretical background and historical development",
      "text": "Diminishing step sizes have a long history in stochastic approximation. Under appropriate assumptions, classical convergence results use conditions such as\n\n\\[\\sum_{t=1}^{\\infty}\\eta_t=\\infty, \\qquad \\sum_{t=1}^{\\infty}\\eta_t^2<\\infty.\\]\n\nThe first condition prevents the total available movement from becoming finite too quickly; the second limits the accumulated effect of stochastic noise. Schedules proportional to \\(t^{-p}\\), with \\(1/2<p\\leq1\\), satisfy these summability conditions.\n\nThese conditions are not universal requirements for successful machine learning. Their conclusions depend on assumptions about the objective, gradient noise, and optimization procedure. They do not directly establish the effectiveness of finite-horizon warmup, cosine, or cyclical schedules in large neural networks.\n\nPractical neural-network training developed along several lines: manually chosen step reductions, smooth decay, metric-triggered reductions, and later cyclical policies and cosine warm restarts. Warmup followed by decay became prominent in Transformer training. These approaches reflect different objectives—long-run convergence, finite-budget training, initial stability, and repeated exploration—rather than a single progression toward one universally best schedule."
    },
    {
      "heading": "Implementation and limitations",
      "text": "Libraries such as PyTorch and TensorFlow provide scheduling mechanisms, but their conventions differ. Important implementation details include whether the scheduler advances before or after an optimizer update, whether its counter begins at zero or one, and whether it expects updates or epochs. These choices can shift the curve or alter its initial value.\n\nReproducible training requires preserving scheduler state as well as model and optimizer state. Restarting a saved model with a fresh schedule can repeat warmup or apply an incorrect decay phase. In distributed training, participating workers must use consistent learning rates and progress counters.\n\nSchedules also interact with:\n\n- Training budget: a policy designed for a short run may decay too early in a longer one.\n- Batch size: changing batch size alters both gradient noise and update counts.\n- Fine-tuning: fine-tuning may use lower rates or different rates for different parameter groups.\n- Data and objective changes: a plateau can reflect changing data difficulty rather than proximity to an optimum.\n- Model selection: a lower training loss does not necessarily imply better validation performance.\n\nNo schedule is uniformly optimal across tasks. Meaningful comparisons account for optimizer settings, batch size, training budget, evaluation procedures, and the extent of hyperparameter optimization, rather than attributing performance differences to the schedule alone."
    }
  ],
  "links": [
    "learning-rate",
    "mathematical-optimization",
    "machine-learning",
    "stochastic-gradient-descent",
    "training-data",
    "deep-learning",
    "loss-function",
    "generalization",
    "hyperparameter",
    "regularization",
    "transformer-architecture",
    "validation-set",
    "test-set",
    "early-stopping",
    "adam-optimizer",
    "pytorch",
    "tensorflow",
    "distributed-computing",
    "fine-tuning",
    "hyperparameter-optimization"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}