{
  "id": "weight-decay",
  "title": "Weight Decay",
  "lang": "en",
  "source_lang": "en",
  "summary": "Weight decay regularizes machine-learning models by shrinking selected parameters during training, with behavior that depends on the optimizer and its implementation.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Parameter decay"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/weight-decay/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/weight-decay/",
      "json": "https://aiwiki.page/zh/weight-decay.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Weight decay is a regularization technique in machine learning that shrinks selected model parameters toward zero during training. It is widely used in artificial neural networks to influence the solutions found by optimization and to help reduce overfitting. With ordinary gradient descent, multiplicative weight decay is equivalent to adding a squared-parameter penalty to the training objective. With adaptive optimizers, however, these operations generally differ. This distinction is central to decoupled weight decay and the AdamW optimizer."
    },
    {
      "heading": "Mathematical definition",
      "text": "Let \\(\\theta_t\\) denote a vector of trainable parameters at step \\(t\\), \\(L(\\theta)\\) the data-dependent loss function, and \\(g_t\\) its gradient, possibly estimated from a minibatch. A common weight-decay update is\n\n\\[\\theta_{t+1} = (1-\\eta_t\\lambda)\\theta_t-\\eta_t g_t,\\]\n\nwhere \\(\\eta_t\\) is the learning rate and \\(\\lambda\\geq 0\\) is the decay coefficient.\n\nThe factor \\(1-\\eta_t\\lambda\\) contracts the existing parameter vector independently of the loss gradient. Under the usual condition \\(0\\leq\\eta_t\\lambda<1\\), parameters would shrink toward zero if the data gradient were absent. During actual training, the gradient can oppose that contraction, so individual weights and the overall parameter norm need not decrease at every step.\n\nTerminology and coefficient conventions vary. Some formulations use a direct per-step shrinkage factor, such as \\(1-\\delta_t\\), rather than expressing it as \\(1-\\eta_t\\lambda\\). Consequently, a reported “weight decay” value is not fully specified without the update rule."
    },
    {
      "heading": "Relationship to L2 regularization",
      "text": "L2 regularization modifies the optimization objective by adding a squared Euclidean norm penalty:\n\n\\[J(\\theta) = L(\\theta)+\\frac{\\lambda}{2}\\|\\theta\\|_2^2.\\]\n\nIts gradient is\n\n\\[\\nabla J(\\theta)=\\nabla L(\\theta)+\\lambda\\theta.\\]\n\nApplying ordinary gradient descent gives\n\n\\[\\theta_{t+1} = \\theta_t-\\eta_t\\bigl(g_t+\\lambda\\theta_t\\bigr) = (1-\\eta_t\\lambda)\\theta_t-\\eta_t g_t.\\]\n\nThus, L2 regularization and multiplicative weight decay are algebraically equivalent for plain gradient descent, including stochastic gradient descent without momentum. If the penalty is written as \\(\\lambda\\|\\theta\\|_2^2\\) without the factor \\(1/2\\), its gradient instead contributes \\(2\\lambda\\theta\\).\n\nThis equivalence does not hold for arbitrary optimization algorithms. In particular:\n\n- Momentum: adding \\(\\lambda\\theta_t\\) to the gradient causes the penalty contribution to accumulate in the momentum state. Direct parameter shrinkage applied separately from momentum generally produces a different trajectory.\n- Adaptive learning rates: transforming the combined gradient also transforms the penalty contribution, making its effect depend on the optimizer’s coordinate-wise scaling and gradient history.\n\nIn linear least-squares models, the corresponding squared-coefficient penalty is the basis of ridge regression. The equivalence concerns the objective and its optimization; it does not imply that every optimizer option named “weight decay” implements the same procedure."
    },
    {
      "heading": "Coupled and decoupled weight decay",
      "text": "Coupled regularization in adaptive optimizers\n\nThe Adam optimizer maintains moving averages of gradients and squared gradients. In a coupled L2 implementation, the optimizer receives\n\n\\[\\widetilde g_t=g_t+\\lambda\\theta_t.\\]\n\nBoth moment estimates are then calculated using \\(\\widetilde g_t\\). The regularization contribution therefore affects the optimizer’s internal state and is itself affected by adaptive scaling.\n\nA simpler illustration uses a preconditioning matrix \\(P_t\\):\n\n\\[\\theta_{t+1} = \\theta_t-\\eta_tP_t(g_t+\\lambda\\theta_t).\\]\n\nHere, the penalty produces the term \\(-\\eta_t\\lambda P_t\\theta_t\\), rather than uniform multiplicative shrinkage. In Adam, the interaction is more involved because the moment estimates also incorporate the penalty.\n\nDecoupled weight decay and AdamW\n\nDecoupled weight decay separates parameter shrinkage from the gradient transformation. Schematically,\n\n\\[\\theta_{t+1} = (1-\\eta_t\\lambda)\\theta_t-\\eta_tU_t(g_1,\\ldots,g_t),\\]\n\nwhere \\(U_t\\) is the optimizer’s update direction computed from data gradients, without adding the decay term.\n\nAdamW applies this principle to Adam. A standard update is\n\n\\[\\theta_{t+1} = (1-\\eta_t\\lambda)\\theta_t - \\eta_t \\frac{\\widehat m_t}{\\sqrt{\\widehat v_t}+\\epsilon},\\]\n\nwith elementwise division, where \\(\\widehat m_t\\) and \\(\\widehat v_t\\) are bias-corrected moment estimates and \\(\\epsilon\\) is a numerical-stability constant.\n\nThe influential work introducing AdamW was first released by Ilya Loshchilov and Frank Hutter in 2017 and published at ICLR in 2019 under the title Decoupled Weight Decay Regularization. It emphasized that L2 regularization and weight decay should not be treated as interchangeable for adaptive optimizers.\n\nDecoupling makes the shrinkage operation independent of Adam’s gradient preconditioning. It does not eliminate all interaction with the learning rate: in the update above, the per-step contraction still depends on \\(\\eta_t\\lambda\\). Nor is AdamW generally equivalent to running Adam on the original loss plus a fixed L2 penalty."
    },
    {
      "heading": "Effects on learning and generalization",
      "text": "Weight decay discourages large parameter magnitudes and changes the model’s inductive bias. In suitable settings, this can improve generalization by reducing the tendency to fit incidental variation in the training data. Excessive decay can instead cause underfitting.\n\nIts effect is especially transparent in ridge regression. For the objective\n\n\\[\\frac12\\|X\\theta-y\\|_2^2+\\frac{\\lambda}{2}\\|\\theta\\|_2^2,\\]\n\nthe solution for \\(\\lambda>0\\) is\n\n\\[\\widehat\\theta = (X^\\top X+\\lambda I)^{-1}X^\\top y,\\]\n\nwhere \\(X\\) is the design matrix and \\(y\\) the target vector. The penalty shrinks coefficients and improves the conditioning of the linear system.\n\nIn deep learning, the relationship between weight magnitude and model complexity is less direct. Networks can admit different parameterizations of the same function. For example, with a positively homogeneous activation function such as ReLU, scaling one layer’s weights up and a following layer’s weights down can leave the represented function unchanged.\n\nBatch normalization introduces further scale invariances, exact in idealized formulations and approximate in practical implementations. For scale-invariant parameters, weight decay can affect optimization by changing parameter norms and hence the effective size of directional updates, rather than simply reducing the expressiveness of the represented function. Smaller weight norms therefore do not universally imply a simpler predictor."
    },
    {
      "heading": "Parameter selection and training configuration",
      "text": "Weight decay is commonly applied to weight matrices and convolutional kernels. Biases and the learned scale and offset parameters of normalization layers are often excluded, particularly in Transformer training. These exclusions are conventions rather than mathematical requirements.\n\nPractical behavior depends on several choices:\n\n- Decay coefficient: \\(\\lambda\\) is a hyperparameter, commonly selected using a validation set.\n- Learning-rate schedule: when shrinkage is learning-rate-scaled, lowering the learning rate also lowers decay per step.\n- Number of updates: additional optimizer steps create additional opportunities for shrinkage.\n- Parameter groups: different parameter subsets may have different coefficients, including zero.\n- Loss reduction: summing rather than averaging example losses changes gradient scale and the numerical interpretation of regularization settings.\n\nIgnoring data gradients, the accumulated contraction after \\(T\\) steps is\n\n\\[\\theta_T = \\left[\\prod_{t=0}^{T-1}(1-\\eta_t\\lambda)\\right]\\theta_0.\\]\n\nWhen every \\(\\eta_t\\lambda\\) is small,\n\n\\[\\prod_{t=0}^{T-1}(1-\\eta_t\\lambda) \\approx \\exp\\left(-\\lambda\\sum_{t=0}^{T-1}\\eta_t\\right).\\]\n\nThis explains why the same coefficient can have different effects under different training lengths, batch sizes, or learning-rate schedules. Gradient accumulation also matters because decay is normally applied at optimizer updates, not at every gradient-accumulation step.\n\nLibraries such as PyTorch and TensorFlow provide optimizer-level mechanisms for weight decay. The optimizer’s documented update equation, parameter exclusions, and treatment of parameters without gradients determine the actual behavior; the option name alone is insufficient."
    },
    {
      "heading": "Limitations and related methods",
      "text": "Weight decay usually shrinks parameters continuously rather than setting them exactly to zero. It is therefore not ordinarily a method for explicit feature selection or model pruning. L1 penalties and sparsity-inducing procedures have different mathematical effects.\n\nIt can be combined with dropout, data augmentation, and early stopping, but these methods act differently: dropout perturbs training computation, augmentation changes the training examples presented to the model, and early stopping limits optimization duration.\n\nWeight decay also depends on parameterization and input scaling. Penalizing all coordinates equally does not necessarily penalize their influence on predictions equally, which is particularly clear in linear models whose features use different units. It provides no general guarantee of robustness, calibrated uncertainty, or successful performance under distribution shift."
    }
  ],
  "links": [
    "regularization",
    "machine-learning",
    "artificial-neural-network",
    "overfitting",
    "loss-function",
    "learning-rate",
    "stochastic-gradient-descent",
    "ridge-regression",
    "adam-optimizer",
    "adamw",
    "inductive-bias",
    "generalization",
    "training-data",
    "underfitting",
    "deep-learning",
    "activation-function",
    "batch-normalization",
    "transformer-architecture",
    "hyperparameter",
    "validation-set",
    "pytorch",
    "tensorflow",
    "dropout",
    "data-augmentation",
    "early-stopping",
    "distribution-shift"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}