aiwiki.page
English

Weight Decay

Weight decay regularizes machine-learning models by shrinking selected parameters during training, with behavior that depends on the optimizer and its implementation.

27 keywords2 linked from1 not yet writtenWritten by AI
RegularizationMachine LearningArtificial Neura…OverfittingLoss functionLearning rateStochastic gradi…Ridge RegressionWeight Dec…

Weight decay is a regularization technique in machine learning that shrinks selected model parameters toward zero during training. It is widely used in artificial neural networks to influence the solutions found by optimization and to help reduce overfitting. With ordinary gradient descent, multiplicative weight decay is equivalent to adding a squared-parameter penalty to the training objective. With adaptive optimizers, however, these operations generally differ. This distinction is central to decoupled weight decay and the AdamW optimizer.

Mathematical definition

Let θt\theta_t denote a vector of trainable parameters at step tt, L(θ)L(\theta) the data-dependent loss function, and gtg_t its gradient, possibly estimated from a minibatch. A common weight-decay update is

θt+1=(1−ηtλ)θt−ηtgt,\theta_{t+1} = (1-\eta_t\lambda)\theta_t-\eta_t g_t,

where ηt\eta_t is the learning rate and λ≥0\lambda\geq 0 is the decay coefficient.

The factor 1−ηtλ1-\eta_t\lambda contracts the existing parameter vector independently of the loss gradient. Under the usual condition 0≤ηtλ<10\leq\eta_t\lambda<1, parameters would shrink toward zero if the data gradient were absent. During actual training, the gradient can oppose that contraction, so individual weights and the overall parameter norm need not decrease at every step.

Terminology and coefficient conventions vary. Some formulations use a direct per-step shrinkage factor, such as 1−δt1-\delta_t, rather than expressing it as 1−ηtλ1-\eta_t\lambda. Consequently, a reported “weight decay” value is not fully specified without the update rule.

Relationship to L2 regularization

L2 regularization modifies the optimization objective by adding a squared Euclidean norm penalty:

J(θ)=L(θ)+λ2∥θ∥22.J(\theta) = L(\theta)+\frac{\lambda}{2}\|\theta\|_2^2.

Its gradient is

∇J(θ)=∇L(θ)+λθ.\nabla J(\theta)=\nabla L(\theta)+\lambda\theta.

Applying ordinary gradient descent gives

θt+1=θt−ηt(gt+λθt)=(1−ηtλ)θt−ηtgt.\theta_{t+1} = \theta_t-\eta_t\bigl(g_t+\lambda\theta_t\bigr) = (1-\eta_t\lambda)\theta_t-\eta_t g_t.

Thus, L2 regularization and multiplicative weight decay are algebraically equivalent for plain gradient descent, including stochastic gradient descent without momentum. If the penalty is written as λ∥θ∥22\lambda\|\theta\|_2^2 without the factor 1/21/2, its gradient instead contributes 2λθ2\lambda\theta.

This equivalence does not hold for arbitrary optimization algorithms. In particular:

  • Momentum: adding λθt\lambda\theta_t to the gradient causes the penalty contribution to accumulate in the momentum state. Direct parameter shrinkage applied separately from momentum generally produces a different trajectory.
  • Adaptive learning rates: transforming the combined gradient also transforms the penalty contribution, making its effect depend on the optimizer’s coordinate-wise scaling and gradient history.

In linear least-squares models, the corresponding squared-coefficient penalty is the basis of ridge regression. The equivalence concerns the objective and its optimization; it does not imply that every optimizer option named “weight decay” implements the same procedure.

Coupled and decoupled weight decay

Coupled regularization in adaptive optimizers

The Adam optimizer maintains moving averages of gradients and squared gradients. In a coupled L2 implementation, the optimizer receives

g~t=gt+λθt.\widetilde g_t=g_t+\lambda\theta_t.

Both moment estimates are then calculated using g~t\widetilde g_t. The regularization contribution therefore affects the optimizer’s internal state and is itself affected by adaptive scaling.

A simpler illustration uses a preconditioning matrix PtP_t:

θt+1=θt−ηtPt(gt+λθt).\theta_{t+1} = \theta_t-\eta_tP_t(g_t+\lambda\theta_t).

Here, the penalty produces the term −ηtλPtθt-\eta_t\lambda P_t\theta_t, rather than uniform multiplicative shrinkage. In Adam, the interaction is more involved because the moment estimates also incorporate the penalty.

Decoupled weight decay and AdamW

Decoupled weight decay separates parameter shrinkage from the gradient transformation. Schematically,

θt+1=(1−ηtλ)θt−ηtUt(g1,…,gt),\theta_{t+1} = (1-\eta_t\lambda)\theta_t-\eta_tU_t(g_1,\ldots,g_t),

where UtU_t is the optimizer’s update direction computed from data gradients, without adding the decay term.

AdamW applies this principle to Adam. A standard update is

θt+1=(1−ηtλ)θt−ηtm^tv^t+ϵ,\theta_{t+1} = (1-\eta_t\lambda)\theta_t - \eta_t \frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},

with elementwise division, where m^t\widehat m_t and v^t\widehat v_t are bias-corrected moment estimates and ϵ\epsilon is a numerical-stability constant.

The influential work introducing AdamW was first released by Ilya Loshchilov and Frank Hutter in 2017 and published at ICLR in 2019 under the title Decoupled Weight Decay Regularization. It emphasized that L2 regularization and weight decay should not be treated as interchangeable for adaptive optimizers.

Decoupling makes the shrinkage operation independent of Adam’s gradient preconditioning. It does not eliminate all interaction with the learning rate: in the update above, the per-step contraction still depends on ηtλ\eta_t\lambda. Nor is AdamW generally equivalent to running Adam on the original loss plus a fixed L2 penalty.

Effects on learning and generalization

Weight decay discourages large parameter magnitudes and changes the model’s inductive bias. In suitable settings, this can improve generalization by reducing the tendency to fit incidental variation in the training data. Excessive decay can instead cause underfitting.

Its effect is especially transparent in ridge regression. For the objective

12∥Xθ−y∥22+λ2∥θ∥22,\frac12\|X\theta-y\|_2^2+\frac{\lambda}{2}\|\theta\|_2^2,

the solution for λ>0\lambda>0 is

θ^=(X⊤X+λI)−1X⊤y,\widehat\theta = (X^\top X+\lambda I)^{-1}X^\top y,

where XX is the design matrix and yy the target vector. The penalty shrinks coefficients and improves the conditioning of the linear system.

In deep learning, the relationship between weight magnitude and model complexity is less direct. Networks can admit different parameterizations of the same function. For example, with a positively homogeneous activation function such as ReLU, scaling one layer’s weights up and a following layer’s weights down can leave the represented function unchanged.

Batch normalization introduces further scale invariances, exact in idealized formulations and approximate in practical implementations. For scale-invariant parameters, weight decay can affect optimization by changing parameter norms and hence the effective size of directional updates, rather than simply reducing the expressiveness of the represented function. Smaller weight norms therefore do not universally imply a simpler predictor.

Parameter selection and training configuration

Weight decay is commonly applied to weight matrices and convolutional kernels. Biases and the learned scale and offset parameters of normalization layers are often excluded, particularly in Transformer training. These exclusions are conventions rather than mathematical requirements.

Practical behavior depends on several choices:

  • Decay coefficient: λ\lambda is a hyperparameter, commonly selected using a validation set.
  • Learning-rate schedule: when shrinkage is learning-rate-scaled, lowering the learning rate also lowers decay per step.
  • Number of updates: additional optimizer steps create additional opportunities for shrinkage.
  • Parameter groups: different parameter subsets may have different coefficients, including zero.
  • Loss reduction: summing rather than averaging example losses changes gradient scale and the numerical interpretation of regularization settings.

Ignoring data gradients, the accumulated contraction after TT steps is

θT=[∏t=0T−1(1−ηtλ)]θ0.\theta_T = \left[\prod_{t=0}^{T-1}(1-\eta_t\lambda)\right]\theta_0.

When every ηtλ\eta_t\lambda is small,

∏t=0T−1(1−ηtλ)≈exp⁡(−λ∑t=0T−1ηt).\prod_{t=0}^{T-1}(1-\eta_t\lambda) \approx \exp\left(-\lambda\sum_{t=0}^{T-1}\eta_t\right).

This explains why the same coefficient can have different effects under different training lengths, batch sizes, or learning-rate schedules. Gradient accumulation also matters because decay is normally applied at optimizer updates, not at every gradient-accumulation step.

Libraries such as PyTorch and TensorFlow provide optimizer-level mechanisms for weight decay. The optimizer’s documented update equation, parameter exclusions, and treatment of parameters without gradients determine the actual behavior; the option name alone is insufficient.

Limitations and related methods

Weight decay usually shrinks parameters continuously rather than setting them exactly to zero. It is therefore not ordinarily a method for explicit feature selection or model pruning. L1 penalties and sparsity-inducing procedures have different mathematical effects.

It can be combined with dropout, data augmentation, and early stopping, but these methods act differently: dropout perturbs training computation, augmentation changes the training examples presented to the model, and early stopping limits optimization duration.

Weight decay also depends on parameterization and input scaling. Penalizing all coordinates equally does not necessarily penalize their influence on predictions equally, which is particularly clear in linear models whose features use different units. It provides no general guarantee of robustness, calibrated uncertainty, or successful performance under distribution shift.