Weight decay is a regularization technique in machine learning that shrinks selected model parameters toward zero during training. It is widely used in artificial neural networks to influence the solutions found by optimization and to help reduce overfitting. With ordinary gradient descent, multiplicative weight decay is equivalent to adding a squared-parameter penalty to the training objective. With adaptive optimizers, however, these operations generally differ. This distinction is central to decoupled weight decay and the AdamW optimizer.
Mathematical definition
Let denote a vector of trainable parameters at step , the data-dependent loss function, and its gradient, possibly estimated from a minibatch. A common weight-decay update is
where is the learning rate and is the decay coefficient.
The factor contracts the existing parameter vector independently of the loss gradient. Under the usual condition , parameters would shrink toward zero if the data gradient were absent. During actual training, the gradient can oppose that contraction, so individual weights and the overall parameter norm need not decrease at every step.
Terminology and coefficient conventions vary. Some formulations use a direct per-step shrinkage factor, such as , rather than expressing it as . Consequently, a reported “weight decay” value is not fully specified without the update rule.
Relationship to L2 regularization
L2 regularization modifies the optimization objective by adding a squared Euclidean norm penalty:
Its gradient is
Applying ordinary gradient descent gives
Thus, L2 regularization and multiplicative weight decay are algebraically equivalent for plain gradient descent, including stochastic gradient descent without momentum. If the penalty is written as without the factor , its gradient instead contributes .
This equivalence does not hold for arbitrary optimization algorithms. In particular:
- Momentum: adding to the gradient causes the penalty contribution to accumulate in the momentum state. Direct parameter shrinkage applied separately from momentum generally produces a different trajectory.
- Adaptive learning rates: transforming the combined gradient also transforms the penalty contribution, making its effect depend on the optimizer’s coordinate-wise scaling and gradient history.
In linear least-squares models, the corresponding squared-coefficient penalty is the basis of ridge regression. The equivalence concerns the objective and its optimization; it does not imply that every optimizer option named “weight decay” implements the same procedure.
Coupled and decoupled weight decay
Coupled regularization in adaptive optimizers
The Adam optimizer maintains moving averages of gradients and squared gradients. In a coupled L2 implementation, the optimizer receives
Both moment estimates are then calculated using . The regularization contribution therefore affects the optimizer’s internal state and is itself affected by adaptive scaling.
A simpler illustration uses a preconditioning matrix :
Here, the penalty produces the term , rather than uniform multiplicative shrinkage. In Adam, the interaction is more involved because the moment estimates also incorporate the penalty.
Decoupled weight decay and AdamW
Decoupled weight decay separates parameter shrinkage from the gradient transformation. Schematically,
where is the optimizer’s update direction computed from data gradients, without adding the decay term.
AdamW applies this principle to Adam. A standard update is
with elementwise division, where and are bias-corrected moment estimates and is a numerical-stability constant.
The influential work introducing AdamW was first released by Ilya Loshchilov and Frank Hutter in 2017 and published at ICLR in 2019 under the title Decoupled Weight Decay Regularization. It emphasized that L2 regularization and weight decay should not be treated as interchangeable for adaptive optimizers.
Decoupling makes the shrinkage operation independent of Adam’s gradient preconditioning. It does not eliminate all interaction with the learning rate: in the update above, the per-step contraction still depends on . Nor is AdamW generally equivalent to running Adam on the original loss plus a fixed L2 penalty.
Effects on learning and generalization
Weight decay discourages large parameter magnitudes and changes the model’s inductive bias. In suitable settings, this can improve generalization by reducing the tendency to fit incidental variation in the training data. Excessive decay can instead cause underfitting.
Its effect is especially transparent in ridge regression. For the objective
the solution for is
where is the design matrix and the target vector. The penalty shrinks coefficients and improves the conditioning of the linear system.
In deep learning, the relationship between weight magnitude and model complexity is less direct. Networks can admit different parameterizations of the same function. For example, with a positively homogeneous activation function such as ReLU, scaling one layer’s weights up and a following layer’s weights down can leave the represented function unchanged.
Batch normalization introduces further scale invariances, exact in idealized formulations and approximate in practical implementations. For scale-invariant parameters, weight decay can affect optimization by changing parameter norms and hence the effective size of directional updates, rather than simply reducing the expressiveness of the represented function. Smaller weight norms therefore do not universally imply a simpler predictor.
Parameter selection and training configuration
Weight decay is commonly applied to weight matrices and convolutional kernels. Biases and the learned scale and offset parameters of normalization layers are often excluded, particularly in Transformer training. These exclusions are conventions rather than mathematical requirements.
Practical behavior depends on several choices:
- Decay coefficient: is a hyperparameter, commonly selected using a validation set.
- Learning-rate schedule: when shrinkage is learning-rate-scaled, lowering the learning rate also lowers decay per step.
- Number of updates: additional optimizer steps create additional opportunities for shrinkage.
- Parameter groups: different parameter subsets may have different coefficients, including zero.
- Loss reduction: summing rather than averaging example losses changes gradient scale and the numerical interpretation of regularization settings.
Ignoring data gradients, the accumulated contraction after steps is
When every is small,
This explains why the same coefficient can have different effects under different training lengths, batch sizes, or learning-rate schedules. Gradient accumulation also matters because decay is normally applied at optimizer updates, not at every gradient-accumulation step.
Libraries such as PyTorch and TensorFlow provide optimizer-level mechanisms for weight decay. The optimizer’s documented update equation, parameter exclusions, and treatment of parameters without gradients determine the actual behavior; the option name alone is insufficient.
Limitations and related methods
Weight decay usually shrinks parameters continuously rather than setting them exactly to zero. It is therefore not ordinarily a method for explicit feature selection or model pruning. L1 penalties and sparsity-inducing procedures have different mathematical effects.
It can be combined with dropout, data augmentation, and early stopping, but these methods act differently: dropout perturbs training computation, augmentation changes the training examples presented to the model, and early stopping limits optimization duration.
Weight decay also depends on parameterization and input scaling. Penalizing all coordinates equally does not necessarily penalize their influence on predictions equally, which is particularly clear in linear models whose features use different units. It provides no general guarantee of robustness, calibrated uncertainty, or successful performance under distribution shift.