{
  "id": "adamw",
  "title": "AdamW",
  "lang": "en",
  "source_lang": "en",
  "summary": "AdamW is an adaptive optimization algorithm that separates weight decay from Adam’s gradient-based updates, providing direct parameter shrinkage without altering gradient-moment estimates.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Adam with decoupled weight decay",
    "Adaptive Moment Estimation with decoupled weight decay"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/adamw/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/adamw/",
      "json": "https://aiwiki.page/zh/adamw.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "AdamW is a optimization algorithm used to train artificial neural networks and other differentiable models. It modifies the Adam optimizer by applying weight decay directly to model parameters, separately from the gradient-based update. This distinction matters because adding a squared-parameter penalty to the training objective—the conventional way to implement weight decay in some optimizers—does not produce the same update under Adam’s coordinate-wise adaptive scaling. AdamW retains Adam’s moving averages of gradients and squared gradients while preventing the weight-decay term from entering those averages."
    },
    {
      "heading": "Background and development",
      "text": "Optimization in machine learning typically adjusts model parameters to reduce a loss function evaluated on batches of training data. Gradients, commonly computed through backpropagation, indicate how the loss changes with each parameter.\n\nAdam combines a moving average of gradients with a moving average of squared gradients. These statistics produce coordinate-wise adaptive updates: different parameters receive different effective step sizes, even when they share the same global learning rate.\n\nAdamW was introduced by Ilya Loshchilov and Frank Hutter in work first released in 2017 and published at the International Conference on Learning Representations in 2019 under the title Decoupled Weight Decay Regularization. Their central observation was that weight decay and an \\(L_2\\) penalty, often treated as interchangeable, cease to be equivalent when the gradient update is adaptively rescaled.\n\nAdamW subsequently became widely used in deep learning, particularly for Transformer models. Implementations are available in frameworks including PyTorch and TensorFlow."
    },
    {
      "heading": "Update rule",
      "text": "Let \\(\\theta_{t-1}\\) denote the parameter vector before optimization step \\(t\\), and let\n\n\\[g_t=\\nabla_\\theta \\mathcal{L}_t(\\theta_{t-1})\\]\n\nbe the gradient of the batch loss. In AdamW, this gradient does not include the decoupled weight-decay contribution.\n\nStarting with \\(m_0=0\\) and \\(v_0=0\\), the first- and second-moment estimates are\n\n\\[m_t=\\beta_1m_{t-1}+(1-\\beta_1)g_t,\\]\n\n\\[v_t=\\beta_2v_{t-1}+(1-\\beta_2)g_t^2.\\]\n\nSquaring and subsequent vector operations are element-wise. The second quantity is an estimate of the uncentered second moment, rather than the variance.\n\nBecause the moving averages start at zero, Adam applies bias correction:\n\n\\[\\hat m_t=\\frac{m_t}{1-\\beta_1^t}, \\qquad \\hat v_t=\\frac{v_t}{1-\\beta_2^t}.\\]\n\nA common AdamW update is\n\n\\[\\theta_t = (1-\\alpha_t\\lambda)\\theta_{t-1} - \\alpha_t \\frac{\\hat m_t}{\\sqrt{\\hat v_t}+\\epsilon},\\]\n\nwhere:\n\n- \\(\\alpha_t\\) is the learning rate at step \\(t\\);\n- \\(\\lambda\\geq 0\\) is the weight-decay coefficient;\n- \\(\\beta_1\\) and \\(\\beta_2\\) control the moving averages;\n- \\(\\epsilon>0\\) provides numerical stabilization.\n\nThe first term shrinks parameters toward zero. The second is the ordinary adaptive Adam update. Crucially, neither \\(m_t\\) nor \\(v_t\\) receives a contribution from \\(\\lambda\\theta_{t-1}\\).\n\nThis equation describes a widely used convention in which the decay coefficient is multiplied by the learning rate. Publications and software can use different parameterizations or numerical arrangements, so matching the name “AdamW” alone does not guarantee identical updates."
    },
    {
      "heading": "Why decoupling matters",
      "text": "Equivalence for plain stochastic gradient descent\n\nConsider an objective with an \\(L_2\\) penalty:\n\n\\[\\mathcal{J}(\\theta) = \\mathcal{L}(\\theta) + \\frac{\\lambda}{2}\\|\\theta\\|_2^2.\\]\n\nIts gradient is\n\n\\[\\nabla\\mathcal{J}(\\theta) = \\nabla\\mathcal{L}(\\theta)+\\lambda\\theta.\\]\n\nFor plain stochastic gradient descent without momentum, the resulting update is\n\n\\[\\theta_t = \\theta_{t-1} - \\alpha_t \\left(g_t+\\lambda\\theta_{t-1}\\right) = (1-\\alpha_t\\lambda)\\theta_{t-1}-\\alpha_tg_t.\\]\n\nThus, under this coefficient convention, adding an \\(L_2\\) penalty is exactly equivalent to multiplicative weight decay.\n\nThis equivalence does not automatically extend to optimizers with momentum or adaptive gradient transformations.\n\nNon-equivalence for Adam\n\nIf an \\(L_2\\) penalty is added to the objective optimized by Adam, the optimizer receives\n\n\\[g_t^{\\mathrm{reg}}=g_t+\\lambda\\theta_{t-1}.\\]\n\nBoth moving averages are then calculated from this modified gradient. The penalty therefore affects not only the immediate update but also the optimizer’s accumulated state. Its influence is mixed with gradient history and rescaled differently across coordinates.\n\nAdamW instead applies direct parameter shrinkage outside the adaptive calculation. For parameters assigned the same decay coefficient, the multiplicative shrinkage factor is the same regardless of their gradient-moment estimates.\n\nThe difference is therefore more substantial than moving an otherwise identical term to another location in code: coupled \\(L_2\\) regularization and decoupled weight decay produce different state trajectories and, generally, different parameter trajectories.\n\nRegularization interpretation\n\nWeight decay is a form of regularization that influences the optimization trajectory and can help reduce overfitting. With AdamW, however, the update is not generally equivalent to running ordinary Adam on the loss plus a fixed Euclidean \\(L_2\\) penalty.\n\nLikewise, parameter shrinkage does not guarantee better generalization. Its effects depend on the model, data, training duration, and other optimization choices."
    },
    {
      "heading": "Hyperparameters and training schedules",
      "text": "AdamW retains Adam’s principal hyperparameters and adds an explicitly decoupled weight-decay coefficient.\n\nHyperparameter | Role\nLearning rate \\(\\alpha_t\\) | Controls the adaptive update scale and, under the displayed convention, the per-step shrinkage\nFirst-moment coefficient \\(\\beta_1\\) | Determines how quickly the gradient moving average responds to new gradients\nSecond-moment coefficient \\(\\beta_2\\) | Determines how quickly the squared-gradient moving average responds\nStabilizer \\(\\epsilon\\) | Affects numerical stability and updates when second-moment estimates are small\nWeight decay \\(\\lambda\\) | Controls direct shrinkage of selected parameters\n\nCommon implementations use \\(\\beta_1=0.9\\) and \\(\\beta_2=0.999\\) as defaults, although particular training systems may use different settings.\n\n“Decoupled” does not mean that weight decay is numerically independent of the learning rate. In the update above, the shrinkage per step depends on \\(\\alpha_t\\lambda\\). Considering only the multiplicative decay contribution over \\(T\\) steps gives\n\n\\[\\theta_T = \\left[ \\prod_{t=1}^{T}(1-\\alpha_t\\lambda) \\right]\\theta_0.\\]\n\nWhen every \\(\\alpha_t\\lambda\\) is small, this factor is approximately\n\n\\[\\exp\\left(-\\lambda\\sum_{t=1}^{T}\\alpha_t\\right).\\]\n\nConsequently, changing the learning-rate schedule, number of optimizer steps, or training duration changes the cumulative decay even if \\(\\lambda\\) is unchanged. This expression isolates the decay contribution; it is not the full trajectory when gradient updates are present."
    },
    {
      "heading": "Parameter groups and implementation",
      "text": "Training systems often divide parameters into groups with different learning rates or decay coefficients. A common Transformer training convention applies decay to weight matrices while excluding bias parameters and the scale and offset parameters of layer normalization. Such exclusions are choices in the training setup, not requirements of AdamW itself.\n\nSeveral implementation details affect reproducibility:\n\n- Penalty placement. Adding an \\(L_2\\) penalty to the loss while enabling AdamW decay applies both mechanisms.\n- Gradient clipping. Clipping the loss gradient before the AdamW update does not normally clip the separate decay contribution.\n- Missing gradients. Libraries may skip parameters whose gradients are absent, whereas a parameter with an explicitly zero gradient may still undergo decay and updates from existing moment estimates.\n- Gradient accumulation. Decay is generally applied once per optimizer step, not once per accumulated batch.\n- Numerical conventions. Epsilon placement, floating-point precision, and fused or parallel implementations can introduce differences in results.\n\nThese details matter when reproducing experiments or transferring configurations between frameworks."
    },
    {
      "heading": "Applications and computational cost",
      "text": "AdamW is widely used in training large language models, vision Transformers, and other deep neural networks. It is also used for fine-tuning, although the appropriate parameter groups and regularization strength may differ from those used in pretraining.\n\nIts computational cost is essentially the same as Adam’s. For \\(n\\) updated parameters, a dense optimizer step takes \\(O(n)\\) time and maintains two additional parameter-sized state arrays for the moment estimates. Weight decay adds an element-wise operation that can often be combined with other update operations.\n\nThe optimizer-state memory can be substantial for large models. Distributed training systems may partition this state, while reduced-precision or compressed optimizer variants can change its storage requirements. Such techniques are separate from AdamW’s defining decoupling rule."
    },
    {
      "heading": "Limitations and comparisons",
      "text": "AdamW does not guarantee convergence to a global minimum for nonconvex neural-network objectives. Decoupling weight decay also does not, by itself, remove known convergence limitations of the underlying Adam method.\n\nCompared with plain stochastic gradient descent, AdamW uses more optimizer-state memory and performs additional element-wise arithmetic. Whether its adaptive updates improve training speed or final predictive performance depends on the task and tuning budget. Neither optimizer is universally superior.\n\nWeight decay is also not a general substitute for other regularization mechanisms. Dropout, data augmentation, and early stopping affect training in different ways. Moreover, in networks with scale-invariant parameterizations, reducing a parameter’s norm can change the effective optimization dynamics without proportionally changing the represented function. Smaller parameter norms therefore do not by themselves establish that a model is simpler or will generalize better."
    }
  ],
  "links": [
    "mathematical-optimization",
    "artificial-neural-network",
    "adam-optimizer",
    "weight-decay",
    "machine-learning",
    "loss-function",
    "training-data",
    "backpropagation",
    "learning-rate",
    "deep-learning",
    "transformer-architecture",
    "pytorch",
    "tensorflow",
    "stochastic-gradient-descent",
    "regularization",
    "overfitting",
    "generalization",
    "hyperparameter",
    "layer-normalization",
    "large-language-model",
    "vision-transformer",
    "fine-tuning",
    "distributed-computing",
    "dropout",
    "data-augmentation",
    "early-stopping"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}