{
  "id": "momentum-optimization",
  "title": "Momentum (Optimization)",
  "lang": "en",
  "source_lang": "en",
  "summary": "Momentum is an optimization technique that combines current gradient information with accumulated past updates to influence the direction and size of subsequent steps.",
  "category": "mathematics",
  "subcategory": "applied",
  "type": "concept",
  "aliases": [
    "Momentum method",
    "gradient descent with momentum",
    "heavy-ball method"
  ],
  "generated_at": "2026-11-06",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/momentum-optimization/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/momentum-optimization/",
      "json": "https://aiwiki.page/zh/momentum-optimization.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Momentum is a technique in mathematical optimization that augments a gradient-based update with a contribution from previous updates. Rather than choosing each step solely from the current gradient, the algorithm maintains a state that carries information across iterations. This can accelerate progress along directions where gradients remain aligned and reduce some of the oscillation associated with poorly conditioned objectives. The classical formulation is the heavy-ball method, introduced by Boris Polyak in 1964. Momentum also appears in stochastic gradient descent, accelerated optimization methods, and algorithms for training artificial neural networks."
    },
    {
      "heading": "Basic formulation",
      "text": "Consider the unconstrained problem\n\n\\[\\min_{x\\in\\mathbb{R}^d} f(x),\\]\n\nwhere \\(f\\) is a differentiable objective function. Ordinary gradient descent uses\n\n\\[x_{t+1}=x_t-\\alpha_t\\nabla f(x_t),\\]\n\nwith step size, or learning rate, \\(\\alpha_t>0\\).\n\nThe heavy-ball momentum method instead uses\n\n\\[x_{t+1} =x_t-\\alpha_t\\nabla f(x_t) +\\beta_t(x_t-x_{t-1}),\\]\n\nwhere \\(\\beta_t\\) is the momentum coefficient, usually chosen in the interval \\(0\\leq\\beta_t<1\\). The additional term preserves part of the preceding displacement.\n\nFor constant parameters, an equivalent velocity formulation is\n\n\\[v_{t+1}=\\beta v_t-\\alpha\\nabla f(x_t), \\qquad x_{t+1}=x_t+v_{t+1}.\\]\n\nA common initialization is \\(v_0=0\\), corresponding to no initial motion. Setting \\(\\beta=0\\) recovers gradient descent.\n\nAnother common convention stores accumulated gradients:\n\n\\[m_{t+1}=\\beta m_t+g_t, \\qquad x_{t+1}=x_t-\\eta m_{t+1},\\]\n\nwhere \\(g_t\\) is either the full gradient or a stochastic estimate. With constant \\(\\eta=\\alpha\\) and compatible initialization, this gives the same iterates as the velocity formulation. If the learning rate varies, the two conventions are not automatically equivalent: one stores past displacements, whereas the other stores past gradients."
    },
    {
      "heading": "Accumulation and smoothing",
      "text": "Expanding the gradient accumulator gives\n\n\\[m_{t+1} =\\beta^{t+1}m_0+ \\sum_{k=0}^{t}\\beta^{t-k}g_k.\\]\n\nThus, past gradients receive geometrically decreasing weights. Consistently aligned gradients reinforce one another; gradients that repeatedly reverse direction can partially cancel.\n\nA normalized exponential moving average instead uses\n\n\\[\\bar m_{t+1} =\\beta\\bar m_t+(1-\\beta)g_t.\\]\n\nFor constant \\(\\beta\\), these two accumulators differ by a fixed scale when initialized at zero. Their learning rates must therefore be rescaled to obtain identical parameter updates.\n\nFor a constant gradient \\(g\\), the unnormalized accumulator approaches \\(g/(1-\\beta)\\). Consequently, increasing \\(\\beta\\) while leaving the learning rate fixed also increases the eventual update magnitude in this setting. Comparisons between momentum coefficients must account for this scaling.\n\nThe characteristic memory length grows roughly as \\(1/(1-\\beta)\\) when \\(\\beta\\) is close to one. A larger coefficient therefore retains information longer, but also makes the update slower to respond when the gradient changes direction.\n\nMechanical interpretation\n\nThe heavy-ball name comes from an analogy with a particle moving in a potential field under friction. A related continuous-time model is\n\n\\[\\ddot x(t)+\\gamma\\dot x(t)+\\nabla f(x(t))=0,\\]\n\nwhere \\(\\gamma>0\\) represents damping. This is a second-order ordinary differential equation, unlike the first-order gradient-flow equation.\n\nThe analogy explains both persistence and overshoot: motion can continue even after the instantaneous force changes direction. It is an interpretation, not an identification of the algorithm’s stored state with physical momentum in every implementation."
    },
    {
      "heading": "Behavior on quadratic objectives",
      "text": "The mechanism is particularly transparent for a strictly convex quadratic:\n\n\\[f(x)=\\frac12 x^\\mathsf{T}Ax-b^\\mathsf{T}x,\\]\n\nwhere \\(A\\) is a symmetric positive-definite matrix. Its unique minimizer is \\(x^\\star=A^{-1}b\\).\n\nWriting \\(e_t=x_t-x^\\star\\), the heavy-ball iteration becomes\n\n\\[e_{t+1} =\\bigl((1+\\beta)I-\\alpha A\\bigr)e_t-\\beta e_{t-1}.\\]\n\nDecomposition into eigenvectors of \\(A\\) separates this into scalar recurrences. For an eigenvalue \\(\\lambda>0\\), an error component satisfies\n\n\\[z_{t+1}=(1+\\beta-\\alpha\\lambda)z_t-\\beta z_{t-1}.\\]\n\nThe roots of the corresponding characteristic polynomial\n\n\\[r^2-(1+\\beta-\\alpha\\lambda)r+\\beta=0\\]\n\ndetermine whether that component decays, oscillates, or diverges.\n\nFor \\(0\\leq\\beta<1\\), all components converge to zero if\n\n\\[0<\\alpha\\lambda<2(1+\\beta)\\]\n\nfor every eigenvalue. In particular, if the largest eigenvalue is \\(L\\), the condition becomes\n\n\\[0<\\alpha<\\frac{2(1+\\beta)}{L}.\\]\n\nThis is a stability statement for the quadratic problem, not a general convergence guarantee for arbitrary nonlinear objectives.\n\nAcceleration and conditioning\n\nSuppose the eigenvalues lie in \\([\\mu,L]\\), with \\(0<\\mu<L\\), and define the spectral condition number\n\n\\[\\kappa=\\frac{L}{\\mu}.\\]\n\nClassical parameters that minimize the worst-case asymptotic root magnitude over this interval are\n\n\\[\\alpha_\\star =\\frac{4}{(\\sqrt L+\\sqrt\\mu)^2}, \\qquad \\beta_\\star =\\left(\\frac{\\sqrt L-\\sqrt\\mu} {\\sqrt L+\\sqrt\\mu}\\right)^2.\\]\n\nThe resulting worst-case asymptotic factor is\n\n\\[q=\\frac{\\sqrt\\kappa-1}{\\sqrt\\kappa+1}.\\]\n\nBy comparison, optimally tuned constant-step gradient descent on the same spectral interval has factor\n\n\\[\\frac{\\kappa-1}{\\kappa+1}.\\]\n\nFor large \\(\\kappa\\), this changes the characteristic iteration scale from dependence on \\(\\kappa\\) to dependence on \\(\\sqrt\\kappa\\).\n\nThese factors describe asymptotic behavior. Repeated characteristic roots can introduce polynomial factors in the iteration count, and substantial transient oscillation can occur before the asymptotic regime."
    },
    {
      "heading": "Heavy-ball momentum and Nesterov acceleration",
      "text": "Nesterov accelerated gradient, introduced in 1983, is related to momentum but is not the same update rule. A common representation is\n\n\\[y_t=x_t+\\beta_t(x_t-x_{t-1}),\\]\n\n\\[x_{t+1}=y_t-\\alpha_t\\nabla f(y_t).\\]\n\nThe gradient is evaluated at the extrapolated point \\(y_t\\), rather than at \\(x_t\\). This changes the interaction between extrapolation and gradient correction.\n\nWith appropriate parameter schedules, Nesterov acceleration has global guarantees for broad classes studied in convex optimization. For an objective with an \\(L\\)-Lipschitz gradient, standard accelerated schemes achieve an objective-error bound of order\n\n\\[f(x_t)-f(x^\\star)=O(t^{-2}),\\]\n\nrather than the \\(O(t^{-1})\\) bound of basic gradient descent under comparable smooth convex assumptions.\n\nStrongly convex accelerated schemes can achieve iteration complexity of order\n\n\\[O\\!\\left(\\sqrt{\\frac{L}{\\mu}}\\log\\frac1\\varepsilon\\right)\\]\n\nfor reducing error to tolerance \\(\\varepsilon\\), with initial-error terms suppressed here.\n\nThe heavy-ball method’s optimal quadratic behavior does not establish the same global guarantees for every smooth, strongly convex nonlinear function. In particular, parameters optimized for quadratic objectives can fail outside that class. “Momentum” and “acceleration” are therefore related terms, not interchangeable claims about convergence rates."
    },
    {
      "heading": "Stochastic momentum",
      "text": "In machine learning, a full gradient may be expensive because the loss function aggregates many training examples. A stochastic or minibatch gradient estimate replaces it:\n\n\\[g_t=\\nabla f(x_t)+\\xi_t,\\]\n\nwhere \\(\\xi_t\\) represents estimation noise.\n\nAccumulation can smooth rapidly fluctuating gradient information. For illustration, consider a normalized exponential average of independent, identically distributed noise vectors with zero mean and covariance matrix \\(\\Sigma\\). Its stationary covariance is\n\n\\[\\operatorname{Cov}(\\bar m) =\\frac{1-\\beta}{1+\\beta}\\Sigma.\\]\n\nThis calculation describes averaging noise at a fixed underlying signal. During optimization, iterates and gradients interact, and successive noise terms need not satisfy these assumptions. It therefore does not by itself establish lower parameter error or faster convergence.\n\nMomentum does not eliminate stochastic uncertainty. With a fixed learning rate, persistent noise generally prevents convergence exactly to a minimizer, even when the deterministic iteration would converge. Decreasing step sizes, changing batch sizes, and additional assumptions lead to different convergence regimes.\n\nThe benefit also depends on how quickly the underlying gradient changes. Long memory can smooth noise but retain outdated directions, especially in regions where the objective’s Hessian varies substantially."
    },
    {
      "heading": "Related adaptive methods",
      "text": "Adam combines an exponential average of gradients with a coordinatewise scale derived from an exponential average of squared gradients. Its first-moment state is momentum-like, but the complete update is not classical heavy-ball momentum because it also includes adaptive scaling and bias corrections.\n\nMomentum is also distinct from:\n\n- Newton’s method, which uses second-order information to transform the gradient.\n- Regularization, which modifies the optimization problem or constrains its solution rather than merely adding update memory.\n- Line search, which selects a step length by evaluating behavior along a direction.\n\nThese techniques can be combined with momentum, but each addresses a different part of the optimization procedure."
    },
    {
      "heading": "Practical limitations and implementation distinctions",
      "text": "Parameter coupling. Learning rate and momentum jointly determine the update dynamics. Increasing momentum without changing the learning rate can increase both persistence and effective step magnitude.\n\nNonmonotonic behavior. The objective need not decrease at every iteration. Temporary increases can be compatible with eventual convergence, but persistent oscillations can also indicate instability.\n\nSensitivity to changing curvature. Favorable results for a fixed quadratic do not automatically transfer to objectives whose curvature changes along the trajectory. Local quadratic reasoning is useful but incomplete.\n\nNo automatic global solution. For nonconvex objectives, momentum alone does not guarantee avoidance of every saddle point or convergence to a global minimizer.\n\nState and conventions. Classical momentum requires an additional vector of the same dimension as the parameters. Implementations differ in initialization, gradient normalization, learning-rate scheduling, and the placement of learning-rate factors. These choices can produce different trajectories even when the same numerical momentum coefficient is used."
    }
  ],
  "links": [
    "mathematical-optimization",
    "gradient",
    "stochastic-gradient-descent",
    "artificial-neural-network",
    "objective-function",
    "gradient-descent",
    "ordinary-differential-equation",
    "positive-definite-matrix",
    "eigenvalues-and-eigenvectors",
    "characteristic-polynomial",
    "condition-number",
    "convex-optimization",
    "machine-learning",
    "loss-function",
    "covariance-matrix",
    "hessian-matrix",
    "adam-optimizer",
    "newtons-method",
    "regularization",
    "line-search",
    "saddle-point"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}