{
  "id": "momentum-method",
  "title": "Momentum method",
  "lang": "en",
  "source_lang": "en",
  "summary": "The momentum method accelerates iterative optimization by combining the current gradient with a weighted memory of previous updates.",
  "category": "mathematics",
  "subcategory": "applied",
  "type": "concept",
  "aliases": [
    "Gradient descent with momentum",
    "Polyak momentum",
    "Heavy-ball method"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/momentum-method/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/momentum-method/",
      "json": "https://aiwiki.page/zh/momentum-method.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The momentum method is a technique in mathematical optimization that updates a candidate solution using both the current gradient and a weighted contribution from earlier updates. By retaining a direction of motion, it can accelerate progress along directions where gradients remain consistent and reduce some of the zigzagging associated with ordinary gradient descent. In its classical form, it is called the heavy-ball method or Polyak momentum. More broadly, “momentum methods” include related algorithms with different gradient-evaluation points and convergence properties."
    },
    {
      "heading": "Update rule",
      "text": "Consider the unconstrained minimization problem\n\n\\[\\min_{x\\in\\mathbb{R}^{d}} f(x),\\]\n\nwhere \\(f\\) is a differentiable objective function. The classical momentum iteration is\n\n\\[x_{k+1} = x_k-\\alpha\\nabla f(x_k)+\\beta(x_k-x_{k-1}),\\]\n\nwhere \\(\\alpha>0\\) is the step size, or learning rate, and \\(\\beta\\) is the momentum coefficient, commonly chosen in the interval \\(0\\leq\\beta<1\\).\n\nIntroducing the displacement \\(v_k=x_k-x_{k-1}\\) gives the equivalent formulation\n\n\\[v_{k+1}=\\beta v_k-\\alpha\\nabla f(x_k), \\qquad x_{k+1}=x_k+v_{k+1}.\\]\n\nA typical initialization is \\(v_0=0\\), equivalent to setting \\(x_{-1}=x_0\\). When \\(\\beta=0\\), the method reduces to ordinary gradient descent.\n\nAnother convention accumulates gradients rather than displacements:\n\n\\[m_{k+1}=\\beta m_k+g_k, \\qquad x_{k+1}=x_k-\\alpha m_{k+1},\\]\n\nwhere \\(g_k=\\nabla f(x_k)\\) for deterministic optimization. With constant \\(\\alpha\\) and matching initial conditions, this is equivalent to the preceding formulation under \\(v_k=-\\alpha m_k\\).\n\nSome implementations instead use\n\n\\[m_{k+1}=\\beta m_k+(1-\\beta)g_k.\\]\n\nThis normalized convention requires a corresponding rescaling of the learning rate to reproduce the same parameter updates. When learning rates vary over time, gradient-buffer and displacement-buffer formulations need not remain equivalent without additional adjustments."
    },
    {
      "heading": "Memory and mechanical interpretation",
      "text": "With \\(v_0=0\\) and constant parameters, expanding the update yields\n\n\\[v_{k+1} = -\\alpha\\sum_{j=0}^{k}\\beta^{k-j}\\nabla f(x_j).\\]\n\nThus, the current displacement contains an exponentially weighted history of gradients. For \\(\\beta\\) close to one, the characteristic memory length is approximately \\(1/(1-\\beta)\\) iterations.\n\nIf successive gradients point in similar directions, their contributions reinforce one another. If their signs alternate, as often occurs when an algorithm crosses a narrow valley repeatedly, the accumulated contributions can partially cancel. This explains why momentum can improve motion through elongated level sets. It does not imply that momentum always reduces oscillation: an unsuitable parameter choice can amplify it.\n\nThe term “heavy ball” comes from an analogy with a particle moving through a potential field under friction. A corresponding ordinary differential equation is\n\n\\[\\ddot{x}(t)+\\gamma\\dot{x}(t)+\\nabla f(x(t))=0,\\]\n\nwhere \\(\\gamma>0\\) represents damping. Approximating acceleration and velocity by finite differences gives\n\n\\[x_{k+1} = x_k-h^2\\nabla f(x_k) +(1-\\gamma h)(x_k-x_{k-1}),\\]\n\nwhich has the momentum form with \\(\\alpha=h^2\\) and \\(\\beta=1-\\gamma h\\).\n\nFor sufficiently regular trajectories, the continuous-time mechanical energy\n\n\\[E(t)=f(x(t))+\\frac12\\|\\dot{x}(t)\\|^2\\]\n\nsatisfies\n\n\\[\\frac{dE}{dt}=-\\gamma\\|\\dot{x}(t)\\|^2\\leq 0.\\]\n\nThe discrete algorithm does not automatically inherit this monotonicity. Its stability depends on the step size, damping, and curvature of the objective."
    },
    {
      "heading": "Behavior on quadratic objectives",
      "text": "The clearest analysis concerns a strictly convex quadratic:\n\n\\[f(x)=\\frac12x^{\\mathsf T}Ax-b^{\\mathsf T}x,\\]\n\nwhere \\(A\\) is a symmetric positive-definite matrix. The unique minimizer is \\(x^\\star=A^{-1}b\\).\n\nWriting \\(e_k=x_k-x^\\star\\) gives\n\n\\[e_{k+1} = \\bigl((1+\\beta)I-\\alpha A\\bigr)e_k-\\beta e_{k-1}.\\]\n\nUsing the eigenvectors of \\(A\\), the iteration separates into scalar error components. Along an eigenvector with eigenvalue \\(\\lambda\\), the error satisfies the recurrence relation\n\n\\[z_{k+1}=(1+\\beta-\\alpha\\lambda)z_k-\\beta z_{k-1}.\\]\n\nIts characteristic equation is\n\n\\[r^2-(1+\\beta-\\alpha\\lambda)r+\\beta=0.\\]\n\nConvergence in that direction requires both roots to have modulus less than one. For \\(0\\leq\\beta<1\\), this is equivalent to\n\n\\[0<\\alpha\\lambda<2(1+\\beta).\\]\n\nConsequently, if the largest eigenvalue is \\(L\\), the full quadratic iteration is stable when\n\n\\[0<\\alpha<\\frac{2(1+\\beta)}{L}.\\]\n\nThese conditions concern positive-definite quadratics; they are not a universal convergence criterion for nonlinear objectives.\n\nOptimal constant parameters for a spectral interval\n\nSuppose the eigenvalues lie in \\([\\mu,L]\\), with \\(0<\\mu<L\\), and define the spectral condition number\n\n\\[\\kappa=\\frac{L}{\\mu}.\\]\n\nThe classical heavy-ball parameters that minimize the worst-case characteristic-root modulus over this interval are\n\n\\[\\alpha_\\star = \\frac{4}{(\\sqrt L+\\sqrt\\mu)^2}, \\qquad \\beta_\\star = \\left( \\frac{\\sqrt L-\\sqrt\\mu} {\\sqrt L+\\sqrt\\mu} \\right)^2.\\]\n\nThe resulting worst-case asymptotic root factor is\n\n\\[\\rho_\\star = \\frac{\\sqrt\\kappa-1}{\\sqrt\\kappa+1}.\\]\n\nFor comparison, optimally tuned constant-step gradient descent on the same quadratic class has error contraction factor\n\n\\[\\rho_{\\mathrm{GD}}=\\frac{\\kappa-1}{\\kappa+1}.\\]\n\nThe square-root dependence on \\(\\kappa\\) explains the potential advantage of momentum on ill-conditioned quadratics.\n\nThe heavy-ball root factor is an asymptotic statement, not necessarily a bound of the form \\(\\|e_k\\|\\leq C\\rho_\\star^k\\) with a fixed constant \\(C\\). Repeated characteristic roots at the spectral endpoints can introduce polynomial factors such as \\(k\\rho_\\star^k\\), and transient behavior can be substantial."
    },
    {
      "heading": "Heavy-ball momentum and Nesterov acceleration",
      "text": "Boris Polyak introduced the classical heavy-ball method in 1964. A related but distinct approach, introduced by Yurii Nesterov in 1983, is Nesterov accelerated gradient. One common representation is\n\n\\[y_k=x_k+\\beta_k(x_k-x_{k-1}),\\]\n\n\\[x_{k+1}=y_k-\\alpha_k\\nabla f(y_k).\\]\n\nThe central distinction is the location of gradient evaluation:\n\n- Heavy-ball momentum evaluates the gradient at the current point \\(x_k\\).\n- Nesterov acceleration evaluates it at the extrapolated point \\(y_k\\).\n\nThis difference has important theoretical consequences. For convex objectives whose gradients satisfy Lipschitz continuity with constant \\(L\\), suitable Nesterov schemes achieve\n\n\\[f(x_k)-f(x^\\star)=O(k^{-2}),\\]\n\ncompared with the standard \\(O(k^{-1})\\) objective-value guarantee for gradient descent on the same smooth convex class. For \\(\\mu\\)-strongly convex, \\(L\\)-smooth objectives, accelerated schemes attain iteration complexity proportional to\n\n\\[\\sqrt{\\frac{L}{\\mu}}\\log\\frac{1}{\\varepsilon}\\]\n\nto reduce the objective error by a prescribed relative factor \\(\\varepsilon\\).\n\nClassical heavy-ball momentum with its quadratic-optimal parameters does not automatically inherit these general guarantees. Parameters that work well on quadratics can fail on nonquadratic objectives, even when those objectives are smooth and strongly convex. Thus, quadratic acceleration and general-purpose convex acceleration are distinct results."
    },
    {
      "heading": "Stochastic optimization and machine learning",
      "text": "Momentum is widely used with stochastic gradient descent in machine learning, particularly for training artificial neural networks. In this setting, an estimated gradient \\(g_k\\), often computed from a minibatch of training examples, replaces the exact gradient:\n\n\\[v_{k+1}=\\beta v_k-\\alpha_k g_k, \\qquad x_{k+1}=x_k+v_{k+1}.\\]\n\nAccumulating gradients can preserve a persistent direction despite fluctuations between minibatches. However, any noise-smoothing interpretation must account for normalization. A normalized moving average smooths its input, whereas an unnormalized momentum buffer also increases its gain. For a constant gradient \\(g\\), the latter approaches\n\n\\[m=\\frac{g}{1-\\beta}.\\]\n\nIncreasing \\(\\beta\\) while holding the learning rate fixed therefore changes both memory and update magnitude.\n\nStochastic momentum introduces several additional considerations:\n\n- Noise near a minimizer. With a fixed nonzero learning rate, persistent gradient noise generally prevents convergence to an exact point. The iterates may fluctuate around a minimizer.\n- Changing directions. Stored momentum can become poorly aligned when the local geometry or sampled gradients change rapidly.\n- Learning-rate schedules. The effect of a schedule depends partly on whether the implementation stores gradients or displacements.\n- Parameter interactions. Learning rate, momentum, batch size, and regularization jointly affect the update dynamics.\n\nAdam also maintains an exponentially weighted first-moment estimate of gradients, but combines it with second-moment scaling and bias correction. It is therefore not simply classical momentum with a different coefficient."
    },
    {
      "heading": "Applications and limitations",
      "text": "Momentum is attractive in large-scale differentiable optimization because it typically needs one gradient evaluation per iteration and one additional vector of storage. It does not require explicitly constructing the Hessian matrix. Applications include minimizing training loss functions, large least-squares objectives, and differentiable models with many parameters.\n\nIts principal limitations are sensitivity to tuning and the gap between simple model problems and general objectives. In particular:\n\n- Objective values need not decrease monotonically. An iterate can pass through a low-value region and temporarily move uphill while retaining velocity.\n- Curvature estimates may be inaccurate. The values \\(\\mu\\) and \\(L\\) required by the classical optimal formulas may be unknown or vary substantially across the domain.\n- Nonconvex problems require separate analysis. Momentum does not guarantee a global minimum, nor does it universally ensure escape from a saddle point.\n- Constraints require modifications. The unconstrained update may leave the feasible set. Projection or other constrained formulations change the algorithm and its analysis.\n- Momentum is not a substitute for preconditioning. Its memory can improve convergence without correcting all coordinate-scaling difficulties.\n\nClaims of acceleration therefore depend on the objective class, parameter choices, gradient accuracy, and whether performance is measured by objective error, distance to a minimizer, or gradient norm."
    }
  ],
  "links": [
    "mathematical-optimization",
    "gradient",
    "gradient-descent",
    "objective-function",
    "ordinary-differential-equation",
    "positive-definite-matrix",
    "eigenvalues-and-eigenvectors",
    "recurrence-relation",
    "condition-number",
    "lipschitz-continuity",
    "stochastic-gradient-descent",
    "machine-learning",
    "artificial-neural-network",
    "regularization",
    "adam-optimizer",
    "hessian-matrix",
    "loss-function",
    "saddle-point",
    "feasible-set"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}