{
  "id": "conjugate-gradient-method",
  "title": "Conjugate Gradient Method",
  "lang": "en",
  "source_lang": "en",
  "summary": "The conjugate gradient method is an iterative algorithm for solving symmetric positive-definite linear systems, particularly large sparse problems, through successive minimization over Krylov subspaces.",
  "category": "mathematics",
  "subcategory": "applied",
  "type": "concept",
  "aliases": [
    "CG method",
    "Conjugate gradients",
    "Linear conjugate gradient method"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/conjugate-gradient-method/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/conjugate-gradient-method/",
      "json": "https://aiwiki.page/zh/conjugate-gradient-method.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The conjugate gradient method (CG) is an iterative algorithm for solving a system of linear equations \\(Ax=b\\) when \\(A\\) is a real symmetric positive-definite matrix, or, in the complex case, a Hermitian positive-definite matrix. It is especially important in numerical linear algebra because it requires matrix–vector products rather than an explicit matrix factorization and stores only a small number of vectors. Equivalently, CG minimizes a strictly convex quadratic function by moving along directions that are mutually conjugate with respect to \\(A\\)."
    },
    {
      "heading": "Mathematical formulation",
      "text": "Consider\n\n\\[Ax=b,\\qquad A\\in\\mathbb{R}^{n\\times n},\\]\n\nwhere\n\n\\[A^{\\mathsf T}=A,\\qquad v^{\\mathsf T}Av>0 \\quad\\text{for every }v\\ne0.\\]\n\nThese assumptions guarantee a unique solution \\(x_*=A^{-1}b\\).\n\nThe same solution is the unique minimizer of the quadratic objective function\n\n\\[f(x)=\\frac12x^{\\mathsf T}Ax-b^{\\mathsf T}x.\\]\n\nIts gradient is\n\n\\[\\nabla f(x)=Ax-b.\\]\n\nThus the residual\n\n\\[r=b-Ax\\]\n\nis the negative gradient. The positive definiteness of \\(A\\) makes this a problem in convex optimization.\n\nThe matrix defines an inner product and an associated energy norm:\n\n\\[\\langle u,v\\rangle_A=u^{\\mathsf T}Av, \\qquad \\|v\\|_A=\\sqrt{v^{\\mathsf T}Av}.\\]\n\nThe objective error satisfies\n\n\\[f(x)-f(x_*)=\\frac12\\|x-x_*\\|_A^2.\\]\n\nConsequently, minimizing the quadratic is equivalent to minimizing the error in the energy norm."
    },
    {
      "heading": "Conjugate directions",
      "text": "Two nonzero directions \\(p_i\\) and \\(p_j\\) are \\(A\\)-conjugate if\n\n\\[p_i^{\\mathsf T}Ap_j=0.\\]\n\nThey need not be perpendicular in the ordinary Euclidean sense. Instead, they are orthogonal in the geometry defined by \\(A\\).\n\nSuppose an iterate is updated along a direction \\(p_k\\):\n\n\\[x_{k+1}=x_k+\\alpha_kp_k.\\]\n\nExact minimization along that line gives\n\n\\[\\alpha_k=\\frac{p_k^{\\mathsf T}r_k}{p_k^{\\mathsf T}Ap_k}.\\]\n\nWhen the directions are mutually \\(A\\)-conjugate, a later update does not undo the minimization already achieved along an earlier direction. A set of \\(n\\) nonzero mutually conjugate directions is linearly independent, so these directions can determine the solution in at most \\(n\\) steps in exact arithmetic.\n\nUnlike steepest descent, which always follows the current negative gradient, CG combines the residual with the preceding search direction. This avoids the repeated zigzagging that steepest descent can exhibit on elongated quadratic level sets."
    },
    {
      "heading": "Standard algorithm",
      "text": "Choose an initial approximation \\(x_0\\), and set\n\n\\[r_0=b-Ax_0,\\qquad p_0=r_0.\\]\n\nIf \\(r_0=0\\), the initial approximation already solves the system. Otherwise, for \\(k=0,1,\\ldots\\), compute\n\n\\[q_k=Ap_k,\\]\n\n\\[\\alpha_k=\\frac{r_k^{\\mathsf T}r_k}{p_k^{\\mathsf T}q_k},\\]\n\n\\[x_{k+1}=x_k+\\alpha_kp_k,\\]\n\n\\[r_{k+1}=r_k-\\alpha_kq_k.\\]\n\nAfter checking for convergence, form\n\n\\[\\beta_k= \\frac{r_{k+1}^{\\mathsf T}r_{k+1}} {r_k^{\\mathsf T}r_k}, \\qquad p_{k+1}=r_{k+1}+\\beta_kp_k.\\]\n\nIn exact arithmetic, as long as convergence has not occurred, the denominator \\(p_k^{\\mathsf T}Ap_k\\) is positive. The recurrence generates\n\n\\[r_i^{\\mathsf T}r_j=0 \\quad(i\\ne j)\\]\n\nand\n\n\\[p_i^{\\mathsf T}Ap_j=0 \\quad(i\\ne j).\\]\n\nThus the residuals are mutually orthogonal in the Euclidean inner product, whereas the search directions are mutually \\(A\\)-conjugate. These are distinct properties.\n\nFor complex Hermitian positive-definite systems, the formulas use the conjugate transpose in place of the ordinary transpose."
    },
    {
      "heading": "Krylov-subspace interpretation",
      "text": "CG belongs to the family of Krylov subspace methods. Starting from \\(r_0=b-Ax_0\\), define\n\n\\[\\mathcal K_k(A,r_0) = \\operatorname{span}\\{r_0,Ar_0,\\ldots,A^{k-1}r_0\\}.\\]\n\nThe \\(k\\)-th iterate satisfies\n\n\\[x_k\\in x_0+\\mathcal K_k(A,r_0)\\]\n\nand is characterized by\n\n\\[x_k = \\underset{x\\in x_0+\\mathcal K_k(A,r_0)} {\\operatorname{argmin}} \\|x-x_*\\|_A.\\]\n\nEquivalently,\n\n\\[r_k\\perp\\mathcal K_k(A,r_0).\\]\n\nThis is a projection condition: the residual is orthogonal to the subspace in which the correction is sought.\n\nCG therefore finds the best energy-norm approximation available in an expanding sequence of spaces, rather than making only a locally optimal choice along the latest gradient. The spaces are nested, so the energy-norm error decreases monotonically in exact arithmetic. The Euclidean norm of the residual, however, need not decrease at every iteration.\n\nCG is closely related to the Lanczos algorithm, which constructs an orthogonal basis of a Krylov subspace and reduces a symmetric matrix to tridiagonal form. This relationship explains how CG can achieve its projection property using short recurrences rather than retaining every previous direction."
    },
    {
      "heading": "Convergence and spectral behavior",
      "text": "Let\n\n\\[\\kappa_2(A)=\\frac{\\lambda_{\\max}(A)}{\\lambda_{\\min}(A)}\\]\n\nbe the spectral condition number of the positive-definite matrix. In exact arithmetic, the standard bound is\n\n\\[\\|x_k-x_*\\|_A \\le 2\\left( \\frac{\\sqrt{\\kappa_2(A)}-1} {\\sqrt{\\kappa_2(A)}+1} \\right)^k \\|x_0-x_*\\|_A.\\]\n\nFor \\(\\kappa_2(A)>1\\), this bound gives a worst-case iteration estimate proportional to\n\n\\[\\sqrt{\\kappa_2(A)}\\log(1/\\varepsilon)\\]\n\nfor reducing the relative energy-norm error to a prescribed tolerance \\(\\varepsilon\\).\n\nThe bound can be pessimistic because it uses only the largest and smallest eigenvalues. Actual convergence depends on the whole spectrum and on the components of the initial error in the corresponding eigenspaces.\n\nTo see this, write \\(e_k=x_*-x_k\\). The error has the form\n\n\\[e_k=p_k(A)e_0,\\]\n\nwhere \\(p_k\\) is a polynomial of degree at most \\(k\\) satisfying \\(p_k(0)=1\\). CG implicitly selects an error polynomial that minimizes the energy norm. Eigenvalue clusters can permit effective polynomials of relatively low degree, producing convergence much faster than the condition-number estimate suggests.\n\nIn exact arithmetic, CG terminates after no more than the number of distinct eigenvalues represented in the initial error, and therefore after at most \\(n\\) iterations. Finite precision generally prevents this finite-termination statement from serving as a practical iteration limit."
    },
    {
      "heading": "Preconditioning",
      "text": "A preconditioner changes the system's effective geometry to improve convergence. For standard preconditioned CG (PCG), a fixed symmetric positive-definite matrix \\(M\\) is chosen so that solving\n\n\\[Mz=r\\]\n\nis inexpensive and the transformed system is better conditioned.\n\nThe symmetric transformed problem is\n\n\\[M^{-1/2}AM^{-1/2}y=M^{-1/2}b, \\qquad x=M^{-1/2}y.\\]\n\nAn implementation does not normally construct the square root. Instead, it applies \\(M^{-1}\\) through solves or an equivalent operator.\n\nWith \\(z_k=M^{-1}r_k\\), initialize \\(p_0=z_0\\) and use\n\n\\[\\alpha_k= \\frac{r_k^{\\mathsf T}z_k} {p_k^{\\mathsf T}Ap_k},\\]\n\nfollowed by the usual iterate and residual updates. Then compute\n\n\\[z_{k+1}=M^{-1}r_{k+1},\\]\n\n\\[\\beta_k= \\frac{r_{k+1}^{\\mathsf T}z_{k+1}} {r_k^{\\mathsf T}z_k}, \\qquad p_{k+1}=z_{k+1}+\\beta_kp_k.\\]\n\nTypical choices include diagonal scaling, incomplete Cholesky factorization, domain-decomposition methods, and suitable symmetric multigrid cycles. Their value depends on total computational cost, not iteration count alone: a more accurate preconditioner can be too expensive to construct or apply.\n\nAlthough \\(M^{-1}A\\) generally is not symmetric in the Euclidean inner product, PCG is not simply ordinary CG applied naively to that matrix. Its justification comes from the symmetric transformed problem, or equivalently from an appropriate weighted inner product. Variable or nonsymmetric preconditioning requires additional care and may call for different algorithms."
    },
    {
      "heading": "Computational cost and implementation",
      "text": "A standard iteration requires one matrix–vector product, a few vector updates, and a small number of inner products. For a sparse matrix with \\(s\\) stored nonzero entries, the matrix–vector product typically costs \\(O(s)\\), with vector work costing \\(O(n)\\). Working-vector storage is \\(O(n)\\), in addition to storage for the matrix and preconditioner.\n\nCG is also suitable for matrix-free computation: only a routine that evaluates \\(Av\\) is needed. The entries of \\(A\\) need not be explicitly assembled.\n\nThese features make CG attractive when direct factorization would require excessive memory because of fill-in. Direct methods may nevertheless be preferable for smaller systems, difficult conditioning, or many right-hand sides when factorization costs can be amortized.\n\nIn parallel computing, matrix–vector products and vector updates can often be distributed effectively. Inner products require global reductions, whose communication latency can become a bottleneck. Pipelined and communication-reducing variants seek to address this cost, but their altered recurrences can also affect numerical behavior."
    },
    {
      "heading": "Finite precision and stopping criteria",
      "text": "Under floating-point arithmetic, residual orthogonality and direction conjugacy gradually deteriorate. The recursively updated residual can differ from the true residual:\n\n\\[r_k^{\\mathrm{rec}}\\ne b-Ax_k.\\]\n\nThis residual gap can limit attainable accuracy or cause a stopping test based only on the recursive residual to be misleading. Explicit residual checks and carefully designed residual-replacement strategies can help monitor this behavior.\n\nA common stopping criterion is\n\n\\[\\|b-Ax_k\\|_2 \\le \\mathrm{atol}+\\mathrm{rtol}\\,\\|b\\|_2,\\]\n\nwhere the two tolerances specify absolute and relative residual thresholds.\n\nA small residual does not necessarily imply a comparably small forward error. For \\(b\\ne0\\),\n\n\\[\\frac{\\|x_*-x_k\\|_2}{\\|x_*\\|_2} \\le \\kappa_2(A) \\frac{\\|b-Ax_k\\|_2}{\\|b\\|_2}.\\]\n\nIll-conditioning can therefore amplify the relationship between residual and solution error. Questions of attainable accuracy and numerical stability must be distinguished from the exact-arithmetic convergence theory."
    },
    {
      "heading": "Applications and limitations",
      "text": "CG is widely used for systems arising from discretizations of partial differential equations, including suitable formulations of the Poisson equation and elliptic boundary-value problems. The finite element method often produces symmetric positive-definite systems when the underlying problem and boundary conditions have the required properties.\n\nOther applications include quadratic optimization, computational mechanics, and inner linear solves in Newton-type methods. It also appears in inverse problems, where early termination can act as a form of regularization by limiting the influence of poorly determined spectral components.\n\nStandard CG is not a general solver for arbitrary matrices:\n\n- Symmetric indefinite systems: positive-definite energy minimization no longer applies, and denominators can vanish or become negative. Methods such as MINRES are designed for this setting.\n- Nonsymmetric systems: methods such as GMRES or BiCGSTAB are generally more appropriate.\n- Positive-semidefinite systems: compatible systems can sometimes be handled within the relevant subspace, but uniqueness and the standard positive-definite guarantees require modification.\n- Least-squares problems: CG can be applied to \\(A^{\\mathsf T}Ax=A^{\\mathsf T}b\\) when \\(A\\) has full column rank, but the normal matrix has \\(\\kappa_2(A^{\\mathsf T}A)=\\kappa_2(A)^2\\). Specialized methods such as CGLS and LSQR avoid explicitly forming that matrix, although they do not remove the underlying conditioning difficulty."
    },
    {
      "heading": "Nonlinear conjugate gradient methods",
      "text": "Nonlinear conjugate gradient methods extend the short-recurrence idea to general smooth optimization. They use directions such as\n\n\\[p_{k+1}=-g_{k+1}+\\beta_kp_k, \\qquad g_k=\\nabla f(x_k),\\]\n\nand select step lengths through a line search.\n\nDifferent formulas for \\(\\beta_k\\) define methods including Fletcher–Reeves and Polak–Ribière. For a positive-definite quadratic with exact line searches, standard versions reduce to linear CG. For general nonlinear objectives, exact conjugacy and finite termination are lost; convergence depends on the update formula, line-search conditions, and possible restarts."
    },
    {
      "heading": "Historical development",
      "text": "Magnus Hestenes and Eduard Stiefel introduced the classical method in their 1952 paper Methods of Conjugate Gradients for Solving Linear Systems. Its exact-arithmetic finite-termination property initially connected it with elimination-style approaches, but its lasting importance developed through its use as an iterative method for large sparse systems.\n\nThe growth of scientific computing, sparse matrix techniques, and effective preconditioners made preconditioned conjugate gradients a central method for large-scale numerical simulation."
    }
  ],
  "links": [
    "system-of-linear-equations",
    "positive-definite-matrix",
    "numerical-linear-algebra",
    "objective-function",
    "gradient",
    "convex-optimization",
    "inner-product",
    "gradient-descent",
    "conjugate-transpose",
    "linear-subspace",
    "condition-number",
    "eigenvalues-and-eigenvectors",
    "polynomial",
    "cholesky-decomposition",
    "sparse-matrix",
    "parallel-computing",
    "floating-point-arithmetic",
    "numerical-stability",
    "partial-differential-equation",
    "poisson-equation",
    "finite-element-method",
    "newtons-method",
    "inverse-problem",
    "regularization",
    "line-search"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}