{
  "id": "gauss-newton-algorithm",
  "title": "Gauss–Newton Algorithm",
  "lang": "en",
  "source_lang": "en",
  "summary": "The Gauss–Newton algorithm solves nonlinear least-squares problems by repeatedly linearizing residuals and solving the resulting linear least-squares problem.",
  "category": "mathematics",
  "subcategory": "applied",
  "type": "concept",
  "aliases": [
    "Gauss–Newton method",
    "Gauss–Newton iteration"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/gauss-newton-algorithm/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/gauss-newton-algorithm/",
      "json": "https://aiwiki.page/zh/gauss-newton-algorithm.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The Gauss–Newton algorithm is an iterative method in mathematical optimization for minimizing a sum of squared residuals. It exploits the structure of nonlinear least-squares problems by linearizing the residual functions at the current parameter estimate and solving the resulting least-squares problem. Unlike a full second-order method, it does not require second derivatives of the residuals. The algorithm is especially effective when the residuals are small near a solution and their derivatives provide enough information to identify the parameters."
    },
    {
      "heading": "Problem formulation",
      "text": "Let \\(x\\in\\mathbb{R}^n\\) be a vector of unknown parameters and let\n\n\\[r(x)= \\begin{bmatrix} r_1(x)&\\cdots&r_m(x) \\end{bmatrix}^{\\mathsf T}\\]\n\nbe a vector of differentiable residual functions. The objective is\n\n\\[\\min_{x\\in\\mathbb{R}^n} F(x), \\qquad F(x)=\\frac12\\|r(x)\\|_2^2 =\\frac12\\sum_{i=1}^{m}r_i(x)^2.\\]\n\nThe factor \\(1/2\\) simplifies derivatives without changing the minimizers. A common application is fitting a model \\(f(t;x)\\) to observations \\((t_i,y_i)\\), with\n\n\\[r_i(x)=f(t_i;x)-y_i.\\]\n\nChoosing the opposite sign for every residual gives the same objective and, when derivatives are defined consistently, the same parameter updates.\n\nUsually \\(m\\geq n\\), so there are at least as many residuals as unknown parameters. The standard uniquely determined step requires the residual Jacobian matrix\n\n\\[J(x)_{ij}=\\frac{\\partial r_i(x)}{\\partial x_j}\\]\n\nto have full column rank."
    },
    {
      "heading": "Derivation",
      "text": "Linearizing the residuals\n\nAt the current iterate \\(x_k\\), approximate the residual vector by its first-order expansion:\n\n\\[r(x_k+p)\\approx r(x_k)+J(x_k)p.\\]\n\nWriting \\(r_k=r(x_k)\\) and \\(J_k=J(x_k)\\), the Gauss–Newton step solves\n\n\\[p_k\\in\\operatorname*{arg\\,min}_{p} \\frac12\\|r_k+J_kp\\|_2^2.\\]\n\nThe normal equations for this linear least-squares problem are\n\n\\[J_k^{\\mathsf T}J_kp_k=-J_k^{\\mathsf T}r_k.\\]\n\nThe basic iteration then takes the full step:\n\n\\[x_{k+1}=x_k+p_k.\\]\n\nAlthough the step can formally be written using the inverse of \\(J_k^{\\mathsf T}J_k\\), practical implementations solve a linear system or least-squares problem rather than explicitly forming that inverse.\n\nRelationship to Newton’s method\n\nFor twice differentiable residuals, the gradient and Hessian of \\(F\\) are\n\n\\[\\nabla F(x)=J(x)^{\\mathsf T}r(x)\\]\n\nand\n\n\\[\\nabla^2F(x) = J(x)^{\\mathsf T}J(x) + \\sum_{i=1}^{m}r_i(x)\\nabla^2r_i(x).\\]\n\nNewton’s method uses the entire Hessian to compute a step. Gauss–Newton instead uses the approximation\n\n\\[\\nabla^2F(x)\\approx J(x)^{\\mathsf T}J(x),\\]\n\ndiscarding the residual-weighted second-derivative term.\n\nThis approximation is attractive for two reasons:\n\n- It requires only residuals and their first derivatives.\n- The approximate Hessian is positive semidefinite and becomes positive definite when \\(J(x)\\) has full column rank.\n\nThe approximation is most accurate when the residual-weighted second-derivative term is small. This can occur because the residuals are small, because the residual functions are nearly linear, or because their curvature contributions cancel."
    },
    {
      "heading": "Basic procedure",
      "text": "A standard implementation follows these steps:\n\n- Choose an initial parameter vector \\(x_0\\).\n- Evaluate \\(r_k\\) and \\(J_k\\).\n- Solve the linear least-squares subproblem for \\(p_k\\).\n- Set \\(x_{k+1}=x_k+p_k\\), or use a safeguarded step.\n- Repeat until a stopping condition is satisfied.\n\nTypical stopping tests examine the gradient norm\n\n\\[\\|J_k^{\\mathsf T}r_k\\|,\\]\n\nthe step size relative to the scale of \\(x_k\\), or the change in the objective.\n\nThese tests detect different phenomena. A small gradient indicates approximate first-order stationarity; a small step can instead result from poor scaling or numerical difficulties. Implementations therefore commonly combine several tests and impose an iteration or evaluation limit.\n\nResidual derivatives can be computed analytically, through automatic differentiation, or approximately by finite differences. Derivative errors can significantly affect convergence, particularly near a solution."
    },
    {
      "heading": "Numerical solution of the step",
      "text": "Normal equations and conditioning\n\nIf \\(J_k\\) has full column rank, the normal equations have a unique solution. However, explicitly forming \\(J_k^{\\mathsf T}J_k\\) can reduce numerical stability. In the Euclidean norm,\n\n\\[\\kappa_2(J_k^{\\mathsf T}J_k)=\\kappa_2(J_k)^2,\\]\n\nwhere \\(\\kappa_2\\) is the spectral condition number.\n\nConsequently, a moderately ill-conditioned Jacobian can lead to a much more ill-conditioned normal-equation system. Forming the product can also increase storage requirements for sparse problems.\n\nA QR decomposition solves the linear least-squares problem without forming the normal equations and is a common choice for dense problems. A singular value decomposition provides additional information about near-dependencies among parameter directions and can produce a minimum-norm step when the Jacobian is rank deficient.\n\nRank deficiency and scaling\n\nWhen \\(J_k\\) lacks full column rank, multiple steps may minimize the linearized objective. A minimum-norm solution is one possible choice, but it does not by itself resolve a lack of information about the model parameters.\n\nScaling also matters. Parameters with very different units or magnitudes can make the linearized problem difficult to solve accurately and can distort step-size tests. Residual weighting and parameter rescaling play different roles: weighting changes how discrepancies are valued, whereas an invertible reparameterization can improve numerical behavior without changing the underlying fit."
    },
    {
      "heading": "Convergence and limitations",
      "text": "Local convergence\n\nGauss–Newton is primarily a local method: convergence depends on the initial estimate and the behavior of the residual functions near a solution.\n\nLet \\(x_*\\) be a stationary point, so that\n\n\\[J(x_*)^{\\mathsf T}r(x_*)=0.\\]\n\nIf \\(J(x_*)\\) has full column rank, define\n\n\\[S_*=\\sum_{i=1}^{m}r_i(x_*)\\nabla^2r_i(x_*).\\]\n\nUnder suitable smoothness conditions, the derivative of the full-step iteration map at \\(x_*\\) is\n\n\\[-\\bigl(J(x_*)^{\\mathsf T}J(x_*)\\bigr)^{-1}S_*.\\]\n\nThis expression explains the method’s characteristic behavior:\n\n- Zero-residual solutions: If \\(r(x_*)=0\\), then \\(S_*=0\\). With a locally Lipschitz Jacobian and full column rank at the solution, sufficiently close initial estimates yield local quadratic convergence.\n- Nonzero-residual solutions: Convergence is generally linear when the iteration map is locally contractive. Small residuals often improve the rate, but smallness must be assessed relative to Jacobian conditioning and residual curvature.\n- Large curvature contributions: If the discarded term is too influential, the full-step iteration may fail to converge even near a stationary point.\n\nA zero residual is sufficient for the discarded term to vanish, but it is not necessary.\n\nDescent does not guarantee a successful full step\n\nWhen \\(J_k\\) has full column rank and the gradient is nonzero,\n\n\\[\\nabla F(x_k)^{\\mathsf T}p_k = -p_k^{\\mathsf T}J_k^{\\mathsf T}J_kp_k = -\\|J_kp_k\\|_2^2<0.\\]\n\nThus the Gauss–Newton direction is a descent direction.\n\nNevertheless, the full update \\(x_k+p_k\\) can increase the objective because the linearized residual model may be inaccurate over a large step. A descent direction guarantees a decrease for sufficiently small positive step lengths, not necessarily for a unit step.\n\nOther limitations include sensitivity to initialization, convergence to nonglobal stationary points, singular Jacobians, and the sensitivity of squared-error objectives to outlying observations."
    },
    {
      "heading": "Safeguarded variants",
      "text": "Damped Gauss–Newton\n\nA damped iteration uses\n\n\\[x_{k+1}=x_k+\\alpha_kp_k, \\qquad 0<\\alpha_k\\leq1.\\]\n\nA line search chooses \\(\\alpha_k\\) to obtain sufficient decrease in the objective. This can improve robustness when the full step is too large.\n\nLevenberg–Marquardt\n\nThe Levenberg–Marquardt algorithm modifies the step equation to\n\n\\[\\bigl(J_k^{\\mathsf T}J_k+\\lambda_kD_k^{\\mathsf T}D_k\\bigr)p_k = -J_k^{\\mathsf T}r_k,\\]\n\nwhere \\(\\lambda_k\\geq0\\) is a damping parameter and \\(D_k\\) is a scaling matrix.\n\nFor small damping, the step resembles Gauss–Newton. With \\(D_k=I\\) and large damping, it approaches a scaled negative-gradient step. Appropriate damping also makes the linear system nonsingular when the Jacobian is rank deficient.\n\nTrust-region methods\n\nA trust-region method restricts the step to a region where the linearized residual model is considered reliable:\n\n\\[\\min_p \\frac12\\|r_k+J_kp\\|_2^2 \\quad\\text{subject to}\\quad \\|D_kp\\|_2\\leq\\Delta_k.\\]\n\nThe radius \\(\\Delta_k\\) is adjusted by comparing actual objective reduction with predicted reduction. Levenberg–Marquardt is closely related to this formulation.\n\nAdaptive damping should be distinguished from regularization of the original estimation problem. A damping term controls an optimization step; a persistent penalty in the objective changes the problem being solved."
    },
    {
      "heading": "Example: fitting an exponential model",
      "text": "Consider observations \\((t_i,y_i)\\) and the model\n\n\\[f(t_i;a,b)=ae^{bt_i}.\\]\n\nWith \\(x=(a,b)^{\\mathsf T}\\), the residuals are\n\n\\[r_i(a,b)=ae^{bt_i}-y_i,\\]\n\nand the \\(i\\)-th Jacobian row is\n\n\\[J_i(a,b)= \\begin{bmatrix} e^{bt_i}&at_i e^{bt_i} \\end{bmatrix}.\\]\n\nAt each iteration, the algorithm solves\n\n\\[\\min_{\\Delta a,\\Delta b} \\sum_{i=1}^{m} \\left[ ae^{bt_i}-y_i +e^{bt_i}\\Delta a +at_i e^{bt_i}\\Delta b \\right]^2\\]\n\nand updates \\(a\\) and \\(b\\).\n\nThe original problem is nonlinear because \\(b\\) appears inside the exponential, but each step is a linear least-squares problem in \\(\\Delta a\\) and \\(\\Delta b\\). If \\(a=0\\), the Jacobian column associated with \\(b\\) vanishes, illustrating how the current parameter estimate can cause rank deficiency.\n\nFor a model linear in all parameters, the linearization is exact. With a full-column-rank design matrix, one full Gauss–Newton step reaches the least-squares solution from any starting point."
    },
    {
      "heading": "Weighted least squares and statistical interpretation",
      "text": "For a fixed symmetric positive-definite weight matrix \\(W\\), the objective becomes\n\n\\[F(x)=\\frac12r(x)^{\\mathsf T}Wr(x).\\]\n\nThe corresponding step satisfies\n\n\\[J_k^{\\mathsf T}WJ_kp_k=-J_k^{\\mathsf T}Wr_k.\\]\n\nIf \\(W=L^{\\mathsf T}L\\), this is ordinary Gauss–Newton applied to the transformed residual vector \\(Lr(x)\\).\n\nIn a statistical model with additive Gaussian errors and a known, parameter-independent covariance matrix \\(\\Sigma\\), choosing \\(W=\\Sigma^{-1}\\) makes weighted least squares equivalent to maximum likelihood estimation for the mean parameters.\n\nNear a well-identified solution, the matrix\n\n\\[\\bigl(J^{\\mathsf T}\\Sigma^{-1}J\\bigr)^{-1}\\]\n\nalso appears as a local approximation to parameter-estimation covariance under suitable regularity and linearization assumptions. This interpretation is not automatically valid for arbitrary residual functions or misspecified error models. If the error covariance depends on the parameters, the likelihood generally includes additional terms and is not simply a fixed weighted least-squares objective."
    },
    {
      "heading": "Applications and historical context",
      "text": "Gauss–Newton is widely used in nonlinear regression, scientific model fitting, calibration, and inverse problems. Large-scale versions exploit sparse Jacobians or compute Jacobian–vector products without storing the entire matrix.\n\nThe method is named after Carl Friedrich Gauss and Isaac Newton. Its defining construction combines the least-squares principle associated with Gauss with a Newton-type local iteration, replacing the exact objective Hessian by a matrix derived from first derivatives of the residuals."
    }
  ],
  "links": [
    "mathematical-optimization",
    "least-squares",
    "jacobian-matrix",
    "gradient",
    "hessian-matrix",
    "newtons-method",
    "positive-definite-matrix",
    "automatic-differentiation",
    "numerical-stability",
    "condition-number",
    "qr-decomposition",
    "singular-value-decomposition",
    "line-search",
    "regularization",
    "covariance-matrix",
    "maximum-likelihood-estimation",
    "calibration",
    "inverse-problem",
    "carl-friedrich-gauss"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}