{
  "id": "iteratively-reweighted-least-squares",
  "title": "Iteratively Reweighted Least Squares",
  "lang": "en",
  "source_lang": "en",
  "summary": "Iteratively reweighted least squares solves estimation and optimization problems through repeated weighted least-squares fits, with weights updated from the current solution.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "IRLS",
    "Iterative reweighted least squares"
  ],
  "generated_at": "2026-11-06",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/iteratively-reweighted-least-squares/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/iteratively-reweighted-least-squares/",
      "json": "https://aiwiki.page/zh/iteratively-reweighted-least-squares.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Iteratively reweighted least squares (IRLS) is a family of algorithms that solve an estimation or optimization problem by repeatedly solving a weighted least-squares subproblem. After each fit, the algorithm recalculates weights—and sometimes a working response—from the current parameter estimates. This allows problems with nonquadratic objectives to be treated using efficient least-squares computations. Major applications include robust statistics, fitting generalized linear models, and approximating solutions to problems involving nonquadratic residual or coefficient penalties.\n\nIRLS is not a single algorithm with universal convergence guarantees. Its weights, interpretation, and behavior depend on the objective being optimized and the way the quadratic subproblem is constructed."
    },
    {
      "heading": "Basic construction",
      "text": "Let \\(y\\in\\mathbb{R}^n\\) be a response vector and let \\(X\\in\\mathbb{R}^{n\\times p}\\) be a design matrix, whose \\(i\\)th row is \\(x_i^\\top\\). A typical regression problem seeks coefficients \\(\\beta\\in\\mathbb{R}^p\\).\n\nAt iteration \\(t\\), IRLS constructs a diagonal matrix of nonnegative weights,\n\n\\[W^{(t)}=\\operatorname{diag}\\!\\left(w_1^{(t)},\\ldots,w_n^{(t)}\\right),\\]\n\nand a working response \\(z^{(t)}\\). It then solves\n\n\\[\\beta^{(t+1)} \\in \\arg\\min_\\beta \\frac12 \\left(z^{(t)}-X\\beta\\right)^\\top W^{(t)} \\left(z^{(t)}-X\\beta\\right).\\]\n\nWhen \\(X^\\top W^{(t)}X\\) is nonsingular, the solution satisfies\n\n\\[X^\\top W^{(t)}X\\,\\beta^{(t+1)} = X^\\top W^{(t)}z^{(t)}.\\]\n\nThe inverse expression\n\n\\[\\beta^{(t+1)} = \\left(X^\\top W^{(t)}X\\right)^{-1} X^\\top W^{(t)}z^{(t)}\\]\n\nis useful for describing the method, but numerical implementations generally solve the corresponding linear system rather than explicitly computing an inverse.\n\nIn robust linear regression, the working response is usually the original response, \\(z^{(t)}=y\\), and only the weights change. In generalized linear models, both the weights and the working response generally change.\n\nDifference from ordinary weighted least squares\n\nIn a single weighted least-squares fit, weights are treated as fixed. They may, for example, represent known differences in observation variances. In IRLS, the weights depend on the current fitted model.\n\nThat distinction matters statistically. A weight can represent estimated inverse variance, local curvature of an objective, or reduced influence of a large residual. These interpretations are not interchangeable. IRLS used for likelihood fitting is not automatically a robust procedure merely because its weights vary between iterations."
    },
    {
      "heading": "Robust regression",
      "text": "From a loss function to weights\n\nConsider the objective\n\n\\[F(\\beta) = \\sum_{i=1}^{n} \\rho\\!\\left(u_i(\\beta)\\right), \\qquad u_i(\\beta)=\\frac{y_i-x_i^\\top\\beta}{s},\\]\n\nwhere \\(s>0\\) is a fixed residual scale and \\(\\rho\\) is a loss function. Minimizing such an objective defines a regression M-estimator.\n\nIf \\(\\rho\\) is differentiable, write\n\n\\[\\psi(u)=\\rho'(u).\\]\n\nThe first-order stationarity condition is\n\n\\[\\sum_{i=1}^{n} x_i\\,\\psi(u_i)=0.\\]\n\nDefine\n\n\\[w(u)=\\frac{\\psi(u)}{u}, \\qquad u\\ne0.\\]\n\nWhen the limit exists, the weight at zero is defined by continuity. Because\n\n\\[\\psi(u)=w(u)u,\\]\n\nthe stationarity condition becomes\n\n\\[X^\\top W(\\beta)(y-X\\beta)=0.\\]\n\nIRLS freezes these parameter-dependent weights at the current iterate:\n\n\\[w_i^{(t)} = w\\!\\left(\\frac{y_i-x_i^\\top\\beta^{(t)}}{s}\\right),\\]\n\nthen fits weighted least squares to obtain the next iterate.\n\nThis derivation shows that a fixed point of the iteration satisfies the estimating equations, subject to the stated differentiability and limiting conditions. It does not, by itself, prove convergence or show that the fixed point is a global minimum.\n\nExamples of robust weights\n\nFor the quadratic loss\n\n\\[\\rho(u)=\\frac12u^2,\\]\n\nthe score is \\(\\psi(u)=u\\), so every weight is one. IRLS reduces to ordinary least squares in a single fit.\n\nFor the Huber loss,\n\n\\[\\rho_c(u)= \\begin{cases} \\frac12u^2, & |u|\\le c,\\\\[2mm] c|u|-\\frac12c^2, & |u|>c, \\end{cases} \\qquad c>0,\\]\n\nthe weights are\n\n\\[w(u)= \\begin{cases} 1, & |u|\\le c,\\\\[2mm] c/|u|, & |u|>c. \\end{cases}\\]\n\nSmall residuals receive ordinary least-squares treatment, while large residuals receive less weight.\n\nSome robust losses have a score that eventually returns to zero as the residual magnitude increases. Their IRLS weights can therefore become zero for sufficiently large residuals. Such losses are generally nonconvex and may produce multiple local minima.\n\nScale and leverage\n\nThe residual scale \\(s\\) determines what counts as a large residual. It can be supplied in advance or estimated during fitting. If scale is updated, the procedure is no longer simply an iteration for the fixed-scale objective above; its behavior depends on the scale-estimation rule as well.\n\nResidual-based downweighting also has a limitation: an observation with an unusual predictor vector can pull the fitted model toward itself and consequently have a small residual. Standard residual-weighted IRLS does not, on its own, provide protection against all high-leverage contamination."
    },
    {
      "heading": "Generalized linear models",
      "text": "In generalized linear models, IRLS commonly implements Fisher scoring, a likelihood-based update that uses Fisher information in place of the observed curvature used by Newton's method.\n\nLet\n\n\\[\\mu_i=\\mathbb{E}[Y_i], \\qquad \\eta_i=x_i^\\top\\beta, \\qquad g(\\mu_i)=\\eta_i,\\]\n\nwhere \\(g\\) is the link function. Suppose\n\n\\[\\operatorname{Var}(Y_i)=\\frac{\\phi V(\\mu_i)}{a_i},\\]\n\nwith dispersion parameter \\(\\phi>0\\), variance function \\(V\\), and known positive observation weights \\(a_i\\).\n\nAt the current estimate, define the working response\n\n\\[z_i^{(t)} = \\eta_i^{(t)} + \\left(y_i-\\mu_i^{(t)}\\right) g'\\!\\left(\\mu_i^{(t)}\\right)\\]\n\nand working weight\n\n\\[w_i^{(t)} = \\frac{a_i} {\\phi V(\\mu_i^{(t)}) \\left[g'(\\mu_i^{(t)})\\right]^2}.\\]\n\nA weighted least-squares fit of \\(z^{(t)}\\) on \\(X\\) gives the Fisher-scoring update for the usual independent-observation generalized linear model. The working response results from a first-order linearization of the relationship between the response mean and the linear predictor. The weights account for both response variance and the sensitivity of the link transformation.\n\nA common dispersion factor multiplies every weight by the same constant and therefore does not affect the unpenalized coefficient update. It does matter for uncertainty calculations and for the relative strength of an unscaled penalty.\n\nLogistic regression\n\nFor binary logistic regression,\n\n\\[\\mu_i=\\frac{1}{1+\\exp(-\\eta_i)}, \\qquad \\operatorname{Var}(Y_i)=\\mu_i(1-\\mu_i).\\]\n\nWith unit observation weights,\n\n\\[w_i=\\mu_i(1-\\mu_i)\\]\n\nand\n\n\\[z_i = \\eta_i+\\frac{y_i-\\mu_i}{\\mu_i(1-\\mu_i)}.\\]\n\nFor this canonical-link model, the Fisher-scoring and Newton updates coincide. Probabilities close to zero or one produce small weights and can cause numerical difficulties. Under complete or certain forms of partial separation, a finite unpenalized maximum-likelihood estimate does not exist, so coefficient growth need not indicate a defect in the linear solver.\n\nPoisson regression\n\nFor a response with a Poisson distribution and a log link,\n\n\\[\\mu_i=\\exp(\\eta_i), \\qquad V(\\mu_i)=\\mu_i.\\]\n\nThe unit-weight IRLS quantities are\n\n\\[w_i=\\mu_i, \\qquad z_i=\\eta_i+\\frac{y_i-\\mu_i}{\\mu_i}.\\]\n\nThese likelihood weights have a different purpose from robust residual weights: their role is to reproduce the scoring update, not to identify or suppress outlying observations."
    },
    {
      "heading": "Nonquadratic residual norms and penalties",
      "text": "IRLS also applies to objectives involving powers of residual magnitudes. For example,\n\n\\[F_p(\\beta) = \\sum_i |y_i-x_i^\\top\\beta|^p.\\]\n\nAway from zero residuals, the corresponding weights are proportional to\n\n\\[w_i^{(t)} = |y_i-x_i^\\top\\beta^{(t)}|^{p-2}.\\]\n\nFor \\(1\\le p<2\\), these weights become unbounded near zero. A common approach instead minimizes a smoothed objective,\n\n\\[F_{p,\\varepsilon}(\\beta) = \\sum_i \\left[(y_i-x_i^\\top\\beta)^2+\\varepsilon^2\\right]^{p/2}, \\qquad \\varepsilon>0,\\]\n\nusing weights proportional to\n\n\\[w_i^{(t)} = \\left[ (y_i-x_i^\\top\\beta^{(t)})^2+\\varepsilon^2 \\right]^{p/2-1}.\\]\n\nThe smoothing parameter changes the objective; it is not merely a numerical detail. For \\(0<p<1\\), the unsmoothed objective is nonconvex, and global optimality is generally not guaranteed.\n\nReweighting can also be applied to coefficients rather than observations. Iteratively reweighted quadratic penalties can approximate absolute-value or other nonquadratic penalties, including those related to lasso regression. Observation weights modify the residual term, whereas coefficient weights modify the penalty.\n\nWith ridge regularization, a subproblem may take the form\n\n\\[\\min_\\beta \\frac12(z-X\\beta)^\\top W(z-X\\beta) + \\frac{\\lambda}{2}\\|L\\beta\\|_2^2,\\]\n\nleading to\n\n\\[(X^\\top WX+\\lambda L^\\top L)\\beta=X^\\top Wz.\\]\n\nThe matrix \\(L\\) specifies which coefficient combinations are penalized; an intercept can be left unpenalized."
    },
    {
      "heading": "Why some IRLS schemes decrease the objective",
      "text": "A useful theoretical interpretation is majorization–minimization. Suppose a scalar residual loss can be written\n\n\\[\\rho(r)=h(r^2),\\]\n\nwhere \\(h\\) is differentiable, concave, and nondecreasing. Concavity gives\n\n\\[h(r^2) \\le h(r_t^2)+h'(r_t^2)(r^2-r_t^2).\\]\n\nThe right-hand side is a quadratic upper bound in \\(r\\), touching the original loss at \\(r=r_t\\).\n\nMinimizing the sum of these bounds produces a weighted least-squares step with weights proportional to \\(h'(r_t^2)\\). If the subproblem is solved exactly, this construction ensures that the original objective does not increase.\n\nNot every IRLS algorithm has this property. A quadratic approximation based on local curvature need not be a global upper bound. Consequently:\n\n- A full update may increase the objective or decrease the likelihood.\n- A line search or step-halving procedure may be needed.\n- Monotonic objective values do not alone establish convergence of the parameters.\n- Convergence to a stationary point does not imply global optimality for a nonconvex objective.\n\nFor convex objectives, stationary solutions are global minimizers under the usual differentiability or subgradient conditions, but uniqueness still depends on the design and loss."
    },
    {
      "heading": "Numerical implementation",
      "text": "Solving the weighted subproblem\n\nWith nonnegative weights, the subproblem can be expressed as ordinary least squares on transformed data:\n\n\\[\\widetilde X=W^{1/2}X, \\qquad \\widetilde z=W^{1/2}z.\\]\n\nA QR decomposition solves this problem without forming \\(X^\\top WX\\). A singular value decomposition is useful when the transformed design is rank deficient or nearly so.\n\nForming normal equations can worsen conditioning because, for a full-column-rank transformed design, their spectral condition number is the square of that of \\(\\widetilde X\\). Zero weights may remove enough information to destroy rank, while a wide range of positive weights can worsen numerical stability.\n\nFor large or sparse problems, iterative linear solvers and sparse factorizations can avoid dense matrix formation. Exact solution of every early subproblem may be unnecessary, but inexact solves must be controlled if convergence guarantees are required.\n\nInitialization and stopping\n\nA typical implementation:\n\n- Chooses initial coefficients and any required scale or dispersion estimates.\n- Computes residuals or fitted means.\n- Updates weights and, where required, the working response.\n- Solves the weighted least-squares subproblem.\n- Applies damping if needed.\n- Tests convergence and repeats.\n\nStopping tests may examine relative parameter change, objective change, or the norm of the estimating equations. Using more than one diagnostic helps distinguish genuine convergence from stagnation caused by extreme weights or an ill-conditioned system.\n\nUnder a dense implementation with \\(n\\ge p\\), a weighted least-squares solve commonly requires work on the order of \\(np^2+p^3\\), although actual costs depend strongly on sparsity and the solver."
    },
    {
      "heading": "Applications and interpretation",
      "text": "IRLS links classical least-squares computation to several broader areas: robust regression, generalized linear modeling, signal reconstruction, and inverse problems with nonquadratic data-fidelity terms or penalties. Its practical appeal is that a difficult objective can often be handled by repeatedly using established linear-algebra routines.\n\nThe final weighted least-squares problem is nevertheless only one stage of the original estimation procedure. Its ordinary covariance formula is not automatically valid for the resulting estimator. Likelihood-based fits require uncertainty calculations appropriate to the likelihood, while robust M-estimators generally require formulas that account for their estimating equations and any estimated scale. Algorithmic weights alone do not determine the sampling uncertainty of the fitted coefficients."
    }
  ],
  "links": [
    "mathematical-optimization",
    "weighted-least-squares",
    "generalized-linear-model",
    "matrix",
    "system-of-linear-equations",
    "loss-function",
    "ordinary-least-squares",
    "fisher-information",
    "newtons-method",
    "logistic-regression",
    "maximum-likelihood-estimation",
    "poisson-distribution",
    "lasso-regression",
    "ridge-regression",
    "line-search",
    "qr-decomposition",
    "singular-value-decomposition",
    "condition-number",
    "numerical-stability",
    "inverse-problem"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}