Iteratively reweighted least squares (IRLS) is a family of algorithms that solve an estimation or optimization problem by repeatedly solving a weighted least-squares subproblem. After each fit, the algorithm recalculates weights—and sometimes a working response—from the current parameter estimates. This allows problems with nonquadratic objectives to be treated using efficient least-squares computations. Major applications include robust statistics, fitting generalized linear models, and approximating solutions to problems involving nonquadratic residual or coefficient penalties.
IRLS is not a single algorithm with universal convergence guarantees. Its weights, interpretation, and behavior depend on the objective being optimized and the way the quadratic subproblem is constructed.
Basic construction
Let be a response vector and let be a design matrix, whose th row is . A typical regression problem seeks coefficients .
At iteration , IRLS constructs a diagonal matrix of nonnegative weights,
and a working response . It then solves
When is nonsingular, the solution satisfies
The inverse expression
is useful for describing the method, but numerical implementations generally solve the corresponding linear system rather than explicitly computing an inverse.
In robust linear regression, the working response is usually the original response, , and only the weights change. In generalized linear models, both the weights and the working response generally change.
Difference from ordinary weighted least squares
In a single weighted least-squares fit, weights are treated as fixed. They may, for example, represent known differences in observation variances. In IRLS, the weights depend on the current fitted model.
That distinction matters statistically. A weight can represent estimated inverse variance, local curvature of an objective, or reduced influence of a large residual. These interpretations are not interchangeable. IRLS used for likelihood fitting is not automatically a robust procedure merely because its weights vary between iterations.
Robust regression
From a loss function to weights
Consider the objective
where is a fixed residual scale and is a loss function. Minimizing such an objective defines a regression M-estimator.
If is differentiable, write
The first-order stationarity condition is
Define
When the limit exists, the weight at zero is defined by continuity. Because
the stationarity condition becomes
IRLS freezes these parameter-dependent weights at the current iterate:
then fits weighted least squares to obtain the next iterate.
This derivation shows that a fixed point of the iteration satisfies the estimating equations, subject to the stated differentiability and limiting conditions. It does not, by itself, prove convergence or show that the fixed point is a global minimum.
Examples of robust weights
For the quadratic loss
the score is , so every weight is one. IRLS reduces to ordinary least squares in a single fit.
For the Huber loss,
the weights are
Small residuals receive ordinary least-squares treatment, while large residuals receive less weight.
Some robust losses have a score that eventually returns to zero as the residual magnitude increases. Their IRLS weights can therefore become zero for sufficiently large residuals. Such losses are generally nonconvex and may produce multiple local minima.
Scale and leverage
The residual scale determines what counts as a large residual. It can be supplied in advance or estimated during fitting. If scale is updated, the procedure is no longer simply an iteration for the fixed-scale objective above; its behavior depends on the scale-estimation rule as well.
Residual-based downweighting also has a limitation: an observation with an unusual predictor vector can pull the fitted model toward itself and consequently have a small residual. Standard residual-weighted IRLS does not, on its own, provide protection against all high-leverage contamination.
Generalized linear models
In generalized linear models, IRLS commonly implements Fisher scoring, a likelihood-based update that uses Fisher information in place of the observed curvature used by Newton's method.
Let
where is the link function. Suppose
with dispersion parameter , variance function , and known positive observation weights .
At the current estimate, define the working response
and working weight
A weighted least-squares fit of on gives the Fisher-scoring update for the usual independent-observation generalized linear model. The working response results from a first-order linearization of the relationship between the response mean and the linear predictor. The weights account for both response variance and the sensitivity of the link transformation.
A common dispersion factor multiplies every weight by the same constant and therefore does not affect the unpenalized coefficient update. It does matter for uncertainty calculations and for the relative strength of an unscaled penalty.
Logistic regression
For binary logistic regression,
With unit observation weights,
and
For this canonical-link model, the Fisher-scoring and Newton updates coincide. Probabilities close to zero or one produce small weights and can cause numerical difficulties. Under complete or certain forms of partial separation, a finite unpenalized maximum-likelihood estimate does not exist, so coefficient growth need not indicate a defect in the linear solver.
Poisson regression
For a response with a Poisson distribution and a log link,
The unit-weight IRLS quantities are
These likelihood weights have a different purpose from robust residual weights: their role is to reproduce the scoring update, not to identify or suppress outlying observations.
Nonquadratic residual norms and penalties
IRLS also applies to objectives involving powers of residual magnitudes. For example,
Away from zero residuals, the corresponding weights are proportional to
For , these weights become unbounded near zero. A common approach instead minimizes a smoothed objective,
using weights proportional to
The smoothing parameter changes the objective; it is not merely a numerical detail. For , the unsmoothed objective is nonconvex, and global optimality is generally not guaranteed.
Reweighting can also be applied to coefficients rather than observations. Iteratively reweighted quadratic penalties can approximate absolute-value or other nonquadratic penalties, including those related to lasso regression. Observation weights modify the residual term, whereas coefficient weights modify the penalty.
With ridge regularization, a subproblem may take the form
leading to
The matrix specifies which coefficient combinations are penalized; an intercept can be left unpenalized.
Why some IRLS schemes decrease the objective
A useful theoretical interpretation is majorization–minimization. Suppose a scalar residual loss can be written
where is differentiable, concave, and nondecreasing. Concavity gives
The right-hand side is a quadratic upper bound in , touching the original loss at .
Minimizing the sum of these bounds produces a weighted least-squares step with weights proportional to . If the subproblem is solved exactly, this construction ensures that the original objective does not increase.
Not every IRLS algorithm has this property. A quadratic approximation based on local curvature need not be a global upper bound. Consequently:
- A full update may increase the objective or decrease the likelihood.
- A line search or step-halving procedure may be needed.
- Monotonic objective values do not alone establish convergence of the parameters.
- Convergence to a stationary point does not imply global optimality for a nonconvex objective.
For convex objectives, stationary solutions are global minimizers under the usual differentiability or subgradient conditions, but uniqueness still depends on the design and loss.
Numerical implementation
Solving the weighted subproblem
With nonnegative weights, the subproblem can be expressed as ordinary least squares on transformed data:
A QR decomposition solves this problem without forming . A singular value decomposition is useful when the transformed design is rank deficient or nearly so.
Forming normal equations can worsen conditioning because, for a full-column-rank transformed design, their spectral condition number is the square of that of . Zero weights may remove enough information to destroy rank, while a wide range of positive weights can worsen numerical stability.
For large or sparse problems, iterative linear solvers and sparse factorizations can avoid dense matrix formation. Exact solution of every early subproblem may be unnecessary, but inexact solves must be controlled if convergence guarantees are required.
Initialization and stopping
A typical implementation:
- Chooses initial coefficients and any required scale or dispersion estimates.
- Computes residuals or fitted means.
- Updates weights and, where required, the working response.
- Solves the weighted least-squares subproblem.
- Applies damping if needed.
- Tests convergence and repeats.
Stopping tests may examine relative parameter change, objective change, or the norm of the estimating equations. Using more than one diagnostic helps distinguish genuine convergence from stagnation caused by extreme weights or an ill-conditioned system.
Under a dense implementation with , a weighted least-squares solve commonly requires work on the order of , although actual costs depend strongly on sparsity and the solver.
Applications and interpretation
IRLS links classical least-squares computation to several broader areas: robust regression, generalized linear modeling, signal reconstruction, and inverse problems with nonquadratic data-fidelity terms or penalties. Its practical appeal is that a difficult objective can often be handled by repeatedly using established linear-algebra routines.
The final weighted least-squares problem is nevertheless only one stage of the original estimation procedure. Its ordinary covariance formula is not automatically valid for the resulting estimator. Likelihood-based fits require uncertainty calculations appropriate to the likelihood, while robust M-estimators generally require formulas that account for their estimating equations and any estimated scale. Algorithmic weights alone do not determine the sampling uncertainty of the fitted coefficients.