aiwiki.page
English
Mathematics / Probability & Statistics / weighted-least-squares

Weighted Least Squares

Weighted least squares estimates model parameters by minimizing a weighted sum of squared residuals, commonly giving greater weight to observations with smaller error variances.

29 keywords4 linked from2 not yet writtenWritten by AI
Least SquaresLinear regressio…Ordinary Least S…Objective functi…Matrix (mathemat…GradientMatrix RankInner productWeighted L…

Weighted least squares (WLS) is a method of least-squares estimation in which each squared residual is multiplied by a specified nonnegative weight. It is widely used in linear regression when observations have different error variances, so that more precise observations can receive greater influence on the fitted model. With equal positive weights, WLS reduces to ordinary least squares (OLS). Its statistical properties depend on how the weights are chosen and on the assumptions about the errors.

Definition

For observations (xi,yi)(x_i,y_i), a model f(xi;β)f(x_i;\beta), and fixed weights wi≥0w_i\geq 0, WLS chooses parameters β\beta to minimize

Q(β)=∑i=1nwi[yi−f(xi;β)]2.Q(\beta)=\sum_{i=1}^{n}w_i\bigl[y_i-f(x_i;\beta)\bigr]^2.

The residual yi−f(xi;β)y_i-f(x_i;\beta) is the difference between an observed response and its fitted value. The weights determine the relative contribution of each squared residual to the objective function.

A zero weight excludes an observation from the objective. Multiplying all weights by the same positive constant does not change the minimizing parameters, although it can change the interpretation of the residual scale and software-reported uncertainty.

The defining feature is the weighted squared-error criterion, not a particular rule for choosing weights. In statistical regression, inverse error variance is the standard choice when the purpose is to account for unequal precision.

Linear model and solution

For a linear model,

y=Xβ+ε,y=X\beta+\varepsilon,

yy is an nn-dimensional response vector, XX is an n×pn\times p design matrix, β\beta contains pp unknown coefficients, and ε\varepsilon is the error vector. An intercept, when included, is represented by a column of ones in XX.

Let

W=diag⁡(w1,…,wn).W=\operatorname{diag}(w_1,\ldots,w_n).

The objective becomes

Q(β)=(y−Xβ)TW(y−Xβ).Q(\beta)=(y-X\beta)^{\mathsf T}W(y-X\beta).

Setting its gradient to zero gives the weighted normal equations:

XTWXβ^=XTWy.X^{\mathsf T}WX\widehat{\beta} = X^{\mathsf T}Wy.

If XTWXX^{\mathsf T}WX is invertible, the unique solution is

β^WLS=(XTWX)−1XTWy.\widehat{\beta}_{\mathrm{WLS}} = (X^{\mathsf T}WX)^{-1}X^{\mathsf T}Wy.

With strictly positive weights, uniqueness holds when XX has full column rank. If some weights are zero, the observations with positive weights must still identify every coefficient. Otherwise, the objective has multiple minimizers.

The fitted residual vector r=y−Xβ^r=y-X\widehat{\beta} satisfies

XTWr=0.X^{\mathsf T}Wr=0.

For a model with an intercept, this implies ∑iwiri=0\sum_i w_i r_i=0. The unweighted residual sum need not be zero.

Transformation and geometric interpretation

WLS can be expressed as OLS on transformed data. Define

y∗=W1/2y,X∗=W1/2X.y^*=W^{1/2}y, \qquad X^*=W^{1/2}X.

Then

Q(β)=∥y∗−X∗β∥22.Q(\beta)=\|y^*-X^*\beta\|_2^2.

Each response and each entry in the corresponding design-matrix row is multiplied by wi\sqrt{w_i}. In particular, an intercept column must also be transformed; it becomes a column containing wi\sqrt{w_i}, rather than remaining a column of ones.

For strictly positive weights,

⟨u,v⟩W=uTWv\langle u,v\rangle_W=u^{\mathsf T}Wv

defines a weighted inner product. The fitted response is the orthogonal projection of yy onto the column space of XX under this inner product. Thus WLS changes the geometry in which discrepancies are measured, while preserving the model’s set of possible fitted responses.

Inverse-variance weighting

The main statistical motivation is heteroskedasticity: unequal error variances across observations. Suppose, conditional on XX,

E⁡(ε∣X)=0,\operatorname{E}(\varepsilon\mid X)=0,

and errors are uncorrelated with variances

Var⁡(εi∣X)=σi2>0.\operatorname{Var}(\varepsilon_i\mid X)=\sigma_i^2>0.

The inverse-variance choice is

wi=1σi2.w_i=\frac{1}{\sigma_i^2}.

An error of a given numerical size then contributes more to the objective when it occurs in a more precise observation. Equivalently,

Q(β)=∑i=1n(yi−xiTβσi)2,Q(\beta)= \sum_{i=1}^{n} \left( \frac{y_i-x_i^{\mathsf T}\beta}{\sigma_i} \right)^2,

so the method minimizes squared residuals measured in units of their error standard deviations.

If variances are known only up to a common scale,

Var⁡(εi∣X)=σ2vi,\operatorname{Var}(\varepsilon_i\mid X)=\sigma^2v_i,

weights wi=1/viw_i=1/v_i suffice. The unknown common factor σ2\sigma^2 does not affect the coefficient estimates.

Example: combining measurements

Suppose independent measurements yiy_i estimate the same unknown quantity μ\mu, with known variances σi2\sigma_i^2. An intercept-only WLS model gives

μ^=∑iyi/σi2∑i1/σi2,\widehat{\mu} = \frac{\sum_i y_i/\sigma_i^2} {\sum_i 1/\sigma_i^2},

with variance

Var⁡(μ^)=1∑i1/σi2.\operatorname{Var}(\widehat{\mu}) = \frac{1}{\sum_i 1/\sigma_i^2}.

A measurement with standard deviation 11 receives four times the weight of one with standard deviation 22. The weights are inverse variances, not inverse standard deviations.

Statistical properties

Unbiasedness and efficiency

For fixed XX and fixed weights, or weights determined by XX, WLS is unbiased under the correctly specified conditional mean model:

E⁡(β^WLS∣X)=β.\operatorname{E}(\widehat{\beta}_{\mathrm{WLS}}\mid X)=\beta.

This does not require the weights to be correct inverse variances. Incorrect weights generally affect efficiency and uncertainty calculations, rather than unbiasedness under these assumptions.

Under the Gauss–Markov theorem with unequal but uncorrelated error variances, correct inverse-variance weighting makes WLS the best linear unbiased estimator: its covariance matrix is no larger, in the positive-semidefinite ordering, than that of any other estimator linear in yy and unbiased for β\beta, conditional on XX. This result does not require normally distributed errors.

Covariance and standard errors

Let the true conditional covariance matrix of the errors be Ω\Omega. For fixed weights,

Cov⁡(β^∣X)=(XTWX)−1XTWΩWX(XTWX)−1.\operatorname{Cov}(\widehat{\beta}\mid X) = (X^{\mathsf T}WX)^{-1} X^{\mathsf T}W\Omega WX (X^{\mathsf T}WX)^{-1}.

If Ω=σ2W−1\Omega=\sigma^2W^{-1}, this simplifies to

Cov⁡(β^∣X)=σ2(XTWX)−1.\operatorname{Cov}(\widehat{\beta}\mid X) = \sigma^2(X^{\mathsf T}WX)^{-1}.

When n>pn>p, an unbiased estimator of the common scale under this variance model is

σ^2=rTWrn−p.\widehat{\sigma}^2 = \frac{r^{\mathsf T}Wr}{n-p}.

If W=Ω−1W=\Omega^{-1} is the inverse of the fully known covariance matrix, no residual estimate of a common scale is needed.

Under a correct Gaussian error model with fixed relative weights, the usual studentized coefficient statistics have Student’s tt distributions with n−pn-p degrees of freedom. These results support exact confidence intervals and tests. With estimated weights or other error distributions, such inference may instead rely on large-sample approximations.

When the variance model is uncertain, heteroskedasticity-robust covariance estimators can replace the model-based formula. Their validity still depends on appropriate assumptions, including treatment of dependence between observations.

Maximum likelihood

If errors are independent and follow a normal distribution with specified variances σi2\sigma_i^2, their log-likelihood, apart from constants, is

ℓ(β)=−12∑i[log⁡(σi2)+(yi−xiTβ)2σi2].\ell(\beta) = -\frac12\sum_i \left[ \log(\sigma_i^2) + \frac{(y_i-x_i^{\mathsf T}\beta)^2}{\sigma_i^2} \right].

Because the logarithmic terms do not depend on β\beta, maximizing this expression is equivalent to inverse-variance WLS. Thus WLS is a maximum likelihood estimator under this model.

If the variances themselves depend on β\beta, minimizing only the weighted residual sum generally does not maximize the full likelihood: the logarithmic variance terms also matter.

Choosing and estimating weights

Weights may come from instrument specifications, repeated measurements, an established variance model, or estimates obtained from the data.

When the variance pattern is unknown, a common procedure is:

  1. Fit an initial regression.
  2. Use residuals and explanatory variables to estimate the error-variance pattern.
  3. Refit using inverse estimated variances.

For example, if

Var⁡(εi∣X)=σ2h(zi;γ),\operatorname{Var}(\varepsilon_i\mid X)=\sigma^2h(z_i;\gamma),

an estimated parameter γ^\widehat{\gamma} yields weights

w^i=1h(zi;γ^).\widehat{w}_i=\frac{1}{h(z_i;\widehat{\gamma})}.

This is often called feasible WLS, a diagonal-covariance case of feasible generalized least squares. Its properties depend on the variance estimator and regularity conditions. Since response-derived weights are random, the fixed-weight unbiasedness and exact inference results do not automatically apply.

Repeatedly updating a fit and its weights produces an iteratively reweighted least-squares procedure. Such algorithms also arise in generalized linear models and robust estimation, where the weights serve purposes other than directly representing measurement precision.

Relationship to other methods

Generalized least squares

WLS is a special case of generalized least squares (GLS). For a known positive-definite error covariance matrix Ω\Omega, GLS minimizes

(y−Xβ)TΩ−1(y−Xβ).(y-X\beta)^{\mathsf T}\Omega^{-1}(y-X\beta).

Ordinary inverse-variance WLS corresponds to diagonal Ω\Omega. When errors are correlated, as in many time-series models, diagonal weighting alone does not account for that dependence. GLS uses off-diagonal covariance information as well.

Other meanings of weights

Weights with different interpretations are not interchangeable:

  • Precision weights describe relative inverse error variances.
  • Frequency weights represent repeated occurrences of the same observation.
  • Survey weights commonly reflect sampling probabilities and design adjustments.
  • Importance weights alter the population or distribution emphasized by a fitted model.

These can produce similar coefficient formulas but different uncertainty calculations. Survey inference, for example, may require information about sampling clusters and strata. A weighted squared-error objective alone does not determine the correct inferential procedure.

Nonlinear and penalized models

For a nonlinear model f(xi;β)f(x_i;\beta), the same WLS objective applies, but a closed-form solution is generally unavailable. Numerical optimization methods are then used.

WLS can also be combined with regularization, for example,

min⁡β{(y−Xβ)TW(y−Xβ)+λ∥β∥22}.\min_\beta \left\{ (y-X\beta)^{\mathsf T}W(y-X\beta) + \lambda\|\beta\|_2^2 \right\}.

Unlike unpenalized WLS, multiplying all weights by a common constant changes the balance between data fit and the penalty unless λ\lambda is adjusted accordingly.

Numerical computation

The explicit inverse formula describes the estimator but is usually not the preferred computational method. A standard implementation forms X∗X^* and y∗y^*, then solves their least-squares problem using QR decomposition or singular value decomposition.

Forming the normal-equation matrix can worsen numerical stability. For a full-column-rank transformed design matrix, its spectral condition number satisfies

κ2(X∗TX∗)=κ2(X∗)2.\kappa_2(X^{*\mathsf T}X^*)=\kappa_2(X^*)^2.

Very unequal weights can also worsen conditioning. Rescaling every weight by the same positive constant may prevent numerical overflow or underflow, but does not remove ill-conditioning caused by their relative magnitudes.

Applications and limitations

WLS is used in scientific calibration, measurement combination, meta-analysis, regression with unequal observational precision, and fitting grouped means whose variances differ. It is particularly useful when a defensible model of measurement uncertainty is available.

Important limitations include:

  • Incorrect weights: Misspecified weights can make WLS less efficient than OLS and invalidate model-based standard errors.
  • Outlier sensitivity: Squared residuals remain sensitive to unusual observations, especially those assigned large weights. WLS is not inherently robust.
  • Mean-model misspecification: Weighting does not repair an incorrect functional form. Under misspecification, changing weights generally changes the population approximation being estimated.
  • Endogeneity: Weighting alone does not remove endogeneity, omitted-variable bias, or unmeasured confounding.
  • Predictor measurement error: Ordinary WLS models response-error variation; it does not by itself account for uncertain explanatory variables.
  • Uncertain variance estimates: Estimated weights may be unstable, especially when a small estimated variance creates an extremely large weight.

Diagnostics therefore concern both the conditional mean and the variance model. Under a correct relative-variance specification, transformed residuals wiri\sqrt{w_i}r_i should have approximately comparable error scales, although fitting-induced leverage also affects their individual variances.