Weighted least squares (WLS) is a method of least-squares estimation in which each squared residual is multiplied by a specified nonnegative weight. It is widely used in linear regression when observations have different error variances, so that more precise observations can receive greater influence on the fitted model. With equal positive weights, WLS reduces to ordinary least squares (OLS). Its statistical properties depend on how the weights are chosen and on the assumptions about the errors.
Definition
For observations , a model , and fixed weights , WLS chooses parameters to minimize
The residual is the difference between an observed response and its fitted value. The weights determine the relative contribution of each squared residual to the objective function.
A zero weight excludes an observation from the objective. Multiplying all weights by the same positive constant does not change the minimizing parameters, although it can change the interpretation of the residual scale and software-reported uncertainty.
The defining feature is the weighted squared-error criterion, not a particular rule for choosing weights. In statistical regression, inverse error variance is the standard choice when the purpose is to account for unequal precision.
Linear model and solution
For a linear model,
is an -dimensional response vector, is an design matrix, contains unknown coefficients, and is the error vector. An intercept, when included, is represented by a column of ones in .
Let
The objective becomes
Setting its gradient to zero gives the weighted normal equations:
If is invertible, the unique solution is
With strictly positive weights, uniqueness holds when has full column rank. If some weights are zero, the observations with positive weights must still identify every coefficient. Otherwise, the objective has multiple minimizers.
The fitted residual vector satisfies
For a model with an intercept, this implies . The unweighted residual sum need not be zero.
Transformation and geometric interpretation
WLS can be expressed as OLS on transformed data. Define
Then
Each response and each entry in the corresponding design-matrix row is multiplied by . In particular, an intercept column must also be transformed; it becomes a column containing , rather than remaining a column of ones.
For strictly positive weights,
defines a weighted inner product. The fitted response is the orthogonal projection of onto the column space of under this inner product. Thus WLS changes the geometry in which discrepancies are measured, while preserving the model’s set of possible fitted responses.
Inverse-variance weighting
The main statistical motivation is heteroskedasticity: unequal error variances across observations. Suppose, conditional on ,
and errors are uncorrelated with variances
The inverse-variance choice is
An error of a given numerical size then contributes more to the objective when it occurs in a more precise observation. Equivalently,
so the method minimizes squared residuals measured in units of their error standard deviations.
If variances are known only up to a common scale,
weights suffice. The unknown common factor does not affect the coefficient estimates.
Example: combining measurements
Suppose independent measurements estimate the same unknown quantity , with known variances . An intercept-only WLS model gives
with variance
A measurement with standard deviation receives four times the weight of one with standard deviation . The weights are inverse variances, not inverse standard deviations.
Statistical properties
Unbiasedness and efficiency
For fixed and fixed weights, or weights determined by , WLS is unbiased under the correctly specified conditional mean model:
This does not require the weights to be correct inverse variances. Incorrect weights generally affect efficiency and uncertainty calculations, rather than unbiasedness under these assumptions.
Under the Gauss–Markov theorem with unequal but uncorrelated error variances, correct inverse-variance weighting makes WLS the best linear unbiased estimator: its covariance matrix is no larger, in the positive-semidefinite ordering, than that of any other estimator linear in and unbiased for , conditional on . This result does not require normally distributed errors.
Covariance and standard errors
Let the true conditional covariance matrix of the errors be . For fixed weights,
If , this simplifies to
When , an unbiased estimator of the common scale under this variance model is
If is the inverse of the fully known covariance matrix, no residual estimate of a common scale is needed.
Under a correct Gaussian error model with fixed relative weights, the usual studentized coefficient statistics have Student’s distributions with degrees of freedom. These results support exact confidence intervals and tests. With estimated weights or other error distributions, such inference may instead rely on large-sample approximations.
When the variance model is uncertain, heteroskedasticity-robust covariance estimators can replace the model-based formula. Their validity still depends on appropriate assumptions, including treatment of dependence between observations.
Maximum likelihood
If errors are independent and follow a normal distribution with specified variances , their log-likelihood, apart from constants, is
Because the logarithmic terms do not depend on , maximizing this expression is equivalent to inverse-variance WLS. Thus WLS is a maximum likelihood estimator under this model.
If the variances themselves depend on , minimizing only the weighted residual sum generally does not maximize the full likelihood: the logarithmic variance terms also matter.
Choosing and estimating weights
Weights may come from instrument specifications, repeated measurements, an established variance model, or estimates obtained from the data.
When the variance pattern is unknown, a common procedure is:
- Fit an initial regression.
- Use residuals and explanatory variables to estimate the error-variance pattern.
- Refit using inverse estimated variances.
For example, if
an estimated parameter yields weights
This is often called feasible WLS, a diagonal-covariance case of feasible generalized least squares. Its properties depend on the variance estimator and regularity conditions. Since response-derived weights are random, the fixed-weight unbiasedness and exact inference results do not automatically apply.
Repeatedly updating a fit and its weights produces an iteratively reweighted least-squares procedure. Such algorithms also arise in generalized linear models and robust estimation, where the weights serve purposes other than directly representing measurement precision.
Relationship to other methods
Generalized least squares
WLS is a special case of generalized least squares (GLS). For a known positive-definite error covariance matrix , GLS minimizes
Ordinary inverse-variance WLS corresponds to diagonal . When errors are correlated, as in many time-series models, diagonal weighting alone does not account for that dependence. GLS uses off-diagonal covariance information as well.
Other meanings of weights
Weights with different interpretations are not interchangeable:
- Precision weights describe relative inverse error variances.
- Frequency weights represent repeated occurrences of the same observation.
- Survey weights commonly reflect sampling probabilities and design adjustments.
- Importance weights alter the population or distribution emphasized by a fitted model.
These can produce similar coefficient formulas but different uncertainty calculations. Survey inference, for example, may require information about sampling clusters and strata. A weighted squared-error objective alone does not determine the correct inferential procedure.
Nonlinear and penalized models
For a nonlinear model , the same WLS objective applies, but a closed-form solution is generally unavailable. Numerical optimization methods are then used.
WLS can also be combined with regularization, for example,
Unlike unpenalized WLS, multiplying all weights by a common constant changes the balance between data fit and the penalty unless is adjusted accordingly.
Numerical computation
The explicit inverse formula describes the estimator but is usually not the preferred computational method. A standard implementation forms and , then solves their least-squares problem using QR decomposition or singular value decomposition.
Forming the normal-equation matrix can worsen numerical stability. For a full-column-rank transformed design matrix, its spectral condition number satisfies
Very unequal weights can also worsen conditioning. Rescaling every weight by the same positive constant may prevent numerical overflow or underflow, but does not remove ill-conditioning caused by their relative magnitudes.
Applications and limitations
WLS is used in scientific calibration, measurement combination, meta-analysis, regression with unequal observational precision, and fitting grouped means whose variances differ. It is particularly useful when a defensible model of measurement uncertainty is available.
Important limitations include:
- Incorrect weights: Misspecified weights can make WLS less efficient than OLS and invalidate model-based standard errors.
- Outlier sensitivity: Squared residuals remain sensitive to unusual observations, especially those assigned large weights. WLS is not inherently robust.
- Mean-model misspecification: Weighting does not repair an incorrect functional form. Under misspecification, changing weights generally changes the population approximation being estimated.
- Endogeneity: Weighting alone does not remove endogeneity, omitted-variable bias, or unmeasured confounding.
- Predictor measurement error: Ordinary WLS models response-error variation; it does not by itself account for uncertain explanatory variables.
- Uncertain variance estimates: Estimated weights may be unstable, especially when a small estimated variance creates an extremely large weight.
Diagnostics therefore concern both the conditional mean and the variance model. Under a correct relative-variance specification, transformed residuals should have approximately comparable error scales, although fitting-induced leverage also affects their individual variances.