{
  "id": "weighted-least-squares",
  "title": "Weighted Least Squares",
  "lang": "en",
  "source_lang": "en",
  "summary": "Weighted least squares estimates model parameters by minimizing a weighted sum of squared residuals, commonly giving greater weight to observations with smaller error variances.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "WLS",
    "Weighted least-squares estimation",
    "Weighted least-squares regression"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/weighted-least-squares/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/weighted-least-squares/",
      "json": "https://aiwiki.page/zh/weighted-least-squares.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Weighted least squares (WLS) is a method of least-squares estimation in which each squared residual is multiplied by a specified nonnegative weight. It is widely used in linear regression when observations have different error variances, so that more precise observations can receive greater influence on the fitted model. With equal positive weights, WLS reduces to ordinary least squares (OLS). Its statistical properties depend on how the weights are chosen and on the assumptions about the errors."
    },
    {
      "heading": "Definition",
      "text": "For observations \\((x_i,y_i)\\), a model \\(f(x_i;\\beta)\\), and fixed weights \\(w_i\\geq 0\\), WLS chooses parameters \\(\\beta\\) to minimize\n\n\\[Q(\\beta)=\\sum_{i=1}^{n}w_i\\bigl[y_i-f(x_i;\\beta)\\bigr]^2.\\]\n\nThe residual \\(y_i-f(x_i;\\beta)\\) is the difference between an observed response and its fitted value. The weights determine the relative contribution of each squared residual to the objective function.\n\nA zero weight excludes an observation from the objective. Multiplying all weights by the same positive constant does not change the minimizing parameters, although it can change the interpretation of the residual scale and software-reported uncertainty.\n\nThe defining feature is the weighted squared-error criterion, not a particular rule for choosing weights. In statistical regression, inverse error variance is the standard choice when the purpose is to account for unequal precision."
    },
    {
      "heading": "Linear model and solution",
      "text": "For a linear model,\n\n\\[y=X\\beta+\\varepsilon,\\]\n\n\\(y\\) is an \\(n\\)-dimensional response vector, \\(X\\) is an \\(n\\times p\\) design matrix, \\(\\beta\\) contains \\(p\\) unknown coefficients, and \\(\\varepsilon\\) is the error vector. An intercept, when included, is represented by a column of ones in \\(X\\).\n\nLet\n\n\\[W=\\operatorname{diag}(w_1,\\ldots,w_n).\\]\n\nThe objective becomes\n\n\\[Q(\\beta)=(y-X\\beta)^{\\mathsf T}W(y-X\\beta).\\]\n\nSetting its gradient to zero gives the weighted normal equations:\n\n\\[X^{\\mathsf T}WX\\widehat{\\beta} = X^{\\mathsf T}Wy.\\]\n\nIf \\(X^{\\mathsf T}WX\\) is invertible, the unique solution is\n\n\\[\\widehat{\\beta}_{\\mathrm{WLS}} = (X^{\\mathsf T}WX)^{-1}X^{\\mathsf T}Wy.\\]\n\nWith strictly positive weights, uniqueness holds when \\(X\\) has full column rank. If some weights are zero, the observations with positive weights must still identify every coefficient. Otherwise, the objective has multiple minimizers.\n\nThe fitted residual vector \\(r=y-X\\widehat{\\beta}\\) satisfies\n\n\\[X^{\\mathsf T}Wr=0.\\]\n\nFor a model with an intercept, this implies \\(\\sum_i w_i r_i=0\\). The unweighted residual sum need not be zero."
    },
    {
      "heading": "Transformation and geometric interpretation",
      "text": "WLS can be expressed as OLS on transformed data. Define\n\n\\[y^*=W^{1/2}y, \\qquad X^*=W^{1/2}X.\\]\n\nThen\n\n\\[Q(\\beta)=\\|y^*-X^*\\beta\\|_2^2.\\]\n\nEach response and each entry in the corresponding design-matrix row is multiplied by \\(\\sqrt{w_i}\\). In particular, an intercept column must also be transformed; it becomes a column containing \\(\\sqrt{w_i}\\), rather than remaining a column of ones.\n\nFor strictly positive weights,\n\n\\[\\langle u,v\\rangle_W=u^{\\mathsf T}Wv\\]\n\ndefines a weighted inner product. The fitted response is the orthogonal projection of \\(y\\) onto the column space of \\(X\\) under this inner product. Thus WLS changes the geometry in which discrepancies are measured, while preserving the model’s set of possible fitted responses."
    },
    {
      "heading": "Inverse-variance weighting",
      "text": "The main statistical motivation is heteroskedasticity: unequal error variances across observations. Suppose, conditional on \\(X\\),\n\n\\[\\operatorname{E}(\\varepsilon\\mid X)=0,\\]\n\nand errors are uncorrelated with variances\n\n\\[\\operatorname{Var}(\\varepsilon_i\\mid X)=\\sigma_i^2>0.\\]\n\nThe inverse-variance choice is\n\n\\[w_i=\\frac{1}{\\sigma_i^2}.\\]\n\nAn error of a given numerical size then contributes more to the objective when it occurs in a more precise observation. Equivalently,\n\n\\[Q(\\beta)= \\sum_{i=1}^{n} \\left( \\frac{y_i-x_i^{\\mathsf T}\\beta}{\\sigma_i} \\right)^2,\\]\n\nso the method minimizes squared residuals measured in units of their error standard deviations.\n\nIf variances are known only up to a common scale,\n\n\\[\\operatorname{Var}(\\varepsilon_i\\mid X)=\\sigma^2v_i,\\]\n\nweights \\(w_i=1/v_i\\) suffice. The unknown common factor \\(\\sigma^2\\) does not affect the coefficient estimates.\n\nExample: combining measurements\n\nSuppose independent measurements \\(y_i\\) estimate the same unknown quantity \\(\\mu\\), with known variances \\(\\sigma_i^2\\). An intercept-only WLS model gives\n\n\\[\\widehat{\\mu} = \\frac{\\sum_i y_i/\\sigma_i^2} {\\sum_i 1/\\sigma_i^2},\\]\n\nwith variance\n\n\\[\\operatorname{Var}(\\widehat{\\mu}) = \\frac{1}{\\sum_i 1/\\sigma_i^2}.\\]\n\nA measurement with standard deviation \\(1\\) receives four times the weight of one with standard deviation \\(2\\). The weights are inverse variances, not inverse standard deviations."
    },
    {
      "heading": "Statistical properties",
      "text": "Unbiasedness and efficiency\n\nFor fixed \\(X\\) and fixed weights, or weights determined by \\(X\\), WLS is unbiased under the correctly specified conditional mean model:\n\n\\[\\operatorname{E}(\\widehat{\\beta}_{\\mathrm{WLS}}\\mid X)=\\beta.\\]\n\nThis does not require the weights to be correct inverse variances. Incorrect weights generally affect efficiency and uncertainty calculations, rather than unbiasedness under these assumptions.\n\nUnder the Gauss–Markov theorem with unequal but uncorrelated error variances, correct inverse-variance weighting makes WLS the best linear unbiased estimator: its covariance matrix is no larger, in the positive-semidefinite ordering, than that of any other estimator linear in \\(y\\) and unbiased for \\(\\beta\\), conditional on \\(X\\). This result does not require normally distributed errors.\n\nCovariance and standard errors\n\nLet the true conditional covariance matrix of the errors be \\(\\Omega\\). For fixed weights,\n\n\\[\\operatorname{Cov}(\\widehat{\\beta}\\mid X) = (X^{\\mathsf T}WX)^{-1} X^{\\mathsf T}W\\Omega WX (X^{\\mathsf T}WX)^{-1}.\\]\n\nIf \\(\\Omega=\\sigma^2W^{-1}\\), this simplifies to\n\n\\[\\operatorname{Cov}(\\widehat{\\beta}\\mid X) = \\sigma^2(X^{\\mathsf T}WX)^{-1}.\\]\n\nWhen \\(n>p\\), an unbiased estimator of the common scale under this variance model is\n\n\\[\\widehat{\\sigma}^2 = \\frac{r^{\\mathsf T}Wr}{n-p}.\\]\n\nIf \\(W=\\Omega^{-1}\\) is the inverse of the fully known covariance matrix, no residual estimate of a common scale is needed.\n\nUnder a correct Gaussian error model with fixed relative weights, the usual studentized coefficient statistics have Student’s t distributions with \\(n-p\\) degrees of freedom. These results support exact confidence intervals and tests. With estimated weights or other error distributions, such inference may instead rely on large-sample approximations.\n\nWhen the variance model is uncertain, heteroskedasticity-robust covariance estimators can replace the model-based formula. Their validity still depends on appropriate assumptions, including treatment of dependence between observations.\n\nMaximum likelihood\n\nIf errors are independent and follow a normal distribution with specified variances \\(\\sigma_i^2\\), their log-likelihood, apart from constants, is\n\n\\[\\ell(\\beta) = -\\frac12\\sum_i \\left[ \\log(\\sigma_i^2) + \\frac{(y_i-x_i^{\\mathsf T}\\beta)^2}{\\sigma_i^2} \\right].\\]\n\nBecause the logarithmic terms do not depend on \\(\\beta\\), maximizing this expression is equivalent to inverse-variance WLS. Thus WLS is a maximum likelihood estimator under this model.\n\nIf the variances themselves depend on \\(\\beta\\), minimizing only the weighted residual sum generally does not maximize the full likelihood: the logarithmic variance terms also matter."
    },
    {
      "heading": "Choosing and estimating weights",
      "text": "Weights may come from instrument specifications, repeated measurements, an established variance model, or estimates obtained from the data.\n\nWhen the variance pattern is unknown, a common procedure is:\n\n- Fit an initial regression.\n- Use residuals and explanatory variables to estimate the error-variance pattern.\n- Refit using inverse estimated variances.\n\nFor example, if\n\n\\[\\operatorname{Var}(\\varepsilon_i\\mid X)=\\sigma^2h(z_i;\\gamma),\\]\n\nan estimated parameter \\(\\widehat{\\gamma}\\) yields weights\n\n\\[\\widehat{w}_i=\\frac{1}{h(z_i;\\widehat{\\gamma})}.\\]\n\nThis is often called feasible WLS, a diagonal-covariance case of feasible generalized least squares. Its properties depend on the variance estimator and regularity conditions. Since response-derived weights are random, the fixed-weight unbiasedness and exact inference results do not automatically apply.\n\nRepeatedly updating a fit and its weights produces an iteratively reweighted least-squares procedure. Such algorithms also arise in generalized linear models and robust estimation, where the weights serve purposes other than directly representing measurement precision."
    },
    {
      "heading": "Relationship to other methods",
      "text": "Generalized least squares\n\nWLS is a special case of generalized least squares (GLS). For a known positive-definite error covariance matrix \\(\\Omega\\), GLS minimizes\n\n\\[(y-X\\beta)^{\\mathsf T}\\Omega^{-1}(y-X\\beta).\\]\n\nOrdinary inverse-variance WLS corresponds to diagonal \\(\\Omega\\). When errors are correlated, as in many time-series models, diagonal weighting alone does not account for that dependence. GLS uses off-diagonal covariance information as well.\n\nOther meanings of weights\n\nWeights with different interpretations are not interchangeable:\n\n- Precision weights describe relative inverse error variances.\n- Frequency weights represent repeated occurrences of the same observation.\n- Survey weights commonly reflect sampling probabilities and design adjustments.\n- Importance weights alter the population or distribution emphasized by a fitted model.\n\nThese can produce similar coefficient formulas but different uncertainty calculations. Survey inference, for example, may require information about sampling clusters and strata. A weighted squared-error objective alone does not determine the correct inferential procedure.\n\nNonlinear and penalized models\n\nFor a nonlinear model \\(f(x_i;\\beta)\\), the same WLS objective applies, but a closed-form solution is generally unavailable. Numerical optimization methods are then used.\n\nWLS can also be combined with regularization, for example,\n\n\\[\\min_\\beta \\left\\{ (y-X\\beta)^{\\mathsf T}W(y-X\\beta) + \\lambda\\|\\beta\\|_2^2 \\right\\}.\\]\n\nUnlike unpenalized WLS, multiplying all weights by a common constant changes the balance between data fit and the penalty unless \\(\\lambda\\) is adjusted accordingly."
    },
    {
      "heading": "Numerical computation",
      "text": "The explicit inverse formula describes the estimator but is usually not the preferred computational method. A standard implementation forms \\(X^*\\) and \\(y^*\\), then solves their least-squares problem using QR decomposition or singular value decomposition.\n\nForming the normal-equation matrix can worsen numerical stability. For a full-column-rank transformed design matrix, its spectral condition number satisfies\n\n\\[\\kappa_2(X^{*\\mathsf T}X^*)=\\kappa_2(X^*)^2.\\]\n\nVery unequal weights can also worsen conditioning. Rescaling every weight by the same positive constant may prevent numerical overflow or underflow, but does not remove ill-conditioning caused by their relative magnitudes."
    },
    {
      "heading": "Applications and limitations",
      "text": "WLS is used in scientific calibration, measurement combination, meta-analysis, regression with unequal observational precision, and fitting grouped means whose variances differ. It is particularly useful when a defensible model of measurement uncertainty is available.\n\nImportant limitations include:\n\n- Incorrect weights: Misspecified weights can make WLS less efficient than OLS and invalidate model-based standard errors.\n- Outlier sensitivity: Squared residuals remain sensitive to unusual observations, especially those assigned large weights. WLS is not inherently robust.\n- Mean-model misspecification: Weighting does not repair an incorrect functional form. Under misspecification, changing weights generally changes the population approximation being estimated.\n- Endogeneity: Weighting alone does not remove endogeneity, omitted-variable bias, or unmeasured confounding.\n- Predictor measurement error: Ordinary WLS models response-error variation; it does not by itself account for uncertain explanatory variables.\n- Uncertain variance estimates: Estimated weights may be unstable, especially when a small estimated variance creates an extremely large weight.\n\nDiagnostics therefore concern both the conditional mean and the variance model. Under a correct relative-variance specification, transformed residuals \\(\\sqrt{w_i}r_i\\) should have approximately comparable error scales, although fitting-induced leverage also affects their individual variances."
    }
  ],
  "links": [
    "least-squares",
    "linear-regression",
    "ordinary-least-squares",
    "objective-function",
    "matrix",
    "gradient",
    "matrix-rank",
    "inner-product",
    "orthogonal-projection",
    "variance",
    "standard-deviation",
    "estimator-bias",
    "gauss-markov-theorem",
    "covariance-matrix",
    "students-t-distribution",
    "confidence-interval",
    "normal-distribution",
    "maximum-likelihood-estimation",
    "iteratively-reweighted-least-squares",
    "time-series",
    "regularization",
    "qr-decomposition",
    "singular-value-decomposition",
    "numerical-stability",
    "calibration",
    "meta-analysis",
    "endogeneity"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}