{
  "id": "gauss-markov-theorem",
  "title": "Gauss–Markov Theorem",
  "lang": "en",
  "source_lang": "en",
  "summary": "The Gauss–Markov theorem states that ordinary least squares is the best linear unbiased estimator in a linear model with zero-mean, uncorrelated errors of equal variance.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "Gauss–Markov theorem",
    "Gauss–Markov–Aitken theorem"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/gauss-markov-theorem/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/gauss-markov-theorem/",
      "json": "https://aiwiki.page/zh/gauss-markov-theorem.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The Gauss–Markov theorem is a result in statistics establishing the optimality of ordinary least squares within a specified class of estimators. In a linear regression model whose errors have zero mean, equal finite variance, and no pairwise correlation, the ordinary least-squares estimator has the smallest covariance matrix among all estimators that are linear in the observed responses and unbiased for the regression coefficients. It is therefore called the best linear unbiased estimator, or BLUE. The theorem does not require normally distributed errors, and its optimality claim does not extend automatically to biased or nonlinear estimators."
    },
    {
      "heading": "Model and assumptions",
      "text": "Consider the model\n\n\\[y=X\\beta+\\varepsilon,\\]\n\nwhere:\n\n- \\(y\\) is an \\(n\\times 1\\) vector of observed responses;\n- \\(X\\) is an \\(n\\times p\\) design matrix;\n- \\(\\beta\\) is a \\(p\\times 1\\) vector of unknown, nonrandom coefficients;\n- \\(\\varepsilon\\) is an \\(n\\times 1\\) vector of random errors.\n\nThe standard fixed-design version treats \\(X\\) as known and nonrandom. Its assumptions are:\n\n- Full column rank: \\(\\operatorname{rank}(X)=p\\), so \\(n\\geq p\\) and the coefficients are uniquely identified.\n- Zero-mean errors: \\(\\mathbb E[\\varepsilon]=0\\).\n- Spherical error covariance:\n\n\\[\\operatorname{Cov}(\\varepsilon)=\\sigma^2 I_n, \\qquad 0<\\sigma^2<\\infty.\\]\n\nHere \\(\\operatorname{Cov}\\) denotes a covariance matrix, and \\(I_n\\) is the identity matrix. The third assumption combines equal error variances, called homoscedasticity, with zero covariance between distinct errors.\n\nZero covariance does not generally imply statistical independence. Independent errors are sufficient for the absence of error correlation, but independence is stronger than the theorem requires.\n\nIf \\(X\\) is random, a common formulation instead assumes\n\n\\[\\mathbb E[\\varepsilon\\mid X]=0, \\qquad \\operatorname{Cov}(\\varepsilon\\mid X)=\\sigma^2 I_n,\\]\n\nwith full column rank almost surely. The theorem then applies conditionally on \\(X\\). The conditional-mean assumption is stronger than merely requiring an unconditional zero mean.\n\n“Linear model” means linear in the unknown coefficients, not necessarily linear in the original explanatory variables. A model containing \\(x\\), \\(x^2\\), and \\(x^3\\), for example, still satisfies this structural requirement when their coefficients enter linearly."
    },
    {
      "heading": "Precise statement",
      "text": "Under these assumptions, the ordinary least-squares estimator is\n\n\\[\\widehat\\beta=(X^{\\mathsf T}X)^{-1}X^{\\mathsf T}y,\\]\n\nwhere \\(X^{\\mathsf T}\\) denotes the transpose of \\(X\\). Full column rank ensures that \\(X^{\\mathsf T}X\\) is invertible.\n\nIts mean and covariance are\n\n\\[\\mathbb E[\\widehat\\beta]=\\beta, \\qquad \\operatorname{Cov}(\\widehat\\beta) =\\sigma^2(X^{\\mathsf T}X)^{-1}.\\]\n\nConsider any other estimator\n\n\\[\\widetilde\\beta=Ay,\\]\n\nwhere \\(A\\) is a fixed \\(p\\times n\\) matrix, possibly depending on the known design \\(X\\), but not on the observed response \\(y\\) or the unknown coefficients. Such an estimator is linear in the observations. It is unbiased for every \\(\\beta\\) precisely when\n\n\\[AX=I_p.\\]\n\nThis is the usual unbiasedness requirement: its expected value equals the parameter throughout the model.\n\nThe theorem states that\n\n\\[\\operatorname{Cov}(\\widetilde\\beta) -\\operatorname{Cov}(\\widehat\\beta) \\succeq 0,\\]\n\nwhere \\(\\succeq 0\\) means positive semidefinite: every associated quadratic form is nonnegative.\n\nEquivalently, for every fixed \\(c\\in\\mathbb R^p\\),\n\n\\[\\operatorname{Var}(c^{\\mathsf T}\\widetilde\\beta) \\geq \\operatorname{Var}(c^{\\mathsf T}\\widehat\\beta).\\]\n\nThus ordinary least squares minimizes the variance of every estimated linear combination of the coefficients among linear unbiased estimators. “Best” refers to this covariance ordering, not to the magnitudes of individual observed errors or to a guarantee of accurate prediction in every dataset."
    },
    {
      "heading": "Proof",
      "text": "Define\n\n\\[B=(X^{\\mathsf T}X)^{-1}X^{\\mathsf T}.\\]\n\nThen \\(\\widehat\\beta=By\\) and \\(BX=I_p\\).\n\nFor any competing linear unbiased estimator \\(Ay\\), write\n\n\\[A=B+D.\\]\n\nBecause \\(AX=BX=I_p\\),\n\n\\[DX=0.\\]\n\nUsing the spherical covariance assumption,\n\n\\[\\operatorname{Cov}(Ay) =\\sigma^2AA^{\\mathsf T} =\\sigma^2(B+D)(B+D)^{\\mathsf T}.\\]\n\nThe cross terms vanish, since\n\n\\[BD^{\\mathsf T} =(X^{\\mathsf T}X)^{-1}X^{\\mathsf T}D^{\\mathsf T} =(X^{\\mathsf T}X)^{-1}(DX)^{\\mathsf T} =0.\\]\n\nConsequently,\n\n\\[\\operatorname{Cov}(Ay) = \\sigma^2(X^{\\mathsf T}X)^{-1} +\\sigma^2DD^{\\mathsf T}.\\]\n\nFor every vector \\(c\\),\n\n\\[c^{\\mathsf T}DD^{\\mathsf T}c =\\lVert D^{\\mathsf T}c\\rVert^2\\geq 0.\\]\n\nThe covariance difference is therefore positive semidefinite, proving the theorem. Since \\(\\sigma^2>0\\), equality of the entire covariance matrices requires \\(D=0\\), establishing uniqueness within this class.\n\nThe argument uses only first and second moments. No particular probability density or distributional shape enters the proof."
    },
    {
      "heading": "Geometric interpretation",
      "text": "The fitted response\n\n\\[\\widehat y=X\\widehat\\beta\\]\n\nis the orthogonal projection of \\(y\\) onto the column space of \\(X\\). The residual vector\n\n\\[e=y-\\widehat y\\]\n\nsatisfies\n\n\\[X^{\\mathsf T}e=0,\\]\n\nso it is perpendicular to every column of the design matrix.\n\nIn the proof, a competing estimator differs from ordinary least squares by \\(Dy\\), with \\(DX=0\\). This extra term has zero mean and is uncorrelated with \\(\\widehat\\beta\\) under spherical error covariance. Adding it therefore cannot reduce covariance; it contributes the nonnegative term \\(\\sigma^2DD^{\\mathsf T}\\).\n\nThis geometric interpretation connects least-squares minimization with statistical efficiency. Orthogonal projection explains why the residual sum of squares is minimized, while the covariance assumptions explain why the resulting coefficients are BLUE."
    },
    {
      "heading": "Example: estimating a common mean",
      "text": "Suppose\n\n\\[y_i=\\mu+\\varepsilon_i, \\qquad i=1,\\ldots,n,\\]\n\nwhere the errors have zero mean, common variance \\(\\sigma^2\\), and zero pairwise covariance. Here \\(X\\) is a single column of ones, so ordinary least squares gives the sample mean\n\n\\[\\widehat\\mu=\\overline y=\\frac1n\\sum_{i=1}^n y_i.\\]\n\nAny linear estimator has the form\n\n\\[\\widetilde\\mu=\\sum_{i=1}^n a_i y_i.\\]\n\nUnbiasedness for every \\(\\mu\\) requires \\(\\sum_i a_i=1\\), and\n\n\\[\\operatorname{Var}(\\widetilde\\mu) =\\sigma^2\\sum_{i=1}^n a_i^2.\\]\n\nSince\n\n\\[\\sum_{i=1}^n a_i^2 = \\frac1n+\\sum_{i=1}^n\\left(a_i-\\frac1n\\right)^2,\\]\n\nthe minimum variance is \\(\\sigma^2/n\\), attained by equal weights.\n\nIf the observations instead have unequal known variances and remain uncorrelated, inverse-variance weighting generally improves on the unweighted mean. This illustrates the importance of the equal-variance assumption."
    },
    {
      "heading": "Generalized least squares",
      "text": "The theorem extends to errors satisfying\n\n\\[\\operatorname{Cov}(\\varepsilon)=\\Omega,\\]\n\nwhere \\(\\Omega\\) is a known positive-definite matrix, not necessarily proportional to the identity.\n\nThe generalized least-squares estimator is\n\n\\[\\widehat\\beta_{\\mathrm{GLS}} = (X^{\\mathsf T}\\Omega^{-1}X)^{-1} X^{\\mathsf T}\\Omega^{-1}y,\\]\n\nwith covariance\n\n\\[\\operatorname{Cov}(\\widehat\\beta_{\\mathrm{GLS}}) = (X^{\\mathsf T}\\Omega^{-1}X)^{-1}.\\]\n\nUnder the zero-mean assumption, it is BLUE for this more general covariance structure. This extension is commonly associated with Aitken.\n\nIt can be understood by transforming the model so that the transformed errors have identity covariance, then applying ordinary least squares. A diagonal \\(\\Omega\\) gives weighted least squares; off-diagonal entries permit correlated errors.\n\nIf \\(\\Omega\\) is estimated from the data, the resulting feasible generalized least-squares estimator does not automatically inherit the same exact finite-sample optimality."
    },
    {
      "heading": "Scope and limitations",
      "text": "Linearity and unbiasedness restrict the comparison\n\nThe theorem compares linear unbiased estimators. It does not establish that ordinary least squares minimizes every measure of estimation error among all possible estimators.\n\nFor a scalar estimator \\(T\\) of \\(\\theta\\), mean squared error satisfies\n\n\\[\\mathbb E[(T-\\theta)^2] = \\operatorname{Var}(T) +\\bigl(\\mathbb E[T]-\\theta\\bigr)^2.\\]\n\nA biased estimator can trade increased bias for reduced variance. Methods such as ridge regression may therefore achieve lower mean squared error in some settings without contradicting Gauss–Markov.\n\nNormality is unnecessary, but exact inference needs more\n\nA normal distribution for the errors is not required for BLUE optimality. If the errors are jointly normal with covariance \\(\\sigma^2I_n\\), ordinary least squares is also the maximum-likelihood estimator of \\(\\beta\\), and familiar exact finite-sample regression tests become available.\n\nWithout normality, the Gauss–Markov theorem alone does not justify exact Student \\(t\\) or \\(F\\) distributions for test statistics. Large-sample inference requires additional conditions.\n\nWhen \\(n>p\\),\n\n\\[s^2=\\frac{e^{\\mathsf T}e}{n-p}\\]\n\nis an unbiased estimator of \\(\\sigma^2\\) under the stated moment assumptions. The usual estimated coefficient covariance matrix is \\(s^2(X^{\\mathsf T}X)^{-1}\\), from which conventional standard errors are obtained.\n\nUnequal variances and correlated errors\n\nIf the zero-mean condition remains valid but error covariance is a general matrix \\(\\Omega\\), ordinary least squares remains unbiased, with\n\n\\[\\operatorname{Cov}(\\widehat\\beta) = (X^{\\mathsf T}X)^{-1} X^{\\mathsf T}\\Omega X (X^{\\mathsf T}X)^{-1}.\\]\n\nIt is no longer generally BLUE. Robust covariance estimators can support inference under specified departures from spherical covariance, but they do not change the ordinary least-squares coefficient estimates or make them efficient automatically.\n\nEndogeneity and causal interpretation\n\nWhen \\(\\mathbb E[\\varepsilon\\mid X]\\neq 0\\), ordinary least squares need not be unbiased. Such endogeneity can arise from omitted variables, measurement error, or simultaneous determination of regressors and responses.\n\nThe theorem also does not establish causal identification. Its efficiency statement concerns estimation within an assumed statistical model; interpreting coefficients as causal effects requires additional substantive assumptions or research design.\n\nRank deficiency and numerical computation\n\nIf \\(X\\) lacks full column rank, the entire coefficient vector is not uniquely identified. Nevertheless, estimable linear combinations of coefficients can still have best linear unbiased estimators.\n\nEven with full rank, near-collinearity can make coefficient variances large and computations sensitive to rounding. BLUE means best within the specified class, not necessarily precise. In computation, QR decomposition or singular value decomposition is generally preferable to explicitly forming and inverting \\(X^{\\mathsf T}X\\)."
    },
    {
      "heading": "Historical context",
      "text": "The theorem is named after Carl Friedrich Gauss and Andrey Markov. Gauss developed foundational results connecting least squares with minimum-variance linear unbiased estimation; Markov later contributed to the theory and its formulations. Aitken extended the framework to general known error covariance.\n\nThe theorem occupies a central place in econometrics, experimental analysis, and regression theory because it distinguishes three separate ideas: minimizing squared residuals, achieving unbiasedness, and attaining minimum variance within a defined estimator class."
    }
  ],
  "links": [
    "statistics",
    "ordinary-least-squares",
    "linear-regression",
    "matrix",
    "covariance-matrix",
    "identity-matrix",
    "statistical-independence",
    "matrix-transpose",
    "estimator-bias",
    "variance",
    "orthogonal-projection",
    "sample-mean",
    "positive-definite-matrix",
    "mean-squared-error",
    "ridge-regression",
    "normal-distribution",
    "maximum-likelihood-estimation",
    "standard-error",
    "endogeneity",
    "causal-inference",
    "qr-decomposition",
    "singular-value-decomposition",
    "carl-friedrich-gauss",
    "econometrics"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}