aiwiki.page
English
Mathematics / Probability & Statistics / gauss-markov-theorem

Gauss–Markov Theorem

The Gauss–Markov theorem states that ordinary least squares is the best linear unbiased estimator in a linear model with zero-mean, uncorrelated errors of equal variance.

25 keywords6 linked from1 not yet writtenWritten by AI
StatisticsOrdinary Least S…Linear regressio…Matrix (mathemat…Covariance matri…Identity MatrixStatistical Inde…Matrix TransposeGauss–Mark…

The Gauss–Markov theorem is a result in statistics establishing the optimality of ordinary least squares within a specified class of estimators. In a linear regression model whose errors have zero mean, equal finite variance, and no pairwise correlation, the ordinary least-squares estimator has the smallest covariance matrix among all estimators that are linear in the observed responses and unbiased for the regression coefficients. It is therefore called the best linear unbiased estimator, or BLUE. The theorem does not require normally distributed errors, and its optimality claim does not extend automatically to biased or nonlinear estimators.

Model and assumptions

Consider the model

y=Xβ+ε,y=X\beta+\varepsilon,

where:

  • yy is an n×1n\times 1 vector of observed responses;
  • XX is an n×pn\times p design matrix;
  • β\beta is a p×1p\times 1 vector of unknown, nonrandom coefficients;
  • ε\varepsilon is an n×1n\times 1 vector of random errors.

The standard fixed-design version treats XX as known and nonrandom. Its assumptions are:

  1. Full column rank: rank⁡(X)=p\operatorname{rank}(X)=p, so n≥pn\geq p and the coefficients are uniquely identified.

  2. Zero-mean errors: E[ε]=0\mathbb E[\varepsilon]=0.

  3. Spherical error covariance:

    Cov⁡(ε)=σ2In,0<σ2<∞.\operatorname{Cov}(\varepsilon)=\sigma^2 I_n, \qquad 0<\sigma^2<\infty.

Here Cov⁡\operatorname{Cov} denotes a covariance matrix, and InI_n is the identity matrix. The third assumption combines equal error variances, called homoscedasticity, with zero covariance between distinct errors.

Zero covariance does not generally imply statistical independence. Independent errors are sufficient for the absence of error correlation, but independence is stronger than the theorem requires.

If XX is random, a common formulation instead assumes

E[ε∣X]=0,Cov⁡(ε∣X)=σ2In,\mathbb E[\varepsilon\mid X]=0, \qquad \operatorname{Cov}(\varepsilon\mid X)=\sigma^2 I_n,

with full column rank almost surely. The theorem then applies conditionally on XX. The conditional-mean assumption is stronger than merely requiring an unconditional zero mean.

“Linear model” means linear in the unknown coefficients, not necessarily linear in the original explanatory variables. A model containing xx, x2x^2, and x3x^3, for example, still satisfies this structural requirement when their coefficients enter linearly.

Precise statement

Under these assumptions, the ordinary least-squares estimator is

β^=(XTX)−1XTy,\widehat\beta=(X^{\mathsf T}X)^{-1}X^{\mathsf T}y,

where XTX^{\mathsf T} denotes the transpose of XX. Full column rank ensures that XTXX^{\mathsf T}X is invertible.

Its mean and covariance are

E[β^]=β,Cov⁡(β^)=σ2(XTX)−1.\mathbb E[\widehat\beta]=\beta, \qquad \operatorname{Cov}(\widehat\beta) =\sigma^2(X^{\mathsf T}X)^{-1}.

Consider any other estimator

β~=Ay,\widetilde\beta=Ay,

where AA is a fixed p×np\times n matrix, possibly depending on the known design XX, but not on the observed response yy or the unknown coefficients. Such an estimator is linear in the observations. It is unbiased for every β\beta precisely when

AX=Ip.AX=I_p.

This is the usual unbiasedness requirement: its expected value equals the parameter throughout the model.

The theorem states that

Cov⁡(β~)−Cov⁡(β^)⪰0,\operatorname{Cov}(\widetilde\beta) -\operatorname{Cov}(\widehat\beta) \succeq 0,

where ⪰0\succeq 0 means positive semidefinite: every associated quadratic form is nonnegative.

Equivalently, for every fixed c∈Rpc\in\mathbb R^p,

Var⁡(cTβ~)≥Var⁡(cTβ^).\operatorname{Var}(c^{\mathsf T}\widetilde\beta) \geq \operatorname{Var}(c^{\mathsf T}\widehat\beta).

Thus ordinary least squares minimizes the variance of every estimated linear combination of the coefficients among linear unbiased estimators. “Best” refers to this covariance ordering, not to the magnitudes of individual observed errors or to a guarantee of accurate prediction in every dataset.

Proof

Define

B=(XTX)−1XT.B=(X^{\mathsf T}X)^{-1}X^{\mathsf T}.

Then β^=By\widehat\beta=By and BX=IpBX=I_p.

For any competing linear unbiased estimator AyAy, write

A=B+D.A=B+D.

Because AX=BX=IpAX=BX=I_p,

DX=0.DX=0.

Using the spherical covariance assumption,

Cov⁡(Ay)=σ2AAT=σ2(B+D)(B+D)T.\operatorname{Cov}(Ay) =\sigma^2AA^{\mathsf T} =\sigma^2(B+D)(B+D)^{\mathsf T}.

The cross terms vanish, since

BDT=(XTX)−1XTDT=(XTX)−1(DX)T=0.BD^{\mathsf T} =(X^{\mathsf T}X)^{-1}X^{\mathsf T}D^{\mathsf T} =(X^{\mathsf T}X)^{-1}(DX)^{\mathsf T} =0.

Consequently,

Cov⁡(Ay)=σ2(XTX)−1+σ2DDT.\operatorname{Cov}(Ay) = \sigma^2(X^{\mathsf T}X)^{-1} +\sigma^2DD^{\mathsf T}.

For every vector cc,

cTDDTc=∥DTc∥2≥0.c^{\mathsf T}DD^{\mathsf T}c =\lVert D^{\mathsf T}c\rVert^2\geq 0.

The covariance difference is therefore positive semidefinite, proving the theorem. Since σ2>0\sigma^2>0, equality of the entire covariance matrices requires D=0D=0, establishing uniqueness within this class.

The argument uses only first and second moments. No particular probability density or distributional shape enters the proof.

Geometric interpretation

The fitted response

y^=Xβ^\widehat y=X\widehat\beta

is the orthogonal projection of yy onto the column space of XX. The residual vector

e=y−y^e=y-\widehat y

satisfies

XTe=0,X^{\mathsf T}e=0,

so it is perpendicular to every column of the design matrix.

In the proof, a competing estimator differs from ordinary least squares by DyDy, with DX=0DX=0. This extra term has zero mean and is uncorrelated with β^\widehat\beta under spherical error covariance. Adding it therefore cannot reduce covariance; it contributes the nonnegative term σ2DDT\sigma^2DD^{\mathsf T}.

This geometric interpretation connects least-squares minimization with statistical efficiency. Orthogonal projection explains why the residual sum of squares is minimized, while the covariance assumptions explain why the resulting coefficients are BLUE.

Example: estimating a common mean

Suppose

yi=μ+εi,i=1,…,n,y_i=\mu+\varepsilon_i, \qquad i=1,\ldots,n,

where the errors have zero mean, common variance σ2\sigma^2, and zero pairwise covariance. Here XX is a single column of ones, so ordinary least squares gives the sample mean

μ^=y‾=1n∑i=1nyi.\widehat\mu=\overline y=\frac1n\sum_{i=1}^n y_i.

Any linear estimator has the form

μ~=∑i=1naiyi.\widetilde\mu=\sum_{i=1}^n a_i y_i.

Unbiasedness for every μ\mu requires ∑iai=1\sum_i a_i=1, and

Var⁡(μ~)=σ2∑i=1nai2.\operatorname{Var}(\widetilde\mu) =\sigma^2\sum_{i=1}^n a_i^2.

Since

∑i=1nai2=1n+∑i=1n(ai−1n)2,\sum_{i=1}^n a_i^2 = \frac1n+\sum_{i=1}^n\left(a_i-\frac1n\right)^2,

the minimum variance is σ2/n\sigma^2/n, attained by equal weights.

If the observations instead have unequal known variances and remain uncorrelated, inverse-variance weighting generally improves on the unweighted mean. This illustrates the importance of the equal-variance assumption.

Generalized least squares

The theorem extends to errors satisfying

Cov⁡(ε)=Ω,\operatorname{Cov}(\varepsilon)=\Omega,

where Ω\Omega is a known positive-definite matrix, not necessarily proportional to the identity.

The generalized least-squares estimator is

β^GLS=(XTΩ−1X)−1XTΩ−1y,\widehat\beta_{\mathrm{GLS}} = (X^{\mathsf T}\Omega^{-1}X)^{-1} X^{\mathsf T}\Omega^{-1}y,

with covariance

Cov⁡(β^GLS)=(XTΩ−1X)−1.\operatorname{Cov}(\widehat\beta_{\mathrm{GLS}}) = (X^{\mathsf T}\Omega^{-1}X)^{-1}.

Under the zero-mean assumption, it is BLUE for this more general covariance structure. This extension is commonly associated with Aitken.

It can be understood by transforming the model so that the transformed errors have identity covariance, then applying ordinary least squares. A diagonal Ω\Omega gives weighted least squares; off-diagonal entries permit correlated errors.

If Ω\Omega is estimated from the data, the resulting feasible generalized least-squares estimator does not automatically inherit the same exact finite-sample optimality.

Scope and limitations

Linearity and unbiasedness restrict the comparison

The theorem compares linear unbiased estimators. It does not establish that ordinary least squares minimizes every measure of estimation error among all possible estimators.

For a scalar estimator TT of θ\theta, mean squared error satisfies

E[(T−θ)2]=Var⁡(T)+(E[T]−θ)2.\mathbb E[(T-\theta)^2] = \operatorname{Var}(T) +\bigl(\mathbb E[T]-\theta\bigr)^2.

A biased estimator can trade increased bias for reduced variance. Methods such as ridge regression may therefore achieve lower mean squared error in some settings without contradicting Gauss–Markov.

Normality is unnecessary, but exact inference needs more

A normal distribution for the errors is not required for BLUE optimality. If the errors are jointly normal with covariance σ2In\sigma^2I_n, ordinary least squares is also the maximum-likelihood estimator of β\beta, and familiar exact finite-sample regression tests become available.

Without normality, the Gauss–Markov theorem alone does not justify exact Student tt or FF distributions for test statistics. Large-sample inference requires additional conditions.

When n>pn>p,

s2=eTen−ps^2=\frac{e^{\mathsf T}e}{n-p}

is an unbiased estimator of σ2\sigma^2 under the stated moment assumptions. The usual estimated coefficient covariance matrix is s2(XTX)−1s^2(X^{\mathsf T}X)^{-1}, from which conventional standard errors are obtained.

Unequal variances and correlated errors

If the zero-mean condition remains valid but error covariance is a general matrix Ω\Omega, ordinary least squares remains unbiased, with

Cov⁡(β^)=(XTX)−1XTΩX(XTX)−1.\operatorname{Cov}(\widehat\beta) = (X^{\mathsf T}X)^{-1} X^{\mathsf T}\Omega X (X^{\mathsf T}X)^{-1}.

It is no longer generally BLUE. Robust covariance estimators can support inference under specified departures from spherical covariance, but they do not change the ordinary least-squares coefficient estimates or make them efficient automatically.

Endogeneity and causal interpretation

When E[ε∣X]≠0\mathbb E[\varepsilon\mid X]\neq 0, ordinary least squares need not be unbiased. Such endogeneity can arise from omitted variables, measurement error, or simultaneous determination of regressors and responses.

The theorem also does not establish causal identification. Its efficiency statement concerns estimation within an assumed statistical model; interpreting coefficients as causal effects requires additional substantive assumptions or research design.

Rank deficiency and numerical computation

If XX lacks full column rank, the entire coefficient vector is not uniquely identified. Nevertheless, estimable linear combinations of coefficients can still have best linear unbiased estimators.

Even with full rank, near-collinearity can make coefficient variances large and computations sensitive to rounding. BLUE means best within the specified class, not necessarily precise. In computation, QR decomposition or singular value decomposition is generally preferable to explicitly forming and inverting XTXX^{\mathsf T}X.

Historical context

The theorem is named after Carl Friedrich Gauss and Andrey Markov. Gauss developed foundational results connecting least squares with minimum-variance linear unbiased estimation; Markov later contributed to the theory and its formulations. Aitken extended the framework to general known error covariance.

The theorem occupies a central place in econometrics, experimental analysis, and regression theory because it distinguishes three separate ideas: minimizing squared residuals, achieving unbiasedness, and attaining minimum variance within a defined estimator class.