The Gauss–Markov theorem is a result in statistics establishing the optimality of ordinary least squares within a specified class of estimators. In a linear regression model whose errors have zero mean, equal finite variance, and no pairwise correlation, the ordinary least-squares estimator has the smallest covariance matrix among all estimators that are linear in the observed responses and unbiased for the regression coefficients. It is therefore called the best linear unbiased estimator, or BLUE. The theorem does not require normally distributed errors, and its optimality claim does not extend automatically to biased or nonlinear estimators.
Model and assumptions
Consider the model
where:
- is an vector of observed responses;
- is an design matrix;
- is a vector of unknown, nonrandom coefficients;
- is an vector of random errors.
The standard fixed-design version treats as known and nonrandom. Its assumptions are:
Full column rank: , so and the coefficients are uniquely identified.
Zero-mean errors: .
Spherical error covariance:
Here denotes a covariance matrix, and is the identity matrix. The third assumption combines equal error variances, called homoscedasticity, with zero covariance between distinct errors.
Zero covariance does not generally imply statistical independence. Independent errors are sufficient for the absence of error correlation, but independence is stronger than the theorem requires.
If is random, a common formulation instead assumes
with full column rank almost surely. The theorem then applies conditionally on . The conditional-mean assumption is stronger than merely requiring an unconditional zero mean.
“Linear model” means linear in the unknown coefficients, not necessarily linear in the original explanatory variables. A model containing , , and , for example, still satisfies this structural requirement when their coefficients enter linearly.
Precise statement
Under these assumptions, the ordinary least-squares estimator is
where denotes the transpose of . Full column rank ensures that is invertible.
Its mean and covariance are
Consider any other estimator
where is a fixed matrix, possibly depending on the known design , but not on the observed response or the unknown coefficients. Such an estimator is linear in the observations. It is unbiased for every precisely when
This is the usual unbiasedness requirement: its expected value equals the parameter throughout the model.
The theorem states that
where means positive semidefinite: every associated quadratic form is nonnegative.
Equivalently, for every fixed ,
Thus ordinary least squares minimizes the variance of every estimated linear combination of the coefficients among linear unbiased estimators. “Best” refers to this covariance ordering, not to the magnitudes of individual observed errors or to a guarantee of accurate prediction in every dataset.
Proof
Define
Then and .
For any competing linear unbiased estimator , write
Because ,
Using the spherical covariance assumption,
The cross terms vanish, since
Consequently,
For every vector ,
The covariance difference is therefore positive semidefinite, proving the theorem. Since , equality of the entire covariance matrices requires , establishing uniqueness within this class.
The argument uses only first and second moments. No particular probability density or distributional shape enters the proof.
Geometric interpretation
The fitted response
is the orthogonal projection of onto the column space of . The residual vector
satisfies
so it is perpendicular to every column of the design matrix.
In the proof, a competing estimator differs from ordinary least squares by , with . This extra term has zero mean and is uncorrelated with under spherical error covariance. Adding it therefore cannot reduce covariance; it contributes the nonnegative term .
This geometric interpretation connects least-squares minimization with statistical efficiency. Orthogonal projection explains why the residual sum of squares is minimized, while the covariance assumptions explain why the resulting coefficients are BLUE.
Example: estimating a common mean
Suppose
where the errors have zero mean, common variance , and zero pairwise covariance. Here is a single column of ones, so ordinary least squares gives the sample mean
Any linear estimator has the form
Unbiasedness for every requires , and
Since
the minimum variance is , attained by equal weights.
If the observations instead have unequal known variances and remain uncorrelated, inverse-variance weighting generally improves on the unweighted mean. This illustrates the importance of the equal-variance assumption.
Generalized least squares
The theorem extends to errors satisfying
where is a known positive-definite matrix, not necessarily proportional to the identity.
The generalized least-squares estimator is
with covariance
Under the zero-mean assumption, it is BLUE for this more general covariance structure. This extension is commonly associated with Aitken.
It can be understood by transforming the model so that the transformed errors have identity covariance, then applying ordinary least squares. A diagonal gives weighted least squares; off-diagonal entries permit correlated errors.
If is estimated from the data, the resulting feasible generalized least-squares estimator does not automatically inherit the same exact finite-sample optimality.
Scope and limitations
Linearity and unbiasedness restrict the comparison
The theorem compares linear unbiased estimators. It does not establish that ordinary least squares minimizes every measure of estimation error among all possible estimators.
For a scalar estimator of , mean squared error satisfies
A biased estimator can trade increased bias for reduced variance. Methods such as ridge regression may therefore achieve lower mean squared error in some settings without contradicting Gauss–Markov.
Normality is unnecessary, but exact inference needs more
A normal distribution for the errors is not required for BLUE optimality. If the errors are jointly normal with covariance , ordinary least squares is also the maximum-likelihood estimator of , and familiar exact finite-sample regression tests become available.
Without normality, the Gauss–Markov theorem alone does not justify exact Student or distributions for test statistics. Large-sample inference requires additional conditions.
When ,
is an unbiased estimator of under the stated moment assumptions. The usual estimated coefficient covariance matrix is , from which conventional standard errors are obtained.
Unequal variances and correlated errors
If the zero-mean condition remains valid but error covariance is a general matrix , ordinary least squares remains unbiased, with
It is no longer generally BLUE. Robust covariance estimators can support inference under specified departures from spherical covariance, but they do not change the ordinary least-squares coefficient estimates or make them efficient automatically.
Endogeneity and causal interpretation
When , ordinary least squares need not be unbiased. Such endogeneity can arise from omitted variables, measurement error, or simultaneous determination of regressors and responses.
The theorem also does not establish causal identification. Its efficiency statement concerns estimation within an assumed statistical model; interpreting coefficients as causal effects requires additional substantive assumptions or research design.
Rank deficiency and numerical computation
If lacks full column rank, the entire coefficient vector is not uniquely identified. Nevertheless, estimable linear combinations of coefficients can still have best linear unbiased estimators.
Even with full rank, near-collinearity can make coefficient variances large and computations sensitive to rounding. BLUE means best within the specified class, not necessarily precise. In computation, QR decomposition or singular value decomposition is generally preferable to explicitly forming and inverting .
Historical context
The theorem is named after Carl Friedrich Gauss and Andrey Markov. Gauss developed foundational results connecting least squares with minimum-variance linear unbiased estimation; Markov later contributed to the theory and its formulations. Aitken extended the framework to general known error covariance.
The theorem occupies a central place in econometrics, experimental analysis, and regression theory because it distinguishes three separate ideas: minimizing squared residuals, achieving unbiasedness, and attaining minimum variance within a defined estimator class.