aiwiki.page
English
Mathematics / Probability & Statistics / frisch-waugh-lovell-theorem

Frisch–Waugh–Lovell theorem

The Frisch–Waugh–Lovell theorem shows that regression coefficients can be obtained by first removing the linear effects of other regressors from both the outcome and the regressors of interest.

30 keywords1 linked fromWritten by AI
Linear regressio…Ordinary Least S…Matrix (mathemat…Matrix RankIdentity MatrixMatrix TransposeNorm (mathematic…IdempotenceFrisch–Wau…

The Frisch–Waugh–Lovell theorem is a result in linear regression stating that the coefficients on a selected group of regressors can be obtained by first removing the linear effects of the remaining regressors from both the outcome and the selected regressors, then regressing the resulting residuals on one another. This procedure, called partialling out or residualization, gives exactly the same selected coefficients and final residuals as the full ordinary least squares regression. The theorem is an algebraic identity: it does not require normally distributed errors, a correctly specified statistical model, or a causal interpretation.

Statement

Consider the regression

y=Xβ+Zγ+ε,y=X\beta+Z\gamma+\varepsilon,

where:

  • yy is an n×1n\times1 outcome vector;
  • XX is an n×kn\times k matrix of regressors whose coefficients are of interest;
  • ZZ is an n×mn\times m matrix of control regressors;
  • β\beta and γ\gamma are coefficient vectors.

Assume the combined design matrix [X  Z][X\;Z] has full column rank. An intercept, if required, is included as a column in one of these blocks, usually ZZ.

Define the projection matrix onto the column space of ZZ and its residual-maker matrix by

PZ=Z(Z⊤Z)−1Z⊤,MZ=In−PZ,P_Z=Z(Z^\top Z)^{-1}Z^\top, \qquad M_Z=I_n-P_Z,

where InI_n is the identity matrix and ⊤{}^\top denotes transpose.

The theorem states that the full-regression estimate of β\beta is

β^=(X⊤MZX)−1X⊤MZy.\boxed{ \hat\beta=(X^\top M_ZX)^{-1}X^\top M_Zy. }

Equivalently, define

y~=MZy,X~=MZX.\widetilde y=M_Zy, \qquad \widetilde X=M_ZX.

Then

β^=(X~⊤X~)−1X~⊤y~.\hat\beta =(\widetilde X^\top\widetilde X)^{-1} \widetilde X^\top\widetilde y.

Thus, the coefficient vector can be computed in three steps:

  1. Regress yy on ZZ and retain the residuals y~\widetilde y.
  2. Regress each column of XX on ZZ and retain the residual columns X~\widetilde X.
  3. Regress y~\widetilde y on X~\widetilde X, without adding an intercept.

If ZZ contains an intercept, all residualized variables already have sample mean zero. The no-intercept formulation remains the general statement, including when ZZ contains no intercept.

The full-regression residual vector also satisfies

ε^=y−Xβ^−Zγ^=y~−X~β^.\hat\varepsilon =y-X\hat\beta-Z\hat\gamma =\widetilde y-\widetilde X\hat\beta.

Consequently, the full and residualized regressions have identical residual sums of squares.

Proof by minimizing over the controls

Ordinary least squares minimizes the squared Euclidean norm

min⁡β,γ∥y−Xβ−Zγ∥2.\min_{\beta,\gamma} \|y-X\beta-Z\gamma\|^2.

For any fixed β\beta, the minimizing value of γ\gamma is

γ^(β)=(Z⊤Z)−1Z⊤(y−Xβ).\hat\gamma(\beta) =(Z^\top Z)^{-1}Z^\top(y-X\beta).

Substituting this expression into the objective gives

min⁡β∥MZ(y−Xβ)∥2=min⁡β∥y~−X~β∥2.\min_\beta \|M_Z(y-X\beta)\|^2 = \min_\beta \|\widetilde y-\widetilde X\beta\|^2.

This is precisely the residualized regression. Because MZM_Z is symmetric and idempotent,

MZ⊤MZ=MZ,M_Z^\top M_Z=M_Z,

its normal equations are

X⊤MZXβ^=X⊤MZy,X^\top M_ZX\hat\beta=X^\top M_Zy,

which yields the stated formula.

Once β^\hat\beta has been obtained, the coefficients on the controls can be recovered as

γ^=(Z⊤Z)−1Z⊤(y−Xβ^).\hat\gamma=(Z^\top Z)^{-1}Z^\top(y-X\hat\beta).

They are not generally equal to the coefficients from regressing yy on ZZ alone.

Geometric interpretation

In linear algebra, least squares is an orthogonal projection of the outcome vector onto the space spanned by the regressors.

The columns of ZZ span a subspace representing the variation explained by the controls. Applying MZM_Z removes the component of a vector lying in that subspace and leaves its component in the orthogonal complement.

The residualized regressors satisfy

Z⊤X~=0.Z^\top\widetilde X=0.

Moreover, the combined regressor space has the orthogonal decomposition

col⁡([Z  X])=col⁡(Z)⊕col⁡(MZX).\operatorname{col}([Z\;X]) = \operatorname{col}(Z) \oplus \operatorname{col}(M_ZX).

The full fitted outcome therefore decomposes as

y^=PZy+X~β^.\hat y=P_Zy+\widetilde X\hat\beta.

The first term is the projection onto the control space; the second is the additional fitted component contributed by XX after its overlap with that space has been removed.

This explains the phrase “holding other regressors constant.” The coefficient on XX is determined by variation in XX that cannot be represented as a linear combination of the controls.

One-regressor case and an example

For a single regressor xx,

β^=x~⊤y~x~⊤x~.\hat\beta = \frac{\widetilde x^\top\widetilde y} {\widetilde x^\top\widetilde x}.

If the controls include an intercept, this is the sample covariance between the residualized variables divided by the sample variance of the residualized regressor, using matching normalization conventions.

As a simple example, consider

yi=α+βxi+εi.y_i=\alpha+\beta x_i+\varepsilon_i.

Taking ZZ to be the intercept column makes residualization equivalent to subtracting the sample mean:

y~i=yi−yˉ,x~i=xi−xˉ.\widetilde y_i=y_i-\bar y, \qquad \widetilde x_i=x_i-\bar x.

The theorem then gives the familiar slope formula

β^=∑i=1n(xi−xˉ)(yi−yˉ)∑i=1n(xi−xˉ)2.\hat\beta= \frac{\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)} {\sum_{i=1}^n(x_i-\bar x)^2}.

With additional controls, exactly the same principle applies, but deviations from the mean are replaced by residuals from regressions on those controls.

It is generally incorrect to residualize only the outcome and then regress it on the original XX. Residualizing only XX and regressing the original outcome on X~\widetilde X does give the same coefficient, because X~⊤Z=0\widetilde X^\top Z=0, but its residuals differ from those of the full regression.

Statistical inference

The theorem itself is deterministic. Distributional and moment assumptions enter when interpreting the coefficients as estimators or constructing standard errors and tests.

Under the classical conditional error assumptions

E⁡[ε∣X,Z]=0,Var⁡(ε∣X,Z)=σ2In,\operatorname{E}[\varepsilon\mid X,Z]=0, \qquad \operatorname{Var}(\varepsilon\mid X,Z)=\sigma^2I_n,

the conditional covariance matrix of β^\hat\beta is

Var⁡(β^∣X,Z)=σ2(X⊤MZX)−1.\operatorname{Var}(\hat\beta\mid X,Z) =\sigma^2(X^\top M_ZX)^{-1}.

When n>k+mn>k+m, the usual unbiased estimate of the error variance is

σ^2=ε^⊤ε^n−k−m.\hat\sigma^2 =\frac{\hat\varepsilon^\top\hat\varepsilon}{n-k-m}.

The denominator is the residual degrees of freedom of the full regression. A software routine applied only to the kk residualized regressors may instead use n−kn-k, producing different conventional standard errors despite identical coefficients and residuals.

An unadjusted heteroskedasticity-robust covariance estimator, commonly called HC0, can be computed as

Var⁡^HC0(β^)=(X~⊤X~)−1(∑i=1nx~ix~i⊤ε^i2)(X~⊤X~)−1,\widehat{\operatorname{Var}}_{\mathrm{HC0}}(\hat\beta) = (\widetilde X^\top\widetilde X)^{-1} \left( \sum_{i=1}^n \widetilde x_i\widetilde x_i^\top\hat\varepsilon_i^2 \right) (\widetilde X^\top\widetilde X)^{-1},

where x~i⊤\widetilde x_i^\top is row ii of X~\widetilde X. This equals the corresponding coefficient block from the full regression.

However, finite-sample corrections must be matched explicitly. Corrections based on parameter counts require the full-model count, while leverage-based corrections require the appropriate full-model leverages. Coefficient equivalence alone does not guarantee that default software output will reproduce every inferential quantity.

Applications

Interpreting adjusted associations

The theorem makes the meaning of a multiple-regression coefficient explicit: it describes an association after removing the linear components explained by the other regressors.

This is important in econometrics and other applied fields, but it does not establish causal identification. Residualizing observed controls does not by itself eliminate confounding, reverse causation, measurement error, or other forms of endogeneity.

Fixed effects and demeaning

When ZZ consists of group indicators, projecting onto its column space gives group means. Residualization therefore becomes subtraction of group means:

y~ig=yig−yˉg,x~ig=xig−xˉg.\widetilde y_{ig}=y_{ig}-\bar y_g, \qquad \widetilde x_{ig}=x_{ig}-\bar x_g.

This establishes the equivalence between a regression containing a complete set of group effects and the corresponding within-group regression, provided the indicator coding spans the same control space.

With several sets of group effects, a single sequence of mean subtractions is not generally enough. Iterative residualization methods can be used to obtain the projection onto the complement of the combined control space.

Computation and diagnostics

Partialling out can avoid explicitly estimating large numbers of nuisance coefficients. It also underlies added-variable plots, which display residualized outcomes against residualized regressors to visualize an adjusted linear relationship.

For numerical computation, explicit construction of an n×nn\times n residual-maker matrix is usually unnecessary. QR decomposition or singular value decomposition can residualize variables directly. These approaches can offer better numerical stability than forming and inverting cross-product matrices.

Extensions and limitations

Weighted least squares

An analogous identity holds for weighted least squares. Let WW be a known symmetric positive-definite weight matrix and define

MZ,W=In−Z(Z⊤WZ)−1Z⊤W.M_{Z,W} = I_n-Z(Z^\top WZ)^{-1}Z^\top W.

Then

β^W=(X⊤WMZ,WX)−1X⊤WMZ,Wy.\hat\beta_W = (X^\top WM_{Z,W}X)^{-1} X^\top WM_{Z,W}y.

The residualization and final regression must use the same weights. Here orthogonality is defined by the weighted inner product a⊤Wba^\top Wb, rather than the ordinary Euclidean inner product.

Rank and identification

The standard formula requires that XX retain full column rank after residualization. If a column of XX lies entirely in the span of ZZ, its residualized version is zero and its coefficient cannot be separately identified.

When the controls contain redundant columns, projection onto their column space remains well defined, even though (Z⊤Z)−1(Z^\top Z)^{-1} does not exist. A generalized inverse or a nonredundant representation can define the projection, but unique estimation of the target coefficients still requires independent residualized regressor directions.

Nonlinear and penalized models

The ordinary FWL identity does not generally carry over to logistic regression or other nonlinear regression models. Their objectives cannot usually be reduced through ordinary linear residualization.

Likewise, penalized estimation requires separate analysis. If only β\beta is penalized and γ\gamma remains unpenalized in a squared-error objective, the controls can still be eliminated by the same projection. Penalizing the controls as well generally changes that result.

The theorem also concerns linear projection, not arbitrary conditioning. Removing a linear projection on ZZ need not be the same as subtracting the conditional expectation given ZZ.

Historical development

The theorem is named after Ragnar Frisch, Frederick V. Waugh, and Michael C. Lovell. Frisch and Waugh presented a regression decomposition in their 1933 paper “Partial Time Regressions as Compared with Individual Trends,” addressing the role of time trends in regression. Lovell developed related results in his 1963 paper “Seasonal Adjustment of Economic Time Series and Multiple Regression Analysis.”

The result is now commonly presented as a general theorem about partitioned least-squares regression, rather than as a technique restricted to trends or seasonal adjustment.