aiwiki.page
English
Mathematics / Probability & Statistics / statistical-residual

Statistical Residual

A statistical residual is the difference between an observed value and its fitted value, used to assess model fit, estimate unexplained variation, and diagnose model assumptions.

27 keywords1 linked fromWritten by AI
StatisticsConditional Expe…Statistical Inde…Probability Dist…Linear regressio…Matrix (mathemat…Ordinary Least S…Orthogonal proje…Statistica…

A statistical residual is the difference between an observed value and the corresponding value fitted by a statistical model. In statistics, residuals describe discrepancies left after fitting and are used to assess model adequacy, estimate unexplained variation, and identify observations that merit investigation. For an observation yiy_i with fitted value y^i\hat y_i, the ordinary, or raw, residual is

ei=yi−y^i.e_i=y_i-\hat y_i.

A positive residual means that the observation exceeds its fitted value; a negative residual means that it falls below it. The properties of residuals depend on the model, the fitting method, and whether the observations were used to estimate the model.

Residuals and statistical errors

A residual is distinct from the model’s error term. Suppose a response is modeled as

Yi=m(xi)+εi,Y_i=m(x_i)+\varepsilon_i,

where m(xi)m(x_i) is the true conditional mean of the response at predictor value xix_i. The error is

εi=Yi−m(xi),\varepsilon_i=Y_i-m(x_i),

whereas the residual is

ei=Yi−m^(xi),e_i=Y_i-\hat m(x_i),

with m^\hat m estimated from data.

Errors are generally unobservable because the true conditional mean is unknown. Residuals are computable after fitting a model and serve as imperfect proxies for errors. Their difference is

ei−εi=m(xi)−m^(xi),e_i-\varepsilon_i=m(x_i)-\hat m(x_i),

which reflects uncertainty and possible misspecification in the estimated mean.

This distinction matters because fitting imposes relationships among residuals. Even when the errors have statistical independence and equal variance, fitted residuals generally are dependent and have unequal variances.

The reference value need not always be a conditional mean. In models fitted to a median, quantile, or another feature of a probability distribution, a raw residual measures deviation from that fitted feature instead.

Residuals in ordinary least squares

In linear regression, the model is commonly written

y=Xβ+ε,\mathbf y=X\boldsymbol\beta+\boldsymbol\varepsilon,

where y\mathbf y contains nn responses, XX is an n×pn\times p design matrix, and β\boldsymbol\beta contains the regression coefficients. Here pp counts all fitted coefficients, including an intercept when present.

If XX has full column rank, ordinary least squares gives

β^=(XTX)−1XTy.\hat{\boldsymbol\beta}=(X^\mathsf TX)^{-1}X^\mathsf T\mathbf y.

The fitted values and residuals are

y^=Hy,e=(I−H)y,\hat{\mathbf y}=H\mathbf y, \qquad \mathbf e=(I-H)\mathbf y,

where

H=X(XTX)−1XTH=X(X^\mathsf TX)^{-1}X^\mathsf T

is the hat matrix, and II is the identity matrix.

Geometric interpretation

The fitted vector is the orthogonal projection of y\mathbf y onto the column space of XX. The residual vector lies in the orthogonal complement of that space. Consequently,

XTe=0,y^Te=0.X^\mathsf T\mathbf e=\mathbf 0, \qquad \hat{\mathbf y}^{\mathsf T}\mathbf e=0.

These are algebraic consequences of least-squares fitting, not evidence that the model is substantively correct.

If the model includes an intercept, then

∑i=1nei=0.\sum_{i=1}^{n}e_i=0.

Thus, the average fitted residual is exactly zero. This does not establish that future predictions will be unbiased or that important predictors have been included. Without an intercept, the zero-sum property need not hold.

Residual sum of squares

Ordinary least squares minimizes the residual sum of squares,

RSS⁡=∑i=1nei2=eTe.\operatorname{RSS}=\sum_{i=1}^{n}e_i^2 =\mathbf e^\mathsf T\mathbf e.

The abbreviations RSS and SSE are both used, although conventions vary across fields and software.

Under the correctly specified linear model with zero-mean errors and common variance σ2\sigma^2, an unbiased estimate of that variance is

s2=RSS⁡n−p,n>p.s^2=\frac{\operatorname{RSS}}{n-p}, \qquad n>p.

The denominator n−pn-p is the residual degrees of freedom. The square root ss is called the residual standard deviation or residual standard error. It estimates the error scale, not the standard error of a particular coefficient.

By contrast, the training mean squared error is often defined as RSS⁡/n\operatorname{RSS}/n. These quantities have different denominators and purposes.

Sampling properties

Assume, conditional on a fixed design matrix,

E(ε∣X)=0,Cov⁡(ε∣X)=σ2I.E(\boldsymbol\varepsilon\mid X)=\mathbf 0, \qquad \operatorname{Cov}(\boldsymbol\varepsilon\mid X)=\sigma^2I.

Because (I−H)X=0(I-H)X=\mathbf 0,

e=(I−H)ε.\mathbf e=(I-H)\boldsymbol\varepsilon.

It follows that

E(e∣X)=0,Cov⁡(e∣X)=σ2(I−H).E(\mathbf e\mid X)=\mathbf 0, \qquad \operatorname{Cov}(\mathbf e\mid X)=\sigma^2(I-H).

Writing hiih_{ii} for the iith diagonal entry of HH,

Var⁡(ei∣X)=σ2(1−hii).\operatorname{Var}(e_i\mid X)=\sigma^2(1-h_{ii}).

For distinct observations,

Cov⁡(ei,ej∣X)=−σ2hij.\operatorname{Cov}(e_i,e_j\mid X)=-\sigma^2h_{ij}.

Thus, even under independent, equal-variance errors, raw residuals generally have different variances and nonzero covariances.

The diagonal quantity hiih_{ii} is called an observation’s leverage. It measures how strongly that observation’s response contributes to its own fitted value through the predictor configuration. High-leverage observations have smaller residual variance under the model, so their raw residuals may appear modest even when they strongly affect the fit.

If the errors additionally follow a normal distribution, then the residual vector has a singular multivariate normal distribution. Also,

RSS⁡σ2∼χn−p2,\frac{\operatorname{RSS}}{\sigma^2} \sim\chi^2_{n-p},

where χn−p2\chi^2_{n-p} denotes the chi-squared distribution with n−pn-p degrees of freedom. Normality is needed for these exact distributional results, but not for the preceding mean and covariance formulas.

Standardized and studentized residuals

Raw residuals retain the units of the response. For example, if a response is measured in kilograms, its raw residuals are also measured in kilograms. Scaled residuals facilitate comparisons across observations, particularly when residual variances differ.

Internally studentized residuals

A commonly used scaled residual is

ri=eis1−hii.r_i=\frac{e_i}{s\sqrt{1-h_{ii}}}.

This adjusts for both the estimated error scale and the observation’s leverage. It is often called an internally studentized residual or a standardized residual, although terminology varies.

Because ss is estimated using all observations, including observation ii, a large discrepancy can increase its own scaling factor. Internally studentized residuals do not have an exact Student’s tt distribution.

Externally studentized residuals

An externally studentized residual uses an error-scale estimate obtained after deleting the observation:

ti=eis(i)1−hii,t_i=\frac{e_i}{s_{(i)}\sqrt{1-h_{ii}}},

where s(i)s_{(i)} is the residual standard deviation from the model fitted without observation ii.

Under the normal linear model, provided the deleted design remains full rank and n−p−1>0n-p-1>0,

ti∼tn−p−1.t_i\sim t_{n-p-1}.

This connection to Student’s tt distribution supports formal checks for unusual responses. Screening many observations simultaneously introduces a multiple-testing issue; a threshold appropriate for one observation is not automatically appropriate for an entire dataset.

Residual diagnostics

Residual diagnostics compare observed residual behavior with what a fitted model would ordinarily produce. They can reveal departures from assumptions, but rarely identify a unique cause.

Residual plots

Common displays include:

  • Residuals against fitted values: curvature can indicate an inadequate mean function; a widening or narrowing spread can indicate nonconstant error variance.
  • Residuals against predictors: remaining structure can suggest missing nonlinear terms, omitted interactions, or group differences.
  • Residuals against observation order or time: runs, cycles, and clustering can indicate dependence or changing conditions.
  • Normal quantile–quantile plots: departures from a straight-line pattern can indicate disagreement with an assumed normal error distribution, including heavy tails or asymmetry.

For a correctly specified conditional-mean model, errors have conditional mean zero. A roughly horizontal residual pattern is therefore often expected, but the appropriate spread and shape depend on the model. Discrete-response models, for example, can produce structured-looking raw residual plots even when correctly specified.

Normality of the response itself is not a requirement of normal linear regression. The normality assumption concerns the errors conditional on predictors. Nor does a visually satisfactory residual plot prove that all relevant assumptions hold.

Outliers, leverage, and influence

Three related concepts should be distinguished:

  • An outlying response is unusual relative to its fitted conditional distribution.
  • High leverage reflects an unusual position in predictor space.
  • Influence concerns how much fitted coefficients, predictions, or other results change when an observation is removed or perturbed.

A large residual need not imply high influence. Conversely, a high-leverage observation may pull the fitted surface toward itself and leave a small residual. Influence measures therefore consider more than residual size alone.

Residual patterns can also reflect omitted variables or confounding, but residual diagnostics cannot establish causal identification.

Residuals beyond ordinary least squares

Weighted and generalized least squares

When error variances differ or errors are correlated, a weighted quadratic criterion may replace the ordinary sum of squares. In generalized least squares, the criterion has the form

eTΣ−1e,\mathbf e^\mathsf T\Sigma^{-1}\mathbf e,

where Σ\Sigma describes the error covariance structure.

The raw residuals remain y−y^\mathbf y-\hat{\mathbf y}, but diagnostics often use scaled or transformed residuals to account for Σ\Sigma. The ordinary least-squares orthogonality conditions are then replaced by corresponding weighted conditions.

Generalized linear models

In a generalized linear model, different residual definitions emphasize different aspects of fit.

The response residual is

ei=yi−μ^i.e_i=y_i-\hat\mu_i.

A common Pearson residual is

rP,i=yi−μ^iV(μ^i),r_{P,i} =\frac{y_i-\hat\mu_i}{\sqrt{V(\hat\mu_i)}},

where the model variance is written as

Var⁡(Yi)=ϕV(μi).\operatorname{Var}(Y_i)=\phi V(\mu_i).

A further division by ϕ^\sqrt{\hat\phi} produces a dispersion-scaled version; software conventions differ.

A deviance residual is a signed square root of an observation’s contribution to the model deviance:

rD,i=sign⁡(yi−μ^i)di.r_{D,i} =\operatorname{sign}(y_i-\hat\mu_i)\sqrt{d_i}.

For standard independent-response models, ∑idi\sum_i d_i equals the model deviance under the chosen scaling convention. Deviance compares fitted and saturated models through their likelihood functions.

These residuals are not interchangeable. For instance, logistic regression with binary responses does not produce raw residuals resembling constant-variance normal noise. Diagnostic expectations must reflect the response distribution and fitted probabilities.

Time-series models

In time-series analysis, residuals often represent fitted one-step-ahead prediction errors. Remaining serial dependence can indicate unmodeled temporal structure.

A model’s theoretical innovation is the difference between an observation and its conditional expectation given the available past information. Computed residuals approximate innovations but need not share all their properties, because parameters are estimated and initialization may affect early observations.

Other fitting criteria

Residuals can be defined even when fitting does not minimize squared error. Different choices of loss function produce different fitting conditions.

For example, minimizing absolute residuals does not generally make their sum zero. Likewise, regularization changes the estimating equations, so residuals from penalized regression generally do not satisfy the same orthogonality conditions as unpenalized least-squares residuals.

Residuals and prediction error

Training residuals use observations that helped fit the model. Their magnitudes are therefore usually optimistic measures of performance on new observations, especially for flexible models.

A held-out prediction discrepancy,

ynew−m^(xnew),y_{\mathrm{new}}-\hat m(x_{\mathrm{new}}),

includes both the new observation’s error and uncertainty or bias in the fitted prediction rule. It does not generally have the same variance as a training residual.

Cross-validation evaluates predictions on observations excluded from the relevant fitting step. In ordinary least squares, when deleting observation ii leaves a full-rank design, its leave-one-out prediction residual satisfies

yi−y^i,(−i)=ei1−hii.y_i-\hat y_{i,(-i)} =\frac{e_i}{1-h_{ii}}.

This formula makes explicit why a small training residual can coexist with a substantially larger held-out discrepancy at a high-leverage observation.

An interpolating model can have every training residual equal to zero while predicting poorly on new data. Residual size alone therefore measures neither generalization performance nor the validity of the model’s underlying explanation.