A statistical residual is the difference between an observed value and the corresponding value fitted by a statistical model. In statistics, residuals describe discrepancies left after fitting and are used to assess model adequacy, estimate unexplained variation, and identify observations that merit investigation. For an observation with fitted value , the ordinary, or raw, residual is
A positive residual means that the observation exceeds its fitted value; a negative residual means that it falls below it. The properties of residuals depend on the model, the fitting method, and whether the observations were used to estimate the model.
Residuals and statistical errors
A residual is distinct from the model’s error term. Suppose a response is modeled as
where is the true conditional mean of the response at predictor value . The error is
whereas the residual is
with estimated from data.
Errors are generally unobservable because the true conditional mean is unknown. Residuals are computable after fitting a model and serve as imperfect proxies for errors. Their difference is
which reflects uncertainty and possible misspecification in the estimated mean.
This distinction matters because fitting imposes relationships among residuals. Even when the errors have statistical independence and equal variance, fitted residuals generally are dependent and have unequal variances.
The reference value need not always be a conditional mean. In models fitted to a median, quantile, or another feature of a probability distribution, a raw residual measures deviation from that fitted feature instead.
Residuals in ordinary least squares
In linear regression, the model is commonly written
where contains responses, is an design matrix, and contains the regression coefficients. Here counts all fitted coefficients, including an intercept when present.
If has full column rank, ordinary least squares gives
The fitted values and residuals are
where
is the hat matrix, and is the identity matrix.
Geometric interpretation
The fitted vector is the orthogonal projection of onto the column space of . The residual vector lies in the orthogonal complement of that space. Consequently,
These are algebraic consequences of least-squares fitting, not evidence that the model is substantively correct.
If the model includes an intercept, then
Thus, the average fitted residual is exactly zero. This does not establish that future predictions will be unbiased or that important predictors have been included. Without an intercept, the zero-sum property need not hold.
Residual sum of squares
Ordinary least squares minimizes the residual sum of squares,
The abbreviations RSS and SSE are both used, although conventions vary across fields and software.
Under the correctly specified linear model with zero-mean errors and common variance , an unbiased estimate of that variance is
The denominator is the residual degrees of freedom. The square root is called the residual standard deviation or residual standard error. It estimates the error scale, not the standard error of a particular coefficient.
By contrast, the training mean squared error is often defined as . These quantities have different denominators and purposes.
Sampling properties
Assume, conditional on a fixed design matrix,
Because ,
It follows that
Writing for the th diagonal entry of ,
For distinct observations,
Thus, even under independent, equal-variance errors, raw residuals generally have different variances and nonzero covariances.
The diagonal quantity is called an observation’s leverage. It measures how strongly that observation’s response contributes to its own fitted value through the predictor configuration. High-leverage observations have smaller residual variance under the model, so their raw residuals may appear modest even when they strongly affect the fit.
If the errors additionally follow a normal distribution, then the residual vector has a singular multivariate normal distribution. Also,
where denotes the chi-squared distribution with degrees of freedom. Normality is needed for these exact distributional results, but not for the preceding mean and covariance formulas.
Standardized and studentized residuals
Raw residuals retain the units of the response. For example, if a response is measured in kilograms, its raw residuals are also measured in kilograms. Scaled residuals facilitate comparisons across observations, particularly when residual variances differ.
Internally studentized residuals
A commonly used scaled residual is
This adjusts for both the estimated error scale and the observation’s leverage. It is often called an internally studentized residual or a standardized residual, although terminology varies.
Because is estimated using all observations, including observation , a large discrepancy can increase its own scaling factor. Internally studentized residuals do not have an exact Student’s distribution.
Externally studentized residuals
An externally studentized residual uses an error-scale estimate obtained after deleting the observation:
where is the residual standard deviation from the model fitted without observation .
Under the normal linear model, provided the deleted design remains full rank and ,
This connection to Student’s distribution supports formal checks for unusual responses. Screening many observations simultaneously introduces a multiple-testing issue; a threshold appropriate for one observation is not automatically appropriate for an entire dataset.
Residual diagnostics
Residual diagnostics compare observed residual behavior with what a fitted model would ordinarily produce. They can reveal departures from assumptions, but rarely identify a unique cause.
Residual plots
Common displays include:
- Residuals against fitted values: curvature can indicate an inadequate mean function; a widening or narrowing spread can indicate nonconstant error variance.
- Residuals against predictors: remaining structure can suggest missing nonlinear terms, omitted interactions, or group differences.
- Residuals against observation order or time: runs, cycles, and clustering can indicate dependence or changing conditions.
- Normal quantile–quantile plots: departures from a straight-line pattern can indicate disagreement with an assumed normal error distribution, including heavy tails or asymmetry.
For a correctly specified conditional-mean model, errors have conditional mean zero. A roughly horizontal residual pattern is therefore often expected, but the appropriate spread and shape depend on the model. Discrete-response models, for example, can produce structured-looking raw residual plots even when correctly specified.
Normality of the response itself is not a requirement of normal linear regression. The normality assumption concerns the errors conditional on predictors. Nor does a visually satisfactory residual plot prove that all relevant assumptions hold.
Outliers, leverage, and influence
Three related concepts should be distinguished:
- An outlying response is unusual relative to its fitted conditional distribution.
- High leverage reflects an unusual position in predictor space.
- Influence concerns how much fitted coefficients, predictions, or other results change when an observation is removed or perturbed.
A large residual need not imply high influence. Conversely, a high-leverage observation may pull the fitted surface toward itself and leave a small residual. Influence measures therefore consider more than residual size alone.
Residual patterns can also reflect omitted variables or confounding, but residual diagnostics cannot establish causal identification.
Residuals beyond ordinary least squares
Weighted and generalized least squares
When error variances differ or errors are correlated, a weighted quadratic criterion may replace the ordinary sum of squares. In generalized least squares, the criterion has the form
where describes the error covariance structure.
The raw residuals remain , but diagnostics often use scaled or transformed residuals to account for . The ordinary least-squares orthogonality conditions are then replaced by corresponding weighted conditions.
Generalized linear models
In a generalized linear model, different residual definitions emphasize different aspects of fit.
The response residual is
A common Pearson residual is
where the model variance is written as
A further division by produces a dispersion-scaled version; software conventions differ.
A deviance residual is a signed square root of an observation’s contribution to the model deviance:
For standard independent-response models, equals the model deviance under the chosen scaling convention. Deviance compares fitted and saturated models through their likelihood functions.
These residuals are not interchangeable. For instance, logistic regression with binary responses does not produce raw residuals resembling constant-variance normal noise. Diagnostic expectations must reflect the response distribution and fitted probabilities.
Time-series models
In time-series analysis, residuals often represent fitted one-step-ahead prediction errors. Remaining serial dependence can indicate unmodeled temporal structure.
A model’s theoretical innovation is the difference between an observation and its conditional expectation given the available past information. Computed residuals approximate innovations but need not share all their properties, because parameters are estimated and initialization may affect early observations.
Other fitting criteria
Residuals can be defined even when fitting does not minimize squared error. Different choices of loss function produce different fitting conditions.
For example, minimizing absolute residuals does not generally make their sum zero. Likewise, regularization changes the estimating equations, so residuals from penalized regression generally do not satisfy the same orthogonality conditions as unpenalized least-squares residuals.
Residuals and prediction error
Training residuals use observations that helped fit the model. Their magnitudes are therefore usually optimistic measures of performance on new observations, especially for flexible models.
A held-out prediction discrepancy,
includes both the new observation’s error and uncertainty or bias in the fitted prediction rule. It does not generally have the same variance as a training residual.
Cross-validation evaluates predictions on observations excluded from the relevant fitting step. In ordinary least squares, when deleting observation leaves a full-rank design, its leave-one-out prediction residual satisfies
This formula makes explicit why a small training residual can coexist with a substantially larger held-out discrepancy at a high-leverage observation.
An interpolating model can have every training residual equal to zero while predicting poorly on new data. Residual size alone therefore measures neither generalization performance nor the validity of the model’s underlying explanation.