{
  "id": "statistical-residual",
  "title": "Statistical Residual",
  "lang": "en",
  "source_lang": "en",
  "summary": "A statistical residual is the difference between an observed value and its fitted value, used to assess model fit, estimate unexplained variation, and diagnose model assumptions.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "Residual",
    "fitted residual",
    "regression residual"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/statistical-residual/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/statistical-residual/",
      "json": "https://aiwiki.page/zh/statistical-residual.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "A statistical residual is the difference between an observed value and the corresponding value fitted by a statistical model. In statistics, residuals describe discrepancies left after fitting and are used to assess model adequacy, estimate unexplained variation, and identify observations that merit investigation. For an observation \\(y_i\\) with fitted value \\(\\hat y_i\\), the ordinary, or raw, residual is\n\n\\[e_i=y_i-\\hat y_i.\\]\n\nA positive residual means that the observation exceeds its fitted value; a negative residual means that it falls below it. The properties of residuals depend on the model, the fitting method, and whether the observations were used to estimate the model."
    },
    {
      "heading": "Residuals and statistical errors",
      "text": "A residual is distinct from the model’s error term. Suppose a response is modeled as\n\n\\[Y_i=m(x_i)+\\varepsilon_i,\\]\n\nwhere \\(m(x_i)\\) is the true conditional mean of the response at predictor value \\(x_i\\). The error is\n\n\\[\\varepsilon_i=Y_i-m(x_i),\\]\n\nwhereas the residual is\n\n\\[e_i=Y_i-\\hat m(x_i),\\]\n\nwith \\(\\hat m\\) estimated from data.\n\nErrors are generally unobservable because the true conditional mean is unknown. Residuals are computable after fitting a model and serve as imperfect proxies for errors. Their difference is\n\n\\[e_i-\\varepsilon_i=m(x_i)-\\hat m(x_i),\\]\n\nwhich reflects uncertainty and possible misspecification in the estimated mean.\n\nThis distinction matters because fitting imposes relationships among residuals. Even when the errors have statistical independence and equal variance, fitted residuals generally are dependent and have unequal variances.\n\nThe reference value need not always be a conditional mean. In models fitted to a median, quantile, or another feature of a probability distribution, a raw residual measures deviation from that fitted feature instead."
    },
    {
      "heading": "Residuals in ordinary least squares",
      "text": "In linear regression, the model is commonly written\n\n\\[\\mathbf y=X\\boldsymbol\\beta+\\boldsymbol\\varepsilon,\\]\n\nwhere \\(\\mathbf y\\) contains \\(n\\) responses, \\(X\\) is an \\(n\\times p\\) design matrix, and \\(\\boldsymbol\\beta\\) contains the regression coefficients. Here \\(p\\) counts all fitted coefficients, including an intercept when present.\n\nIf \\(X\\) has full column rank, ordinary least squares gives\n\n\\[\\hat{\\boldsymbol\\beta}=(X^\\mathsf TX)^{-1}X^\\mathsf T\\mathbf y.\\]\n\nThe fitted values and residuals are\n\n\\[\\hat{\\mathbf y}=H\\mathbf y, \\qquad \\mathbf e=(I-H)\\mathbf y,\\]\n\nwhere\n\n\\[H=X(X^\\mathsf TX)^{-1}X^\\mathsf T\\]\n\nis the hat matrix, and \\(I\\) is the identity matrix.\n\nGeometric interpretation\n\nThe fitted vector is the orthogonal projection of \\(\\mathbf y\\) onto the column space of \\(X\\). The residual vector lies in the orthogonal complement of that space. Consequently,\n\n\\[X^\\mathsf T\\mathbf e=\\mathbf 0, \\qquad \\hat{\\mathbf y}^{\\mathsf T}\\mathbf e=0.\\]\n\nThese are algebraic consequences of least-squares fitting, not evidence that the model is substantively correct.\n\nIf the model includes an intercept, then\n\n\\[\\sum_{i=1}^{n}e_i=0.\\]\n\nThus, the average fitted residual is exactly zero. This does not establish that future predictions will be unbiased or that important predictors have been included. Without an intercept, the zero-sum property need not hold.\n\nResidual sum of squares\n\nOrdinary least squares minimizes the residual sum of squares,\n\n\\[\\operatorname{RSS}=\\sum_{i=1}^{n}e_i^2 =\\mathbf e^\\mathsf T\\mathbf e.\\]\n\nThe abbreviations RSS and SSE are both used, although conventions vary across fields and software.\n\nUnder the correctly specified linear model with zero-mean errors and common variance \\(\\sigma^2\\), an unbiased estimate of that variance is\n\n\\[s^2=\\frac{\\operatorname{RSS}}{n-p}, \\qquad n>p.\\]\n\nThe denominator \\(n-p\\) is the residual degrees of freedom. The square root \\(s\\) is called the residual standard deviation or residual standard error. It estimates the error scale, not the standard error of a particular coefficient.\n\nBy contrast, the training mean squared error is often defined as \\(\\operatorname{RSS}/n\\). These quantities have different denominators and purposes."
    },
    {
      "heading": "Sampling properties",
      "text": "Assume, conditional on a fixed design matrix,\n\n\\[E(\\boldsymbol\\varepsilon\\mid X)=\\mathbf 0, \\qquad \\operatorname{Cov}(\\boldsymbol\\varepsilon\\mid X)=\\sigma^2I.\\]\n\nBecause \\((I-H)X=\\mathbf 0\\),\n\n\\[\\mathbf e=(I-H)\\boldsymbol\\varepsilon.\\]\n\nIt follows that\n\n\\[E(\\mathbf e\\mid X)=\\mathbf 0, \\qquad \\operatorname{Cov}(\\mathbf e\\mid X)=\\sigma^2(I-H).\\]\n\nWriting \\(h_{ii}\\) for the \\(i\\)th diagonal entry of \\(H\\),\n\n\\[\\operatorname{Var}(e_i\\mid X)=\\sigma^2(1-h_{ii}).\\]\n\nFor distinct observations,\n\n\\[\\operatorname{Cov}(e_i,e_j\\mid X)=-\\sigma^2h_{ij}.\\]\n\nThus, even under independent, equal-variance errors, raw residuals generally have different variances and nonzero covariances.\n\nThe diagonal quantity \\(h_{ii}\\) is called an observation’s leverage. It measures how strongly that observation’s response contributes to its own fitted value through the predictor configuration. High-leverage observations have smaller residual variance under the model, so their raw residuals may appear modest even when they strongly affect the fit.\n\nIf the errors additionally follow a normal distribution, then the residual vector has a singular multivariate normal distribution. Also,\n\n\\[\\frac{\\operatorname{RSS}}{\\sigma^2} \\sim\\chi^2_{n-p},\\]\n\nwhere \\(\\chi^2_{n-p}\\) denotes the chi-squared distribution with \\(n-p\\) degrees of freedom. Normality is needed for these exact distributional results, but not for the preceding mean and covariance formulas."
    },
    {
      "heading": "Standardized and studentized residuals",
      "text": "Raw residuals retain the units of the response. For example, if a response is measured in kilograms, its raw residuals are also measured in kilograms. Scaled residuals facilitate comparisons across observations, particularly when residual variances differ.\n\nInternally studentized residuals\n\nA commonly used scaled residual is\n\n\\[r_i=\\frac{e_i}{s\\sqrt{1-h_{ii}}}.\\]\n\nThis adjusts for both the estimated error scale and the observation’s leverage. It is often called an internally studentized residual or a standardized residual, although terminology varies.\n\nBecause \\(s\\) is estimated using all observations, including observation \\(i\\), a large discrepancy can increase its own scaling factor. Internally studentized residuals do not have an exact Student’s \\(t\\) distribution.\n\nExternally studentized residuals\n\nAn externally studentized residual uses an error-scale estimate obtained after deleting the observation:\n\n\\[t_i=\\frac{e_i}{s_{(i)}\\sqrt{1-h_{ii}}},\\]\n\nwhere \\(s_{(i)}\\) is the residual standard deviation from the model fitted without observation \\(i\\).\n\nUnder the normal linear model, provided the deleted design remains full rank and \\(n-p-1>0\\),\n\n\\[t_i\\sim t_{n-p-1}.\\]\n\nThis connection to Student’s t distribution supports formal checks for unusual responses. Screening many observations simultaneously introduces a multiple-testing issue; a threshold appropriate for one observation is not automatically appropriate for an entire dataset."
    },
    {
      "heading": "Residual diagnostics",
      "text": "Residual diagnostics compare observed residual behavior with what a fitted model would ordinarily produce. They can reveal departures from assumptions, but rarely identify a unique cause.\n\nResidual plots\n\nCommon displays include:\n\n- Residuals against fitted values: curvature can indicate an inadequate mean function; a widening or narrowing spread can indicate nonconstant error variance.\n- Residuals against predictors: remaining structure can suggest missing nonlinear terms, omitted interactions, or group differences.\n- Residuals against observation order or time: runs, cycles, and clustering can indicate dependence or changing conditions.\n- Normal quantile–quantile plots: departures from a straight-line pattern can indicate disagreement with an assumed normal error distribution, including heavy tails or asymmetry.\n\nFor a correctly specified conditional-mean model, errors have conditional mean zero. A roughly horizontal residual pattern is therefore often expected, but the appropriate spread and shape depend on the model. Discrete-response models, for example, can produce structured-looking raw residual plots even when correctly specified.\n\nNormality of the response itself is not a requirement of normal linear regression. The normality assumption concerns the errors conditional on predictors. Nor does a visually satisfactory residual plot prove that all relevant assumptions hold.\n\nOutliers, leverage, and influence\n\nThree related concepts should be distinguished:\n\n- An outlying response is unusual relative to its fitted conditional distribution.\n- High leverage reflects an unusual position in predictor space.\n- Influence concerns how much fitted coefficients, predictions, or other results change when an observation is removed or perturbed.\n\nA large residual need not imply high influence. Conversely, a high-leverage observation may pull the fitted surface toward itself and leave a small residual. Influence measures therefore consider more than residual size alone.\n\nResidual patterns can also reflect omitted variables or confounding, but residual diagnostics cannot establish causal identification."
    },
    {
      "heading": "Residuals beyond ordinary least squares",
      "text": "Weighted and generalized least squares\n\nWhen error variances differ or errors are correlated, a weighted quadratic criterion may replace the ordinary sum of squares. In generalized least squares, the criterion has the form\n\n\\[\\mathbf e^\\mathsf T\\Sigma^{-1}\\mathbf e,\\]\n\nwhere \\(\\Sigma\\) describes the error covariance structure.\n\nThe raw residuals remain \\(\\mathbf y-\\hat{\\mathbf y}\\), but diagnostics often use scaled or transformed residuals to account for \\(\\Sigma\\). The ordinary least-squares orthogonality conditions are then replaced by corresponding weighted conditions.\n\nGeneralized linear models\n\nIn a generalized linear model, different residual definitions emphasize different aspects of fit.\n\nThe response residual is\n\n\\[e_i=y_i-\\hat\\mu_i.\\]\n\nA common Pearson residual is\n\n\\[r_{P,i} =\\frac{y_i-\\hat\\mu_i}{\\sqrt{V(\\hat\\mu_i)}},\\]\n\nwhere the model variance is written as\n\n\\[\\operatorname{Var}(Y_i)=\\phi V(\\mu_i).\\]\n\nA further division by \\(\\sqrt{\\hat\\phi}\\) produces a dispersion-scaled version; software conventions differ.\n\nA deviance residual is a signed square root of an observation’s contribution to the model deviance:\n\n\\[r_{D,i} =\\operatorname{sign}(y_i-\\hat\\mu_i)\\sqrt{d_i}.\\]\n\nFor standard independent-response models, \\(\\sum_i d_i\\) equals the model deviance under the chosen scaling convention. Deviance compares fitted and saturated models through their likelihood functions.\n\nThese residuals are not interchangeable. For instance, logistic regression with binary responses does not produce raw residuals resembling constant-variance normal noise. Diagnostic expectations must reflect the response distribution and fitted probabilities.\n\nTime-series models\n\nIn time-series analysis, residuals often represent fitted one-step-ahead prediction errors. Remaining serial dependence can indicate unmodeled temporal structure.\n\nA model’s theoretical innovation is the difference between an observation and its conditional expectation given the available past information. Computed residuals approximate innovations but need not share all their properties, because parameters are estimated and initialization may affect early observations.\n\nOther fitting criteria\n\nResiduals can be defined even when fitting does not minimize squared error. Different choices of loss function produce different fitting conditions.\n\nFor example, minimizing absolute residuals does not generally make their sum zero. Likewise, regularization changes the estimating equations, so residuals from penalized regression generally do not satisfy the same orthogonality conditions as unpenalized least-squares residuals."
    },
    {
      "heading": "Residuals and prediction error",
      "text": "Training residuals use observations that helped fit the model. Their magnitudes are therefore usually optimistic measures of performance on new observations, especially for flexible models.\n\nA held-out prediction discrepancy,\n\n\\[y_{\\mathrm{new}}-\\hat m(x_{\\mathrm{new}}),\\]\n\nincludes both the new observation’s error and uncertainty or bias in the fitted prediction rule. It does not generally have the same variance as a training residual.\n\nCross-validation evaluates predictions on observations excluded from the relevant fitting step. In ordinary least squares, when deleting observation \\(i\\) leaves a full-rank design, its leave-one-out prediction residual satisfies\n\n\\[y_i-\\hat y_{i,(-i)} =\\frac{e_i}{1-h_{ii}}.\\]\n\nThis formula makes explicit why a small training residual can coexist with a substantially larger held-out discrepancy at a high-leverage observation.\n\nAn interpolating model can have every training residual equal to zero while predicting poorly on new data. Residual size alone therefore measures neither generalization performance nor the validity of the model’s underlying explanation."
    }
  ],
  "links": [
    "statistics",
    "conditional-expectation",
    "statistical-independence",
    "probability-distribution",
    "linear-regression",
    "matrix",
    "ordinary-least-squares",
    "orthogonal-projection",
    "orthogonal-complement",
    "variance",
    "degrees-of-freedom",
    "standard-error",
    "mean-squared-error",
    "covariance",
    "normal-distribution",
    "chi-squared-distribution",
    "students-t-distribution",
    "multiple-testing",
    "confounding",
    "causal-inference",
    "generalized-linear-model",
    "likelihood-function",
    "logistic-regression",
    "time-series",
    "loss-function",
    "regularization",
    "cross-validation"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}