aiwiki.page
English
Mathematics / Probability & Statistics / regression-to-the-mean

Regression to the mean

Regression to the mean is the tendency for unusually extreme observations to be followed by less extreme observations when repeated measurements are imperfectly correlated.

32 keywords12 linked from2 not yet writtenWritten by AI
CorrelationLinear regressio…Standard Deviati…Conditional Expe…Multivariate Nor…Mean squared err…Prediction Inter…Measurement Erro…Regression…

Regression to the mean is a statistical phenomenon in which observations selected for being unusually high or low tend, on average, to be less extreme when measured again. It occurs when the measurements are imperfectly correlated and the initial extremeness partly reflects variation that is not reproduced on the second occasion. A high examination score, an exceptional sporting performance, or an unusually elevated laboratory measurement may combine a persistent characteristic with a temporary fluctuation; selecting the extreme observation also tends to select an unusually large fluctuation.

The phenomenon is important because an expected statistical change can be mistaken for the effect of a treatment, punishment, reward, or policy. It also underlies useful methods for predicting future observations and estimating differences among individuals or institutions. Regression to the mean describes conditional averages, not an inevitable movement of every observation toward a common value. Its precise form depends on the relevant population, the relationship between measurements, and the process by which observations were selected.

Historical origins and the meaning of “regression”

The concept emerged from the nineteenth-century investigations of Francis Galton into heredity and variation. In experiments reported in 1877, Galton examined the relationship between the sizes of parent and offspring sweet peas. Seeds descended from unusually large parent seeds were, on average, less unusually large; offspring of unusually small seeds were, on average, less unusually small. Galton initially described the pattern as “reversion,” interpreting it in relation to the distribution of inherited characteristics.

His paper “Regression towards Mediocrity in Hereditary Stature,” published in 1886, developed the idea using human height. Galton compared the heights of adult children with a combined parental height, adjusting the measurements to make male and female heights comparable. Children of exceptionally tall parents tended to be taller than average, but their average height was less exceptional than that of their parents. Likewise, children of unusually short parents were generally short without being, on average, equally extreme. The historical word “mediocrity” referred to an average or central condition, not a judgment of quality.

This work helped establish the language of correlation and linear regression. Modern regression analysis is much broader: it models how the distribution or average of an outcome depends on one or more explanatory variables. A fitted regression equation need not describe any temporal movement toward an average. The historical term therefore survives in a technical framework whose applications extend far beyond the phenomenon that originally motivated it.

An apparent paradox in the height example is that regression toward the mean can occur in both directions. Tall parents have, on average, less exceptionally tall children, but tall children also have, on average, less exceptionally tall parents. There is no contradiction because the two statements condition on different groups. Selecting families by parental height does not produce the same sample as selecting them by the height of their children.

Nor does regression imply that the population’s variation must steadily disappear. A stable distribution can contain both a conditional tendency toward its center and continuing individual departures from it. Random variation and imperfect correspondence between generations can preserve the population spread even though averages conditional on extreme parental characteristics are less extreme. This distinction between a conditional relationship and the behavior of an entire distribution remains central to the modern interpretation.

Mathematical formulation and its assumptions

Let XX be an initial measurement and YY a second measurement, with means μX,μY\mu_X,\mu_Y, positive standard deviations σX,σY\sigma_X,\sigma_Y, and correlation ρ\rho. If their conditional expectation is linear, then

E⁡[Y∣X=x]=μY+ρσYσX(x−μX).\operatorname{E}[Y\mid X=x] = \mu_Y+\rho\frac{\sigma_Y}{\sigma_X}(x-\mu_X).

This formula holds exactly for a nondegenerate jointly normal distribution. More generally, the same expression is the best linear predictor under mean squared error, although the true conditional expectation may be nonlinear. The distinction matters: knowledge of the correlation alone does not determine the complete conditional relationship.

Define standardized measurements by

ZX=X−μXσX,ZY=Y−μYσY.Z_X=\frac{X-\mu_X}{\sigma_X}, \qquad Z_Y=\frac{Y-\mu_Y}{\sigma_Y}.

Under the linear conditional-mean assumption,

E⁡[ZY∣ZX=z]=ρz.\operatorname{E}[Z_Y\mid Z_X=z]=\rho z.

For 0<ρ<10<\rho<1, the conditional average has the same sign but a smaller magnitude than the initial standardized observation. If an initial result is two standard deviations above its mean and ρ=0.6\rho=0.6, the expected second result is 1.21.2 standard deviations above its own mean. It remains above average; regression does not mean a complete return to the center.

When both measurements have the same mean μ\mu and standard deviation, the expected change has the particularly simple form

E⁡[Y−X∣X=x]=−(1−ρ)(x−μ).\operatorname{E}[Y-X\mid X=x] = -(1-\rho)(x-\mu).

An initial value above μ\mu is therefore followed by an expected decrease, and a value below μ\mu by an expected increase. With ρ=0\rho=0, the initial value provides no linear predictive information; under independence, the second conditional mean is simply μ\mu. With ρ=1\rho=1, there is no contraction in standardized units. Negative correlations reverse the sign of the conditional deviation, requiring care with informal descriptions of a “return.”

Regression concerns a distribution, not just a predicted value. In the jointly normal case,

Var⁡(Y∣X=x)=σY2(1−ρ2).\operatorname{Var}(Y\mid X=x)=\sigma_Y^2(1-\rho^2).

Individual follow-up observations can consequently be more extreme than their initial values, even when their conditional average is less extreme. A prediction interval expresses this uncertainty; the conditional mean alone does not.

Comparing raw deviations can also be misleading when the two measurements have different spreads. Their raw regression slope is ρσY/σX\rho\sigma_Y/\sigma_X, which may exceed one despite ∣ρ∣<1|\rho|<1. The clean contraction statement applies to standardized conditional deviations. Neither an arbitrary joint distribution nor a single extreme observation, by itself, establishes that regression toward a particular mean must occur.

Stable characteristics, temporary variation, and measurement error

A useful model separates a stable individual characteristic from occasion-specific variation:

X=θ+ε1,Y=θ+ε2.X=\theta+\varepsilon_1, \qquad Y=\theta+\varepsilon_2.

Here θ\theta varies across individuals, while ε1\varepsilon_1 and ε2\varepsilon_2 represent temporary influences or measurement error. Suppose the errors have mean zero, are independent of each other and of θ\theta, and have equal variance σε2\sigma_\varepsilon^2. If θ\theta has mean μ\mu and variance σθ2\sigma_\theta^2, then

Var⁡(X)=σθ2+σε2,Cov⁡(X,Y)=σθ2.\operatorname{Var}(X)=\sigma_\theta^2+\sigma_\varepsilon^2, \qquad \operatorname{Cov}(X,Y)=\sigma_\theta^2.

The correlation between repeated measurements is therefore

ρ=σθ2σθ2+σε2.\rho= \frac{\sigma_\theta^2} {\sigma_\theta^2+\sigma_\varepsilon^2}.

In this model, the ratio is also a reliability coefficient: it measures the fraction of observed variation attributable to stable differences. High reliability leaves relatively little scope for regression; low reliability means that exceptional observations contain a larger temporary component. Reliability is population-dependent because the same measurement error can represent a small fraction of variation in a heterogeneous population and a large fraction in a homogeneous one.

If the stable characteristics and errors are normally distributed, the expected stable value given an observation is

E⁡[θ∣X=x]=μ+ρ(x−μ).\operatorname{E}[\theta\mid X=x] = \mu+\rho(x-\mu).

The expected repeat measurement has the same value because the second error has conditional mean zero. An unusually high observation is thus interpreted partly as evidence of a high stable characteristic and partly as evidence of a favorable temporary fluctuation. The model does not identify either component exactly for a particular individual.

For example, consider a hypothetical test with population mean 100, observed standard deviation 15, and repeat-measurement correlation 0.8. Someone initially scoring 130 has an expected repeat score of

100+0.8(130−100)=124.100+0.8(130-100)=124.

The predicted decrease of six points does not require a loss of underlying ability. In this model, selecting the score of 130 preferentially selects people whose first-occasion error was positive. A score below 130 on the next occasion is compatible with entirely unchanged stable ability.

“Error” in this decomposition need not mean a defective instrument. Variation in sleep, motivation, weather, opponents, or short-term biological conditions can produce the same statistical structure. Correlated temporary influences, changing underlying characteristics, and unequal measurement precision alter the formulas. Averaging several independent measurements reduces the error variance of the average, but it does not remove genuine changes over time or errors shared across occasions.

Selection of extremes and apparent relationships with change

Regression to the mean becomes especially visible when inclusion depends on an extreme baseline measurement. A program might enroll the lowest-scoring pupils, inspect factories after unusually many accidents, or evaluate a service following a particularly poor year. Even with no effective intervention, later measurements can improve because the selection rule preferentially captured unfavorable temporary fluctuations. This is one way that selection bias can distort an apparent intervention effect.

Suppose XX and YY have the same mean μ\mu, standard deviation σ\sigma, and a linear conditional relationship with correlation ρ\rho. For observations selected because X>cX>c,

E⁡[Y−X∣X>c]=−(1−ρ){E⁡[X∣X>c]−μ}.\operatorname{E}[Y-X\mid X>c] = -(1-\rho) \left\{\operatorname{E}[X\mid X>c]-\mu\right\}.

The average selected baseline exceeds the population mean, so the expected follow-up change is negative when ρ<1\rho<1. More extreme selection generally produces a larger expected change. Selecting a group for low values produces the corresponding expected increase.

For a normal distribution, write a=(c−μ)/σa=(c-\mu)/\sigma. The selected baseline mean is

E⁡[X∣X>c]=μ+σϕ(a)1−Φ(a),\operatorname{E}[X\mid X>c] = \mu+\sigma\frac{\phi(a)}{1-\Phi(a)},

where ϕ\phi and Φ\Phi are the standard normal density and cumulative distribution functions. If selection requires a baseline more than two standard deviations above average, the selected group’s mean is approximately 2.372.37 standard deviations above average. With ρ=0.5\rho=0.5, its expected follow-up mean is approximately 1.191.19 standard deviations above average. That substantial decline follows from the selection rule and joint distribution, without any treatment effect.

A related difficulty arises when change is plotted against baseline. Define D=Y−XD=Y-X. By the algebra of covariance,

Cov⁡(X,D)=Cov⁡(X,Y)−Var⁡(X).\operatorname{Cov}(X,D) = \operatorname{Cov}(X,Y)-\operatorname{Var}(X).

With equal variances, this becomes

Cov⁡(X,D)=−(1−ρ)σ2.\operatorname{Cov}(X,D)=-(1-\rho)\sigma^2.

Thus a negative relationship between baseline and change can arise partly because baseline appears with a minus sign inside the change score. Observing that those who started highest fell most does not, by itself, establish a biological feedback mechanism or show that an intervention worked best for them.

The relevant selection process can extend beyond a stated cutoff. A manager may initiate action after an unusually bad month, a researcher may investigate a subgroup because its result was striking, or an institution may be chosen because it occupied the bottom of a ranking. Identifying the population from which that exceptional observation was selected is essential: a mean calculated from the selected extremes is generally not the reference mean governing their expected repeat measurements.

Consequences for experiments and causal interpretation

Regression to the mean is a problem for causal inference when an expected change is attributed to an action taken after an extreme observation. Consider a group receiving an intervention because its initial outcomes are poor. The observed improvement combines any intervention effect with changes that would have happened without the intervention. Regression is one component of that untreated change, alongside trends, maturation, seasonal effects, and other influences.

A one-group before-and-after comparison cannot generally separate these components. In particular, a statistically significant difference between baseline and follow-up does not establish that the intervention caused the difference. A test of zero average change addresses a different question from whether the treated group changed more than it would have changed untreated. Greater sample size may estimate the before-and-after difference very precisely while leaving this causal problem unresolved.

A randomized controlled trial addresses the problem by allocating eligible participants to treatment and control groups after applying the same entry criteria. If eligibility depends on an extreme baseline value, both groups are subject to regression. Under valid randomization, it contributes equally to their expected untreated outcomes, rather than systematically favoring the treatment group. The between-group comparison can therefore identify a treatment effect even when both groups improve markedly from baseline.

Baseline adjustment can increase precision and account for chance imbalances. A common model is

Yfollow-up=α+τT+βXbaseline+ε,Y_{\text{follow-up}} = \alpha+\tau T+\beta X_{\text{baseline}}+\varepsilon,

where TT indicates treatment assignment. The coefficient τ\tau compares follow-up outcomes conditional on baseline under the model. This approach is often described as analysis of covariance, within the broader family of analysis of variance methods. It does not assume that the baseline coefficient is one, as a simple unadjusted change-score formulation effectively does. Interactions or nonlinear terms may be needed when relationships differ across groups or baseline levels.

Nonrandomized designs require additional assumptions. A comparison group selected without the same extreme-baseline rule may experience a different amount of regression. Likewise, difference-in-differences can be misleading if treatment begins specifically in response to an unusually unfavorable temporary outcome: the treated group’s expected rebound can violate the assumed untreated parallel trend.

Repeated measurements before intervention help distinguish a persistent level from a temporary excursion. They can improve estimates of baseline status and reveal pre-existing trends, although they do not independently guarantee causal identification. The design question remains whether the comparison captures the outcome trajectory that the treated group would have followed without treatment, including any regression induced by its selection.

Prediction, shrinkage, and Bayesian estimation

Regression to the mean is not merely a source of error in interpretation. It also explains why predictions based directly on extreme observations are often too extreme. If a measured value combines a persistent signal with transient noise, a prediction that treats the entire observed deviation as persistent will tend to overstate future differences. Moving the prediction toward an appropriate reference mean can reduce expected prediction error.

A normal model makes this connection explicit. Suppose an unknown individual parameter has prior distribution

θ∼N(μ,τ2),\theta\sim N(\mu,\tau^2),

and an observed estimate satisfies

X∣θ∼N(θ,s2).X\mid\theta\sim N(\theta,s^2).

Under Bayesian inference, the posterior distribution is normal, with mean

E⁡[θ∣X=x]=μ+τ2τ2+s2(x−μ)\operatorname{E}[\theta\mid X=x] = \mu+ \frac{\tau^2}{\tau^2+s^2}(x-\mu)

and variance

Var⁡(θ∣X=x)=τ2s2τ2+s2.\operatorname{Var}(\theta\mid X=x) = \frac{\tau^2s^2}{\tau^2+s^2}.

The factor τ2/(τ2+s2)\tau^2/(\tau^2+s^2) determines how much of the observed deviation is retained. Precise observations receive little shrinkage; noisy observations receive more.

This yields a distinction between regression to the mean as a property of repeated observations and shrinkage as an estimation procedure. Regression is present in the relevant joint distribution whether or not an analyst models it. Shrinkage deliberately uses information about that distribution to estimate a latent parameter or predict a later observation. A suitable estimator anticipates the expected regression instead of interpreting the observed extreme as wholly stable.

For example, an institution with only a small number of observed cases may have a highly variable outcome rate. Another institution with many cases may have a much more precise rate. A hierarchical model can shrink the first estimate more strongly toward the mean of comparable institutions while leaving the second largely unchanged. The appropriate reference population matters: institutions serving systematically different populations need not share the same underlying distribution.

In empirical Bayes methods, features such as the population mean and between-unit variance are estimated from the ensemble of observations. These methods connect regression to the mean with partial pooling: individual units retain distinct estimates, but those estimates borrow information from the larger group. The shrinkage weights depend on both measurement uncertainty and genuine between-unit variation.

Connections also exist with regularization in predictive modeling. For example, ridge regression contracts fitted coefficients to control instability, though its target and mathematical setting differ from predicting a repeat measurement toward a population mean. Shrinkage is not automatically justified merely because a value is extreme. An inappropriate reference group, an incorrect model of uncertainty, or a genuine structural shift can make a centralizing estimate misleading.

Regression to the mean and temporal mean reversion

Regression to the mean is often confused with mean reversion in a time series. The ideas overlap mathematically but are not identical. Regression can arise between two measurements without any time ordering, as in the relationship between parent and offspring heights or two different tests taken at one occasion. Temporal mean reversion concerns the evolution of a process and whether deviations from a reference level predict subsequent changes.

A simple autoregressive model illustrates the overlap:

Xt+1−μ=ϕ(Xt−μ)+ηt+1,X_{t+1}-\mu = \phi(X_t-\mu)+\eta_{t+1},

where the innovations have conditional mean zero given the past. If 0<ϕ<10<\phi<1, then

E⁡[Xt+1−Xt∣Xt]=−(1−ϕ)(Xt−μ).\operatorname{E}[X_{t+1}-X_t\mid X_t] = -(1-\phi)(X_t-\mu).

The conditional mean moves toward μ\mu, and after kk steps,

E⁡[Xt+k∣Xt]=μ+ϕk(Xt−μ).\operatorname{E}[X_{t+k}\mid X_t] = \mu+\phi^k(X_t-\mu).

Under the usual finite-variance innovation assumptions, ∣ϕ∣<1|\phi|<1 supports a stationary model. A negative ϕ\phi produces alternating conditional deviations rather than a monotone approach.

In this model there is an explicit temporal transition rule. By contrast, repeated noisy measurements of an unchanged latent characteristic can show regression without any restorative movement in the characteristic itself. A high instrument reading may be followed by a lower reading because its first measurement error was unusually positive, not because the underlying object changed in response to being high.

A random walk provides a useful contrast. If

Xt+1=Xt+ηt+1X_{t+1}=X_t+\eta_{t+1}

with conditionally mean-zero increments, then

E⁡[Xt+1∣Xt]=Xt.\operatorname{E}[X_{t+1}\mid X_t]=X_t.

There is no expected return to a fixed long-run mean. A series does not become mean-reverting simply because its current value appears historically extreme. Deterministic trends, structural breaks, and changing reference levels further complicate attempts to infer a stable center from observed reversals.

The law of large numbers is another distinct concept. It describes the convergence of averages under appropriate assumptions; it does not require future observations to compensate for past extremes. With independent fair coin tosses, a run of heads does not make tails more likely on the next toss. The observed proportion can approach one-half because the fixed early imbalance becomes less important as the total number of tosses grows.

Consequently, a physical restoring force, regression of repeat measurements, temporal mean reversion, and convergence of sample averages require separate explanations. Similar-looking graphs can be generated by different mechanisms, and the existence of a historical average alone does not identify which mechanism is operating.

Applications in performance assessment and scientific research

Performance rankings provide a prominent application. An athlete’s season statistics combine persistent skill, health, playing conditions, opponent quality, and chance. Selecting the highest performer selects both strong skill and an unusually favorable realization of those other influences. If the next season’s temporary influences are less favorable, performance declines without requiring a loss of skill. The same reasoning applies to unusually poor seasons, which may be followed by improvement without a special corrective event.

An award or prominent media appearance often occurs after an exceptional performance. A subsequent decline can therefore be associated with the recognition even if recognition has no causal effect. This is a selection-based explanation of some apparent “jinxes,” not a claim that all performance changes are statistical. Changes in age, training, injury, competition, or opportunity can also matter, and their contributions require separate evidence.

Educational comparisons have a related structure. Schools selected for very low examination results may improve on the next assessment partly because the first cohort or occasion was unusually unfavorable. Year-to-year changes also depend on cohort composition and the number of pupils assessed. An extreme average from a small school is usually less precise than the same average from a much larger school, making sample size and uncertainty important in interpreting rankings. Repeated cohorts and appropriately matched comparisons help distinguish persistent differences from temporary extremes.

Scientific research produces another form of selection. Suppose many studies estimate effects with sampling error, and attention concentrates on those with the largest estimates or smallest p-values. If selection favors an extreme positive estimate, that estimate is likely to combine a genuine effect with positive sampling error. An independent replication can consequently produce a smaller estimate even when the underlying effect is unchanged. This pattern is often called the winner’s curse; it is related to, but not identical with, publication bias, which concerns selective availability of results.

The amount of expected shrinkage in replication depends on the selection rule, sampling uncertainty, and distribution of underlying effects. It is not correct to assume that every published estimate must decline by a fixed proportion. Selection by statistical significance alone also differs from selection of the largest estimate across many analyses. Multiple testing can increase opportunities to find an extreme result, while preregistration records hypotheses and analysis plans before outcomes are known, helping distinguish planned analyses from data-dependent selection.

Across these applications, the relevant mean may be conditional rather than universal: expected performance can depend on age, institution type, case mix, or historical period. Evaluation therefore requires the original selection rule, the precision of the measurements, the relationship between repeated outcomes, and an appropriate reference population. Without those ingredients, neither a rebound nor a decline identifies how much of the observed change reflects regression to the mean.