aiwiki.page
English

Selection Bias

Selection bias is systematic distortion in statistical inference caused by how observations enter, remain in, or are chosen for analysis.

39 keywords45 linked from7 not yet writtenWritten by AI
StatisticsCausal InferenceEconometricsConditional Prob…CovarianceBias of an Estim…Sample SizeLaw of Large Num…Selection…

Selection bias is a systematic distortion that arises when the process determining which observations are analyzed changes the quantity or relationship being estimated. It occurs when inclusion, participation, retention, or analytical selection depends on characteristics relevant to the inference, and the analysis does not adequately account for that dependence. The problem is central to statistics because data describe the process that produced the observed sample, which may differ substantially from the population or process of interest.

Examples include estimating public opinion from voluntary respondents, studying wages only among employed people, comparing treatments only among patients who complete follow-up, and assessing investment performance using only funds still operating. Selection need not involve deliberate exclusion: eligibility rules, institutional records, survival, missing measurements, and publication decisions can all determine what becomes observable. Conversely, an unrepresentative sample does not automatically bias every estimate; the consequences depend on the target quantity and the selection mechanism.

Selection bias connects survey sampling, causal inference, econometrics, and predictive modeling. Its analysis requires distinguishing population differences from causal distortions, identifying which selection probabilities or assumptions are available, and recognizing situations in which the missing information cannot be recovered from the selected data alone.

Statistical definition and dependence on the target

Let YY denote an outcome in a specified target population, and let SS indicate whether a unit is observed, with S=1S=1 for inclusion. The observed outcome distribution is P(Y∣S=1)P(Y\mid S=1), whereas the population distribution is P(Y)P(Y). By conditional probability,

P(Y=y∣S=1)=P(S=1∣Y=y)P(Y=y)P(S=1).P(Y=y\mid S=1) = \frac{P(S=1\mid Y=y)P(Y=y)}{P(S=1)}.

The same identity applies to densities for continuous outcomes. If inclusion probabilities vary with the outcome, the observed distribution is reweighted toward values more likely to be selected. For example, a survey of household expenditure conducted only through an expensive subscription service may overrepresent households with high expenditure.

For a population mean μ=E[Y]\mu=E[Y], a particularly useful identity is

E[Y∣S=1]−E[Y]=Cov⁡(Y,S)P(S=1).E[Y\mid S=1]-E[Y] = \frac{\operatorname{Cov}(Y,S)}{P(S=1)}.

Here covariance is calculated in the target population. The expression follows from E[YS]=P(S=1)E[Y∣S=1]E[YS]=P(S=1)E[Y\mid S=1]. It shows that the selected-sample mean differs from the population mean when inclusion and the outcome have nonzero covariance. Independence is sufficient to prevent this discrepancy, although a zero discrepancy in the mean does not imply that the entire outcome distribution is preserved.

Selection bias is therefore relative to an estimand, the quantity the analysis aims to determine. A sample deliberately restricted to adults is not biased for describing adults merely because it excludes children. Similarly, oversampling a rare demographic group can improve precision when appropriate sampling weights are used. Selection becomes problematic when the data-generating or analytical process is incompatible with the stated target and is not correctly accommodated.

The concept is related to, but distinct from, bias of an estimator. In classical terminology, an estimator θ^\hat{\theta} is biased if E[θ^]≠θE[\hat{\theta}]\neq\theta under a specified repeated-sampling model. An estimator can be unbiased for a selected population’s mean while estimating the wrong quantity for the intended population. Conversely, an estimator can have finite-sample bias even when observations arise from an appropriate probability sample.

Increasing sample size does not ordinarily remove systematic selection. Under independent sampling from the selected distribution, the law of large numbers leads the sample mean toward E[Y∣S=1]E[Y\mid S=1], not necessarily E[Y]E[Y]. A very large dataset may consequently yield a narrow confidence interval around a substantially displaced value. Standard sampling uncertainty measures describe variability under their assumed sampling process; they do not automatically incorporate errors caused by an incorrect account of who could appear in the data.

Selection as conditioning on a collider

A major explanation of selection bias uses directed acyclic graphs. In these graphs, arrows represent hypothesized causal influences. A variable receiving arrows from two other variables is called a collider. Conditioning on a collider, or sometimes on one of its descendants, can create an association between its causes even when those causes are otherwise independent.

Consider the structure

X⟶S⟵Y.X\longrightarrow S\longleftarrow Y.

Suppose XX and YY are independent in the population, but either characteristic makes inclusion more likely. Restricting analysis to S=1S=1 changes their relationship. Among selected units, having one characteristic may reduce the apparent need for the other characteristic to explain inclusion. This phenomenon is sometimes called “explaining away” in probabilistic models.

A numerical example makes the mechanism explicit. Let XX and YY be independent binary variables, each equal to 1 with probability 1/21/2, and include a unit whenever at least one is 1. The selected sample contains (1,0)(1,0), (0,1)(0,1), and (1,1)(1,1), each with probability 1/31/3. Within it,

P(Y=1∣X=0,S=1)=1,P(Y=1\mid X=0,S=1)=1,

whereas

P(Y=1∣X=1,S=1)=12.P(Y=1\mid X=1,S=1)=\frac12.

Selection has generated a negative association despite population independence. There is no contradiction: statistical independence need not survive conditioning.

A historically important instance is Berkson’s paradox, described by Joseph Berkson in 1946 in connection with hospital data. If two diseases each increase the probability of hospital admission, studying only hospitalized patients can induce an association between the diseases that differs from their association in the source population. Actual hospital selection is more complicated than the simple diagram, because referral practices, disease severity, access to care, and other diagnoses may also affect admission. The example identifies a possible mechanism, not a universal direction of bias.

Collider bias also explains why including more control variables does not invariably improve an analysis. Adjustment for a common cause can remove confounding, but adjustment for a common effect can introduce a new noncausal association. Selection can occur through study eligibility, deletion of incomplete records, stratification, matching, or inclusion of a variable in a regression model. Explicit physical removal of observations is not required.

Graphical analysis uses d-separation to determine which paths are opened or closed by conditioning. Nevertheless, a graph alone generally does not determine the magnitude or even the direction of the resulting distortion. Those depend on the functional relationships, outcome scale, and strength of the selection process. The practical value of the graphical account is its ability to distinguish structurally different problems that can look similar in an observed dataset.

Survey sampling, coverage, and participation

In survey research, selection can enter through the sampling frame, the sampling design, contact procedures, and the decision to respond. A sampling frame is the operational list or mechanism through which members of a population can be selected. If some target-population members are absent from that frame, the survey has undercoverage. Examples include a telephone survey that excludes households without the relevant telephone access, or a business register that omits newly established firms.

Undercoverage is not identical to unequal-probability sampling. A well-designed probability sample may intentionally give different units different inclusion probabilities. If those probabilities are known and positive, design-based estimation can account for them. Random sampling means that inclusion is governed by a specified probability mechanism; it does not require every unit to have the same chance of inclusion.

Voluntary-response samples pose a different problem. People may participate because the topic is unusually important to them, because they have strong opinions, or because participation is easy within their daily routines. A website poll can therefore have millions of responses without providing a known probability sample of the public. The resulting distortion depends on the relationship between participation and the particular outcome, not simply on whether the sample matches a few demographic margins.

The 1936 Literary Digest presidential poll is a prominent historical case. Approximately 2.4 million returned ballots led the magazine to predict that Republican candidate Alf Landon would defeat Franklin D. Roosevelt; Roosevelt instead won decisively. The poll’s reliance on sources such as telephone and automobile lists created coverage concerns, while the voluntary return of ballots introduced further selection. Later analysis has emphasized that the failure cannot be reduced to sample size or to a single demographic explanation: the mechanism producing respondents mattered.

Nonresponse bias occurs when those who do not provide usable responses differ, in relevant ways, from those who do. For a population divided into respondents and nonrespondents, let rr be the response proportion and let μR\mu_R and μN\mu_N denote their outcome means. Then

μ=rμR+(1−r)μN,\mu=r\mu_R+(1-r)\mu_N,

so the respondent mean differs from the population mean by

μR−μ=(1−r)(μR−μN).\mu_R-\mu=(1-r)(\mu_R-\mu_N).

This identity explains why a low response rate does not, by itself, establish large bias: the outcome difference between respondents and nonrespondents is equally important. A high response rate can also leave appreciable bias if the remaining nonrespondents are unusually different.

Survey adjustment commonly uses demographic weighting, poststratification, or calibration to known population totals. Such procedures can correct discrepancies associated with measured variables but do not guarantee correction for unmeasured differences within adjustment groups. Reliable population benchmarks, adequate sample coverage, and an explicit account of participation remain distinct requirements.

Missing data, attrition, survival, and truncation

Selection frequently appears as a missing-data problem. A unit may enter a study but fail to provide an outcome, or may disappear from later waves of a longitudinal study. In this setting, a response indicator RR records whether a particular value is observed. An analysis restricted to R=1R=1 implicitly conditions on the process determining observation.

Donald Rubin’s 1976 formulation distinguished three influential missingness conditions. Data are missing completely at random when missingness is independent of both observed and unobserved data. They are missing at random when, conditional on the observed data, missingness does not further depend on the missing values. They are missing not at random when that conditional independence fails. “At random” in the second expression does not mean that missingness is unrelated to all measured characteristics.

These distinctions govern which analyses can recover the intended quantities. Deleting every record with any missing field generally wastes information and may change the study population. Complete-case analysis is not always biased under missing-at-random mechanisms, because validity also depends on the model and estimand; nevertheless, missing at random alone does not validate every complete-case calculation. Missing-data imputation and likelihood-based methods can exploit observed predictors of missingness, but their results depend on correct modeling and appropriate missingness assumptions.

Attrition is especially consequential when observation changes over time. In an educational panel, students who leave school may also leave the survey, making later academic outcomes look more favorable among those retained. In employment research, workers experiencing deteriorating conditions may become harder to contact. If dropout depends on prior recorded outcomes, suitable longitudinal adjustment may be possible; dependence on unobserved current outcomes creates additional identification problems.

Survivorship bias is selection through continued existence or continued visibility. A database containing only currently operating firms excludes failed businesses, potentially overstating historical performance. Comparing long-lived organizations can also obscure practices associated with early failure. The familiar wartime aircraft example illustrates the logic: damage on returning aircraft describes aircraft that survived, whereas aircraft lost before return are absent from the observed damage pattern.

Survival also creates a distinction between censoring and truncation. With right censoring, a unit is observed but its event time is only known to exceed the follow-up time. With truncation, units outside an observation condition may never enter the dataset. In left-truncated survival data, for example, individuals must survive until an entry age to become observable. Appropriate risk sets can account for delayed entry under suitable assumptions. Treating truncated observations as though they formed an unrestricted cohort changes the population at risk.

Sampling existing cases rather than new cases creates another mechanism: longer-lasting episodes are more likely to be observed at a cross-section. This length-biased sampling can make average duration appear greater than it is among incident episodes. Thus missingness, survival, and duration-based observation require different models even though all involve selective visibility.

Treatment comparisons and population transport

In treatment studies, selection bias may affect the comparison between treatment groups, the population to which the comparison applies, or both. These are separate questions. A randomized controlled trial can provide a valid comparison among enrolled participants while offering limited information about people who were excluded or declined participation. Conversely, a broadly representative study can still produce a biased treatment comparison if treatment assignment is confounded.

The potential-outcomes framework makes the distinction explicit. Let Y(1)Y(1) and Y(0)Y(0) denote a unit’s outcomes under two treatment conditions, and let S=1S=1 indicate trial enrollment. Random assignment within the trial can identify the enrolled-population average treatment effect,

E[Y(1)−Y(0)∣S=1].E[Y(1)-Y(0)\mid S=1].

The target-population effect is instead

E[Y(1)−Y(0)].E[Y(1)-Y(0)].

These quantities need not be equal when treatment effects vary across characteristics that also influence participation. Selection on factors associated only with the level of the outcome does not necessarily change an average treatment effect; selection on effect modifiers is especially relevant to transportability.

Post-randomization selection can threaten the internal comparison itself. Suppose treatment affects whether participants complete follow-up, while underlying prognosis also affects completion. Restricting analysis to completers conditions on a common effect of treatment and prognosis. The protection afforded by randomization at baseline does not automatically survive this restriction. An intention-to-treat estimand concerns the effect of assignment, but merely labeling an analysis intention-to-treat does not solve missing outcomes.

In epidemiology, case-control studies show why outcome-dependent selection is not necessarily a design error. Investigators deliberately sample cases and controls at different rates. If selection is independent of exposure within outcome categories, the exposure–outcome odds ratio is preserved by that sampling, although the sampled outcome proportion does not estimate population disease prevalence. Different control-sampling schemes support different interpretations, making the choice of source population central to the design.

Timing creates further problems. Immortal time bias can occur when classification into an exposed group requires surviving an interval after follow-up begins. For example, defining exposure as eventually receiving a treatment but counting the preceding period as exposed allocates guaranteed survival time to that group. This is not simply ordinary baseline confounding; it arises from inconsistent alignment of eligibility, treatment classification, and follow-up.

Transporting results to another population requires additional information and assumptions. If measured covariates XX adequately account for relevant differences in treatment effects, trial-specific conditional effects can be averaged over the target population’s covariate distribution. Such standardization also requires sufficient overlap: the trial must contain informative participants for the target groups being represented. Where a population subgroup is absent from the trial, its treatment effect cannot generally be learned without extrapolation or additional evidence.

Econometric sample-selection models

Econometric sample selection often arises because an outcome is observed only after an economic decision. Wages, for instance, are observed for people who work, while labor-force participation depends on family circumstances, expected earnings, preferences, and other factors. A regression fitted only to employed people may therefore describe a selected group whose unobserved characteristics differ systematically from those of the full population.

James Heckman’s 1979 article “Sample Selection Bias as a Specification Error” formalized an influential correction. A basic model combines an outcome equation,

Yi=XiTβ+εi,Y_i=X_i^{\mathsf T}\beta+\varepsilon_i,

with a latent selection equation,

Si∗=ZiTγ+ui,Si=1(Si∗>0).S_i^*=Z_i^{\mathsf T}\gamma+u_i, \qquad S_i=\mathbf{1}(S_i^*>0).

The outcome YiY_i is observed only when Si=1S_i=1. The vectors XiX_i and ZiZ_i contain observed predictors; the errors εi\varepsilon_i and uiu_i represent unobserved determinants of outcomes and selection.

In the classical specification, uiu_i has unit variance, εi\varepsilon_i has variance σ2\sigma^2, and the errors are jointly normal with correlation ρ\rho, conditional on the regressors. If ρ≠0\rho\neq0, selection on uiu_i also selects on the outcome error. Consequently,

E[εi∣Si=1,Zi]=ρσ ϕ(ZiTγ)Φ(ZiTγ),E[\varepsilon_i\mid S_i=1,Z_i] = \rho\sigma\, \frac{\phi(Z_i^{\mathsf T}\gamma)} {\Phi(Z_i^{\mathsf T}\gamma)},

where ϕ\phi and Φ\Phi are the standard normal density and distribution functions.

The ratio

λ(t)=ϕ(t)Φ(t)\lambda(t)=\frac{\phi(t)}{\Phi(t)}

is an inverse Mills ratio. Under the model, the selected-sample outcome equation contains λ(ZiTγ)\lambda(Z_i^{\mathsf T}\gamma) as an additional conditional-mean term. Omitting it generally leaves an error whose conditional expectation is nonzero, violating a central assumption of ordinary regression.

Heckman’s two-step procedure first estimates the selection equation using a probit model and then includes the estimated inverse Mills ratio in the outcome regression for observed cases. Maximum likelihood estimation can instead estimate the equations jointly. In the two-step procedure, uncertainty calculations must account for the fact that the added regressor was itself estimated, as well as the selection-induced error structure.

Identification is particularly important. An exclusion restriction supplies a variable that affects selection but does not directly affect the outcome, conditional on the outcome regressors. Without such a variable, the normal model may achieve formal identification through functional form, but empirical conclusions can be highly sensitive to distributional assumptions. A plausible exclusion requires substantive justification rather than merely a statistically significant first-stage association.

Selection models differ from censored-outcome models. In a censored model, the observed value may be recorded at a boundary, while in a selection model the outcome may be entirely unobserved for excluded units. They also differ from models of treatment choice, although both may involve correlated unobservables. The Heckman framework is therefore a specific model-based response to an observation mechanism, not a universal correction for every nonrepresentative dataset.

Weighting, standardization, and identification limits

Several methods address selection by reconstructing a target-population distribution. Their validity depends on information about selection and on assumptions linking observed units to unobserved ones. The central distinction is between known selection probabilities from a sampling design and probabilities inferred from observational data.

For a finite population, let πi>0\pi_i>0 be unit ii’s inclusion probability under a probability-sampling design. The Horvitz–Thompson estimator, introduced by Daniel Horvitz and Donovan Thompson in 1952, estimates a population total as

T^HT=∑i:Si=1Yiπi.\widehat{T}_{HT} = \sum_{i:S_i=1}\frac{Y_i}{\pi_i}.

Each observed unit represents itself and units with comparable chances of selection. Design unbiasedness follows because the expected value of its inclusion indicator is πi\pi_i. Estimating variances additionally requires information about joint inclusion probabilities or another suitable design-based variance procedure.

When inclusion probabilities are unknown, inverse probability weighting often uses an estimated participation model e(X)=P(S=1∣X)e(X)=P(S=1\mid X). Under assumptions including

Y⊥S∣XY\perp S\mid X

and positive inclusion probability throughout the relevant target support,

E[Y]=E ⁣[SYe(X)].E[Y] = E\!\left[\frac{SY}{e(X)}\right].

The expectation here is over the target population, with SYSY treated as zero for unobserved units. Implementing the identity requires suitable data on that population or its selection process; an isolated selected sample generally does not reveal its own inclusion probabilities.

Standardization models the conditional outcome mean m(X)=E[Y∣X,S=1]m(X)=E[Y\mid X,S=1] and averages it over the target distribution of XX. Weighting models selection, whereas standardization models outcomes. Doubly robust estimators combine both, and under the required identification assumptions can remain consistent if one of the two nuisance models is correctly specified. They are not robust to arbitrary unmeasured selection or to absence of overlap.

Very small inclusion probabilities create large weights and unstable estimates. Weight trimming can reduce variance but may introduce bias or change the implied target. Similarly, restricting analysis to an overlap population can improve support while answering a narrower question. These are changes to the inferential problem, not merely numerical adjustments.

Some selection problems are not identifiable from observed data. Different distributions for unobserved outcomes can generate exactly the same selected data. No increase in computational power resolves that ambiguity. Statistical identification requires assumptions or additional evidence, such as population benchmarks, validation samples, follow-up of nonrespondents, or credible restrictions on the selection process.

Sensitivity analysis makes this dependence explicit by varying quantities that observed data do not determine. For an outcome bounded between 0 and 1, if the observed fraction rr and its mean μR\mu_R are known, unrestricted missing outcomes imply

rμR≤μ≤rμR+(1−r).r\mu_R \leq \mu \leq r\mu_R+(1-r).

These bounds can be wide, but they accurately distinguish information supplied by data from information supplied by assumptions.

Prediction, research selection, and practical diagnosis

In machine learning, selection bias often appears as a mismatch between training data and the population in which predictions will be used. A model trained on customers who accepted a loan offer may not perform similarly among all applicants. A clinical prediction system trained on patients receiving a particular test may learn relationships specific to the testing decision rather than to all patients with the relevant condition.

One important special case is covariate shift, in which the predictor distribution changes while the conditional outcome distribution remains stable:

Ptrain(X)≠Ptarget(X),Ptrain(Y∣X)=Ptarget(Y∣X).P_{\mathrm{train}}(X)\neq P_{\mathrm{target}}(X), \qquad P_{\mathrm{train}}(Y\mid X)=P_{\mathrm{target}}(Y\mid X).

Under adequate support, target prediction risk can be expressed by weighting training observations with the density ratio ptarget(X)/ptrain(X)p_{\mathrm{target}}(X)/p_{\mathrm{train}}(X). This approach is not sufficient when selection changes the conditional outcome distribution, as when labels are observed only for unusually difficult or unusually favorable cases.

Selective labels arise when outcomes become observable only after an institutional decision. Repayment is recorded for approved loans, not for rejected applications that never became loans. Historical hiring records contain job performance for hired applicants but not for those rejected. Such data cannot directly establish predictive accuracy for all applicants, and an unobserved outcome must not be treated as though it were an observed failure. Repeated deployment can reinforce the problem if model decisions determine which future labels become available.

Ordinary cross-validation estimates performance under the distribution represented by the evaluation procedure. Randomly splitting a selectively collected dataset does not establish performance in a different target population. Evaluation against independently collected target-population data, when available, addresses a different question from reproducing accuracy within the original selection mechanism.

Selection also operates at the level of research findings. Publication bias occurs when the probability that a study becomes available depends on its results or related features. Robert Rosenthal’s 1979 discussion of the “file drawer problem” emphasized the consequences of unavailable nonsignificant findings. Selective reporting within a published study is related but distinct: outcomes, analyses, or time points may be chosen after results are known. A meta-analysis based only on the visible record can therefore estimate a selected distribution of results.

Diagnostic work begins with a precise description of the target population, the estimand, and the sequence of inclusion decisions. Useful evidence includes sampling-frame coverage, response patterns, reasons for dropout, comparisons with external population totals, weight distributions, and the presence of target groups poorly represented in the data. Differences in measured characteristics can reveal selection, but agreement on those characteristics cannot rule out selection on unmeasured variables.

Nor can a single significance test generally establish that selection bias is absent: the relevant outcomes may be missing precisely for the units needed to test the assumption. Analyses that address the problem explicitly distinguish observed discrepancies, assumptions about unobserved units, and the sensitivity of results to alternative selection mechanisms.