aiwiki.page
English
Mathematics / Probability & Statistics / multiple-comparisons-problem

Multiple Comparisons Problem

The multiple comparisons problem arises when testing or selecting among many hypotheses increases the risk of misleading statistical conclusions.

41 keywords1 linked from9 not yet writtenWritten by AI
Statistical Hypo…Experimental Des…Significance Lev…Null HypothesisStatistical Inde…Expected valueP-valueAnalysis of Vari…Multiple C…

The multiple comparisons problem arises when several statistical comparisons are conducted, or when a reported comparison is selected from several possibilities, so that error guarantees for an individual comparison do not automatically apply to the collection. In statistical hypothesis testing, a procedure that rarely produces a false positive in one prespecified test may produce at least one false positive quite frequently when repeated across many outcomes, groups, models, or analytical choices. The problem also affects estimation: ordinary confidence intervals need not retain their advertised coverage when many are examined simultaneously or only the most favorable results are reported.

Multiplicity is central to experimental design, high-dimensional research, and the interpretation of statistical evidence. Its treatment depends on the scientific claim being made. Confirming any one of several possible treatment benefits, identifying a list of promising genes, and estimating whichever effect looks largest are different inferential tasks and need not use the same error criterion. Methods therefore range from simultaneous inference and family-wise error control to false discovery rate procedures, selective inference, and hierarchical modeling.

How multiplicity produces misleading evidence

Consider mm tests, each conducted at significance level α\alpha, with every null hypothesis true. If the tests have exact rejection probability α\alpha and satisfy statistical independence, the probability of at least one false rejection is

Pr⁡(at least one false rejection)=1−(1−α)m.\Pr(\text{at least one false rejection})=1-(1-\alpha)^m.

At α=0.05\alpha=0.05, this probability is approximately 0.4010.401 for 10 tests, 0.6420.642 for 20 tests, and 0.9940.994 for 100 tests. The individual tests have not become incorrectly calibrated. Rather, the event “at least one test rejects” is much more likely than the event “this particular test rejects.”

The expected number of false rejections is a different quantity. If VV denotes their number, then, under the same exact marginal rejection probabilities,

E⁡[V]=mα.\operatorname{E}[V]=m\alpha.

This result follows from linearity of expectation and does not require independence. Thus, 100 true null hypotheses tested at level 0.050.05 produce five false rejections on average, although the actual number in any one experiment varies. Dependence changes the probability of obtaining at least one false rejection and the distribution of the total, without changing this expectation when the marginal probabilities remain fixed.

The same phenomenon appears through the smallest observed p-value. For independent continuous tests whose null p-values are uniformly distributed,

Pr⁡ ⁣(min⁡ipi≤t)=1−(1−t)m.\Pr\!\left(\min_i p_i\leq t\right)=1-(1-t)^m.

Consequently, a minimum p-value of 0.010.01 is not especially unusual among 100 independent true-null tests: the probability of obtaining a value at least that small is about 0.6340.634. Treating the selected minimum as though it came from a single prespecified test ignores the selection process.

Multiplicity need not involve a visibly printed table of tests. Searching across outcome definitions, covariate adjustments, subgroup boundaries, transformations, or stopping times can create additional opportunities for favorable results. Conversely, measuring many variables does not automatically invalidate a genuinely prespecified inference about one of them. The relevant issue is how the claim was generated and what repeated-sampling guarantee is attached to it.

Historically, comparisons among treatment means made the issue prominent in analysis of variance. A significant overall test can establish that not all means are equal, but it does not by itself license unrestricted individual comparisons at their ordinary significance levels. Later applications extended the same reasoning to thousands or millions of simultaneous tests.

Families of hypotheses and competing error criteria

A family is the collection of hypotheses treated together for an inferential purpose. Its definition depends on the claims, not simply on whether results appear in one table or come from one dataset. Comparisons supporting interchangeable routes to the same conclusion often belong together. Separate, independently specified scientific questions may instead have separate inferential guarantees. Defining the family only after inspecting the results can conceal the very selection that adjustment is intended to address.

Let m0m_0 be the number of true null hypotheses, RR the total number rejected, and VV the number of true nulls rejected. The family-wise error rate is

FWER⁡=Pr⁡(V≥1).\operatorname{FWER}=\Pr(V\geq 1).

A procedure provides weak control at level α\alpha if this probability is bounded by α\alpha when every null hypothesis is true. It provides strong control if the bound holds under every configuration of true and false null hypotheses. Strong control is important because real applications commonly contain a mixture of genuine and absent effects.

Other criteria answer different questions. The per-comparison error rate is E⁡[V]/m\operatorname{E}[V]/m, whereas the per-family expected error count is E⁡[V]\operatorname{E}[V]. Neither directly bounds the probability of making any false discovery. The realized false discovery proportion is

FDP⁡=Vmax⁡(R,1),\operatorname{FDP}=\frac{V}{\max(R,1)},

and the false discovery rate is its expectation,

FDR⁡=E⁡[FDP⁡].\operatorname{FDR}=\operatorname{E}[\operatorname{FDP}].

The convention makes the proportion zero when no hypotheses are rejected. Under the complete null, every rejection is false, so FDR and FWER coincide. When many alternatives are true, FDR control can permit substantially more rejections.

These distinctions prevent several common misinterpretations. FDR control at 0.050.05 does not mean that each reported finding has a 5 percent probability of being false. Nor does it guarantee that no more than 5 percent of the discoveries in a particular experiment are false. It bounds an average over repetitions of the entire procedure. Criteria such as Pr⁡(FDP⁡>γ)\Pr(\operatorname{FDP}>\gamma) instead address the probability that the realized false-discovery proportion exceeds a chosen tolerance.

Error control also has a cost in statistical power, the probability of detecting a genuine effect. A procedure protecting against even one false rejection generally demands stronger evidence than one controlling an average proportion. The choice is therefore partly scientific: a small set of confirmatory claims may call for different protection from a large exploratory screening list. Mathematical guarantees clarify this trade-off but do not determine which errors matter most.

Single-step corrections and simultaneous confidence intervals

The Bonferroni correction tests each of mm hypotheses at level α/m\alpha/m. If I0I_0 indexes the true null hypotheses and each null p-value is valid, meaning Pr⁡(pi≤t)≤t\Pr(p_i\leq t)\leq t, the union bound gives

Pr⁡ ⁣(⋃i∈I0{pi≤α/m})≤∑i∈I0Pr⁡(pi≤α/m)≤m0mα≤α.\Pr\!\left(\bigcup_{i\in I_0}\{p_i\leq \alpha/m\}\right) \leq \sum_{i\in I_0}\Pr(p_i\leq \alpha/m) \leq \frac{m_0}{m}\alpha \leq \alpha.

This establishes strong FWER control without assumptions about dependence among tests. Equivalently, a Bonferroni-adjusted p-value is min⁡(mpi,1)\min(mp_i,1). For 20 comparisons and a family-wise level of 0.050.05, each comparison must have an unadjusted p-value no larger than 0.00250.0025.

The Šidák correction uses the individual threshold

αindividual=1−(1−α)1/m.\alpha_{\mathrm{individual}}=1-(1-\alpha)^{1/m}.

For independent exact tests, this makes the complete-null FWER exactly α\alpha; with independent valid tests, it also provides strong control. Its threshold is slightly larger than Bonferroni’s. Independence is sufficient, and some dependent settings also permit the correction, but it is not valid under arbitrary dependence.

Weighted versions allocate different portions of the error budget to different hypotheses. With fixed nonnegative weights wiw_i satisfying ∑iwi=1\sum_i w_i=1, rejecting when pi≤αwip_i\leq \alpha w_i retains Bonferroni’s guarantee. Larger weights provide less stringent thresholds for hypotheses assigned greater priority. Weights chosen using the same outcome evidence being tested require additional justification; the fixed-weight argument does not automatically cover that adaptation.

Multiplicity corrections also apply to confidence intervals. Constructing mm intervals, each with coverage at least 1−α/m1-\alpha/m, yields simultaneous coverage of at least 1−α1-\alpha. For estimates with an appropriate normal approximation, Bonferroni intervals take the form

θ^i±z1−α/(2m)SE⁡(θ^i),\widehat{\theta}_i \pm z_{1-\alpha/(2m)} \operatorname{SE}(\widehat{\theta}_i),

where zuz_u is the uu-quantile of the standard normal distribution and SE⁡\operatorname{SE} is the standard error. Finite-sample procedures may instead require Student’s t quantiles or another model-specific distribution.

A simultaneous statement concerns the whole collection: with the stated coverage probability, every interval covers its corresponding fixed parameter. It does not assign a probability to a parameter lying inside a particular observed interval. Simultaneous intervals are also typically wider than ordinary marginal intervals because they protect against inspecting many estimates and emphasizing whichever appears most extreme. Bonferroni’s generality makes it useful, but procedures that exploit known joint structure can often achieve narrower intervals.

Stepwise procedures, closed testing, and comparisons of means

Single-step procedures use predetermined thresholds for all hypotheses. Stepwise procedures use the ordered evidence to obtain greater power while retaining a specified error guarantee. Write the ordered p-values as p(1)≤⋯≤p(m)p_{(1)}\leq\cdots\leq p_{(m)}. The Holm method, published by Sture Holm in 1979, begins with the smallest and compares successive values with

αm,αm−1,…,α.\frac{\alpha}{m},\quad \frac{\alpha}{m-1},\quad \ldots,\quad \alpha.

Testing stops at the first failure; that hypothesis and all subsequent hypotheses remain unrejected. Holm’s procedure strongly controls FWER under arbitrary dependence and rejects every hypothesis that ordinary Bonferroni rejects, sometimes rejecting additional ones.

The Hochberg procedure, introduced in 1988, works in the opposite direction. It finds the largest index kk for which

p(k)≤αm−k+1p_{(k)}\leq \frac{\alpha}{m-k+1}

and rejects hypotheses 1,…,k1,\ldots,k. Unlike Holm’s method, its general guarantee requires independence or appropriate positive-dependence conditions. Although the two procedures use the same sequence of numerical thresholds, their logic differs: Holm requires all preceding tests to pass, whereas Hochberg uses a passing ordered p-value to justify rejection of smaller ones.

A general organizing principle is the closed testing procedure. For every nonempty subset of elementary hypotheses, form the intersection hypothesis that all hypotheses in that subset are true. An elementary hypothesis is rejected only if every intersection containing it is rejected by a valid level-α\alpha local test. If an elementary true null is rejected, the intersection of all true nulls must also have been rejected. The level bound for that intersection establishes strong FWER control. Holm’s method can be derived as an efficient shortcut to closed testing with Bonferroni local tests.

Specialized procedures can be more efficient when the comparisons have known structure. Tukey’s procedure, originating in John Tukey’s work in 1949, uses the studentized range distribution for all pairwise comparisons among normal population means with a common variance. The Tukey–Kramer extension accommodates unequal group sizes. Dunnett’s test, introduced in 1955, targets comparisons of several treatments with one shared control, exploiting their correlation rather than protecting every treatment-versus-treatment comparison.

Scheffé’s method, introduced in 1953, protects all contrasts among group means, not merely a predetermined finite list of pairwise differences. This broader protection can make it more conservative for a narrow comparison set. Likewise, using an overall analysis-of-variance test as a preliminary gate does not automatically produce strong control for arbitrary follow-up testing. The entire rejection rule, including which comparisons become available after the gate opens, determines the guarantee.

False discovery rate procedures for large-scale screening

The Benjamini–Hochberg procedure, introduced by Yoav Benjamini and Yosef Hochberg in 1995, changed the standard approach to large collections of hypotheses. Instead of primarily protecting against any false rejection, it controls the expected fraction of false rejections among all discoveries. This can provide much greater power when many genuine signals are present.

For a target FDR qq, order the p-values and find

k=max⁡{i:p(i)≤imq}.k=\max\left\{i:p_{(i)}\leq \frac{i}{m}q\right\}.

Reject the hypotheses corresponding to p(1),…,p(k)p_{(1)},\ldots,p_{(k)}; reject none if the set is empty. Under independence and valid null p-values, the procedure controls FDR at no more than (m0/m)q(m_0/m)q, and hence at no more than qq. Control also holds under a particular positive-dependence condition known as positive regression dependence on the subset of true nulls. Arbitrary correlation is not covered by that condition.

The step-up construction is important. With ten hypotheses and q=0.05q=0.05, the first three ordered thresholds are 0.0050.005, 0.0100.010, and 0.0150.015. If the first three p-values are 0.0040.004, 0.0090.009, and 0.0140.014, and no later p-value passes its threshold, all three hypotheses are rejected. A fixed Bonferroni threshold of 0.0050.005 would reject only the first. Moreover, a passing later p-value can justify rejection of earlier values even if an earlier value does not pass its own ranked threshold.

Benjamini and Daniel Yekutieli established an arbitrary-dependence version in 2001. It replaces qq by q/Hmq/H_m, where

Hm=∑j=1m1j.H_m=\sum_{j=1}^{m}\frac{1}{j}.

The resulting thresholds are iq/(mHm)iq/(mH_m). Since HmH_m grows approximately as log⁡m\log m, the additional dependence protection can be substantial. The procedure offers a broad guarantee, but it may be unnecessarily conservative when more specific information about the joint distribution is justified.

FDR-adjusted p-values for the ordinary Benjamini–Hochberg procedure can be expressed, in ranked order, as

p~(i)=min⁡{1, min⁡j≥imjp(j)}.\widetilde p_{(i)} = \min\left\{ 1,\, \min_{j\geq i}\frac{m}{j}p_{(j)} \right\}.

They identify the smallest target FDR level at which each hypothesis would be rejected by that procedure. Related methods estimate the proportion of true null hypotheses to improve power. Such adaptive procedures require their own assumptions and safeguards; simply replacing mm with an informal estimate of the number of “relevant” tests does not inherit the original theorem.

Dependence, resampling, and structured hypotheses

Tests often share observations, outcomes, controls, or model parameters. Their dependence is not merely a technical nuisance: it determines how much distinct opportunity there is for an extreme result. Perfectly identical tests offer only one such opportunity, whereas independent tests offer many. A covariance matrix can describe aspects of this relationship, but pairwise correlations alone do not generally determine the joint tail probabilities needed for simultaneous inference.

One direct strategy calibrates the largest test statistic. If TiT_i measures evidence against hypothesis ii, a common critical value cαc_\alpha may be chosen to satisfy

Pr⁡ ⁣(max⁡1≤i≤m∣Ti∣>cα)≤α\Pr\!\left(\max_{1\leq i\leq m}|T_i|>c_\alpha\right)\leq\alpha

under a relevant null model. This max-statistic approach uses the joint sampling distribution rather than treating every comparison as unrelated. Under strong positive dependence, the resulting threshold may be appreciably less stringent than a Bonferroni threshold.

Permutation procedures estimate such distributions by rearranging labels in ways permitted by the design or null hypothesis. For example, treatment assignments can be reassigned according to the randomization scheme, with the maximum statistic recomputed for every reassignment. Observations from clustered or repeated-measures data cannot generally be shuffled as though they were independent. The valid rearrangements must preserve the required exchangeability or follow the actual assignment mechanism.

Resampling under the complete null does not automatically yield strong FWER control when some effects are present. A condition often used for resampling-based multiple testing is subset pivotality: the joint distribution of statistics for any subset of true nulls does not depend on which other hypotheses are false. Westfall–Young methods, developed prominently in their 1993 treatment of resampling-based multiple testing, use joint resampling distributions and step-down constructions. Without the needed assumptions, a complete-null calibration may provide only weak control. Bootstrap methods offer other approximations, with validity depending on the model, statistic, and resampling scheme.

Hypotheses may also have logical or scientific structure. A hierarchy might first test whether a biological pathway contains any signal and then examine individual genes. In spatial imaging, a procedure may target voxels, clusters, or regions; these are distinct inferential objects. A cluster-level discovery does not establish that every location within the cluster has an effect.

There is therefore no universal “effective number of independent tests” that solves dependence in every setting. Such approximations can be useful when derived and validated for a specified model, but joint-distribution methods and structurally tailored procedures make their assumptions more explicit.

Selection, model search, and post-selection inference

Multiplicity becomes particularly subtle when the set of reported questions is chosen after seeing the data. A researcher might inspect many subgroup effects, choose a favorable regression model, or report the outcome showing the largest treatment difference. The nominal test for the selected result then omits a selection event that helped make the result look unusual. The relevant reference distribution is not necessarily the distribution for a hypothesis specified before examining the observations.

Selection also distorts estimation. When many noisy estimates are ranked, the largest observed estimates tend to combine genuine effects with unusually favorable noise. This “winner’s curse” is related to selection bias and regression to the mean. A selected effect can be genuinely nonzero and still be substantially exaggerated. Adjusting p-values does not, by itself, remove this estimation bias or restore ordinary confidence-interval coverage.

One response is simultaneous inference over a prespecified candidate set. Suppose intervals CiC_i cover all fixed parameters θi\theta_i simultaneously with probability at least 1−α1-\alpha. Then any data-dependent choice i^\widehat i from that set satisfies

Pr⁡ ⁣(θi^∈Ci^)≥1−α.\Pr\!\left(\theta_{\widehat i}\in C_{\widehat i}\right)\geq 1-\alpha.

Whenever all intervals cover, the selected interval covers too. This reasoning protects selection among the specified targets, but it does not automatically cover a newly constructed model-dependent parameter absent from the original family.

Another response is selective inference, which explicitly accounts for the selection rule. In certain Gaussian linear-model problems, selection events generated by procedures such as lasso regression can be represented by linear inequalities in the observed data. Conditional on the selection event and suitable additional quantities, a selected linear contrast follows a truncated normal distribution. Tests and intervals can then be constructed for the stated conditional target. Their interpretation depends on exactly what was selected and conditioned upon.

Sample splitting separates the data used to discover a question from the data used to test it. A hypothesis selected on one part can be tested on an independent part without charging that test for every search performed in the first part, provided the second part remains genuinely untouched and the test is valid conditional on the first-stage choices. The cost is reduced information for both selection and confirmation. Repeatedly consulting the holdout set undermines the separation.

Cross-validation primarily estimates predictive performance; it does not automatically validate coefficient p-values after extensive tuning. Choosing the best-performing model among many can also bias the reported performance estimate. Nested evaluation and independent replication address aspects of this problem, but their guarantees depend on the full search-and-evaluation procedure.

Experimental planning, repeated looks, and application-specific claims

Multiplicity is often easier to characterize when the claims and decision rules are specified before outcomes are examined. A preregistered analysis plan can identify primary outcomes, comparison families, directions of testing, subgroup analyses, and adjustment procedures. This does not make the statistical model correct, but it distinguishes planned inference from questions discovered through exploration and makes the relevant search space more visible.

In a randomized controlled trial, the logical form of a success criterion matters. If success can be declared when any one of several endpoints is significant, there are several routes to a false success claim. If success requires all of several co-primary endpoints to be significant, the logic is different. Under an intersection-union test, whenever the overall claim is false, at least one required component null is true; the chance of rejecting all components is bounded by the level of that true component’s valid test. This can protect the conjunction without a Bonferroni division, although separate claims about individual endpoints may require additional treatment.

Repeated looks at accumulating data create multiplicity across time. Applying an ordinary fixed-sample 0.050.05 test after every new batch does not generally preserve a 5 percent false-positive probability. Group-sequential methods instead allocate crossing probabilities across planned analyses, often through an alpha-spending function. Their calibration uses the joint distribution of interim statistics, which are strongly correlated because later analyses include earlier observations.

In high-dimensional biology, the comparison family can include genes, genetic variants, tissues, phenotypes, or combinations of them. Tests of gene expression commonly use FDR control because the immediate objective is a list for further investigation. Genome-wide association studies often use a conventional threshold near 5×10−85\times10^{-8} for common-variant analyses, reflecting the very large search over correlated variants. That threshold is a domain convention grounded in particular study structures, not a universal solution for every genomic analysis or ancestry composition.

A further distinction arises when hypotheses arrive indefinitely rather than in one fixed batch. Online multiple testing assigns a testing level to each new hypothesis using information available before that hypothesis is tested. Alpha-investing and related procedures allow the testing budget to evolve, sometimes rewarding discoveries with additional future testing capacity. Different procedures control different quantities, and guarantees may rely on independence or conditional validity.

Across these settings, a valid multiplicity procedure cannot repair invalid component tests. Unmodeled clustering, inappropriate stopping rules, measurement artifacts, or confounding can invalidate the input p-values or the scientific interpretation. Error control describes the specified statistical procedure under its assumptions, not the entire reliability of the research process.

Bayesian approaches, estimation, and limits of interpretation

Bayesian inference approaches multiplicity through joint modeling, prior probabilities, and posterior decision rules rather than through a mandatory universal correction to p-values. A useful model for many hypotheses places probability π0\pi_0 on a null state and 1−π01-\pi_0 on an alternative. If an observation xix_i has respective densities f0f_0 and f1f_1, its posterior null probability is

Pr⁡(Hi null∣xi)=π0f0(xi)π0f0(xi)+(1−π0)f1(xi).\Pr(H_i\text{ null}\mid x_i) = \frac{\pi_0 f_0(x_i)} {\pi_0 f_0(x_i)+(1-\pi_0)f_1(x_i)}.

Evidence is interpreted together with the prevalence and distribution of effects, not solely through a tail probability under the null.

Hierarchical models let related effects share information. For example, effect estimates may be modeled as noisy observations of parameters drawn from a common distribution. Estimated population-scale variation then influences how strongly individual estimates shrink toward a shared center. Empirical Bayes methods estimate aspects of this distribution from the collection of observations. Such borrowing can reduce exaggeration among extreme estimates, although performance depends on the adequacy of the model and the treatment of hyperparameter uncertainty.

For a selected set SS of size r>0r>0, suppose ℓi\ell_i is the posterior probability that hypothesis ii is null. Conditional on the observed data, the posterior expected false-discovery proportion is

E⁡ ⁣[#{null hypotheses in S}r | data]=1r∑i∈Sℓi.\operatorname{E}\!\left[ \frac{\#\{\text{null hypotheses in }S\}}{r} \,\middle|\,\text{data} \right] = \frac{1}{r}\sum_{i\in S}\ell_i.

This equality follows from linearity of conditional expectation and does not require posterior independence. It supports decision rules based on posterior expected error. It is nevertheless distinct from a frequentist FDR theorem, which averages over repeated datasets under specified fixed configurations of effects.

Bayesian modeling does not automatically neutralize unreported searches or incomplete observation mechanisms. If a model covers only the selected result while ignoring how that result became available, the interpretation may omit relevant selection information. Whether conditioning on selection changes a posterior analysis depends on the sampling model, the observed information, and the inferential target. Likewise, assigning independent diffuse priors to thousands of effects is not equivalent to fitting a sparsity-aware hierarchical model.

A separate limitation concerns the meaning of discovery. Neither a multiplicity-adjusted p-value nor a posterior null probability measures effect size, practical importance, or causation. Large samples can detect negligible departures from a null hypothesis; stringent correction can miss effects that matter substantively. Estimates and appropriate uncertainty statements remain necessary to characterize what was found.

Selective publication adds another layer: even well-adjusted analyses within individual studies can enter a literature shaped by publication bias. The statistical family used inside one paper need not represent all searches across laboratories, datasets, and unpublished analyses. Multiplicity guarantees therefore require an explicit scope: the hypotheses considered, the selection rule, the error criterion, and the assumptions under which that criterion is controlled.