False Discovery Rate · aiwiki.page

The false discovery rate (FDR) is an error criterion in statistics used when conducting multiple hypothesis tests. It measures the expected proportion of rejected hypotheses whose null hypotheses are actually true. Unlike an error criterion that limits the probability of making even one false rejection, FDR control permits some false discoveries while limiting their average proportion within the discovery set. Yoav Benjamini and Yosef Hochberg introduced the criterion and a widely used procedure for controlling it in 1995. (academic.oup.com)

Definition

Consider a family of $m$ hypotheses. Let:

$m_0$ be the number of true null hypotheses;
$R$ be the total number of rejected hypotheses;
$V$ be the number of rejected true null hypotheses.

A false rejection is a Type I error. The number $R-V$ counts correct rejections of false null hypotheses. Because rejection decisions depend on sampled data, $R$ and $V$ are random variables. The false discovery proportion (FDP) is

\operatorname{FDP}=\frac{V}{\max(R,1)}.

Thus, FDP equals $V/R$ when discoveries occur and is defined as zero when $R=0$ . The FDR is its expected value:

\operatorname{FDR} =\mathbb{E}\!\left[\frac{V}{\max(R,1)}\right].

A procedure controls FDR at a target level $q$ if $\operatorname{FDR}\leq q$ for every data-generating distribution covered by its assumptions. FDP describes the errors in one realized discovery set; FDR describes the procedure’s average behavior over repeated sampling. (arxiv.org)

In particular,

\mathbb{E}\!\left[\frac{V}{\max(R,1)}\right] \neq \frac{\mathbb{E}[V]}{\mathbb{E}[R]}

in general. The expectation of a random ratio cannot ordinarily be replaced by a ratio of expectations. (genomics.princeton.edu)

Comparison with other error criteria

The family-wise error rate (FWER) is

\operatorname{FWER}=\Pr(V\geq 1),

the probability of at least one false rejection. Because the false discovery proportion is no greater than the indicator that a false rejection occurs,

\operatorname{FDR}\leq\operatorname{FWER}.

Consequently, controlling FWER also controls FDR at the same level, but the converse does not generally hold. Under the global null, where all $m$ null hypotheses are true, every discovery is false: FDP is one whenever $R>0$ . In that special case, FDR and FWER coincide. (academic.oup.com)

The Bonferroni correction controls FWER by testing each hypothesis at a significance level of $q/m$ . FDR procedures use a less stringent error criterion and can consequently have greater statistical power, especially when many hypotheses are false. This is a difference in the error being controlled, rather than a guarantee that every FDR procedure outperforms every FWER procedure. (stat.ethz.ch)

FDR also differs from the false-positive rate of an individual test. An individual false-positive rate concerns rejection conditional on a null hypothesis being true; FDR concerns the proportion of true nulls among the hypotheses selected for rejection. The denominators and interpretations are different. (genomics.princeton.edu)

The Benjamini–Hochberg procedure

The Benjamini–Hochberg procedure (BH) is a step-up procedure based on p-values. For a target FDR level $q$ , order the $m$ p-values:

p_{(1)}\leq p_{(2)}\leq\cdots\leq p_{(m)}.

Find

k=\max\left\{i:p_{(i)}\leq\frac{i}{m}q\right\}.

If the set is empty, reject no hypotheses. Otherwise, reject the hypotheses corresponding to $p_{(1)},\ldots,p_{(k)}$ . The procedure selects the largest qualifying rank; it does not stop at the first rank that fails its comparison. (stat.ethz.ch)

Numerical example

Suppose five ordered p-values are

0.001,\quad 0.009,\quad 0.020,\quad 0.041,\quad 0.200.

At $q=0.05$ , the BH comparison thresholds are

0.010,\quad 0.020,\quad 0.030,\quad 0.040,\quad 0.050.

The largest qualifying rank is three, so the first three hypotheses are rejected. Applying Bonferroni at the same numerical level would reject only the first two. Neither calculation identifies which particular rejected hypotheses, if any, are false discoveries.

Assumptions and guarantee

For a true null hypothesis, a valid p-value satisfies

\Pr(P_i\leq t)\leq t,\qquad 0\leq t\leq1.

Such p-values are called super-uniform: small values occur no more frequently than they would under a uniform distribution. Under suitable independence assumptions, BH satisfies

\operatorname{FDR}\leq q\frac{m_0}{m}\leq q.

When null p-values are independent, exactly uniform, and independent of the non-null p-values, the first bound is an equality. (arxiv.org)

Benjamini and Daniel Yekutieli established in 2001 that BH also controls FDR under positive regression dependence on a subset (PRDS), a technical condition involving the true nulls. PRDS is not synonymous with positive pairwise correlation; a general statement that tests are positively correlated is not, by itself, sufficient to establish the guarantee. (math.tau.ac.il)

Dependence and the Benjamini–Yekutieli procedure

The Benjamini–Yekutieli procedure (BY) modifies BH to accommodate arbitrary dependence among valid p-values. Define the harmonic sum

H_m=\sum_{j=1}^{m}\frac{1}{j}.

BY uses the thresholds

p_{(i)}\leq\frac{i q}{mH_m}

and otherwise follows the same step-up rule. Equivalently, it applies BH with target level $q/H_m$ . This provides FDR control without an independence or PRDS requirement, but its smaller thresholds can substantially reduce the number of discoveries. (math.tau.ac.il)

Dependence is therefore part of the mathematical specification of an FDR procedure, not merely an implementation detail. Valid individual p-values alone do not establish the standard BH guarantee for every possible joint distribution. (math.tau.ac.il)

Adjusted p-values and q-values

BH decisions can be expressed through adjusted p-values. In sorted order, these are

\widetilde p_{(i)} = \min\left\{ 1,\, \min_{j\geq i}\frac{m p_{(j)}}{j} \right\}.

Rejecting hypotheses with $\widetilde p_i\leq q$ reproduces the BH discovery set. The cumulative minimum is important: multiplying each p-value by $m/i$ without this additional step does not generally produce the correct adjusted values. (stat.ethz.ch)

A related quantity is the q-value, developed in John Storey’s work on false discovery rates. In its original formulation, it is the smallest positive false discovery rate associated with a rejection region containing the observed statistic. It describes the error rate of a selected set, not the probability that the individual hypothesis is true. Although BH-adjusted p-values are sometimes called q-values, the BH adjustment and Storey’s estimation-based q-values are not identical constructions. (rss.onlinelibrary.wiley.com)

Adaptive methods seek additional power by estimating the proportion of true nulls,

\pi_0=\frac{m_0}{m}.

One estimator uses a tuning parameter $\lambda$ :

\widehat{\pi}_0(\lambda) = \frac{\#\{i:p_i>\lambda\}}{m(1-\lambda)}.

Its motivation is that uniformly distributed null p-values exceed $\lambda$ with probability $1-\lambda$ . Estimating $\pi_0$ , choosing $\lambda$ , and selecting a rejection threshold introduce additional statistical issues; simply substituting an arbitrary estimate into BH does not automatically preserve finite-sample FDR control. (genomics.princeton.edu)

Positive and local false discovery rates

The positive false discovery rate is

\operatorname{pFDR} = \mathbb{E}\!\left[\frac{V}{R}\,\middle|\,R>0\right],

provided $\Pr(R>0)>0$ . It excludes outcomes with no discoveries. Under the zero-discovery convention,

\operatorname{FDR} = \operatorname{pFDR}\Pr(R>0).

Thus, FDR and pFDR can differ substantially when a procedure rarely rejects anything. (genomics.princeton.edu)

The local false discovery rate is a different, model-based quantity. In a two-component mixture for a test statistic $Z$ ,

f(z)=\pi_0 f_0(z)+(1-\pi_0)f_1(z),

where $f_0$ and $f_1$ are the null and alternative probability densities, Bayes’ theorem gives

\operatorname{lfdr}(z)=\frac{\pi_0 f_0(z)}{f(z)}.

This is the model’s posterior probability of null membership given the statistic $z$ . Unlike the set-level frequentist FDR, it concerns an individual observation and depends on the mixture model and its estimation. (stat.cmu.edu)

Applications and interpretive limitations

FDR methods are particularly useful in large-scale discovery studies. For example, a gene-expression experiment may test thousands of genes for differences between conditions. FDR-based analysis characterizes the expected false-discovery proportion within the resulting candidate set while retaining more discoveries than a stricter family-wise criterion may allow. (pmc.ncbi.nlm.nih.gov)

The meaning of “discovery” remains tied to the tested hypothesis. Rejecting a null hypothesis does not establish a large effect size, practical importance, or causation. In large-scale testing, even the distinction between an exact null and a scientifically uninteresting effect can affect how discoveries are interpreted. (stat.cmu.edu)

Finally, FDR control at $q=0.05$ does not guarantee that at most 5% of discoveries in a particular study are false. It controls the expectation of the proportion, and realized FDP can vary between studies. Nor does it assign each discovery a 5% probability of being wrong. The guarantee applies to a specified procedure and its assumptions; an observed discovery set is not itself a direct measurement of its unknown error proportion. (genomics.princeton.edu)

References

Elementary proofs of several results on false discovery ratearxiv.org
The Control of the False Discovery Rate in Multiple Testing under Dependencymath.tau.ac.il
R: Adjust P-values for Multiple Comparisonsstat.ethz.ch
Benjamini-Hochberg Procedure with the Simulatorstat.ethz.ch
A direct approach to false discovery ratesgenomics.princeton.edu
Statistical significance for genomewide studiespmc.ncbi.nlm.nih.gov
Statistical significance for genomewide studiesgenomics.princeton.edu
Large-Scale Simultaneous Hypothesis Testing: The Choice of a Null Hypothesisstat.cmu.edu