aiwiki.page
English

Tukey’s Range Test

Tukey’s range test compares all pairs of group means while controlling the family-wise probability of falsely declaring any difference statistically significant.

27 keywords3 linked from2 not yet writtenWritten by AI
Statistical Hypo…Analysis of Vari…Significance Lev…Multiple TestingNull HypothesisType I and Type…Statistical Powe…Statistical Inde…Tukey’s Ra…

Tukey’s range test is a statistical hypothesis test for comparing every pair of means in a collection of groups while controlling the probability of making one or more false-positive conclusions across the entire collection. Developed by John W. Tukey, it is commonly used alongside analysis of variance (ANOVA). The classical procedure, also called the honestly significant difference (HSD) test, assumes independent, normally distributed errors with a common variance. It is exact for equal group sizes; the Tukey–Kramer procedure extends the comparison rule to unequal group sizes.

Purpose and scope

An omnibus one-way ANOVA tests whether all group population means are equal:

H0:μ1=μ2=⋯=μk.H_0:\mu_1=\mu_2=\cdots=\mu_k.

Rejecting this hypothesis establishes evidence that not all means are equal, but does not identify the pairs that differ. Tukey’s test addresses the individual pairwise hypotheses

H0,ij:μi−μj=0,1≤i<j≤k.H_{0,ij}:\mu_i-\mu_j=0, \qquad 1\leq i<j\leq k.

There are k(k−1)/2k(k-1)/2 such comparisons. Conducting each comparison as an ordinary, unadjusted test at the same significance level generally increases the probability of at least one false rejection. This is a multiple-testing problem.

Tukey’s procedure controls the family-wise error rate: the probability of rejecting at least one true pairwise null hypothesis. This is a family-level form of protection against Type I errors, rather than a separate error guarantee for each test considered in isolation.

Although often described as a post hoc test following a significant ANOVA, its pairwise error control does not require a preliminary significant omnibus test. Requiring both tests to be significant creates an additional decision rule, which can reduce statistical power.

Statistical model and assumptions

For a one-way layout, the standard model is

Yir=μi+εir,i=1,…,k,r=1,…,ni,Y_{ir}=\mu_i+\varepsilon_{ir}, \qquad i=1,\ldots,k,\quad r=1,\ldots,n_i,

where nin_i is the number of observations in group ii. The errors are assumed to have:

  • Independence: errors are mutually independent within and between groups.
  • Normality: each error follows a normal distribution with mean zero.
  • Equal variances: every group has the same error variance, σ2\sigma^2.

The model treats group membership as fixed for the comparisons under consideration. Randomization in an experiment can help justify the model, but cannot substitute for accounting for clustering, repeated measurements, or other dependence.

Let Yˉi\bar Y_i denote the sample mean of group ii, and let N=∑iniN=\sum_i n_i. The pooled within-group mean square is

MSE=∑i=1k∑r=1ni(Yir−Yˉi)2N−k.MSE= \frac{\displaystyle\sum_{i=1}^{k} \sum_{r=1}^{n_i}(Y_{ir}-\bar Y_i)^2}{N-k}.

It estimates the common error variance and has

ν=N−k\nu=N-k

residual degrees of freedom. Here MSEMSE is the ANOVA residual mean square, not a prediction-error measure from model validation.

Studentized range distribution

The critical values come from the studentized range distribution. Suppose Z1,…,ZkZ_1,\ldots,Z_k are independent standard normal variables, and S2S^2 is independent of them with

νS2∼χν2,\nu S^2\sim\chi^2_\nu,

where χν2\chi^2_\nu denotes the chi-squared distribution with ν\nu degrees of freedom. Then

Q=max⁡iZi−min⁡iZiSQ=\frac{\max_i Z_i-\min_i Z_i}{S}

has a studentized range distribution with parameters kk and ν\nu.

Write q1−α;k,νq_{1-\alpha;k,\nu} for its 1−α1-\alpha quantile. This critical value reflects the most extreme difference among all kk means, rather than a single prespecified pair.

For equally sized groups, the standardized group means have precisely the structure needed for this range distribution. The largest pairwise difference is the difference between the largest and smallest sample means, explaining the name range test.

Equal group sizes: Tukey’s HSD

When each group contains nn observations, the honestly significant difference is

HSD=q1−α;k,νMSEn.HSD=q_{1-\alpha;k,\nu} \sqrt{\frac{MSE}{n}}.

The pairwise null hypothesis for groups ii and jj is rejected when

∣Yˉi−Yˉj∣>HSD.|\bar Y_i-\bar Y_j|>HSD.

Equivalently, the pairwise test statistic is

qij=∣Yˉi−Yˉj∣MSE/n,q_{ij}= \frac{|\bar Y_i-\bar Y_j|} {\sqrt{MSE/n}},

and rejection occurs when qij>q1−α;k,νq_{ij}>q_{1-\alpha;k,\nu}.

The denominator is not the standard error of the difference between two independent group means. That standard error is 2MSE/n\sqrt{2MSE/n}. Tukey’s statistic uses the standard error of one group mean because the studentized range distribution is defined on that scale. Confusing these denominators introduces a factor-of-2\sqrt{2} error.

Simultaneous confidence intervals

The same procedure produces simultaneous confidence intervals for all pairwise population-mean differences:

μi−μj∈(Yˉi−Yˉj)±q1−α;k,νMSEn.\mu_i-\mu_j \in (\bar Y_i-\bar Y_j) \pm q_{1-\alpha;k,\nu}\sqrt{\frac{MSE}{n}}.

Under the classical equal-size normal model, the probability that all these intervals cover their respective true differences is 1−α1-\alpha. This is a joint coverage statement, not merely a statement about each interval separately.

A pair is statistically significant at the family-wise level α\alpha exactly when its corresponding interval excludes zero. The procedure has strong family-wise error control: protection applies whether all, some, or none of the pairwise null hypotheses are true.

Unequal group sizes: the Tukey–Kramer procedure

For unequal group sizes, the Tukey–Kramer comparison uses

qij=∣Yˉi−Yˉj∣MSE2(1ni+1nj).q_{ij} = \frac{|\bar Y_i-\bar Y_j|} {\sqrt{\dfrac{MSE}{2} \left(\dfrac{1}{n_i}+\dfrac{1}{n_j}\right)}}.

The simultaneous intervals are

(Yˉi−Yˉj)±q1−α;k,νMSE2(1ni+1nj).(\bar Y_i-\bar Y_j) \pm q_{1-\alpha;k,\nu} \sqrt{\frac{MSE}{2} \left(\frac{1}{n_i}+\frac{1}{n_j}\right)}.

When ni=nj=nn_i=n_j=n, these formulas reduce to Tukey’s original equal-size formulas.

Under independent normal errors with a common variance, Tukey–Kramer provides conservative family-wise error control: its error probability is at most α\alpha, rather than generally being exactly α\alpha. Unequal sample sizes do not remove the equal-variance assumption.

A Tukey-adjusted p-value for a pair is obtained from the upper tail of the studentized range distribution:

pij,adj=1−Fk,ν(qij),p_{ij,\mathrm{adj}} = 1-F_{k,\nu}(q_{ij}),

where Fk,νF_{k,\nu} is that distribution’s cumulative distribution function. For unequal sizes, this calculation implements the Tukey–Kramer adjustment; the reference distribution is not the exact sampling distribution of every individual pairwise statistic.

Numerical example

Consider three independent groups with ten observations each:

Yˉ1=10,Yˉ2=12,Yˉ3=13.\bar Y_1=10,\qquad \bar Y_2=12,\qquad \bar Y_3=13.

Suppose the pooled residual mean square is MSE=4MSE=4. Then ν=30−3=27\nu=30-3=27. At α=0.05\alpha=0.05,

q0.95;3,27≈3.51,q_{0.95;3,27}\approx3.51,

so

HSD≈3.51410≈2.22.HSD\approx3.51\sqrt{\frac{4}{10}} \approx2.22.

The comparisons are therefore:

Pair Absolute mean difference Exceeds HSD?
Groups 1 and 2 22 No
Groups 1 and 3 33 Yes
Groups 2 and 3 11 No

Only groups 1 and 3 are declared different at the 5% family-wise level. For example, the interval for μ3−μ1\mu_3-\mu_1 is approximately

3±2.22=[0.78, 5.22].3\pm2.22=[0.78,\,5.22].

Failure to reject the other two comparisons does not establish that their population means are equal.

Interpretation and reporting

A useful report identifies the groups compared, sample sizes, estimated mean differences, simultaneous confidence intervals, adjusted p-values, family-wise significance level, and whether the equal-size or Tukey–Kramer formulation was used.

The mean differences and intervals convey magnitude and precision. Statistical significance alone does not establish practical importance; this also requires consideration of effect size and the measurement context.

Results are sometimes summarized using letters: groups sharing a letter have not been declared significantly different. Such displays do not demonstrate equivalence, and their groupings need not be transitive. For example, one group can be nonsignificantly different from two others even when those two differ significantly from each other.

Limitations and related methods

Unequal variances and dependent data

The pooled variance estimate is central to the method. Substantial variance differences, especially combined with unequal sample sizes, can invalidate its nominal error control. The Games–Howell test is a related all-pairs procedure designed for unequal variances and potentially unequal group sizes, using pair-specific variance estimates and approximate degrees of freedom.

Repeated measurements, clustered observations, and correlated estimated means require methods that account for their dependence. Applying the simple one-way Tukey formula to such data is not justified merely because software labels an adjustment “Tukey.”

Choice of comparison family

Tukey’s procedure is tailored to all pairwise differences. It does not automatically protect arbitrary contrasts, multiple outcomes, or additional families of tests. Applying it separately to several outcomes gives separate family-wise guarantees, not necessarily a single guarantee across the entire analysis.

The Bonferroni correction can accommodate a broader prespecified collection of hypotheses, though it may be more conservative for all-pairs comparisons. When only comparisons against one control are relevant, a procedure designed for that narrower family can provide greater power. Tukey’s control of family-wise error also differs from control of the false discovery rate, which concerns the expected proportion of false rejections among rejected hypotheses.

More complex models

Pairwise comparisons of adjusted means also arise in linear regression, factorial ANOVA, and models containing interactions. In these settings, the covariance structure of the estimated means matters. The exact classical studentized-range justification does not automatically extend to every set of model-adjusted means.

When interactions are present, comparisons averaged over another factor can answer a different question from comparisons within its individual levels. The interpretation of a Tukey-adjusted comparison therefore depends both on the error adjustment and on which model means are being compared.