Tukey’s range test is a statistical hypothesis test for comparing every pair of means in a collection of groups while controlling the probability of making one or more false-positive conclusions across the entire collection. Developed by John W. Tukey, it is commonly used alongside analysis of variance (ANOVA). The classical procedure, also called the honestly significant difference (HSD) test, assumes independent, normally distributed errors with a common variance. It is exact for equal group sizes; the Tukey–Kramer procedure extends the comparison rule to unequal group sizes.
Purpose and scope
An omnibus one-way ANOVA tests whether all group population means are equal:
Rejecting this hypothesis establishes evidence that not all means are equal, but does not identify the pairs that differ. Tukey’s test addresses the individual pairwise hypotheses
There are such comparisons. Conducting each comparison as an ordinary, unadjusted test at the same significance level generally increases the probability of at least one false rejection. This is a multiple-testing problem.
Tukey’s procedure controls the family-wise error rate: the probability of rejecting at least one true pairwise null hypothesis. This is a family-level form of protection against Type I errors, rather than a separate error guarantee for each test considered in isolation.
Although often described as a post hoc test following a significant ANOVA, its pairwise error control does not require a preliminary significant omnibus test. Requiring both tests to be significant creates an additional decision rule, which can reduce statistical power.
Statistical model and assumptions
For a one-way layout, the standard model is
where is the number of observations in group . The errors are assumed to have:
- Independence: errors are mutually independent within and between groups.
- Normality: each error follows a normal distribution with mean zero.
- Equal variances: every group has the same error variance, .
The model treats group membership as fixed for the comparisons under consideration. Randomization in an experiment can help justify the model, but cannot substitute for accounting for clustering, repeated measurements, or other dependence.
Let denote the sample mean of group , and let . The pooled within-group mean square is
It estimates the common error variance and has
residual degrees of freedom. Here is the ANOVA residual mean square, not a prediction-error measure from model validation.
Studentized range distribution
The critical values come from the studentized range distribution. Suppose are independent standard normal variables, and is independent of them with
where denotes the chi-squared distribution with degrees of freedom. Then
has a studentized range distribution with parameters and .
Write for its quantile. This critical value reflects the most extreme difference among all means, rather than a single prespecified pair.
For equally sized groups, the standardized group means have precisely the structure needed for this range distribution. The largest pairwise difference is the difference between the largest and smallest sample means, explaining the name range test.
Equal group sizes: Tukey’s HSD
When each group contains observations, the honestly significant difference is
The pairwise null hypothesis for groups and is rejected when
Equivalently, the pairwise test statistic is
and rejection occurs when .
The denominator is not the standard error of the difference between two independent group means. That standard error is . Tukey’s statistic uses the standard error of one group mean because the studentized range distribution is defined on that scale. Confusing these denominators introduces a factor-of- error.
Simultaneous confidence intervals
The same procedure produces simultaneous confidence intervals for all pairwise population-mean differences:
Under the classical equal-size normal model, the probability that all these intervals cover their respective true differences is . This is a joint coverage statement, not merely a statement about each interval separately.
A pair is statistically significant at the family-wise level exactly when its corresponding interval excludes zero. The procedure has strong family-wise error control: protection applies whether all, some, or none of the pairwise null hypotheses are true.
Unequal group sizes: the Tukey–Kramer procedure
For unequal group sizes, the Tukey–Kramer comparison uses
The simultaneous intervals are
When , these formulas reduce to Tukey’s original equal-size formulas.
Under independent normal errors with a common variance, Tukey–Kramer provides conservative family-wise error control: its error probability is at most , rather than generally being exactly . Unequal sample sizes do not remove the equal-variance assumption.
A Tukey-adjusted p-value for a pair is obtained from the upper tail of the studentized range distribution:
where is that distribution’s cumulative distribution function. For unequal sizes, this calculation implements the Tukey–Kramer adjustment; the reference distribution is not the exact sampling distribution of every individual pairwise statistic.
Numerical example
Consider three independent groups with ten observations each:
Suppose the pooled residual mean square is . Then . At ,
so
The comparisons are therefore:
| Pair | Absolute mean difference | Exceeds HSD? |
|---|---|---|
| Groups 1 and 2 | No | |
| Groups 1 and 3 | Yes | |
| Groups 2 and 3 | No |
Only groups 1 and 3 are declared different at the 5% family-wise level. For example, the interval for is approximately
Failure to reject the other two comparisons does not establish that their population means are equal.
Interpretation and reporting
A useful report identifies the groups compared, sample sizes, estimated mean differences, simultaneous confidence intervals, adjusted p-values, family-wise significance level, and whether the equal-size or Tukey–Kramer formulation was used.
The mean differences and intervals convey magnitude and precision. Statistical significance alone does not establish practical importance; this also requires consideration of effect size and the measurement context.
Results are sometimes summarized using letters: groups sharing a letter have not been declared significantly different. Such displays do not demonstrate equivalence, and their groupings need not be transitive. For example, one group can be nonsignificantly different from two others even when those two differ significantly from each other.
Limitations and related methods
Unequal variances and dependent data
The pooled variance estimate is central to the method. Substantial variance differences, especially combined with unequal sample sizes, can invalidate its nominal error control. The Games–Howell test is a related all-pairs procedure designed for unequal variances and potentially unequal group sizes, using pair-specific variance estimates and approximate degrees of freedom.
Repeated measurements, clustered observations, and correlated estimated means require methods that account for their dependence. Applying the simple one-way Tukey formula to such data is not justified merely because software labels an adjustment “Tukey.”
Choice of comparison family
Tukey’s procedure is tailored to all pairwise differences. It does not automatically protect arbitrary contrasts, multiple outcomes, or additional families of tests. Applying it separately to several outcomes gives separate family-wise guarantees, not necessarily a single guarantee across the entire analysis.
The Bonferroni correction can accommodate a broader prespecified collection of hypotheses, though it may be more conservative for all-pairs comparisons. When only comparisons against one control are relevant, a procedure designed for that narrower family can provide greater power. Tukey’s control of family-wise error also differs from control of the false discovery rate, which concerns the expected proportion of false rejections among rejected hypotheses.
More complex models
Pairwise comparisons of adjusted means also arise in linear regression, factorial ANOVA, and models containing interactions. In these settings, the covariance structure of the estimated means matters. The exact classical studentized-range justification does not automatically extend to every set of model-adjusted means.
When interactions are present, comparisons averaged over another factor can answer a different question from comparisons within its individual levels. The interpretation of a Tukey-adjusted comparison therefore depends both on the error adjustment and on which model means are being compared.