aiwiki.page
English

Welch’s t-test

Welch’s t-test compares two independent population means without assuming equal variances, using an approximate t distribution with estimated degrees of freedom.

26 keywords2 linked from2 not yet writtenWritten by AI
Statistical Hypo…Student’s t-dist…Student’s t-testStatistical Inde…Null HypothesisAlternative Hypo…Sample MeanVarianceWelch’s t-…

Welch’s t-test is a method of statistical hypothesis testing for comparing the means of two independent populations without assuming that their variances are equal. It divides the difference between two sample means by an estimated standard error and compares the resulting statistic with an approximate Student’s t distribution. Unlike the conventional pooled-variance Student’s t-test, it estimates each population’s variance separately and adjusts the reference distribution’s degrees of freedom.

Statistical setting

Suppose two samples contain observations X11,…,X1n1X_{11},\ldots,X_{1n_1} and X21,…,X2n2X_{21},\ldots,X_{2n_2}, drawn from populations with means μ1,μ2\mu_1,\mu_2 and variances σ12,σ22\sigma_1^2,\sigma_2^2. The observations are assumed to be independent within each sample, and the two samples are assumed to have statistical independence from one another.

The usual null hypothesis is

H0:μ1−μ2=0.H_0:\mu_1-\mu_2=0.

More generally, the hypothesized difference may be any specified value Δ0\Delta_0:

H0:μ1−μ2=Δ0.H_0:\mu_1-\mu_2=\Delta_0.

The alternative hypothesis may be two-sided, μ1−μ2≠Δ0\mu_1-\mu_2\ne\Delta_0, or directional, such as μ1−μ2>Δ0\mu_1-\mu_2>\Delta_0.

Welch’s test addresses a difference in population arithmetic means. It does not test whether the populations have identical distributions, equal medians, or equal variances.

Test statistic

For group ii, let the sample mean and sample variance be

Xˉi=1ni∑j=1niXij,si2=1ni−1∑j=1ni(Xij−Xˉi)2.\bar X_i=\frac{1}{n_i}\sum_{j=1}^{n_i}X_{ij}, \qquad s_i^2=\frac{1}{n_i-1} \sum_{j=1}^{n_i}(X_{ij}-\bar X_i)^2.

The estimated standard error of the difference in sample means is

SE⁡^(Xˉ1−Xˉ2)=s12n1+s22n2.\widehat{\operatorname{SE}}(\bar X_1-\bar X_2) = \sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}.

Welch’s test statistic is therefore

T=(Xˉ1−Xˉ2)−Δ0s12/n1+s22/n2.T= \frac{(\bar X_1-\bar X_2)-\Delta_0} {\sqrt{s_1^2/n_1+s_2^2/n_2}}.

The denominator estimates the standard deviation of the difference between independent sample means. Each group contributes according to both its variability and its sample size; no common population variance is estimated.

Approximate degrees of freedom

The statistic is compared with a t distribution having estimated degrees of freedom

ν=(s12/n1+s22/n2)2(s12/n1)2n1−1+(s22/n2)2n2−1.\nu= \frac{ \left(s_1^2/n_1+s_2^2/n_2\right)^2 }{ \frac{(s_1^2/n_1)^2}{n_1-1} + \frac{(s_2^2/n_2)^2}{n_2-1} }.

This is the Welch–Satterthwaite equation. The value of ν\nu is generally not an integer; modern software evaluates the t distribution at the fractional value directly.

For a two-sided test, the approximate p-value is

p=2[1−Fν ⁣(∣Tobs∣)],p=2\left[1-F_\nu\!\left(|T_{\mathrm{obs}}|\right)\right],

where FνF_\nu is the cumulative distribution function of a t distribution with ν\nu degrees of freedom. At a specified significance level α\alpha, the null hypothesis is rejected when

∣Tobs∣>t1−α/2,ν,|T_{\mathrm{obs}}|>t_{1-\alpha/2,\nu},

where tq,νt_{q,\nu} denotes the t-distribution quantile with cumulative probability qq.

Why the approximation is needed

If the populations have normal distributions, each sample mean is independent of its own sample variance. The difference between sample means is normal, with variance

Var⁡(Xˉ1−Xˉ2)=σ12n1+σ22n2.\operatorname{Var}(\bar X_1-\bar X_2) = \frac{\sigma_1^2}{n_1}+\frac{\sigma_2^2}{n_2}.

However, the denominator of Welch’s statistic involves a weighted sum of two independent variance estimates. Each sample variance is related to a chi-squared distribution, but their weighted sum is generally not a single scaled chi-squared variable. Consequently, the statistic does not generally have an exact t distribution, even for normal populations.

The Welch–Satterthwaite approximation replaces the estimated variance by an approximately equivalent scaled chi-squared variable. Writing

a=s12n1,b=s22n2,a=\frac{s_1^2}{n_1}, \qquad b=\frac{s_2^2}{n_2},

the resulting degrees of freedom are

ν=(a+b)2a2/(n1−1)+b2/(n2−1).\nu=\frac{(a+b)^2}{a^2/(n_1-1)+b^2/(n_2-1)}.

This moment-matching construction accounts for uncertainty in both variance estimates. If one group dominates the estimated variance of the difference, the effective degrees of freedom tend toward those of that group. In general,

min⁡(n1−1,n2−1)≤ν≤n1+n2−2.\min(n_1-1,n_2-1) \le \nu \le n_1+n_2-2.

The test is thus an approximate solution to an unequal-variance inference problem, not an exact finite-sample test for all normal populations.

Confidence interval

The same calculation gives an approximate 100(1−α)%100(1-\alpha)\% confidence interval for the population mean difference:

(Xˉ1−Xˉ2)±t1−α/2,νs12n1+s22n2.(\bar X_1-\bar X_2) \pm t_{1-\alpha/2,\nu} \sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}.

For a two-sided test using the same significance level and approximation, the null hypothesis is rejected exactly when Δ0\Delta_0 lies outside this interval.

The interval expresses the estimated difference in the original measurement units. It is distinct from a standardized effect size, which divides a difference by a chosen measure of variability. When population variances differ, the choice of standardizing quantity is a separate methodological issue.

Comparison with the pooled-variance t-test

The conventional independent-samples Student’s t-test assumes a common population variance and estimates it by

sp2=(n1−1)s12+(n2−1)s22n1+n2−2.s_p^2= \frac{(n_1-1)s_1^2+(n_2-1)s_2^2} {n_1+n_2-2}.

Its standard error is

sp1n1+1n2,s_p\sqrt{\frac{1}{n_1}+\frac{1}{n_2}},

and its reference t distribution has n1+n2−2n_1+n_2-2 degrees of freedom. Under independent normal sampling and equal population variances, this reference distribution is exact.

Welch’s test differs in two ways:

  • It uses separate variance estimates rather than a pooled estimate.
  • It adjusts the degrees of freedom to reflect the uncertainty in those estimates.

Unequal variances are especially consequential for the pooled test when sample sizes also differ. A smaller sample drawn from the more variable population can make the pooled procedure too liberal, increasing its Type I error rate. The reverse arrangement can make it conservative.

If the sample sizes are equal, the two procedures have the same numerical test statistic, but their degrees of freedom can differ. If the sample sizes and sample variances are both equal, the usual formulas coincide.

Welch’s test often has statistical power close to that of the pooled test when population variances are equal, while providing better error-rate control across many unequal-variance settings. Nevertheless, it is not uniformly superior under every distribution and sampling arrangement. Choosing between the procedures by first testing equality of variances creates a two-stage decision rule whose overall behavior need not match that of either test used alone.

Assumptions and limitations

Independence and sampling design

Welch’s variance calculation relies on independent observations. It does not accommodate repeated measurements, matched observations, or clustering merely by allowing unequal variances. For paired observations, the variance of the difference includes a covariance term that is absent from the independent-samples formula.

The interpretation also depends on the sampling or assignment design. A statistically significant difference does not, by itself, establish causation or eliminate confounding and selection effects.

Normality and large-sample behavior

Under normal sampling, the t approximation is often accurate, but it remains an approximation. For nonnormal populations, the central limit theorem supports large-sample inference when the relevant finite-variance and sampling conditions hold.

There is no universal sample-size threshold guaranteeing accuracy. Strong skewness, heavy tails, extreme observations, or a very small sample in one group can impair the approximation. Increasing only the larger group’s size may provide little improvement if uncertainty is dominated by the smaller group.

Allowing unequal variances does not make Welch’s test resistant to outliers: both sample means and sample variances are sensitive to extreme observations.

Degenerate data

The standard formulas require at least two observations per group. If both sample variances are zero, the denominator and degrees-of-freedom calculation are degenerate, and the usual procedure is not defined.

Interpretation

A small p-value indicates incompatibility with the specified null hypothesis under the model and approximation; it is not the probability that the null hypothesis is true. Failure to reject does not establish equality of the population means. Statistical significance also does not measure the practical importance of a difference.

Historical development and related methods

The test is named after the statistician Bernard Lewis Welch, whose work in 1938 and 1947 developed approximate methods for comparing means when population variances are unknown and not assumed equal. Franklin E. Satterthwaite’s 1946 work on approximating the distribution of variance estimates underlies the associated degrees-of-freedom formula.

Welch’s test is a widely used approximate approach to the Behrens–Fisher problem: inference about the difference between two normal population means when both variances are unknown and potentially unequal.

For comparisons involving more than two groups, Welch’s unequal-variance extension of analysis of variance provides a related procedure. The two-group test also has a linear regression interpretation: the coefficient on a binary group indicator estimates the difference in means. With an appropriate heteroskedasticity-consistent variance estimator, its estimated standard error matches Welch’s, although reproducing the full test also requires the Welch–Satterthwaite degrees-of-freedom adjustment.