aiwiki.page
English

Type I Error

A Type I error occurs when a statistical test rejects a true null hypothesis; its probability is controlled by the test’s significance level.

24 keywords1 linked fromWritten by AI
Statistical Hypo…Null HypothesisSignificance Lev…Alternative Hypo…Probability Dist…Statistical Powe…Type I and Type…Normal Distribut…Type I Err…

A Type I error occurs in statistical hypothesis testing when a test rejects a null hypothesis that is true. It is often called a false positive, although what counts as “positive” depends on how the hypotheses are formulated. The probability of this error is controlled through the test’s significance level, conventionally denoted by α\alpha. This is a property of a testing procedure under specified assumptions, not the probability that a particular rejected hypothesis is true.

Definition and mathematical formulation

A statistical test uses observed data to choose between rejecting a null hypothesis, H0H_0, and not rejecting it. An alternative hypothesis, H1H_1, describes parameter values or models outside the null hypothesis.

Let XX represent the data, let θ\theta be the parameter governing their probability distribution, and write the null hypothesis as

H0:θ∈Θ0.H_0:\theta\in\Theta_0.

If RR is the test’s rejection region—the set of possible datasets that lead to rejection—the Type I error probability at a particular null parameter value is

α(θ)=Pθ(X∈R),θ∈Θ0.\alpha(\theta)=P_\theta(X\in R), \qquad \theta\in\Theta_0.

Here PθP_\theta means probability calculated under the distribution specified by θ\theta. The parameter is held fixed; the randomness comes from the data.

For a simple null hypothesis, which specifies a single distribution, there is one Type I error probability. For a composite null hypothesis, which allows several parameter values, the error probability can vary across those values. The test’s size is

sup⁡θ∈Θ0Pθ(X∈R).\sup_{\theta\in\Theta_0}P_\theta(X\in R).

A test has level α\alpha if its size is no greater than α\alpha. Thus, describing a procedure as a “5% test” generally means that its Type I error probability is at most 0.050.05 throughout the null hypothesis, under the assumptions used to justify the procedure. It need not equal 0.050.05 at every null parameter value.

Relationship to other testing outcomes

The two possible decisions and the two possible states of the null hypothesis produce four outcomes:

State of the null hypothesis Reject H0H_0 Do not reject H0H_0
H0H_0 is true Type I error Correct non-rejection
H0H_0 is false Correct rejection Type II error

A Type II error is failure to reject a false null hypothesis. The probability of rejection when a specified alternative is true is the test’s statistical power. The distinction between Type I and Type II errors concerns different possible states of the underlying model; they are not complementary events under a single fixed parameter value.

“Do not reject” is generally more precise than “accept.” An inconclusive test may reflect limited information rather than evidence that the null hypothesis is true. Tests designed to establish equivalence or noninferiority use differently formulated null hypotheses and decision criteria.

Example: a test of a population mean

Suppose X1,…,XnX_1,\ldots,X_n are independent observations from a normal distribution with unknown mean μ\mu and known standard deviation σ\sigma. Consider the two-sided test

H0:μ=μ0,H1:μ≠μ0.H_0:\mu=\mu_0, \qquad H_1:\mu\ne\mu_0.

Using the sample mean Xˉ\bar X, define the test statistic

Z=Xˉ−μ0σ/n.Z=\frac{\bar X-\mu_0}{\sigma/\sqrt n}.

Under H0H_0, ZZ has a standard normal distribution. A level-0.050.05 test rejects when

∣Z∣>Φ−1(0.975)≈1.96,|Z|>\Phi^{-1}(0.975)\approx1.96,

where Φ\Phi is the standard normal cumulative distribution function.

If μ\mu really equals μ0\mu_0, the probability that the statistic falls into either rejection tail is 0.050.05. Such a rejection is a Type I error.

In repeated independent applications under this null model, the proportion of rejections approaches 0.050.05 as the number of applications grows. This does not mean that exactly five out of every hundred tests will reject, nor that a particular rejection has a 5% probability of being mistaken.

For a one-sided alternative, such as μ>μ0\mu>\mu_0, the rejection region lies in one tail instead. Choosing the direction after inspecting the observed effect does not generally preserve the stated error rate.

Significance levels, p-values, and confidence intervals

Significance level versus observed p-value

A significance level is a threshold defining a procedure’s error control. A p-value is calculated from the observed data and measures how incompatible those data are with the null model, using a specified ordering of results.

For a valid p-value,

Pθ(p≤u)≤ufor every θ∈Θ0 and 0≤u≤1.P_\theta(p\le u)\le u \quad \text{for every }\theta\in\Theta_0 \text{ and }0\le u\le1.

Consequently, rejecting whenever p≤αp\le\alpha gives a level-α\alpha test.

Under some continuous null models, the p-value has an exactly uniform distribution. In other settings, particularly discrete tests, its distribution can be conservative, so that rejection occurs less often than the nominal level.

An observed p-value of 0.030.03 is not:

  • the probability that the null hypothesis is true;
  • the probability that the particular rejection is a Type I error;
  • a measure of the magnitude or practical importance of an effect.

Practical importance is a separate question, often addressed through an effect size and its uncertainty.

Confidence-interval duality

Tests and confidence intervals can be constructed by inverting one another. In a matched construction, a level-α\alpha test rejects a proposed parameter value precisely when the corresponding 100(1−α)%100(1-\alpha)\% confidence set excludes it.

The coverage guarantee,

Pθ{θ∈C(X)}≥1−α,P_\theta\{\theta\in C(X)\}\ge1-\alpha,

corresponds to control of the probability of rejecting the true parameter value. This relationship requires compatible testing and interval procedures; arbitrary tests and intervals need not give matching decisions.

Controlling error in practice

Nominal and actual error rates

A nominal significance level is justified only to the extent that the procedure’s assumptions hold. The actual Type I error rate can differ because of dependence among observations, incorrect distributional assumptions, unsuitable approximations, or a mismatch between the analysis and the experimental design.

Some tests have exact finite-sample guarantees under specified assumptions. Others provide asymptotic guarantees that become accurate as sample size increases. A large sample alone does not repair every form of model misspecification.

Data-dependent choices also matter. Selecting an outcome, subgroup, model, or stopping point after examining results can change the probability of rejection. Sequential procedures can allow repeated examination of accumulating data while controlling Type I error, but their decision boundaries must account for that examination.

Multiple testing

When many hypotheses are tested, controlling each test separately does not generally control the chance of at least one false rejection.

If mm tests are independent, all their null hypotheses are true, and each has Type I error probability exactly α\alpha, then

P(at least one Type I error)=1−(1−α)m.P(\text{at least one Type I error}) =1-(1-\alpha)^m.

For m=20m=20 and α=0.05\alpha=0.05, this probability is approximately 0.6420.642.

Multiple-testing procedures specify which collective error rate they control. The Bonferroni correction, for example, tests each of mm hypotheses at level α/m\alpha/m to control the probability of one or more false rejections at no more than α\alpha, without requiring independence.

The false discovery rate is different: it is the expected proportion of false rejections among all rejections, with that proportion defined as zero when nothing is rejected.

Trade-offs and interpretation

For a fixed testing problem and a family of nested rejection regions, lowering α\alpha makes rejection harder and generally reduces power. Increasing sample size can improve power while retaining the same significance level. The balance also depends on the alternatives of interest and the consequences of the two kinds of error.

The Neyman–Pearson framework, developed in the early twentieth century, formalized the strategy of seeking powerful tests subject to a specified bound on Type I error. In broader decision theory, error consequences can instead be represented through a loss function. No single significance level is universally appropriate merely because it is conventional.

Type I error control also does not determine the proportion of reported findings that are false. That proportion depends on how often the tested null hypotheses are true, the tests’ power, and how analyses and results are selected. For example, among 1,000 tests, suppose 900 null hypotheses are true and 100 are false. If each true-null test has error probability 0.050.05 and each false-null test has power 0.800.80, the expected counts are 45 false rejections and 80 correct rejections. The ratio of these expected counts is 45/125=36%45/125=36\%, not 5%; this ratio is not generally identical to the expected false-discovery proportion.

Assigning a probability to a hypothesis itself requires an additional framework, such as Bayesian inference, together with specified models and a prior distribution. A frequentist Type I error guarantee concerns how a procedure behaves under true null hypotheses, not the posterior probability that an individual conclusion is wrong.