The Kolmogorov–Smirnov test is a family of statistical hypothesis tests that measure discrepancies between cumulative distributions. The one-sample test assesses whether observations follow a specified probability distribution, while the two-sample test assesses whether two independent samples come from the same distribution. Its statistic is the greatest vertical separation between the relevant cumulative distribution functions. Under independent sampling and appropriate continuity assumptions, its null distribution does not depend on the particular population distribution.
Definition and intuition
A cumulative distribution function (CDF), , gives the probability that a random variable is at most . For observations , the empirical distribution function is
where is an indicator function. This step function records the fraction of observations at or below each value.
The Kolmogorov–Smirnov statistic measures the largest difference between cumulative probabilities across all thresholds . Unlike a comparison of means, it can respond to differences in location, dispersion, or shape. Unlike a histogram-based test, it does not require observations to be divided into bins.
The statistic is expressed in probability units. For example, a two-sample value of means that at some threshold the proportions of observations at or below that threshold differ by 20 percentage points. It does not mean that the measured values differ by 20 percent.
One-sample test
Hypotheses and statistic
Suppose are independent and identically distributed observations with unknown CDF . The null hypothesis is
where is a fully specified reference CDF. The two-sided alternative is that the two functions differ somewhere.
The test statistic is
Here, the supremum is the least upper bound over all real thresholds. Large values provide evidence against .
When is continuous, sorting the observations as gives
and
Both sides of each empirical jump matter. Evaluating only can miss the largest discrepancy.
One-sided alternatives
The statistics and also support directional tests. A large indicates that the empirical CDF exceeds the reference CDF somewhere; a large indicates the reverse.
CDF direction must be interpreted carefully: a distribution concentrated on smaller values tends to have a larger CDF. Thus, “larger CDF” does not mean “larger observations.”
Two-sample test
For independent samples and , let and be their empirical CDFs. The null hypothesis is
and the two-sided statistic is
No particular common distribution needs to be specified. The test compares entire distributions rather than only their means or medians.
The statistic can be computed by sorting the pooled observations and tracking the difference between the two empirical CDFs. When tied observations occur, all observations at the same value must be treated consistently; an arbitrary ordering within a tied group must not create artificial CDF differences.
The usual formulation assumes independence both within and between samples. Paired measurements, repeated observations from the same individual, and serially dependent data do not generally satisfy these assumptions.
Distribution theory and calibration
Why the classical test is distribution-free
For a continuous, fully specified , the probability integral transform implies that follows the uniform distribution on under the null hypothesis. Consequently, the sampling distribution of depends on sample size but not on the shape of .
For the two-sample test under a common continuous distribution, the ordering of the pooled observations has a distribution determined by sample sizes alone. This gives the corresponding distribution-free property.
“Distribution-free” therefore describes a property under specified assumptions. It does not mean that continuity, independence, or the treatment of estimated parameters can be ignored.
Large-sample distribution
Under the continuous one-sample null hypothesis,
where denotes convergence in distribution and is a standard Brownian bridge. This is a zero-mean Gaussian process with covariance
If , then for ,
This limiting law is called the Kolmogorov distribution.
For the two-sample test,
as both sample sizes grow under the usual assumptions.
At significance level , large-sample rejection rules are therefore
and
where . Common two-sided values are approximately and . These are asymptotic values, not universally exact finite-sample thresholds.
Exact and simulated probabilities
Finite-sample distributions can be evaluated using specialized numerical algorithms. In the continuous two-sample case, exact probabilities can be obtained by counting admissible arrangements of the two sample labels in the pooled ordering.
The p-value is the null probability of a statistic at least as large as the observed one. Exact and asymptotic calculations can differ, particularly for small samples. Monte Carlo methods can provide approximate calibration when suitable null simulations are available.
Estimated parameters and the Lilliefors problem
The classical one-sample test requires a fully specified reference distribution. Testing against a normal distribution with a mean and variance fixed independently of the sample is not the same problem as testing against a normal distribution whose mean and variance were estimated from that sample.
For example, consider
where is the standard normal CDF and are fitted from the observations. Fitting makes the reference distribution adapt to the data, changing the statistic’s null distribution. Ordinary Kolmogorov–Smirnov critical values are therefore invalid for this procedure and typically make it overly conservative.
The Lilliefors test provides a calibration for important fitted-distribution cases, including normality with estimated parameters. More generally, a parametric bootstrap can simulate samples from the fitted null model, refit the parameters in every simulated sample, and recompute the statistic. Repeating the fitting step is essential to reproduce the procedure being calibrated.
Discrete data, ties, and dependence
Discrete and rounded observations
For a discrete reference distribution, the probability integral transform is not continuously uniform, and the standard continuous-null distribution no longer applies. The statistic remains meaningful, but its calibration generally depends on the specified distribution.
Using continuous-null critical values for a fixed discrete one-sample null is generally conservative: the actual rejection probability is no greater than the nominal level, though it may be substantially smaller.
Ties may reflect genuinely discrete data or rounding of continuous measurements. Software implementations differ in how they handle such cases. For the two-sample problem, a permutation test based on relabeling the pooled observations can accommodate ties when sample labels are exchangeable under the null. The conventional no-ties exact calculation is not the same as a calculation conditional on a tied pooled sample.
Dependent observations
In time series, clustered samples, and other dependent data, the fluctuations of the empirical CDF can differ from those assumed by the classical test. Ordinary critical values can then give incorrect rejection rates.
This issue also arises when the test is applied to fitted regression residuals. Residuals are generally dependent and constrained by the fitting procedure, so their interpretation is not equivalent to that of an independent sample from a fully specified distribution.
Interpretation, power, and limitations
The test is omnibus: under standard conditions it is consistent against any fixed alternative distribution different from the null. Nevertheless, its power varies with the form of the discrepancy.
Important limitations include:
- Tail sensitivity. The statistic uses unweighted CDF differences. Discrepancies confined to extreme tails may produce small probability differences and can be harder to detect than with tail-weighted tests.
- Loss of detail. A single maximum does not describe all the ways distributions differ. Distinct discrepancies can produce the same statistic.
- Sample-size sensitivity. Large samples can detect small differences, while small samples may fail to detect substantial ones.
- No proof of equality. Failure to reject does not establish that the null distribution is correct or that two populations are identical.
- No diagnosis of cause. Rejection does not by itself identify whether the difference concerns location, dispersion, skewness, or another feature.
In the two-sample setting, can also be reported as a descriptive effect size: the maximum observed difference in cumulative proportions. Its practical importance depends on the application, not solely on statistical significance.
Related methods and extensions
The Anderson–Darling test weights squared CDF discrepancies more heavily near the tails. The Cramér–von Mises test aggregates squared discrepancies over the distribution rather than retaining only the largest one. These methods emphasize different aspects of departure from a reference distribution.
The Kuiper test uses rather than their maximum. It is particularly useful for circular observations, where the choice of origin should not determine the result.
The classical Kolmogorov–Smirnov test is fundamentally one-dimensional. Proposed multivariate extensions do not generally inherit its simple distribution-free calibration.
Historical development
Andrey Kolmogorov established the limiting distribution underlying the one-sample statistic in 1933. Nikolai Smirnov developed related one-sided and two-sample distributional results during the 1930s. Their work made the maximum distance between empirical and theoretical cumulative distributions a central tool in nonparametric statistics.
The broader theory connects the statistic to uniform convergence of empirical distributions, formalized by the Glivenko–Cantelli theorem, and to stochastic-process limits represented by the Brownian bridge.