An empirical distribution is the probability distribution that assigns equal probability to each observation in a dataset, combining those probabilities when values repeat. It describes the data without assuming a parametric model, such as a normal or exponential distribution. For numerical observations, it is commonly represented by an empirical cumulative distribution function (ECDF), which gives the proportion of observations at or below each value. Empirical distributions are fundamental in statistics, connecting observed frequencies with probability models, estimation, and resampling.
Definition
Let be observations in a measurable space , with . Their empirical distribution is the probability measure defined by
where is an indicator function. Thus, the empirical probability of a set is the fraction of observations belonging to that set.
Equivalently,
where denotes the point-mass probability measure concentrated at .
Each observation contributes mass . If a value occurs times, the total mass at that value is . The empirical distribution is therefore discrete, even when the observations were sampled from a continuous population distribution.
The definition itself requires no assumptions about how the data were obtained. Assumptions such as independence and identical distribution become important when using the empirical distribution to infer properties of a population.
Empirical cumulative distribution function
For real-valued observations, the empirical cumulative distribution function is
It is a nondecreasing, right-continuous step function. It equals zero below the smallest observation and one at and above the largest observation. At each distinct observed value, it jumps by that value’s relative frequency.
If the observations are sorted as
then, when all observations are distinct, between and . Repeated observations produce larger jumps.
Example
For the dataset
the empirical probabilities are
The corresponding ECDF is
Consequently, the empirical probability of the interval is
The empirical distribution and its ECDF contain the same information for real-valued data. The former emphasizes probability assigned to sets; the latter gives a convenient numerical and graphical representation.
Empirical averages, moments, and quantiles
For any real-valued function that is finite at the observed values, integration against the empirical distribution gives
Thus, many familiar sample statistics are properties of the empirical distribution.
Its expected value is the sample mean:
Its variance is
This uses denominator , because it is the variance of the empirical probability distribution itself. By contrast, the usual unbiased estimator of a population variance uses denominator , under independent, identically distributed sampling with finite variance. The distinction is the basis of Bessel’s correction.
For , an empirical quantile can be defined through the generalized inverse
Under this convention,
Statistical software also uses quantile conventions that interpolate between adjacent observations. Such interpolated values need not coincide with the generalized inverse of the step-function ECDF. For example, the common convention of averaging the two middle observations in an even-sized dataset differs from the generalized-inverse definition of the median.
Statistical properties
Suppose are independent and identically distributed observations with population CDF . Before the data are observed, and are random objects.
Behavior at a fixed point
For any fixed , the count of observations not exceeding has a binomial distribution:
It follows that
Thus, is an unbiased estimator of . The law of large numbers gives
at every fixed .
When , the central limit theorem also gives
Uniform convergence
A stronger result, the Glivenko–Cantelli theorem, states that
This is uniform convergence: the largest discrepancy anywhere on the real line tends to zero, not merely the discrepancy at any one fixed point.
A finite-sample bound is provided by the Dvoretzky–Kiefer–Wolfowitz inequality:
For , setting
produces a simultaneous confidence band with coverage at least :
for every , with the stated probability. Unlike a pointwise confidence interval, this band concerns the entire CDF at once.
Applications
Nonparametric estimation
The empirical distribution is a basic nonparametric estimator: it estimates a distribution without selecting a finite-dimensional parametric family.
The plug-in principle estimates a population quantity by applying the same mathematical operation to the empirical distribution:
This yields sample moments, quantiles, proportions, and many other statistics. The usefulness of a plug-in estimate depends on both the sampling assumptions and the behavior of the functional .
Bootstrap resampling
In the ordinary nonparametric bootstrap, new samples are drawn independently from , conditional on the observed data. Operationally, this means sampling the original observations with replacement.
Repeatedly recomputing a statistic on these samples approximates its sampling distribution. The empirical distribution therefore acts as a substitute population. Bootstrap validity is not automatic: it depends on the statistic and sampling setting, and the ordinary bootstrap can fail for some extreme-value or other nonregular problems.
Goodness-of-fit testing
Empirical CDFs are central to goodness-of-fit testing. The one-sample Kolmogorov–Smirnov test compares an empirical CDF with a specified CDF , using
For a fully specified continuous null distribution, the null distribution of this statistic does not depend on the particular form of . Standard calibration generally changes if the null distribution is discrete or its parameters are estimated from the same data.
Two-sample versions compare empirical CDFs from separate datasets.
Empirical risk minimization
In statistical learning, an empirical distribution turns a population-average loss into an observed-data average. For data and loss ,
Minimizing this empirical risk is a central estimation method. Good performance on observed data does not by itself guarantee good performance on new data; model complexity and sampling conditions also matter.
Extensions
A weighted empirical distribution assigns nonnegative weights satisfying :
Weights can represent unequal sampling probabilities, importance weights, or other adjustments. Its statistical properties depend on how the weights are constructed; results for the ordinary empirical CDF cannot automatically be applied unchanged.
For vector-valued observations in , the same point-mass definition applies. A multivariate empirical CDF is
The measure-based definition also accommodates categorical data and other measurable observation spaces, even when no natural ordering or one-dimensional CDF exists.
Distinctions and limitations
An empirical distribution is not a histogram. A histogram groups observations into bins, and its appearance depends on the bin boundaries and widths. The empirical distribution preserves the exact observed values and their frequencies, though not their original order.
Nor is it a probability density function with respect to ordinary length measure. A finite empirical distribution has point masses. Density estimation methods, such as kernel smoothing, construct a different object by spreading probability around observations.
The empirical distribution is also distinct from the sampling distribution of a statistic. The former describes the observed data; the latter describes how a statistic varies across repeated samples.
Its main limitations concern what the observations can support:
- Unobserved outcomes receive zero empirical probability. This is not evidence that their population probability is zero. In particular, the empirical distribution cannot directly extrapolate beyond the observed minimum and maximum.
- Representativeness is not guaranteed. Selection bias or an unsuitable sampling design can prevent the empirical distribution from approximating the intended population.
- Dependence changes uncertainty. For time series or clustered observations, independent-sampling variance formulas and confidence bounds generally do not apply without modification.
- CDF convergence does not ensure convergence of every statistic. Consistency of moments and other unbounded functionals requires additional conditions, such as integrability.
- Order is discarded. Datasets with the same values and frequencies have the same empirical distribution, even if their temporal or spatial arrangements differ.