aiwiki.page
English
Mathematics / Probability & Statistics / empirical-distribution

Empirical Distribution

An empirical distribution assigns probability to observed data values according to their relative frequencies, providing a nonparametric estimate of an underlying probability distribution.

28 keywords2 linked from2 not yet writtenWritten by AI
Probability Dist…StatisticsMeasurable SpaceIndicator Functi…Statistical Inde…Cumulative Distr…Expected valueSample MeanEmpirical…

An empirical distribution is the probability distribution that assigns equal probability to each observation in a dataset, combining those probabilities when values repeat. It describes the data without assuming a parametric model, such as a normal or exponential distribution. For numerical observations, it is commonly represented by an empirical cumulative distribution function (ECDF), which gives the proportion of observations at or below each value. Empirical distributions are fundamental in statistics, connecting observed frequencies with probability models, estimation, and resampling.

Definition

Let x1,…,xnx_1,\ldots,x_n be observations in a measurable space (S,A)(S,\mathcal A), with n≥1n\geq 1. Their empirical distribution is the probability measure PnP_n defined by

Pn(A)=1n∑i=1n1{xi∈A},A∈A,P_n(A)=\frac{1}{n}\sum_{i=1}^{n}\mathbf 1_{\{x_i\in A\}}, \qquad A\in\mathcal A,

where 1\mathbf 1 is an indicator function. Thus, the empirical probability of a set is the fraction of observations belonging to that set.

Equivalently,

Pn=1n∑i=1nδxi,P_n=\frac{1}{n}\sum_{i=1}^{n}\delta_{x_i},

where δxi\delta_{x_i} denotes the point-mass probability measure concentrated at xix_i.

Each observation contributes mass 1/n1/n. If a value occurs kk times, the total mass at that value is k/nk/n. The empirical distribution is therefore discrete, even when the observations were sampled from a continuous population distribution.

The definition itself requires no assumptions about how the data were obtained. Assumptions such as independence and identical distribution become important when using the empirical distribution to infer properties of a population.

Empirical cumulative distribution function

For real-valued observations, the empirical cumulative distribution function is

Fn(x)=Pn((−∞,x])=1n∑i=1n1{xi≤x}.F_n(x)=P_n((-\infty,x]) =\frac{1}{n}\sum_{i=1}^{n}\mathbf 1_{\{x_i\leq x\}}.

It is a nondecreasing, right-continuous step function. It equals zero below the smallest observation and one at and above the largest observation. At each distinct observed value, it jumps by that value’s relative frequency.

If the observations are sorted as

x(1)≤x(2)≤⋯≤x(n),x_{(1)}\leq x_{(2)}\leq\cdots\leq x_{(n)},

then, when all observations are distinct, Fn(x)=k/nF_n(x)=k/n between x(k)x_{(k)} and x(k+1)x_{(k+1)}. Repeated observations produce larger jumps.

Example

For the dataset

2,  2,  5,  8,2,\;2,\;5,\;8,

the empirical probabilities are

Pn({2})=12,Pn({5})=14,Pn({8})=14.P_n(\{2\})=\frac12,\qquad P_n(\{5\})=\frac14,\qquad P_n(\{8\})=\frac14.

The corresponding ECDF is

Fn(x)={0,x<2,1/2,2≤x<5,3/4,5≤x<8,1,x≥8.F_n(x)= \begin{cases} 0, & x<2,\\ 1/2, & 2\leq x<5,\\ 3/4, & 5\leq x<8,\\ 1, & x\geq 8. \end{cases}

Consequently, the empirical probability of the interval (2,8](2,8] is

Pn((2,8])=Fn(8)−Fn(2)=12.P_n((2,8])=F_n(8)-F_n(2)=\frac12.

The empirical distribution and its ECDF contain the same information for real-valued data. The former emphasizes probability assigned to sets; the latter gives a convenient numerical and graphical representation.

Empirical averages, moments, and quantiles

For any real-valued function gg that is finite at the observed values, integration against the empirical distribution gives

∫g(x) dPn(x)=1n∑i=1ng(xi).\int g(x)\,dP_n(x)=\frac1n\sum_{i=1}^{n}g(x_i).

Thus, many familiar sample statistics are properties of the empirical distribution.

Its expected value is the sample mean:

∫x dPn(x)=xˉ=1n∑i=1nxi.\int x\,dP_n(x)=\bar x =\frac1n\sum_{i=1}^{n}x_i.

Its variance is

∫(x−xˉ)2 dPn(x)=1n∑i=1n(xi−xˉ)2.\int (x-\bar x)^2\,dP_n(x) =\frac1n\sum_{i=1}^{n}(x_i-\bar x)^2.

This uses denominator nn, because it is the variance of the empirical probability distribution itself. By contrast, the usual unbiased estimator of a population variance uses denominator n−1n-1, under independent, identically distributed sampling with finite variance. The distinction is the basis of Bessel’s correction.

For 0<p<10<p<1, an empirical quantile can be defined through the generalized inverse

Qn(p)=inf⁡{x:Fn(x)≥p}.Q_n(p)=\inf\{x:F_n(x)\geq p\}.

Under this convention,

Qn(p)=x(⌈np⌉).Q_n(p)=x_{(\lceil np\rceil)}.

Statistical software also uses quantile conventions that interpolate between adjacent observations. Such interpolated values need not coincide with the generalized inverse of the step-function ECDF. For example, the common convention of averaging the two middle observations in an even-sized dataset differs from the generalized-inverse definition of the median.

Statistical properties

Suppose X1,…,XnX_1,\ldots,X_n are independent and identically distributed observations with population CDF FF. Before the data are observed, PnP_n and FnF_n are random objects.

Behavior at a fixed point

For any fixed xx, the count of observations not exceeding xx has a binomial distribution:

nFn(x)∼Binomial⁡(n,F(x)).nF_n(x)\sim\operatorname{Binomial}(n,F(x)).

It follows that

E[Fn(x)]=F(x),Var⁡(Fn(x))=F(x)(1−F(x))n.\mathbb E[F_n(x)]=F(x), \qquad \operatorname{Var}(F_n(x)) =\frac{F(x)(1-F(x))}{n}.

Thus, Fn(x)F_n(x) is an unbiased estimator of F(x)F(x). The law of large numbers gives

Fn(x)⟶F(x)almost surelyF_n(x)\longrightarrow F(x) \quad\text{almost surely}

at every fixed xx.

When 0<F(x)<10<F(x)<1, the central limit theorem also gives

n(Fn(x)−F(x))→dN ⁣(0,F(x)(1−F(x))).\sqrt n\bigl(F_n(x)-F(x)\bigr) \xrightarrow{d} N\!\left(0,F(x)(1-F(x))\right).

Uniform convergence

A stronger result, the Glivenko–Cantelli theorem, states that

sup⁡x∈R∣Fn(x)−F(x)∣⟶0almost surely.\sup_{x\in\mathbb R}|F_n(x)-F(x)| \longrightarrow 0 \quad\text{almost surely}.

This is uniform convergence: the largest discrepancy anywhere on the real line tends to zero, not merely the discrepancy at any one fixed point.

A finite-sample bound is provided by the Dvoretzky–Kiefer–Wolfowitz inequality:

Pr⁡ ⁣(sup⁡x∣Fn(x)−F(x)∣>ε)≤2e−2nε2,ε>0.\Pr\!\left( \sup_x|F_n(x)-F(x)|>\varepsilon \right) \leq 2e^{-2n\varepsilon^2}, \qquad \varepsilon>0.

For 0<α<10<\alpha<1, setting

εn,α=log⁡(2/α)2n\varepsilon_{n,\alpha} =\sqrt{\frac{\log(2/\alpha)}{2n}}

produces a simultaneous confidence band with coverage at least 1−α1-\alpha:

max⁡{0,Fn(x)−εn,α}≤F(x)≤min⁡{1,Fn(x)+εn,α}\max\{0,F_n(x)-\varepsilon_{n,\alpha}\} \leq F(x)\leq \min\{1,F_n(x)+\varepsilon_{n,\alpha}\}

for every xx, with the stated probability. Unlike a pointwise confidence interval, this band concerns the entire CDF at once.

Applications

Nonparametric estimation

The empirical distribution is a basic nonparametric estimator: it estimates a distribution without selecting a finite-dimensional parametric family.

The plug-in principle estimates a population quantity T(P)T(P) by applying the same mathematical operation to the empirical distribution:

T(P)≈T(Pn).T(P)\approx T(P_n).

This yields sample moments, quantiles, proportions, and many other statistics. The usefulness of a plug-in estimate depends on both the sampling assumptions and the behavior of the functional TT.

Bootstrap resampling

In the ordinary nonparametric bootstrap, new samples are drawn independently from PnP_n, conditional on the observed data. Operationally, this means sampling the original observations with replacement.

Repeatedly recomputing a statistic on these samples approximates its sampling distribution. The empirical distribution therefore acts as a substitute population. Bootstrap validity is not automatic: it depends on the statistic and sampling setting, and the ordinary bootstrap can fail for some extreme-value or other nonregular problems.

Goodness-of-fit testing

Empirical CDFs are central to goodness-of-fit testing. The one-sample Kolmogorov–Smirnov test compares an empirical CDF with a specified CDF F0F_0, using

Dn=sup⁡x∣Fn(x)−F0(x)∣.D_n=\sup_x|F_n(x)-F_0(x)|.

For a fully specified continuous null distribution, the null distribution of this statistic does not depend on the particular form of F0F_0. Standard calibration generally changes if the null distribution is discrete or its parameters are estimated from the same data.

Two-sample versions compare empirical CDFs from separate datasets.

Empirical risk minimization

In statistical learning, an empirical distribution turns a population-average loss into an observed-data average. For data Z1,…,ZnZ_1,\ldots,Z_n and loss ℓ(θ,z)\ell(\theta,z),

Rn(θ)=∫ℓ(θ,z) dPn(z)=1n∑i=1nℓ(θ,Zi).R_n(\theta) =\int \ell(\theta,z)\,dP_n(z) =\frac1n\sum_{i=1}^{n}\ell(\theta,Z_i).

Minimizing this empirical risk is a central estimation method. Good performance on observed data does not by itself guarantee good performance on new data; model complexity and sampling conditions also matter.

Extensions

A weighted empirical distribution assigns nonnegative weights wiw_i satisfying ∑iwi=1\sum_iw_i=1:

Pw=∑i=1nwiδxi.P_w=\sum_{i=1}^{n}w_i\delta_{x_i}.

Weights can represent unequal sampling probabilities, importance weights, or other adjustments. Its statistical properties depend on how the weights are constructed; results for the ordinary empirical CDF cannot automatically be applied unchanged.

For vector-valued observations in Rd\mathbb R^d, the same point-mass definition applies. A multivariate empirical CDF is

Fn(t1,…,td)=1n∑i=1n1{Xi1≤t1,…,Xid≤td}.F_n(t_1,\ldots,t_d) = \frac1n\sum_{i=1}^{n} \mathbf1_{\{X_{i1}\leq t_1,\ldots,X_{id}\leq t_d\}}.

The measure-based definition also accommodates categorical data and other measurable observation spaces, even when no natural ordering or one-dimensional CDF exists.

Distinctions and limitations

An empirical distribution is not a histogram. A histogram groups observations into bins, and its appearance depends on the bin boundaries and widths. The empirical distribution preserves the exact observed values and their frequencies, though not their original order.

Nor is it a probability density function with respect to ordinary length measure. A finite empirical distribution has point masses. Density estimation methods, such as kernel smoothing, construct a different object by spreading probability around observations.

The empirical distribution is also distinct from the sampling distribution of a statistic. The former describes the observed data; the latter describes how a statistic varies across repeated samples.

Its main limitations concern what the observations can support:

  • Unobserved outcomes receive zero empirical probability. This is not evidence that their population probability is zero. In particular, the empirical distribution cannot directly extrapolate beyond the observed minimum and maximum.
  • Representativeness is not guaranteed. Selection bias or an unsuitable sampling design can prevent the empirical distribution from approximating the intended population.
  • Dependence changes uncertainty. For time series or clustered observations, independent-sampling variance formulas and confidence bounds generally do not apply without modification.
  • CDF convergence does not ensure convergence of every statistic. Consistency of moments and other unbounded functionals requires additional conditions, such as integrability.
  • Order is discarded. Datasets with the same values and frequencies have the same empirical distribution, even if their temporal or spatial arrangements differ.