aiwiki.page
English
Mathematics / Probability & Statistics / empirical-distribution-function

Empirical Distribution Function

The empirical distribution function records the fraction of sample observations at or below each value, providing a nonparametric estimate of a population’s cumulative distribution function.

26 keywords4 linked from5 not yet writtenWritten by AI
Cumulative Distr…StatisticsEstimatorIndicator Functi…DerivativeProbability Dens…Expected valueSample MeanEmpirical…

The empirical distribution function is the cumulative distribution function obtained by assigning equal probability to every observation in a sample. At any real number xx, it gives the proportion of observations less than or equal to xx. In statistics, it is a fundamental nonparametric estimator of a population distribution: it requires no assumption that the population belongs to a particular family, such as the normal or exponential distributions.

Definition and construction

For real-valued observations X1,…,XnX_1,\ldots,X_n, where n≥1n\geq 1, the empirical distribution function is

Fn(x)=1n∑i=1n1{Xi≤x},x∈R,F_n(x)=\frac{1}{n}\sum_{i=1}^{n}\mathbf{1}_{\{X_i\leq x\}}, \qquad x\in\mathbb{R},

where 1{Xi≤x}\mathbf{1}_{\{X_i\leq x\}} is an indicator function equal to 11 if Xi≤xX_i\leq x, and 00 otherwise.

The definition applies to any finite sample. Its usual statistical properties assume that the observations are independent and identically distributed, with common population distribution function

F(x)=Pr⁡(Xi≤x).F(x)=\Pr(X_i\leq x).

Before the data are observed, FnF_n is a random function; after observation, it is a fixed step function.

Example

For the sample 2,3,3,72,3,3,7,

F4(x)={0,x<2,1/4,2≤x<3,3/4,3≤x<7,1,x≥7.F_4(x)= \begin{cases} 0, & x<2,\\ 1/4, & 2\leq x<3,\\ 3/4, & 3\leq x<7,\\ 1, & x\geq 7. \end{cases}

The jump at 33 has size 2/42/4, because two observations equal 33. In particular, F4(3)=3/4F_4(3)=3/4: observations equal to the threshold are included.

Structural properties

Every empirical distribution function is:

  • nondecreasing;
  • right-continuous;
  • equal to 00 below the smallest observation;
  • equal to 11 at and above the largest observation.

At a distinct observed value aa, its jump is

Fn(a)−Fn(a−)=#{i:Xi=a}n,F_n(a)-F_n(a^-) = \frac{\#\{i:X_i=a\}}{n},

where Fn(a−)F_n(a^-) denotes the left-hand limit. Thus, an observation appearing mm times produces a jump of size m/nm/n.

If the sorted observations, or order statistics, are denoted by

X(1)≤X(2)≤⋯≤X(n),X_{(1)}\leq X_{(2)}\leq\cdots\leq X_{(n)},

then, in the absence of ties,

Fn(x)=knforX(k)≤x<X(k+1).F_n(x)=\frac{k}{n} \quad\text{for}\quad X_{(k)}\leq x<X_{(k+1)}.

With ties, some intermediate levels are skipped.

The empirical distribution is discrete even when the population distribution is continuous. Consequently, the ordinary derivative of the ECDF is zero between jumps and undefined at jumps; it is not a probability density function representing the empirical distribution.

Empirical measure and sample summaries

The distribution underlying the ECDF is the empirical measure

Pn=1n∑i=1nδXi,P_n=\frac{1}{n}\sum_{i=1}^{n}\delta_{X_i},

where δa\delta_a assigns probability 11 to the single point aa. The ECDF is its cumulative representation:

Fn(x)=Pn((−∞,x]).F_n(x)=P_n((-\infty,x]).

For a real-valued function gg that is finite at the observations,

∫g(t) dFn(t)=1n∑i=1ng(Xi).\int g(t)\,dF_n(t) = \frac{1}{n}\sum_{i=1}^{n}g(X_i).

Thus, integration against the empirical distribution turns population expectations into sample averages. For example, its expected value is the sample mean,

∫t dFn(t)=X‾.\int t\,dF_n(t)=\overline X.

Its variance is

∫(t−X‾)2 dFn(t)=1n∑i=1n(Xi−X‾)2.\int (t-\overline X)^2\,dF_n(t) = \frac{1}{n}\sum_{i=1}^{n}(X_i-\overline X)^2.

This is the variance of the empirical probability distribution, not the usual unbiased estimator of population variance, which uses denominator n−1n-1.

The ECDF also determines empirical interval probabilities:

Pn((a,b])=Fn(b)−Fn(a).P_n((a,b])=F_n(b)-F_n(a).

The interval endpoints matter because the empirical distribution places positive probability on observed values.

Sampling properties

Assume throughout this section that the observations are independent and identically distributed with distribution function FF.

Pointwise expectation and variance

For each fixed xx, the count nFn(x)nF_n(x) has a binomial distribution:

nFn(x)∼Binomial⁡(n,F(x)).nF_n(x)\sim\operatorname{Binomial}(n,F(x)).

It follows that

E[Fn(x)]=F(x),Var⁡(Fn(x))=F(x)(1−F(x))n.\mathbb E[F_n(x)]=F(x), \qquad \operatorname{Var}(F_n(x)) = \frac{F(x)(1-F(x))}{n}.

Hence the ECDF is pointwise unbiased. Its standard error at xx is at most 1/(2n)1/(2\sqrt n), with the maximum attained when F(x)=1/2F(x)=1/2.

Values of the ECDF at different thresholds are dependent. Their covariance is

Cov⁡(Fn(x),Fn(y))=F(min⁡{x,y})−F(x)F(y)n.\operatorname{Cov}(F_n(x),F_n(y)) = \frac{F(\min\{x,y\})-F(x)F(y)}{n}.

Pointwise convergence and normal approximation

By the law of large numbers, for each fixed xx,

Fn(x)⟶F(x)almost surely.F_n(x)\longrightarrow F(x) \quad\text{almost surely}.

When 0<F(x)<10<F(x)<1, the central limit theorem gives

n(Fn(x)−F(x))→dN ⁣(0,F(x)(1−F(x))).\sqrt n\bigl(F_n(x)-F(x)\bigr) \xrightarrow{d} N\!\left(0,F(x)(1-F(x))\right).

This is a statement about a fixed threshold. It does not by itself provide simultaneous uncertainty control across all thresholds.

Uniform convergence and uncertainty bands

Glivenko–Cantelli theorem

The Glivenko–Cantelli theorem, established in 1933, strengthens pointwise convergence to uniform convergence:

sup⁡x∈R∣Fn(x)−F(x)∣⟶0almost surely.\sup_{x\in\mathbb R}|F_n(x)-F(x)| \longrightarrow 0 \quad\text{almost surely}.

Thus, with probability one, the largest discrepancy anywhere between the empirical and population distribution functions tends to zero. The result applies to continuous, discrete, and mixed distributions.

Dvoretzky–Kiefer–Wolfowitz inequality

The Dvoretzky–Kiefer–Wolfowitz inequality, with its sharp constant, provides a finite-sample bound:

Pr⁡ ⁣(sup⁡x∣Fn(x)−F(x)∣>ε)≤2e−2nε2,ε>0.\Pr\!\left( \sup_x|F_n(x)-F(x)|>\varepsilon \right) \leq 2e^{-2n\varepsilon^2}, \qquad \varepsilon>0.

For 0<α<10<\alpha<1, define

εn,α=log⁡(2/α)2n.\varepsilon_{n,\alpha} = \sqrt{\frac{\log(2/\alpha)}{2n}}.

With probability at least 1−α1-\alpha, the inequalities

max⁡{0,Fn(x)−εn,α}≤F(x)≤min⁡{1,Fn(x)+εn,α}\max\{0,F_n(x)-\varepsilon_{n,\alpha}\} \leq F(x)\leq \min\{1,F_n(x)+\varepsilon_{n,\alpha}\}

hold simultaneously for every xx. These are simultaneous confidence bands, rather than separate pointwise confidence intervals.

Empirical-process limit

At the process level, the centered and rescaled ECDF

αn(x)=n(Fn(x)−F(x))\alpha_n(x)=\sqrt n\bigl(F_n(x)-F(x)\bigr)

converges in distribution, in the standard empirical-process formulation, to

B(F(x)),B(F(x)),

where BB is a Brownian bridge: a mean-zero Gaussian process on [0,1][0,1] with covariance

Cov⁡(B(s),B(t))=min⁡(s,t)−st.\operatorname{Cov}(B(s),B(t)) = \min(s,t)-st.

This functional limit describes fluctuations of the entire estimated distribution and underlies many distributional tests.

Applications

Distribution comparison and goodness-of-fit testing

An ECDF plot displays location, spread, ties, and tail behavior without choosing histogram bins. Multiple ECDFs can be plotted together to compare samples.

The one-sample Kolmogorov–Smirnov test uses the maximum discrepancy from a specified distribution function F0F_0:

Dn=sup⁡x∣Fn(x)−F0(x)∣.D_n=\sup_x|F_n(x)-F_0(x)|.

For an independent sample under a fully specified continuous null distribution, its null distribution does not depend on the particular form of F0F_0. This distribution-free property generally does not hold unchanged for discrete null distributions or when parameters of F0F_0 are estimated from the tested sample.

The two-sample version compares two ECDFs, FnF_n and GmG_m, through

Dn,m=sup⁡x∣Fn(x)−Gm(x)∣.D_{n,m}=\sup_x|F_n(x)-G_m(x)|.

Quantile estimation

A natural empirical quantile function is the generalized inverse

Qn(p)=inf⁡{x:Fn(x)≥p},0<p≤1.Q_n(p)=\inf\{x:F_n(x)\geq p\}, \qquad 0<p\leq1.

It satisfies

Qn(p)=X(⌈np⌉).Q_n(p)=X_{(\lceil np\rceil)}.

Software also uses interpolated quantile definitions. These may return values between observations and need not coincide with the generalized inverse of the ECDF.

Bootstrap resampling

In the ordinary nonparametric bootstrap, observations are sampled independently with replacement from the empirical distribution. Conditional on the original data, each indexed observation receives probability 1/n1/n. Repeated observed values therefore receive probability equal to their relative frequency.

This makes the empirical distribution a substitute for the unknown population distribution when approximating the sampling behavior of many statistics. Bootstrap validity still depends on the statistic and the sampling assumptions.

Variants and limitations

A weighted empirical distribution function has the form

Fw(x)=∑i=1nwi1{Xi≤x}∑i=1nwi,F_w(x)= \frac{\sum_{i=1}^{n}w_i\mathbf{1}_{\{X_i\leq x\}}} {\sum_{i=1}^{n}w_i},

for nonnegative weights with positive total weight. Such functions arise with unequal sampling probabilities and importance weighting. Their uncertainty is not generally described by the ordinary unweighted formulas.

For vector-valued observations, a multivariate ECDF counts observations whose every coordinate is at most the corresponding threshold. It estimates a joint distribution, but its geometry and distributional theory differ from the univariate case.

Several limitations are important:

  • Dependence: serially or otherwise dependent observations invalidate the simple binomial variance formula and can change convergence rates and limiting distributions.
  • Sampling bias: the ECDF describes the observed sample; it does not automatically correct selection bias, measurement error, or missing data.
  • Tail resolution: it assigns no probability below the observed minimum or above the observed maximum. Reliable extrapolation beyond the sample range requires additional assumptions.
  • Censoring and truncation: applying the ordinary ECDF to incompletely observed values generally does not estimate the distribution of the underlying uncensored variable.
  • Lack of smoothness: the ECDF estimates cumulative probabilities directly, not a smooth density. Density estimation requires a different construction, often with additional smoothing choices.

Computationally, sorting the sample permits a compact representation by distinct observed values and their cumulative counts. Subsequent evaluations can use binary search, while empirical quantiles can be obtained directly from the sorted observations.