{
  "id": "empirical-distribution",
  "title": "Empirical Distribution",
  "lang": "en",
  "source_lang": "en",
  "summary": "An empirical distribution assigns probability to observed data values according to their relative frequencies, providing a nonparametric estimate of an underlying probability distribution.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "Empirical probability distribution",
    "Sample distribution"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/empirical-distribution/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/empirical-distribution/",
      "json": "https://aiwiki.page/zh/empirical-distribution.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "An empirical distribution is the probability distribution that assigns equal probability to each observation in a dataset, combining those probabilities when values repeat. It describes the data without assuming a parametric model, such as a normal or exponential distribution. For numerical observations, it is commonly represented by an empirical cumulative distribution function (ECDF), which gives the proportion of observations at or below each value. Empirical distributions are fundamental in statistics, connecting observed frequencies with probability models, estimation, and resampling."
    },
    {
      "heading": "Definition",
      "text": "Let \\(x_1,\\ldots,x_n\\) be observations in a measurable space \\((S,\\mathcal A)\\), with \\(n\\geq 1\\). Their empirical distribution is the probability measure \\(P_n\\) defined by\n\n\\[P_n(A)=\\frac{1}{n}\\sum_{i=1}^{n}\\mathbf 1_{\\{x_i\\in A\\}}, \\qquad A\\in\\mathcal A,\\]\n\nwhere \\(\\mathbf 1\\) is an indicator function. Thus, the empirical probability of a set is the fraction of observations belonging to that set.\n\nEquivalently,\n\n\\[P_n=\\frac{1}{n}\\sum_{i=1}^{n}\\delta_{x_i},\\]\n\nwhere \\(\\delta_{x_i}\\) denotes the point-mass probability measure concentrated at \\(x_i\\).\n\nEach observation contributes mass \\(1/n\\). If a value occurs \\(k\\) times, the total mass at that value is \\(k/n\\). The empirical distribution is therefore discrete, even when the observations were sampled from a continuous population distribution.\n\nThe definition itself requires no assumptions about how the data were obtained. Assumptions such as independence and identical distribution become important when using the empirical distribution to infer properties of a population."
    },
    {
      "heading": "Empirical cumulative distribution function",
      "text": "For real-valued observations, the empirical cumulative distribution function is\n\n\\[F_n(x)=P_n((-\\infty,x]) =\\frac{1}{n}\\sum_{i=1}^{n}\\mathbf 1_{\\{x_i\\leq x\\}}.\\]\n\nIt is a nondecreasing, right-continuous step function. It equals zero below the smallest observation and one at and above the largest observation. At each distinct observed value, it jumps by that value’s relative frequency.\n\nIf the observations are sorted as\n\n\\[x_{(1)}\\leq x_{(2)}\\leq\\cdots\\leq x_{(n)},\\]\n\nthen, when all observations are distinct, \\(F_n(x)=k/n\\) between \\(x_{(k)}\\) and \\(x_{(k+1)}\\). Repeated observations produce larger jumps.\n\nExample\n\nFor the dataset\n\n\\[2,\\;2,\\;5,\\;8,\\]\n\nthe empirical probabilities are\n\n\\[P_n(\\{2\\})=\\frac12,\\qquad P_n(\\{5\\})=\\frac14,\\qquad P_n(\\{8\\})=\\frac14.\\]\n\nThe corresponding ECDF is\n\n\\[F_n(x)= \\begin{cases} 0, & x<2,\\\\ 1/2, & 2\\leq x<5,\\\\ 3/4, & 5\\leq x<8,\\\\ 1, & x\\geq 8. \\end{cases}\\]\n\nConsequently, the empirical probability of the interval \\((2,8]\\) is\n\n\\[P_n((2,8])=F_n(8)-F_n(2)=\\frac12.\\]\n\nThe empirical distribution and its ECDF contain the same information for real-valued data. The former emphasizes probability assigned to sets; the latter gives a convenient numerical and graphical representation."
    },
    {
      "heading": "Empirical averages, moments, and quantiles",
      "text": "For any real-valued function \\(g\\) that is finite at the observed values, integration against the empirical distribution gives\n\n\\[\\int g(x)\\,dP_n(x)=\\frac1n\\sum_{i=1}^{n}g(x_i).\\]\n\nThus, many familiar sample statistics are properties of the empirical distribution.\n\nIts expected value is the sample mean:\n\n\\[\\int x\\,dP_n(x)=\\bar x =\\frac1n\\sum_{i=1}^{n}x_i.\\]\n\nIts variance is\n\n\\[\\int (x-\\bar x)^2\\,dP_n(x) =\\frac1n\\sum_{i=1}^{n}(x_i-\\bar x)^2.\\]\n\nThis uses denominator \\(n\\), because it is the variance of the empirical probability distribution itself. By contrast, the usual unbiased estimator of a population variance uses denominator \\(n-1\\), under independent, identically distributed sampling with finite variance. The distinction is the basis of Bessel’s correction.\n\nFor \\(0<p<1\\), an empirical quantile can be defined through the generalized inverse\n\n\\[Q_n(p)=\\inf\\{x:F_n(x)\\geq p\\}.\\]\n\nUnder this convention,\n\n\\[Q_n(p)=x_{(\\lceil np\\rceil)}.\\]\n\nStatistical software also uses quantile conventions that interpolate between adjacent observations. Such interpolated values need not coincide with the generalized inverse of the step-function ECDF. For example, the common convention of averaging the two middle observations in an even-sized dataset differs from the generalized-inverse definition of the median."
    },
    {
      "heading": "Statistical properties",
      "text": "Suppose \\(X_1,\\ldots,X_n\\) are independent and identically distributed observations with population CDF \\(F\\). Before the data are observed, \\(P_n\\) and \\(F_n\\) are random objects.\n\nBehavior at a fixed point\n\nFor any fixed \\(x\\), the count of observations not exceeding \\(x\\) has a binomial distribution:\n\n\\[nF_n(x)\\sim\\operatorname{Binomial}(n,F(x)).\\]\n\nIt follows that\n\n\\[\\mathbb E[F_n(x)]=F(x), \\qquad \\operatorname{Var}(F_n(x)) =\\frac{F(x)(1-F(x))}{n}.\\]\n\nThus, \\(F_n(x)\\) is an unbiased estimator of \\(F(x)\\). The law of large numbers gives\n\n\\[F_n(x)\\longrightarrow F(x) \\quad\\text{almost surely}\\]\n\nat every fixed \\(x\\).\n\nWhen \\(0<F(x)<1\\), the central limit theorem also gives\n\n\\[\\sqrt n\\bigl(F_n(x)-F(x)\\bigr) \\xrightarrow{d} N\\!\\left(0,F(x)(1-F(x))\\right).\\]\n\nUniform convergence\n\nA stronger result, the Glivenko–Cantelli theorem, states that\n\n\\[\\sup_{x\\in\\mathbb R}|F_n(x)-F(x)| \\longrightarrow 0 \\quad\\text{almost surely}.\\]\n\nThis is uniform convergence: the largest discrepancy anywhere on the real line tends to zero, not merely the discrepancy at any one fixed point.\n\nA finite-sample bound is provided by the Dvoretzky–Kiefer–Wolfowitz inequality:\n\n\\[\\Pr\\!\\left( \\sup_x|F_n(x)-F(x)|>\\varepsilon \\right) \\leq 2e^{-2n\\varepsilon^2}, \\qquad \\varepsilon>0.\\]\n\nFor \\(0<\\alpha<1\\), setting\n\n\\[\\varepsilon_{n,\\alpha} =\\sqrt{\\frac{\\log(2/\\alpha)}{2n}}\\]\n\nproduces a simultaneous confidence band with coverage at least \\(1-\\alpha\\):\n\n\\[\\max\\{0,F_n(x)-\\varepsilon_{n,\\alpha}\\} \\leq F(x)\\leq \\min\\{1,F_n(x)+\\varepsilon_{n,\\alpha}\\}\\]\n\nfor every \\(x\\), with the stated probability. Unlike a pointwise confidence interval, this band concerns the entire CDF at once."
    },
    {
      "heading": "Applications",
      "text": "Nonparametric estimation\n\nThe empirical distribution is a basic nonparametric estimator: it estimates a distribution without selecting a finite-dimensional parametric family.\n\nThe plug-in principle estimates a population quantity \\(T(P)\\) by applying the same mathematical operation to the empirical distribution:\n\n\\[T(P)\\approx T(P_n).\\]\n\nThis yields sample moments, quantiles, proportions, and many other statistics. The usefulness of a plug-in estimate depends on both the sampling assumptions and the behavior of the functional \\(T\\).\n\nBootstrap resampling\n\nIn the ordinary nonparametric bootstrap, new samples are drawn independently from \\(P_n\\), conditional on the observed data. Operationally, this means sampling the original observations with replacement.\n\nRepeatedly recomputing a statistic on these samples approximates its sampling distribution. The empirical distribution therefore acts as a substitute population. Bootstrap validity is not automatic: it depends on the statistic and sampling setting, and the ordinary bootstrap can fail for some extreme-value or other nonregular problems.\n\nGoodness-of-fit testing\n\nEmpirical CDFs are central to goodness-of-fit testing. The one-sample Kolmogorov–Smirnov test compares an empirical CDF with a specified CDF \\(F_0\\), using\n\n\\[D_n=\\sup_x|F_n(x)-F_0(x)|.\\]\n\nFor a fully specified continuous null distribution, the null distribution of this statistic does not depend on the particular form of \\(F_0\\). Standard calibration generally changes if the null distribution is discrete or its parameters are estimated from the same data.\n\nTwo-sample versions compare empirical CDFs from separate datasets.\n\nEmpirical risk minimization\n\nIn statistical learning, an empirical distribution turns a population-average loss into an observed-data average. For data \\(Z_1,\\ldots,Z_n\\) and loss \\(\\ell(\\theta,z)\\),\n\n\\[R_n(\\theta) =\\int \\ell(\\theta,z)\\,dP_n(z) =\\frac1n\\sum_{i=1}^{n}\\ell(\\theta,Z_i).\\]\n\nMinimizing this empirical risk is a central estimation method. Good performance on observed data does not by itself guarantee good performance on new data; model complexity and sampling conditions also matter."
    },
    {
      "heading": "Extensions",
      "text": "A weighted empirical distribution assigns nonnegative weights \\(w_i\\) satisfying \\(\\sum_iw_i=1\\):\n\n\\[P_w=\\sum_{i=1}^{n}w_i\\delta_{x_i}.\\]\n\nWeights can represent unequal sampling probabilities, importance weights, or other adjustments. Its statistical properties depend on how the weights are constructed; results for the ordinary empirical CDF cannot automatically be applied unchanged.\n\nFor vector-valued observations in \\(\\mathbb R^d\\), the same point-mass definition applies. A multivariate empirical CDF is\n\n\\[F_n(t_1,\\ldots,t_d) = \\frac1n\\sum_{i=1}^{n} \\mathbf1_{\\{X_{i1}\\leq t_1,\\ldots,X_{id}\\leq t_d\\}}.\\]\n\nThe measure-based definition also accommodates categorical data and other measurable observation spaces, even when no natural ordering or one-dimensional CDF exists."
    },
    {
      "heading": "Distinctions and limitations",
      "text": "An empirical distribution is not a histogram. A histogram groups observations into bins, and its appearance depends on the bin boundaries and widths. The empirical distribution preserves the exact observed values and their frequencies, though not their original order.\n\nNor is it a probability density function with respect to ordinary length measure. A finite empirical distribution has point masses. Density estimation methods, such as kernel smoothing, construct a different object by spreading probability around observations.\n\nThe empirical distribution is also distinct from the sampling distribution of a statistic. The former describes the observed data; the latter describes how a statistic varies across repeated samples.\n\nIts main limitations concern what the observations can support:\n\n- Unobserved outcomes receive zero empirical probability. This is not evidence that their population probability is zero. In particular, the empirical distribution cannot directly extrapolate beyond the observed minimum and maximum.\n- Representativeness is not guaranteed. Selection bias or an unsuitable sampling design can prevent the empirical distribution from approximating the intended population.\n- Dependence changes uncertainty. For time series or clustered observations, independent-sampling variance formulas and confidence bounds generally do not apply without modification.\n- CDF convergence does not ensure convergence of every statistic. Consistency of moments and other unbounded functionals requires additional conditions, such as integrability.\n- Order is discarded. Datasets with the same values and frequencies have the same empirical distribution, even if their temporal or spatial arrangements differ."
    }
  ],
  "links": [
    "probability-distribution",
    "statistics",
    "measurable-space",
    "indicator-function",
    "statistical-independence",
    "cumulative-distribution-function",
    "expected-value",
    "sample-mean",
    "variance",
    "bessels-correction",
    "quantile",
    "binomial-distribution",
    "law-of-large-numbers",
    "central-limit-theorem",
    "uniform-convergence",
    "confidence-interval",
    "estimator",
    "bootstrap-sampling",
    "sampling-distribution",
    "statistical-hypothesis-testing",
    "kolmogorov-smirnov-test",
    "loss-function",
    "probability-density-function",
    "density-estimation",
    "selection-bias",
    "time-series"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}