aiwiki.page
English
Mathematics / Probability & Statistics / effective-sample-size

Effective Sample Size

Effective sample size expresses the precision of an estimator as the number of independent, equally weighted observations needed to achieve comparable precision.

20 keywords6 linked from1 not yet writtenWritten by AI
StatisticsVarianceSample MeanQuantileBias of an Estim…Mean squared err…Cauchy–Schwarz i…Covariance matri…Effective…

Effective sample size (ESS) is a measure in statistics that translates the precision of an estimate into an equivalent number of observations under a specified reference sampling scheme. It is commonly used when observations are correlated, carry unequal weights, or arise from a complex sampling design. A dataset containing nn observations may provide substantially less information than nn independent, equally weighted observations, although some designs and dependence structures can produce an effective sample size greater than nn. ESS is not a single universal quantity: its definition depends on the estimator, the reference experiment, and the aspect of precision being measured.

Variance-based definition

For nn independent observations from the same distribution, each with variance σ2\sigma^2, the sample mean has variance

Var⁡(Xˉ)=σ2n.\operatorname{Var}(\bar X)=\frac{\sigma^2}{n}.

If an estimator μ^\hat\mu estimates the same population mean under a different sampling scheme, a natural variance-matching definition is

neff=σ2Var⁡(μ^).n_{\mathrm{eff}} = \frac{\sigma^2}{\operatorname{Var}(\hat\mu)}.

Thus, if an estimator has variance 0.01σ20.01\sigma^2, its effective sample size is 100 relative to independent sampling from that population.

More generally, suppose the reference estimator based on mm independent observations has variance approximately V/mV/m. Then

neff≈VVar⁡(θ^).n_{\mathrm{eff}}\approx \frac{V}{\operatorname{Var}(\hat\theta)}.

Here VV depends on the target quantity and the reference estimator. Consequently, the same observations can have different effective sample sizes for a mean, a proportion, a regression coefficient, or a quantile.

This interpretation concerns sampling precision, not every aspect of information. In particular, it does not account directly for bias. A highly precise estimate can still be systematically wrong, so a large ESS does not by itself imply small mean squared error.

Unequal observation weights

Kish effective sample size

Suppose observations X1,…,XnX_1,\ldots,X_n have a common mean, common variance σ2\sigma^2, and are independent. For fixed, nonnegative weights wiw_i, with at least one positive weight, consider the weighted mean

μ^w=∑i=1nwiXi∑i=1nwi.\hat\mu_w = \frac{\sum_{i=1}^{n}w_iX_i} {\sum_{i=1}^{n}w_i}.

Its variance is

Var⁡(μ^w)=σ2∑i=1nwi2(∑i=1nwi)2.\operatorname{Var}(\hat\mu_w) = \sigma^2 \frac{\sum_{i=1}^{n}w_i^2} {\left(\sum_{i=1}^{n}w_i\right)^2}.

Matching this to σ2/neff\sigma^2/n_{\mathrm{eff}} gives the Kish effective sample size:

neff=(∑i=1nwi)2∑i=1nwi2.n_{\mathrm{eff}} = \frac{\left(\sum_{i=1}^{n}w_i\right)^2} {\sum_{i=1}^{n}w_i^2}.

For normalized weights

ai=wi∑jwj,a_i=\frac{w_i}{\sum_jw_j},

the same expression becomes

neff=1∑iai2.n_{\mathrm{eff}}=\frac{1}{\sum_i a_i^2}.

Multiplying all weights by the same positive constant leaves ESS unchanged. Equal weights give neff=nn_{\mathrm{eff}}=n; a single nonzero weight gives neff=1n_{\mathrm{eff}}=1. For nonnegative weights,

1≤neff≤n,1\le n_{\mathrm{eff}}\le n,

with the upper bound following from the Cauchy–Schwarz inequality.

For example, normalized weights 0.7,0.1,0.1,0.10.7,0.1,0.1,0.1 give

neff=10.72+3(0.12)≈1.92.n_{\mathrm{eff}} = \frac{1}{0.7^2+3(0.1^2)} \approx1.92.

Although four observations are present, their weighted average has the same variance as approximately 1.92 independent, equally weighted observations under the stated assumptions.

Relationship to weight variability

Define the coefficient of variation of the weights using divisor nn:

CV⁡w2=n−1∑i(wi−wˉ)2wˉ2.\operatorname{CV}_w^2 = \frac{n^{-1}\sum_i(w_i-\bar w)^2}{\bar w^2}.

Then

neff=n1+CV⁡w2.n_{\mathrm{eff}} = \frac{n}{1+\operatorname{CV}_w^2}.

This identity expresses how concentration of weight reduces the effective number of contributing observations.

Its interpretation requires care. If weights depend on observed outcomes, observations have different variances, or observations are dependent, the weight-only formula need not equal a variance-matched ESS. For fixed normalized weights and covariance matrix Σ\Sigma, the general variance is

Var⁡(μ^w)=aTΣa,\operatorname{Var}(\hat\mu_w)=a^\mathsf{T}\Sigma a,

which cannot generally be inferred from weight dispersion alone.

Complex survey sampling

In survey sampling, precision depends on features such as clustering, stratification, unequal selection probabilities, and weighting adjustments.

The design effect for an estimator is

DEFF⁡=Var⁡design(θ^)Var⁡reference(θ^).\operatorname{DEFF} = \frac{\operatorname{Var}_{\mathrm{design}}(\hat\theta)} {\operatorname{Var}_{\mathrm{reference}}(\hat\theta)}.

When the reference variance for a nominal sample of size nn scales as 1/n1/n, a corresponding effective sample size is

neff=nDEFF⁡.n_{\mathrm{eff}} = \frac{n}{\operatorname{DEFF}}.

The reference is commonly simple random sampling, but whether sampling is with or without replacement, and whether a finite-population correction is included, matters. If the reference variance does not scale exactly as 1/n1/n, the expression is an approximation or a reporting convention rather than an exact variance match.

Clustered observations

Under a simple model with equal cluster size mm, equal individual variances, common within-cluster correlation ρ\rho, and independent clusters, the design effect for the overall mean is

DEFF⁡=1+(m−1)ρ.\operatorname{DEFF}=1+(m-1)\rho.

Hence

neff=n1+(m−1)ρ.n_{\mathrm{eff}} = \frac{n}{1+(m-1)\rho}.

For example, 1,000 observations arranged in independent clusters of 10, with within-cluster correlation 0.10.1, have

neff=10001.9≈526.n_{\mathrm{eff}} = \frac{1000}{1.9} \approx526.

The result does not extend unchanged to unequal cluster sizes or arbitrary correlation structures.

Clustering with positive within-cluster correlation usually reduces precision relative to independent sampling. Effective stratification can instead reduce variance below the reference variance, giving DEFF⁡<1\operatorname{DEFF}<1 and neff>nn_{\mathrm{eff}}>n. This is greater efficiency relative to the chosen reference, not an increase in the actual number of observations.

Correlated sequences

For observations from a stationary process, the variance of a mean depends on correlations across time. Let each observation have variance σ2\sigma^2, and define the lag-kk autocorrelation as

ρk=Cov⁡(Xt,Xt+k)σ2.\rho_k = \frac{\operatorname{Cov}(X_t,X_{t+k})}{\sigma^2}.

The exact variance of the mean of nn consecutive observations is

Var⁡(Xˉn)=σ2n[1+2∑k=1n−1(1−kn)ρk].\operatorname{Var}(\bar X_n) = \frac{\sigma^2}{n} \left[ 1+2\sum_{k=1}^{n-1} \left(1-\frac{k}{n}\right)\rho_k \right].

The corresponding effective sample size is therefore

neff=n1+2∑k=1n−1(1−kn)ρk.n_{\mathrm{eff}} = \frac{n}{ 1+2\sum_{k=1}^{n-1} \left(1-\frac{k}{n}\right)\rho_k }.

This expression shows that correlations at many lags can matter; adjusting only for the correlation between neighboring observations is generally insufficient.

When the autocorrelations are absolutely summable and an appropriate central limit theorem holds, the large-sample approximation is

neff≈nτint,τint=1+2∑k=1∞ρk.n_{\mathrm{eff}} \approx \frac{n}{\tau_{\mathrm{int}}}, \qquad \tau_{\mathrm{int}} = 1+2\sum_{k=1}^{\infty}\rho_k.

Here τint\tau_{\mathrm{int}} is the integrated autocorrelation time under a convention in which independent observations have τint=1\tau_{\mathrm{int}}=1.

Positive serial correlation generally reduces ESS. Negative correlation can make τint<1\tau_{\mathrm{int}}<1, producing ESS greater than the number of observations.

First-order autoregressive example

For a stationary first-order autoregressive model with ρk=ϕk\rho_k=\phi^k, where ∣ϕ∣<1|\phi|<1,

τint=1+ϕ1−ϕ,\tau_{\mathrm{int}} = \frac{1+\phi}{1-\phi},

so

neff≈n1−ϕ1+ϕ.n_{\mathrm{eff}} \approx n\frac{1-\phi}{1+\phi}.

If ϕ=0.8\phi=0.8, the effective sample size is approximately n/9n/9. This is a large-sample approximation; the finite-sample formula retains the factors 1−k/n1-k/n.

Markov chain Monte Carlo

In Markov chain Monte Carlo (MCMC), successive simulation draws are usually dependent. ESS describes the precision of a Monte Carlo estimate relative to independent draws from the target distribution.

For a target function ff, define

μ^f=1n∑t=1nf(Xt).\hat\mu_f=\frac{1}{n}\sum_{t=1}^{n}f(X_t).

Under suitable stationarity and limit-theorem conditions,

neff,f≈n1+2∑k=1∞ρk(f),n_{\mathrm{eff},f} \approx \frac{n}{ 1+2\sum_{k=1}^{\infty}\rho_k^{(f)} },

where ρk(f)\rho_k^{(f)} is the autocorrelation of the transformed sequence f(Xt)f(X_t). The Monte Carlo standard error is then approximately

MCSE⁡(μ^f)≈Var⁡π(f(X))neff,f.\operatorname{MCSE}(\hat\mu_f) \approx \sqrt{ \frac{\operatorname{Var}_{\pi}(f(X))} {n_{\mathrm{eff},f}} }.

This dependence on ff is essential. A chain may estimate a mean efficiently while estimating tail probabilities poorly.

Practical diagnostics distinguish bulk ESS, commonly calculated from rank-normalized draws, from tail ESS, which assesses sampling efficiency in distributional tails. These diagnostics need not equal the ESS for the original-scale mean.

Autocorrelations must be estimated from finite runs. Summing noisy estimates over many lags is unstable, so software uses truncation, paired-sequence methods, or other long-run variance estimators. Multi-chain methods can also account for disagreement between chains.

ESS alone does not establish convergence to the posterior distribution or another target. A chain trapped in one region can yield an apparently favorable autocorrelation-based ESS while failing to explore important regions. Convergence diagnostics and ESS address related but distinct questions.

Importance sampling and particle methods

In importance sampling, draws from a proposal distribution are assigned weights to represent a target distribution. For normalized weights w~i\tilde w_i, a common diagnostic is

neff,weights=1∑iw~i2.n_{\mathrm{eff,weights}} = \frac{1}{\sum_i\tilde w_i^2}.

This measures weight concentration and is algebraically identical to the Kish formula. It approaches nn for nearly uniform weights and approaches 1 when one weight dominates.

However, it is not generally the exact ESS for an importance-sampling estimate. For independent proposal draws Xi∼qX_i\sim q, normalized target density π\pi, and weights w(x)=π(x)/q(x)w(x)=\pi(x)/q(x), the self-normalized estimate of

μ=E⁡π[f(X)]\mu=\operatorname{E}_{\pi}[f(X)]

has, under suitable moment and support conditions, asymptotic variance

1nE⁡q[w(X)2(f(X)−μ)2].\frac{1}{n} \operatorname{E}_{q} \left[w(X)^2\bigl(f(X)-\mu\bigr)^2\right].

Its variance-matched ESS therefore depends on both the weights and the integrand ff. A weight-only diagnostic cannot capture that dependence.

In sequential Monte Carlo, weight-based ESS is often used to trigger resampling. Resampling can restore equal weights, but it may create repeated particles sharing the same ancestors. An ESS equal to the particle count immediately after resampling does not imply that all particles contain independent information.

Interpretation and limitations

An ESS value is meaningful only with its definition and reference specified.

  • It is estimator-specific. Means, quantiles, tail probabilities, and model parameters can have different effective sample sizes.
  • It is not necessarily an integer. ESS is an equivalence measure, not a literal count of retained observations.
  • It need not be bounded by nominal sample size. The nonnegative-weight Kish formula is bounded by nn, whereas variance-based definitions can exceed nn.
  • It does not correct systematic error. Selection bias, model misspecification, and incomplete simulation exploration are not removed by reporting ESS.
  • It is itself estimated. Short sequences, slowly decaying dependence, and unstable weights can make ESS estimates unreliable.
  • It is not automatically a degrees-of-freedom adjustment. Replacing nominal sample size with ESS in a test or confidence interval does not by itself establish the correct reference distribution or coverage.
  • Its standard interpretation can fail without finite variances. Heavy-tailed importance weights or dependence that prevents an ordinary central limit theorem may invalidate conventional variance-based ESS calculations.

A reproducible report identifies the estimator or diagnostic being assessed, the nominal sample size, the reference sampling scheme, and the method used to calculate ESS.