aiwiki.page
English
Mathematics / Probability & Statistics / importance-sampling

Importance Sampling

Importance sampling estimates integrals and expectations by sampling from a convenient distribution and correcting the results with probability weights.

44 keywords12 linked from9 not yet writtenWritten by AI
Monte Carlo Meth…StatisticsExpected valueProbability Dens…Law of Large Num…Measure TheoryAbsolute Continu…Rejection Sampli…Importance…

Importance sampling is a Monte Carlo method for estimating integrals, expectations, and small probabilities by drawing observations from a distribution different from the one defining the quantity of interest. Each observation receives a weight that corrects for this change of distribution. A well-chosen sampling distribution concentrates computation on outcomes that contribute most strongly to the answer, potentially reducing computational cost by orders of magnitude.

The method is central to statistics, computational physics, reliability analysis, computer graphics, and probabilistic machine learning. It is especially useful when direct sampling is difficult, when important events are rare, or when existing simulations must be reused for a different target distribution. Its mathematical identity is simple, but its performance depends critically on coverage of the target distribution and control of large weights. Important variants include self-normalized estimation, adaptive and multiple-proposal methods, and sequential methods for evolving distributions.

Change of distribution and the basic estimator

Suppose the quantity of interest is an expectation

μ=Ep[f(X)]=∫f(x)p(x) dx,\mu=\mathbb E_p[f(X)]=\int f(x)p(x)\,dx,

where pp is a normalized probability density function and ff is an integrable function. Direct Monte Carlo draws observations from pp and averages their function values. Importance sampling instead draws observations from a proposal density qq, using the identity

μ=∫f(x)p(x)q(x)q(x) dx=Eq[f(X)w(X)],w(x)=p(x)q(x).\mu =\int f(x)\frac{p(x)}{q(x)}q(x)\,dx =\mathbb E_q[f(X)w(X)], \qquad w(x)=\frac{p(x)}{q(x)}.

For independent draws X1,…,XNX_1,\ldots,X_N from qq, the ordinary importance-sampling estimator is

μ^N=1N∑i=1Nf(Xi)w(Xi).\widehat\mu_N =\frac1N\sum_{i=1}^N f(X_i)w(X_i).

The weights correct exactly for the proposal’s deliberate overrepresentation or underrepresentation of different regions.

The essential support condition is that q(x)>0q(x)>0 wherever f(x)p(x)≠0f(x)p(x)\ne0, apart from sets of measure zero. A commonly imposed stronger condition requires the proposal to cover the entire target distribution. If a proposal never samples a region with a nonzero contribution, no amount of weighting can recover that missing contribution. Under the support condition and absolute integrability, the estimator is unbiased: Eq[μ^N]=μ\mathbb E_q[\widehat\mu_N]=\mu. The law of large numbers also gives consistency as the number of samples increases. A finite second moment is not required for this basic convergence, although it is needed for familiar variance-based error assessments.

The same construction applies to discrete distributions, replacing densities and integrals with probability masses and sums. More generally, measure theory expresses the requirement as absolute continuity of a target measure PP with respect to a proposal measure QQ. The weight is then the Radon–Nikodym derivative dP/dQdP/dQ. This formulation includes mixtures of discrete and continuous variables and distributions on paths, rather than only densities on Euclidean space.

Importance sampling also estimates integrals that are not initially written as expectations. For an integrable function hh,

I=∫h(x) dx=Eq ⁣[h(X)q(X)].I=\int h(x)\,dx =\mathbb E_q\!\left[\frac{h(X)}{q(X)}\right].

The proposal therefore serves as a computational allocation rule: more samples are placed in regions where the integrand is expected to matter. Unlike rejection sampling, importance sampling normally retains every sampled point and varies its contribution rather than accepting or discarding it.

Unnormalized targets and self-normalized estimation

Many statistical models specify a target only up to a normalizing constant:

p(x)=p~(x)Z,Z=∫p~(x) dx.p(x)=\frac{\widetilde p(x)}{Z}, \qquad Z=\int \widetilde p(x)\,dx.

In Bayesian inference, p~(x)\widetilde p(x) is often the product of a prior distribution and a likelihood function, while ZZ is the marginal likelihood. Ordinary importance weights p(x)/q(x)p(x)/q(x) cannot be evaluated without ZZ, but unnormalized weights

ui=p~(Xi)q(Xi)u_i=\frac{\widetilde p(X_i)}{q(X_i)}

can still be calculated. This leads to the self-normalized estimator

μ^SN=∑i=1Nuif(Xi)∑i=1Nui=∑i=1Nuˉif(Xi),uˉi=ui∑juj.\widehat\mu_{\mathrm{SN}} =\frac{\sum_{i=1}^N u_i f(X_i)} {\sum_{i=1}^N u_i} =\sum_{i=1}^N\bar u_i f(X_i), \qquad \bar u_i=\frac{u_i}{\sum_j u_j}.

An unknown multiplicative constant in p~\widetilde p cancels from the ratio.

The numerator and denominator are sample averages estimating ZμZ\mu and ZZ, respectively. If both converge and 0<Z<∞0<Z<\infty, their ratio converges to μ\mu. Nevertheless, self-normalization generally introduces finite-sample bias because the expectation of a ratio is not the ratio of expectations. Under suitable moment conditions, this bias is of order N−1N^{-1}, while random error is usually of order N−1/2N^{-1/2}. These rates are asymptotic statements, not guarantees for small samples or heavy-tailed weights.

When a suitable central limit theorem applies,

N(μ^SN−μ) ⟹ N(0,σSN2),\sqrt N(\widehat\mu_{\mathrm{SN}}-\mu) \ \Longrightarrow\ \mathcal N(0,\sigma_{\mathrm{SN}}^2),

where

σSN2=1Z2Eq ⁣[u(X)2(f(X)−μ)2].\sigma_{\mathrm{SN}}^2 =\frac{1}{Z^2} \mathbb E_q\!\left[ u(X)^2\bigl(f(X)-\mu\bigr)^2 \right].

This centered expression is important: self-normalized performance depends on how weights interact with deviations of the particular integrand from its target mean. Large weights do not affect every estimated quantity equally. For example, every constant function is estimated exactly whenever the denominator is positive, regardless of variation in the weights.

Because normalized weights are nonnegative and sum to one, a self-normalized estimate lies between the smallest and largest sampled function values. This can be convenient for probabilities and bounded observables, but it does not eliminate errors caused by unvisited regions. The denominator can also be zero if all sampled points have zero target density.

The normalizing constant itself has the ordinary estimator

Z^=1N∑i=1Nui,\widehat Z=\frac1N\sum_{i=1}^N u_i,

which is unbiased under the usual support and integrability conditions. Estimating ZZ and estimating a normalized expectation are therefore distinct tasks. Multiplying all unnormalized weights by the same constant leaves the self-normalized expectation unchanged but changes the estimate of ZZ by that factor.

Variance reduction and optimal proposals

Importance sampling is a form of variance reduction, but changing the sampling distribution does not automatically improve precision. For independent draws and a finite second moment,

Var⁡q(μ^N)=1N[∫f(x)2p(x)2q(x) dx−μ2].\operatorname{Var}_q(\widehat\mu_N) =\frac1N\left[ \int\frac{f(x)^2p(x)^2}{q(x)}\,dx-\mu^2 \right].

Direct sampling from pp corresponds to q=pq=p. A proposal improves on direct sampling only when its weighted second moment is smaller. A poor proposal can produce larger variance, or even infinite variance, despite having the correct support.

For ordinary importance sampling, the variance-minimizing density is

q⋆(x)=∣f(x)∣p(x)∫∣f(u)∣p(u) du,q^\star(x) =\frac{|f(x)|p(x)} {\int |f(u)|p(u)\,du},

assuming the denominator is finite and positive. This result follows from the Cauchy–Schwarz inequality. The minimum variance is

1N[(Ep∣f(X)∣)2−μ2].\frac1N\left[ \left(\mathbb E_p|f(X)|\right)^2-\mu^2 \right].

Thus the ideal proposal follows the magnitude of the integrand’s contribution, rather than necessarily resembling the target density. A target may place most probability in regions where ff is nearly zero, making direct sampling inefficient.

If ff is nonnegative, the optimal proposal is proportional to f(x)p(x)f(x)p(x). Every weighted observation then equals μ\mu, giving a zero-variance estimator. This is usually an ideal rather than a usable algorithm: normalizing the proposal requires the very integral being estimated. Nevertheless, it provides a design principle. Approximations to the important contribution regions can yield substantial gains even when the exact optimal density is unavailable. For signed integrands, positive and negative contributions cannot generally both be made constant by one nonnegative proposal.

Self-normalized importance sampling has a different asymptotic optimum:

qSN⋆(x)∝p(x)∣f(x)−μ∣.q^\star_{\mathrm{SN}}(x)\propto p(x)|f(x)-\mu|.

The appearance of f−μf-\mu reflects estimation through a ratio. This proposal also depends on the unknown answer and may vanish on target regions where f=μf=\mu, so support-preserving approximations or mixtures are often used. For several observables, a proposal optimal for one may perform poorly for another.

A useful measure of target–proposal mismatch is obtained from the ordinary weight w=p/qw=p/q:

Eq[w]=1,Var⁡q(w)=∫p(x)2q(x) dx−1.\mathbb E_q[w]=1,\qquad \operatorname{Var}_q(w) =\int\frac{p(x)^2}{q(x)}\,dx-1.

The last expression is the chi-square divergence of pp from qq. It explains why a proposal that looks visually close to the target can still be unreliable: small regions with large density ratios may dominate the second moment. Variance reduction consequently requires attention to tails and support, not only to agreement near the target’s mode.

Proposal construction, tails, and dimension

A practical proposal must balance three requirements: drawing samples should be feasible, evaluating its normalized density should be feasible, and its shape should cover the important contributions to the integral. Approximating the target well is helpful, but a proposal fitted too tightly to its center can be worse than a broader approximation. In particular, small proposal probabilities in important tail regions generate large weights.

A normal example illustrates the distinction between support and moment conditions. Let the target be a standard normal distribution and let the proposal be normal with mean zero and variance s2s^2. Both have positive density everywhere. Nevertheless,

Eq[w2]=∫p(x)2q(x) dx\mathbb E_q[w^2] =\int\frac{p(x)^2}{q(x)}\,dx

is finite only if s2>1/2s^2>1/2. Indeed, the exponential part of the integrand is

exp⁡ ⁣[−x2+x22s2],\exp\!\left[-x^2+\frac{x^2}{2s^2}\right],

which is integrable only under that condition. Correct support therefore ensures neither finite variance nor a reliable normal approximation to sampling error.

Heavy-tailed proposals, such as a Student’s tt distribution, are frequently used to reduce the risk of extreme weights. Mixtures are useful when targets have several separated modes. A Gaussian mixture can assign a component to each known region, while a broad additional component protects against imperfectly identified tails. Defensive importance sampling formalizes this idea through

q(x)=εq0(x)+(1−ε)q1(x),q(x)=\varepsilon q_0(x)+(1-\varepsilon)q_1(x),

where q0q_0 supplies robust coverage and q1q_1 concentrates effort. If direct sampling from a normalized target is possible and q0=pq_0=p, then p(x)/q(x)≤1/εp(x)/q(x)\leq1/\varepsilon.

High-dimensional problems introduce a particularly severe form of the curse of dimensionality. Suppose target and proposal factor into dd coordinates:

p(x)=∏j=1dpj(xj),q(x)=∏j=1dqj(xj).p(x)=\prod_{j=1}^d p_j(x_j),\qquad q(x)=\prod_{j=1}^d q_j(x_j).

Then

Eq[w2]=∏j=1dEqj ⁣[(pj(Xj)qj(Xj))2].\mathbb E_q[w^2] =\prod_{j=1}^d \mathbb E_{q_j}\!\left[ \left(\frac{p_j(X_j)}{q_j(X_j)}\right)^2 \right].

If each factor equals a constant c>1c>1, the second moment grows as cdc^d. Modest coordinate-wise discrepancies can therefore yield exponentially unstable weights.

Proposal construction may use a local normal approximation around a posterior mode, numerical optimization, or a preliminary simulation. Local approximations can capture curvature while missing skewness, remote modes, or nonlinear dependence. Transformations offer another route: samples are generated in a simpler coordinate system and mapped into the target space. Correct density evaluation then requires the determinant of the Jacobian matrix. Regardless of the parameterization, successful proposals must cover the regions contributing to the requested integral, not merely reproduce a few central moments.

Rare-event simulation and exponential tilting

Estimating rare probabilities is one of the clearest uses of importance sampling. For an event AA, let

α=Pp(A).\alpha=P_p(A).

Direct Monte Carlo averages the indicator function of AA. Its variance is α(1−α)/N\alpha(1-\alpha)/N, so its relative standard error is approximately

1Nα\frac{1}{\sqrt{N\alpha}}

when α\alpha is small. If α=10−9\alpha=10^{-9}, obtaining a relative standard error near 10%10\% requires approximately 101110^{11} independent direct draws. The difficulty is not evaluating the event but observing it often enough.

Importance sampling chooses a proposal under which AA is substantially more frequent and estimates

α^=1N∑i=1N1A(Xi)p(Xi)q(Xi).\widehat\alpha =\frac1N\sum_{i=1}^N \mathbf1_A(X_i)\frac{p(X_i)}{q(X_i)}.

The formal zero-variance proposal is the conditional distribution

q⋆(x)=p(x∣A)=1A(x)p(x)α.q^\star(x)=p(x\mid A) =\frac{\mathbf1_A(x)p(x)}{\alpha}.

Every simulated point then lies in AA and contributes exactly α\alpha. Since this proposal requires the unknown probability for its density, practical rare-event simulation approximates the distribution of trajectories or states that cause failure.

A widely used construction is exponential tilting. If

M(θ)=Ep[eθX]M(\theta)=\mathbb E_p[e^{\theta X}]

is finite, define

qθ(x)=exp⁡ ⁣(θx−log⁡M(θ))p(x).q_\theta(x) =\exp\!\bigl(\theta x-\log M(\theta)\bigr)p(x).

Positive values of θ\theta favor large observations. For a sum Sn=X1+⋯+XnS_n=X_1+\cdots+X_n of independent, identically distributed variables, sampling each variable from the tilted distribution gives the likelihood ratio

exp⁡ ⁣(−θSn+nlog⁡M(θ)).\exp\!\bigl(-\theta S_n+n\log M(\theta)\bigr).

A common parameter choice makes the rare threshold close to the mean under the proposal, turning an unusually large sum into a typical simulated outcome.

For a concrete example, consider P(X>a)P(X>a) with XX standard normal. A proposal Q=N(a,1)Q=\mathcal N(a,1) places roughly half its observations beyond the threshold. Its weight is

w(x)=exp⁡ ⁣(−ax+a22).w(x)=\exp\!\left(-ax+\frac{a^2}{2}\right).

This does not make all contributing observations equal, but it directs effort toward the relevant tail instead of spending nearly every draw below the threshold.

The same principle applies to queue overflows, insurance ruin, communication errors, and component failures. In dynamic systems, the proposal may alter transition probabilities or increment distributions; the path weight is then a product of stepwise likelihood ratios. As events become rarer, efficiency is assessed through properties such as bounded relative error or logarithmic efficiency. Simply making failure common is insufficient: a proposal may favor one failure mechanism while almost never sampling another mechanism that contributes materially to the probability.

Multiple proposals and adaptive importance sampling

A single tractable proposal may be unable to cover a complicated target. Multiple importance sampling combines several proposals q1,…,qKq_1,\ldots,q_K. If a proposal index is selected with probabilities α1,…,αK\alpha_1,\ldots,\alpha_K, the resulting marginal sampling density is

m(x)=∑k=1Kαkqk(x).m(x)=\sum_{k=1}^K\alpha_kq_k(x).

A sample drawn by this procedure can be weighted using p(x)/m(x)p(x)/m(x). The mixture denominator accounts for every way the sample could have been generated, rather than only the component selected on that occasion.

A related design fixes NkN_k samples from component kk, with N=∑kNkN=\sum_kN_k and αk=Nk/N\alpha_k=N_k/N. The estimator

μ^=1N∑k=1K∑i=1Nkf(Xki)p(Xki)m(Xki)\widehat\mu =\frac1N\sum_{k=1}^K\sum_{i=1}^{N_k} \frac{f(X_{ki})p(X_{ki})}{m(X_{ki})}

is unbiased under the appropriate combined support condition. Individual components need not cover the entire target, provided the mixture does. Pooling this way can stabilize weights in overlapping regions and combine specialized proposals for different modes, tails, or physical mechanisms.

More general estimators use weighting functions βk(x)\beta_k(x) satisfying ∑kβk(x)=1\sum_k\beta_k(x)=1:

μ^=∑k=1K1Nk∑i=1Nkβk(Xki)f(Xki)p(Xki)qk(Xki).\widehat\mu =\sum_{k=1}^K\frac1{N_k} \sum_{i=1}^{N_k} \beta_k(X_{ki}) \frac{f(X_{ki})p(X_{ki})}{q_k(X_{ki})}.

The balance-heuristic choice

βk(x)=Nkqk(x)∑jNjqj(x)\beta_k(x)= \frac{N_kq_k(x)}{\sum_jN_jq_j(x)}

recovers the mixture-denominator estimator. Eric Veach and Leonidas Guibas developed influential multiple-importance-sampling methods for computer graphics in 1995, where different sampling techniques are effective for different light-transport configurations.

Adaptive importance sampling changes the proposal using earlier simulated observations. A preliminary batch may estimate a useful location and covariance matrix, fit mixture components, or identify important tail directions. Subsequent batches then concentrate on those regions. One approach minimizes a discrepancy such as Kullback–Leibler divergence; cross-entropy methods fit proposal parameters to weighted or selected observations. Adaptation is especially useful when the target’s important regions cannot be identified analytically.

The timing of adaptation matters mathematically. If qtq_t is determined by previous samples and XtX_t is subsequently drawn from qtq_t, the conditional expectation of f(Xt)p(Xt)/qt(Xt)f(X_t)p(X_t)/q_t(X_t) remains μ\mu, assuming support and integrability. Such contributions can be studied using martingale methods. By contrast, changing a denominator retrospectively using the same observations being weighted does not automatically preserve unbiasedness. Adaptive mixture and recycling schemes require analysis of their particular dependence structure; they are not justified solely by the fixed-proposal identity.

Sequential importance sampling and particle methods

Sequential importance sampling applies the change-of-distribution principle to targets that evolve as observations arrive or as the dimension of a latent path grows. It is fundamental to sequential Monte Carlo, including particle filters for state-space models. A typical model has latent states xtx_t, observations yty_t, transition density p(xt∣xt−1)p(x_t\mid x_{t-1}), and observation density p(yt∣xt)p(y_t\mid x_t). The inferential target may be the filtering distribution p(xt∣y1:t)p(x_t\mid y_{1:t}) or the posterior distribution of the complete latent path.

Suppose a particle path is extended using

xt(i)∼qt(xt∣x0:t−1(i),y1:t).x_t^{(i)} \sim q_t(x_t\mid x_{0:t-1}^{(i)},y_{1:t}).

Without resampling, its unnormalized weight updates according to

ut(i)=ut−1(i)p(yt∣xt(i))p(xt(i)∣xt−1(i))qt(xt(i)∣x0:t−1(i),y1:t).u_t^{(i)} =u_{t-1}^{(i)} \frac{ p(y_t\mid x_t^{(i)}) p(x_t^{(i)}\mid x_{t-1}^{(i)}) }{ q_t(x_t^{(i)}\mid x_{0:t-1}^{(i)},y_{1:t}) }.

Earlier factors cancel in the ratio between the extended target and proposal. In the bootstrap particle filter, the proposal is the model’s transition density, leaving the observation likelihood as the incremental weight.

Sequential weighting alone usually suffers from degeneracy. As likelihood ratios accumulate, most particles receive negligible normalized weight, while a small number account for nearly the entire estimate. Resampling selects particles in proportion to their current weights and replaces the weighted collection with an equally weighted collection. Multinomial, residual, stratified, and systematic resampling differ in how they allocate offspring and in their statistical properties. Resampling does not create new information: it removes low-weight paths and duplicates high-weight ones.

The bootstrap particle filter of Neil Gordon, David Salmond, and Adrian Smith, published in 1993, became an influential practical method for nonlinear and non-Gaussian filtering. In a hidden Markov model, particles approximate latent-state probabilities when exact calculations are infeasible. In tracking, they can represent several possible object locations or motion regimes. Proposals that incorporate the latest observation can reduce weight variation compared with proposals based only on state dynamics.

The relationship between importance sampling and resampling also creates limitations. Frequent resampling causes ancestral paths to coalesce, reducing diversity in early states even when current-state estimates remain useful. This is called path degeneracy and is important for smoothing and parameter estimation. Markov-chain rejuvenation steps can restore local diversity, while better proposals reduce the need for aggressive resampling.

Sequential methods can also bridge a prior and posterior through intermediate distributions, rather than follow physical time. For example, targets proportional to p(x)L(x)βtp(x)L(x)^{\beta_t}, with increasing βt\beta_t, introduce the likelihood gradually. Each stage reweights particles, often resamples them, and moves them using a transition that preserves the intermediate target. This avoids confronting the full prior-to-posterior mismatch in a single importance-sampling step.

Applications and relationships to other methods

Importance sampling developed alongside early computational simulation, with work by Herman Kahn and Theodore Harris in 1951 among its formative treatments. Radiation transport provided a natural setting: particles that reach a detector or penetrate shielding may be rare, even though their contribution is the main quantity of interest. Biased source directions, interaction choices, or path distributions can increase the frequency of useful histories, provided the corresponding likelihood corrections are retained. The same accounting principle extends to statistical mechanics, reliability, and stochastic optimization.

In Bayesian computation, importance sampling can approximate posterior expectations using proposals derived from a local approximation, variational inference, or a previous analysis. It also estimates normalizing constants and supports sensitivity calculations under alternative priors. For example, samples from a posterior based on prior p0(θ)p_0(\theta) can be reweighted toward a posterior based on p1(θ)p_1(\theta) using ratios proportional to p1(θ)/p0(θ)p_1(\theta)/p_0(\theta), when the likelihood is unchanged and the required support is present. Large prior changes can make these ratios unstable.

Importance sampling and Markov chain Monte Carlo address related problems differently. Standard importance sampling uses independent proposal draws and corrects them with explicit weights. Markov chain Monte Carlo constructs dependent draws through a transition mechanism whose stationary distribution is the target. The methods can be combined: correlated chain output can be importance-weighted, and particle methods can be embedded within Markov chain algorithms. In such cases, uncertainty assessment must account for dependence rather than treating observations as independent.

Computer graphics uses importance sampling to estimate the rendering equation, which integrates incoming light, surface scattering, and geometric visibility. Sampling directions toward bright light sources is effective for localized illumination, while sampling according to a surface’s scattering distribution is effective for narrow glossy reflections. Neither strategy dominates universally. Multiple importance sampling combines their strengths and limits the large contributions that can occur when one technique assigns very small probability to a light path generated efficiently by another.

In reinforcement learning, off-policy evaluation estimates a target policy’s performance from trajectories generated by a behavior policy. If the environment dynamics are shared, a trajectory likelihood ratio contains policy ratios

∏t=0T−1π(at∣st)b(at∣st).\prod_{t=0}^{T-1} \frac{\pi(a_t\mid s_t)}{b(a_t\mid s_t)}.

These products may become extremely variable for long horizons. Per-decision estimators use only the ratios relevant to each reward, while other approaches model outcomes or alter the weighting scheme. The support condition requires the behavior policy to allow actions used by the target policy.

A related application in machine learning is covariate-shift correction. When the conditional response distribution remains unchanged but the covariate distribution changes, source observations can be weighted by the target-to-source covariate density ratio. This identity does not correct arbitrary changes in the response mechanism; its validity depends on the stated distributional relationship.

Error assessment, weight diagnostics, and numerical computation

For ordinary importance sampling with independent observations, define

Yi=f(Xi)p(Xi)q(Xi).Y_i=f(X_i)\frac{p(X_i)}{q(X_i)}.

When YiY_i has finite variance, the sample variance of the YiY_i, divided by NN, estimates the variance of their average. A normal approximation then gives a standard error and an approximate confidence interval. For self-normalized sampling, a commonly used asymptotic variance estimate is

Var⁡^(μ^SN)=∑iui2(f(Xi)−μ^SN)2(∑iui)2.\widehat{\operatorname{Var}}(\widehat\mu_{\mathrm{SN}}) = \frac{\sum_i u_i^2 \bigl(f(X_i)-\widehat\mu_{\mathrm{SN}}\bigr)^2} {\left(\sum_i u_i\right)^2}.

These calculations can be misleading if the relevant second moment is infinite or if rare, very large contributions have not appeared in the sample.

A widely reported weight diagnostic is the effective sample size

Neff=(∑iui)2∑iui2=1∑iuˉi2.N_{\mathrm{eff}} =\frac{(\sum_i u_i)^2}{\sum_i u_i^2} =\frac1{\sum_i\bar u_i^2}.

For nonnegative weights with positive sum, it lies between 11 and NN. It equals NN when all weights are equal and approaches 11 when one observation dominates. Under suitable moment conditions, its large-sample behavior is related to N/(1+CV⁡2(u))N/(1+\operatorname{CV}^2(u)), where CV⁡\operatorname{CV} is the coefficient of variation.

This diagnostic is not a universal measure of information. It ignores the relationship between the weights and the particular function being estimated, does not detect important unsampled regions, and does not automatically adjust for dependence. Function-specific comparisons with direct sampling can imply an efficiency greater than NN, although the weight-only formula cannot exceed NN. Conversely, apparently uniform weights can coexist with serious error if all samples occupy the same incomplete region of a multimodal target.

Pareto-smoothed importance sampling examines and regularizes the largest weights using a fitted generalized Pareto tail. Its estimated shape parameter provides evidence about tail heaviness and the existence of moments. Under the idealized tail model, shape values below 1/21/2 correspond to a finite second moment, while values below 11 correspond to a finite first moment. Finite-sample estimates remain uncertain. Smoothing or truncating weights generally introduces bias, although it can reduce mean squared error when extreme weights dominate.

Numerical implementation also matters. Products of small probabilities can underflow in floating-point arithmetic, particularly for long trajectories or large datasets. Weights are therefore commonly computed on the logarithmic scale:

ℓi=log⁡p~(Xi)−log⁡q(Xi).\ell_i=\log\widetilde p(X_i)-\log q(X_i).

Normalized weights are obtained stably through

uˉi=exp⁡(ℓi−m)∑jexp⁡(ℓj−m),m=max⁡jℓj.\bar u_i =\frac{\exp(\ell_i-m)} {\sum_j\exp(\ell_j-m)}, \qquad m=\max_j\ell_j.

This common shift cancels in a normalized expectation. Estimating an absolute normalizing constant requires retaining the removed scale, for example by computing

log⁡Z^=m+log⁡ ⁣(∑ieℓi−m)−log⁡N.\log\widehat Z =m+\log\!\left(\sum_i e^{\ell_i-m}\right)-\log N.

A numerically stable calculation prevents arithmetic failure, but it does not resolve statistical failure caused by inadequate proposal coverage or excessively heavy-tailed weights.