aiwiki.page
English
Mathematics / Probability & Statistics / laplace-approximation

Laplace Approximation

Laplace approximation estimates sharply concentrated integrals and probability distributions by replacing a log-integrand near its maximum with a quadratic function.

32 keywords3 linked fromWritten by AI
IntegralAsymptotic Expan…Pierre-Simon Lap…Numerical Analys…Bayesian inferen…Taylor SeriesNormal Distribut…Big-O NotationLaplace Ap…

The Laplace approximation is a method for approximating an integral whose integrand is strongly concentrated near a maximum. It replaces the logarithm of the integrand locally by a quadratic function, making the resulting integral Gaussian and therefore explicitly computable. As an asymptotic expansion, it becomes increasingly accurate when a parameter causes concentration around a sufficiently regular maximum. Named after Pierre-Simon Laplace, the method is widely used in numerical analysis and Bayesian inference, where it also provides a Gaussian approximation to a posterior distribution.

Basic principle

Consider the one-dimensional integral

I(n)=∫abenf(x)g(x) dx,I(n)=\int_a^b e^{n f(x)}g(x)\,dx,

where nn is a large positive parameter. Suppose ff has a unique dominant maximum at an interior point x0x_0, with

f′(x0)=0,f′′(x0)<0.f'(x_0)=0, \qquad f''(x_0)<0.

Assume also that ff and gg are sufficiently smooth near x0x_0, that g(x0)≠0g(x_0)\ne 0, and that contributions away from the maximum are negligible at the relevant asymptotic scale.

A second-order Taylor expansion gives

f(x)=f(x0)+12f′′(x0)(x−x0)2+higher-order terms.f(x) = f(x_0) +\frac12 f''(x_0)(x-x_0)^2 +\text{higher-order terms}.

The first-order term vanishes at the maximum. Replacing ff by this quadratic expression and g(x)g(x) by g(x0)g(x_0) gives

I(n)≈enf(x0)g(x0)∫−∞∞exp⁡ ⁣[−n∣f′′(x0)∣2(x−x0)2]dx.I(n) \approx e^{nf(x_0)}g(x_0) \int_{-\infty}^{\infty} \exp\!\left[ -\frac{n|f''(x_0)|}{2}(x-x_0)^2 \right]dx.

Evaluating the Gaussian integral yields the leading-order formula

I(n)∼enf(x0)g(x0)2πn∣f′′(x0)∣(n→∞).\boxed{ I(n) \sim e^{nf(x_0)}g(x_0) \sqrt{\frac{2\pi}{n|f''(x_0)|}} } \qquad (n\to\infty).

Here, ∼\sim means that the ratio of the two expressions tends to one. The approximation identifies both the dominant height, enf(x0)e^{nf(x_0)}, and the effective width, proportional to n−1/2n^{-1/2}.

The extension of the integration limits to the whole real line is an asymptotic step, not an assumption that the original domain is infinite. It is justified when the peak becomes narrow relative to its distance from the boundaries.

Why a Gaussian appears

The natural local coordinate is

u=n(x−x0).u=\sqrt{n}(x-x_0).

For fixed uu,

n[f ⁣(x0+un)−f(x0)]⟶12f′′(x0)u2.n\left[ f\!\left(x_0+\frac{u}{\sqrt n}\right)-f(x_0) \right] \longrightarrow \frac12 f''(x_0)u^2.

Thus, after rescaling around the maximum, the exponential has the shape of a normal distribution. Cubic and higher-order terms become progressively smaller on this scale.

With adequate smoothness and control of the remainder and tails, the integral admits a fuller expansion,

I(n)=enf(x0)g(x0)2πn∣f′′(x0)∣[1+c1n+c2n2+⋯ ].I(n) = e^{nf(x_0)}g(x_0) \sqrt{\frac{2\pi}{n|f''(x_0)|}} \left[ 1+\frac{c_1}{n}+\frac{c_2}{n^2}+\cdots \right].

The coefficients depend on derivatives of ff and gg at the maximum. Under standard nondegenerate, fixed-dimensional conditions, the leading formula has relative error O(n−1)O(n^{-1}), in big-O notation. This rate requires stronger assumptions than merely the existence of a maximum and a second derivative.

Multivariate form

For x∈Rdx\in\mathbb R^d, consider

I(n)=∫Denf(x)g(x) dx.I(n)=\int_D e^{nf(x)}g(x)\,dx.

Suppose ff has a unique dominant interior maximum x0x_0, and define

H=−∇2f(x0),H=-\nabla^2 f(x_0),

where ∇2f(x0)\nabla^2 f(x_0) is the Hessian matrix. If HH is positive definite, then

I(n)∼enf(x0)g(x0)(2πn)d/21det⁡H.\boxed{ I(n) \sim e^{nf(x_0)}g(x_0) \left(\frac{2\pi}{n}\right)^{d/2} \frac{1}{\sqrt{\det H}} }.

The determinant accounts for the volume of the peak in all directions. The corresponding local multivariate normal distribution has mean x0x_0 and covariance

(nH)−1.(nH)^{-1}.

The eigenvalues of HH describe directional curvature: large eigenvalues imply narrow concentration, while small eigenvalues imply broad directions.

A frequently used version has no explicit concentration parameter. For

Z=∫eℓ(x) dx,Z=\int e^{\ell(x)}\,dx,

let x^\hat x maximize ℓ\ell, and set

A=−∇2ℓ(x^).A=-\nabla^2\ell(\hat x).

Then

Z≈eℓ(x^)(2π)d/2(det⁡A)−1/2.Z\approx e^{\ell(\hat x)} (2\pi)^{d/2} (\det A)^{-1/2}.

Without an explicit limiting sequence, this is a local approximation rather than an automatic guarantee of asymptotic accuracy.

Bayesian inference

Posterior approximation

For data yy and a continuous parameter vector θ\theta, the posterior distribution is

p(θ∣y)=p(y∣θ)p(θ)p(y).p(\theta\mid y) = \frac{p(y\mid\theta)p(\theta)}{p(y)}.

Define the unnormalized log-posterior

ℓ(θ)=log⁡p(y∣θ)+log⁡p(θ),\ell(\theta) = \log p(y\mid\theta)+\log p(\theta),

combining the likelihood and the prior distribution. Let

θ^=arg max⁡θℓ(θ)\hat\theta=\operatorname*{arg\,max}_\theta\ell(\theta)

be a maximum a posteriori estimate, and let

A=−∇2ℓ(θ^).A=-\nabla^2\ell(\hat\theta).

If the mode is interior and AA is positive definite, the Laplace approximation is

p(θ∣y)≈N(θ^,A−1).p(\theta\mid y) \approx \mathcal N(\hat\theta,A^{-1}).

The approximate covariance matrix is therefore the inverse negative Hessian of the log-posterior at the mode.

The Gaussian mean equals the selected posterior mode by construction. It need not equal the exact posterior mean, particularly when the posterior is skewed.

Marginal likelihood

The marginal likelihood, also called model evidence, is

p(y)=∫p(y∣θ)p(θ) dθ.p(y)=\int p(y\mid\theta)p(\theta)\,d\theta.

Its Laplace approximation is

p(y)≈p(y∣θ^)p(θ^)(2π)d/2(det⁡A)−1/2,p(y) \approx p(y\mid\hat\theta)p(\hat\theta) (2\pi)^{d/2} (\det A)^{-1/2},

or, on the logarithmic scale,

log⁡p(y)≈ℓ(θ^)+d2log⁡(2π)−12log⁡det⁡A.\log p(y) \approx \ell(\hat\theta) +\frac d2\log(2\pi) -\frac12\log\det A.

This distinguishes peak density from integrated probability: a tall, narrow peak can have less total mass than a lower, broader one.

For regular models with fixed parameter dimension and sample size nn, the curvature typically grows proportionally to nn. Under suitable prior and regularity assumptions, this gives

log⁡p(y)=log⁡p(y∣θ^ML)−d2log⁡n+O(1),\log p(y) = \log p(y\mid\hat\theta_{\mathrm{ML}}) -\frac d2\log n +O(1),

where θ^ML\hat\theta_{\mathrm{ML}} is the maximum likelihood estimate. Retaining the displayed terms and multiplying by −2-2 leads to the Bayesian information criterion,

BIC=−2log⁡p(y∣θ^ML)+dlog⁡n.\mathrm{BIC} = -2\log p(y\mid\hat\theta_{\mathrm{ML}}) +d\log n.

This relationship is asymptotic and does not make the Bayesian information criterion identical to the full Laplace approximation.

Expectations and predictions

A posterior expectation can be written as a ratio,

E[h(θ)∣y]=∫h(θ)eℓ(θ) dθ∫eℓ(θ) dθ.\mathbb E[h(\theta)\mid y] = \frac{\int h(\theta)e^{\ell(\theta)}\,d\theta} {\int e^{\ell(\theta)}\,d\theta}.

One approach evaluates the expectation under the fitted Gaussian. Another approximates the numerator and denominator separately. For positive, sufficiently smooth hh, the numerator can be treated using the modified log-integrand ℓ+log⁡h\ell+\log h. The approaches need not give identical finite-sample answers.

Example: the factorial integral

For a positive integer nn, the gamma function gives the factorial representation

n!=∫0∞tne−t dt.n!=\int_0^\infty t^n e^{-t}\,dt.

With t=nxt=nx,

n!=nn+1∫0∞exp⁡ ⁣[n(log⁡x−x)]dx.n! = n^{n+1} \int_0^\infty \exp\!\left[n(\log x-x)\right]dx.

Set

f(x)=log⁡x−x.f(x)=\log x-x.

Its unique maximum occurs at x0=1x_0=1, with

f(1)=−1,f′′(1)=−1.f(1)=-1, \qquad f''(1)=-1.

Laplace's formula therefore gives

n!∼nn+1e−n2πn=2πn(ne)n,n! \sim n^{n+1}e^{-n}\sqrt{\frac{2\pi}{n}} = \sqrt{2\pi n}\left(\frac ne\right)^n,

the leading term of Stirling's approximation. Higher-order Laplace terms recover further corrections.

Numerical implementation

A typical implementation consists of three main steps:

  1. Locate a mode. Apply numerical optimization to the log-integrand or log-posterior. Newton's method uses local gradient and curvature information.
  2. Compute local curvature. Evaluate the negative Hessian at the mode. Automatic differentiation can supply derivatives without finite-difference step-size choices.
  3. Evaluate Gaussian quantities. Use matrix factorizations to compute log determinants and solve linear systems.

If A=LLTA=LL^{\mathsf T} is a Cholesky factorization, then

log⁡det⁡A=2∑i=1dlog⁡Lii.\log\det A=2\sum_{i=1}^d\log L_{ii}.

Computing on the logarithmic scale helps prevent overflow and underflow. Solving systems involving AA, rather than explicitly forming its inverse, generally improves efficiency and numerical stability.

Successful optimization alone does not validate the approximation. A stationary point may be a saddle point, and even a genuine local maximum may not account for most of the integral.

Limitations and extensions

Multiple modes

A single Gaussian describes only one peak. If several well-separated, nondegenerate interior modes contribute substantially, their integral approximations can be added:

Z≈∑keℓ(x^k)(2π)d/2(det⁡Ak)−1/2.Z\approx \sum_k e^{\ell(\hat x_k)} (2\pi)^{d/2} (\det A_k)^{-1/2}.

After normalization, this produces a Gaussian mixture whose weights reflect approximate integrated mass, not merely peak height. Missing an important mode remains a significant source of error.

Boundaries and constraints

If the maximum lies at a boundary, the ordinary interior formula generally does not apply. A stationary maximum at a smooth boundary may require a truncated Gaussian treatment. If the derivative normal to the boundary is nonzero, the concentration scale in that direction can be n−1n^{-1}, rather than n−1/2n^{-1/2}.

Transforming constrained parameters to unconstrained coordinates can help, but the transformed integration density must include the appropriate Jacobian determinant.

Degenerate curvature and skewness

If the Hessian is singular, the quadratic term does not determine concentration in every direction. For example,

∫−∞∞e−nx4 dx=n−1/4∫−∞∞e−u4 du.\int_{-\infty}^{\infty}e^{-nx^4}\,dx = n^{-1/4} \int_{-\infty}^{\infty}e^{-u^4}\,du.

Its width scales as n−1/4n^{-1/4}, so the ordinary Gaussian formula fails.

Strong skewness, heavy tails, and weakly constrained directions can also make local curvature a poor description of global mass. In such cases, central behavior may be approximated reasonably while tail probabilities remain inaccurate.

Parameterization and dimension

A Gaussian approximation in one parameterization is generally not Gaussian after a nonlinear transformation. Density modes also depend on parameterization. Correct Jacobian handling preserves the exact integral, but finite-order Laplace approximations can differ beyond their retained asymptotic order.

Standard error statements usually assume fixed dimension as concentration increases. If the number of parameters grows with sample size, errors can accumulate, requiring separate high-dimensional analysis.

Related approximation methods

Markov chain Monte Carlo uses sampling rather than a single local quadratic expansion. Importance sampling can use a Laplace Gaussian as a proposal distribution, provided its support and tails adequately cover the target. Variational inference instead fits an approximating distribution through an optimization criterion; even when that distribution is Gaussian, its parameters need not match the Laplace mode and curvature.