The Laplace approximation is a method for approximating an integral whose integrand is strongly concentrated near a maximum. It replaces the logarithm of the integrand locally by a quadratic function, making the resulting integral Gaussian and therefore explicitly computable. As an asymptotic expansion, it becomes increasingly accurate when a parameter causes concentration around a sufficiently regular maximum. Named after Pierre-Simon Laplace, the method is widely used in numerical analysis and Bayesian inference, where it also provides a Gaussian approximation to a posterior distribution.
Basic principle
Consider the one-dimensional integral
where is a large positive parameter. Suppose has a unique dominant maximum at an interior point , with
Assume also that and are sufficiently smooth near , that , and that contributions away from the maximum are negligible at the relevant asymptotic scale.
A second-order Taylor expansion gives
The first-order term vanishes at the maximum. Replacing by this quadratic expression and by gives
Evaluating the Gaussian integral yields the leading-order formula
Here, means that the ratio of the two expressions tends to one. The approximation identifies both the dominant height, , and the effective width, proportional to .
The extension of the integration limits to the whole real line is an asymptotic step, not an assumption that the original domain is infinite. It is justified when the peak becomes narrow relative to its distance from the boundaries.
Why a Gaussian appears
The natural local coordinate is
For fixed ,
Thus, after rescaling around the maximum, the exponential has the shape of a normal distribution. Cubic and higher-order terms become progressively smaller on this scale.
With adequate smoothness and control of the remainder and tails, the integral admits a fuller expansion,
The coefficients depend on derivatives of and at the maximum. Under standard nondegenerate, fixed-dimensional conditions, the leading formula has relative error , in big-O notation. This rate requires stronger assumptions than merely the existence of a maximum and a second derivative.
Multivariate form
For , consider
Suppose has a unique dominant interior maximum , and define
where is the Hessian matrix. If is positive definite, then
The determinant accounts for the volume of the peak in all directions. The corresponding local multivariate normal distribution has mean and covariance
The eigenvalues of describe directional curvature: large eigenvalues imply narrow concentration, while small eigenvalues imply broad directions.
A frequently used version has no explicit concentration parameter. For
let maximize , and set
Then
Without an explicit limiting sequence, this is a local approximation rather than an automatic guarantee of asymptotic accuracy.
Bayesian inference
Posterior approximation
For data and a continuous parameter vector , the posterior distribution is
Define the unnormalized log-posterior
combining the likelihood and the prior distribution. Let
be a maximum a posteriori estimate, and let
If the mode is interior and is positive definite, the Laplace approximation is
The approximate covariance matrix is therefore the inverse negative Hessian of the log-posterior at the mode.
The Gaussian mean equals the selected posterior mode by construction. It need not equal the exact posterior mean, particularly when the posterior is skewed.
Marginal likelihood
The marginal likelihood, also called model evidence, is
Its Laplace approximation is
or, on the logarithmic scale,
This distinguishes peak density from integrated probability: a tall, narrow peak can have less total mass than a lower, broader one.
For regular models with fixed parameter dimension and sample size , the curvature typically grows proportionally to . Under suitable prior and regularity assumptions, this gives
where is the maximum likelihood estimate. Retaining the displayed terms and multiplying by leads to the Bayesian information criterion,
This relationship is asymptotic and does not make the Bayesian information criterion identical to the full Laplace approximation.
Expectations and predictions
A posterior expectation can be written as a ratio,
One approach evaluates the expectation under the fitted Gaussian. Another approximates the numerator and denominator separately. For positive, sufficiently smooth , the numerator can be treated using the modified log-integrand . The approaches need not give identical finite-sample answers.
Example: the factorial integral
For a positive integer , the gamma function gives the factorial representation
With ,
Set
Its unique maximum occurs at , with
Laplace's formula therefore gives
the leading term of Stirling's approximation. Higher-order Laplace terms recover further corrections.
Numerical implementation
A typical implementation consists of three main steps:
- Locate a mode. Apply numerical optimization to the log-integrand or log-posterior. Newton's method uses local gradient and curvature information.
- Compute local curvature. Evaluate the negative Hessian at the mode. Automatic differentiation can supply derivatives without finite-difference step-size choices.
- Evaluate Gaussian quantities. Use matrix factorizations to compute log determinants and solve linear systems.
If is a Cholesky factorization, then
Computing on the logarithmic scale helps prevent overflow and underflow. Solving systems involving , rather than explicitly forming its inverse, generally improves efficiency and numerical stability.
Successful optimization alone does not validate the approximation. A stationary point may be a saddle point, and even a genuine local maximum may not account for most of the integral.
Limitations and extensions
Multiple modes
A single Gaussian describes only one peak. If several well-separated, nondegenerate interior modes contribute substantially, their integral approximations can be added:
After normalization, this produces a Gaussian mixture whose weights reflect approximate integrated mass, not merely peak height. Missing an important mode remains a significant source of error.
Boundaries and constraints
If the maximum lies at a boundary, the ordinary interior formula generally does not apply. A stationary maximum at a smooth boundary may require a truncated Gaussian treatment. If the derivative normal to the boundary is nonzero, the concentration scale in that direction can be , rather than .
Transforming constrained parameters to unconstrained coordinates can help, but the transformed integration density must include the appropriate Jacobian determinant.
Degenerate curvature and skewness
If the Hessian is singular, the quadratic term does not determine concentration in every direction. For example,
Its width scales as , so the ordinary Gaussian formula fails.
Strong skewness, heavy tails, and weakly constrained directions can also make local curvature a poor description of global mass. In such cases, central behavior may be approximated reasonably while tail probabilities remain inaccurate.
Parameterization and dimension
A Gaussian approximation in one parameterization is generally not Gaussian after a nonlinear transformation. Density modes also depend on parameterization. Correct Jacobian handling preserves the exact integral, but finite-order Laplace approximations can differ beyond their retained asymptotic order.
Standard error statements usually assume fixed dimension as concentration increases. If the number of parameters grows with sample size, errors can accumulate, requiring separate high-dimensional analysis.
Related approximation methods
Markov chain Monte Carlo uses sampling rather than a single local quadratic expansion. Importance sampling can use a Laplace Gaussian as a proposal distribution, provided its support and tails adequately cover the target. Variational inference instead fits an approximating distribution through an optimization criterion; even when that distribution is Gaussian, its parameters need not match the Laplace mode and curvature.