{
  "id": "laplace-approximation",
  "title": "Laplace Approximation",
  "lang": "en",
  "source_lang": "en",
  "summary": "Laplace approximation estimates sharply concentrated integrals and probability distributions by replacing a log-integrand near its maximum with a quadratic function.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "Laplace's approximation",
    "Laplace's method",
    "Method of Laplace"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/laplace-approximation/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/laplace-approximation/",
      "json": "https://aiwiki.page/zh/laplace-approximation.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The Laplace approximation is a method for approximating an integral whose integrand is strongly concentrated near a maximum. It replaces the logarithm of the integrand locally by a quadratic function, making the resulting integral Gaussian and therefore explicitly computable. As an asymptotic expansion, it becomes increasingly accurate when a parameter causes concentration around a sufficiently regular maximum. Named after Pierre-Simon Laplace, the method is widely used in numerical analysis and Bayesian inference, where it also provides a Gaussian approximation to a posterior distribution."
    },
    {
      "heading": "Basic principle",
      "text": "Consider the one-dimensional integral\n\n\\[I(n)=\\int_a^b e^{n f(x)}g(x)\\,dx,\\]\n\nwhere \\(n\\) is a large positive parameter. Suppose \\(f\\) has a unique dominant maximum at an interior point \\(x_0\\), with\n\n\\[f'(x_0)=0, \\qquad f''(x_0)<0.\\]\n\nAssume also that \\(f\\) and \\(g\\) are sufficiently smooth near \\(x_0\\), that \\(g(x_0)\\ne 0\\), and that contributions away from the maximum are negligible at the relevant asymptotic scale.\n\nA second-order Taylor expansion gives\n\n\\[f(x) = f(x_0) +\\frac12 f''(x_0)(x-x_0)^2 +\\text{higher-order terms}.\\]\n\nThe first-order term vanishes at the maximum. Replacing \\(f\\) by this quadratic expression and \\(g(x)\\) by \\(g(x_0)\\) gives\n\n\\[I(n) \\approx e^{nf(x_0)}g(x_0) \\int_{-\\infty}^{\\infty} \\exp\\!\\left[ -\\frac{n|f''(x_0)|}{2}(x-x_0)^2 \\right]dx.\\]\n\nEvaluating the Gaussian integral yields the leading-order formula\n\n\\[\\boxed{ I(n) \\sim e^{nf(x_0)}g(x_0) \\sqrt{\\frac{2\\pi}{n|f''(x_0)|}} } \\qquad (n\\to\\infty).\\]\n\nHere, \\(\\sim\\) means that the ratio of the two expressions tends to one. The approximation identifies both the dominant height, \\(e^{nf(x_0)}\\), and the effective width, proportional to \\(n^{-1/2}\\).\n\nThe extension of the integration limits to the whole real line is an asymptotic step, not an assumption that the original domain is infinite. It is justified when the peak becomes narrow relative to its distance from the boundaries."
    },
    {
      "heading": "Why a Gaussian appears",
      "text": "The natural local coordinate is\n\n\\[u=\\sqrt{n}(x-x_0).\\]\n\nFor fixed \\(u\\),\n\n\\[n\\left[ f\\!\\left(x_0+\\frac{u}{\\sqrt n}\\right)-f(x_0) \\right] \\longrightarrow \\frac12 f''(x_0)u^2.\\]\n\nThus, after rescaling around the maximum, the exponential has the shape of a normal distribution. Cubic and higher-order terms become progressively smaller on this scale.\n\nWith adequate smoothness and control of the remainder and tails, the integral admits a fuller expansion,\n\n\\[I(n) = e^{nf(x_0)}g(x_0) \\sqrt{\\frac{2\\pi}{n|f''(x_0)|}} \\left[ 1+\\frac{c_1}{n}+\\frac{c_2}{n^2}+\\cdots \\right].\\]\n\nThe coefficients depend on derivatives of \\(f\\) and \\(g\\) at the maximum. Under standard nondegenerate, fixed-dimensional conditions, the leading formula has relative error \\(O(n^{-1})\\), in big-O notation. This rate requires stronger assumptions than merely the existence of a maximum and a second derivative."
    },
    {
      "heading": "Multivariate form",
      "text": "For \\(x\\in\\mathbb R^d\\), consider\n\n\\[I(n)=\\int_D e^{nf(x)}g(x)\\,dx.\\]\n\nSuppose \\(f\\) has a unique dominant interior maximum \\(x_0\\), and define\n\n\\[H=-\\nabla^2 f(x_0),\\]\n\nwhere \\(\\nabla^2 f(x_0)\\) is the Hessian matrix. If \\(H\\) is positive definite, then\n\n\\[\\boxed{ I(n) \\sim e^{nf(x_0)}g(x_0) \\left(\\frac{2\\pi}{n}\\right)^{d/2} \\frac{1}{\\sqrt{\\det H}} }.\\]\n\nThe determinant accounts for the volume of the peak in all directions. The corresponding local multivariate normal distribution has mean \\(x_0\\) and covariance\n\n\\[(nH)^{-1}.\\]\n\nThe eigenvalues of \\(H\\) describe directional curvature: large eigenvalues imply narrow concentration, while small eigenvalues imply broad directions.\n\nA frequently used version has no explicit concentration parameter. For\n\n\\[Z=\\int e^{\\ell(x)}\\,dx,\\]\n\nlet \\(\\hat x\\) maximize \\(\\ell\\), and set\n\n\\[A=-\\nabla^2\\ell(\\hat x).\\]\n\nThen\n\n\\[Z\\approx e^{\\ell(\\hat x)} (2\\pi)^{d/2} (\\det A)^{-1/2}.\\]\n\nWithout an explicit limiting sequence, this is a local approximation rather than an automatic guarantee of asymptotic accuracy."
    },
    {
      "heading": "Bayesian inference",
      "text": "Posterior approximation\n\nFor data \\(y\\) and a continuous parameter vector \\(\\theta\\), the posterior distribution is\n\n\\[p(\\theta\\mid y) = \\frac{p(y\\mid\\theta)p(\\theta)}{p(y)}.\\]\n\nDefine the unnormalized log-posterior\n\n\\[\\ell(\\theta) = \\log p(y\\mid\\theta)+\\log p(\\theta),\\]\n\ncombining the likelihood and the prior distribution. Let\n\n\\[\\hat\\theta=\\operatorname*{arg\\,max}_\\theta\\ell(\\theta)\\]\n\nbe a maximum a posteriori estimate, and let\n\n\\[A=-\\nabla^2\\ell(\\hat\\theta).\\]\n\nIf the mode is interior and \\(A\\) is positive definite, the Laplace approximation is\n\n\\[p(\\theta\\mid y) \\approx \\mathcal N(\\hat\\theta,A^{-1}).\\]\n\nThe approximate covariance matrix is therefore the inverse negative Hessian of the log-posterior at the mode.\n\nThe Gaussian mean equals the selected posterior mode by construction. It need not equal the exact posterior mean, particularly when the posterior is skewed.\n\nMarginal likelihood\n\nThe marginal likelihood, also called model evidence, is\n\n\\[p(y)=\\int p(y\\mid\\theta)p(\\theta)\\,d\\theta.\\]\n\nIts Laplace approximation is\n\n\\[p(y) \\approx p(y\\mid\\hat\\theta)p(\\hat\\theta) (2\\pi)^{d/2} (\\det A)^{-1/2},\\]\n\nor, on the logarithmic scale,\n\n\\[\\log p(y) \\approx \\ell(\\hat\\theta) +\\frac d2\\log(2\\pi) -\\frac12\\log\\det A.\\]\n\nThis distinguishes peak density from integrated probability: a tall, narrow peak can have less total mass than a lower, broader one.\n\nFor regular models with fixed parameter dimension and sample size \\(n\\), the curvature typically grows proportionally to \\(n\\). Under suitable prior and regularity assumptions, this gives\n\n\\[\\log p(y) = \\log p(y\\mid\\hat\\theta_{\\mathrm{ML}}) -\\frac d2\\log n +O(1),\\]\n\nwhere \\(\\hat\\theta_{\\mathrm{ML}}\\) is the maximum likelihood estimate. Retaining the displayed terms and multiplying by \\(-2\\) leads to the Bayesian information criterion,\n\n\\[\\mathrm{BIC} = -2\\log p(y\\mid\\hat\\theta_{\\mathrm{ML}}) +d\\log n.\\]\n\nThis relationship is asymptotic and does not make the Bayesian information criterion identical to the full Laplace approximation.\n\nExpectations and predictions\n\nA posterior expectation can be written as a ratio,\n\n\\[\\mathbb E[h(\\theta)\\mid y] = \\frac{\\int h(\\theta)e^{\\ell(\\theta)}\\,d\\theta} {\\int e^{\\ell(\\theta)}\\,d\\theta}.\\]\n\nOne approach evaluates the expectation under the fitted Gaussian. Another approximates the numerator and denominator separately. For positive, sufficiently smooth \\(h\\), the numerator can be treated using the modified log-integrand \\(\\ell+\\log h\\). The approaches need not give identical finite-sample answers."
    },
    {
      "heading": "Example: the factorial integral",
      "text": "For a positive integer \\(n\\), the gamma function gives the factorial representation\n\n\\[n!=\\int_0^\\infty t^n e^{-t}\\,dt.\\]\n\nWith \\(t=nx\\),\n\n\\[n! = n^{n+1} \\int_0^\\infty \\exp\\!\\left[n(\\log x-x)\\right]dx.\\]\n\nSet\n\n\\[f(x)=\\log x-x.\\]\n\nIts unique maximum occurs at \\(x_0=1\\), with\n\n\\[f(1)=-1, \\qquad f''(1)=-1.\\]\n\nLaplace's formula therefore gives\n\n\\[n! \\sim n^{n+1}e^{-n}\\sqrt{\\frac{2\\pi}{n}} = \\sqrt{2\\pi n}\\left(\\frac ne\\right)^n,\\]\n\nthe leading term of Stirling's approximation. Higher-order Laplace terms recover further corrections."
    },
    {
      "heading": "Numerical implementation",
      "text": "A typical implementation consists of three main steps:\n\n- Locate a mode. Apply numerical optimization to the log-integrand or log-posterior. Newton's method uses local gradient and curvature information.\n- Compute local curvature. Evaluate the negative Hessian at the mode. Automatic differentiation can supply derivatives without finite-difference step-size choices.\n- Evaluate Gaussian quantities. Use matrix factorizations to compute log determinants and solve linear systems.\n\nIf \\(A=LL^{\\mathsf T}\\) is a Cholesky factorization, then\n\n\\[\\log\\det A=2\\sum_{i=1}^d\\log L_{ii}.\\]\n\nComputing on the logarithmic scale helps prevent overflow and underflow. Solving systems involving \\(A\\), rather than explicitly forming its inverse, generally improves efficiency and numerical stability.\n\nSuccessful optimization alone does not validate the approximation. A stationary point may be a saddle point, and even a genuine local maximum may not account for most of the integral."
    },
    {
      "heading": "Limitations and extensions",
      "text": "Multiple modes\n\nA single Gaussian describes only one peak. If several well-separated, nondegenerate interior modes contribute substantially, their integral approximations can be added:\n\n\\[Z\\approx \\sum_k e^{\\ell(\\hat x_k)} (2\\pi)^{d/2} (\\det A_k)^{-1/2}.\\]\n\nAfter normalization, this produces a Gaussian mixture whose weights reflect approximate integrated mass, not merely peak height. Missing an important mode remains a significant source of error.\n\nBoundaries and constraints\n\nIf the maximum lies at a boundary, the ordinary interior formula generally does not apply. A stationary maximum at a smooth boundary may require a truncated Gaussian treatment. If the derivative normal to the boundary is nonzero, the concentration scale in that direction can be \\(n^{-1}\\), rather than \\(n^{-1/2}\\).\n\nTransforming constrained parameters to unconstrained coordinates can help, but the transformed integration density must include the appropriate Jacobian determinant.\n\nDegenerate curvature and skewness\n\nIf the Hessian is singular, the quadratic term does not determine concentration in every direction. For example,\n\n\\[\\int_{-\\infty}^{\\infty}e^{-nx^4}\\,dx = n^{-1/4} \\int_{-\\infty}^{\\infty}e^{-u^4}\\,du.\\]\n\nIts width scales as \\(n^{-1/4}\\), so the ordinary Gaussian formula fails.\n\nStrong skewness, heavy tails, and weakly constrained directions can also make local curvature a poor description of global mass. In such cases, central behavior may be approximated reasonably while tail probabilities remain inaccurate.\n\nParameterization and dimension\n\nA Gaussian approximation in one parameterization is generally not Gaussian after a nonlinear transformation. Density modes also depend on parameterization. Correct Jacobian handling preserves the exact integral, but finite-order Laplace approximations can differ beyond their retained asymptotic order.\n\nStandard error statements usually assume fixed dimension as concentration increases. If the number of parameters grows with sample size, errors can accumulate, requiring separate high-dimensional analysis.\n\nRelated approximation methods\n\nMarkov chain Monte Carlo uses sampling rather than a single local quadratic expansion. Importance sampling can use a Laplace Gaussian as a proposal distribution, provided its support and tails adequately cover the target. Variational inference instead fits an approximating distribution through an optimization criterion; even when that distribution is Gaussian, its parameters need not match the Laplace mode and curvature."
    }
  ],
  "links": [
    "integral",
    "asymptotic-expansion",
    "pierre-simon-laplace",
    "numerical-analysis",
    "bayesian-inference",
    "taylor-series",
    "normal-distribution",
    "big-o-notation",
    "hessian-matrix",
    "positive-definite-matrix",
    "determinant",
    "multivariate-normal-distribution",
    "eigenvalues-and-eigenvectors",
    "posterior-distribution",
    "likelihood-function",
    "prior-distribution",
    "maximum-a-posteriori-estimation",
    "covariance-matrix",
    "marginal-likelihood",
    "maximum-likelihood-estimation",
    "expected-value",
    "gamma-function",
    "factorial",
    "mathematical-optimization",
    "newtons-method",
    "automatic-differentiation",
    "cholesky-decomposition",
    "numerical-stability",
    "gaussian-mixture-model",
    "markov-chain-monte-carlo",
    "importance-sampling",
    "variational-inference"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}