Evidence Lower Bound · aiwiki.page

The evidence lower bound (ELBO) is an objective function used in variational inference to approximate otherwise intractable probability distributions. It is a lower bound on the logarithm of a model’s marginal likelihood, also called its evidence. Maximizing the ELBO makes an approximate distribution closer to the exact posterior in a specific divergence, while avoiding direct evaluation of the evidence. It is central to approximate Bayesian inference and to learning latent-variable models in machine learning. (cs.columbia.edu)

Mathematical definition

Let $x$ denote observed data, $z$ unobserved variables, and $\theta$ model parameters. A joint probability distribution is written

p_\theta(x,z)=p_\theta(x\mid z)p_\theta(z),

where $p_\theta(z)$ is the prior distribution and $p_\theta(x\mid z)$ supplies the likelihood. Computing the evidence requires marginalizing $z$ :

p_\theta(x)=\int p_\theta(x,z)\,dz.

For discrete variables, the integral is replaced by a sum. The exact posterior distribution is $p_\theta(z\mid x)=p_\theta(x,z)/p_\theta(x)$ . These quantities may be computationally difficult even when the joint density is evaluable. (cs.columbia.edu)

For an approximate distribution $q(z)$ , define

\mathcal L(q,\theta;x) =\mathbb E_q\!\left[ \log p_\theta(x,z)-\log q(z) \right].

Here $\mathbb E_q$ denotes an expectation under $q$ . The defining relation is

\log p_\theta(x) =\mathcal L(q,\theta;x) +D_{\mathrm{KL}}\!\left(q(z)\,\|\,p_\theta(z\mid x)\right).

Because Kullback–Leibler divergence is nonnegative, $\mathcal L\leq\log p_\theta(x)$ . Equality holds precisely when the approximate and exact posterior distributions agree almost everywhere. The identity assumes a positive, finite evidence and appropriately defined expectations; incompatible supports can instead yield an ELBO of negative infinity. (cs.columbia.edu)

Derivation and equivalent forms

The decomposition follows by substituting the posterior into the divergence:

\begin{aligned} D_{\mathrm{KL}}(q\|p_\theta(z\mid x)) &=\mathbb E_q[ \log q(z)-\log p_\theta(z\mid x)]\\ &=\mathbb E_q[ \log q(z)-\log p_\theta(x,z)] +\log p_\theta(x). \end{aligned}

Rearranging produces the ELBO identity. Alternatively, when $q$ covers the joint distribution’s support in $z$ , Jensen’s inequality applied to the concave logarithm gives

\log\mathbb E_q\!\left[ \frac{p_\theta(x,z)}{q(z)} \right] \geq \mathbb E_q\!\left[ \log\frac{p_\theta(x,z)}{q(z)} \right].

The left side is then $\log p_\theta(x)$ . (cs.columbia.edu)

Two equivalent expressions highlight different interpretations:

\mathcal L =\mathbb E_q[\log p_\theta(x,z)]+H(q)

and

\mathcal L =\mathbb E_q[\log p_\theta(x\mid z)] -D_{\mathrm{KL}}(q(z)\|p_\theta(z)).

The first combines expected log joint density with entropy; for continuous variables, $H(q)$ is differential entropy. The second balances data fit against departure from the prior. Its negative is commonly called variational free energy, although sign conventions differ across fields. (arxiv.org)

Optimization and expectation–maximization

With $\theta$ fixed, maximizing the ELBO over a family $\mathcal Q$ is equivalent to minimizing $D_{\mathrm{KL}}(q\|p_\theta(z\mid x))$ . Thus inference becomes mathematical optimization. A restricted family can leave a nonzero gap even at its global optimum. (jmlr.org)

A common choice is a factorized, or mean-field, family $q(z)=\prod_jq_j(z_j)$ . Coordinate-ascent updates optimize one factor while holding the others fixed:

\log q_j^*(z_j) =\mathbb E_{q_{-j}}[\log p_\theta(x,z)] +\text{constant}.

The constant normalizes the factor. Such updates improve the objective when performed exactly, but generally need not find its global maximum. (jmlr.org)

The expectation–maximization algorithm admits the same variational interpretation. Its E-step sets $q$ to the current exact posterior, making the bound tight. Its M-step increases the bound with respect to $\theta$ , thereby increasing or preserving the observed-data log likelihood. Variational EM instead restricts $q$ , allowing approximate E-steps when exact inference is unavailable. Increasing a loose bound alone does not guarantee an increase in the true evidence. (cs.toronto.edu)

Neural models and stochastic computation

In a variational autoencoder, an inference network supplies $q_\phi(z\mid x)$ , while a generative network defines $p_\theta(x\mid z)$ . Both parameter sets are optimized through the ELBO. The expected log likelihood is often called the reconstruction term, while the prior divergence acts as regularization. Minimizing the negative ELBO expresses the same optimization as a loss function. (arxiv.org)

The reparameterization trick allows differentiable sampling constructions, such as

z=\mu_\phi(x)+\sigma_\phi(x)\odot\epsilon, \qquad \epsilon\sim\mathcal N(0,I),

where $\odot$ denotes elementwise multiplication. This permits gradient estimates through sampled latent variables. Automatic differentiation and stochastic gradient methods support scalable optimization; stochastic variational inference also uses data subsampling to fit models with large datasets. (arxiv.org)

Interpretation and limitations

The exact ELBO is a bound, but a finite-sample stochastic estimate can exceed the log evidence because of sampling variability. Numerical optimization therefore distinguishes the mathematical objective from its noisy estimates. (arxiv.org)

The approximation also depends on the divergence’s direction. Minimizing $D_{\mathrm{KL}}(q\|p)$ can favor concentration in a posterior mode; factorization can suppress dependencies and underestimate uncertainty. A high ELBO is consequently not, by itself, proof of accurate posterior uncertainty. (cs.columbia.edu)

Importance-weighted objectives extend the construction by averaging several importance weights inside a logarithm. Their expected bounds are no looser than the single-sample ELBO and can better accommodate complex posteriors. They remain distinct objectives: tighter evidence bounds and improved optimization behavior are separate questions. (arxiv.org)