Variational autoencoder

A variational autoencoder (VAE) is a generative model in machine learning that combines a probabilistic model of observed data with a learned approximation to inference over hidden variables. Typically implemented using artificial neural networks, it learns both how to generate observations from a latent space and how to infer latent distributions from observations. Unlike a conventional autoencoder, it optimizes a probabilistic objective rather than reconstruction accuracy alone. (arxiv.org)

Origins and theoretical basis

Diederik P. Kingma and Max Welling introduced the framework in Auto-Encoding Variational Bayes, first submitted in December 2013. Closely related work by Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra appeared in January 2014 and was published at the International Conference on Machine Learning that year. These contributions made joint learning of generative and inference networks practical through stochastic gradient estimation. (arxiv.org)

The framework connects deep learning with Bayesian inference and variational inference. Its central problem is that posterior distributions over hidden variables are generally computationally intractable in nonlinear generative models. Instead of calculating the exact posterior, a VAE learns a tractable distribution that approximates it. (arxiv.org)

Generative model and encoder

For an observation $x$ and a latent random variable $z$, the generative model is

Here, $p(z)$ is a prior distribution, commonly a standard multivariate normal distribution, and $p_\theta(x\mid z)$ is a decoder likelihood with learned parameters $\theta$. Generation involves sampling $z$ from the prior and then sampling an observation from the decoder distribution. The decoder therefore specifies a distribution, not merely a reconstructed data point. (arxiv.org)

The encoder, or recognition network, produces $q_\phi(z\mid x)$, an approximation to the posterior $p_\theta(z\mid x)$. In a common implementation, it outputs the mean and diagonal covariance of a Gaussian. Both networks form an encoder–decoder architecture, but the encoder is an inference mechanism rather than part of the generative process itself. Sharing encoder parameters across observations is called amortized inference: one learned mapping replaces separate optimization for each input. (arxiv.org)

Variational training objective

Training maximizes the evidence lower bound (ELBO):

The first term is the expected log-likelihood of reconstructing the observation. The second is the Kullback–Leibler divergence between the approximate posterior and prior. Their relationship to the marginal likelihood is

Because KL divergence is nonnegative, the ELBO is a lower bound on log-likelihood. Maximizing it supports approximate maximum likelihood estimation while improving posterior approximation. (arxiv.org)

The negative ELBO serves as the loss function. Its reconstruction component depends on the observation model: a Gaussian with fixed variance yields a term proportional to mean squared error, whereas a Bernoulli distribution yields binary cross-entropy. The KL component acts as regularization, discouraging unrestricted information storage in the latent code. (arxiv.org)

Reparameterization and optimization

Sampling from a parameter-dependent distribution complicates gradient calculation. For a diagonal Gaussian encoder, the reparameterization trick expresses the sample as

where $\odot$ denotes elementwise multiplication. Randomness is isolated in parameter-independent noise, while $z$ remains differentiable with respect to the encoder parameters. This permits backpropagation through the sampled latent representation. (arxiv.org)

Expectations are estimated using samples, and encoder and decoder parameters are updated jointly with stochastic gradient methods. Mini-batches allow learning from large datasets. The Gaussian construction is especially convenient, but the broader approach applies to other distributions with suitable differentiable sampling representations; discrete latent variables require alternative estimators or relaxations. (proceedings.mlr.press)

Applications and limitations

VAEs support data generation, representation learning, visualization, and missing-data imputation. Their probabilistic encoder provides a distribution over possible latent explanations rather than a single deterministic encoding. Early experiments demonstrated generation, imputation, and visualization using deep latent Gaussian models. (proceedings.mlr.press)

A significant failure mode is posterior collapse, in which the approximate posterior closely matches the prior for some latent variables. Those variables then convey little information about individual observations. Analyses of linear VAEs show that collapse can arise from local maxima of the marginal likelihood, rather than solely from the variational bound. Related linear models connect VAEs to principal component analysis. (arxiv.org)

A restricted approximate posterior can also limit learned representations. Good reconstruction, useful latent features, and accurate density modeling are distinct objectives; optimizing one does not automatically optimize the others. (arxiv.org)

Important extensions

Conditional VAEs incorporate additional observations or labels into the generative and inference models. They model conditional distributions and can produce several plausible outputs for the same input, including structured image predictions. (papers.nips.cc)

β-VAEs multiply the KL term by a tunable coefficient $\beta$. Increasing this coefficient changes the balance between reconstruction accuracy and constraints on latent information, encouraging more factorized representations in the settings studied by their developers. (openreview.net)

Importance-weighted autoencoders use multiple latent samples and importance weighting to construct a likelihood bound at least as tight as the standard ELBO. Their original experiments showed improved held-out likelihoods and richer latent representations on density-estimation benchmarks. (arxiv.org)