Autoencoder

An autoencoder is an artificial neural network trained to reconstruct its input through an intermediate representation, often called a code or latent representation. It is used in machine learning for compression, denoising, and representation learning. Rather than merely copying data, a useful autoencoder employs constraints that encourage it to capture recurring structure. Its reconstruction task normally requires no externally supplied labels: the input itself provides the target output. (deeplearningbook.org)

Architecture and training

An autoencoder follows an encoder–decoder architecture. The encoder transforms an input $x$ into a code $z=f_\theta(x)$; the decoder produces a reconstruction $\hat{x}=g_\phi(z)$. The parameters $\theta$ and $\phi$ are learned jointly. For example, an image containing hundreds of pixel values can be encoded into a smaller vector and reconstructed from that vector. The reconstruction approximates the original rather than necessarily reproducing it exactly. (deeplearningbook.org)

A typical objective is

Here, the $x_i$ are examples from the training data, $L$ is a reconstruction loss function, and $R$ represents an optional regularization term. Mean squared error is a common reconstruction criterion. Training generally uses backpropagation to calculate gradients and minibatch stochastic gradient descent to update parameters. (deeplearningbook.org)

Autoencoders are traditionally categorized as unsupervised learning methods because they can learn from unlabeled examples. Their reconstruction objective can also be described as self-supervised learning, since targets are derived from the data. These descriptions concern the source of supervision, not whether the network has an explicit numerical target during training. Denoising objectives similarly obtain targets from clean versions of corrupted inputs. (deeplearningbook.org)

Bottlenecks and dimensionality reduction

An undercomplete autoencoder has a code with fewer dimensions than its input. This bottleneck encourages the model to retain information useful for reconstruction while discarding other details. It provides a learned form of dimensionality reduction. An overcomplete autoencoder has a code at least as large as the input and therefore generally needs additional restrictions to avoid learning an uninformative copying operation. A small code alone does not guarantee useful features when the surrounding network has substantial capacity. (deeplearningbook.org)

There is a precise connection with principal component analysis (PCA). For centered data, an undercomplete autoencoder with linear encoder and decoder transformations, optimized globally under squared reconstruction error, recovers a principal subspace. Its code coordinates need not equal the conventional PCA scores: the learned basis may differ by an invertible transformation. Nonlinear activation functions allow autoencoders to represent relationships beyond a single linear subspace, although nonlinear models do not automatically outperform PCA. (arxiv.org)

Regularized variants

A sparse autoencoder penalizes hidden-unit activity so that relatively few units are strongly active for a given example. Sparsity concerns the activity pattern, rather than necessarily requiring a small total number of code dimensions. It can therefore constrain an overcomplete representation without imposing a narrow architectural bottleneck. (deeplearningbook.org)

A denoising autoencoder receives a corrupted input $\tilde{x}$ but is trained to reconstruct the corresponding clean input $x$. Corruption may involve additive noise or masking selected values. The task discourages straightforward copying and encourages representations that preserve predictable structure despite perturbations. Denoising autoencoders can also be stacked: the representation learned by one layer becomes the input for another, producing a hierarchy of features. (jmlr.org)

A contractive autoencoder adds a penalty on the sensitivity of the encoder to small input changes. Specifically, it penalizes the squared Frobenius norm of the encoder’s Jacobian matrix. The reconstruction term preserves distinctions needed to reproduce examples, while the contraction term encourages local stability. This creates a connection with manifold learning: the representation can remain sensitive to directions of meaningful variation while becoming less sensitive to other local directions. (icml.cc)

Variational autoencoders

A variational autoencoder (VAE) combines an encoder–decoder arrangement with a probabilistic latent-variable model. Instead of assigning only a deterministic code, its encoder specifies an approximate posterior distribution $q_\phi(z\mid x)$. The decoder defines an observation distribution $p_\theta(x\mid z)$, and a prior $p(z)$ specifies how latent variables are distributed before observing an input. (arxiv.org)

Training commonly maximizes an evidence lower bound,

The first term rewards reconstruction; the Kullback–Leibler divergence term constrains the approximate posterior relative to the prior. A reparameterization expresses latent sampling in a form suitable for gradient-based optimization. Sampling from the prior and decoding supports generation, whereas an ordinary reconstruction-only autoencoder does not by itself specify a comparable probability model over its codes. (arxiv.org)

Applications and limitations

Autoencoders provide feature extraction and compact representations for subsequent analysis or classification. Denoising models can reconstruct cleaner images from noisy observations. Historically, stacked autoencoders also served as a means of initializing deep networks before supervised training, with the reconstruction objectives learning intermediate representations layer by layer. Such features are optimized for their training objectives and are not guaranteed to correspond to human-interpretable categories. (jmlr.org)

In anomaly detection, an autoencoder may be trained on examples regarded as normal, with unusually large reconstruction errors used as anomaly scores. Thresholds determine which observations are flagged. However, unfamiliar or anomalous inputs can sometimes be reconstructed accurately, so low error does not establish normality. Reconstruction quality and anomaly-detection quality are therefore distinct properties; model capacity and behavior outside the training distribution affect whether the reconstruction score separates normal from abnormal examples. (tensorflow.org)