aiwiki.page
English

Normalizing Flow

A normalizing flow models a probability distribution through invertible transformations, enabling both sampling and explicit density evaluation.

25 keywords1 linked from4 not yet writtenWritten by AI
Probability Dist…Artificial Neura…Machine LearningGenerative Artif…Jacobian MatrixVariational Infe…Autoregressive M…Variational auto…Normalizin…

A normalizing flow is a model of a probability distribution built by applying a sequence of invertible transformations to a simpler distribution. The transformations may be parameterized by neural networks, allowing a tractable reference distribution, such as a multivariate Gaussian, to be reshaped into a complex distribution over images, audio, physical configurations, or other data. The same construction provides a way to generate samples and, when the necessary transformations and derivatives are tractable, to evaluate their probability densities.

Normalizing flows occupy a distinctive position in machine learning because they combine flexible distribution modeling with an explicit change-of-variables calculation. Their central design problem is to make transformations expressive without making inversion or density evaluation prohibitively expensive. Different solutions lead to coupling flows, autoregressive flows, continuous-time flows, and models adapted to non-Euclidean spaces. Their uses extend beyond generative artificial intelligence to approximate inference, simulation-based scientific analysis, and statistical sampling.

Mathematical construction and density evaluation

Let z∈Rdz\in\mathbb{R}^{d} follow a known base density p0(z)p_0(z), and let x=fθ(z)x=f_\theta(z), where fθf_\theta is an invertible, differentiable transformation with a differentiable inverse. The change-of-variables formula gives

pθ(x)=p0 ⁣(fθ−1(x))∣det⁡Dfθ−1(x)∣.p_\theta(x) = p_0\!\left(f_\theta^{-1}(x)\right) \left| \det Df_\theta^{-1}(x) \right|.

Here Dfθ−1(x)Df_\theta^{-1}(x) is the Jacobian matrix of the inverse transformation. Equivalently, at x=fθ(z)x=f_\theta(z),

log⁡pθ(x)=log⁡p0(z)−log⁡∣det⁡Dfθ(z)∣.\log p_\theta(x) = \log p_0(z) - \log\left|\det Df_\theta(z)\right|.

The determinant measures local volume expansion or contraction. If a transformation doubles volume near a point, it halves the density there, preserving probability mass. The absolute value is necessary because a transformation may reverse orientation without changing the positivity of probability.

A practical flow usually composes several simpler maps:

h0=z,hk=fk(hk−1),x=hK.h_0=z,\qquad h_k=f_k(h_{k-1}),\qquad x=h_K.

The chain rule converts the determinant of the full transformation into a product of layerwise determinants. In logarithmic form,

log⁡pθ(x)=log⁡p0(h0)−∑k=1Klog⁡∣det⁡Dfk(hk−1)∣.\log p_\theta(x) = \log p_0(h_0) - \sum_{k=1}^{K} \log\left|\det Df_k(h_{k-1})\right|.

This decomposition is the main computational advantage of the architecture. Each layer can have a relatively simple structure, while the composition produces complicated nonlinear dependencies. The inverse is obtained by applying the inverse layers in reverse order.

For example, suppose zz is a standard Gaussian and x=Az+bx=Az+b, with nonsingular AA. The resulting distribution is Gaussian with mean bb and covariance AATAA^\mathsf{T}, and its log density contains the correction −log⁡∣det⁡A∣-\log|\det A|. Nonlinear flows generalize this familiar affine construction. Unlike an unrestricted nonlinear neural network, however, every layer must satisfy the appropriate invertibility conditions.

An arbitrary dense Jacobian is costly: computing its determinant by standard matrix factorization generally takes cubic time in dimension. Explicitly constructing the Jacobian can also require quadratic storage. Flow architectures therefore exploit triangular, block-triangular, diagonal, or otherwise structured derivatives. The important property is not merely that a map is invertible in principle, but that inversion and the log absolute determinant can be computed efficiently and stably.

Under these assumptions, the density is normalized automatically whenever the base density is normalized. No unknown partition function has to be estimated. “Exact likelihood” in flow literature refers primarily to this explicit probabilistic calculation; floating-point arithmetic, iterative inversion, and numerical differential-equation solvers can still introduce approximation error.

Historical development

Normalizing flows bring together classical probability transformations and modern neural density estimation. The underlying change-of-variables theorem predates machine learning, as do triangular transformations between distributions. In 1952, Murray Rosenblatt described a transformation constructed from successive conditional cumulative distribution functions. For continuous distributions satisfying appropriate regularity conditions, this construction maps a multivariate distribution to independent uniform variables. Its conditional, coordinate-by-coordinate structure anticipates important features of autoregressive flows.

An early neural precursor was the nonlinear independent component estimation model introduced by Gustavo Deco and Wilfried Brauer in 1995. It used invertible nonlinear transformations for density modeling. Such work established that neural transformations could be connected directly to probability densities, although the computational architectures and large-scale training methods used in later flows were not yet available.

The contemporary research program became prominent in 2014 and 2015. Laurent Dinh, David Krueger, and Yoshua Bengio introduced NICE, or nonlinear independent components estimation, in 2014. Its additive coupling layers provided simple inverses and unit Jacobian determinants. In 2015, Danilo Jimenez Rezende and Shakir Mohamed presented Variational Inference with Normalizing Flows, developing sequences of invertible transformations as a way to enrich approximate posterior distributions. Their planar and radial flows showed how relatively simple transformations could improve the flexibility of variational inference.

Subsequent work separated into several influential architectural families. Dinh, Jascha Sohl-Dickstein, and Bengio introduced Real NVP in a 2016 preprint, published at the 2017 International Conference on Learning Representations. Its affine coupling layers combined easy inversion with nonconstant volume changes. Inverse autoregressive flow was introduced by Diederik Kingma and colleagues in 2016, while masked autoregressive flow was presented by George Papamakarios, Theo Pavlakou, and Iain Murray in 2017. These models exposed a central trade-off between fast density evaluation and fast sampling.

In 2018, Kingma and Prafulla Dhariwal introduced Glow, which used learned invertible 1×11\times1 convolutions and activation normalization in a multiscale image model. The same year, Ricky T. Q. Chen and colleagues introduced neural ordinary differential equations, including the continuous normalizing-flow formulation. Will Grathwohl and colleagues subsequently developed FFJORD, using stochastic trace estimation to make continuous-flow likelihood calculations more scalable.

Research in 2019 included neural spline flows and Flow++, which improved the flexibility of individual transformations and the treatment of discrete image data. Later developments broadened the subject toward constrained geometries, scientific sampling, and flow-matching methods. These developments share transport-based ideas but differ in their training objectives, numerical requirements, and whether density evaluation is central to their intended use.

Coupling transformations and multiscale architectures

A coupling layer divides its input into two groups, uau_a and ubu_b. One group remains unchanged while the other is transformed using parameters computed from the unchanged group. In an affine coupling layer,

va=ua,vb=ub⊙exp⁡(sθ(ua))+tθ(ua),v_a=u_a,\qquad v_b=u_b\odot\exp(s_\theta(u_a))+t_\theta(u_a),

where ⊙\odot denotes elementwise multiplication. The functions sθs_\theta and tθt_\theta may be complex neural networks. They do not themselves need to be invertible because the conditioning variables uau_a are preserved.

The inverse is explicit:

ua=va,ub=(vb−tθ(va))⊙exp⁡(−sθ(va)).u_a=v_a,\qquad u_b= \bigl(v_b-t_\theta(v_a)\bigr) \odot\exp\bigl(-s_\theta(v_a)\bigr).

The Jacobian is block triangular, so its determinant depends only on the diagonal scaling terms:

log⁡∣det⁡J∣=∑isθ,i(ua).\log|\det J| = \sum_i s_{\theta,i}(u_a).

Derivatives of the potentially large conditioning networks do not enter this determinant directly. This is the architectural device that permits expressive neural computation while retaining inexpensive density evaluation. In an additive coupling layer, sθ=0s_\theta=0, making the determinant exactly one.

A single layer leaves part of its input unchanged, so repeated layers must exchange information across partitions. Real NVP used alternating masks, including checkerboard partitions over image positions and channel-based partitions. Fixed permutations are another option. Glow introduced learned invertible 1×11\times1 convolutions: an invertible matrix mixes the channels at each spatial location. For an image feature map with height HH and width WW, the log determinant of such a channel-mixing operation is HWlog⁡∣det⁡Wc∣HW\log|\det W_c|, where WcW_c is the channel transformation matrix.

Multiscale architectures reduce computation by progressively separating some variables from the active transformation pathway. Other variables continue through deeper layers. Image “squeeze” operations rearrange spatial positions into channels without discarding information, allowing local image structure to interact through channel mixing. These operations differ from ordinary pooling: a bijective rearrangement preserves every input value.

Affine coupling is computationally attractive, but its transformation of each conditioned coordinate is limited to scaling and translation. More expressive coupling layers replace the affine map with a monotone nonlinear function. Spline transformations are a prominent example. Rational-quadratic spline flows parameterize bin widths, bin heights, and derivatives subject to positivity constraints, providing a flexible monotone map with an analytically computable inverse.

The practical trade-off is between the complexity of each scalar transform and the number of layers needed. A richer coupling transform may reduce depth, but increases parameter generation, numerical bookkeeping, and implementation complexity. Coupling models remain important because both their forward and inverse computations can process many coordinates simultaneously.

Autoregressive flows and directional computation

An autoregressive transformation defines each output coordinate using earlier coordinates in an ordering. One common affine form is

xi=μi(z<i)+σi(z<i)zi,σi(z<i)>0,x_i=\mu_i(z_{<i})+\sigma_i(z_{<i})z_i, \qquad \sigma_i(z_{<i})>0,

where z<iz_{<i} denotes the coordinates preceding ii. The Jacobian is triangular because xix_i depends only on z1,…,ziz_1,\ldots,z_i. Its determinant is therefore the product of the diagonal scales. This connects flow construction to autoregressive modeling, in which a joint density is factorized into conditional distributions.

Triangular structure makes the determinant inexpensive, but it does not guarantee that both transformation directions are equally fast. In the displayed construction, all conditioning inputs zz are known during sampling. A suitably masked neural network can calculate the transformation parameters in parallel. Recovering zz from a given xx, however, generally requires solving coordinates sequentially: z1z_1 must be recovered before evaluating parameters for z2z_2, and so forth.

Inverse autoregressive flow exploits this directionality for approximate posterior sampling. In a variational autoencoder, optimization commonly requires samples from an approximate posterior together with their densities. A flow can transform a simple posterior sample efficiently while accumulating the density corrections along that sampled trajectory. It need not support equally efficient density evaluation at every arbitrary point.

Masked autoregressive flow makes the opposite trade-off attractive for density estimation. Its data-to-base computation can be written as

zi=xi−μi(x<i)σi(x<i).z_i=\frac{x_i-\mu_i(x_{<i})}{\sigma_i(x_{<i})}.

Given a complete observed vector xx, a masked network can compute all conditional means and scales in one evaluation. Generation is usually sequential because each newly generated coordinate supplies conditioning information for subsequent coordinates. These models can thus score large batches efficiently while requiring many network evaluations to generate each high-dimensional sample.

Masking enforces the required dependency pattern inside a feedforward network. The MADE architecture, introduced by Mathieu Germain and colleagues in 2015, provided a widely used mechanism for constructing such masked autoregressive networks. Different layers of a flow can use different coordinate orderings or permutations, avoiding a permanent dependence on one arbitrary ordering.

Autoregressive transformations need not be affine. Monotone splines and other scalar bijections can model more complicated conditional distributions. Their computational behavior still depends on the direction in which conditioning information becomes available. Consequently, comparisons between flow models distinguish training throughput, arbitrary-point density evaluation, and sampling speed rather than treating all three as a single measure of efficiency.

Continuous normalizing flows and flow matching

A continuous normalizing flow replaces a finite sequence of transformations with an evolving state governed by an ordinary differential equation:

dx(t)dt=vθ(x(t),t).\frac{dx(t)}{dt}=v_\theta(x(t),t).

The time-dependent vector field vθv_\theta transports an initial distribution at time 00 to a final distribution at time TT. Under conditions ensuring unique solutions over the interval, integrating the equation backward recovers the initial state. Invertibility follows from the uniqueness of trajectories rather than from a specially constructed algebraic layer.

The density evolves according to the instantaneous change-of-variables formula:

ddtlog⁡pt(x(t))=−∇⋅vθ(x(t),t)=−tr⁡(∂vθ∂x).\frac{d}{dt}\log p_t(x(t)) = -\nabla\cdot v_\theta(x(t),t) = -\operatorname{tr} \left( \frac{\partial v_\theta}{\partial x} \right).

Thus a log density can be computed by integrating the state and its accumulated divergence together. A finite transformation requires a log determinant; its continuous-time counterpart requires an integral of a Jacobian trace. A vector field with zero divergence preserves local volume and keeps density constant along each trajectory.

Computing the full trace can be expensive in high dimensions. A stochastic trace estimator uses a random vector ϵ\epsilon satisfying E[ϵϵT]=I\mathbb{E}[\epsilon\epsilon^\mathsf{T}]=I:

Eϵ[ϵTJϵ]=tr⁡(J).\mathbb{E}_\epsilon \left[ \epsilon^\mathsf{T}J\epsilon \right] = \operatorname{tr}(J).

Automatic differentiation evaluates the required products without explicitly constructing the whole Jacobian. FFJORD used this principle to permit relatively unrestricted neural vector fields. The estimator reduces computational requirements but introduces stochastic error into the divergence calculation.

Numerical integration creates a different set of costs from those in discrete flows. An adaptive solver may require many vector-field evaluations, particularly when the learned dynamics are stiff or rapidly varying. Solver tolerances affect accuracy, runtime, and the agreement between forward and reverse calculations. The mathematical flow can be exactly invertible even when its numerical implementation is only approximately reversible.

Flow matching, prominently developed in work by Yaron Lipman and colleagues in 2022–2023, trains a vector field by matching target velocities along selected probability paths. Related conditional formulations permit training without simulating the learned differential equation at every optimization step. Sampling still generally integrates a learned vector field. Likelihood evaluation, when desired, requires the additional density-evolution calculation.

This creates a distinction between an invertible continuous-time generative mechanism and a model trained directly by maximum likelihood. Flow matching can produce the former without using the latter objective. Certain diffusion models also admit a deterministic probability-flow differential equation with the same time-dependent marginal distributions as their stochastic process. This mathematical relationship does not make all diffusion models and normalizing flows identical in architecture or training.

Training objectives and statistical interpretation

A density-estimation flow is commonly trained by maximum likelihood estimation. Given observations x1,…,xNx_1,\ldots,x_N, the objective is

L(θ)=−1N∑n=1Nlog⁡pθ(xn).\mathcal{L}(\theta) = -\frac{1}{N} \sum_{n=1}^{N}\log p_\theta(x_n).

The flow evaluates each observation in base coordinates and combines the base log density with the relevant volume corrections. Backpropagation differentiates this objective through the transformation and determinant calculations. Gradients do not require sampling from the model when the training observations and data-to-base transformation are directly available.

At the population level, minimizing expected negative log likelihood is equivalent, up to a constant independent of the model, to minimizing

DKL(pdata∥pθ),D_{\mathrm{KL}}(p_{\mathrm{data}}\|p_\theta),

the Kullback–Leibler divergence from the data distribution to the model, assuming the quantities are well-defined. This direction strongly penalizes assigning extremely low density where the data distribution has positive mass. Finite samples, limited model capacity, and imperfect optimization nevertheless mean that maximum likelihood does not guarantee recovery of every mode.

Flows also parameterize approximate distributions in variational inference. If an unnormalized target density is p~(x)\tilde p(x), a flow qθq_\theta can minimize

Ex∼qθ[log⁡qθ(x)−log⁡p~(x)].\mathbb{E}_{x\sim q_\theta} \left[ \log q_\theta(x)-\log \tilde p(x) \right].

The target’s unknown normalization constant disappears from the optimization. This corresponds to minimizing DKL(qθ∥p)D_{\mathrm{KL}}(q_\theta\|p), which reverses the divergence direction used in ordinary density estimation. For restricted approximating families, it can favor concentrating on a subset of modes rather than spreading probability into low-target-density regions.

Conditional flows model pθ(x∣c)p_\theta(x\mid c), where cc may be a class label, an audio conditioning signal, or a measurement vector. Transformation parameters depend on cc, but the transformation must be invertible in xx for each fixed condition. The conditioning network itself need not be invertible. This permits supervised applications such as conditional uncertainty estimation and posterior approximation.

Optimization must also control numerical conditioning. Very large or small affine scales can cause unstable activations, extreme log determinants, or poor inversion accuracy. Implementations commonly constrain scale parameterizations, initialize transformations near the identity, or use normalization layers with tractable inverses. Glow’s activation normalization uses learned per-channel affine parameters initialized from a data batch, rather than continually relying on changing batch statistics.

Model assessment separates the training objective from held-out performance. Negative log likelihood on a test set measures density generalization, while sample quality, calibration, computational cost, and task-specific performance measure different properties. Good likelihood alone does not establish semantic realism or reliable behavior under distribution shift.

Discrete data, dimensionality, and geometric constraints

The standard change-of-variables formula applies to continuous densities on spaces of equal dimension. Images stored as integer-valued pixels are discrete, so fitting a continuous density directly to their raw grid points creates a mismatch. A continuous density can become arbitrarily concentrated near observed points without defining a satisfactory discrete probability model. The distinction matters because density values depend on measurement scale, whereas probabilities assigned to discrete outcomes do not.

A common solution is dequantization. For an integer-valued observation xx, noise uu is added within its associated quantization cell, often with uu uniform on [0,1)d[0,1)^d. The modeled discrete probability is then

Pθ(x)=∫[0,1)dpθ(x+u) du.P_\theta(x) = \int_{[0,1)^d}p_\theta(x+u)\,du.

Jensen’s inequality gives

log⁡Pθ(x)≥Eu[log⁡pθ(x+u)].\log P_\theta(x) \geq \mathbb{E}_{u} \left[\log p_\theta(x+u)\right].

Consequently, uniform dequantization trains a lower bound on the discrete log likelihood. Rescaling pixels introduces an additional density correction, which must be included when comparing reported coding costs.

Variational dequantization replaces uniform noise with a learned conditional density q(u∣x)q(u\mid x), yielding the bound

log⁡Pθ(x)≥Eq(u∣x)[log⁡pθ(x+u)−log⁡q(u∣x)].\log P_\theta(x) \geq \mathbb{E}_{q(u\mid x)} \left[ \log p_\theta(x+u)-\log q(u\mid x) \right].

Flow++ used this approach to improve image density modeling. Image likelihoods are often reported as bits per dimension, obtained by converting negative log probabilities from natural logarithms to base two and dividing by the number of scalar image components. Comparisons require consistent quantization, preprocessing, and bound conventions.

Dimensionality presents a separate restriction. A conventional bijection maps between spaces of the same dimension; it cannot directly compress a full-dimensional continuous distribution into a lower-dimensional latent code while retaining an ordinary inverse. Data concentrated exactly on a lower-dimensional manifold do not have a full-dimensional density with respect to ambient volume. Adding observation noise, introducing auxiliary variables, or using a manifold-specific construction changes the modeling problem so that an appropriate density can be defined.

Topology also constrains expressiveness. A continuous invertible image of a connected support remains connected. A standard Gaussian base therefore cannot be transformed by a global diffeomorphism into a distribution whose support consists of truly disconnected components. It can approximate separated clusters by assigning very little probability to connecting regions, but this may require sharply varying transformations.

For angular variables, rotations, spheres, or molecular constraints, Euclidean flows can impose artificial discontinuities or ignore important structure. Manifold flows instead define transformations and density corrections relative to the relevant geometric volume measure. Augmented and non-bijective extensions relax other restrictions, but marginalizing added variables or accounting for many-to-one maps may replace a simple exact likelihood with an integral or variational bound.

Applications in inference, generation, and scientific sampling

Flows serve as general-purpose density estimators for vectors, images, and time series. Their ability to return a normalized density makes them useful when a downstream calculation requires more than plausible samples. In anomaly detection, for example, a fitted model can assign a score based on negative log likelihood. Such scores describe atypicality under the learned distribution, however, and do not automatically identify semantic novelty, physical faults, or malicious behavior.

Audio synthesis illustrates the value of rapid inversion. WaveGlow, introduced by Ryan Prenger, Rafael Valle, and Bryan Catanzaro in 2018, used flow-based transformations to generate speech waveforms conditioned on mel-spectrograms. FloWaveNet was another contemporaneous flow-based vocoder. These systems exploited coupling-style computation to avoid the sample-by-sample generation required by many autoregressive waveform models, while retaining a likelihood-based training objective.

In Bayesian inference, flows can approximate posterior distributions with correlated, skewed, or multimodal structure. A conditional flow can learn a family of posteriors across many observations, amortizing the cost of inference. In simulation-based inference, synthetic parameter–observation pairs train a conditional density estimator even when the simulator does not provide an analytically tractable likelihood. Accuracy still depends on simulation coverage, model capacity, and validation; invertibility alone does not establish posterior correctness.

Scientific sampling uses flows to propose configurations for distributions defined by an energy function. Boltzmann generators, introduced by Frank Noé and colleagues in 2019, combined invertible neural transformations with statistical-mechanical objectives to generate molecular configurations. For a target proportional to exp⁡[−βU(x)]\exp[-\beta U(x)], the evaluable proposal density enables importance weights involving exp⁡[−βU(x)]/qθ(x)\exp[-\beta U(x)]/q_\theta(x). This can correct discrepancies in principle, but highly variable weights can make estimates inefficient.

Flow proposals can also be incorporated into Markov chain Monte Carlo. A Metropolis–Hastings acceptance rule accounts for the mismatch between a learned proposal and the target distribution. In lattice field theory, flow-based samplers have been investigated as ways to produce global proposals and reduce autocorrelation. Their effectiveness depends on learning relevant structure and preserving, or correctly handling, the symmetries of the physical system.

Because a flow provides probabilities, it can also support lossless compression schemes that convert a probabilistic model into code lengths. Practical compression additionally requires a suitable discrete representation and an actual coding procedure; continuous density values alone are not code lengths. Across these applications, the decisive feature is the combination of sampling and probability accounting, rather than generation alone.

Evaluation, limitations, and related generative models

Normalizing flows are often contrasted with variational autoencoders and generative adversarial networks. A conventional variational autoencoder introduces latent variables and an observation model, generally optimizing a lower bound on data likelihood. A conventional adversarial model learns through a discriminator and does not ordinarily provide a tractable normalized data density. A bijective flow instead obtains that density from an explicit transformation. Hybrid models can combine these components, so the distinctions concern standard formulations rather than mutually exclusive categories.

Autoregressive density models also provide normalized likelihoods, but do so through a product of conditional probabilities rather than necessarily through an explicit invertible transformation. For continuous variables, suitable conditional cumulative distribution functions connect autoregressive factorization to triangular flows. Operationally, both approaches may face sequential generation costs, while coupling flows permit more parallel computation at the price of structural restrictions on each layer.

Likelihood is a demanding but incomplete evaluation criterion. Research on deep generative models has shown that out-of-distribution inputs can receive higher likelihood than the training distribution’s own examples. Eric Nalisnick and colleagues documented this behavior in 2018–2019, including experiments with flow-based image models. Density can be strongly influenced by low-level statistics such as background intensity or local variation rather than the semantic categories that an evaluator considers important.

High-dimensional geometry further complicates interpretation. A distribution’s highest-density point need not lie in the region containing most of its probability mass. For a high-dimensional standard Gaussian, density is greatest at the origin, but a typical sample lies at a radius of approximately d\sqrt d. Consequently, ranking observations only by pointwise density is not equivalent to assessing whether they have typical joint properties. This distinction helps explain why likelihood thresholds can be unreliable as universal novelty detectors.

Invertibility also does not guarantee efficient learning or numerical robustness. A flow may approximate a target only by developing severe stretching and compression. Deep compositions can become poorly conditioned, and continuous flows may require expensive numerical integration. Ordinary flows also make marginalization difficult: although the full joint density is explicit, integrating out an arbitrary subset of coordinates usually has no closed-form solution.

Evaluation therefore reports several quantities separately: held-out likelihood, sample fidelity and diversity, sampling latency, density-evaluation cost, parameter count, and numerical accuracy. Scientific applications additionally examine observables, effective sample size, importance-weight variance, or Markov-chain acceptance rates. For conditional inference, calibration and coverage test whether nominal uncertainty matches empirical behavior.

The choice of flow family depends on which operation dominates the intended task. Masked autoregressive flows favor density evaluation; inverse autoregressive flows favor transformation of known base samples; coupling flows balance both directions; continuous flows offer flexible dynamics at the cost of numerical integration. Geometric and discrete extensions address domains where the standard Euclidean bijection would impose the wrong probability structure.