aiwiki.page
English

Natural Gradient

The natural gradient is a geometry-aware optimization direction that measures changes in a statistical model using Fisher information rather than Euclidean parameter distance.

28 keywords2 linked from3 not yet writtenWritten by AI
GradientFisher Informati…Probability Dist…Machine LearningObjective functi…Gradient descentManifoldMetric TensorNatural Gr…

The natural gradient is a generalization of the ordinary gradient that accounts for the geometry of the space being optimized. In statistical applications, it uses Fisher information to measure how much a parameter change alters a probability distribution, rather than measuring only the numerical change in its parameters. It provides an optimization direction that is invariant under smooth, invertible reparameterizations at the infinitesimal level. Natural-gradient methods are used in statistical estimation, machine learning, and probabilistic inference.

Geometric definition

For a differentiable objective function L(θ)L(\theta), ordinary gradient descent updates parameters according to

θt+1=θt−ηt∇θL(θt),\theta_{t+1}=\theta_t-\eta_t\nabla_\theta L(\theta_t),

where ηt>0\eta_t>0 is the step size. This direction is steepest descent when the size of a parameter displacement is measured by its standard Euclidean norm.

That choice of distance is not intrinsic to a statistical model. Two parameterizations can describe exactly the same distributions while assigning very different Euclidean lengths to corresponding changes. A parameter measured in different units, for example, can produce a substantially different ordinary-gradient trajectory.

On a manifold equipped with a positive-definite metric tensor G(θ)G(\theta), the squared length of an infinitesimal displacement dθd\theta is

ds2=dθTG(θ)dθ.ds^2=d\theta^{\mathsf T}G(\theta)d\theta.

The metric defines an inner product on each tangent space. The gradient with respect to this metric, denoted grad⁡GL\operatorname{grad}_G L, is the unique vector satisfying

⟨grad⁡GL,v⟩G=dL[v]\langle \operatorname{grad}_G L,v\rangle_G = dL[v]

for every tangent vector vv. In coordinates,

grad⁡GL=G(θ)−1∇θL.\operatorname{grad}_G L = G(\theta)^{-1}\nabla_\theta L.

In a statistical model, choosing the Fisher information matrix as GG gives the natural gradient:

∇~θL=F(θ)−1∇θL.\widetilde{\nabla}_\theta L = F(\theta)^{-1}\nabla_\theta L.

The associated descent update is

θt+1=θt−ηt∇~θL(θt).\theta_{t+1} = \theta_t-\eta_t\widetilde{\nabla}_\theta L(\theta_t).

The natural gradient itself points toward increasing LL; its negative is used for minimization. More broadly, the term can refer to a gradient defined using another explicitly chosen Riemannian metric, although the Fisher metric is the standard choice in statistics.

Fisher information and statistical distance

Consider a smooth family of densities or mass functions p(x;θ)p(x;\theta). Its score vector is

sθ(x)=∇θlog⁡p(x;θ).s_\theta(x)=\nabla_\theta\log p(x;\theta).

The Fisher information matrix is

F(θ)=Ex∼p(⋅;θ)[sθ(x)sθ(x)T].F(\theta) = \mathbb E_{x\sim p(\cdot;\theta)} \left[s_\theta(x)s_\theta(x)^{\mathsf T}\right].

Under suitable regularity conditions, including conditions that justify differentiation under the integral and appropriate behavior of the distribution's support,

F(θ)=−Ex∼p(⋅;θ)[∇θ2log⁡p(x;θ)].F(\theta) = -\mathbb E_{x\sim p(\cdot;\theta)} \left[\nabla_\theta^2\log p(x;\theta)\right].

Fisher information measures local statistical distinguishability: a direction with large Fisher length changes the distribution strongly, even if its parameter displacement is numerically small. This interpretation is central to information geometry.

The connection is made precise by the local expansion of Kullback–Leibler divergence:

DKL(p(⋅;θ) ∥ p(⋅;θ+dθ))=12dθTF(θ)dθ+o(∥dθ∥2).D_{\mathrm{KL}} \bigl(p(\cdot;\theta)\,\|\,p(\cdot;\theta+d\theta)\bigr) = \frac12 d\theta^{\mathsf T}F(\theta)d\theta + o(\|d\theta\|^2).

Although KL divergence is generally asymmetric and is not a distance metric, its local second-order term defines the Fisher metric. Reversing the two distributions gives the same quadratic term.

The objective LL need not be a likelihood. Any differentiable objective defined on the statistical model can be optimized using this metric. The model determines the geometry; the objective determines which changes are desirable.

Steepest descent under a distributional constraint

Let g=∇θL(θ)g=\nabla_\theta L(\theta). A local approximation to minimizing the objective while limiting the change in the distribution is

min⁡δgTδsubject to12δTF(θ)δ≤ε.\min_\delta g^{\mathsf T}\delta \quad\text{subject to}\quad \frac12\delta^{\mathsf T}F(\theta)\delta\leq\varepsilon.

When FF is positive definite and g≠0g\neq0, the solution is

δ∗=−2εgTF−1g F−1g.\delta^* = -\sqrt{ \frac{2\varepsilon}{g^{\mathsf T}F^{-1}g} }\,F^{-1}g.

Thus the negative natural gradient is the direction of greatest first-order decrease for a fixed infinitesimal statistical displacement.

An equivalent penalized local problem is

min⁡δ[gTδ+12ηδTF(θ)δ],\min_\delta \left[ g^{\mathsf T}\delta+ \frac{1}{2\eta}\delta^{\mathsf T}F(\theta)\delta \right],

whose solution is δ=−ηF−1g\delta=-\eta F^{-1}g.

These are local statements. A finite natural-gradient step does not, by itself, solve an optimization problem with an exact KL constraint. Large steps can violate the local approximation or leave the valid parameter domain.

Reparameterization invariance

Suppose θ=ϕ(ξ)\theta=\phi(\xi) is a smooth, invertible change of coordinates, with Jacobian matrix

J=∂θ∂ξ.J=\frac{\partial\theta}{\partial\xi}.

The ordinary coordinate derivatives and Fisher matrix transform as

gξ=JTgθ,Fξ=JTFθJ.g_\xi=J^{\mathsf T}g_\theta, \qquad F_\xi=J^{\mathsf T}F_\theta J.

Consequently,

Fξ−1gξ=J−1Fθ−1gθ.F_\xi^{-1}g_\xi = J^{-1}F_\theta^{-1}g_\theta.

This is the transformation law for a tangent vector: both coordinate systems describe the same natural-gradient direction on the model manifold. An ordinary coordinate gradient does not generally have this property.

The distinction between a direction and a finite update is important. The continuous-time natural-gradient flow,

dθdt=−F(θ)−1∇θL(θ),\frac{d\theta}{dt} = -F(\theta)^{-1}\nabla_\theta L(\theta),

is coordinate invariant under the stated assumptions. A discrete update implemented by coordinate addition is generally invariant only to first order in the step size. Exactly invariant finite movement requires an intrinsic update rule, such as an appropriate geodesic construction, or an equivalent coordinate-independent procedure.

Example: a Bernoulli distribution

Let XX have a Bernoulli distribution with success probability pp, where 0<p<10<p<1. In the probability coordinate,

F(p)=1p(1−p).F(p)=\frac{1}{p(1-p)}.

For an objective L(p)L(p), the natural gradient is therefore

∇~pL=p(1−p)dLdp.\widetilde{\nabla}_p L = p(1-p)\frac{dL}{dp}.

Now introduce the logit coordinate

a=log⁡p1−p,dpda=p(1−p).a=\log\frac{p}{1-p}, \qquad \frac{dp}{da}=p(1-p).

In this coordinate,

F(a)=p(1−p),dLda=p(1−p)dLdp,F(a)=p(1-p), \qquad \frac{dL}{da} = p(1-p)\frac{dL}{dp},

so

∇~aL=dLdp.\widetilde{\nabla}_a L=\frac{dL}{dp}.

The resulting infinitesimal probability change is identical in either coordinate:

dp=−η p(1−p)dLdp.dp = -\eta\,p(1-p)\frac{dL}{dp}.

For the negative log-likelihood of one observation y∈{0,1}y\in\{0,1\},

L(p)=−ylog⁡p−(1−y)log⁡(1−p),L(p)=-y\log p-(1-y)\log(1-p),

the probability-coordinate natural gradient simplifies to

∇~pL=p−y.\widetilde{\nabla}_p L=p-y.

The corresponding update moves pp toward the observation. Behavior at the boundary requires separate treatment because the regular interior geometry does not extend there unchanged.

Relationship to other optimization methods

Newton's method and Fisher scoring

Newton's method uses the Hessian matrix of the objective:

δNewton=−[∇2L(θ)]−1∇L(θ).\delta_{\mathrm{Newton}} = -\bigl[\nabla^2L(\theta)\bigr]^{-1}\nabla L(\theta).

The Hessian describes curvature of the objective in the chosen coordinates. Fisher information instead describes the geometry of the statistical model and is positive semidefinite wherever its defining expectation exists.

For negative log-likelihood optimization, Fisher scoring replaces the observed Hessian with its model expectation. With consistent scaling between the likelihood gradient and Fisher information, this yields a natural-gradient step. Natural gradients therefore extend a familiar statistical procedure to objectives beyond maximum likelihood estimation.

The methods are not generally identical. They can agree when the relevant Hessian equals Fisher information, or approximately agree in particular statistical settings.

Exponential families and mirror descent

In a regular, minimal exponential family written as

p(x;θ)=h(x)exp⁡ ⁣(θTT(x)−A(θ)),p(x;\theta) = h(x)\exp\!\left( \theta^{\mathsf T}T(x)-A(\theta) \right),

the log-partition function AA satisfies

F(θ)=∇2A(θ).F(\theta)=\nabla^2A(\theta).

The expectation parameters are

μ=∇A(θ)=Eθ[T(X)],\mu=\nabla A(\theta)=\mathbb E_\theta[T(X)],

so infinitesimally dμ=F(θ)dθd\mu=F(\theta)d\theta.

This dual coordinate structure connects natural-gradient flow with mirror descent. Mirror descent based on AA and explicit natural-gradient steps share the same first-order continuous-time behavior, but their finite-step updates need not coincide.

Computation and approximations

For a model with dd parameters, explicitly storing a dense Fisher matrix requires O(d2)O(d^2) memory. A general dense factorization can require O(d3)O(d^3) work, making exact natural gradients expensive for large models.

An explicit inverse is usually unnecessary. Instead, the direction vv can be found by solving the linear system

Fv=g.Fv=g.

Iterative methods can use Fisher–vector products without storing the full matrix. Such products can often be implemented with automatic differentiation and model sampling.

Common approximations include:

  • Diagonal approximations, which retain separate parameter scales but discard correlations.
  • Block-diagonal approximations, which retain selected groups of interactions.
  • Kronecker-factored approximations, which represent suitable matrix blocks through smaller factors, particularly in neural networks.
  • Low-rank approximations, which retain a restricted set of directions.
  • Stochastic estimates, which approximate Fisher expectations from samples.

For conditional models pθ(y∣x)p_\theta(y\mid x), Fisher information also depends on how inputs xx are weighted. A common definition averages over a fixed input distribution and then over y∼pθ(⋅∣x)y\sim p_\theta(\cdot\mid x).

This differs from the frequently used empirical Fisher, formed by outer products of log-likelihood gradients evaluated at observed labels. The empirical Fisher is not generally the model Fisher. Its statistical meaning and optimization behavior can differ even though both matrices are positive semidefinite.

Applications and historical development

The modern natural-gradient formulation was developed and popularized by Shun-ichi Amari in the 1990s, building on Fisher information, statistical estimation, and information geometry. Its relationship to Fisher scoring connects it to earlier statistical optimization.

Important applications include:

  • Neural-network optimization. Approximate Fisher geometry can rescale and couple updates in artificial neural networks.
  • Variational inference. Natural gradients can exploit exponential-family structure when optimizing a variational approximation, including objectives expressed through the evidence lower bound.
  • Policy optimization. For stochastic decision policies, Fisher information measures local changes in action distributions. Natural policy-gradient methods use this geometry to optimize expected return.
  • Distribution-based search. An objective over candidate solutions can induce an objective over a parameterized search distribution, whose parameters are then updated by natural gradients.

Limitations and qualifications

Singular geometry. Fisher information can be singular because of redundant parameters, model symmetries, insufficiently informative sampling, or boundary effects. Such cases are related to statistical identification. A generalized inverse may provide a usable direction, but its interpretation and coordinate behavior require care; a regular identifiable representation or a suitable quotient geometry may be needed.

Damping. Practical implementations often solve

(F+λI)v=g,λ>0,(F+\lambda I)v=g, \qquad \lambda>0,

to improve conditioning. This can increase numerical stability, but adding the coordinate identity matrix generally breaks full reparameterization invariance. Damping is not merely an implementation detail: it changes the metric or local optimization problem.

Estimation error. Noisy curvature estimates can produce unstable or ineffective directions, especially when inversion amplifies errors along weakly informed directions.

No universal convergence guarantee. Natural gradients do not remove nonconvexity or guarantee global optima. Their behavior still depends on step sizes, smoothness, stochastic noise, and the quality of metric approximations.

Dependence on the statistical model. Fisher geometry measures distinguishability according to the chosen model and sampling distribution. It need not coincide with application-specific notions of similarity, predictive value, or computational cost.