The natural gradient is a generalization of the ordinary gradient that accounts for the geometry of the space being optimized. In statistical applications, it uses Fisher information to measure how much a parameter change alters a probability distribution, rather than measuring only the numerical change in its parameters. It provides an optimization direction that is invariant under smooth, invertible reparameterizations at the infinitesimal level. Natural-gradient methods are used in statistical estimation, machine learning, and probabilistic inference.
Geometric definition
For a differentiable objective function , ordinary gradient descent updates parameters according to
where is the step size. This direction is steepest descent when the size of a parameter displacement is measured by its standard Euclidean norm.
That choice of distance is not intrinsic to a statistical model. Two parameterizations can describe exactly the same distributions while assigning very different Euclidean lengths to corresponding changes. A parameter measured in different units, for example, can produce a substantially different ordinary-gradient trajectory.
On a manifold equipped with a positive-definite metric tensor , the squared length of an infinitesimal displacement is
The metric defines an inner product on each tangent space. The gradient with respect to this metric, denoted , is the unique vector satisfying
for every tangent vector . In coordinates,
In a statistical model, choosing the Fisher information matrix as gives the natural gradient:
The associated descent update is
The natural gradient itself points toward increasing ; its negative is used for minimization. More broadly, the term can refer to a gradient defined using another explicitly chosen Riemannian metric, although the Fisher metric is the standard choice in statistics.
Fisher information and statistical distance
Consider a smooth family of densities or mass functions . Its score vector is
The Fisher information matrix is
Under suitable regularity conditions, including conditions that justify differentiation under the integral and appropriate behavior of the distribution's support,
Fisher information measures local statistical distinguishability: a direction with large Fisher length changes the distribution strongly, even if its parameter displacement is numerically small. This interpretation is central to information geometry.
The connection is made precise by the local expansion of Kullback–Leibler divergence:
Although KL divergence is generally asymmetric and is not a distance metric, its local second-order term defines the Fisher metric. Reversing the two distributions gives the same quadratic term.
The objective need not be a likelihood. Any differentiable objective defined on the statistical model can be optimized using this metric. The model determines the geometry; the objective determines which changes are desirable.
Steepest descent under a distributional constraint
Let . A local approximation to minimizing the objective while limiting the change in the distribution is
When is positive definite and , the solution is
Thus the negative natural gradient is the direction of greatest first-order decrease for a fixed infinitesimal statistical displacement.
An equivalent penalized local problem is
whose solution is .
These are local statements. A finite natural-gradient step does not, by itself, solve an optimization problem with an exact KL constraint. Large steps can violate the local approximation or leave the valid parameter domain.
Reparameterization invariance
Suppose is a smooth, invertible change of coordinates, with Jacobian matrix
The ordinary coordinate derivatives and Fisher matrix transform as
Consequently,
This is the transformation law for a tangent vector: both coordinate systems describe the same natural-gradient direction on the model manifold. An ordinary coordinate gradient does not generally have this property.
The distinction between a direction and a finite update is important. The continuous-time natural-gradient flow,
is coordinate invariant under the stated assumptions. A discrete update implemented by coordinate addition is generally invariant only to first order in the step size. Exactly invariant finite movement requires an intrinsic update rule, such as an appropriate geodesic construction, or an equivalent coordinate-independent procedure.
Example: a Bernoulli distribution
Let have a Bernoulli distribution with success probability , where . In the probability coordinate,
For an objective , the natural gradient is therefore
Now introduce the logit coordinate
In this coordinate,
so
The resulting infinitesimal probability change is identical in either coordinate:
For the negative log-likelihood of one observation ,
the probability-coordinate natural gradient simplifies to
The corresponding update moves toward the observation. Behavior at the boundary requires separate treatment because the regular interior geometry does not extend there unchanged.
Relationship to other optimization methods
Newton's method and Fisher scoring
Newton's method uses the Hessian matrix of the objective:
The Hessian describes curvature of the objective in the chosen coordinates. Fisher information instead describes the geometry of the statistical model and is positive semidefinite wherever its defining expectation exists.
For negative log-likelihood optimization, Fisher scoring replaces the observed Hessian with its model expectation. With consistent scaling between the likelihood gradient and Fisher information, this yields a natural-gradient step. Natural gradients therefore extend a familiar statistical procedure to objectives beyond maximum likelihood estimation.
The methods are not generally identical. They can agree when the relevant Hessian equals Fisher information, or approximately agree in particular statistical settings.
Exponential families and mirror descent
In a regular, minimal exponential family written as
the log-partition function satisfies
The expectation parameters are
so infinitesimally .
This dual coordinate structure connects natural-gradient flow with mirror descent. Mirror descent based on and explicit natural-gradient steps share the same first-order continuous-time behavior, but their finite-step updates need not coincide.
Computation and approximations
For a model with parameters, explicitly storing a dense Fisher matrix requires memory. A general dense factorization can require work, making exact natural gradients expensive for large models.
An explicit inverse is usually unnecessary. Instead, the direction can be found by solving the linear system
Iterative methods can use Fisher–vector products without storing the full matrix. Such products can often be implemented with automatic differentiation and model sampling.
Common approximations include:
- Diagonal approximations, which retain separate parameter scales but discard correlations.
- Block-diagonal approximations, which retain selected groups of interactions.
- Kronecker-factored approximations, which represent suitable matrix blocks through smaller factors, particularly in neural networks.
- Low-rank approximations, which retain a restricted set of directions.
- Stochastic estimates, which approximate Fisher expectations from samples.
For conditional models , Fisher information also depends on how inputs are weighted. A common definition averages over a fixed input distribution and then over .
This differs from the frequently used empirical Fisher, formed by outer products of log-likelihood gradients evaluated at observed labels. The empirical Fisher is not generally the model Fisher. Its statistical meaning and optimization behavior can differ even though both matrices are positive semidefinite.
Applications and historical development
The modern natural-gradient formulation was developed and popularized by Shun-ichi Amari in the 1990s, building on Fisher information, statistical estimation, and information geometry. Its relationship to Fisher scoring connects it to earlier statistical optimization.
Important applications include:
- Neural-network optimization. Approximate Fisher geometry can rescale and couple updates in artificial neural networks.
- Variational inference. Natural gradients can exploit exponential-family structure when optimizing a variational approximation, including objectives expressed through the evidence lower bound.
- Policy optimization. For stochastic decision policies, Fisher information measures local changes in action distributions. Natural policy-gradient methods use this geometry to optimize expected return.
- Distribution-based search. An objective over candidate solutions can induce an objective over a parameterized search distribution, whose parameters are then updated by natural gradients.
Limitations and qualifications
Singular geometry. Fisher information can be singular because of redundant parameters, model symmetries, insufficiently informative sampling, or boundary effects. Such cases are related to statistical identification. A generalized inverse may provide a usable direction, but its interpretation and coordinate behavior require care; a regular identifiable representation or a suitable quotient geometry may be needed.
Damping. Practical implementations often solve
to improve conditioning. This can increase numerical stability, but adding the coordinate identity matrix generally breaks full reparameterization invariance. Damping is not merely an implementation detail: it changes the metric or local optimization problem.
Estimation error. Noisy curvature estimates can produce unstable or ineffective directions, especially when inversion amplifies errors along weakly informed directions.
No universal convergence guarantee. Natural gradients do not remove nonconvexity or guarantee global optima. Their behavior still depends on step sizes, smoothness, stochastic noise, and the quality of metric approximations.
Dependence on the statistical model. Fisher geometry measures distinguishability according to the chosen model and sampling distribution. It need not coincide with application-specific notions of similarity, predictive value, or computational cost.