{
  "id": "natural-gradient",
  "title": "Natural Gradient",
  "lang": "en",
  "source_lang": "en",
  "summary": "The natural gradient is a geometry-aware optimization direction that measures changes in a statistical model using Fisher information rather than Euclidean parameter distance.",
  "category": "mathematics",
  "subcategory": "applied",
  "type": "concept",
  "aliases": [
    "Fisher natural gradient",
    "information-geometric gradient"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/natural-gradient/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/natural-gradient/",
      "json": "https://aiwiki.page/zh/natural-gradient.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The natural gradient is a generalization of the ordinary gradient that accounts for the geometry of the space being optimized. In statistical applications, it uses Fisher information to measure how much a parameter change alters a probability distribution, rather than measuring only the numerical change in its parameters. It provides an optimization direction that is invariant under smooth, invertible reparameterizations at the infinitesimal level. Natural-gradient methods are used in statistical estimation, machine learning, and probabilistic inference."
    },
    {
      "heading": "Geometric definition",
      "text": "For a differentiable objective function \\(L(\\theta)\\), ordinary gradient descent updates parameters according to\n\n\\[\\theta_{t+1}=\\theta_t-\\eta_t\\nabla_\\theta L(\\theta_t),\\]\n\nwhere \\(\\eta_t>0\\) is the step size. This direction is steepest descent when the size of a parameter displacement is measured by its standard Euclidean norm.\n\nThat choice of distance is not intrinsic to a statistical model. Two parameterizations can describe exactly the same distributions while assigning very different Euclidean lengths to corresponding changes. A parameter measured in different units, for example, can produce a substantially different ordinary-gradient trajectory.\n\nOn a manifold equipped with a positive-definite metric tensor \\(G(\\theta)\\), the squared length of an infinitesimal displacement \\(d\\theta\\) is\n\n\\[ds^2=d\\theta^{\\mathsf T}G(\\theta)d\\theta.\\]\n\nThe metric defines an inner product on each tangent space. The gradient with respect to this metric, denoted \\(\\operatorname{grad}_G L\\), is the unique vector satisfying\n\n\\[\\langle \\operatorname{grad}_G L,v\\rangle_G = dL[v]\\]\n\nfor every tangent vector \\(v\\). In coordinates,\n\n\\[\\operatorname{grad}_G L = G(\\theta)^{-1}\\nabla_\\theta L.\\]\n\nIn a statistical model, choosing the Fisher information matrix as \\(G\\) gives the natural gradient:\n\n\\[\\widetilde{\\nabla}_\\theta L = F(\\theta)^{-1}\\nabla_\\theta L.\\]\n\nThe associated descent update is\n\n\\[\\theta_{t+1} = \\theta_t-\\eta_t\\widetilde{\\nabla}_\\theta L(\\theta_t).\\]\n\nThe natural gradient itself points toward increasing \\(L\\); its negative is used for minimization. More broadly, the term can refer to a gradient defined using another explicitly chosen Riemannian metric, although the Fisher metric is the standard choice in statistics."
    },
    {
      "heading": "Fisher information and statistical distance",
      "text": "Consider a smooth family of densities or mass functions \\(p(x;\\theta)\\). Its score vector is\n\n\\[s_\\theta(x)=\\nabla_\\theta\\log p(x;\\theta).\\]\n\nThe Fisher information matrix is\n\n\\[F(\\theta) = \\mathbb E_{x\\sim p(\\cdot;\\theta)} \\left[s_\\theta(x)s_\\theta(x)^{\\mathsf T}\\right].\\]\n\nUnder suitable regularity conditions, including conditions that justify differentiation under the integral and appropriate behavior of the distribution's support,\n\n\\[F(\\theta) = -\\mathbb E_{x\\sim p(\\cdot;\\theta)} \\left[\\nabla_\\theta^2\\log p(x;\\theta)\\right].\\]\n\nFisher information measures local statistical distinguishability: a direction with large Fisher length changes the distribution strongly, even if its parameter displacement is numerically small. This interpretation is central to information geometry.\n\nThe connection is made precise by the local expansion of Kullback–Leibler divergence:\n\n\\[D_{\\mathrm{KL}} \\bigl(p(\\cdot;\\theta)\\,\\|\\,p(\\cdot;\\theta+d\\theta)\\bigr) = \\frac12 d\\theta^{\\mathsf T}F(\\theta)d\\theta + o(\\|d\\theta\\|^2).\\]\n\nAlthough KL divergence is generally asymmetric and is not a distance metric, its local second-order term defines the Fisher metric. Reversing the two distributions gives the same quadratic term.\n\nThe objective \\(L\\) need not be a likelihood. Any differentiable objective defined on the statistical model can be optimized using this metric. The model determines the geometry; the objective determines which changes are desirable."
    },
    {
      "heading": "Steepest descent under a distributional constraint",
      "text": "Let \\(g=\\nabla_\\theta L(\\theta)\\). A local approximation to minimizing the objective while limiting the change in the distribution is\n\n\\[\\min_\\delta g^{\\mathsf T}\\delta \\quad\\text{subject to}\\quad \\frac12\\delta^{\\mathsf T}F(\\theta)\\delta\\leq\\varepsilon.\\]\n\nWhen \\(F\\) is positive definite and \\(g\\neq0\\), the solution is\n\n\\[\\delta^* = -\\sqrt{ \\frac{2\\varepsilon}{g^{\\mathsf T}F^{-1}g} }\\,F^{-1}g.\\]\n\nThus the negative natural gradient is the direction of greatest first-order decrease for a fixed infinitesimal statistical displacement.\n\nAn equivalent penalized local problem is\n\n\\[\\min_\\delta \\left[ g^{\\mathsf T}\\delta+ \\frac{1}{2\\eta}\\delta^{\\mathsf T}F(\\theta)\\delta \\right],\\]\n\nwhose solution is \\(\\delta=-\\eta F^{-1}g\\).\n\nThese are local statements. A finite natural-gradient step does not, by itself, solve an optimization problem with an exact KL constraint. Large steps can violate the local approximation or leave the valid parameter domain."
    },
    {
      "heading": "Reparameterization invariance",
      "text": "Suppose \\(\\theta=\\phi(\\xi)\\) is a smooth, invertible change of coordinates, with Jacobian matrix\n\n\\[J=\\frac{\\partial\\theta}{\\partial\\xi}.\\]\n\nThe ordinary coordinate derivatives and Fisher matrix transform as\n\n\\[g_\\xi=J^{\\mathsf T}g_\\theta, \\qquad F_\\xi=J^{\\mathsf T}F_\\theta J.\\]\n\nConsequently,\n\n\\[F_\\xi^{-1}g_\\xi = J^{-1}F_\\theta^{-1}g_\\theta.\\]\n\nThis is the transformation law for a tangent vector: both coordinate systems describe the same natural-gradient direction on the model manifold. An ordinary coordinate gradient does not generally have this property.\n\nThe distinction between a direction and a finite update is important. The continuous-time natural-gradient flow,\n\n\\[\\frac{d\\theta}{dt} = -F(\\theta)^{-1}\\nabla_\\theta L(\\theta),\\]\n\nis coordinate invariant under the stated assumptions. A discrete update implemented by coordinate addition is generally invariant only to first order in the step size. Exactly invariant finite movement requires an intrinsic update rule, such as an appropriate geodesic construction, or an equivalent coordinate-independent procedure."
    },
    {
      "heading": "Example: a Bernoulli distribution",
      "text": "Let \\(X\\) have a Bernoulli distribution with success probability \\(p\\), where \\(0<p<1\\). In the probability coordinate,\n\n\\[F(p)=\\frac{1}{p(1-p)}.\\]\n\nFor an objective \\(L(p)\\), the natural gradient is therefore\n\n\\[\\widetilde{\\nabla}_p L = p(1-p)\\frac{dL}{dp}.\\]\n\nNow introduce the logit coordinate\n\n\\[a=\\log\\frac{p}{1-p}, \\qquad \\frac{dp}{da}=p(1-p).\\]\n\nIn this coordinate,\n\n\\[F(a)=p(1-p), \\qquad \\frac{dL}{da} = p(1-p)\\frac{dL}{dp},\\]\n\nso\n\n\\[\\widetilde{\\nabla}_a L=\\frac{dL}{dp}.\\]\n\nThe resulting infinitesimal probability change is identical in either coordinate:\n\n\\[dp = -\\eta\\,p(1-p)\\frac{dL}{dp}.\\]\n\nFor the negative log-likelihood of one observation \\(y\\in\\{0,1\\}\\),\n\n\\[L(p)=-y\\log p-(1-y)\\log(1-p),\\]\n\nthe probability-coordinate natural gradient simplifies to\n\n\\[\\widetilde{\\nabla}_p L=p-y.\\]\n\nThe corresponding update moves \\(p\\) toward the observation. Behavior at the boundary requires separate treatment because the regular interior geometry does not extend there unchanged."
    },
    {
      "heading": "Relationship to other optimization methods",
      "text": "Newton's method and Fisher scoring\n\nNewton's method uses the Hessian matrix of the objective:\n\n\\[\\delta_{\\mathrm{Newton}} = -\\bigl[\\nabla^2L(\\theta)\\bigr]^{-1}\\nabla L(\\theta).\\]\n\nThe Hessian describes curvature of the objective in the chosen coordinates. Fisher information instead describes the geometry of the statistical model and is positive semidefinite wherever its defining expectation exists.\n\nFor negative log-likelihood optimization, Fisher scoring replaces the observed Hessian with its model expectation. With consistent scaling between the likelihood gradient and Fisher information, this yields a natural-gradient step. Natural gradients therefore extend a familiar statistical procedure to objectives beyond maximum likelihood estimation.\n\nThe methods are not generally identical. They can agree when the relevant Hessian equals Fisher information, or approximately agree in particular statistical settings.\n\nExponential families and mirror descent\n\nIn a regular, minimal exponential family written as\n\n\\[p(x;\\theta) = h(x)\\exp\\!\\left( \\theta^{\\mathsf T}T(x)-A(\\theta) \\right),\\]\n\nthe log-partition function \\(A\\) satisfies\n\n\\[F(\\theta)=\\nabla^2A(\\theta).\\]\n\nThe expectation parameters are\n\n\\[\\mu=\\nabla A(\\theta)=\\mathbb E_\\theta[T(X)],\\]\n\nso infinitesimally \\(d\\mu=F(\\theta)d\\theta\\).\n\nThis dual coordinate structure connects natural-gradient flow with mirror descent. Mirror descent based on \\(A\\) and explicit natural-gradient steps share the same first-order continuous-time behavior, but their finite-step updates need not coincide."
    },
    {
      "heading": "Computation and approximations",
      "text": "For a model with \\(d\\) parameters, explicitly storing a dense Fisher matrix requires \\(O(d^2)\\) memory. A general dense factorization can require \\(O(d^3)\\) work, making exact natural gradients expensive for large models.\n\nAn explicit inverse is usually unnecessary. Instead, the direction \\(v\\) can be found by solving the linear system\n\n\\[Fv=g.\\]\n\nIterative methods can use Fisher–vector products without storing the full matrix. Such products can often be implemented with automatic differentiation and model sampling.\n\nCommon approximations include:\n\n- Diagonal approximations, which retain separate parameter scales but discard correlations.\n- Block-diagonal approximations, which retain selected groups of interactions.\n- Kronecker-factored approximations, which represent suitable matrix blocks through smaller factors, particularly in neural networks.\n- Low-rank approximations, which retain a restricted set of directions.\n- Stochastic estimates, which approximate Fisher expectations from samples.\n\nFor conditional models \\(p_\\theta(y\\mid x)\\), Fisher information also depends on how inputs \\(x\\) are weighted. A common definition averages over a fixed input distribution and then over \\(y\\sim p_\\theta(\\cdot\\mid x)\\).\n\nThis differs from the frequently used empirical Fisher, formed by outer products of log-likelihood gradients evaluated at observed labels. The empirical Fisher is not generally the model Fisher. Its statistical meaning and optimization behavior can differ even though both matrices are positive semidefinite."
    },
    {
      "heading": "Applications and historical development",
      "text": "The modern natural-gradient formulation was developed and popularized by Shun-ichi Amari in the 1990s, building on Fisher information, statistical estimation, and information geometry. Its relationship to Fisher scoring connects it to earlier statistical optimization.\n\nImportant applications include:\n\n- Neural-network optimization. Approximate Fisher geometry can rescale and couple updates in artificial neural networks.\n- Variational inference. Natural gradients can exploit exponential-family structure when optimizing a variational approximation, including objectives expressed through the evidence lower bound.\n- Policy optimization. For stochastic decision policies, Fisher information measures local changes in action distributions. Natural policy-gradient methods use this geometry to optimize expected return.\n- Distribution-based search. An objective over candidate solutions can induce an objective over a parameterized search distribution, whose parameters are then updated by natural gradients."
    },
    {
      "heading": "Limitations and qualifications",
      "text": "Singular geometry. Fisher information can be singular because of redundant parameters, model symmetries, insufficiently informative sampling, or boundary effects. Such cases are related to statistical identification. A generalized inverse may provide a usable direction, but its interpretation and coordinate behavior require care; a regular identifiable representation or a suitable quotient geometry may be needed.\n\nDamping. Practical implementations often solve\n\n\\[(F+\\lambda I)v=g, \\qquad \\lambda>0,\\]\n\nto improve conditioning. This can increase numerical stability, but adding the coordinate identity matrix generally breaks full reparameterization invariance. Damping is not merely an implementation detail: it changes the metric or local optimization problem.\n\nEstimation error. Noisy curvature estimates can produce unstable or ineffective directions, especially when inversion amplifies errors along weakly informed directions.\n\nNo universal convergence guarantee. Natural gradients do not remove nonconvexity or guarantee global optima. Their behavior still depends on step sizes, smoothness, stochastic noise, and the quality of metric approximations.\n\nDependence on the statistical model. Fisher geometry measures distinguishability according to the chosen model and sampling distribution. It need not coincide with application-specific notions of similarity, predictive value, or computational cost."
    }
  ],
  "links": [
    "gradient",
    "fisher-information",
    "probability-distribution",
    "machine-learning",
    "objective-function",
    "gradient-descent",
    "manifold",
    "metric-tensor",
    "inner-product",
    "tangent-space",
    "kullback-leibler-divergence",
    "jacobian-matrix",
    "geodesic",
    "bernoulli-distribution",
    "logit",
    "newtons-method",
    "hessian-matrix",
    "maximum-likelihood-estimation",
    "system-of-linear-equations",
    "automatic-differentiation",
    "artificial-neural-network",
    "variational-inference",
    "evidence-lower-bound",
    "statistical-identification",
    "numerical-stability"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}