Fisher Information

Fisher information is a mathematical measure of how sensitively a probability distribution changes when an unknown parameter changes. It is defined through the expected squared derivative of the log-likelihood function, or through a matrix of such expectations when several parameters are involved. Under suitable regularity conditions, it also equals the expected negative curvature of the log-likelihood. It describes a property of a statistical model at a specified parameter value, rather than the information contained in one particular realized observation. (stat210a.berkeley.edu)

Its central importance in statistics comes from the connection between this sensitivity and estimation precision. The Cramér–Rao inequality uses inverse Fisher information to bound the variance of unbiased estimators, while large-sample theory uses it to describe the uncertainty of maximum-likelihood estimates. Fisher information also supports invariant prior construction and the geometric interpretation of statistical models. (arxiv.org)

Named after Ronald Fisher, the concept developed alongside likelihood, sufficiency, and efficiency in twentieth-century statistical theory. Its uses now extend to experimental design, optimization in machine learning, and quantum parameter estimation. The distinctions between expected, observed, and empirical information—and the assumptions underlying each—are essential to these applications. (digital.library.adelaide.edu.au)

Score Functions and the Definition of Information

Let $\{P_\theta:\theta\in\Theta\}$ be a dominated parametric model, with scalar parameter $\theta$ in an open subset $\Theta\subseteq\mathbb R$. Dominated means that all model distributions have densities $p(x\mid\theta)$ relative to one parameter-independent measure $\mu$: counting measure for discrete observations, or typically Lebesgue measure for continuous observations. For a fixed observation $x$, the log-likelihood is $\ell(\theta;x)=\log p(x\mid\theta)$. Wherever the density is positive and differentiable, the score function is

Differentiation concerns the parameter, not the observation: $x$ remains fixed as $\theta$ varies. The score is thus the local rate of change of the log-likelihood; $s_\theta(X)$ becomes a random variable when the observation is drawn from $P_\theta$. (stat210a.berkeley.edu)

The scalar Fisher information is the expected value of the squared score,

For a probability mass function on a countable support $\mathcal X$, this becomes

for a probability density function on a continuous support, it becomes

These expressions measure squared local sensitivity, averaged over observations generated at the same parameter value where the derivative is evaluated. They are nonnegative; finiteness requires the score to be square-integrable. (stat.cmu.edu)

The score has mean zero under suitable regularity conditions. A standard sufficient setup assumes a parameter-independent support, positive densities there, and differentiability in a neighborhood of the parameter under consideration. An integrable bound on $\lvert\partial_\theta p(x\mid\theta)\rvert$, uniform over that neighborhood, permits differentiation under the integral through the dominated convergence theorem. Differentiating the normalization identity then gives

Consequently, $I(\theta)=\operatorname{Var}_\theta[s_\theta(X)]$: information is the variance of the score, not merely its second moment. The matching parameter values are essential; a score evaluated at $\theta$ need not have zero expectation under $P_{\theta'}$ when $\theta'\ne\theta$. Parameter-dependent support can invalidate this calculation by introducing boundary terms. (stat210a.berkeley.edu)

A curvature representation needs additional assumptions: twice-differentiable densities, justification for interchanging the second derivative and integration, and integrability of the relevant quantities. For example, an integrable neighborhood-wise bound on $\lvert\partial_\theta^2p(x\mid\theta)\rvert$, together with finite information, suffices in the common-support setting. Direct differentiation yields

Taking expectations and using $\int\partial_\theta^2p(x\mid\theta)\,d\mu(x)=\partial_\theta^2 1=0$ therefore establishes

This equality is a regular-model identity, not an unconditional alternative definition valid for every likelihood. (stat.cmu.edu)

Expected information $I(\theta)$ averages over possible data. By contrast, observed information

is calculated from the realized observation and generally depends on it. Under the curvature identity, $\mathbb E_\theta[j(\theta;X)]=I(\theta)$, but the two need not agree for an individual sample. Nor must observed information be nonnegative at every parameter value: it is negative wherever that particular log-likelihood has positive second derivative, whereas expected information remains nonnegative by its squared-score definition. (stat.umn.edu)

Multiparameter Information and Nuisance Parameters

For a parameter vector $\theta=(\theta_1,\ldots,\theta_d)^{\mathsf T}$, the score is the column vector of parameter derivatives of the log-likelihood:

Assume that the score has finite second moments and that differentiation under the integral is valid, giving $E_\theta[s_\theta]=0$. Then $I$ is its covariance matrix: $I_{jj}=E_\theta[s_j^2]$ measures sensitivity to $\theta_j$ with the remaining coordinates fixed, while $I_{jk}=E_\theta[s_js_k]$ records cross-parameter score covariance. These cross-terms describe overlap between likelihood responses to different parameter changes, not correlations between the parameters themselves. (stat210a.berkeley.edu)

The matrix is symmetric and positive semidefinite because, for every $a\in\mathbb R^d$,

Here $a^{\mathsf T}s_\theta$ is the directional score along $\theta+ta$. In fixed parameter units, eigenvalues therefore quantify sensitivity along their eigenvectors. A zero eigenvalue identifies a direction whose score vanishes almost surely: the model has no first-order sensitivity there. This local statement does not establish that distinct parameter values generate identical distributions. Eigenvalue magnitudes also depend on the chosen parameter coordinates. (stat210a.berkeley.edu)

Partition $\theta=(\psi^{\mathsf T},\lambda^{\mathsf T})^{\mathsf T}$, where $\psi$ is the inferential target and $\lambda$ contains nuisance parameters, and write

If $\lambda$ is known, the relevant information is $I_{\psi\psi}$. If it must be estimated, changes in $\lambda$ can partly reproduce the likelihood response to changes in $\psi$. The information available for distinguishing the target from these nuisance changes is consequently different from the target’s diagonal block alone. (web.stanford.edu)

When $I_{\lambda\lambda}$ is invertible, remove the linear projection of the target score onto the nuisance scores:

Indeed,

Expanding the residual’s covariance gives the efficient information

the Schur complement of the nuisance block. The subtracted matrix is positive semidefinite, so estimating nuisance parameters cannot increase information relative to knowing them. For a nonsingular full matrix, block inversion yields $(I^{-1})_{\psi\psi}=I_{\mathrm{eff}}^{-1}$, rather than $I_{\psi\psi}^{-1}$. (sciencedirect.com)

A numerical illustration makes the distinction explicit. Suppose the information for two scalar parameters is

with the first parameter of interest. Its information is $4$ when the second parameter is known, but $4-2^2/3=8/3$ when the second must be estimated. Correspondingly, the target entry of the inverse full matrix is $3/8$, whereas the inverse known-nuisance information is $1/4$. These are consequences of the block-inverse identity; their interpretation as variance bounds or asymptotic covariance scales requires the relevant estimation assumptions. (web.stanford.edu)

The blocks are information orthogonal when $I_{\psi\lambda}=0$, in which case $I_{\mathrm{eff}}=I_{\psi\psi}$. Orthogonality at one parameter value is a pointwise property; orthogonality throughout a model depends on its parameterization. For example, the mean and standard deviation of a normal distribution have zero cross-information. Thus estimating the standard deviation does not reduce efficient information about the mean, although unknown scale still affects finite-sample confidence procedures. Information orthogonality means uncorrelated score blocks, not necessarily independent scores or independent estimators. (arxiv.org)

Worked Models and Parameterization Choices

For one observation $X$ from a Bernoulli distribution, with $P(X=1)=p$ and $0<p<1$, the log-likelihood and score are

Since $\operatorname{Var}(X)=p(1-p)$, taking the expected square of the score gives

For a count $K$ having a binomial distribution with known trial number $n$, the score is $(K-np)/[p(1-p)]$. Its variance, using $\operatorname{Var}(K)=np(1-p)$, yields $I_p(p)=n/[p(1-p)]$, the same information as the underlying $n$ independent Bernoulli trials. (pages.stat.wisc.edu)

In log-odds coordinates, $\eta=\log[p/(1-p)]$, the Bernoulli log-likelihood becomes $x\eta-\log(1+e^\eta)$. Differentiation now gives $s_\eta(x)=x-p$, and therefore

Thus information in probability coordinates diverges as $p$ approaches either endpoint, whereas information in log-odds coordinates tends to zero. This difference reflects the coordinate scale: $dp/d\eta=p(1-p)$, so the chain rule gives $I_\eta=(dp/d\eta)^2I_p$. For example, at $p=1/2$, the respective one-trial values are $4$ and $1/4$. These are different descriptions of the same experiment, not contradictory assessments of its information; the formulas concern interior probabilities, not the boundary models themselves. (stat.berkeley.edu)

For $X$ from a Poisson distribution with mean $\lambda>0$,

Because $\operatorname{Var}(X)=\lambda$, $I_\lambda(\lambda)=1/\lambda$. If instead $\eta=\log\lambda$, then $\ell(\eta;x)=x\eta-e^\eta-\log(x!)$, the score is $x-\lambda$, and $I_\eta(\eta)=\lambda$. Increasing $\lambda$ consequently decreases information about additive changes in the mean but increases information about changes on the logarithmic, or relative, scale. Here $\lambda$ is the expected count over the specified observation interval; identifying it with a rate assumes a unit interval. (stat.berkeley.edu)

For one normal observation $X\sim N(\mu,\sigma^2)$, with $\sigma>0$, differentiating the log-density gives

Writing $Z=(X-\mu)/\sigma$, these scores are $Z/\sigma$ and $(Z^2-1)/\sigma$. Using $E(Z^2)=1$, $E(Z^3)=0$, and $E(Z^4)=3$, their expected products give

The cross-entry vanishes because $E[Z(Z^2-1)]=0$. In variance coordinates $v=\sigma^2$, the second score becomes $[(X-\mu)^2-v]/(2v^2)$, giving

In particular, information about standard deviation is $2/\sigma^2$, whereas information about variance is $1/(2\sigma^4)$; these quantities must not be interchanged. (stat.berkeley.edu)

These calculations share a general mechanism in a regular exponential family expressed in natural coordinates:

Here $T(X)$ is the canonical sufficient statistic, and $A$ normalizes the density. At interior natural-parameter values, differentiation under the normalizing integral yields

The score is therefore the centered statistic $T(X)-E_\eta[T(X)]$, and information is both its covariance matrix and the Hessian of $A$. Bernoulli and Poisson models have $A(\eta)=\log(1+e^\eta)$ and $A(\eta)=e^\eta$, respectively. The Hessian identity specifically requires natural coordinates: differentiating $A(\eta(\theta))$ twice in an arbitrary parameter $\theta$ does not generally produce its information matrix. (stat210a.berkeley.edu)

The Cramér–Rao Inequality and Efficiency

Let $X$ denote the entire observed sample, with joint density or mass function $p_\theta(x)$, and let $S_\theta(X)=\partial_\theta\log p_\theta(X)$ be its score. Assume an interior scalar parameter, parameter-independent support, differentiation under the integral or sum for both normalization and the estimator’s expectation, and finite second moments with $0<I(\theta)<\infty$. For a parameter-independent estimator $T(X)$ satisfying $E_\theta[T]=g(\theta)$ throughout a neighborhood, differentiation gives the covariance identity

where the last equality uses $E_\theta[S_\theta]=0$; in a discrete model, replace the integral by a sum. The Cauchy–Schwarz inequality then yields

This is the Cramér–Rao inequality, also called the information inequality. (stat210a.berkeley.edu)

The information in this bound is that of the whole sample, not one observation. For $n$ independent, identically distributed observations with information $I_1(\theta)$, it becomes $[g'(\theta)]^2/[nI_1(\theta)]$. For example, if $X_i\sim N(\mu,\sigma^2)$ with known $\sigma^2$, the sample score is $S_\mu=n(\bar X-\mu)/\sigma^2$, and the information is $n/\sigma^2$. The sample mean has variance $\sigma^2/n$, exactly attaining the bound for estimating $\mu$. This is an exact finite-sample result, not a large-sample approximation. (stat.cmu.edu)

For a $d$-dimensional parameter vector $\theta$, suppose $E_\theta[T]=\theta$, the corresponding differentiation identities hold componentwise, and the sample information matrix $I(\theta)=E_\theta[SS^{\mathsf T}]$ is finite and nonsingular. Then

The ordering means that $\operatorname{Cov}_\theta(T)-I(\theta)^{-1}$ is positive semidefinite: every linear combination $a^{\mathsf T}T$ has variance at least $a^{\mathsf T}I(\theta)^{-1}a$. Indeed, the covariance matrix of the residual $T-\theta-I(\theta)^{-1}S$ equals this difference, providing a direct proof. It is not an entrywise inequality between matrices. (stat.cmu.edu)

In the scalar case, equality holds precisely when

Thus, the centered estimator must be proportional to the score; matrix equality requires $T-\theta=I(\theta)^{-1}S$ almost surely. For a nonzero bound and positive estimator variance, finite-sample efficiency can be expressed as $e_\theta(T)=[g'(\theta)]^2/\{I(\theta)\operatorname{Var}_\theta(T)\}$. An unbiased estimator attaining equality throughout the parameter space is called efficient. The bound need not be attainable, however: failure to attain it does not establish that a better unbiased estimator exists. Finite-sample attainment is distinct from asymptotic efficiency, which concerns limiting performance as sample size increases. (stat210a.berkeley.edu)

For a biased estimator of $g(\theta)$, define its bias by $b(\theta)=E_\theta[T]-g(\theta)$. Applying the covariance argument to its actual expectation gives

For estimating $\theta$ itself, the numerator is $[1+b'(\theta)]^2$. The mean squared error includes squared bias as well as variance, so a biased estimator may have MSE below the unbiased Cramér–Rao bound without contradiction. Unbiasedness at a single parameter value is insufficient for the unbiased formula: the derivative of the expectation, and therefore of the bias, also matters. (cpb-us-e1.wpmucdn.com)

Likelihood Inference in Large Samples

For independent, identically distributed observations $X_1,\ldots,X_n$ from a correctly specified, identifiable model with a fixed-dimensional parameter $\theta\in\mathbb R^d$, a regular maximum-likelihood estimator satisfies

Here $\theta_0$ is the true parameter, $I_1(\theta_0)$ is expected information per observation, and $\Rightarrow$ denotes convergence in distribution. Thus the large-sample normal approximation has covariance $I_1(\theta_0)^{-1}/n$, not $I_1(\theta_0)^{-1}$. The theorem requires more than differentiability: typical assumptions include consistency of the selected maximizer, an interior true parameter, finite nonsingular information, and sufficient control of likelihood derivatives near $\theta_0$. These conditions exclude routine application at parameter boundaries or in singular models. (stat210a.berkeley.edu)

The mechanism is a local expansion of the likelihood equation. Writing $\ell_n(\theta)=\sum_{i=1}^n\log p(X_i\mid\theta)$ and $U_n=\nabla\ell_n$, regularity yields

where $o_p(1)$ vanishes in probability. The central limit theorem makes the normalized score asymptotically normal with covariance $I_1(\theta_0)$, while normalized negative likelihood curvature converges to that same matrix. Consistency keeps the expansion in the appropriate neighborhood; nonsingularity permits inversion. The resulting inverse-information covariance expresses asymptotic efficiency among regular estimators. It is not a claim of exact unbiasedness, minimum finite-sample mean-squared error, or superiority to every estimator at an isolated parameter value. (stat210a.berkeley.edu)

Because $\theta_0$ is unknown, two standard approximations to the estimator’s covariance matrix are

The first substitutes the estimate into expected information; the second uses the observed likelihood Hessian. Under the relevant continuity and convergence assumptions, both have the same first-order asymptotic limit, although they need not agree for a particular sample. Estimated standard errors are square roots of diagonal entries of either matrix. With several jointly estimated parameters, the relevant quantity is a diagonal entry of the inverse full matrix, not generally the reciprocal of a diagonal information entry. (stat.cmu.edu)

For a scalar parameter, a Wald confidence interval is $\hat\theta_n\pm z_{1-\alpha/2}\sqrt{\widehat V}$, where $z_{1-\alpha/2}$ is a standard-normal quantile. For the simple null hypothesis $H_0:\theta=\theta_\ast$, the vector Wald statistic is

Inverting this test gives an approximate confidence ellipsoid. For example, in a Bernoulli sample with $n=400$ and $\hat p=0.30$, the information-based standard error is $\sqrt{0.30(0.70)/400}\approx0.0229$; the approximate 95% Wald interval is $0.255$ to $0.345$. This calculation illustrates the approximation rather than establishing exact 95% coverage. (stat210a.berkeley.edu)

Information also standardizes the score under the null:

Unlike the Wald statistic, this score statistic uses derivatives evaluated at the null rather than at the unrestricted estimate. The likelihood-ratio statistic,

compares unrestricted and null-constrained maximizers. For a smooth nested null imposing $r$ independent restrictions, its limiting distribution is $\chi_r^2$; corresponding Wald and score tests have the same first-order asymptotic behavior. Information enters this result through the local quadratic likelihood expansion. With nuisance parameters, the score calculation uses restricted estimates and information adjusted for their estimation. Despite asymptotic equivalence, these tests can yield different finite-sample results, and neither their chi-squared calibration nor confidence-region coverage is an exact general guarantee. (katsevich-teaching.github.io)

Combining Observations and Preserving Information

For independent observations $X_1,\ldots,X_n$, possibly with different distributions $p_i(x_i\mid\theta)$, the joint likelihood factors and its score is the sum of the individual scores:

Under regularity conditions giving each score zero mean and finite second moments, independence makes the cross terms $E_\theta[s_i s_j^{\mathsf T}]$ vanish for $i\ne j$. Consequently,

Identically distributed observations give the special case $I_n=nI_1$. With dependent observations, the joint distribution must instead determine the score: summing marginal information generally omits the dependence structure. For example, recording the same random outcome twice does not provide two independent observations or double its information. These conclusions follow from likelihood factorization and score covariance, rather than from sample size alone. (stat210a.berkeley.edu)

Let $T$ be a parameter-independent reduction of data $X$, including a randomized observation channel with conditional density $k(t\mid x)$ independent of $\theta$. Its density is $q_\theta(t)=\int k(t\mid x)p_\theta(x)\,dx$. If differentiation can pass through this integral and the score is square-integrable, then

Thus the reduced-data score is the conditional expectation of the original score, not generally its value at an estimated or reconstructed observation. A deterministic statistic $T=T(X)$ satisfies the same identity, interpreted through its induced distribution. The conditional-score identity also holds under the more general regularity framework of differentiability in quadratic mean. (arxiv.org)

The conditional covariance decomposition now yields

Data reduction therefore cannot increase expected information in any parameter direction. Equality holds precisely when the score is determined by $T$, almost surely at the parameter value concerned. A sufficient statistic preserves information because the conditional distribution of $X$ given $T$ is parameter-independent. For independent Bernoulli trials, retaining only the success count preserves the score $(T-np)/[p(1-p)]$ and hence the information $n/[p(1-p)]$. Preservation alone, however, need not imply sufficiency: Abram Kagan and Lawrence Shepp’s 2005 example uses a binary label and a continuous location observation whose component densities occupy opposite sides of the location parameter. Discarding the label preserves Fisher information, although its conditional distribution still depends on the parameter. Additional assumptions are therefore needed for the converse implication. (arxiv.org)

For incomplete observations, write the complete data as $X=(Y,Z)$, where $Y$ is observed and $Z$ is missing or latent. If $s_c$ is the complete-data score, the observed-data score is $s_o(Y)=E_\theta[s_c\mid Y]$. The same decomposition gives

Here $I_m$ measures the score uncertainty remaining after observing $Y$. All three quantities are expected information matrices; $I_o$ is not the sample-specific negative Hessian. (arxiv.org)

Thomas A. Louis’s 1982 method supplies the corresponding sample-specific identity. With $\ell_c(\theta;Y,Z)=\log p_\theta(Y,Z)$, $H_c=-\nabla_\theta^2\ell_c$, and observed log-likelihood $\ell_o(\theta;y)$,

The conditional covariance includes subtraction of the conditional score’s outer product; that score vanishes at an interior stationary likelihood estimate. After fitting with the expectation–maximization algorithm, Louis’s identity obtains observed information from complete-data derivatives and conditional moments of latent variables, without directly differentiating the marginal likelihood. The conditional expected complete-data curvature alone is insufficient: the missing-score covariance must also be subtracted. (markirwin.net)

Statistical Geometry and Bayesian Connections

Let $\theta=h(\eta)$ be a smooth, locally invertible change of coordinates for a $k$-parameter model, and let $J=\partial\theta/\partial\eta$ be its Jacobian matrix. The chain rule gives $s_\eta=J^{\mathsf T}s_\theta$, so taking expected score outer products yields

Information therefore does not have coordinate-independent numerical entries: in one dimension, $I_\eta=I_\theta(d\theta/d\eta)^2$. For example, expressing a normal mean in centimeters rather than meters divides its information by $10^4$, because $\mu_{\mathrm m}=\mu_{\mathrm{cm}}/100$. Nevertheless, the quadratic form is unchanged: $d\eta^{\mathsf T}I_\eta d\eta=d\theta^{\mathsf T}I_\theta d\theta$. Changing units changes the coordinate description, not the statistical sensitivity represented by that form. (pmc.ncbi.nlm.nih.gov)

On a smooth, regular, identifiable statistical manifold with finite, positive-definite information, this transformation law makes $I(\theta)$ a metric tensor. The resulting Fisher–Rao metric assigns tangent vectors $u,v$ the inner product $g_\theta(u,v)=u^{\mathsf T}I(\theta)v$, with infinitesimal line element

A curve $\theta(t)$ consequently has length $\int\sqrt{\dot\theta(t)^{\mathsf T}I(\theta(t))\dot\theta(t)}\,dt$; minimizing lengths over connecting curves defines the associated geodesic distance. Positive definiteness is essential here: identifiability alone does not rule out degeneracy at exceptional parameter values. Thus a singular information matrix does not define an ordinary Riemannian metric on the full coordinate space. (pmc.ncbi.nlm.nih.gov)

The metric also describes the local behavior of Kullback–Leibler divergence. Using natural logarithms, set $F(\delta)=E_\theta[\log p_\theta(X)-\log p_{\theta+\delta}(X)]$. Under common-support, differentiability, and integrability assumptions that justify differentiating the expectation, $F(0)=0$, $\nabla F(0)=-E_\theta[s_\theta]=0$, and $\nabla^2F(0)=-E_\theta[\nabla_\theta^2\log p_\theta]=I(\theta)$. A second-order Taylor expansion therefore gives

Reversing the divergence has the same quadratic term. This is a local relation, not an equality between KL divergence and squared geodesic distance for separated distributions; KL divergence is generally asymmetric. (jmlr.org)

In Bayesian inference, Jeffreys’s prior, proposed by Harold Jeffreys in 1946, uses the metric volume density:

Indeed, the coordinate-transformation formula implies $\sqrt{\det I_\eta}=|\det J|\sqrt{\det I_\theta}$, exactly the Jacobian factor required to transform a probability density. Invariance means that the same measure is obtained in either coordinate system, not that its density has the same numerical expression. For a Bernoulli probability $p$, the normalized density is $1/[\pi\sqrt{p(1-p)}]$. In the coordinate $\phi=2\arcsin\sqrt p$, $0<\phi<\pi$, information equals $1$, so this measure becomes uniform. (arxiv.org)

Metric volume need not be finite. For a normal mean $\mu$ with known variance $\sigma^2$, $I(\mu)=1/\sigma^2$, giving the improper prior $\pi_J(\mu)\propto1$ on the real line. Nevertheless, $n\geq1$ independent observations yield the proper posterior $N(\bar x,\sigma^2/n)$. In general, posterior propriety requires

Reparameterization invariance does not establish this condition. For example, with known normal mean $\mu_0$ and unknown standard deviation, Jeffreys’s density is $1/\sigma$; if every observation equals $\mu_0$, the posterior kernel is $\sigma^{-(n+1)}$, whose integral diverges at zero. This exceptional dataset demonstrates why normalization must be checked separately. (www2.stat.duke.edu)

Numerical Computation and Learning Algorithms

Scores and information matrices can be calculated by analytical differentiation when the likelihood has a tractable form, or by automatic differentiation, which applies the chain rule through a computational graph rather than using finite differences. Reverse-mode differentiation efficiently evaluates the score of a scalar log-likelihood with many parameters. When expectations are unavailable analytically, a Monte Carlo approximation at a fixed parameter value uses independent draws $Z_m\sim p_\theta$:

Under suitable integrability conditions this converges to $I(\theta)$. Averaging $-\nabla_\theta^2\log p_\theta(Z_m)$ is another option when the information identity holds. The outer-product estimate is positive semidefinite for every simulated sample; the Hessian estimate need not be. Differentiation treats the sampled observation as fixed. (jmlr.org)

Fisher scoring maximizes a sample log-likelihood $\ell_n$ through the update

Here $I_n$ is expected information for the entire sample and $\alpha_k$ controls step length. Newton’s method instead uses the observed negative Hessian, $j_n=-\nabla_\theta^2\ell_n$. Expected information is positive semidefinite, whereas observed curvature can be indefinite away from a maximum. Neither method guarantees improvement with an unrestricted full step. In canonical generalized linear models with fixed dispersion, expected and observed curvature coincide. For logistic regression, both equal $X^{\mathsf T}WX$, where $W_{ii}=p_i(1-p_i)$; thus scoring and Newton updates agree. (stat.cmu.edu)

For a differentiable objective $L$ to be maximized and nonsingular information, the natural-gradient direction is $I(\theta)^{-1}\nabla_\theta L$; minimizing a loss reverses its sign. Locally, this direction maximizes first-order improvement subject to a fixed quadratic cost $d^{\mathsf T}I(\theta)d$, corresponding to a small Kullback–Leibler divergence between model distributions. For a predictor $f_\theta(x)$, the generalized Gauss–Newton matrix pulls output-loss curvature back through its Jacobian. It equals the conditional Fisher when the loss is negative log-likelihood for an exponential-family output distribution and $f_\theta(x)$ supplies its natural parameters—for example, softmax logits or a Gaussian mean with fixed variance. This equality does not hold for arbitrary losses or output parameterizations. (arxiv.org)

In supervised learning, averaging over observed inputs does not require replacing model-distributed labels with training labels. Writing $s_i(y)=\nabla_\theta\log p_\theta(y\mid x_i)$, the model-expectation Fisher and the commonly named empirical Fisher are respectively

The latter averages gradients at observed labels, not simulated model labels, so it is not generally a Monte Carlo estimate of $F$. Frederik Kunstner, Lukas Balles, and Philipp Hennig examined this distinction in 2019. In Gaussian regression with mean $f_\theta(x_i)$, fixed variance $\sigma^2$, and $J_i=\nabla_\theta f_\theta(x_i)$, Fisher contributions are $J_iJ_i^{\mathsf T}/\sigma^2$, whereas empirical-Fisher contributions are $(y_i-f_\theta(x_i))^2J_iJ_i^{\mathsf T}/\sigma^4$. At exact interpolation, the latter vanish even if the former remain nonzero. (papers.nips.cc)

A dense matrix for $d$ parameters requires $O(d^2)$ storage. Matrix-vector products instead use $Fv=E[s(s^{\mathsf T}v)]$, enabling iterative linear solves without explicitly forming $F$. Diagonal, block-diagonal, and low-rank approximations reduce computational costs but discard structure. James Martens and Roger Grosse’s 2015 Kronecker-factored Approximate Curvature method approximates neural-network Fisher blocks using Kronecker products of smaller activation and backpropagated-derivative moment matrices, making approximate inversion practical. (arxiv.org)

Small eigenvalues and noisy estimates can make these solves poorly conditioned. Damping replaces the system by $(\widehat F+\lambda\mathrm{Id})d=\nabla L$, with $\lambda>0$, limiting amplification along weakly informed directions. Step-size control addresses the additional error from treating local curvature as valid over a finite update. Damping and structured approximations are computational regularizations, not the exact information matrix, and generally alter its reparameterization properties. (arxiv.org)

Experimental Design and Measurement

In experimental design, information depends on controllable observation settings as well as on the unknown parameter. Let $u$ denote a setting, such as a dose, sampling time, or instrument configuration, and let $I(\theta;u)$ be the information from one observation with density $p(y\mid\theta,u)$. A design $\xi=\{(u_k,w_k)\}_{k=1}^{K}$ specifies settings and allocation proportions, with $w_k\geq0$ and $\sum_k w_k=1$. For $n$ independent observations allocated in these proportions, the total information is

An exact design requires each $nw_k$ to be an integer; an approximate design treats the weights as continuous optimization variables. The density must include the measurement-noise model: changing noise variance, correlation, or the response distribution changes the information associated with the same settings. Allocation-weight formulations are used both for linear measurements and for more general regression models. (web.stanford.edu)

Optimality criteria reduce this matrix comparison to a scalar objective, usually with sample size or cost held fixed. D-optimality maximizes the determinant $\det I(\theta;\xi)$, or equivalently $\log\det I$ when information is positive definite. It minimizes the determinant of inverse information and, under a normal approximation, the volume of a fixed-level confidence ellipsoid. A-optimality minimizes the trace $\operatorname{tr}(I^{-1})$, corresponding to the sum of parameter-variance bounds. E-optimality maximizes the smallest eigenvalue $\lambda_{\min}(I)$; equivalently, it minimizes $\lambda_{\max}(I^{-1})$, the largest variance bound among unit-length parameter contrasts. These objectives need not select the same design. A- and E-optimality also require a specified parameter scaling, since changing units changes the relative weighting of parameter directions. (itl.nist.gov)

For Gaussian linear regression, suppose $Y=X\beta+\varepsilon$, where the design matrix $X$ is fixed and the errors are independent with distribution $N(0,\sigma^2)$, with known $\sigma^2$. Differentiating the log-likelihood gives the score $X^{\mathsf T}(Y-X\beta)/\sigma^2$, whose covariance is

When $X$ has full column rank, the least-squares estimator has exact covariance $\sigma^2(X^{\mathsf T}X)^{-1}$. Rank deficiency leaves some coefficient combinations unidentifiable; near-collinearity produces small information eigenvalues and large variances in the corresponding directions. Thus replication increases information but cannot remove a rank deficiency if every replicate repeats the same insufficient set of regressor vectors. These results follow directly from the Gaussian linear-measurement formulation. (web.stanford.edu)

For a concrete calculation, consider an intercept and slope with settings $x_i\in[-1,1]$. The preceding formula gives

For even $n$, equal allocation to $-1$ and $1$ attains the maximum determinant $n^2/\sigma^4$. With $n=10$ and $\sigma^2=4$, this design yields $I_\beta=\operatorname{diag}(2.5,2.5)$, so both coefficient variances are $0.4$. Taking all ten observations at $x=0$ instead gives $\operatorname{diag}(2.5,0)$, providing no slope information. This endpoint result concerns the specified straight-line model: adding a quadratic term makes the same two-setting design rank deficient because $x^2=1$ at both endpoints. (web.stanford.edu)

In nonlinear or heteroscedastic models, information commonly depends on unknown parameter values. A locally optimal design optimizes $I(\theta_0;\xi)$ at a specified nominal value $\theta_0$. An information-based Bayesian alternative averages a criterion over a prior distribution, for example maximizing $E_\pi[\log\det I(\theta;\xi)]$; this is not generally equivalent to maximizing the log-determinant of the prior-averaged matrix. A robust alternative optimizes worst-case performance over a specified parameter region, sometimes using efficiency relative to each parameter value’s local optimum. The prior or uncertainty region, admissible settings, noise assumptions, and chosen criterion are therefore part of any optimality claim, rather than incidental details. (conservancy.umn.edu)

Information-Theoretic and Quantum Extensions

Fisher information and Shannon entropy quantify different properties of a probability model. Entropy describes uncertainty in outcomes: for a discrete variable, $H(X)=-\sum_x p(x)\log p(x)$. Mutual information describes dependence through $I(X;Y)=D_{\mathrm{KL}}(p_{XY}\Vert p_Xp_Y)$. Neither definition requires an unknown parameter. Fisher information instead measures sensitivity to a specified parameter through derivatives of the log-density; its numerical value depends on the parameter’s units. These distinctions persist for continuous distributions, where differential entropy replaces discrete entropy. Relations between entropy and Fisher information therefore connect distinct quantities rather than identify them. (arxiv.org)

For a differentiable probability density $f$ on $\mathbb R^d$, the translation-information functional is

Here differentiation is with respect to the observation coordinate. Sufficient regularity conditions include a positive, continuously differentiable density, finite $J(f)$, integrable first derivatives, and boundary decay allowing integration by parts without boundary contributions. In the location family $p(x\mid\theta)=f(x-\theta)$, the parameter score is $-\nabla\log f(x-\theta)$; consequently, the location information matrix is $\int(\nabla\log f)(\nabla\log f)^{\mathsf T}f\,dx$, and $J(f)$ is its trace. A direct connection to entropy is the de Bruijn identity: if $Z$ is an independent standard Gaussian vector and the relevant differentiations and integrals are justified, then, for $t>0$,

with natural logarithms. Thus translation information measures the rate of entropy growth under Gaussian smoothing. (arxiv.org)

In quantum mechanics, a parameterized state is represented by a density matrix $\rho_\theta$. Its symmetric logarithmic derivative (SLD) is a self-adjoint operator $L_\theta$ satisfying

This defines the SLD version of quantum Fisher information. For a full-rank state the SLD is unique; for a regular fixed-rank family, its arbitrary kernel–kernel components do not affect $F_Q$. For a pure state $\rho_\theta=|\psi_\theta\rangle\langle\psi_\theta|$, the formula becomes $F_Q=4(\langle\partial_\theta\psi|\partial_\theta\psi\rangle-|\langle\psi|\partial_\theta\psi\rangle|^2)$. In particular, unitary encoding $|\psi_\theta\rangle=e^{-i\theta G}|\psi_0\rangle$ gives $F_Q=4\operatorname{Var}_{\psi_0}(G)$, making sensitivity depend explicitly on fluctuations of the generator. (arxiv.org)

A specified measurement, described by positive operators $M_y$ with $\sum_yM_y=\mathbf1$, produces classical probabilities $p(y\mid\theta)=\operatorname{Tr}(\rho_\theta M_y)$ and classical information $I_M(\theta)$. For scalar estimation, $I_M(\theta)\leq F_Q(\theta)$; Samuel L. Braunstein and Carlton M. Caves established the measurement-optimization characterization in 1994. Locally, measuring in an SLD eigenbasis attains the maximum, but the required basis generally depends on the unknown parameter. For $n$ independent identical preparations and a locally unbiased estimator, the quantum Cramér–Rao bound is

Optimizing measurement information does not itself guarantee finite-sample estimator efficiency; adaptive measurements can address the parameter dependence in suitable regular large-sample settings. (doi.org)

For several parameters, separate SLDs define

When this matrix is nonsingular, locally unbiased estimation satisfies $\operatorname{Cov}(\hat\theta)\succeq(nF_Q)^{-1}$. Unlike scalar estimation, however, one measurement need not attain the entire matrix bound: different SLDs may require incompatible measurements. Commuting SLDs permit a common eigenbasis and provide a sufficient compatibility condition. Noncommutation alone is not decisive; the weaker condition $\operatorname{Tr}(\rho[L_i,L_j])=0$ governs asymptotic attainability in regular collective-measurement formulations. When compatibility fails, attainable precision requires measurement-dependent trade-offs or tighter bounds rather than simply interpreting every diagonal entry of $F_Q^{-1}$ as simultaneously achievable. (quanestimation.github.io)

Nonregular Models and Limits of Interpretation

For $X$ following a uniform distribution on $(0,\theta)$, with $\theta>0$, the density is $p_\theta(x)=\theta^{-1}\mathbf 1_{\{0<x<\theta\}}$. At an interior observation, differentiating the log-density gives $s_\theta(x)=-1/\theta$, so

The score therefore has nonzero mean, its second moment is not its variance, and the curvature identity fails. Differentiating the normalization integral involves a moving-boundary term:

Ignoring that term invalidates the usual Cramér–Rao argument. For $n$ independent observations, the interior sample score is $-n/\theta$, whose squared expectation is $n^2/\theta^2$, not $n/\theta^2$; the usual additivity argument also fails. Writing $M=\max_iX_i$, the unbiased estimator $T=(n+1)M/n$ has $\operatorname{Var}_\theta(T)=\theta^2/[n(n+2)]$, below even the purported bound $\theta^2/n^2$. This contradicts no valid inequality: its regularity assumptions are absent. (ocw.mit.edu)

Parameter boundaries create a different obstruction. A variance component constrained to be nonnegative, for example, cannot fluctuate symmetrically around a true value of zero. Steven Self and Kung-Yee Liang’s 1987 analysis describes boundary limits through projections of Gaussian variables onto admissible cones; likelihood-ratio statistics can have mixtures of chi-squared distributions rather than the ordinary interior-parameter limit. Nondifferentiable models likewise require an explicit examination of the relevant smoothness conditions, rather than automatic use of Hessian formulas. If $E_\theta[s_\theta^2]=\infty$, the finite-information quadratic approximation and its usual asymptotic covariance formula are unavailable: treating $1/\infty=0$ as an attainable zero variance is unjustified. Alternative local-distance and nonregular asymptotic theories can describe such problems. (stat.cmu.edu)

A singular information matrix concerns local sensitivity, not necessarily global identifiability. As a direct example, parameterize a unit-variance normal distribution by mean $a+b$. Its information matrix is

with null direction $(1,-1)^{\mathsf T}$: changing $a$ and $b$ oppositely leaves the distribution unchanged. In a Gaussian mixture model, coincident components similarly make their mixing weight unidentified and introduce degeneracy. By contrast, exchanging two distinct component labels gives the same density at separated parameter points; this discrete symmetry does not by itself force local singularity at each point. Thus label switching can produce multiple equivalent likelihood modes even where local information is nonsingular. (arxiv.org)

Under model misspecification, expectations taken within the fitted model need not describe actual sampling variability. For independent, identically distributed observations from $Q$, let the interior pseudo-true parameter $\theta^*$ maximize $E_Q[\log p_\theta(X)]$, equivalently minimizing Kullback–Leibler divergence when that expression is defined. Set

Under suitable smoothness, moment, identification, and nonsingularity assumptions, maximum-likelihood estimation satisfies