{
  "id": "fisher-information",
  "title": "Fisher Information",
  "lang": "en",
  "source_lang": "en",
  "summary": "Fisher information measures a statistical model’s local sensitivity to its parameters and determines bounds on estimation precision.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "Fisher's information",
    "expected Fisher information",
    "Fisher information matrix"
  ],
  "generated_at": "2026-10-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/fisher-information/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/fisher-information/",
      "json": "https://aiwiki.page/zh/fisher-information.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Fisher information is a mathematical measure of how sensitively a probability distribution changes when an unknown parameter changes. It is defined through the expected squared derivative of the log-likelihood function, or through a matrix of such expectations when several parameters are involved. Under suitable regularity conditions, it also equals the expected negative curvature of the log-likelihood. It describes a property of a statistical model at a specified parameter value, rather than the information contained in one particular realized observation. (stat210a.berkeley.edu)\n\nIts central importance in statistics comes from the connection between this sensitivity and estimation precision. The Cramér–Rao inequality uses inverse Fisher information to bound the variance of unbiased estimators, while large-sample theory uses it to describe the uncertainty of maximum-likelihood estimates. Fisher information also supports invariant prior construction and the geometric interpretation of statistical models. (arxiv.org)\n\nNamed after Ronald Fisher, the concept developed alongside likelihood, sufficiency, and efficiency in twentieth-century statistical theory. Its uses now extend to experimental design, optimization in machine learning, and quantum parameter estimation. The distinctions between expected, observed, and empirical information—and the assumptions underlying each—are essential to these applications. (digital.library.adelaide.edu.au)"
    },
    {
      "heading": "Score Functions and the Definition of Information",
      "text": "Let \\(\\{P_\\theta:\\theta\\in\\Theta\\}\\) be a dominated parametric model, with scalar parameter \\(\\theta\\) in an open subset \\(\\Theta\\subseteq\\mathbb R\\). Dominated means that all model distributions have densities \\(p(x\\mid\\theta)\\) relative to one parameter-independent measure \\(\\mu\\): counting measure for discrete observations, or typically Lebesgue measure for continuous observations. For a fixed observation \\(x\\), the log-likelihood is \\(\\ell(\\theta;x)=\\log p(x\\mid\\theta)\\). Wherever the density is positive and differentiable, the score function is\n\n\\[s_\\theta(x)=\\partial_\\theta\\ell(\\theta;x) =\\frac{\\partial_\\theta p(x\\mid\\theta)}{p(x\\mid\\theta)}.\\]\n\nDifferentiation concerns the parameter, not the observation: \\(x\\) remains fixed as \\(\\theta\\) varies. The score is thus the local rate of change of the log-likelihood; \\(s_\\theta(X)\\) becomes a random variable when the observation is drawn from \\(P_\\theta\\). (stat210a.berkeley.edu)\n\nThe scalar Fisher information is the expected value of the squared score,\n\n\\[I(\\theta)=\\mathbb E_\\theta[s_\\theta(X)^2].\\]\n\nFor a probability mass function on a countable support \\(\\mathcal X\\), this becomes\n\n\\[I(\\theta)=\\sum_{x\\in\\mathcal X} \\bigl[\\partial_\\theta\\log p(x\\mid\\theta)\\bigr]^2p(x\\mid\\theta);\\]\n\nfor a probability density function on a continuous support, it becomes\n\n\\[I(\\theta)=\\int_{\\mathcal X} \\bigl[\\partial_\\theta\\log p(x\\mid\\theta)\\bigr]^2p(x\\mid\\theta)\\,dx.\\]\n\nThese expressions measure squared local sensitivity, averaged over observations generated at the same parameter value where the derivative is evaluated. They are nonnegative; finiteness requires the score to be square-integrable. (stat.cmu.edu)\n\nThe score has mean zero under suitable regularity conditions. A standard sufficient setup assumes a parameter-independent support, positive densities there, and differentiability in a neighborhood of the parameter under consideration. An integrable bound on \\(\\lvert\\partial_\\theta p(x\\mid\\theta)\\rvert\\), uniform over that neighborhood, permits differentiation under the integral through the dominated convergence theorem. Differentiating the normalization identity then gives\n\n\\[\\mathbb E_\\theta[s_\\theta(X)] =\\int_{\\mathcal X}\\partial_\\theta p(x\\mid\\theta)\\,d\\mu(x) =\\partial_\\theta\\int_{\\mathcal X}p(x\\mid\\theta)\\,d\\mu(x) =\\partial_\\theta 1=0.\\]\n\nConsequently, \\(I(\\theta)=\\operatorname{Var}_\\theta[s_\\theta(X)]\\): information is the variance of the score, not merely its second moment. The matching parameter values are essential; a score evaluated at \\(\\theta\\) need not have zero expectation under \\(P_{\\theta'}\\) when \\(\\theta'\\ne\\theta\\). Parameter-dependent support can invalidate this calculation by introducing boundary terms. (stat210a.berkeley.edu)\n\nA curvature representation needs additional assumptions: twice-differentiable densities, justification for interchanging the second derivative and integration, and integrability of the relevant quantities. For example, an integrable neighborhood-wise bound on \\(\\lvert\\partial_\\theta^2p(x\\mid\\theta)\\rvert\\), together with finite information, suffices in the common-support setting. Direct differentiation yields\n\n\\[\\partial_\\theta^2\\ell(\\theta;x) =\\frac{\\partial_\\theta^2p(x\\mid\\theta)}{p(x\\mid\\theta)} -s_\\theta(x)^2.\\]\n\nTaking expectations and using\n\\(\\int\\partial_\\theta^2p(x\\mid\\theta)\\,d\\mu(x)=\\partial_\\theta^2 1=0\\)\ntherefore establishes\n\n\\[I(\\theta)=-\\mathbb E_\\theta[\\partial_\\theta^2\\ell(\\theta;X)].\\]\n\nThis equality is a regular-model identity, not an unconditional alternative definition valid for every likelihood. (stat.cmu.edu)\n\nExpected information \\(I(\\theta)\\) averages over possible data. By contrast, observed information\n\n\\[j(\\theta;x)=-\\partial_\\theta^2\\ell(\\theta;x)\\]\n\nis calculated from the realized observation and generally depends on it. Under the curvature identity, \\(\\mathbb E_\\theta[j(\\theta;X)]=I(\\theta)\\), but the two need not agree for an individual sample. Nor must observed information be nonnegative at every parameter value: it is negative wherever that particular log-likelihood has positive second derivative, whereas expected information remains nonnegative by its squared-score definition. (stat.umn.edu)"
    },
    {
      "heading": "Multiparameter Information and Nuisance Parameters",
      "text": "For a parameter vector \\(\\theta=(\\theta_1,\\ldots,\\theta_d)^{\\mathsf T}\\), the score is the column vector of parameter derivatives of the log-likelihood:\n\n\\[s_\\theta(X)=\\nabla_\\theta\\log p(X\\mid\\theta), \\qquad I(\\theta)=E_\\theta[s_\\theta(X)s_\\theta(X)^{\\mathsf T}].\\]\n\nAssume that the score has finite second moments and that differentiation under the integral is valid, giving \\(E_\\theta[s_\\theta]=0\\). Then \\(I\\) is its covariance matrix: \\(I_{jj}=E_\\theta[s_j^2]\\) measures sensitivity to \\(\\theta_j\\) with the remaining coordinates fixed, while \\(I_{jk}=E_\\theta[s_js_k]\\) records cross-parameter score covariance. These cross-terms describe overlap between likelihood responses to different parameter changes, not correlations between the parameters themselves. (stat210a.berkeley.edu)\n\nThe matrix is symmetric and positive semidefinite because, for every \\(a\\in\\mathbb R^d\\),\n\n\\[a^{\\mathsf T}I(\\theta)a =E_\\theta[(a^{\\mathsf T}s_\\theta(X))^2]\\geq0.\\]\n\nHere \\(a^{\\mathsf T}s_\\theta\\) is the directional score along \\(\\theta+ta\\). In fixed parameter units, eigenvalues therefore quantify sensitivity along their eigenvectors. A zero eigenvalue identifies a direction whose score vanishes almost surely: the model has no first-order sensitivity there. This local statement does not establish that distinct parameter values generate identical distributions. Eigenvalue magnitudes also depend on the chosen parameter coordinates. (stat210a.berkeley.edu)\n\nPartition \\(\\theta=(\\psi^{\\mathsf T},\\lambda^{\\mathsf T})^{\\mathsf T}\\), where \\(\\psi\\) is the inferential target and \\(\\lambda\\) contains nuisance parameters, and write\n\n\\[s_\\theta= \\begin{pmatrix}s_\\psi\\\\s_\\lambda\\end{pmatrix}, \\qquad I= \\begin{pmatrix} I_{\\psi\\psi}&I_{\\psi\\lambda}\\\\ I_{\\lambda\\psi}&I_{\\lambda\\lambda} \\end{pmatrix}.\\]\n\nIf \\(\\lambda\\) is known, the relevant information is \\(I_{\\psi\\psi}\\). If it must be estimated, changes in \\(\\lambda\\) can partly reproduce the likelihood response to changes in \\(\\psi\\). The information available for distinguishing the target from these nuisance changes is consequently different from the target’s diagonal block alone. (web.stanford.edu)\n\nWhen \\(I_{\\lambda\\lambda}\\) is invertible, remove the linear projection of the target score onto the nuisance scores:\n\n\\[s_\\psi^{\\mathrm{eff}} =s_\\psi-I_{\\psi\\lambda}I_{\\lambda\\lambda}^{-1}s_\\lambda.\\]\n\nIndeed,\n\n\\[E_\\theta[s_\\psi^{\\mathrm{eff}}s_\\lambda^{\\mathsf T}] =I_{\\psi\\lambda} -I_{\\psi\\lambda}I_{\\lambda\\lambda}^{-1}I_{\\lambda\\lambda} =0.\\]\n\nExpanding the residual’s covariance gives the efficient information\n\n\\[I_{\\mathrm{eff}} =E_\\theta[s_\\psi^{\\mathrm{eff}} (s_\\psi^{\\mathrm{eff}})^{\\mathsf T}] =I_{\\psi\\psi} -I_{\\psi\\lambda}I_{\\lambda\\lambda}^{-1}I_{\\lambda\\psi},\\]\n\nthe Schur complement of the nuisance block. The subtracted matrix is positive semidefinite, so estimating nuisance parameters cannot increase information relative to knowing them. For a nonsingular full matrix, block inversion yields \\((I^{-1})_{\\psi\\psi}=I_{\\mathrm{eff}}^{-1}\\), rather than \\(I_{\\psi\\psi}^{-1}\\). (sciencedirect.com)\n\nA numerical illustration makes the distinction explicit. Suppose the information for two scalar parameters is\n\n\\[I=\\begin{pmatrix}4&2\\\\2&3\\end{pmatrix},\\]\n\nwith the first parameter of interest. Its information is \\(4\\) when the second parameter is known, but \\(4-2^2/3=8/3\\) when the second must be estimated. Correspondingly, the target entry of the inverse full matrix is \\(3/8\\), whereas the inverse known-nuisance information is \\(1/4\\). These are consequences of the block-inverse identity; their interpretation as variance bounds or asymptotic covariance scales requires the relevant estimation assumptions. (web.stanford.edu)\n\nThe blocks are information orthogonal when \\(I_{\\psi\\lambda}=0\\), in which case \\(I_{\\mathrm{eff}}=I_{\\psi\\psi}\\). Orthogonality at one parameter value is a pointwise property; orthogonality throughout a model depends on its parameterization. For example, the mean and standard deviation of a normal distribution have zero cross-information. Thus estimating the standard deviation does not reduce efficient information about the mean, although unknown scale still affects finite-sample confidence procedures. Information orthogonality means uncorrelated score blocks, not necessarily independent scores or independent estimators. (arxiv.org)"
    },
    {
      "heading": "Worked Models and Parameterization Choices",
      "text": "For one observation \\(X\\) from a Bernoulli distribution, with \\(P(X=1)=p\\) and \\(0<p<1\\), the log-likelihood and score are\n\n\\[\\ell(p;x)=x\\log p+(1-x)\\log(1-p),\\qquad s_p(x)=\\frac{x-p}{p(1-p)}.\\]\n\nSince \\(\\operatorname{Var}(X)=p(1-p)\\), taking the expected square of the score gives\n\n\\[I_p(p)=\\frac{\\operatorname{Var}(X)}{p^2(1-p)^2} =\\frac{1}{p(1-p)}.\\]\n\nFor a count \\(K\\) having a binomial distribution with known trial number \\(n\\), the score is \\((K-np)/[p(1-p)]\\). Its variance, using \\(\\operatorname{Var}(K)=np(1-p)\\), yields \\(I_p(p)=n/[p(1-p)]\\), the same information as the underlying \\(n\\) independent Bernoulli trials. (pages.stat.wisc.edu)\n\nIn log-odds coordinates, \\(\\eta=\\log[p/(1-p)]\\), the Bernoulli log-likelihood becomes \\(x\\eta-\\log(1+e^\\eta)\\). Differentiation now gives \\(s_\\eta(x)=x-p\\), and therefore\n\n\\[I_\\eta(\\eta)=p(1-p),\\qquad I_\\eta^{\\mathrm{binomial}}(\\eta)=np(1-p).\\]\n\nThus information in probability coordinates diverges as \\(p\\) approaches either endpoint, whereas information in log-odds coordinates tends to zero. This difference reflects the coordinate scale: \\(dp/d\\eta=p(1-p)\\), so the chain rule gives \\(I_\\eta=(dp/d\\eta)^2I_p\\). For example, at \\(p=1/2\\), the respective one-trial values are \\(4\\) and \\(1/4\\). These are different descriptions of the same experiment, not contradictory assessments of its information; the formulas concern interior probabilities, not the boundary models themselves. (stat.berkeley.edu)\n\nFor \\(X\\) from a Poisson distribution with mean \\(\\lambda>0\\),\n\n\\[\\ell(\\lambda;x)=x\\log\\lambda-\\lambda-\\log(x!),\\qquad s_\\lambda(x)=\\frac{x-\\lambda}{\\lambda}.\\]\n\nBecause \\(\\operatorname{Var}(X)=\\lambda\\), \\(I_\\lambda(\\lambda)=1/\\lambda\\). If instead \\(\\eta=\\log\\lambda\\), then \\(\\ell(\\eta;x)=x\\eta-e^\\eta-\\log(x!)\\), the score is \\(x-\\lambda\\), and \\(I_\\eta(\\eta)=\\lambda\\). Increasing \\(\\lambda\\) consequently decreases information about additive changes in the mean but increases information about changes on the logarithmic, or relative, scale. Here \\(\\lambda\\) is the expected count over the specified observation interval; identifying it with a rate assumes a unit interval. (stat.berkeley.edu)\n\nFor one normal observation \\(X\\sim N(\\mu,\\sigma^2)\\), with \\(\\sigma>0\\), differentiating the log-density gives\n\n\\[s_\\mu(X)=\\frac{X-\\mu}{\\sigma^2},\\qquad s_\\sigma(X)=-\\frac1\\sigma+\\frac{(X-\\mu)^2}{\\sigma^3}.\\]\n\nWriting \\(Z=(X-\\mu)/\\sigma\\), these scores are \\(Z/\\sigma\\) and \\((Z^2-1)/\\sigma\\). Using \\(E(Z^2)=1\\), \\(E(Z^3)=0\\), and \\(E(Z^4)=3\\), their expected products give\n\n\\[I_{(\\mu,\\sigma)} =\\begin{pmatrix}\\sigma^{-2}&0\\\\0&2\\sigma^{-2}\\end{pmatrix}.\\]\n\nThe cross-entry vanishes because \\(E[Z(Z^2-1)]=0\\). In variance coordinates \\(v=\\sigma^2\\), the second score becomes \\([(X-\\mu)^2-v]/(2v^2)\\), giving\n\n\\[I_{(\\mu,v)} =\\begin{pmatrix}v^{-1}&0\\\\0&(2v^2)^{-1}\\end{pmatrix}.\\]\n\nIn particular, information about standard deviation is \\(2/\\sigma^2\\), whereas information about variance is \\(1/(2\\sigma^4)\\); these quantities must not be interchanged. (stat.berkeley.edu)\n\nThese calculations share a general mechanism in a regular exponential family expressed in natural coordinates:\n\n\\[p_\\eta(x)=h(x)\\exp\\{\\eta^{\\mathsf T}T(x)-A(\\eta)\\}.\\]\n\nHere \\(T(X)\\) is the canonical sufficient statistic, and \\(A\\) normalizes the density. At interior natural-parameter values, differentiation under the normalizing integral yields\n\n\\[\\nabla A(\\eta)=E_\\eta[T(X)],\\qquad I_\\eta(\\eta)=\\operatorname{Cov}_\\eta[T(X)] =\\nabla^2A(\\eta).\\]\n\nThe score is therefore the centered statistic \\(T(X)-E_\\eta[T(X)]\\), and information is both its covariance matrix and the Hessian of \\(A\\). Bernoulli and Poisson models have \\(A(\\eta)=\\log(1+e^\\eta)\\) and \\(A(\\eta)=e^\\eta\\), respectively. The Hessian identity specifically requires natural coordinates: differentiating \\(A(\\eta(\\theta))\\) twice in an arbitrary parameter \\(\\theta\\) does not generally produce its information matrix. (stat210a.berkeley.edu)"
    },
    {
      "heading": "The Cramér–Rao Inequality and Efficiency",
      "text": "Let \\(X\\) denote the entire observed sample, with joint density or mass function \\(p_\\theta(x)\\), and let \\(S_\\theta(X)=\\partial_\\theta\\log p_\\theta(X)\\) be its score. Assume an interior scalar parameter, parameter-independent support, differentiation under the integral or sum for both normalization and the estimator’s expectation, and finite second moments with \\(0<I(\\theta)<\\infty\\). For a parameter-independent estimator \\(T(X)\\) satisfying \\(E_\\theta[T]=g(\\theta)\\) throughout a neighborhood, differentiation gives the covariance identity\n\n\\[g'(\\theta) =\\int T(x)\\,\\partial_\\theta p_\\theta(x)\\,dx =E_\\theta[T S_\\theta] =\\operatorname{Cov}_\\theta(T,S_\\theta),\\]\n\nwhere the last equality uses \\(E_\\theta[S_\\theta]=0\\); in a discrete model, replace the integral by a sum. The Cauchy–Schwarz inequality then yields\n\n\\[[g'(\\theta)]^2 \\leq \\operatorname{Var}_\\theta(T)\\,I(\\theta), \\qquad \\operatorname{Var}_\\theta(T)\\geq \\frac{[g'(\\theta)]^2}{I(\\theta)}.\\]\n\nThis is the Cramér–Rao inequality, also called the information inequality. (stat210a.berkeley.edu)\n\nThe information in this bound is that of the whole sample, not one observation. For \\(n\\) independent, identically distributed observations with information \\(I_1(\\theta)\\), it becomes \\([g'(\\theta)]^2/[nI_1(\\theta)]\\). For example, if \\(X_i\\sim N(\\mu,\\sigma^2)\\) with known \\(\\sigma^2\\), the sample score is \\(S_\\mu=n(\\bar X-\\mu)/\\sigma^2\\), and the information is \\(n/\\sigma^2\\). The sample mean has variance \\(\\sigma^2/n\\), exactly attaining the bound for estimating \\(\\mu\\). This is an exact finite-sample result, not a large-sample approximation. (stat.cmu.edu)\n\nFor a \\(d\\)-dimensional parameter vector \\(\\theta\\), suppose \\(E_\\theta[T]=\\theta\\), the corresponding differentiation identities hold componentwise, and the sample information matrix \\(I(\\theta)=E_\\theta[SS^{\\mathsf T}]\\) is finite and nonsingular. Then\n\n\\[E_\\theta[(T-\\theta)S^{\\mathsf T}]=\\operatorname{Id}_d, \\qquad \\operatorname{Cov}_\\theta(T)\\succeq I(\\theta)^{-1}.\\]\n\nThe ordering means that \\(\\operatorname{Cov}_\\theta(T)-I(\\theta)^{-1}\\) is positive semidefinite: every linear combination \\(a^{\\mathsf T}T\\) has variance at least \\(a^{\\mathsf T}I(\\theta)^{-1}a\\). Indeed, the covariance matrix of the residual \\(T-\\theta-I(\\theta)^{-1}S\\) equals this difference, providing a direct proof. It is not an entrywise inequality between matrices. (stat.cmu.edu)\n\nIn the scalar case, equality holds precisely when\n\n\\[T-g(\\theta)=\\frac{g'(\\theta)}{I(\\theta)}S_\\theta \\quad\\text{almost surely under }P_\\theta.\\]\n\nThus, the centered estimator must be proportional to the score; matrix equality requires \\(T-\\theta=I(\\theta)^{-1}S\\) almost surely. For a nonzero bound and positive estimator variance, finite-sample efficiency can be expressed as \\(e_\\theta(T)=[g'(\\theta)]^2/\\{I(\\theta)\\operatorname{Var}_\\theta(T)\\}\\). An unbiased estimator attaining equality throughout the parameter space is called efficient. The bound need not be attainable, however: failure to attain it does not establish that a better unbiased estimator exists. Finite-sample attainment is distinct from asymptotic efficiency, which concerns limiting performance as sample size increases. (stat210a.berkeley.edu)\n\nFor a biased estimator of \\(g(\\theta)\\), define its bias by \\(b(\\theta)=E_\\theta[T]-g(\\theta)\\). Applying the covariance argument to its actual expectation gives\n\n\\[\\operatorname{Var}_\\theta(T)\\geq \\frac{[g'(\\theta)+b'(\\theta)]^2}{I(\\theta)}, \\qquad \\operatorname{MSE}_\\theta(T)\\geq b(\\theta)^2+\\frac{[g'(\\theta)+b'(\\theta)]^2}{I(\\theta)}.\\]\n\nFor estimating \\(\\theta\\) itself, the numerator is \\([1+b'(\\theta)]^2\\). The mean squared error includes squared bias as well as variance, so a biased estimator may have MSE below the unbiased Cramér–Rao bound without contradiction. Unbiasedness at a single parameter value is insufficient for the unbiased formula: the derivative of the expectation, and therefore of the bias, also matters. (cpb-us-e1.wpmucdn.com)"
    },
    {
      "heading": "Likelihood Inference in Large Samples",
      "text": "For independent, identically distributed observations \\(X_1,\\ldots,X_n\\) from a correctly specified, identifiable model with a fixed-dimensional parameter \\(\\theta\\in\\mathbb R^d\\), a regular maximum-likelihood estimator satisfies\n\n\\[\\sqrt n(\\hat\\theta_n-\\theta_0) \\Rightarrow N_d\\!\\left(0,I_1(\\theta_0)^{-1}\\right).\\]\n\nHere \\(\\theta_0\\) is the true parameter, \\(I_1(\\theta_0)\\) is expected information per observation, and \\(\\Rightarrow\\) denotes convergence in distribution. Thus the large-sample normal approximation has covariance \\(I_1(\\theta_0)^{-1}/n\\), not \\(I_1(\\theta_0)^{-1}\\). The theorem requires more than differentiability: typical assumptions include consistency of the selected maximizer, an interior true parameter, finite nonsingular information, and sufficient control of likelihood derivatives near \\(\\theta_0\\). These conditions exclude routine application at parameter boundaries or in singular models. (stat210a.berkeley.edu)\n\nThe mechanism is a local expansion of the likelihood equation. Writing \\(\\ell_n(\\theta)=\\sum_{i=1}^n\\log p(X_i\\mid\\theta)\\) and \\(U_n=\\nabla\\ell_n\\), regularity yields\n\n\\[\\sqrt n(\\hat\\theta_n-\\theta_0) = I_1(\\theta_0)^{-1}\\frac{U_n(\\theta_0)}{\\sqrt n} +o_p(1),\\]\n\nwhere \\(o_p(1)\\) vanishes in probability. The central limit theorem makes the normalized score asymptotically normal with covariance \\(I_1(\\theta_0)\\), while normalized negative likelihood curvature converges to that same matrix. Consistency keeps the expansion in the appropriate neighborhood; nonsingularity permits inversion. The resulting inverse-information covariance expresses asymptotic efficiency among regular estimators. It is not a claim of exact unbiasedness, minimum finite-sample mean-squared error, or superiority to every estimator at an isolated parameter value. (stat210a.berkeley.edu)\n\nBecause \\(\\theta_0\\) is unknown, two standard approximations to the estimator’s covariance matrix are\n\n\\[\\widehat V_{\\mathrm{exp}} =\\bigl[nI_1(\\hat\\theta_n)\\bigr]^{-1}, \\qquad \\widehat V_{\\mathrm{obs}} =j_n(\\hat\\theta_n)^{-1}, \\quad j_n(\\theta)=-\\nabla^2\\ell_n(\\theta).\\]\n\nThe first substitutes the estimate into expected information; the second uses the observed likelihood Hessian. Under the relevant continuity and convergence assumptions, both have the same first-order asymptotic limit, although they need not agree for a particular sample. Estimated standard errors are square roots of diagonal entries of either matrix. With several jointly estimated parameters, the relevant quantity is a diagonal entry of the inverse full matrix, not generally the reciprocal of a diagonal information entry. (stat.cmu.edu)\n\nFor a scalar parameter, a Wald confidence interval is \\(\\hat\\theta_n\\pm z_{1-\\alpha/2}\\sqrt{\\widehat V}\\), where \\(z_{1-\\alpha/2}\\) is a standard-normal quantile. For the simple null hypothesis \\(H_0:\\theta=\\theta_\\ast\\), the vector Wald statistic is\n\n\\[W_n=(\\hat\\theta_n-\\theta_\\ast)^{\\mathsf T} \\widehat V^{-1}(\\hat\\theta_n-\\theta_\\ast) \\Rightarrow\\chi_d^2.\\]\n\nInverting this test gives an approximate confidence ellipsoid. For example, in a Bernoulli sample with \\(n=400\\) and \\(\\hat p=0.30\\), the information-based standard error is \\(\\sqrt{0.30(0.70)/400}\\approx0.0229\\); the approximate 95% Wald interval is \\(0.255\\) to \\(0.345\\). This calculation illustrates the approximation rather than establishing exact 95% coverage. (stat210a.berkeley.edu)\n\nInformation also standardizes the score under the null:\n\n\\[S_n=U_n(\\theta_\\ast)^{\\mathsf T} \\bigl[nI_1(\\theta_\\ast)\\bigr]^{-1}U_n(\\theta_\\ast) \\Rightarrow\\chi_d^2.\\]\n\nUnlike the Wald statistic, this score statistic uses derivatives evaluated at the null rather than at the unrestricted estimate. The likelihood-ratio statistic,\n\n\\[R_n=2\\{\\ell_n(\\hat\\theta_n)-\\ell_n(\\tilde\\theta_n)\\},\\]\n\ncompares unrestricted and null-constrained maximizers. For a smooth nested null imposing \\(r\\) independent restrictions, its limiting distribution is \\(\\chi_r^2\\); corresponding Wald and score tests have the same first-order asymptotic behavior. Information enters this result through the local quadratic likelihood expansion. With nuisance parameters, the score calculation uses restricted estimates and information adjusted for their estimation. Despite asymptotic equivalence, these tests can yield different finite-sample results, and neither their chi-squared calibration nor confidence-region coverage is an exact general guarantee. (katsevich-teaching.github.io)"
    },
    {
      "heading": "Combining Observations and Preserving Information",
      "text": "For independent observations \\(X_1,\\ldots,X_n\\), possibly with different distributions \\(p_i(x_i\\mid\\theta)\\), the joint likelihood factors and its score is the sum of the individual scores:\n\n\\[p(x_1,\\ldots,x_n\\mid\\theta)=\\prod_{i=1}^n p_i(x_i\\mid\\theta), \\qquad s_{1:n}=\\sum_{i=1}^n s_i.\\]\n\nUnder regularity conditions giving each score zero mean and finite second moments, independence makes the cross terms \\(E_\\theta[s_i s_j^{\\mathsf T}]\\) vanish for \\(i\\ne j\\). Consequently,\n\n\\[I_{1:n}(\\theta)=\\sum_{i=1}^n I_i(\\theta).\\]\n\nIdentically distributed observations give the special case \\(I_n=nI_1\\). With dependent observations, the joint distribution must instead determine the score: summing marginal information generally omits the dependence structure. For example, recording the same random outcome twice does not provide two independent observations or double its information. These conclusions follow from likelihood factorization and score covariance, rather than from sample size alone. (stat210a.berkeley.edu)\n\nLet \\(T\\) be a parameter-independent reduction of data \\(X\\), including a randomized observation channel with conditional density \\(k(t\\mid x)\\) independent of \\(\\theta\\). Its density is \\(q_\\theta(t)=\\int k(t\\mid x)p_\\theta(x)\\,dx\\). If differentiation can pass through this integral and the score is square-integrable, then\n\n\\[\\nabla_\\theta q_\\theta(t) =\\int k(t\\mid x)p_\\theta(x)s_X(x)\\,dx, \\qquad s_T(t)=E_\\theta[s_X(X)\\mid T=t].\\]\n\nThus the reduced-data score is the conditional expectation of the original score, not generally its value at an estimated or reconstructed observation. A deterministic statistic \\(T=T(X)\\) satisfies the same identity, interpreted through its induced distribution. The conditional-score identity also holds under the more general regularity framework of differentiability in quadratic mean. (arxiv.org)\n\nThe conditional covariance decomposition now yields\n\n\\[I_X(\\theta)-I_T(\\theta) =E_\\theta\\!\\left[ \\operatorname{Cov}_\\theta\\{s_X(X)\\mid T\\} \\right]\\succeq0.\\]\n\nData reduction therefore cannot increase expected information in any parameter direction. Equality holds precisely when the score is determined by \\(T\\), almost surely at the parameter value concerned. A sufficient statistic preserves information because the conditional distribution of \\(X\\) given \\(T\\) is parameter-independent. For independent Bernoulli trials, retaining only the success count preserves the score \\((T-np)/[p(1-p)]\\) and hence the information \\(n/[p(1-p)]\\). Preservation alone, however, need not imply sufficiency: Abram Kagan and Lawrence Shepp’s 2005 example uses a binary label and a continuous location observation whose component densities occupy opposite sides of the location parameter. Discarding the label preserves Fisher information, although its conditional distribution still depends on the parameter. Additional assumptions are therefore needed for the converse implication. (arxiv.org)\n\nFor incomplete observations, write the complete data as \\(X=(Y,Z)\\), where \\(Y\\) is observed and \\(Z\\) is missing or latent. If \\(s_c\\) is the complete-data score, the observed-data score is \\(s_o(Y)=E_\\theta[s_c\\mid Y]\\). The same decomposition gives\n\n\\[I_c(\\theta)=I_o(\\theta)+I_m(\\theta), \\qquad I_m(\\theta)=E_\\theta[ \\operatorname{Cov}_\\theta(s_c\\mid Y)].\\]\n\nHere \\(I_m\\) measures the score uncertainty remaining after observing \\(Y\\). All three quantities are expected information matrices; \\(I_o\\) is not the sample-specific negative Hessian. (arxiv.org)\n\nThomas A. Louis’s 1982 method supplies the corresponding sample-specific identity. With \\(\\ell_c(\\theta;Y,Z)=\\log p_\\theta(Y,Z)\\), \\(H_c=-\\nabla_\\theta^2\\ell_c\\), and observed log-likelihood \\(\\ell_o(\\theta;y)\\),\n\n\\[j_o(\\theta;y) =-\\nabla_\\theta^2\\ell_o(\\theta;y) =E_\\theta[H_c\\mid Y=y] -\\operatorname{Cov}_\\theta(s_c\\mid Y=y).\\]\n\nThe conditional covariance includes subtraction of the conditional score’s outer product; that score vanishes at an interior stationary likelihood estimate. After fitting with the expectation–maximization algorithm, Louis’s identity obtains observed information from complete-data derivatives and conditional moments of latent variables, without directly differentiating the marginal likelihood. The conditional expected complete-data curvature alone is insufficient: the missing-score covariance must also be subtracted. (markirwin.net)"
    },
    {
      "heading": "Statistical Geometry and Bayesian Connections",
      "text": "Let \\(\\theta=h(\\eta)\\) be a smooth, locally invertible change of coordinates for a \\(k\\)-parameter model, and let \\(J=\\partial\\theta/\\partial\\eta\\) be its Jacobian matrix. The chain rule gives \\(s_\\eta=J^{\\mathsf T}s_\\theta\\), so taking expected score outer products yields\n\n\\[I_\\eta(\\eta)=J^{\\mathsf T}I_\\theta(h(\\eta))J.\\]\n\nInformation therefore does not have coordinate-independent numerical entries: in one dimension, \\(I_\\eta=I_\\theta(d\\theta/d\\eta)^2\\). For example, expressing a normal mean in centimeters rather than meters divides its information by \\(10^4\\), because \\(\\mu_{\\mathrm m}=\\mu_{\\mathrm{cm}}/100\\). Nevertheless, the quadratic form is unchanged: \\(d\\eta^{\\mathsf T}I_\\eta d\\eta=d\\theta^{\\mathsf T}I_\\theta d\\theta\\). Changing units changes the coordinate description, not the statistical sensitivity represented by that form. (pmc.ncbi.nlm.nih.gov)\n\nOn a smooth, regular, identifiable statistical manifold with finite, positive-definite information, this transformation law makes \\(I(\\theta)\\) a metric tensor. The resulting Fisher–Rao metric assigns tangent vectors \\(u,v\\) the inner product \\(g_\\theta(u,v)=u^{\\mathsf T}I(\\theta)v\\), with infinitesimal line element\n\n\\[ds^2=d\\theta^{\\mathsf T}I(\\theta)d\\theta.\\]\n\nA curve \\(\\theta(t)\\) consequently has length \\(\\int\\sqrt{\\dot\\theta(t)^{\\mathsf T}I(\\theta(t))\\dot\\theta(t)}\\,dt\\); minimizing lengths over connecting curves defines the associated geodesic distance. Positive definiteness is essential here: identifiability alone does not rule out degeneracy at exceptional parameter values. Thus a singular information matrix does not define an ordinary Riemannian metric on the full coordinate space. (pmc.ncbi.nlm.nih.gov)\n\nThe metric also describes the local behavior of Kullback–Leibler divergence. Using natural logarithms, set \\(F(\\delta)=E_\\theta[\\log p_\\theta(X)-\\log p_{\\theta+\\delta}(X)]\\). Under common-support, differentiability, and integrability assumptions that justify differentiating the expectation, \\(F(0)=0\\), \\(\\nabla F(0)=-E_\\theta[s_\\theta]=0\\), and \\(\\nabla^2F(0)=-E_\\theta[\\nabla_\\theta^2\\log p_\\theta]=I(\\theta)\\). A second-order Taylor expansion therefore gives\n\n\\[D_{\\mathrm{KL}}(p_\\theta\\Vert p_{\\theta+d\\theta}) =\\tfrac12d\\theta^{\\mathsf T}I(\\theta)d\\theta +o(\\|d\\theta\\|^2).\\]\n\nReversing the divergence has the same quadratic term. This is a local relation, not an equality between KL divergence and squared geodesic distance for separated distributions; KL divergence is generally asymmetric. (jmlr.org)\n\nIn Bayesian inference, Jeffreys’s prior, proposed by Harold Jeffreys in 1946, uses the metric volume density:\n\n\\[\\pi_J(\\theta)\\propto\\sqrt{\\det I(\\theta)},\\qquad dV_I=\\sqrt{\\det I(\\theta)}\\,d\\theta_1\\cdots d\\theta_k.\\]\n\nIndeed, the coordinate-transformation formula implies \\(\\sqrt{\\det I_\\eta}=|\\det J|\\sqrt{\\det I_\\theta}\\), exactly the Jacobian factor required to transform a probability density. Invariance means that the same measure is obtained in either coordinate system, not that its density has the same numerical expression. For a Bernoulli probability \\(p\\), the normalized density is \\(1/[\\pi\\sqrt{p(1-p)}]\\). In the coordinate \\(\\phi=2\\arcsin\\sqrt p\\), \\(0<\\phi<\\pi\\), information equals \\(1\\), so this measure becomes uniform. (arxiv.org)\n\nMetric volume need not be finite. For a normal mean \\(\\mu\\) with known variance \\(\\sigma^2\\), \\(I(\\mu)=1/\\sigma^2\\), giving the improper prior \\(\\pi_J(\\mu)\\propto1\\) on the real line. Nevertheless, \\(n\\geq1\\) independent observations yield the proper posterior \\(N(\\bar x,\\sigma^2/n)\\). In general, posterior propriety requires\n\n\\[0<\\int_\\Theta p(x\\mid\\theta)\\sqrt{\\det I(\\theta)}\\,d\\theta<\\infty.\\]\n\nReparameterization invariance does not establish this condition. For example, with known normal mean \\(\\mu_0\\) and unknown standard deviation, Jeffreys’s density is \\(1/\\sigma\\); if every observation equals \\(\\mu_0\\), the posterior kernel is \\(\\sigma^{-(n+1)}\\), whose integral diverges at zero. This exceptional dataset demonstrates why normalization must be checked separately. (www2.stat.duke.edu)"
    },
    {
      "heading": "Numerical Computation and Learning Algorithms",
      "text": "Scores and information matrices can be calculated by analytical differentiation when the likelihood has a tractable form, or by automatic differentiation, which applies the chain rule through a computational graph rather than using finite differences. Reverse-mode differentiation efficiently evaluates the score of a scalar log-likelihood with many parameters. When expectations are unavailable analytically, a Monte Carlo approximation at a fixed parameter value uses independent draws \\(Z_m\\sim p_\\theta\\):\n\n\\[\\widehat I_M(\\theta)=\\frac1M\\sum_{m=1}^{M} s_\\theta(Z_m)s_\\theta(Z_m)^{\\mathsf T}.\\]\n\nUnder suitable integrability conditions this converges to \\(I(\\theta)\\). Averaging \\(-\\nabla_\\theta^2\\log p_\\theta(Z_m)\\) is another option when the information identity holds. The outer-product estimate is positive semidefinite for every simulated sample; the Hessian estimate need not be. Differentiation treats the sampled observation as fixed. (jmlr.org)\n\nFisher scoring maximizes a sample log-likelihood \\(\\ell_n\\) through the update\n\n\\[I_n(\\theta_k)d_k=\\nabla_\\theta\\ell_n(\\theta_k), \\qquad \\theta_{k+1}=\\theta_k+\\alpha_k d_k.\\]\n\nHere \\(I_n\\) is expected information for the entire sample and \\(\\alpha_k\\) controls step length. Newton’s method instead uses the observed negative Hessian, \\(j_n=-\\nabla_\\theta^2\\ell_n\\). Expected information is positive semidefinite, whereas observed curvature can be indefinite away from a maximum. Neither method guarantees improvement with an unrestricted full step. In canonical generalized linear models with fixed dispersion, expected and observed curvature coincide. For logistic regression, both equal \\(X^{\\mathsf T}WX\\), where \\(W_{ii}=p_i(1-p_i)\\); thus scoring and Newton updates agree. (stat.cmu.edu)\n\nFor a differentiable objective \\(L\\) to be maximized and nonsingular information, the natural-gradient direction is \\(I(\\theta)^{-1}\\nabla_\\theta L\\); minimizing a loss reverses its sign. Locally, this direction maximizes first-order improvement subject to a fixed quadratic cost \\(d^{\\mathsf T}I(\\theta)d\\), corresponding to a small Kullback–Leibler divergence between model distributions. For a predictor \\(f_\\theta(x)\\), the generalized Gauss–Newton matrix pulls output-loss curvature back through its Jacobian. It equals the conditional Fisher when the loss is negative log-likelihood for an exponential-family output distribution and \\(f_\\theta(x)\\) supplies its natural parameters—for example, softmax logits or a Gaussian mean with fixed variance. This equality does not hold for arbitrary losses or output parameterizations. (arxiv.org)\n\nIn supervised learning, averaging over observed inputs does not require replacing model-distributed labels with training labels. Writing \\(s_i(y)=\\nabla_\\theta\\log p_\\theta(y\\mid x_i)\\), the model-expectation Fisher and the commonly named empirical Fisher are respectively\n\n\\[F=\\frac1n\\sum_i E_{Y\\sim p_\\theta(\\cdot\\mid x_i)} [s_i(Y)s_i(Y)^{\\mathsf T}], \\qquad \\widetilde F=\\frac1n\\sum_i s_i(y_i)s_i(y_i)^{\\mathsf T}.\\]\n\nThe latter averages gradients at observed labels, not simulated model labels, so it is not generally a Monte Carlo estimate of \\(F\\). Frederik Kunstner, Lukas Balles, and Philipp Hennig examined this distinction in 2019. In Gaussian regression with mean \\(f_\\theta(x_i)\\), fixed variance \\(\\sigma^2\\), and \\(J_i=\\nabla_\\theta f_\\theta(x_i)\\), Fisher contributions are \\(J_iJ_i^{\\mathsf T}/\\sigma^2\\), whereas empirical-Fisher contributions are \\((y_i-f_\\theta(x_i))^2J_iJ_i^{\\mathsf T}/\\sigma^4\\). At exact interpolation, the latter vanish even if the former remain nonzero. (papers.nips.cc)\n\nA dense matrix for \\(d\\) parameters requires \\(O(d^2)\\) storage. Matrix-vector products instead use \\(Fv=E[s(s^{\\mathsf T}v)]\\), enabling iterative linear solves without explicitly forming \\(F\\). Diagonal, block-diagonal, and low-rank approximations reduce computational costs but discard structure. James Martens and Roger Grosse’s 2015 Kronecker-factored Approximate Curvature method approximates neural-network Fisher blocks using Kronecker products of smaller activation and backpropagated-derivative moment matrices, making approximate inversion practical. (arxiv.org)\n\nSmall eigenvalues and noisy estimates can make these solves poorly conditioned. Damping replaces the system by \\((\\widehat F+\\lambda\\mathrm{Id})d=\\nabla L\\), with \\(\\lambda>0\\), limiting amplification along weakly informed directions. Step-size control addresses the additional error from treating local curvature as valid over a finite update. Damping and structured approximations are computational regularizations, not the exact information matrix, and generally alter its reparameterization properties. (arxiv.org)"
    },
    {
      "heading": "Experimental Design and Measurement",
      "text": "In experimental design, information depends on controllable observation settings as well as on the unknown parameter. Let \\(u\\) denote a setting, such as a dose, sampling time, or instrument configuration, and let \\(I(\\theta;u)\\) be the information from one observation with density \\(p(y\\mid\\theta,u)\\). A design \\(\\xi=\\{(u_k,w_k)\\}_{k=1}^{K}\\) specifies settings and allocation proportions, with \\(w_k\\geq0\\) and \\(\\sum_k w_k=1\\). For \\(n\\) independent observations allocated in these proportions, the total information is\n\n\\[I(\\theta;\\xi)=n\\sum_{k=1}^{K}w_k I(\\theta;u_k).\\]\n\nAn exact design requires each \\(nw_k\\) to be an integer; an approximate design treats the weights as continuous optimization variables. The density must include the measurement-noise model: changing noise variance, correlation, or the response distribution changes the information associated with the same settings. Allocation-weight formulations are used both for linear measurements and for more general regression models. (web.stanford.edu)\n\nOptimality criteria reduce this matrix comparison to a scalar objective, usually with sample size or cost held fixed. D-optimality maximizes the determinant \\(\\det I(\\theta;\\xi)\\), or equivalently \\(\\log\\det I\\) when information is positive definite. It minimizes the determinant of inverse information and, under a normal approximation, the volume of a fixed-level confidence ellipsoid. A-optimality minimizes the trace \\(\\operatorname{tr}(I^{-1})\\), corresponding to the sum of parameter-variance bounds. E-optimality maximizes the smallest eigenvalue \\(\\lambda_{\\min}(I)\\); equivalently, it minimizes \\(\\lambda_{\\max}(I^{-1})\\), the largest variance bound among unit-length parameter contrasts. These objectives need not select the same design. A- and E-optimality also require a specified parameter scaling, since changing units changes the relative weighting of parameter directions. (itl.nist.gov)\n\nFor Gaussian linear regression, suppose \\(Y=X\\beta+\\varepsilon\\), where the design matrix \\(X\\) is fixed and the errors are independent with distribution \\(N(0,\\sigma^2)\\), with known \\(\\sigma^2\\). Differentiating the log-likelihood gives the score \\(X^{\\mathsf T}(Y-X\\beta)/\\sigma^2\\), whose covariance is\n\n\\[I_\\beta=\\frac{X^{\\mathsf T}X}{\\sigma^2}.\\]\n\nWhen \\(X\\) has full column rank, the least-squares estimator has exact covariance \\(\\sigma^2(X^{\\mathsf T}X)^{-1}\\). Rank deficiency leaves some coefficient combinations unidentifiable; near-collinearity produces small information eigenvalues and large variances in the corresponding directions. Thus replication increases information but cannot remove a rank deficiency if every replicate repeats the same insufficient set of regressor vectors. These results follow directly from the Gaussian linear-measurement formulation. (web.stanford.edu)\n\nFor a concrete calculation, consider an intercept and slope with settings \\(x_i\\in[-1,1]\\). The preceding formula gives\n\n\\[I_\\beta=\\frac{1}{\\sigma^2} \\begin{pmatrix} n&\\sum_i x_i\\\\ \\sum_i x_i&\\sum_i x_i^2 \\end{pmatrix}, \\qquad \\det I_\\beta=\\frac{n\\sum_i x_i^2-(\\sum_i x_i)^2}{\\sigma^4}.\\]\n\nFor even \\(n\\), equal allocation to \\(-1\\) and \\(1\\) attains the maximum determinant \\(n^2/\\sigma^4\\). With \\(n=10\\) and \\(\\sigma^2=4\\), this design yields \\(I_\\beta=\\operatorname{diag}(2.5,2.5)\\), so both coefficient variances are \\(0.4\\). Taking all ten observations at \\(x=0\\) instead gives \\(\\operatorname{diag}(2.5,0)\\), providing no slope information. This endpoint result concerns the specified straight-line model: adding a quadratic term makes the same two-setting design rank deficient because \\(x^2=1\\) at both endpoints. (web.stanford.edu)\n\nIn nonlinear or heteroscedastic models, information commonly depends on unknown parameter values. A locally optimal design optimizes \\(I(\\theta_0;\\xi)\\) at a specified nominal value \\(\\theta_0\\). An information-based Bayesian alternative averages a criterion over a prior distribution, for example maximizing \\(E_\\pi[\\log\\det I(\\theta;\\xi)]\\); this is not generally equivalent to maximizing the log-determinant of the prior-averaged matrix. A robust alternative optimizes worst-case performance over a specified parameter region, sometimes using efficiency relative to each parameter value’s local optimum. The prior or uncertainty region, admissible settings, noise assumptions, and chosen criterion are therefore part of any optimality claim, rather than incidental details. (conservancy.umn.edu)"
    },
    {
      "heading": "Information-Theoretic and Quantum Extensions",
      "text": "Fisher information and Shannon entropy quantify different properties of a probability model. Entropy describes uncertainty in outcomes: for a discrete variable, \\(H(X)=-\\sum_x p(x)\\log p(x)\\). Mutual information describes dependence through \\(I(X;Y)=D_{\\mathrm{KL}}(p_{XY}\\Vert p_Xp_Y)\\). Neither definition requires an unknown parameter. Fisher information instead measures sensitivity to a specified parameter through derivatives of the log-density; its numerical value depends on the parameter’s units. These distinctions persist for continuous distributions, where differential entropy replaces discrete entropy. Relations between entropy and Fisher information therefore connect distinct quantities rather than identify them. (arxiv.org)\n\nFor a differentiable probability density \\(f\\) on \\(\\mathbb R^d\\), the translation-information functional is\n\n\\[J(f)=\\int_{\\{f>0\\}}\\|\\nabla\\log f(x)\\|^2f(x)\\,dx.\\]\n\nHere differentiation is with respect to the observation coordinate. Sufficient regularity conditions include a positive, continuously differentiable density, finite \\(J(f)\\), integrable first derivatives, and boundary decay allowing integration by parts without boundary contributions. In the location family \\(p(x\\mid\\theta)=f(x-\\theta)\\), the parameter score is \\(-\\nabla\\log f(x-\\theta)\\); consequently, the location information matrix is \\(\\int(\\nabla\\log f)(\\nabla\\log f)^{\\mathsf T}f\\,dx\\), and \\(J(f)\\) is its trace. A direct connection to entropy is the de Bruijn identity: if \\(Z\\) is an independent standard Gaussian vector and the relevant differentiations and integrals are justified, then, for \\(t>0\\),\n\n\\[\\frac{d}{dt}h(X+\\sqrt t\\,Z) =\\frac12J(f_{X+\\sqrt t\\,Z}),\\]\n\nwith natural logarithms. Thus translation information measures the rate of entropy growth under Gaussian smoothing. (arxiv.org)\n\nIn quantum mechanics, a parameterized state is represented by a density matrix \\(\\rho_\\theta\\). Its symmetric logarithmic derivative (SLD) is a self-adjoint operator \\(L_\\theta\\) satisfying\n\n\\[\\partial_\\theta\\rho_\\theta =\\frac12(L_\\theta\\rho_\\theta+\\rho_\\theta L_\\theta), \\qquad F_Q(\\theta)=\\operatorname{Tr}(\\rho_\\theta L_\\theta^2).\\]\n\nThis defines the SLD version of quantum Fisher information. For a full-rank state the SLD is unique; for a regular fixed-rank family, its arbitrary kernel–kernel components do not affect \\(F_Q\\). For a pure state \\(\\rho_\\theta=|\\psi_\\theta\\rangle\\langle\\psi_\\theta|\\), the formula becomes \\(F_Q=4(\\langle\\partial_\\theta\\psi|\\partial_\\theta\\psi\\rangle-|\\langle\\psi|\\partial_\\theta\\psi\\rangle|^2)\\). In particular, unitary encoding \\(|\\psi_\\theta\\rangle=e^{-i\\theta G}|\\psi_0\\rangle\\) gives \\(F_Q=4\\operatorname{Var}_{\\psi_0}(G)\\), making sensitivity depend explicitly on fluctuations of the generator. (arxiv.org)\n\nA specified measurement, described by positive operators \\(M_y\\) with \\(\\sum_yM_y=\\mathbf1\\), produces classical probabilities \\(p(y\\mid\\theta)=\\operatorname{Tr}(\\rho_\\theta M_y)\\) and classical information \\(I_M(\\theta)\\). For scalar estimation, \\(I_M(\\theta)\\leq F_Q(\\theta)\\); Samuel L. Braunstein and Carlton M. Caves established the measurement-optimization characterization in 1994. Locally, measuring in an SLD eigenbasis attains the maximum, but the required basis generally depends on the unknown parameter. For \\(n\\) independent identical preparations and a locally unbiased estimator, the quantum Cramér–Rao bound is\n\n\\[\\operatorname{Var}_\\theta(\\hat\\theta)\\geq\\frac{1}{nF_Q(\\theta)}.\\]\n\nOptimizing measurement information does not itself guarantee finite-sample estimator efficiency; adaptive measurements can address the parameter dependence in suitable regular large-sample settings. (doi.org)\n\nFor several parameters, separate SLDs define\n\n\\[(F_Q)_{ij} =\\frac12\\operatorname{Tr}\\!\\left[\\rho(L_iL_j+L_jL_i)\\right].\\]\n\nWhen this matrix is nonsingular, locally unbiased estimation satisfies \\(\\operatorname{Cov}(\\hat\\theta)\\succeq(nF_Q)^{-1}\\). Unlike scalar estimation, however, one measurement need not attain the entire matrix bound: different SLDs may require incompatible measurements. Commuting SLDs permit a common eigenbasis and provide a sufficient compatibility condition. Noncommutation alone is not decisive; the weaker condition \\(\\operatorname{Tr}(\\rho[L_i,L_j])=0\\) governs asymptotic attainability in regular collective-measurement formulations. When compatibility fails, attainable precision requires measurement-dependent trade-offs or tighter bounds rather than simply interpreting every diagonal entry of \\(F_Q^{-1}\\) as simultaneously achievable. (quanestimation.github.io)"
    },
    {
      "heading": "Nonregular Models and Limits of Interpretation",
      "text": "For \\(X\\) following a uniform distribution on \\((0,\\theta)\\), with \\(\\theta>0\\), the density is \\(p_\\theta(x)=\\theta^{-1}\\mathbf 1_{\\{0<x<\\theta\\}}\\). At an interior observation, differentiating the log-density gives \\(s_\\theta(x)=-1/\\theta\\), so\n\n\\[E_\\theta[s_\\theta(X)]=-\\frac1\\theta,\\qquad E_\\theta[s_\\theta(X)^2]=\\frac1{\\theta^2},\\qquad -E_\\theta[\\partial_\\theta^2\\log p_\\theta(X)]=-\\frac1{\\theta^2}.\\]\n\nThe score therefore has nonzero mean, its second moment is not its variance, and the curvature identity fails. Differentiating the normalization integral involves a moving-boundary term:\n\n\\[\\frac{d}{d\\theta}\\int_0^\\theta\\frac{dx}{\\theta} =-\\frac1\\theta+\\frac1\\theta=0.\\]\n\nIgnoring that term invalidates the usual Cramér–Rao argument. For \\(n\\) independent observations, the interior sample score is \\(-n/\\theta\\), whose squared expectation is \\(n^2/\\theta^2\\), not \\(n/\\theta^2\\); the usual additivity argument also fails. Writing \\(M=\\max_iX_i\\), the unbiased estimator \\(T=(n+1)M/n\\) has \\(\\operatorname{Var}_\\theta(T)=\\theta^2/[n(n+2)]\\), below even the purported bound \\(\\theta^2/n^2\\). This contradicts no valid inequality: its regularity assumptions are absent. (ocw.mit.edu)\n\nParameter boundaries create a different obstruction. A variance component constrained to be nonnegative, for example, cannot fluctuate symmetrically around a true value of zero. Steven Self and Kung-Yee Liang’s 1987 analysis describes boundary limits through projections of Gaussian variables onto admissible cones; likelihood-ratio statistics can have mixtures of chi-squared distributions rather than the ordinary interior-parameter limit. Nondifferentiable models likewise require an explicit examination of the relevant smoothness conditions, rather than automatic use of Hessian formulas. If \\(E_\\theta[s_\\theta^2]=\\infty\\), the finite-information quadratic approximation and its usual asymptotic covariance formula are unavailable: treating \\(1/\\infty=0\\) as an attainable zero variance is unjustified. Alternative local-distance and nonregular asymptotic theories can describe such problems. (stat.cmu.edu)\n\nA singular information matrix concerns local sensitivity, not necessarily global identifiability. As a direct example, parameterize a unit-variance normal distribution by mean \\(a+b\\). Its information matrix is\n\n\\[I(a,b)= \\begin{pmatrix}1&1\\\\1&1\\end{pmatrix},\\]\n\nwith null direction \\((1,-1)^{\\mathsf T}\\): changing \\(a\\) and \\(b\\) oppositely leaves the distribution unchanged. In a Gaussian mixture model, coincident components similarly make their mixing weight unidentified and introduce degeneracy. By contrast, exchanging two distinct component labels gives the same density at separated parameter points; this discrete symmetry does not by itself force local singularity at each point. Thus label switching can produce multiple equivalent likelihood modes even where local information is nonsingular. (arxiv.org)\n\nUnder model misspecification, expectations taken within the fitted model need not describe actual sampling variability. For independent, identically distributed observations from \\(Q\\), let the interior pseudo-true parameter \\(\\theta^*\\) maximize \\(E_Q[\\log p_\\theta(X)]\\), equivalently minimizing Kullback–Leibler divergence when that expression is defined. Set\n\n\\[A=-E_Q[\\nabla_\\theta^2\\log p_{\\theta^*}(X)], \\qquad B=\\operatorname{Cov}_Q[\\nabla_\\theta\\log p_{\\theta^*}(X)].\\]\n\nUnder suitable smoothness, moment, identification, and nonsingularity assumptions, maximum-likelihood estimation satisfies\n\n\\[\\sqrt n(\\hat\\theta-\\theta^*)\\Rightarrow N(0,A^{-1}BA^{-1}).\\]\n\nThe resulting sandwich covariance is approximately \\(A^{-1}BA^{-1}/n\\) for the estimator itself. Correct specification restores \\(A=B=I(\\theta^*)\\); misspecification generally does not. Halbert White’s 1982 treatment formalized these distinctions, including the fact that convergence to \\(\\theta^*\\) need not mean consistency for a scientifically intended parameter. (faculty.utrgv.edu)\n\nInformation describes local distinguishability within a specified model. It does not determine an estimator’s finite-sample mean squared error, which also depends on bias, or resolve separated likelihood modes. Nor does it supply a target-independent ranking of experiments: in a multiparameter problem, one design can increase information in one direction while decreasing it in another, so comparison requires an inferential target or an explicit matrix criterion. These limitations persist even when every local information matrix is finite and invertible. (arxiv.org)"
    },
    {
      "heading": "Historical Development",
      "text": "Ronald A. Fisher’s 1922 paper “On the Mathematical Foundations of Theoretical Statistics,” published in Philosophical Transactions of the Royal Society, volume 222, pages 309–368, organized estimation around consistency, efficiency, and sufficiency. It distinguished unknown population parameters from statistics calculated from observations and developed maximum-likelihood estimation as a general method. Fisher’s terminology was not identical to its later textbook formulation: efficiency concerned comparative accuracy in large samples, while sufficiency concerned retaining all the sample’s information about the parameter. Information therefore entered through the problem of reducing observations without sacrificing inferential accuracy, rather than as an isolated definition. His discussion included the estimation of normal-distribution scale and the efficiency of moment-based estimates for Pearson frequency curves. (stats.org.uk)\n\n“Theory of Statistical Estimation,” published in 1925 in the Proceedings of the Cambridge Philosophical Society, volume 22, pages 700–725, refined this framework through a more explicit treatment of information and information loss. It examined how replacing the full sample by a statistic could reduce attainable accuracy, while a sufficient statistic preserved the relevant information. The paper also introduced the method subsequently known as Fisher scoring. In modern notation, this method updates a parameter estimate using the likelihood derivative multiplied by inverse expected information, rather than inverse sample-dependent curvature. Its computational role was to obtain likelihood estimates through successive corrections. These contributions developed across Fisher’s papers; the complete modern apparatus of information matrices, variance inequalities, and geometric interpretations should not be attributed wholesale to the 1922 publication. (digital.library.adelaide.edu.au)\n\nDuring the 1940s, information became central to explicit lower bounds on estimation variance. C. R. Rao’s 1945 paper “Information and the Accuracy Attainable in the Estimation of Statistical Parameters” established an information inequality for unbiased estimation and treated its multiparameter extension. Under the required differentiability and integrability conditions, the scalar result takes the modern form \\(\\operatorname{Var}_\\theta(T)\\geq 1/I(\\theta)\\) for an unbiased estimator of \\(\\theta\\). Harald Cramér’s Mathematical Methods of Statistics, issued in Sweden in 1945 and by Princeton University Press in 1946, supplied another foundational treatment; the bound became known as the Cramér–Rao inequality. This finite-sample inequality gave a precise counterpart to earlier comparisons based on large-sample efficiency. (gwern.net)\n\nRao’s same 1945 paper also interpreted the information matrix as a metric tensor on a space of probability distributions. His “population space” used the quadratic differential form \\(ds^2=\\sum_{i,j}I_{ij}\\,d\\theta_i\\,d\\theta_j\\) to measure the local discrepancy between neighboring populations, explicitly connecting statistical estimation with Riemannian geometry. Harold Jeffreys pursued a related invariance problem in “An Invariant Form for the Prior Probability in Estimation Problems” (1946). The resulting Jeffreys prior, expressed in modern notation as \\(\\pi(\\theta)\\propto\\sqrt{\\det I(\\theta)}\\), connected information with Bayesian inference by constructing a prior measure invariant under smooth changes of parameter coordinates. (gwern.net)\n\nInformation-based computation subsequently expanded to incomplete observations. Thomas A. Louis’s 1982 paper “Finding the Observed Information Matrix when Using the EM Algorithm” showed how to extract observed information when expectation–maximization was used for likelihood estimation. The calculation combined conditional expectations of complete-data derivatives with a correction involving the conditional covariance of the complete-data score. This made uncertainty calculations possible without treating unobserved quantities as if they had actually been measured. (markirwin.net)\n\nLater publications connected information geometry with quantum measurement and learning algorithms. Samuel L. Braunstein and Carlton M. Caves’s 1994 “Statistical Distance and the Geometry of Quantum States” optimized statistical distinguishability over quantum measurements, obtaining a Riemannian metric on density operators. Shun-ichi Amari’s 1998 “Natural Gradient Works Efficiently in Learning,” published in Neural Computation, used the natural gradient to account for the geometry of parameter spaces rather than their arbitrary coordinates. Its applications included perceptron learning, blind source separation, and linear dynamical systems, extending information-based reasoning from estimation accuracy to the design of optimization procedures. (doi.org)"
    }
  ],
  "links": [
    "probability-distribution",
    "likelihood-function",
    "statistics",
    "maximum-likelihood-estimation",
    "ronald-fisher",
    "experimental-design",
    "machine-learning",
    "expected-value",
    "probability-mass-function",
    "probability-density-function",
    "dominated-convergence-theorem",
    "variance",
    "covariance-matrix",
    "eigenvalues-and-eigenvectors",
    "normal-distribution",
    "bernoulli-distribution",
    "binomial-distribution",
    "logit",
    "chain-rule",
    "poisson-distribution",
    "sufficient-statistic",
    "hessian-matrix",
    "estimator",
    "covariance",
    "cauchy-schwarz-inequality",
    "sample-mean",
    "estimator-bias",
    "mean-squared-error",
    "central-limit-theorem",
    "standard-error",
    "confidence-interval",
    "statistical-independence",
    "joint-probability-distribution",
    "conditional-expectation",
    "expectation-maximization-algorithm",
    "jacobian-matrix",
    "manifold",
    "metric-tensor",
    "geodesic",
    "kullback-leibler-divergence",
    "taylor-series",
    "bayesian-inference",
    "posterior-distribution",
    "automatic-differentiation",
    "monte-carlo-method",
    "newtons-method",
    "generalized-linear-model",
    "logistic-regression",
    "regularization",
    "determinant",
    "matrix-trace",
    "linear-regression",
    "matrix-rank",
    "prior-distribution",
    "information-entropy",
    "mutual-information",
    "quantum-mechanics",
    "density-matrix",
    "self-adjoint-operator",
    "uniform-distribution",
    "statistical-identification",
    "null-space",
    "gaussian-mixture-model"
  ],
  "sources": [
    {
      "title": "Lecture 8: Fisher Information",
      "url": "https://stat210a.berkeley.edu/fall-2024/handwritten/lecture08-fisherinfo.pdf"
    },
    {
      "title": "Score Function and Fisher Information",
      "url": "https://stat210a.berkeley.edu/fall-2024/reader/score-fisher.html"
    },
    {
      "title": "1 Fisher Information",
      "url": "https://www.stat.cmu.edu/~larry/%3Dstat705/Lecture13.pdf"
    },
    {
      "title": "Stat 5101 (Geyer) Course Notes",
      "url": "https://www.stat.umn.edu/geyer/5101/notes/n2.pdf"
    },
    {
      "title": "Stats 300b: Theory of Statistics",
      "url": "https://web.stanford.edu/class/stats300b/ScribeNotes/2021/lecture-06.pdf"
    },
    {
      "title": "A Tutorial on Fisher Information",
      "url": "https://arxiv.org/abs/1705.01064"
    },
    {
      "title": "STATS426: Introduction to Theoretical Statistics",
      "url": "https://pages.stat.wisc.edu/~kdlevin/teaching/Fall2019/STATS426/hw/homework7.pdf"
    },
    {
      "title": "Proof of Asymptotic Normality, and Confidence Intervals",
      "url": "https://stat135.berkeley.edu/spring-2026/lectures/lecture-15.html"
    },
    {
      "title": "Exponential Families",
      "url": "https://www.stat.berkeley.edu/~wfithian/courses/stat210a/exponential-families.html"
    },
    {
      "title": "528-5.dvi",
      "url": "https://www.stat.purdue.edu/~dasgupta/528-5.pdf"
    },
    {
      "title": "Lecture 15: October 2—Cramér–Rao Bound",
      "url": "https://www.stat.cmu.edu/~siva/teaching/705/lec15.pdf"
    },
    {
      "title": "Statistical Signal Processing",
      "url": "https://cpb-us-e1.wpmucdn.com/blogs.rice.edu/dist/7/3490/files/2014/07/notes.pdf"
    },
    {
      "title": "Analysis of Neural Data",
      "url": "https://www.stat.cmu.edu/~kass/sam/all/all-old.pdf"
    },
    {
      "title": "Likelihood Based Inference",
      "url": "https://stat210a.berkeley.edu/fall-2024/handwritten/lecture23-likelihoodbasedinference.pdf"
    },
    {
      "title": "Large Sample Theory for the Maximum Likelihood Estimator",
      "url": "https://stat135.berkeley.edu/fall-2026/Modules/10-MLEAsyNormal.html"
    },
    {
      "title": "22 Inference in GLMs – STAT 9610 Lecture Notes",
      "url": "https://katsevich-teaching.github.io/stat-9610-notes/glm-inference.html"
    },
    {
      "title": "Score Function and Fisher Information",
      "url": "https://stat210a.berkeley.edu/fall-2025/reader/score-fisher.html"
    },
    {
      "title": "A note on insufficiency and the preservation of Fisher information",
      "url": "https://arxiv.org/abs/1107.3797"
    },
    {
      "title": "Sufficiency",
      "url": "https://stat210a.berkeley.edu/fall-2024/reader/sufficiency.html"
    },
    {
      "title": "A Sufficiency Paradox: An Insufficient Statistic Preserving the Fisher Information",
      "url": "https://faculty.wharton.upenn.edu/wp-content/uploads/2012/04/Sufficiency-paradox.pdf"
    },
    {
      "title": "An Elementary Introduction to Information Geometry",
      "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC7650632/"
    },
    {
      "title": "Fisher Information",
      "url": "https://www2.stat.duke.edu/courses/Spring15/sta250/lec/fish.pdf"
    },
    {
      "title": "Information-Geometric Optimization",
      "url": "https://jmlr.org/papers/volume18/14-467/14-467.pdf"
    },
    {
      "title": "A Tutorial on Fisher Information",
      "url": "https://arxiv.org/html/1705.01064v2"
    },
    {
      "title": "STAT 203: Lecture 4",
      "url": "https://faculty.ucr.edu/~jflegal/203/04-priors.html"
    },
    {
      "title": "Automatic Differentiation in Machine Learning: a Survey",
      "url": "https://jmlr.org/papers/volume18/17-468/17-468.pdf"
    },
    {
      "title": "New Insights and Perspectives on the Natural Gradient Method",
      "url": "https://arxiv.org/html/1412.1193v11"
    },
    {
      "title": "Comparing Newton-Raphson with Fisher Scoring",
      "url": "https://www.stat.cmu.edu/~brian/711/week13/mle-2.pdf"
    },
    {
      "title": "Limitations of the Empirical Fisher Approximation for Natural Gradient Descent",
      "url": "https://papers.nips.cc/paper_files/paper/2019/file/46a558d97954d0692411c861cf78ef79-Paper.pdf"
    },
    {
      "title": "Optimizing Neural Networks with Kronecker-factored Approximate Curvature",
      "url": "https://proceedings.mlr.press/v37/martens15.pdf"
    },
    {
      "title": "5.2.1. D-Optimal designs",
      "url": "https://www.itl.nist.gov/div898/handbook/pri/section5/pri521.htm"
    },
    {
      "title": "Convex Optimization",
      "url": "https://web.stanford.edu/class/ee364a/lectures/stat.pdf"
    },
    {
      "title": "Generalization of the de Bruijn's identity to general φ-entropies and φ-Fisher informations",
      "url": "https://arxiv.org/pdf/1611.09400"
    },
    {
      "title": "Quantum estimation for quantum technology",
      "url": "https://arxiv.org/pdf/0804.2981"
    },
    {
      "title": "Statistical distance and the geometry of quantum states",
      "url": "https://doi.org/10.1103/PhysRevLett.72.3439"
    },
    {
      "title": "Fisher information in quantum statistics",
      "url": "https://arxiv.org/abs/quant-ph/9808009"
    },
    {
      "title": "Quantum Fisher information matrix and multiparameter estimation",
      "url": "https://quanestimation.github.io/group/Publications/pdf/JPA_53_023001.pdf"
    },
    {
      "title": "381 Statistical Method in Economics, Lec 6: Efficient Estimators, Rao-Cramer Bound",
      "url": "https://ocw.mit.edu/courses/14-381-statistical-method-in-economics-fall-2018/15224973dfc1c2b4287804c8712681f7_MIT14_381F18_lec6.pdf"
    },
    {
      "title": "Asymptotic Properties of Maximum Likelihood Estimators and Likelihood Ratio Tests Under Nonstandard Conditions",
      "url": "https://www.stat.cmu.edu/~brian/763-2015/week06/papers/self-liang-1987.pdf"
    },
    {
      "title": "On optimal designs for non-regular models",
      "url": "https://arxiv.org/abs/1802.10024"
    },
    {
      "title": "Singularity, Misspecification, and the Convergence Rate of EM",
      "url": "https://arxiv.org/abs/1810.00828"
    },
    {
      "title": "Identifying Bayesian Mixture Models",
      "url": "https://mc-stan.org/learn-stan/case-studies/identifying_mixture_models.html"
    },
    {
      "title": "Maximum Likelihood Estimation of Misspecified Models",
      "url": "https://faculty.utrgv.edu/diego.escobari/teaching/Econ8370/Papers/White%281982%29Econometrica-Corey.pdf"
    },
    {
      "title": "042: Theory of Statistical Estimation.",
      "url": "https://digital.library.adelaide.edu.au/items/5d9e2da4-f384-47f6-9aac-056832ea5f1d"
    },
    {
      "title": "Information and the Accuracy Attainable in the Estimation of Statistical Parameters",
      "url": "https://gwern.net/doc/statistics/1992-rao.pdf"
    },
    {
      "title": "Mathematical methods of statistics",
      "url": "https://ci.nii.ac.jp/ncid/BA01001228"
    },
    {
      "title": "An invariant form for the prior probability in estimation problems",
      "url": "https://pubmed.ncbi.nlm.nih.gov/20998741/"
    },
    {
      "title": "An invariant form for the prior probability in estimation problems",
      "url": "https://scispace.com/papers/an-invariant-form-for-the-prior-probability-in-estimation-4aws3zhzza?citations_page=104"
    },
    {
      "title": "Noninformative priors for one parameter of many",
      "url": "https://scispace.com/papers/noninformative-priors-for-one-parameter-of-many-1xiy3ldfr4"
    }
  ],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}