{
  "id": "cramer-rao-inequality",
  "title": "Cramér–Rao Inequality",
  "lang": "en",
  "source_lang": "en",
  "summary": "The Cramér–Rao inequality bounds an estimator’s variance in terms of Fisher information, establishing a fundamental limit on statistical precision under regularity conditions.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "Cramér–Rao bound",
    "Cramér–Rao lower bound",
    "Information inequality",
    "Rao–Cramér inequality"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/cramer-rao-inequality/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/cramer-rao-inequality/",
      "json": "https://aiwiki.page/zh/cramer-rao-inequality.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The Cramér–Rao inequality is a theorem in statistics that places a lower bound on the variance of an estimator in terms of the information contained in the observations. In its best-known form, an unbiased estimator \\(T\\) of a scalar parameter \\(\\theta\\) satisfies\n\n\\[\\operatorname{Var}_{\\theta}(T)\\geq \\frac{1}{I(\\theta)},\\]\n\nwhere \\(I(\\theta)\\) is the Fisher information in the entire observed sample. The inequality requires regularity conditions connecting differentiation of the statistical model with integration over possible observations; it is not a universal bound for every estimation problem.\n\nThe result provides a benchmark for statistical precision independently of the particular algorithm used to construct an estimator. It helps characterize efficient estimators, explains the large-sample behavior of maximum likelihood estimation, and guides experimental design. Its extensions address vector parameters, biased estimators, nuisance parameters, Bayesian risk, and quantum measurements. A central distinction throughout these applications is between a mathematical lower bound and an attainable level of performance: the bound may hold exactly without any estimator attaining it."
    },
    {
      "heading": "Historical development and statistical setting",
      "text": "The inequality emerged from the development of mathematical estimation theory in the first half of the twentieth century. Ronald Fisher introduced and developed the concept of information in connection with likelihood, sufficiency, and the precision of statistical estimates. Information expresses how strongly a probability distribution changes when its parameter changes. If two nearby parameter values produce markedly different distributions, observations can generally distinguish them more precisely than if the distributions change only slightly.\n\nMaurice Fréchet obtained an early form of the information inequality in 1943. C. R. Rao published his formulation in the 1945 paper “Information and the Accuracy Attainable in the Estimation of Statistical Parameters,” and Harald Cramér developed the result in his 1946 book Mathematical Methods of Statistics. These contributions account for the alternative names Cramér–Rao, Rao–Cramér, and occasionally Fréchet–Cramér–Rao inequality. Rao’s work also connected statistical information with geometry, a connection that later became central to information geometry.\n\nThe underlying statistical setting is a family of distributions\n\n\\[\\mathcal P=\\{P_\\theta:\\theta\\in\\Theta\\}.\\]\n\nThe observations \\(X\\) have distribution \\(P_\\theta\\), but \\(\\theta\\) is unknown. An estimator is a measurable function \\(T(X)\\) of the observations, constructed without knowledge of the true parameter. Repeated samples at a fixed \\(\\theta\\) produce a sampling distribution for \\(T\\), whose spread measures one aspect of estimation uncertainty. The Cramér–Rao inequality relates this spread to a property of the model rather than to a specific computational method.\n\nHistorically, the inequality helped distinguish several meanings of statistical optimality. An estimator may have minimum variance among unbiased estimators, attain the information bound, or be efficient only as sample size tends to infinity. These properties are not equivalent. A minimum-variance unbiased estimator need not attain the Cramér–Rao bound, and an estimator with small finite-sample bias may have lower mean squared error than any unbiased estimator. The theorem therefore belongs to a broader framework of decision theory, in which performance depends on the estimator class and the chosen loss function."
    },
    {
      "heading": "Scalar formulation and regularity conditions",
      "text": "Suppose the model has a density or mass function \\(p_\\theta(x)\\) with respect to a fixed dominating measure. The density may be a probability density function for continuous observations or a probability mass function for discrete observations. The likelihood function treats \\(p_\\theta(x)\\) as a function of \\(\\theta\\) for the observed value \\(x\\). Its logarithmic derivative,\n\n\\[U_\\theta(X)=\\frac{\\partial}{\\partial\\theta}\\log p_\\theta(X),\\]\n\nis called the score function. Under suitable regularity conditions,\n\n\\[E_\\theta[U_\\theta]=0, \\qquad I(\\theta)=E_\\theta[U_\\theta^2].\\]\n\nThus information is the variance of the score.\n\nLet \\(T\\) estimate a differentiable scalar function \\(g(\\theta)\\) without bias:\n\n\\[E_\\theta[T]=g(\\theta).\\]\n\nIf \\(T\\) has finite variance and differentiation may be interchanged with integration in the relevant expectations, then\n\n\\[\\operatorname{Var}_\\theta(T) \\geq \\frac{[g'(\\theta)]^2}{I(\\theta)}, \\qquad 0<I(\\theta)<\\infty.\\]\n\nThe familiar reciprocal-information bound is the special case \\(g(\\theta)=\\theta\\). The derivative in the numerator matters: estimating \\(\\theta^2\\), for example, is not the same precision problem as estimating \\(\\theta\\).\n\nA common sufficient framework assumes that \\(\\Theta\\) is an open interval, the support of the distribution does not depend on \\(\\theta\\), and the density is differentiable with suitable integrable bounds on its derivatives. These assumptions allow a derivative to pass through an integral, often through the dominated convergence theorem. Fixed support alone is not sufficient, and parameter-dependent support is not by itself a complete diagnosis; the decisive issue is whether the identities used in the proof are valid. Under stronger second-order assumptions,\n\n\\[I(\\theta) = -E_\\theta\\!\\left[ \\frac{\\partial^2}{\\partial\\theta^2}\\log p_\\theta(X) \\right].\\]\n\nThis expected curvature should be distinguished from the random observed curvature evaluated at a particular sample.\n\nFor \\(n\\) observations that satisfy statistical independence and have a common distribution, the sample score is the sum of the individual scores. Their zero means make the cross terms vanish, giving\n\n\\[I_n(\\theta)=nI_1(\\theta).\\]\n\nConsequently,\n\n\\[\\operatorname{Var}_\\theta(T)\\geq\\frac{1}{nI_1(\\theta)}\\]\n\nfor unbiased estimation of \\(\\theta\\). Independent but nonidentically distributed observations contribute information additively as well. Dependent observations require the information of their full joint distribution; multiplying a marginal information by sample size is generally unjustified."
    },
    {
      "heading": "Proof and geometric interpretation",
      "text": "The proof rests on differentiating an expected value and applying the Cauchy–Schwarz inequality. Starting with \\(E_\\theta[T]=g(\\theta)\\), the regularity assumptions give\n\n\\[\\begin{aligned} g'(\\theta) &=\\frac{\\partial}{\\partial\\theta} \\int T(x)p_\\theta(x)\\,dx\\\\ &=\\int T(x)\\frac{\\partial p_\\theta(x)}{\\partial\\theta}\\,dx\\\\ &=E_\\theta[TU_\\theta]. \\end{aligned}\\]\n\nBecause \\(E_\\theta[U_\\theta]=0\\), the final expression equals\n\n\\[E_\\theta[(T-g(\\theta))U_\\theta] =\\operatorname{Cov}_\\theta(T,U_\\theta).\\]\n\nHere the integration notation also represents summation in a discrete model.\n\nApplying Cauchy–Schwarz to the centered estimator and the score yields\n\n\\[[g'(\\theta)]^2 \\leq E_\\theta[(T-g(\\theta))^2]\\, E_\\theta[U_\\theta^2] = \\operatorname{Var}_\\theta(T)\\,I(\\theta).\\]\n\nDivision by positive information establishes the result. The proof identifies the precise statistical role of regularity: it ensures that a change in the estimator’s expectation is represented by its covariance with the score. The final inequality is then a general fact about square-integrable random variables.\n\nThere is also a direct interpretation in a Hilbert space. At a fixed parameter value, centered square-integrable functions of the data form an inner-product space with\n\n\\[\\langle A,B\\rangle=E_\\theta[AB].\\]\n\nThe component of \\(T-g(\\theta)\\) along the score is\n\n\\[\\frac{g'(\\theta)}{I(\\theta)}U_\\theta.\\]\n\nWriting the estimator’s centered error as this component plus an orthogonal residual \\(R_\\theta\\) gives\n\n\\[T-g(\\theta) = \\frac{g'(\\theta)}{I(\\theta)}U_\\theta+R_\\theta, \\qquad E_\\theta[R_\\theta U_\\theta]=0.\\]\n\nBy the identity for the squared length of orthogonal components,\n\n\\[\\operatorname{Var}_\\theta(T) = \\frac{[g'(\\theta)]^2}{I(\\theta)} + E_\\theta[R_\\theta^2].\\]\n\nThis decomposition makes the inequality more informative than a numerical bound alone. The excess variance is exactly the squared size of a component orthogonal to the score. The score describes the first-order change in the distribution as the parameter moves; the estimator must have a specified response to that change. Any remaining fluctuation increases variance without contributing to this required local response. Attainment means that no such residual remains.\n\nThe argument is local in the parameter: expectations and inner products are evaluated under one \\(P_\\theta\\). Nevertheless, an estimator claimed to be unbiased throughout a model must satisfy its expectation identity across the parameter space. Confusing pointwise statements with requirements on a single estimator valid for all parameter values can lead to incorrect claims of attainability."
    },
    {
      "heading": "Equality, efficient estimators, and exact examples",
      "text": "Equality holds precisely when the centered estimator is proportional to the score:\n\n\\[T-g(\\theta) = \\frac{g'(\\theta)}{I(\\theta)}U_\\theta \\quad P_\\theta\\text{-almost surely}.\\]\n\nThe qualification almost surely allows differences on events of probability zero. An unbiased estimator satisfying this condition is called efficient at that parameter value. Efficiency throughout the model requires one statistic, independent of the unknown parameter, to satisfy the equality condition for every parameter value. This is a restrictive requirement.\n\nFor independent observations from a normal distribution with mean \\(\\mu\\) and known variance \\(\\sigma^2\\),\n\n\\[U_\\mu=\\frac{1}{\\sigma^2}\\sum_{i=1}^n(X_i-\\mu), \\qquad I_n(\\mu)=\\frac{n}{\\sigma^2}.\\]\n\nThe sample mean satisfies\n\n\\[\\overline X-\\mu=\\frac{\\sigma^2}{n}U_\\mu, \\qquad \\operatorname{Var}(\\overline X)=\\frac{\\sigma^2}{n},\\]\n\nand therefore attains the bound exactly. Two other standard examples have the same property. For Bernoulli observations with success probability \\(p\\in(0,1)\\),\n\n\\[I_n(p)=\\frac{n}{p(1-p)}, \\qquad \\operatorname{Var}(\\overline X)=\\frac{p(1-p)}{n}.\\]\n\nFor Poisson observations with mean \\(\\lambda>0\\),\n\n\\[I_n(\\lambda)=\\frac{n}{\\lambda}, \\qquad \\operatorname{Var}(\\overline X)=\\frac{\\lambda}{n}.\\]\n\nBoth sample means are unbiased and efficient.\n\nExact attainment is not automatic even in an elementary regular model. Suppose \\(X_1,\\ldots,X_n\\) follow an exponential distribution with rate \\(\\lambda\\), and let \\(S=\\sum_iX_i\\). The score and information are\n\n\\[U_\\lambda=\\frac{n}{\\lambda}-S, \\qquad I_n(\\lambda)=\\frac{n}{\\lambda^2}.\\]\n\nFor \\(n>2\\), the estimator\n\n\\[T=\\frac{n-1}{S}\\]\n\nis unbiased for \\(\\lambda\\), with\n\n\\[\\operatorname{Var}_\\lambda(T)=\\frac{\\lambda^2}{n-2}.\\]\n\nThis exceeds the Cramér–Rao bound \\(\\lambda^2/n\\). The estimator depends reciprocally on \\(S\\), whereas equality would require an affine relationship with the score.\n\nEquality has a close connection with the exponential family of distributions. In a scalar problem, if an estimator of \\(\\theta\\) attains the bound throughout an interval and the necessary smoothness conditions hold, the equality condition implies\n\n\\[\\frac{\\partial}{\\partial\\theta}\\log p_\\theta(x) = I(\\theta)\\bigl(T(x)-\\theta\\bigr).\\]\n\nIntegrating with respect to \\(\\theta\\) produces a representation of the form\n\n\\[p_\\theta(x)=h(x)\\exp\\{A(\\theta)T(x)-B(\\theta)\\}.\\]\n\nThus exact efficiency often reflects a special algebraic structure of the model. Membership in an exponential family does not, however, guarantee an unbiased efficient estimator for every function of its parameter."
    },
    {
      "heading": "Vector parameters and nuisance parameters",
      "text": "For a parameter vector \\(\\theta\\in\\mathbb R^p\\), the score is the gradient of the log likelihood:\n\n\\[U_\\theta=\\nabla_\\theta\\log p_\\theta(X).\\]\n\nIts covariance matrix,\n\n\\[I(\\theta)=E_\\theta[U_\\theta U_\\theta^{\\mathsf T}],\\]\n\nis the Fisher information matrix. Under regularity it is symmetric and positive semidefinite. Suppose \\(T\\in\\mathbb R^q\\) is unbiased for a differentiable vector-valued function \\(g(\\theta)\\), and let \\(G(\\theta)\\) be its \\(q\\times p\\) Jacobian matrix. If \\(I(\\theta)\\) is nonsingular, the matrix Cramér–Rao inequality is\n\n\\[\\operatorname{Cov}_\\theta(T) \\succeq G(\\theta)I(\\theta)^{-1}G(\\theta)^{\\mathsf T}.\\]\n\nThe symbol \\(\\succeq\\) means that the difference is positive semidefinite.\n\nThis is stronger than separate lower bounds on the component variances. For every fixed vector \\(a\\),\n\n\\[\\operatorname{Var}_\\theta(a^{\\mathsf T}T) \\geq a^{\\mathsf T}GI^{-1}G^{\\mathsf T}a.\\]\n\nThe inequality therefore constrains every linear combination of the estimated quantities, including combinations affected by covariance. When \\(g(\\theta)=\\theta\\), the Jacobian is the identity and the bound reduces to \\(I^{-1}\\). The matrix proof follows the scalar proof by projecting the centered estimator onto the span of all score components.\n\nNuisance parameters are parameters needed to describe the observations but not themselves the estimation target. Write \\(\\theta=(\\psi,\\lambda)\\), where \\(\\psi\\) is the target and \\(\\lambda\\) is nuisance, and partition the information:\n\n\\[I= \\begin{pmatrix} I_{\\psi\\psi}&I_{\\psi\\lambda}\\\\ I_{\\lambda\\psi}&I_{\\lambda\\lambda} \\end{pmatrix}.\\]\n\nFor a scalar target, when the relevant inverses exist,\n\n\\[\\operatorname{Var}(\\widehat\\psi) \\geq \\left( I_{\\psi\\psi} -I_{\\psi\\lambda}I_{\\lambda\\lambda}^{-1}I_{\\lambda\\psi} \\right)^{-1}.\\]\n\nThe denominator is the information remaining after projecting the target score away from the nuisance-score directions. It is no greater than \\(I_{\\psi\\psi}\\), the information that would apply if the nuisance parameter were known.\n\nFor a normal distribution parameterized by its mean \\(\\mu\\) and variance \\(v\\), one observation has information\n\n\\[I(\\mu,v)= \\begin{pmatrix} 1/v&0\\\\ 0&1/(2v^2) \\end{pmatrix}.\\]\n\nThe zero off-diagonal terms mean that the mean and variance scores are orthogonal. With \\(n\\) observations, the bound for unbiased estimation of \\(v\\) is \\(2v^2/n\\). Yet the usual unbiased sample variance has variance \\(2v^2/(n-1)\\). This is another case in which the information bound is valid but unattained.\n\nSingular information requires separate treatment. It may indicate redundant coordinates or a failure of local statistical identification, although a singular first-order information matrix does not always mean global nonidentifiability. Generalized-inverse bounds require compatibility conditions between the target derivative and the range of the information matrix; replacing an inverse mechanically is not sufficient."
    },
    {
      "heading": "Biased estimation and mean squared error",
      "text": "Unbiasedness is a condition on an estimator’s expectation, not a universal criterion of quality. Define the bias of an estimator \\(T\\) of \\(\\theta\\) by\n\n\\[b(\\theta)=E_\\theta[T]-\\theta.\\]\n\nThe same score argument applies to the mean function \\(m(\\theta)=\\theta+b(\\theta)\\), giving\n\n\\[\\operatorname{Var}_\\theta(T) \\geq \\frac{[1+b'(\\theta)]^2}{I(\\theta)}.\\]\n\nConsequently, its mean squared error satisfies\n\n\\[E_\\theta[(T-\\theta)^2] \\geq b(\\theta)^2+ \\frac{[1+b'(\\theta)]^2}{I(\\theta)}.\\]\n\nFor estimation of \\(g(\\theta)\\), the numerator becomes \\([g'(\\theta)+b'(\\theta)]^2\\).\n\nThe derivative of bias is essential. An estimator can have small bias at one parameter value while its mean responds quite differently to nearby changes in the parameter. The Cramér–Rao argument measures this local response through the derivative of the mean. Knowing the bias at one isolated point therefore does not determine the appropriate variance bound there. Local unbiasedness for \\(\\theta\\) ordinarily requires both the correct mean and unit derivative of the mean at the point in question.\n\nA simple normal example illustrates the bias–variance tradeoff. If \\(\\overline X\\) estimates a normal mean with known variance \\(\\sigma^2\\), consider\n\n\\[T_a=a\\overline X\\]\n\nfor a fixed constant \\(a\\). Its bias and variance are\n\n\\[b(\\mu)=(a-1)\\mu, \\qquad \\operatorname{Var}_\\mu(T_a)=\\frac{a^2\\sigma^2}{n}.\\]\n\nSince \\(1+b'(\\mu)=a\\), this estimator attains the biased variance bound. Its mean squared error is\n\n\\[\\operatorname{MSE}_\\mu(T_a) = \\frac{a^2\\sigma^2}{n}+(a-1)^2\\mu^2.\\]\n\nFor \\(0<a<1\\), it improves on the sample mean near \\(\\mu=0\\), while the bias term makes it worse sufficiently far away.\n\nA constant estimator is an extreme but useful illustration. If \\(T=c\\), its variance is zero, its bias is \\(c-\\theta\\), and \\(1+b'(\\theta)=0\\). The biased inequality is consistent with its zero variance, while its squared error is \\((c-\\theta)^2\\). No positive pointwise lower bound can apply to the mean squared error of every possible estimator at every parameter value without additional restrictions.\n\nThese distinctions matter for regularization methods, including ridge regression and other shrinkage procedures. Such estimators may deliberately accept bias to reduce error. The unbiased information bound is not a universal lower bound for their risk. Their analysis requires a bias-sensitive inequality or another criterion, such as integrated risk, minimax risk, or performance over a specified parameter neighborhood."
    },
    {
      "heading": "Large-sample efficiency and likelihood methods",
      "text": "In many regular models, exact finite-sample attainment is unavailable, but maximum likelihood estimators approach the information bound as sample size increases. Under conditions ensuring identification, consistency, and appropriate smoothness,\n\n\\[\\sqrt n(\\widehat\\theta_n-\\theta_0) \\xrightarrow{d} N\\!\\left(0,I_1(\\theta_0)^{-1}\\right).\\]\n\nThe notation denotes convergence in distribution, and \\(I_1\\) is the information in one observation. For a vector parameter, the limiting law is a multivariate normal distribution. This statement concerns asymptotic distributions; convergence of the actual variances requires additional control of moments or tails.\n\nThe mechanism comes from expanding the likelihood score near the true parameter. If an interior maximum satisfies \\(U_n(\\widehat\\theta_n)=0\\), a first-order expansion gives, schematically,\n\n\\[0\\approx U_n(\\theta_0) + H_n(\\theta_0)(\\widehat\\theta_n-\\theta_0),\\]\n\nwhere \\(H_n\\) is the Hessian matrix of the log likelihood. Typically,\n\n\\[n^{-1/2}U_n(\\theta_0) \\xrightarrow{d}N(0,I_1(\\theta_0)), \\qquad -\\frac{1}{n}H_n(\\theta_0) \\xrightarrow{P}I_1(\\theta_0).\\]\n\nThe first limit follows from a central limit theorem, and the second commonly from a law of large numbers. Together they yield\n\n\\[\\sqrt n(\\widehat\\theta_n-\\theta_0) \\approx I_1(\\theta_0)^{-1}\\frac{U_n(\\theta_0)}{\\sqrt n}.\\]\n\nThis explains the frequent appearance of inverse information in a reported standard error. Expected information or observed likelihood curvature can provide a large-sample covariance approximation, leading to approximate confidence intervals. These approximations can be inaccurate in small samples, near parameter boundaries, or when the likelihood is far from quadratic. An inverse Hessian is therefore not, by itself, a finite-sample precision guarantee.\n\nThe local asymptotic normality framework gives a more general interpretation. It studies alternatives of the form\n\n\\[\\theta=\\theta_0+\\frac{h}{\\sqrt n},\\]\n\nunder which a regular model approaches a Gaussian shift experiment. Appropriate notions of estimator regularity then produce asymptotic information bounds that do not require exact finite-sample unbiasedness. This is the setting in which inverse information becomes a broad limit on regular large-sample estimation.\n\nRegularity of the estimator remains important. Hodges-type superefficient estimators can have smaller limiting error than the usual information benchmark at selected parameter values, while performing poorly along nearby parameter sequences. This does not contradict either the finite-sample unbiased inequality or the asymptotic theory. Pointwise asymptotic behavior is weaker than uniform behavior over shrinking neighborhoods."
    },
    {
      "heading": "Nonregular models and limits of applicability",
      "text": "A standard failure occurs when the support depends on the parameter. Let \\(X_1,\\ldots,X_n\\) be independent observations from the uniform distribution on \\((0,\\theta)\\), where \\(\\theta>0\\). Away from the moving boundary, differentiating the log density appears to give\n\n\\[U_\\theta=-\\frac{n}{\\theta}.\\]\n\nThis cannot be a regular score because its expectation is not zero. The missing contribution comes from the changing support, and ordinary differentiation under the integral does not capture the full change in probability.\n\nThe sample maximum \\(M=\\max_iX_i\\) has expectation \\(n\\theta/(n+1)\\), so\n\n\\[T=\\frac{n+1}{n}M\\]\n\nis unbiased for \\(\\theta\\). Its variance is\n\n\\[\\operatorname{Var}_\\theta(T) = \\frac{\\theta^2}{n(n+2)}.\\]\n\nThis decreases at order \\(n^{-2}\\), rather than the usual regular-model order \\(n^{-1}\\). It does not violate the Cramér–Rao theorem because the required score identities fail. The apparent contradiction results from applying a regular-model formula outside its assumptions.\n\nBoundary parameters create a different difficulty. For a Bernoulli probability \\(p\\), the regular information formula applies on \\(0<p<1\\); at \\(p=0\\) or \\(p=1\\), the distribution is degenerate and two-sided parameter perturbations are unavailable. Variance components constrained to be nonnegative can likewise have nonnormal limiting distributions when the true value is zero. A model may be smooth in its interior yet require different asymptotic analysis at its boundary.\n\nOther complications include infinite information, nondifferentiable models, and information matrices that become singular at particular parameter values. Mixture models can lose regularity when components coincide or a mixing weight vanishes. Increasing parameter dimension or allowing an expanding collection of nuisance parameters also requires assumptions beyond the fixed-dimensional theory. A symbolic likelihood derivative does not establish that a particular information bound is meaningful.\n\nModel misspecification raises a separate issue. If the actual data distribution lies outside the assumed family, a likelihood estimator may converge to the parameter minimizing Kullback–Leibler divergence from the true distribution to the model. Under suitable conditions, its asymptotic covariance has the sandwich form\n\n\\[A^{-1}BA^{-1},\\]\n\nwhere \\(A\\) is the negative expected log-likelihood Hessian and \\(B\\) is the score covariance at that limiting parameter. Under correct specification, \\(A=B=I\\); otherwise, the identity need not hold. Ordinary inverse-information calculations can then misrepresent uncertainty even when the numerical likelihood maximization succeeds."
    },
    {
      "heading": "Experimental design and measurement applications",
      "text": "Information depends on how data are collected as well as on the parameter. If an experiment is controlled by a design variable \\(d\\), its information can be written \\(I(\\theta;d)\\). Independent measurements contribute additive information, enabling a comparison of measurement times, sensor positions, stimulus settings, or sample allocations. For scalar estimation, maximizing information minimizes the Cramér–Rao variance benchmark. This comparison is meaningful only relative to the assumed observation model and any nuisance parameters.\n\nFor example, in the normal linear regression model\n\n\\[Y=X\\beta+\\varepsilon, \\qquad \\varepsilon\\sim N(0,\\sigma^2I_n),\\]\n\nwith known \\(\\sigma^2\\) and a fixed full-column-rank design matrix,\n\n\\[I(\\beta)=\\frac{1}{\\sigma^2}X^{\\mathsf T}X.\\]\n\nThe ordinary least squares estimator has covariance\n\n\\[\\sigma^2(X^{\\mathsf T}X)^{-1}\\]\n\nand attains the bound. Nearly dependent design columns make the information poorly conditioned, producing high variance in some parameter combinations. This is a statistical consequence of the same geometry that can cause numerical difficulties in solving the regression equations.\n\nThere is no single best design criterion for every vector estimation problem. D-optimal design maximizes the determinant of the information matrix, corresponding to reducing the volume of a local uncertainty ellipsoid. A-optimal design minimizes the trace of the inverse information, while E-optimal design maximizes the smallest information eigenvalue. A criterion for one specified contrast \\(c^{\\mathsf T}\\beta\\) instead focuses on \\(c^{\\mathsf T}I^{-1}c\\). These objectives can favor different designs. In nonlinear models, information usually depends on unknown parameter values, so local design calculations may use nominal values, while Bayesian or robust formulations average over or protect against parameter uncertainty.\n\nMeasurement applications often have an especially transparent derivative structure. Suppose\n\n\\[Y_i=s_i(\\theta)+\\varepsilon_i, \\qquad \\varepsilon_i\\sim N(0,\\sigma^2)\\]\n\nindependently, with known noise variance. Then\n\n\\[I(\\theta)=\\frac{1}{\\sigma^2} \\sum_i\\left(\\frac{\\partial s_i(\\theta)}{\\partial\\theta}\\right)^2.\\]\n\nMeasurements are informative where the predicted signal changes strongly with the target parameter relative to noise. For a time delay \\(\\tau\\) in a signal \\(s(t-\\tau)\\), the derivative is proportional to the signal’s time derivative. This connects delay precision with signal bandwidth and explains the use of information bounds in radar, localization, and communications.\n\nUnknown amplitudes, offsets, or noise characteristics can reduce usable information through nuisance-parameter coupling. In photon-counting imaging, Poisson models yield analogous formulas involving derivatives of expected counts. Such bounds characterize idealized statistical precision, not every source of measurement uncertainty. Calibration errors, an inaccurate signal model, unresolved ambiguities, and estimator failure can produce errors not represented by the calculated bound."
    },
    {
      "heading": "Generalizations and related information bounds",
      "text": "The classical result is local and derivative-based. The Hammersley–Chapman–Robbins bound instead compares distributions at separated parameter values. If \\(P_{\\theta'}\\) is absolutely continuous with respect to \\(P_\\theta\\), an unbiased estimator of \\(g(\\theta)\\) satisfies, under the relevant integrability conditions,\n\n\\[\\operatorname{Var}_\\theta(T) \\geq \\frac{[g(\\theta')-g(\\theta)]^2} {E_\\theta\\!\\left[ \\left(\\frac{p_{\\theta'}(X)}{p_\\theta(X)}-1\\right)^2 \\right]}.\\]\n\nTaking the supremum over admissible \\(\\theta'\\) can improve the bound. In regular models, the limit as \\(\\theta'\\to\\theta\\) recovers the Cramér–Rao expression. Finite-difference bounds can remain useful when differentiability is unavailable, though support and absolute-continuity conditions still matter.\n\nThe Barankin bound uses collections of parameter alternatives and their likelihood ratios to obtain sharper constraints on unbiased estimation. Another complementary result, the Rao–Blackwell theorem, improves an unbiased estimator by taking its conditional expectation given a sufficient statistic. This operation cannot increase variance, but it need not produce an estimator attaining the information bound. Sufficient-statistic methods and information inequalities address related but distinct questions: how to improve an estimator, and how small its variance can possibly be.\n\nFor Bayesian inference, the van Trees inequality bounds risk averaged over a prior distribution. In a scalar form, with prior density \\(\\pi\\), suitable smoothness, and boundary terms that vanish,\n\n\\[E[(T(X)-\\theta)^2] \\geq \\frac{1}{E_\\pi[I(\\theta)]+I_\\pi}, \\qquad I_\\pi= \\int\\left(\\frac{d}{d\\theta}\\log\\pi(\\theta)\\right)^2 \\pi(\\theta)\\,d\\theta.\\]\n\nUnlike the classical unbiased bound, it applies to integrated squared error and does not require the estimator to be unbiased. It is not a pointwise inequality at each fixed parameter value. Boundary conditions are part of the theorem and cannot be dropped when applying the displayed formula.\n\nIn information geometry, Fisher information defines a metric tensor on a regular statistical model. A small displacement has squared statistical length\n\n\\[ds^2=d\\theta^{\\mathsf T}I(\\theta)\\,d\\theta.\\]\n\nThe transformation laws for this metric explain why information bounds behave consistently under smooth changes of parameter coordinates. In semiparametric statistics, nuisance parameters may be infinite-dimensional. Projecting the target score away from the closure of nuisance-score directions yields an efficient score and an associated lower bound, extending the finite-dimensional nuisance-parameter construction.\n\nIn quantum mechanics, the distribution of observed outcomes depends on both the quantum state and the measurement performed. Quantum Fisher information provides a measurement-independent limit on the classical information obtainable about a state parameter. For a single parameter, the quantum Cramér–Rao bound constrains locally unbiased estimation; attainability depends on the measurements and repetition scheme allowed. In multiparameter problems, measurements optimal for different parameters can be incompatible, so an inverse quantum information matrix need not be jointly attainable. More refined quantum bounds address this incompatibility explicitly."
    }
  ],
  "links": [
    "statistics",
    "variance",
    "estimator",
    "fisher-information",
    "maximum-likelihood-estimation",
    "experimental-design",
    "ronald-fisher",
    "probability-distribution",
    "sampling-distribution",
    "decision-theory",
    "loss-function",
    "probability-density-function",
    "probability-mass-function",
    "likelihood-function",
    "derivative",
    "integral",
    "dominated-convergence-theorem",
    "statistical-independence",
    "expected-value",
    "cauchy-schwarz-inequality",
    "covariance",
    "hilbert-space",
    "almost-surely",
    "normal-distribution",
    "sample-mean",
    "bernoulli-distribution",
    "poisson-distribution",
    "exponential-distribution",
    "gradient",
    "covariance-matrix",
    "jacobian-matrix",
    "statistical-identification",
    "estimator-bias",
    "mean-squared-error",
    "bias-variance-tradeoff",
    "regularization",
    "ridge-regression",
    "multivariate-normal-distribution",
    "hessian-matrix",
    "central-limit-theorem",
    "law-of-large-numbers",
    "standard-error",
    "confidence-interval",
    "uniform-distribution",
    "kullback-leibler-divergence",
    "linear-regression",
    "ordinary-least-squares",
    "measurement-uncertainty",
    "rao-blackwell-theorem",
    "conditional-expectation",
    "sufficient-statistic",
    "bayesian-inference",
    "prior-distribution",
    "metric-tensor",
    "quantum-mechanics",
    "quantum-fisher-information"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}