aiwiki.page
English
Mathematics / Probability & Statistics / cramer-rao-inequality

Cramér–Rao Inequality

The Cramér–Rao inequality bounds an estimator’s variance in terms of Fisher information, establishing a fundamental limit on statistical precision under regularity conditions.

63 keywords1 linked from7 not yet writtenWritten by AI
StatisticsVarianceEstimatorFisher Informati…Maximum likeliho…Experimental Des…Ronald FisherProbability Dist…Cramér–Rao…

The Cramér–Rao inequality is a theorem in statistics that places a lower bound on the variance of an estimator in terms of the information contained in the observations. In its best-known form, an unbiased estimator TT of a scalar parameter θ\theta satisfies

Var⁡θ(T)≥1I(θ),\operatorname{Var}_{\theta}(T)\geq \frac{1}{I(\theta)},

where I(θ)I(\theta) is the Fisher information in the entire observed sample. The inequality requires regularity conditions connecting differentiation of the statistical model with integration over possible observations; it is not a universal bound for every estimation problem.

The result provides a benchmark for statistical precision independently of the particular algorithm used to construct an estimator. It helps characterize efficient estimators, explains the large-sample behavior of maximum likelihood estimation, and guides experimental design. Its extensions address vector parameters, biased estimators, nuisance parameters, Bayesian risk, and quantum measurements. A central distinction throughout these applications is between a mathematical lower bound and an attainable level of performance: the bound may hold exactly without any estimator attaining it.

Historical development and statistical setting

The inequality emerged from the development of mathematical estimation theory in the first half of the twentieth century. Ronald Fisher introduced and developed the concept of information in connection with likelihood, sufficiency, and the precision of statistical estimates. Information expresses how strongly a probability distribution changes when its parameter changes. If two nearby parameter values produce markedly different distributions, observations can generally distinguish them more precisely than if the distributions change only slightly.

Maurice Fréchet obtained an early form of the information inequality in 1943. C. R. Rao published his formulation in the 1945 paper “Information and the Accuracy Attainable in the Estimation of Statistical Parameters,” and Harald Cramér developed the result in his 1946 book Mathematical Methods of Statistics. These contributions account for the alternative names Cramér–Rao, Rao–Cramér, and occasionally Fréchet–Cramér–Rao inequality. Rao’s work also connected statistical information with geometry, a connection that later became central to information geometry.

The underlying statistical setting is a family of distributions

P={Pθ:θ∈Θ}.\mathcal P=\{P_\theta:\theta\in\Theta\}.

The observations XX have distribution PθP_\theta, but θ\theta is unknown. An estimator is a measurable function T(X)T(X) of the observations, constructed without knowledge of the true parameter. Repeated samples at a fixed θ\theta produce a sampling distribution for TT, whose spread measures one aspect of estimation uncertainty. The Cramér–Rao inequality relates this spread to a property of the model rather than to a specific computational method.

Historically, the inequality helped distinguish several meanings of statistical optimality. An estimator may have minimum variance among unbiased estimators, attain the information bound, or be efficient only as sample size tends to infinity. These properties are not equivalent. A minimum-variance unbiased estimator need not attain the Cramér–Rao bound, and an estimator with small finite-sample bias may have lower mean squared error than any unbiased estimator. The theorem therefore belongs to a broader framework of decision theory, in which performance depends on the estimator class and the chosen loss function.

Scalar formulation and regularity conditions

Suppose the model has a density or mass function pθ(x)p_\theta(x) with respect to a fixed dominating measure. The density may be a probability density function for continuous observations or a probability mass function for discrete observations. The likelihood function treats pθ(x)p_\theta(x) as a function of θ\theta for the observed value xx. Its logarithmic derivative,

Uθ(X)=∂∂θlog⁡pθ(X),U_\theta(X)=\frac{\partial}{\partial\theta}\log p_\theta(X),

is called the score function. Under suitable regularity conditions,

Eθ[Uθ]=0,I(θ)=Eθ[Uθ2].E_\theta[U_\theta]=0, \qquad I(\theta)=E_\theta[U_\theta^2].

Thus information is the variance of the score.

Let TT estimate a differentiable scalar function g(θ)g(\theta) without bias:

Eθ[T]=g(θ).E_\theta[T]=g(\theta).

If TT has finite variance and differentiation may be interchanged with integration in the relevant expectations, then

Var⁡θ(T)≥[g′(θ)]2I(θ),0<I(θ)<∞.\operatorname{Var}_\theta(T) \geq \frac{[g'(\theta)]^2}{I(\theta)}, \qquad 0<I(\theta)<\infty.

The familiar reciprocal-information bound is the special case g(θ)=θg(\theta)=\theta. The derivative in the numerator matters: estimating θ2\theta^2, for example, is not the same precision problem as estimating θ\theta.

A common sufficient framework assumes that Θ\Theta is an open interval, the support of the distribution does not depend on θ\theta, and the density is differentiable with suitable integrable bounds on its derivatives. These assumptions allow a derivative to pass through an integral, often through the dominated convergence theorem. Fixed support alone is not sufficient, and parameter-dependent support is not by itself a complete diagnosis; the decisive issue is whether the identities used in the proof are valid. Under stronger second-order assumptions,

I(θ)=−Eθ ⁣[∂2∂θ2log⁡pθ(X)].I(\theta) = -E_\theta\!\left[ \frac{\partial^2}{\partial\theta^2}\log p_\theta(X) \right].

This expected curvature should be distinguished from the random observed curvature evaluated at a particular sample.

For nn observations that satisfy statistical independence and have a common distribution, the sample score is the sum of the individual scores. Their zero means make the cross terms vanish, giving

In(θ)=nI1(θ).I_n(\theta)=nI_1(\theta).

Consequently,

Var⁡θ(T)≥1nI1(θ)\operatorname{Var}_\theta(T)\geq\frac{1}{nI_1(\theta)}

for unbiased estimation of θ\theta. Independent but nonidentically distributed observations contribute information additively as well. Dependent observations require the information of their full joint distribution; multiplying a marginal information by sample size is generally unjustified.

Proof and geometric interpretation

The proof rests on differentiating an expected value and applying the Cauchy–Schwarz inequality. Starting with Eθ[T]=g(θ)E_\theta[T]=g(\theta), the regularity assumptions give

g′(θ)=∂∂θ∫T(x)pθ(x) dx=∫T(x)∂pθ(x)∂θ dx=Eθ[TUθ].\begin{aligned} g'(\theta) &=\frac{\partial}{\partial\theta} \int T(x)p_\theta(x)\,dx\\ &=\int T(x)\frac{\partial p_\theta(x)}{\partial\theta}\,dx\\ &=E_\theta[TU_\theta]. \end{aligned}

Because Eθ[Uθ]=0E_\theta[U_\theta]=0, the final expression equals

Eθ[(T−g(θ))Uθ]=Cov⁡θ(T,Uθ).E_\theta[(T-g(\theta))U_\theta] =\operatorname{Cov}_\theta(T,U_\theta).

Here the integration notation also represents summation in a discrete model.

Applying Cauchy–Schwarz to the centered estimator and the score yields

[g′(θ)]2≤Eθ[(T−g(θ))2] Eθ[Uθ2]=Var⁡θ(T) I(θ).[g'(\theta)]^2 \leq E_\theta[(T-g(\theta))^2]\, E_\theta[U_\theta^2] = \operatorname{Var}_\theta(T)\,I(\theta).

Division by positive information establishes the result. The proof identifies the precise statistical role of regularity: it ensures that a change in the estimator’s expectation is represented by its covariance with the score. The final inequality is then a general fact about square-integrable random variables.

There is also a direct interpretation in a Hilbert space. At a fixed parameter value, centered square-integrable functions of the data form an inner-product space with

⟨A,B⟩=Eθ[AB].\langle A,B\rangle=E_\theta[AB].

The component of T−g(θ)T-g(\theta) along the score is

g′(θ)I(θ)Uθ.\frac{g'(\theta)}{I(\theta)}U_\theta.

Writing the estimator’s centered error as this component plus an orthogonal residual RθR_\theta gives

T−g(θ)=g′(θ)I(θ)Uθ+Rθ,Eθ[RθUθ]=0.T-g(\theta) = \frac{g'(\theta)}{I(\theta)}U_\theta+R_\theta, \qquad E_\theta[R_\theta U_\theta]=0.

By the identity for the squared length of orthogonal components,

Var⁡θ(T)=[g′(θ)]2I(θ)+Eθ[Rθ2].\operatorname{Var}_\theta(T) = \frac{[g'(\theta)]^2}{I(\theta)} + E_\theta[R_\theta^2].

This decomposition makes the inequality more informative than a numerical bound alone. The excess variance is exactly the squared size of a component orthogonal to the score. The score describes the first-order change in the distribution as the parameter moves; the estimator must have a specified response to that change. Any remaining fluctuation increases variance without contributing to this required local response. Attainment means that no such residual remains.

The argument is local in the parameter: expectations and inner products are evaluated under one PθP_\theta. Nevertheless, an estimator claimed to be unbiased throughout a model must satisfy its expectation identity across the parameter space. Confusing pointwise statements with requirements on a single estimator valid for all parameter values can lead to incorrect claims of attainability.

Equality, efficient estimators, and exact examples

Equality holds precisely when the centered estimator is proportional to the score:

T−g(θ)=g′(θ)I(θ)UθPθ-almost surely.T-g(\theta) = \frac{g'(\theta)}{I(\theta)}U_\theta \quad P_\theta\text{-almost surely}.

The qualification almost surely allows differences on events of probability zero. An unbiased estimator satisfying this condition is called efficient at that parameter value. Efficiency throughout the model requires one statistic, independent of the unknown parameter, to satisfy the equality condition for every parameter value. This is a restrictive requirement.

For independent observations from a normal distribution with mean μ\mu and known variance σ2\sigma^2,

Uμ=1σ2∑i=1n(Xi−μ),In(μ)=nσ2.U_\mu=\frac{1}{\sigma^2}\sum_{i=1}^n(X_i-\mu), \qquad I_n(\mu)=\frac{n}{\sigma^2}.

The sample mean satisfies

X‾−μ=σ2nUμ,Var⁡(X‾)=σ2n,\overline X-\mu=\frac{\sigma^2}{n}U_\mu, \qquad \operatorname{Var}(\overline X)=\frac{\sigma^2}{n},

and therefore attains the bound exactly. Two other standard examples have the same property. For Bernoulli observations with success probability p∈(0,1)p\in(0,1),

In(p)=np(1−p),Var⁡(X‾)=p(1−p)n.I_n(p)=\frac{n}{p(1-p)}, \qquad \operatorname{Var}(\overline X)=\frac{p(1-p)}{n}.

For Poisson observations with mean λ>0\lambda>0,

In(λ)=nλ,Var⁡(X‾)=λn.I_n(\lambda)=\frac{n}{\lambda}, \qquad \operatorname{Var}(\overline X)=\frac{\lambda}{n}.

Both sample means are unbiased and efficient.

Exact attainment is not automatic even in an elementary regular model. Suppose X1,…,XnX_1,\ldots,X_n follow an exponential distribution with rate λ\lambda, and let S=∑iXiS=\sum_iX_i. The score and information are

Uλ=nλ−S,In(λ)=nλ2.U_\lambda=\frac{n}{\lambda}-S, \qquad I_n(\lambda)=\frac{n}{\lambda^2}.

For n>2n>2, the estimator

T=n−1ST=\frac{n-1}{S}

is unbiased for λ\lambda, with

Var⁡λ(T)=λ2n−2.\operatorname{Var}_\lambda(T)=\frac{\lambda^2}{n-2}.

This exceeds the Cramér–Rao bound λ2/n\lambda^2/n. The estimator depends reciprocally on SS, whereas equality would require an affine relationship with the score.

Equality has a close connection with the exponential family of distributions. In a scalar problem, if an estimator of θ\theta attains the bound throughout an interval and the necessary smoothness conditions hold, the equality condition implies

∂∂θlog⁡pθ(x)=I(θ)(T(x)−θ).\frac{\partial}{\partial\theta}\log p_\theta(x) = I(\theta)\bigl(T(x)-\theta\bigr).

Integrating with respect to θ\theta produces a representation of the form

pθ(x)=h(x)exp⁡{A(θ)T(x)−B(θ)}.p_\theta(x)=h(x)\exp\{A(\theta)T(x)-B(\theta)\}.

Thus exact efficiency often reflects a special algebraic structure of the model. Membership in an exponential family does not, however, guarantee an unbiased efficient estimator for every function of its parameter.

Vector parameters and nuisance parameters

For a parameter vector θ∈Rp\theta\in\mathbb R^p, the score is the gradient of the log likelihood:

Uθ=∇θlog⁡pθ(X).U_\theta=\nabla_\theta\log p_\theta(X).

Its covariance matrix,

I(θ)=Eθ[UθUθT],I(\theta)=E_\theta[U_\theta U_\theta^{\mathsf T}],

is the Fisher information matrix. Under regularity it is symmetric and positive semidefinite. Suppose T∈RqT\in\mathbb R^q is unbiased for a differentiable vector-valued function g(θ)g(\theta), and let G(θ)G(\theta) be its q×pq\times p Jacobian matrix. If I(θ)I(\theta) is nonsingular, the matrix Cramér–Rao inequality is

Cov⁡θ(T)⪰G(θ)I(θ)−1G(θ)T.\operatorname{Cov}_\theta(T) \succeq G(\theta)I(\theta)^{-1}G(\theta)^{\mathsf T}.

The symbol ⪰\succeq means that the difference is positive semidefinite.

This is stronger than separate lower bounds on the component variances. For every fixed vector aa,

Var⁡θ(aTT)≥aTGI−1GTa.\operatorname{Var}_\theta(a^{\mathsf T}T) \geq a^{\mathsf T}GI^{-1}G^{\mathsf T}a.

The inequality therefore constrains every linear combination of the estimated quantities, including combinations affected by covariance. When g(θ)=θg(\theta)=\theta, the Jacobian is the identity and the bound reduces to I−1I^{-1}. The matrix proof follows the scalar proof by projecting the centered estimator onto the span of all score components.

Nuisance parameters are parameters needed to describe the observations but not themselves the estimation target. Write θ=(ψ,λ)\theta=(\psi,\lambda), where ψ\psi is the target and λ\lambda is nuisance, and partition the information:

I=(IψψIψλIλψIλλ).I= \begin{pmatrix} I_{\psi\psi}&I_{\psi\lambda}\\ I_{\lambda\psi}&I_{\lambda\lambda} \end{pmatrix}.

For a scalar target, when the relevant inverses exist,

Var⁡(ψ^)≥(Iψψ−IψλIλλ−1Iλψ)−1.\operatorname{Var}(\widehat\psi) \geq \left( I_{\psi\psi} -I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi} \right)^{-1}.

The denominator is the information remaining after projecting the target score away from the nuisance-score directions. It is no greater than IψψI_{\psi\psi}, the information that would apply if the nuisance parameter were known.

For a normal distribution parameterized by its mean μ\mu and variance vv, one observation has information

I(μ,v)=(1/v001/(2v2)).I(\mu,v)= \begin{pmatrix} 1/v&0\\ 0&1/(2v^2) \end{pmatrix}.

The zero off-diagonal terms mean that the mean and variance scores are orthogonal. With nn observations, the bound for unbiased estimation of vv is 2v2/n2v^2/n. Yet the usual unbiased sample variance has variance 2v2/(n−1)2v^2/(n-1). This is another case in which the information bound is valid but unattained.

Singular information requires separate treatment. It may indicate redundant coordinates or a failure of local statistical identification, although a singular first-order information matrix does not always mean global nonidentifiability. Generalized-inverse bounds require compatibility conditions between the target derivative and the range of the information matrix; replacing an inverse mechanically is not sufficient.

Biased estimation and mean squared error

Unbiasedness is a condition on an estimator’s expectation, not a universal criterion of quality. Define the bias of an estimator TT of θ\theta by

b(θ)=Eθ[T]−θ.b(\theta)=E_\theta[T]-\theta.

The same score argument applies to the mean function m(θ)=θ+b(θ)m(\theta)=\theta+b(\theta), giving

Var⁡θ(T)≥[1+b′(θ)]2I(θ).\operatorname{Var}_\theta(T) \geq \frac{[1+b'(\theta)]^2}{I(\theta)}.

Consequently, its mean squared error satisfies

Eθ[(T−θ)2]≥b(θ)2+[1+b′(θ)]2I(θ).E_\theta[(T-\theta)^2] \geq b(\theta)^2+ \frac{[1+b'(\theta)]^2}{I(\theta)}.

For estimation of g(θ)g(\theta), the numerator becomes [g′(θ)+b′(θ)]2[g'(\theta)+b'(\theta)]^2.

The derivative of bias is essential. An estimator can have small bias at one parameter value while its mean responds quite differently to nearby changes in the parameter. The Cramér–Rao argument measures this local response through the derivative of the mean. Knowing the bias at one isolated point therefore does not determine the appropriate variance bound there. Local unbiasedness for θ\theta ordinarily requires both the correct mean and unit derivative of the mean at the point in question.

A simple normal example illustrates the bias–variance tradeoff. If X‾\overline X estimates a normal mean with known variance σ2\sigma^2, consider

Ta=aX‾T_a=a\overline X

for a fixed constant aa. Its bias and variance are

b(μ)=(a−1)μ,Var⁡μ(Ta)=a2σ2n.b(\mu)=(a-1)\mu, \qquad \operatorname{Var}_\mu(T_a)=\frac{a^2\sigma^2}{n}.

Since 1+b′(μ)=a1+b'(\mu)=a, this estimator attains the biased variance bound. Its mean squared error is

MSE⁡μ(Ta)=a2σ2n+(a−1)2μ2.\operatorname{MSE}_\mu(T_a) = \frac{a^2\sigma^2}{n}+(a-1)^2\mu^2.

For 0<a<10<a<1, it improves on the sample mean near μ=0\mu=0, while the bias term makes it worse sufficiently far away.

A constant estimator is an extreme but useful illustration. If T=cT=c, its variance is zero, its bias is c−θc-\theta, and 1+b′(θ)=01+b'(\theta)=0. The biased inequality is consistent with its zero variance, while its squared error is (c−θ)2(c-\theta)^2. No positive pointwise lower bound can apply to the mean squared error of every possible estimator at every parameter value without additional restrictions.

These distinctions matter for regularization methods, including ridge regression and other shrinkage procedures. Such estimators may deliberately accept bias to reduce error. The unbiased information bound is not a universal lower bound for their risk. Their analysis requires a bias-sensitive inequality or another criterion, such as integrated risk, minimax risk, or performance over a specified parameter neighborhood.

Large-sample efficiency and likelihood methods

In many regular models, exact finite-sample attainment is unavailable, but maximum likelihood estimators approach the information bound as sample size increases. Under conditions ensuring identification, consistency, and appropriate smoothness,

n(θ^n−θ0)→dN ⁣(0,I1(θ0)−1).\sqrt n(\widehat\theta_n-\theta_0) \xrightarrow{d} N\!\left(0,I_1(\theta_0)^{-1}\right).

The notation denotes convergence in distribution, and I1I_1 is the information in one observation. For a vector parameter, the limiting law is a multivariate normal distribution. This statement concerns asymptotic distributions; convergence of the actual variances requires additional control of moments or tails.

The mechanism comes from expanding the likelihood score near the true parameter. If an interior maximum satisfies Un(θ^n)=0U_n(\widehat\theta_n)=0, a first-order expansion gives, schematically,

0≈Un(θ0)+Hn(θ0)(θ^n−θ0),0\approx U_n(\theta_0) + H_n(\theta_0)(\widehat\theta_n-\theta_0),

where HnH_n is the Hessian matrix of the log likelihood. Typically,

n−1/2Un(θ0)→dN(0,I1(θ0)),−1nHn(θ0)→PI1(θ0).n^{-1/2}U_n(\theta_0) \xrightarrow{d}N(0,I_1(\theta_0)), \qquad -\frac{1}{n}H_n(\theta_0) \xrightarrow{P}I_1(\theta_0).

The first limit follows from a central limit theorem, and the second commonly from a law of large numbers. Together they yield

n(θ^n−θ0)≈I1(θ0)−1Un(θ0)n.\sqrt n(\widehat\theta_n-\theta_0) \approx I_1(\theta_0)^{-1}\frac{U_n(\theta_0)}{\sqrt n}.

This explains the frequent appearance of inverse information in a reported standard error. Expected information or observed likelihood curvature can provide a large-sample covariance approximation, leading to approximate confidence intervals. These approximations can be inaccurate in small samples, near parameter boundaries, or when the likelihood is far from quadratic. An inverse Hessian is therefore not, by itself, a finite-sample precision guarantee.

The local asymptotic normality framework gives a more general interpretation. It studies alternatives of the form

θ=θ0+hn,\theta=\theta_0+\frac{h}{\sqrt n},

under which a regular model approaches a Gaussian shift experiment. Appropriate notions of estimator regularity then produce asymptotic information bounds that do not require exact finite-sample unbiasedness. This is the setting in which inverse information becomes a broad limit on regular large-sample estimation.

Regularity of the estimator remains important. Hodges-type superefficient estimators can have smaller limiting error than the usual information benchmark at selected parameter values, while performing poorly along nearby parameter sequences. This does not contradict either the finite-sample unbiased inequality or the asymptotic theory. Pointwise asymptotic behavior is weaker than uniform behavior over shrinking neighborhoods.

Nonregular models and limits of applicability

A standard failure occurs when the support depends on the parameter. Let X1,…,XnX_1,\ldots,X_n be independent observations from the uniform distribution on (0,θ)(0,\theta), where θ>0\theta>0. Away from the moving boundary, differentiating the log density appears to give

Uθ=−nθ.U_\theta=-\frac{n}{\theta}.

This cannot be a regular score because its expectation is not zero. The missing contribution comes from the changing support, and ordinary differentiation under the integral does not capture the full change in probability.

The sample maximum M=max⁡iXiM=\max_iX_i has expectation nθ/(n+1)n\theta/(n+1), so

T=n+1nMT=\frac{n+1}{n}M

is unbiased for θ\theta. Its variance is

Var⁡θ(T)=θ2n(n+2).\operatorname{Var}_\theta(T) = \frac{\theta^2}{n(n+2)}.

This decreases at order n−2n^{-2}, rather than the usual regular-model order n−1n^{-1}. It does not violate the Cramér–Rao theorem because the required score identities fail. The apparent contradiction results from applying a regular-model formula outside its assumptions.

Boundary parameters create a different difficulty. For a Bernoulli probability pp, the regular information formula applies on 0<p<10<p<1; at p=0p=0 or p=1p=1, the distribution is degenerate and two-sided parameter perturbations are unavailable. Variance components constrained to be nonnegative can likewise have nonnormal limiting distributions when the true value is zero. A model may be smooth in its interior yet require different asymptotic analysis at its boundary.

Other complications include infinite information, nondifferentiable models, and information matrices that become singular at particular parameter values. Mixture models can lose regularity when components coincide or a mixing weight vanishes. Increasing parameter dimension or allowing an expanding collection of nuisance parameters also requires assumptions beyond the fixed-dimensional theory. A symbolic likelihood derivative does not establish that a particular information bound is meaningful.

Model misspecification raises a separate issue. If the actual data distribution lies outside the assumed family, a likelihood estimator may converge to the parameter minimizing Kullback–Leibler divergence from the true distribution to the model. Under suitable conditions, its asymptotic covariance has the sandwich form

A−1BA−1,A^{-1}BA^{-1},

where AA is the negative expected log-likelihood Hessian and BB is the score covariance at that limiting parameter. Under correct specification, A=B=IA=B=I; otherwise, the identity need not hold. Ordinary inverse-information calculations can then misrepresent uncertainty even when the numerical likelihood maximization succeeds.

Experimental design and measurement applications

Information depends on how data are collected as well as on the parameter. If an experiment is controlled by a design variable dd, its information can be written I(θ;d)I(\theta;d). Independent measurements contribute additive information, enabling a comparison of measurement times, sensor positions, stimulus settings, or sample allocations. For scalar estimation, maximizing information minimizes the Cramér–Rao variance benchmark. This comparison is meaningful only relative to the assumed observation model and any nuisance parameters.

For example, in the normal linear regression model

Y=Xβ+ε,ε∼N(0,σ2In),Y=X\beta+\varepsilon, \qquad \varepsilon\sim N(0,\sigma^2I_n),

with known σ2\sigma^2 and a fixed full-column-rank design matrix,

I(β)=1σ2XTX.I(\beta)=\frac{1}{\sigma^2}X^{\mathsf T}X.

The ordinary least squares estimator has covariance

σ2(XTX)−1\sigma^2(X^{\mathsf T}X)^{-1}

and attains the bound. Nearly dependent design columns make the information poorly conditioned, producing high variance in some parameter combinations. This is a statistical consequence of the same geometry that can cause numerical difficulties in solving the regression equations.

There is no single best design criterion for every vector estimation problem. D-optimal design maximizes the determinant of the information matrix, corresponding to reducing the volume of a local uncertainty ellipsoid. A-optimal design minimizes the trace of the inverse information, while E-optimal design maximizes the smallest information eigenvalue. A criterion for one specified contrast cTβc^{\mathsf T}\beta instead focuses on cTI−1cc^{\mathsf T}I^{-1}c. These objectives can favor different designs. In nonlinear models, information usually depends on unknown parameter values, so local design calculations may use nominal values, while Bayesian or robust formulations average over or protect against parameter uncertainty.

Measurement applications often have an especially transparent derivative structure. Suppose

Yi=si(θ)+εi,εi∼N(0,σ2)Y_i=s_i(\theta)+\varepsilon_i, \qquad \varepsilon_i\sim N(0,\sigma^2)

independently, with known noise variance. Then

I(θ)=1σ2∑i(∂si(θ)∂θ)2.I(\theta)=\frac{1}{\sigma^2} \sum_i\left(\frac{\partial s_i(\theta)}{\partial\theta}\right)^2.

Measurements are informative where the predicted signal changes strongly with the target parameter relative to noise. For a time delay τ\tau in a signal s(t−τ)s(t-\tau), the derivative is proportional to the signal’s time derivative. This connects delay precision with signal bandwidth and explains the use of information bounds in radar, localization, and communications.

Unknown amplitudes, offsets, or noise characteristics can reduce usable information through nuisance-parameter coupling. In photon-counting imaging, Poisson models yield analogous formulas involving derivatives of expected counts. Such bounds characterize idealized statistical precision, not every source of measurement uncertainty. Calibration errors, an inaccurate signal model, unresolved ambiguities, and estimator failure can produce errors not represented by the calculated bound.

Generalizations and related information bounds

The classical result is local and derivative-based. The Hammersley–Chapman–Robbins bound instead compares distributions at separated parameter values. If Pθ′P_{\theta'} is absolutely continuous with respect to PθP_\theta, an unbiased estimator of g(θ)g(\theta) satisfies, under the relevant integrability conditions,

Var⁡θ(T)≥[g(θ′)−g(θ)]2Eθ ⁣[(pθ′(X)pθ(X)−1)2].\operatorname{Var}_\theta(T) \geq \frac{[g(\theta')-g(\theta)]^2} {E_\theta\!\left[ \left(\frac{p_{\theta'}(X)}{p_\theta(X)}-1\right)^2 \right]}.

Taking the supremum over admissible θ′\theta' can improve the bound. In regular models, the limit as θ′→θ\theta'\to\theta recovers the Cramér–Rao expression. Finite-difference bounds can remain useful when differentiability is unavailable, though support and absolute-continuity conditions still matter.

The Barankin bound uses collections of parameter alternatives and their likelihood ratios to obtain sharper constraints on unbiased estimation. Another complementary result, the Rao–Blackwell theorem, improves an unbiased estimator by taking its conditional expectation given a sufficient statistic. This operation cannot increase variance, but it need not produce an estimator attaining the information bound. Sufficient-statistic methods and information inequalities address related but distinct questions: how to improve an estimator, and how small its variance can possibly be.

For Bayesian inference, the van Trees inequality bounds risk averaged over a prior distribution. In a scalar form, with prior density π\pi, suitable smoothness, and boundary terms that vanish,

E[(T(X)−θ)2]≥1Eπ[I(θ)]+Iπ,Iπ=∫(ddθlog⁡π(θ))2π(θ) dθ.E[(T(X)-\theta)^2] \geq \frac{1}{E_\pi[I(\theta)]+I_\pi}, \qquad I_\pi= \int\left(\frac{d}{d\theta}\log\pi(\theta)\right)^2 \pi(\theta)\,d\theta.

Unlike the classical unbiased bound, it applies to integrated squared error and does not require the estimator to be unbiased. It is not a pointwise inequality at each fixed parameter value. Boundary conditions are part of the theorem and cannot be dropped when applying the displayed formula.

In information geometry, Fisher information defines a metric tensor on a regular statistical model. A small displacement has squared statistical length

ds2=dθTI(θ) dθ.ds^2=d\theta^{\mathsf T}I(\theta)\,d\theta.

The transformation laws for this metric explain why information bounds behave consistently under smooth changes of parameter coordinates. In semiparametric statistics, nuisance parameters may be infinite-dimensional. Projecting the target score away from the closure of nuisance-score directions yields an efficient score and an associated lower bound, extending the finite-dimensional nuisance-parameter construction.

In quantum mechanics, the distribution of observed outcomes depends on both the quantum state and the measurement performed. Quantum Fisher information provides a measurement-independent limit on the classical information obtainable about a state parameter. For a single parameter, the quantum Cramér–Rao bound constrains locally unbiased estimation; attainability depends on the measurements and repetition scheme allowed. In multiparameter problems, measurements optimal for different parameters can be incompatible, so an inverse quantum information matrix need not be jointly attainable. More refined quantum bounds address this incompatibility explicitly.