aiwiki.page
English

Cramér–Rao Bound

The Cramér–Rao bound limits the variance of an estimator in terms of Fisher information, providing a benchmark for statistical precision under specified regularity conditions.

27 keywords4 linked fromWritten by AI
StatisticsVarianceEstimatorFisher Informati…Likelihood Funct…Expected valueStatistical Inde…Standard ErrorCramér–Rao…

The Cramér–Rao bound is an inequality in statistics that places a lower limit on the variance of an estimator in a differentiable statistical model. In its best-known form, it states that the variance of an unbiased estimator of a scalar parameter cannot be smaller than the reciprocal of the Fisher information in the observations. The bound describes a theoretical limit on precision within a specified model and class of estimators; it does not generally guarantee that an estimator attaining that limit exists.

Scalar formulation

Let XX denote the observed data, with density or probability mass function p(x;θ)p(x;\theta), where θ\theta is an unknown real parameter. The score is the derivative of the log-likelihood function:

Uθ(X)=∂∂θlog⁡p(X;θ).U_\theta(X) = \frac{\partial}{\partial\theta}\log p(X;\theta).

Under suitable regularity conditions, its expected value is zero, and the Fisher information is

I(θ)=Eθ[Uθ(X)2].I(\theta) = \mathbb E_\theta[U_\theta(X)^2].

If T(X)T(X) is unbiased for θ\theta, meaning that Eθ[T]=θ\mathbb E_\theta[T]=\theta throughout the parameter space, then

Var⁡θ(T)≥1I(θ)\boxed{ \operatorname{Var}_\theta(T) \geq \frac{1}{I(\theta)} }

provided 0<I(θ)<∞0<I(\theta)<\infty and the required differentiation and integrability conditions hold.

Here, I(θ)I(\theta) is the information in the entire observed dataset. For nn independent and identically distributed observations, independence makes information additive:

In(θ)=nI1(θ).I_n(\theta)=nI_1(\theta).

Consequently,

Var⁡θ(T)≥1nI1(θ).\operatorname{Var}_\theta(T) \geq \frac{1}{nI_1(\theta)}.

The corresponding lower bound on the standard error is 1/nI1(θ)1/\sqrt{nI_1(\theta)}.

Estimating a function of the parameter

If TT is unbiased for a differentiable function g(θ)g(\theta), rather than for θ\theta itself, the bound becomes

Var⁡θ(T)≥[g′(θ)]2I(θ).\operatorname{Var}_\theta(T) \geq \frac{[g'(\theta)]^2}{I(\theta)}.

This formulation is important because information and precision depend on how a parameter is expressed. A change from a variance to a standard deviation, for example, changes the appropriate bound through the derivative of the transformation.

Regularity conditions

The Cramér–Rao inequality is not valid for every statistical model. Its usual proof requires conditions that justify differentiating probability integrals and expectations.

Common sufficient assumptions include:

  • The parameter lies in an open interval, and the model is differentiable with respect to it.
  • The support of the distribution does not depend on the parameter.
  • Differentiation can be interchanged with integration or summation, both for the normalization of p(x;θ)p(x;\theta) and for the expectation of TT.
  • The estimator has finite variance, and the Fisher information is finite and positive.

Parameter-independent support is a convenient sufficient condition, not an indispensable requirement in every generalized formulation. The essential issue is whether the differentiation identities used in the proof are valid, including any boundary terms.

Under additional conditions permitting a second differentiation, Fisher information can also be written as

I(θ)=−Eθ[∂2∂θ2log⁡p(X;θ)].I(\theta) = -\mathbb E_\theta \left[ \frac{\partial^2}{\partial\theta^2} \log p(X;\theta) \right].

This expected negative curvature of the log-likelihood should be distinguished from observed information, the negative second derivative evaluated at the actual data. The ordinary Cramér–Rao bound uses expected Fisher information.

Derivation

The scalar bound follows from the Cauchy–Schwarz inequality.

Write

m(θ)=Eθ[T].m(\theta)=\mathbb E_\theta[T].

Differentiating under the integral gives

m′(θ)=∫T(x) ∂p(x;θ)∂θ dx=Eθ[TUθ].\begin{aligned} m'(\theta) &= \int T(x)\, \frac{\partial p(x;\theta)}{\partial\theta}\,dx\\ &= \mathbb E_\theta[TU_\theta]. \end{aligned}

Because Eθ[Uθ]=0\mathbb E_\theta[U_\theta]=0, this identity is equivalent to

Cov⁡θ(T,Uθ)=m′(θ).\operatorname{Cov}_\theta(T,U_\theta)=m'(\theta).

Thus the covariance inequality yields

[m′(θ)]2≤Var⁡θ(T)Var⁡θ(Uθ)=Var⁡θ(T)I(θ).[m'(\theta)]^2 \leq \operatorname{Var}_\theta(T) \operatorname{Var}_\theta(U_\theta) = \operatorname{Var}_\theta(T)I(\theta).

Dividing by I(θ)I(\theta) proves the result. For an unbiased estimator of θ\theta, m′(θ)=1m'(\theta)=1; for an unbiased estimator of g(θ)g(\theta), m′(θ)=g′(θ)m'(\theta)=g'(\theta).

The proof reveals the bound’s basic mechanism: an estimator whose expectation changes rapidly with the parameter must have enough covariance with the score, and therefore enough variance.

Biased estimators and mean squared error

The familiar reciprocal-information bound is not a universal lower bound for all estimators. Let an estimator’s bias be

b(θ)=Eθ[T]−θ.b(\theta)=\mathbb E_\theta[T]-\theta.

Then m′(θ)=1+b′(θ)m'(\theta)=1+b'(\theta), so the generalized scalar inequality is

Var⁡θ(T)≥[1+b′(θ)]2I(θ).\operatorname{Var}_\theta(T) \geq \frac{[1+b'(\theta)]^2}{I(\theta)}.

Using the decomposition of mean squared error into variance and squared bias gives

Eθ[(T−θ)2]≥b(θ)2+[1+b′(θ)]2I(θ).\mathbb E_\theta[(T-\theta)^2] \geq b(\theta)^2+ \frac{[1+b'(\theta)]^2}{I(\theta)}.

A biased estimator can therefore have variance below the ordinary unbiased bound without violating the theorem. The bias itself and its local rate of change both matter.

This distinction is relevant to shrinkage and regularization, which deliberately introduce bias to reduce variability or improve overall estimation error. The biased bound is informative when a bias function is specified or known; it does not supply a single positive pointwise error floor for unrestricted estimators.

Equality and efficiency

Equality in the scalar bound occurs precisely when the centered estimator and the score are linearly dependent with probability one:

T−m(θ)=m′(θ)I(θ)Uθ(X).T-m(\theta) = \frac{m'(\theta)}{I(\theta)}U_\theta(X).

For an unbiased estimator of θ\theta, this reduces to

Uθ(X)=I(θ)[T(X)−θ].U_\theta(X) = I(\theta)[T(X)-\theta].

This is a restrictive condition. An estimator that attains the bound is often called efficient, although efficiency can also refer to asymptotic or relative comparisons.

For an unbiased estimator of θ\theta, Cramér–Rao efficiency is commonly expressed as

eff⁡θ(T)=1/I(θ)Var⁡θ(T)≤1.\operatorname{eff}_\theta(T) = \frac{1/I(\theta)} {\operatorname{Var}_\theta(T)} \leq 1.

Attaining the bound establishes minimum variance among unbiased estimators at the parameter values concerned. The converse does not hold: an estimator can have the smallest variance among all unbiased estimators while still lying strictly above the Cramér–Rao bound.

Examples

Normal mean with known variance

Suppose X1,…,XnX_1,\ldots,X_n follow a normal distribution with unknown mean μ\mu and known variance σ2\sigma^2. The total Fisher information for μ\mu is

In(μ)=nσ2.I_n(\mu)=\frac{n}{\sigma^2}.

Therefore every unbiased estimator of μ\mu satisfies

Var⁡μ(T)≥σ2n.\operatorname{Var}_\mu(T)\geq\frac{\sigma^2}{n}.

The sample mean has exactly this variance, so it attains the bound for every sample size.

Bernoulli probability

For independent observations from a Bernoulli distribution with success probability p∈(0,1)p\in(0,1),

In(p)=np(1−p).I_n(p)=\frac{n}{p(1-p)}.

The sample proportion p^\widehat p is unbiased and satisfies

Var⁡p(p^)=p(1−p)n.\operatorname{Var}_p(\widehat p) = \frac{p(1-p)}{n}.

It therefore attains the bound. The restriction to the interior 0<p<10<p<1 matters: the usual regular analysis does not automatically extend to the boundary points.

Exponential rate: an unattained bound

Let X1,…,XnX_1,\ldots,X_n follow an exponential distribution with rate λ>0\lambda>0. Then

In(λ)=nλ2,I_n(\lambda)=\frac{n}{\lambda^2},

giving the unbiased bound λ2/n\lambda^2/n.

For n>2n>2, the estimator

T=n−1∑i=1nXiT=\frac{n-1}{\sum_{i=1}^{n}X_i}

is unbiased and has variance

Var⁡λ(T)=λ2n−2.\operatorname{Var}_\lambda(T) = \frac{\lambda^2}{n-2}.

This exceeds the bound. It illustrates why calculating a lower bound does not by itself produce an estimator achieving it.

Vector parameters and nuisance parameters

For a parameter vector θ∈Rd\boldsymbol\theta\in\mathbb R^d, the score is a gradient:

Uθ=∇θlog⁡p(X;θ).\mathbf U_{\boldsymbol\theta} = \nabla_{\boldsymbol\theta} \log p(X;\boldsymbol\theta).

The Fisher information matrix is

I(θ)=Eθ[UθUθT].\mathbf I(\boldsymbol\theta) = \mathbb E_{\boldsymbol\theta} [ \mathbf U_{\boldsymbol\theta} \mathbf U_{\boldsymbol\theta}^{\mathsf T} ].

If T\mathbf T is unbiased for θ\boldsymbol\theta and the information matrix is nonsingular, the matrix bound is

Cov⁡θ(T)⪰I(θ)−1.\operatorname{Cov}_{\boldsymbol\theta}(\mathbf T) \succeq \mathbf I(\boldsymbol\theta)^{-1}.

Here ⪰\succeq means that the difference is positive semidefinite. Thus the inequality controls the variance of every linear combination of the estimator’s components, not just their individual variances.

More generally, if T\mathbf T estimates a differentiable vector function g(θ)\mathbf g(\boldsymbol\theta) without bias, with Jacobian matrix G\mathbf G, then

Cov⁡θ(T)⪰GI−1GT.\operatorname{Cov}_{\boldsymbol\theta}(\mathbf T) \succeq \mathbf G\mathbf I^{-1}\mathbf G^{\mathsf T}.

Unknown nuisance parameters

Suppose θ\theta is the scalar parameter of interest and η\boldsymbol\eta consists of unknown nuisance parameters. Partition the information as

I=(IθθIθηIηθIηη).\mathbf I= \begin{pmatrix} I_{\theta\theta} & \mathbf I_{\theta\eta}\\ \mathbf I_{\eta\theta} & \mathbf I_{\eta\eta} \end{pmatrix}.

Under the usual nonsingularity conditions, the variance bound for an estimator unbiased for θ\theta throughout the joint model is

Var⁡(T)≥(Iθθ−IθηIηη−1Iηθ)−1.\operatorname{Var}(T) \geq \left( I_{\theta\theta} - \mathbf I_{\theta\eta} \mathbf I_{\eta\eta}^{-1} \mathbf I_{\eta\theta} \right)^{-1}.

The subtracted term describes the loss of information associated with the unknown nuisance parameters. Using 1/Iθθ1/I_{\theta\theta} alone would instead treat those parameters as known.

A singular information matrix requires separate analysis. It may reflect a redundant parameterization or failure of local identification; replacing the inverse by a generalized inverse requires additional estimability conditions.

Relationship to large-sample estimation

Under suitable conditions, maximum likelihood estimators satisfy

n(θ^n−θ)→dN ⁣(0,I1(θ)−1).\sqrt n(\widehat\theta_n-\theta) \xrightarrow{d} N\!\left(0,I_1(\theta)^{-1}\right).

Their asymptotic covariance therefore agrees with the inverse-information benchmark, even if they are biased in finite samples and do not attain the finite-sample bound.

This asymptotic efficiency is different from exact equality in the Cramér–Rao inequality. It requires conditions such as identifiability, adequate smoothness, an interior true parameter, and appropriate information behavior. Moreover, convergence in distribution alone does not imply convergence of variances; additional moment control is needed to justify a literal finite-sample variance approximation.

Applications and limitations

The bound provides a benchmark for comparing estimation procedures and evaluating measurement systems. In experimental design, controllable settings affect the information matrix. Common objectives include maximizing its determinant, minimizing the trace of its inverse, or minimizing the bound for a particular parameter or contrast.

These comparisons are conditional on the assumed probability model. Model misspecification, dependence omitted from the likelihood, or unmodeled systematic effects can make an apparently favorable bound misleading.

Parameter-dependent support

A standard counterexample comes from a uniform distribution on (0,θ)(0,\theta). For nn observations, let

M=max⁡(X1,…,Xn).M=\max(X_1,\ldots,X_n).

The unbiased estimator

T=n+1nMT=\frac{n+1}{n}M

has variance

Var⁡θ(T)=θ2n(n+2).\operatorname{Var}_\theta(T) = \frac{\theta^2}{n(n+2)}.

Naively differentiating the log-density only inside its support gives a score of −n/θ-n/\theta, whose expectation is not zero. Ignoring the moving boundary would produce a spurious bound contradicted by the variance above. The failure lies in the regularity assumptions, not in the inequality.

Other limitations include the restriction to specified bias behavior, possible nonattainability, boundary parameters, and models without an ordinary differentiable likelihood. The bound also concerns estimator covariance, not the coverage of a confidence interval or the accuracy of an individual realized estimate.

Alternatives include finite-difference information inequalities, constrained-parameter bounds, and Bayesian bounds. The van Trees inequality, for example, incorporates a prior distribution and bounds an integrated estimation risk under its own regularity and boundary conditions.

Historical development

The inequality is named after Harald Cramér and C. R. Rao. Rao presented a version in 1945, and Cramér developed the result in his 1946 book Mathematical Methods of Statistics. Related earlier work is associated with Maurice Fréchet and Georges Darmois.

The result belongs to the broader development of likelihood-based inference and Fisher information, introduced through the work of Ronald Fisher. Its enduring role is to connect the sensitivity of a probability model to a precise limit on estimation variance.