The Cramér–Rao bound is an inequality in statistics that places a lower limit on the variance of an estimator in a differentiable statistical model. In its best-known form, it states that the variance of an unbiased estimator of a scalar parameter cannot be smaller than the reciprocal of the Fisher information in the observations. The bound describes a theoretical limit on precision within a specified model and class of estimators; it does not generally guarantee that an estimator attaining that limit exists.
Scalar formulation
Let denote the observed data, with density or probability mass function , where is an unknown real parameter. The score is the derivative of the log-likelihood function:
Under suitable regularity conditions, its expected value is zero, and the Fisher information is
If is unbiased for , meaning that throughout the parameter space, then
provided and the required differentiation and integrability conditions hold.
Here, is the information in the entire observed dataset. For independent and identically distributed observations, independence makes information additive:
Consequently,
The corresponding lower bound on the standard error is .
Estimating a function of the parameter
If is unbiased for a differentiable function , rather than for itself, the bound becomes
This formulation is important because information and precision depend on how a parameter is expressed. A change from a variance to a standard deviation, for example, changes the appropriate bound through the derivative of the transformation.
Regularity conditions
The Cramér–Rao inequality is not valid for every statistical model. Its usual proof requires conditions that justify differentiating probability integrals and expectations.
Common sufficient assumptions include:
- The parameter lies in an open interval, and the model is differentiable with respect to it.
- The support of the distribution does not depend on the parameter.
- Differentiation can be interchanged with integration or summation, both for the normalization of and for the expectation of .
- The estimator has finite variance, and the Fisher information is finite and positive.
Parameter-independent support is a convenient sufficient condition, not an indispensable requirement in every generalized formulation. The essential issue is whether the differentiation identities used in the proof are valid, including any boundary terms.
Under additional conditions permitting a second differentiation, Fisher information can also be written as
This expected negative curvature of the log-likelihood should be distinguished from observed information, the negative second derivative evaluated at the actual data. The ordinary Cramér–Rao bound uses expected Fisher information.
Derivation
The scalar bound follows from the Cauchy–Schwarz inequality.
Write
Differentiating under the integral gives
Because , this identity is equivalent to
Thus the covariance inequality yields
Dividing by proves the result. For an unbiased estimator of , ; for an unbiased estimator of , .
The proof reveals the bound’s basic mechanism: an estimator whose expectation changes rapidly with the parameter must have enough covariance with the score, and therefore enough variance.
Biased estimators and mean squared error
The familiar reciprocal-information bound is not a universal lower bound for all estimators. Let an estimator’s bias be
Then , so the generalized scalar inequality is
Using the decomposition of mean squared error into variance and squared bias gives
A biased estimator can therefore have variance below the ordinary unbiased bound without violating the theorem. The bias itself and its local rate of change both matter.
This distinction is relevant to shrinkage and regularization, which deliberately introduce bias to reduce variability or improve overall estimation error. The biased bound is informative when a bias function is specified or known; it does not supply a single positive pointwise error floor for unrestricted estimators.
Equality and efficiency
Equality in the scalar bound occurs precisely when the centered estimator and the score are linearly dependent with probability one:
For an unbiased estimator of , this reduces to
This is a restrictive condition. An estimator that attains the bound is often called efficient, although efficiency can also refer to asymptotic or relative comparisons.
For an unbiased estimator of , Cramér–Rao efficiency is commonly expressed as
Attaining the bound establishes minimum variance among unbiased estimators at the parameter values concerned. The converse does not hold: an estimator can have the smallest variance among all unbiased estimators while still lying strictly above the Cramér–Rao bound.
Examples
Normal mean with known variance
Suppose follow a normal distribution with unknown mean and known variance . The total Fisher information for is
Therefore every unbiased estimator of satisfies
The sample mean has exactly this variance, so it attains the bound for every sample size.
Bernoulli probability
For independent observations from a Bernoulli distribution with success probability ,
The sample proportion is unbiased and satisfies
It therefore attains the bound. The restriction to the interior matters: the usual regular analysis does not automatically extend to the boundary points.
Exponential rate: an unattained bound
Let follow an exponential distribution with rate . Then
giving the unbiased bound .
For , the estimator
is unbiased and has variance
This exceeds the bound. It illustrates why calculating a lower bound does not by itself produce an estimator achieving it.
Vector parameters and nuisance parameters
For a parameter vector , the score is a gradient:
The Fisher information matrix is
If is unbiased for and the information matrix is nonsingular, the matrix bound is
Here means that the difference is positive semidefinite. Thus the inequality controls the variance of every linear combination of the estimator’s components, not just their individual variances.
More generally, if estimates a differentiable vector function without bias, with Jacobian matrix , then
Unknown nuisance parameters
Suppose is the scalar parameter of interest and consists of unknown nuisance parameters. Partition the information as
Under the usual nonsingularity conditions, the variance bound for an estimator unbiased for throughout the joint model is
The subtracted term describes the loss of information associated with the unknown nuisance parameters. Using alone would instead treat those parameters as known.
A singular information matrix requires separate analysis. It may reflect a redundant parameterization or failure of local identification; replacing the inverse by a generalized inverse requires additional estimability conditions.
Relationship to large-sample estimation
Under suitable conditions, maximum likelihood estimators satisfy
Their asymptotic covariance therefore agrees with the inverse-information benchmark, even if they are biased in finite samples and do not attain the finite-sample bound.
This asymptotic efficiency is different from exact equality in the Cramér–Rao inequality. It requires conditions such as identifiability, adequate smoothness, an interior true parameter, and appropriate information behavior. Moreover, convergence in distribution alone does not imply convergence of variances; additional moment control is needed to justify a literal finite-sample variance approximation.
Applications and limitations
The bound provides a benchmark for comparing estimation procedures and evaluating measurement systems. In experimental design, controllable settings affect the information matrix. Common objectives include maximizing its determinant, minimizing the trace of its inverse, or minimizing the bound for a particular parameter or contrast.
These comparisons are conditional on the assumed probability model. Model misspecification, dependence omitted from the likelihood, or unmodeled systematic effects can make an apparently favorable bound misleading.
Parameter-dependent support
A standard counterexample comes from a uniform distribution on . For observations, let
The unbiased estimator
has variance
Naively differentiating the log-density only inside its support gives a score of , whose expectation is not zero. Ignoring the moving boundary would produce a spurious bound contradicted by the variance above. The failure lies in the regularity assumptions, not in the inequality.
Other limitations include the restriction to specified bias behavior, possible nonattainability, boundary parameters, and models without an ordinary differentiable likelihood. The bound also concerns estimator covariance, not the coverage of a confidence interval or the accuracy of an individual realized estimate.
Alternatives include finite-difference information inequalities, constrained-parameter bounds, and Bayesian bounds. The van Trees inequality, for example, incorporates a prior distribution and bounds an integrated estimation risk under its own regularity and boundary conditions.
Historical development
The inequality is named after Harald Cramér and C. R. Rao. Rao presented a version in 1945, and Cramér developed the result in his 1946 book Mathematical Methods of Statistics. Related earlier work is associated with Maurice Fréchet and Georges Darmois.
The result belongs to the broader development of likelihood-based inference and Fisher information, introduced through the work of Ronald Fisher. Its enduring role is to connect the sensitivity of a probability model to a precise limit on estimation variance.