The Cramér–Rao inequality is a theorem in statistics that places a lower bound on the variance of an estimator in terms of the information contained in the observations. In its best-known form, an unbiased estimator of a scalar parameter satisfies
where is the Fisher information in the entire observed sample. The inequality requires regularity conditions connecting differentiation of the statistical model with integration over possible observations; it is not a universal bound for every estimation problem.
The result provides a benchmark for statistical precision independently of the particular algorithm used to construct an estimator. It helps characterize efficient estimators, explains the large-sample behavior of maximum likelihood estimation, and guides experimental design. Its extensions address vector parameters, biased estimators, nuisance parameters, Bayesian risk, and quantum measurements. A central distinction throughout these applications is between a mathematical lower bound and an attainable level of performance: the bound may hold exactly without any estimator attaining it.
Historical development and statistical setting
The inequality emerged from the development of mathematical estimation theory in the first half of the twentieth century. Ronald Fisher introduced and developed the concept of information in connection with likelihood, sufficiency, and the precision of statistical estimates. Information expresses how strongly a probability distribution changes when its parameter changes. If two nearby parameter values produce markedly different distributions, observations can generally distinguish them more precisely than if the distributions change only slightly.
Maurice Fréchet obtained an early form of the information inequality in 1943. C. R. Rao published his formulation in the 1945 paper “Information and the Accuracy Attainable in the Estimation of Statistical Parameters,” and Harald Cramér developed the result in his 1946 book Mathematical Methods of Statistics. These contributions account for the alternative names Cramér–Rao, Rao–Cramér, and occasionally Fréchet–Cramér–Rao inequality. Rao’s work also connected statistical information with geometry, a connection that later became central to information geometry.
The underlying statistical setting is a family of distributions
The observations have distribution , but is unknown. An estimator is a measurable function of the observations, constructed without knowledge of the true parameter. Repeated samples at a fixed produce a sampling distribution for , whose spread measures one aspect of estimation uncertainty. The Cramér–Rao inequality relates this spread to a property of the model rather than to a specific computational method.
Historically, the inequality helped distinguish several meanings of statistical optimality. An estimator may have minimum variance among unbiased estimators, attain the information bound, or be efficient only as sample size tends to infinity. These properties are not equivalent. A minimum-variance unbiased estimator need not attain the Cramér–Rao bound, and an estimator with small finite-sample bias may have lower mean squared error than any unbiased estimator. The theorem therefore belongs to a broader framework of decision theory, in which performance depends on the estimator class and the chosen loss function.
Scalar formulation and regularity conditions
Suppose the model has a density or mass function with respect to a fixed dominating measure. The density may be a probability density function for continuous observations or a probability mass function for discrete observations. The likelihood function treats as a function of for the observed value . Its logarithmic derivative,
is called the score function. Under suitable regularity conditions,
Thus information is the variance of the score.
Let estimate a differentiable scalar function without bias:
If has finite variance and differentiation may be interchanged with integration in the relevant expectations, then
The familiar reciprocal-information bound is the special case . The derivative in the numerator matters: estimating , for example, is not the same precision problem as estimating .
A common sufficient framework assumes that is an open interval, the support of the distribution does not depend on , and the density is differentiable with suitable integrable bounds on its derivatives. These assumptions allow a derivative to pass through an integral, often through the dominated convergence theorem. Fixed support alone is not sufficient, and parameter-dependent support is not by itself a complete diagnosis; the decisive issue is whether the identities used in the proof are valid. Under stronger second-order assumptions,
This expected curvature should be distinguished from the random observed curvature evaluated at a particular sample.
For observations that satisfy statistical independence and have a common distribution, the sample score is the sum of the individual scores. Their zero means make the cross terms vanish, giving
Consequently,
for unbiased estimation of . Independent but nonidentically distributed observations contribute information additively as well. Dependent observations require the information of their full joint distribution; multiplying a marginal information by sample size is generally unjustified.
Proof and geometric interpretation
The proof rests on differentiating an expected value and applying the Cauchy–Schwarz inequality. Starting with , the regularity assumptions give
Because , the final expression equals
Here the integration notation also represents summation in a discrete model.
Applying Cauchy–Schwarz to the centered estimator and the score yields
Division by positive information establishes the result. The proof identifies the precise statistical role of regularity: it ensures that a change in the estimator’s expectation is represented by its covariance with the score. The final inequality is then a general fact about square-integrable random variables.
There is also a direct interpretation in a Hilbert space. At a fixed parameter value, centered square-integrable functions of the data form an inner-product space with
The component of along the score is
Writing the estimator’s centered error as this component plus an orthogonal residual gives
By the identity for the squared length of orthogonal components,
This decomposition makes the inequality more informative than a numerical bound alone. The excess variance is exactly the squared size of a component orthogonal to the score. The score describes the first-order change in the distribution as the parameter moves; the estimator must have a specified response to that change. Any remaining fluctuation increases variance without contributing to this required local response. Attainment means that no such residual remains.
The argument is local in the parameter: expectations and inner products are evaluated under one . Nevertheless, an estimator claimed to be unbiased throughout a model must satisfy its expectation identity across the parameter space. Confusing pointwise statements with requirements on a single estimator valid for all parameter values can lead to incorrect claims of attainability.
Equality, efficient estimators, and exact examples
Equality holds precisely when the centered estimator is proportional to the score:
The qualification almost surely allows differences on events of probability zero. An unbiased estimator satisfying this condition is called efficient at that parameter value. Efficiency throughout the model requires one statistic, independent of the unknown parameter, to satisfy the equality condition for every parameter value. This is a restrictive requirement.
For independent observations from a normal distribution with mean and known variance ,
The sample mean satisfies
and therefore attains the bound exactly. Two other standard examples have the same property. For Bernoulli observations with success probability ,
For Poisson observations with mean ,
Both sample means are unbiased and efficient.
Exact attainment is not automatic even in an elementary regular model. Suppose follow an exponential distribution with rate , and let . The score and information are
For , the estimator
is unbiased for , with
This exceeds the Cramér–Rao bound . The estimator depends reciprocally on , whereas equality would require an affine relationship with the score.
Equality has a close connection with the exponential family of distributions. In a scalar problem, if an estimator of attains the bound throughout an interval and the necessary smoothness conditions hold, the equality condition implies
Integrating with respect to produces a representation of the form
Thus exact efficiency often reflects a special algebraic structure of the model. Membership in an exponential family does not, however, guarantee an unbiased efficient estimator for every function of its parameter.
Vector parameters and nuisance parameters
For a parameter vector , the score is the gradient of the log likelihood:
Its covariance matrix,
is the Fisher information matrix. Under regularity it is symmetric and positive semidefinite. Suppose is unbiased for a differentiable vector-valued function , and let be its Jacobian matrix. If is nonsingular, the matrix Cramér–Rao inequality is
The symbol means that the difference is positive semidefinite.
This is stronger than separate lower bounds on the component variances. For every fixed vector ,
The inequality therefore constrains every linear combination of the estimated quantities, including combinations affected by covariance. When , the Jacobian is the identity and the bound reduces to . The matrix proof follows the scalar proof by projecting the centered estimator onto the span of all score components.
Nuisance parameters are parameters needed to describe the observations but not themselves the estimation target. Write , where is the target and is nuisance, and partition the information:
For a scalar target, when the relevant inverses exist,
The denominator is the information remaining after projecting the target score away from the nuisance-score directions. It is no greater than , the information that would apply if the nuisance parameter were known.
For a normal distribution parameterized by its mean and variance , one observation has information
The zero off-diagonal terms mean that the mean and variance scores are orthogonal. With observations, the bound for unbiased estimation of is . Yet the usual unbiased sample variance has variance . This is another case in which the information bound is valid but unattained.
Singular information requires separate treatment. It may indicate redundant coordinates or a failure of local statistical identification, although a singular first-order information matrix does not always mean global nonidentifiability. Generalized-inverse bounds require compatibility conditions between the target derivative and the range of the information matrix; replacing an inverse mechanically is not sufficient.
Biased estimation and mean squared error
Unbiasedness is a condition on an estimator’s expectation, not a universal criterion of quality. Define the bias of an estimator of by
The same score argument applies to the mean function , giving
Consequently, its mean squared error satisfies
For estimation of , the numerator becomes .
The derivative of bias is essential. An estimator can have small bias at one parameter value while its mean responds quite differently to nearby changes in the parameter. The Cramér–Rao argument measures this local response through the derivative of the mean. Knowing the bias at one isolated point therefore does not determine the appropriate variance bound there. Local unbiasedness for ordinarily requires both the correct mean and unit derivative of the mean at the point in question.
A simple normal example illustrates the bias–variance tradeoff. If estimates a normal mean with known variance , consider
for a fixed constant . Its bias and variance are
Since , this estimator attains the biased variance bound. Its mean squared error is
For , it improves on the sample mean near , while the bias term makes it worse sufficiently far away.
A constant estimator is an extreme but useful illustration. If , its variance is zero, its bias is , and . The biased inequality is consistent with its zero variance, while its squared error is . No positive pointwise lower bound can apply to the mean squared error of every possible estimator at every parameter value without additional restrictions.
These distinctions matter for regularization methods, including ridge regression and other shrinkage procedures. Such estimators may deliberately accept bias to reduce error. The unbiased information bound is not a universal lower bound for their risk. Their analysis requires a bias-sensitive inequality or another criterion, such as integrated risk, minimax risk, or performance over a specified parameter neighborhood.
Large-sample efficiency and likelihood methods
In many regular models, exact finite-sample attainment is unavailable, but maximum likelihood estimators approach the information bound as sample size increases. Under conditions ensuring identification, consistency, and appropriate smoothness,
The notation denotes convergence in distribution, and is the information in one observation. For a vector parameter, the limiting law is a multivariate normal distribution. This statement concerns asymptotic distributions; convergence of the actual variances requires additional control of moments or tails.
The mechanism comes from expanding the likelihood score near the true parameter. If an interior maximum satisfies , a first-order expansion gives, schematically,
where is the Hessian matrix of the log likelihood. Typically,
The first limit follows from a central limit theorem, and the second commonly from a law of large numbers. Together they yield
This explains the frequent appearance of inverse information in a reported standard error. Expected information or observed likelihood curvature can provide a large-sample covariance approximation, leading to approximate confidence intervals. These approximations can be inaccurate in small samples, near parameter boundaries, or when the likelihood is far from quadratic. An inverse Hessian is therefore not, by itself, a finite-sample precision guarantee.
The local asymptotic normality framework gives a more general interpretation. It studies alternatives of the form
under which a regular model approaches a Gaussian shift experiment. Appropriate notions of estimator regularity then produce asymptotic information bounds that do not require exact finite-sample unbiasedness. This is the setting in which inverse information becomes a broad limit on regular large-sample estimation.
Regularity of the estimator remains important. Hodges-type superefficient estimators can have smaller limiting error than the usual information benchmark at selected parameter values, while performing poorly along nearby parameter sequences. This does not contradict either the finite-sample unbiased inequality or the asymptotic theory. Pointwise asymptotic behavior is weaker than uniform behavior over shrinking neighborhoods.
Nonregular models and limits of applicability
A standard failure occurs when the support depends on the parameter. Let be independent observations from the uniform distribution on , where . Away from the moving boundary, differentiating the log density appears to give
This cannot be a regular score because its expectation is not zero. The missing contribution comes from the changing support, and ordinary differentiation under the integral does not capture the full change in probability.
The sample maximum has expectation , so
is unbiased for . Its variance is
This decreases at order , rather than the usual regular-model order . It does not violate the Cramér–Rao theorem because the required score identities fail. The apparent contradiction results from applying a regular-model formula outside its assumptions.
Boundary parameters create a different difficulty. For a Bernoulli probability , the regular information formula applies on ; at or , the distribution is degenerate and two-sided parameter perturbations are unavailable. Variance components constrained to be nonnegative can likewise have nonnormal limiting distributions when the true value is zero. A model may be smooth in its interior yet require different asymptotic analysis at its boundary.
Other complications include infinite information, nondifferentiable models, and information matrices that become singular at particular parameter values. Mixture models can lose regularity when components coincide or a mixing weight vanishes. Increasing parameter dimension or allowing an expanding collection of nuisance parameters also requires assumptions beyond the fixed-dimensional theory. A symbolic likelihood derivative does not establish that a particular information bound is meaningful.
Model misspecification raises a separate issue. If the actual data distribution lies outside the assumed family, a likelihood estimator may converge to the parameter minimizing Kullback–Leibler divergence from the true distribution to the model. Under suitable conditions, its asymptotic covariance has the sandwich form
where is the negative expected log-likelihood Hessian and is the score covariance at that limiting parameter. Under correct specification, ; otherwise, the identity need not hold. Ordinary inverse-information calculations can then misrepresent uncertainty even when the numerical likelihood maximization succeeds.
Experimental design and measurement applications
Information depends on how data are collected as well as on the parameter. If an experiment is controlled by a design variable , its information can be written . Independent measurements contribute additive information, enabling a comparison of measurement times, sensor positions, stimulus settings, or sample allocations. For scalar estimation, maximizing information minimizes the Cramér–Rao variance benchmark. This comparison is meaningful only relative to the assumed observation model and any nuisance parameters.
For example, in the normal linear regression model
with known and a fixed full-column-rank design matrix,
The ordinary least squares estimator has covariance
and attains the bound. Nearly dependent design columns make the information poorly conditioned, producing high variance in some parameter combinations. This is a statistical consequence of the same geometry that can cause numerical difficulties in solving the regression equations.
There is no single best design criterion for every vector estimation problem. D-optimal design maximizes the determinant of the information matrix, corresponding to reducing the volume of a local uncertainty ellipsoid. A-optimal design minimizes the trace of the inverse information, while E-optimal design maximizes the smallest information eigenvalue. A criterion for one specified contrast instead focuses on . These objectives can favor different designs. In nonlinear models, information usually depends on unknown parameter values, so local design calculations may use nominal values, while Bayesian or robust formulations average over or protect against parameter uncertainty.
Measurement applications often have an especially transparent derivative structure. Suppose
independently, with known noise variance. Then
Measurements are informative where the predicted signal changes strongly with the target parameter relative to noise. For a time delay in a signal , the derivative is proportional to the signal’s time derivative. This connects delay precision with signal bandwidth and explains the use of information bounds in radar, localization, and communications.
Unknown amplitudes, offsets, or noise characteristics can reduce usable information through nuisance-parameter coupling. In photon-counting imaging, Poisson models yield analogous formulas involving derivatives of expected counts. Such bounds characterize idealized statistical precision, not every source of measurement uncertainty. Calibration errors, an inaccurate signal model, unresolved ambiguities, and estimator failure can produce errors not represented by the calculated bound.
Generalizations and related information bounds
The classical result is local and derivative-based. The Hammersley–Chapman–Robbins bound instead compares distributions at separated parameter values. If is absolutely continuous with respect to , an unbiased estimator of satisfies, under the relevant integrability conditions,
Taking the supremum over admissible can improve the bound. In regular models, the limit as recovers the Cramér–Rao expression. Finite-difference bounds can remain useful when differentiability is unavailable, though support and absolute-continuity conditions still matter.
The Barankin bound uses collections of parameter alternatives and their likelihood ratios to obtain sharper constraints on unbiased estimation. Another complementary result, the Rao–Blackwell theorem, improves an unbiased estimator by taking its conditional expectation given a sufficient statistic. This operation cannot increase variance, but it need not produce an estimator attaining the information bound. Sufficient-statistic methods and information inequalities address related but distinct questions: how to improve an estimator, and how small its variance can possibly be.
For Bayesian inference, the van Trees inequality bounds risk averaged over a prior distribution. In a scalar form, with prior density , suitable smoothness, and boundary terms that vanish,
Unlike the classical unbiased bound, it applies to integrated squared error and does not require the estimator to be unbiased. It is not a pointwise inequality at each fixed parameter value. Boundary conditions are part of the theorem and cannot be dropped when applying the displayed formula.
In information geometry, Fisher information defines a metric tensor on a regular statistical model. A small displacement has squared statistical length
The transformation laws for this metric explain why information bounds behave consistently under smooth changes of parameter coordinates. In semiparametric statistics, nuisance parameters may be infinite-dimensional. Projecting the target score away from the closure of nuisance-score directions yields an efficient score and an associated lower bound, extending the finite-dimensional nuisance-parameter construction.
In quantum mechanics, the distribution of observed outcomes depends on both the quantum state and the measurement performed. Quantum Fisher information provides a measurement-independent limit on the classical information obtainable about a state parameter. For a single parameter, the quantum Cramér–Rao bound constrains locally unbiased estimation; attainability depends on the measurements and repetition scheme allowed. In multiparameter problems, measurements optimal for different parameters can be incompatible, so an inverse quantum information matrix need not be jointly attainable. More refined quantum bounds address this incompatibility explicitly.