{
  "id": "cramer-rao-bound",
  "title": "Cramér–Rao Bound",
  "lang": "en",
  "source_lang": "en",
  "summary": "The Cramér–Rao bound limits the variance of an estimator in terms of Fisher information, providing a benchmark for statistical precision under specified regularity conditions.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "Cramér–Rao lower bound",
    "CRLB",
    "Cramér–Rao inequality",
    "Information inequality"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/cramer-rao-bound/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/cramer-rao-bound/",
      "json": "https://aiwiki.page/zh/cramer-rao-bound.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The Cramér–Rao bound is an inequality in statistics that places a lower limit on the variance of an estimator in a differentiable statistical model. In its best-known form, it states that the variance of an unbiased estimator of a scalar parameter cannot be smaller than the reciprocal of the Fisher information in the observations. The bound describes a theoretical limit on precision within a specified model and class of estimators; it does not generally guarantee that an estimator attaining that limit exists."
    },
    {
      "heading": "Scalar formulation",
      "text": "Let \\(X\\) denote the observed data, with density or probability mass function \\(p(x;\\theta)\\), where \\(\\theta\\) is an unknown real parameter. The score is the derivative of the log-likelihood function:\n\n\\[U_\\theta(X) = \\frac{\\partial}{\\partial\\theta}\\log p(X;\\theta).\\]\n\nUnder suitable regularity conditions, its expected value is zero, and the Fisher information is\n\n\\[I(\\theta) = \\mathbb E_\\theta[U_\\theta(X)^2].\\]\n\nIf \\(T(X)\\) is unbiased for \\(\\theta\\), meaning that \\(\\mathbb E_\\theta[T]=\\theta\\) throughout the parameter space, then\n\n\\[\\boxed{ \\operatorname{Var}_\\theta(T) \\geq \\frac{1}{I(\\theta)} }\\]\n\nprovided \\(0<I(\\theta)<\\infty\\) and the required differentiation and integrability conditions hold.\n\nHere, \\(I(\\theta)\\) is the information in the entire observed dataset. For \\(n\\) independent and identically distributed observations, independence makes information additive:\n\n\\[I_n(\\theta)=nI_1(\\theta).\\]\n\nConsequently,\n\n\\[\\operatorname{Var}_\\theta(T) \\geq \\frac{1}{nI_1(\\theta)}.\\]\n\nThe corresponding lower bound on the standard error is \\(1/\\sqrt{nI_1(\\theta)}\\).\n\nEstimating a function of the parameter\n\nIf \\(T\\) is unbiased for a differentiable function \\(g(\\theta)\\), rather than for \\(\\theta\\) itself, the bound becomes\n\n\\[\\operatorname{Var}_\\theta(T) \\geq \\frac{[g'(\\theta)]^2}{I(\\theta)}.\\]\n\nThis formulation is important because information and precision depend on how a parameter is expressed. A change from a variance to a standard deviation, for example, changes the appropriate bound through the derivative of the transformation."
    },
    {
      "heading": "Regularity conditions",
      "text": "The Cramér–Rao inequality is not valid for every statistical model. Its usual proof requires conditions that justify differentiating probability integrals and expectations.\n\nCommon sufficient assumptions include:\n\n- The parameter lies in an open interval, and the model is differentiable with respect to it.\n- The support of the distribution does not depend on the parameter.\n- Differentiation can be interchanged with integration or summation, both for the normalization of \\(p(x;\\theta)\\) and for the expectation of \\(T\\).\n- The estimator has finite variance, and the Fisher information is finite and positive.\n\nParameter-independent support is a convenient sufficient condition, not an indispensable requirement in every generalized formulation. The essential issue is whether the differentiation identities used in the proof are valid, including any boundary terms.\n\nUnder additional conditions permitting a second differentiation, Fisher information can also be written as\n\n\\[I(\\theta) = -\\mathbb E_\\theta \\left[ \\frac{\\partial^2}{\\partial\\theta^2} \\log p(X;\\theta) \\right].\\]\n\nThis expected negative curvature of the log-likelihood should be distinguished from observed information, the negative second derivative evaluated at the actual data. The ordinary Cramér–Rao bound uses expected Fisher information."
    },
    {
      "heading": "Derivation",
      "text": "The scalar bound follows from the Cauchy–Schwarz inequality.\n\nWrite\n\n\\[m(\\theta)=\\mathbb E_\\theta[T].\\]\n\nDifferentiating under the integral gives\n\n\\[\\begin{aligned} m'(\\theta) &= \\int T(x)\\, \\frac{\\partial p(x;\\theta)}{\\partial\\theta}\\,dx\\\\ &= \\mathbb E_\\theta[TU_\\theta]. \\end{aligned}\\]\n\nBecause \\(\\mathbb E_\\theta[U_\\theta]=0\\), this identity is equivalent to\n\n\\[\\operatorname{Cov}_\\theta(T,U_\\theta)=m'(\\theta).\\]\n\nThus the covariance inequality yields\n\n\\[[m'(\\theta)]^2 \\leq \\operatorname{Var}_\\theta(T) \\operatorname{Var}_\\theta(U_\\theta) = \\operatorname{Var}_\\theta(T)I(\\theta).\\]\n\nDividing by \\(I(\\theta)\\) proves the result. For an unbiased estimator of \\(\\theta\\), \\(m'(\\theta)=1\\); for an unbiased estimator of \\(g(\\theta)\\), \\(m'(\\theta)=g'(\\theta)\\).\n\nThe proof reveals the bound’s basic mechanism: an estimator whose expectation changes rapidly with the parameter must have enough covariance with the score, and therefore enough variance."
    },
    {
      "heading": "Biased estimators and mean squared error",
      "text": "The familiar reciprocal-information bound is not a universal lower bound for all estimators. Let an estimator’s bias be\n\n\\[b(\\theta)=\\mathbb E_\\theta[T]-\\theta.\\]\n\nThen \\(m'(\\theta)=1+b'(\\theta)\\), so the generalized scalar inequality is\n\n\\[\\operatorname{Var}_\\theta(T) \\geq \\frac{[1+b'(\\theta)]^2}{I(\\theta)}.\\]\n\nUsing the decomposition of mean squared error into variance and squared bias gives\n\n\\[\\mathbb E_\\theta[(T-\\theta)^2] \\geq b(\\theta)^2+ \\frac{[1+b'(\\theta)]^2}{I(\\theta)}.\\]\n\nA biased estimator can therefore have variance below the ordinary unbiased bound without violating the theorem. The bias itself and its local rate of change both matter.\n\nThis distinction is relevant to shrinkage and regularization, which deliberately introduce bias to reduce variability or improve overall estimation error. The biased bound is informative when a bias function is specified or known; it does not supply a single positive pointwise error floor for unrestricted estimators."
    },
    {
      "heading": "Equality and efficiency",
      "text": "Equality in the scalar bound occurs precisely when the centered estimator and the score are linearly dependent with probability one:\n\n\\[T-m(\\theta) = \\frac{m'(\\theta)}{I(\\theta)}U_\\theta(X).\\]\n\nFor an unbiased estimator of \\(\\theta\\), this reduces to\n\n\\[U_\\theta(X) = I(\\theta)[T(X)-\\theta].\\]\n\nThis is a restrictive condition. An estimator that attains the bound is often called efficient, although efficiency can also refer to asymptotic or relative comparisons.\n\nFor an unbiased estimator of \\(\\theta\\), Cramér–Rao efficiency is commonly expressed as\n\n\\[\\operatorname{eff}_\\theta(T) = \\frac{1/I(\\theta)} {\\operatorname{Var}_\\theta(T)} \\leq 1.\\]\n\nAttaining the bound establishes minimum variance among unbiased estimators at the parameter values concerned. The converse does not hold: an estimator can have the smallest variance among all unbiased estimators while still lying strictly above the Cramér–Rao bound."
    },
    {
      "heading": "Examples",
      "text": "Normal mean with known variance\n\nSuppose \\(X_1,\\ldots,X_n\\) follow a normal distribution with unknown mean \\(\\mu\\) and known variance \\(\\sigma^2\\). The total Fisher information for \\(\\mu\\) is\n\n\\[I_n(\\mu)=\\frac{n}{\\sigma^2}.\\]\n\nTherefore every unbiased estimator of \\(\\mu\\) satisfies\n\n\\[\\operatorname{Var}_\\mu(T)\\geq\\frac{\\sigma^2}{n}.\\]\n\nThe sample mean has exactly this variance, so it attains the bound for every sample size.\n\nBernoulli probability\n\nFor independent observations from a Bernoulli distribution with success probability \\(p\\in(0,1)\\),\n\n\\[I_n(p)=\\frac{n}{p(1-p)}.\\]\n\nThe sample proportion \\(\\widehat p\\) is unbiased and satisfies\n\n\\[\\operatorname{Var}_p(\\widehat p) = \\frac{p(1-p)}{n}.\\]\n\nIt therefore attains the bound. The restriction to the interior \\(0<p<1\\) matters: the usual regular analysis does not automatically extend to the boundary points.\n\nExponential rate: an unattained bound\n\nLet \\(X_1,\\ldots,X_n\\) follow an exponential distribution with rate \\(\\lambda>0\\). Then\n\n\\[I_n(\\lambda)=\\frac{n}{\\lambda^2},\\]\n\ngiving the unbiased bound \\(\\lambda^2/n\\).\n\nFor \\(n>2\\), the estimator\n\n\\[T=\\frac{n-1}{\\sum_{i=1}^{n}X_i}\\]\n\nis unbiased and has variance\n\n\\[\\operatorname{Var}_\\lambda(T) = \\frac{\\lambda^2}{n-2}.\\]\n\nThis exceeds the bound. It illustrates why calculating a lower bound does not by itself produce an estimator achieving it."
    },
    {
      "heading": "Vector parameters and nuisance parameters",
      "text": "For a parameter vector \\(\\boldsymbol\\theta\\in\\mathbb R^d\\), the score is a gradient:\n\n\\[\\mathbf U_{\\boldsymbol\\theta} = \\nabla_{\\boldsymbol\\theta} \\log p(X;\\boldsymbol\\theta).\\]\n\nThe Fisher information matrix is\n\n\\[\\mathbf I(\\boldsymbol\\theta) = \\mathbb E_{\\boldsymbol\\theta} [ \\mathbf U_{\\boldsymbol\\theta} \\mathbf U_{\\boldsymbol\\theta}^{\\mathsf T} ].\\]\n\nIf \\(\\mathbf T\\) is unbiased for \\(\\boldsymbol\\theta\\) and the information matrix is nonsingular, the matrix bound is\n\n\\[\\operatorname{Cov}_{\\boldsymbol\\theta}(\\mathbf T) \\succeq \\mathbf I(\\boldsymbol\\theta)^{-1}.\\]\n\nHere \\(\\succeq\\) means that the difference is positive semidefinite. Thus the inequality controls the variance of every linear combination of the estimator’s components, not just their individual variances.\n\nMore generally, if \\(\\mathbf T\\) estimates a differentiable vector function \\(\\mathbf g(\\boldsymbol\\theta)\\) without bias, with Jacobian matrix \\(\\mathbf G\\), then\n\n\\[\\operatorname{Cov}_{\\boldsymbol\\theta}(\\mathbf T) \\succeq \\mathbf G\\mathbf I^{-1}\\mathbf G^{\\mathsf T}.\\]\n\nUnknown nuisance parameters\n\nSuppose \\(\\theta\\) is the scalar parameter of interest and \\(\\boldsymbol\\eta\\) consists of unknown nuisance parameters. Partition the information as\n\n\\[\\mathbf I= \\begin{pmatrix} I_{\\theta\\theta} & \\mathbf I_{\\theta\\eta}\\\\ \\mathbf I_{\\eta\\theta} & \\mathbf I_{\\eta\\eta} \\end{pmatrix}.\\]\n\nUnder the usual nonsingularity conditions, the variance bound for an estimator unbiased for \\(\\theta\\) throughout the joint model is\n\n\\[\\operatorname{Var}(T) \\geq \\left( I_{\\theta\\theta} - \\mathbf I_{\\theta\\eta} \\mathbf I_{\\eta\\eta}^{-1} \\mathbf I_{\\eta\\theta} \\right)^{-1}.\\]\n\nThe subtracted term describes the loss of information associated with the unknown nuisance parameters. Using \\(1/I_{\\theta\\theta}\\) alone would instead treat those parameters as known.\n\nA singular information matrix requires separate analysis. It may reflect a redundant parameterization or failure of local identification; replacing the inverse by a generalized inverse requires additional estimability conditions."
    },
    {
      "heading": "Relationship to large-sample estimation",
      "text": "Under suitable conditions, maximum likelihood estimators satisfy\n\n\\[\\sqrt n(\\widehat\\theta_n-\\theta) \\xrightarrow{d} N\\!\\left(0,I_1(\\theta)^{-1}\\right).\\]\n\nTheir asymptotic covariance therefore agrees with the inverse-information benchmark, even if they are biased in finite samples and do not attain the finite-sample bound.\n\nThis asymptotic efficiency is different from exact equality in the Cramér–Rao inequality. It requires conditions such as identifiability, adequate smoothness, an interior true parameter, and appropriate information behavior. Moreover, convergence in distribution alone does not imply convergence of variances; additional moment control is needed to justify a literal finite-sample variance approximation."
    },
    {
      "heading": "Applications and limitations",
      "text": "The bound provides a benchmark for comparing estimation procedures and evaluating measurement systems. In experimental design, controllable settings affect the information matrix. Common objectives include maximizing its determinant, minimizing the trace of its inverse, or minimizing the bound for a particular parameter or contrast.\n\nThese comparisons are conditional on the assumed probability model. Model misspecification, dependence omitted from the likelihood, or unmodeled systematic effects can make an apparently favorable bound misleading.\n\nParameter-dependent support\n\nA standard counterexample comes from a uniform distribution on \\((0,\\theta)\\). For \\(n\\) observations, let\n\n\\[M=\\max(X_1,\\ldots,X_n).\\]\n\nThe unbiased estimator\n\n\\[T=\\frac{n+1}{n}M\\]\n\nhas variance\n\n\\[\\operatorname{Var}_\\theta(T) = \\frac{\\theta^2}{n(n+2)}.\\]\n\nNaively differentiating the log-density only inside its support gives a score of \\(-n/\\theta\\), whose expectation is not zero. Ignoring the moving boundary would produce a spurious bound contradicted by the variance above. The failure lies in the regularity assumptions, not in the inequality.\n\nOther limitations include the restriction to specified bias behavior, possible nonattainability, boundary parameters, and models without an ordinary differentiable likelihood. The bound also concerns estimator covariance, not the coverage of a confidence interval or the accuracy of an individual realized estimate.\n\nAlternatives include finite-difference information inequalities, constrained-parameter bounds, and Bayesian bounds. The van Trees inequality, for example, incorporates a prior distribution and bounds an integrated estimation risk under its own regularity and boundary conditions."
    },
    {
      "heading": "Historical development",
      "text": "The inequality is named after Harald Cramér and C. R. Rao. Rao presented a version in 1945, and Cramér developed the result in his 1946 book Mathematical Methods of Statistics. Related earlier work is associated with Maurice Fréchet and Georges Darmois.\n\nThe result belongs to the broader development of likelihood-based inference and Fisher information, introduced through the work of Ronald Fisher. Its enduring role is to connect the sensitivity of a probability model to a precise limit on estimation variance."
    }
  ],
  "links": [
    "statistics",
    "variance",
    "estimator",
    "fisher-information",
    "likelihood-function",
    "expected-value",
    "statistical-independence",
    "standard-error",
    "cauchy-schwarz-inequality",
    "covariance",
    "estimator-bias",
    "mean-squared-error",
    "regularization",
    "normal-distribution",
    "sample-mean",
    "bernoulli-distribution",
    "exponential-distribution",
    "gradient",
    "jacobian-matrix",
    "statistical-identification",
    "maximum-likelihood-estimation",
    "experimental-design",
    "probability-distribution",
    "uniform-distribution",
    "confidence-interval",
    "prior-distribution",
    "ronald-fisher"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}