{
  "id": "rao-blackwell-theorem",
  "title": "Rao–Blackwell theorem",
  "lang": "en",
  "source_lang": "en",
  "summary": "The Rao–Blackwell theorem improves an estimator by conditioning it on a sufficient statistic, preserving its expectation while reducing variance and, more generally, risk under convex loss.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "Rao–Blackwellization theorem",
    "Rao–Blackwell inequality"
  ],
  "generated_at": "2026-11-06",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/rao-blackwell-theorem/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/rao-blackwell-theorem/",
      "json": "https://aiwiki.page/zh/rao-blackwell-theorem.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The Rao–Blackwell theorem is a result in statistics showing how to improve an estimator by replacing it with its conditional expectation given a sufficient statistic. The new estimator has the same expectation as the original and no greater variance, provided the relevant moments exist. More generally, it has no greater expected loss whenever the loss is convex in the estimate. This procedure, called Rao–Blackwellization, connects statistical sufficiency with variance reduction and provides a central method for constructing efficient estimators."
    },
    {
      "heading": "Statistical setting and statement",
      "text": "Let \\(X\\) denote observed data with distribution \\(P_\\theta\\), where \\(\\theta\\in\\Theta\\) is an unknown parameter. Suppose \\(U=U(X)\\) estimates a real-valued quantity \\(g(\\theta)\\), and let \\(T=T(X)\\) be sufficient for \\(\\theta\\).\n\nSufficiency means that the conditional distribution of the full data \\(X\\), given \\(T\\), can be chosen not to depend on \\(\\theta\\). Consequently, under the usual regularity conditions for conditional distributions, the expression\n\n\\[U^*(T)=\\mathbb E_\\theta[U(X)\\mid T]\\]\n\ncan be represented by a single function of \\(T\\) that does not involve the unknown parameter. Thus \\(U^*\\) is an implementable estimator, rather than a formula requiring knowledge of \\(\\theta\\).\n\nIf \\(\\mathbb E_\\theta[U^2]<\\infty\\) for every \\(\\theta\\), then\n\n\\[\\mathbb E_\\theta[U^*]=\\mathbb E_\\theta[U]\\]\n\nand\n\n\\[\\operatorname{Var}_\\theta(U^*) \\leq \\operatorname{Var}_\\theta(U).\\]\n\nIn particular, if \\(U\\) is unbiased for \\(g(\\theta)\\), then \\(U^*\\) is also unbiased and has no greater variance at any parameter value.\n\nThe theorem does not require the original estimator to be unbiased. Since the bias is unchanged, its mean squared error also cannot increase:\n\n\\[\\mathbb E_\\theta\\!\\left[(U^*-g(\\theta))^2\\right] \\leq \\mathbb E_\\theta\\!\\left[(U-g(\\theta))^2\\right].\\]\n\nHere and throughout, comparisons are made separately at each fixed value of \\(\\theta\\)."
    },
    {
      "heading": "Convex-loss formulation",
      "text": "The more general formulation uses a loss function \\(L(\\theta,a)\\), where \\(a\\) is the reported estimate or action. Its risk is\n\n\\[R(\\theta,U)=\\mathbb E_\\theta[L(\\theta,U)].\\]\n\nSuppose the action space is a convex subset of a finite-dimensional real vector space, \\(U\\) is integrable, and \\(L(\\theta,a)\\) is a convex function of \\(a\\) for every fixed \\(\\theta\\). Under conditions ensuring that the expectations are well defined,\n\n\\[R(\\theta,U^*)\\leq R(\\theta,U) \\qquad\\text{for every }\\theta.\\]\n\nThis includes squared-error loss and absolute-error loss. It also applies to vector-valued estimators when the loss is convex in the estimated vector.\n\nConvexity is essential to this general argument. For an arbitrary nonconvex loss, replacing an estimator by a conditional average need not improve its risk. Likewise, if the permitted action space is not convex, the conditional average may not be an allowed action."
    },
    {
      "heading": "Proof and equality conditions",
      "text": "Preservation of expectation\n\nThe tower property of expectation gives\n\n\\[\\mathbb E_\\theta[U^*] = \\mathbb E_\\theta\\!\\left[ \\mathbb E_\\theta[U\\mid T] \\right] = \\mathbb E_\\theta[U].\\]\n\nThus conditioning preserves the estimator’s expectation and hence its bias.\n\nVariance reduction\n\nThe conditional variance decomposition states that\n\n\\[\\operatorname{Var}_\\theta(U) = \\operatorname{Var}_\\theta\\!\\left( \\mathbb E_\\theta[U\\mid T] \\right) + \\mathbb E_\\theta\\!\\left[ \\operatorname{Var}_\\theta(U\\mid T) \\right].\\]\n\nTherefore,\n\n\\[\\operatorname{Var}_\\theta(U) - \\operatorname{Var}_\\theta(U^*) = \\mathbb E_\\theta\\!\\left[ \\operatorname{Var}_\\theta(U\\mid T) \\right] \\geq 0.\\]\n\nThe reduction is strict at a given \\(\\theta\\) precisely when the original estimator retains positive conditional variance with positive probability. Equality holds exactly when\n\n\\[U=U^* \\qquad P_\\theta\\text{-almost surely}.\\]\n\nEquivalently, at that parameter value, \\(U\\) is already a function of \\(T\\), apart from events of probability zero.\n\nFor any fixed target \\(g(\\theta)\\), a stronger identity holds:\n\n\\[\\mathbb E_\\theta[(U-g(\\theta))^2] = \\mathbb E_\\theta[(U^*-g(\\theta))^2] + \\mathbb E_\\theta[(U-U^*)^2].\\]\n\nThis identifies the exact reduction in squared-error risk.\n\nConvex-loss proof\n\nConditional Jensen’s inequality gives\n\n\\[L\\!\\left(\\theta,\\mathbb E_\\theta[U\\mid T]\\right) \\leq \\mathbb E_\\theta[L(\\theta,U)\\mid T].\\]\n\nTaking expectations establishes the risk inequality. If the loss is strictly convex, equality, under the relevant integrability conditions, requires \\(U\\) to be conditionally constant given \\(T\\)."
    },
    {
      "heading": "Why sufficiency matters",
      "text": "The probabilistic inequality\n\n\\[\\operatorname{Var}_\\theta\\!\\left( \\mathbb E_\\theta[U\\mid T] \\right) \\leq \\operatorname{Var}_\\theta(U)\\]\n\nholds for any statistic \\(T\\), not just a sufficient one. Sufficiency has a different role: it ensures that conditional averaging can be performed without knowing \\(\\theta\\).\n\nFor a nonsufficient statistic, \\(\\mathbb E_\\theta[U\\mid T]\\) may depend on the unknown parameter. For example, if \\(T\\) is constant, then conditioning produces \\(\\mathbb E_\\theta[U]\\). An unbiased estimator of \\(\\theta\\) would thereby become the constant \\(\\theta\\), which has zero variance but cannot be calculated from the data.\n\nSufficiency is therefore a general guarantee that the variance-reduced expression remains a valid statistical procedure. It is not logically necessary in every individual application: a particular conditional expectation may happen to be parameter-free even when the conditioning statistic is not sufficient."
    },
    {
      "heading": "Examples",
      "text": "Estimating a Bernoulli success probability\n\nLet \\(X_1,\\ldots,X_n\\) be independent observations from a Bernoulli distribution with success probability \\(p\\), and define\n\n\\[S=\\sum_{i=1}^{n}X_i.\\]\n\nThe statistic \\(S\\) is sufficient for \\(p\\) and has a binomial distribution. A simple unbiased estimator of \\(p\\) is \\(U=X_1\\), which ignores all observations except the first.\n\nGiven \\(S=s\\), each of the \\(n\\) positions is equally likely to contain one of the \\(s\\) successes. Hence\n\n\\[\\mathbb E_p[X_1\\mid S=s]=\\frac{s}{n},\\]\n\nso Rao–Blackwellization gives the sample mean:\n\n\\[U^*=\\frac{S}{n}=\\overline X.\\]\n\nTheir variances are\n\n\\[\\operatorname{Var}_p(X_1)=p(1-p), \\qquad \\operatorname{Var}_p(\\overline X)=\\frac{p(1-p)}{n}.\\]\n\nFor \\(n>1\\) and \\(0<p<1\\), the improvement is strict.\n\nEstimating the square of a probability\n\nIn the same model, suppose the target is \\(p^2\\), with \\(n\\geq2\\). The estimator\n\n\\[U=X_1X_2\\]\n\nis unbiased because the observations are independent. Conditional on \\(S=s\\), the probability that both specified positions contain successes is\n\n\\[\\mathbb E_p[X_1X_2\\mid S=s] = \\frac{s(s-1)}{n(n-1)}.\\]\n\nThus\n\n\\[U^*=\\frac{S(S-1)}{n(n-1)}\\]\n\nis unbiased for \\(p^2\\) and has variance no greater than that of \\(X_1X_2\\). This illustrates how the theorem can transform an inefficient estimator into a useful nonlinear function of a sufficient statistic."
    },
    {
      "heading": "Completeness and minimum-variance unbiased estimation",
      "text": "The Rao–Blackwell theorem guarantees improvement, not necessarily optimality among all unbiased estimators. An additional property, completeness, supplies the standard route to that stronger conclusion.\n\nA statistic \\(T\\) is complete if, for every measurable \\(h\\) satisfying the required integrability conditions,\n\n\\[\\mathbb E_\\theta[h(T)]=0 \\quad\\text{for all }\\theta\\]\n\nimplies\n\n\\[h(T)=0 \\quad P_\\theta\\text{-almost surely for every }\\theta.\\]\n\nThe Lehmann–Scheffé theorem states that an unbiased function of a complete sufficient statistic is the unique uniformly minimum-variance unbiased estimator, up to almost-sure equality, among unbiased estimators with finite variance.\n\nThe two results serve distinct purposes:\n\n- Rao–Blackwell: conditioning on a sufficient statistic improves or preserves risk under convex loss.\n- Lehmann–Scheffé: completeness establishes uniqueness and minimum variance within the unbiased class.\n\nNeither statement implies optimality over all biased and unbiased estimators. Under squared-error loss, a biased estimator can sometimes have smaller risk than a minimum-variance unbiased estimator."
    },
    {
      "heading": "Geometric interpretation",
      "text": "For a fixed \\(\\theta\\), square-integrable random variables form a Hilbert space with inner product\n\n\\[\\langle A,B\\rangle_\\theta=\\mathbb E_\\theta[AB].\\]\n\nThe random variables measurable with respect to \\(T\\) form a closed linear subspace. Conditional expectation is the orthogonal projection onto this subspace.\n\nWriting\n\n\\[U=U^*+(U-U^*),\\]\n\nthe residual is orthogonal to every square-integrable function \\(h(T)\\):\n\n\\[\\mathbb E_\\theta[(U-U^*)h(T)]=0.\\]\n\nThe squared-error decomposition is consequently a Pythagorean identity. This interpretation explains why removing variation within groups of observations having the same sufficient statistic cannot increase squared-error risk."
    },
    {
      "heading": "Computational applications and limitations",
      "text": "In Monte Carlo methods, Rao–Blackwellization often refers more broadly to replacing a simulated quantity \\(H(X,Z)\\) by an analytically calculated conditional expectation\n\n\\[m(Z)=\\mathbb E[H(X,Z)\\mid Z].\\]\n\nThen\n\n\\[\\mathbb E[m(Z)]=\\mathbb E[H(X,Z)], \\qquad \\operatorname{Var}(m(Z)) \\leq \\operatorname{Var}(H(X,Z)).\\]\n\nFor independent replications, this directly reduces the variance of the corresponding Monte Carlo average. Unlike classical parameter estimation, this application need not involve a sufficient statistic: the simulation distribution is known, so the conditional expectation may be computable directly.\n\nConditional averaging is also used in Bayesian computation and Markov chain Monte Carlo. For correlated simulation output, however, reducing the variance of each individual output does not by itself establish a smaller asymptotic variance for its average; serial covariances must also be considered.\n\nPractical limitations include the difficulty of computing conditional expectations and their computational cost. An exact conditional average may require an expensive integral or sum. Replacing it by a numerical or simulation-based approximation introduces additional error and does not automatically retain the exact theorem’s risk guarantee."
    },
    {
      "heading": "Historical development",
      "text": "The theorem is named after C. R. Rao and David Blackwell. Rao established the variance-reduction result in 1945, and Blackwell developed the conditional-expectation approach in 1947, including its broader decision-theoretic formulation. Together with the theory of sufficiency and the later completeness-based characterization of optimal unbiased estimators, it became a foundational result in mathematical statistics."
    }
  ],
  "links": [
    "statistics",
    "estimator",
    "conditional-expectation",
    "sufficient-statistic",
    "variance",
    "estimator-bias",
    "mean-squared-error",
    "loss-function",
    "convex-function",
    "expected-value",
    "jensens-inequality",
    "bernoulli-distribution",
    "binomial-distribution",
    "sample-mean",
    "hilbert-space",
    "orthogonal-projection",
    "monte-carlo-method",
    "bayesian-inference",
    "markov-chain-monte-carlo"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}