The Rao–Blackwell theorem is a result in statistics showing how to improve an estimator by replacing it with its conditional expectation given a sufficient statistic. The new estimator has the same expectation as the original and no greater variance, provided the relevant moments exist. More generally, it has no greater expected loss whenever the loss is convex in the estimate. This procedure, called Rao–Blackwellization, connects statistical sufficiency with variance reduction and provides a central method for constructing efficient estimators.
Statistical setting and statement
Let denote observed data with distribution , where is an unknown parameter. Suppose estimates a real-valued quantity , and let be sufficient for .
Sufficiency means that the conditional distribution of the full data , given , can be chosen not to depend on . Consequently, under the usual regularity conditions for conditional distributions, the expression
can be represented by a single function of that does not involve the unknown parameter. Thus is an implementable estimator, rather than a formula requiring knowledge of .
If for every , then
and
In particular, if is unbiased for , then is also unbiased and has no greater variance at any parameter value.
The theorem does not require the original estimator to be unbiased. Since the bias is unchanged, its mean squared error also cannot increase:
Here and throughout, comparisons are made separately at each fixed value of .
Convex-loss formulation
The more general formulation uses a loss function , where is the reported estimate or action. Its risk is
Suppose the action space is a convex subset of a finite-dimensional real vector space, is integrable, and is a convex function of for every fixed . Under conditions ensuring that the expectations are well defined,
This includes squared-error loss and absolute-error loss. It also applies to vector-valued estimators when the loss is convex in the estimated vector.
Convexity is essential to this general argument. For an arbitrary nonconvex loss, replacing an estimator by a conditional average need not improve its risk. Likewise, if the permitted action space is not convex, the conditional average may not be an allowed action.
Proof and equality conditions
Preservation of expectation
The tower property of expectation gives
Thus conditioning preserves the estimator’s expectation and hence its bias.
Variance reduction
The conditional variance decomposition states that
Therefore,
The reduction is strict at a given precisely when the original estimator retains positive conditional variance with positive probability. Equality holds exactly when
Equivalently, at that parameter value, is already a function of , apart from events of probability zero.
For any fixed target , a stronger identity holds:
This identifies the exact reduction in squared-error risk.
Convex-loss proof
Conditional Jensen’s inequality gives
Taking expectations establishes the risk inequality. If the loss is strictly convex, equality, under the relevant integrability conditions, requires to be conditionally constant given .
Why sufficiency matters
The probabilistic inequality
holds for any statistic , not just a sufficient one. Sufficiency has a different role: it ensures that conditional averaging can be performed without knowing .
For a nonsufficient statistic, may depend on the unknown parameter. For example, if is constant, then conditioning produces . An unbiased estimator of would thereby become the constant , which has zero variance but cannot be calculated from the data.
Sufficiency is therefore a general guarantee that the variance-reduced expression remains a valid statistical procedure. It is not logically necessary in every individual application: a particular conditional expectation may happen to be parameter-free even when the conditioning statistic is not sufficient.
Examples
Estimating a Bernoulli success probability
Let be independent observations from a Bernoulli distribution with success probability , and define
The statistic is sufficient for and has a binomial distribution. A simple unbiased estimator of is , which ignores all observations except the first.
Given , each of the positions is equally likely to contain one of the successes. Hence
so Rao–Blackwellization gives the sample mean:
Their variances are
For and , the improvement is strict.
Estimating the square of a probability
In the same model, suppose the target is , with . The estimator
is unbiased because the observations are independent. Conditional on , the probability that both specified positions contain successes is
Thus
is unbiased for and has variance no greater than that of . This illustrates how the theorem can transform an inefficient estimator into a useful nonlinear function of a sufficient statistic.
Completeness and minimum-variance unbiased estimation
The Rao–Blackwell theorem guarantees improvement, not necessarily optimality among all unbiased estimators. An additional property, completeness, supplies the standard route to that stronger conclusion.
A statistic is complete if, for every measurable satisfying the required integrability conditions,
implies
The Lehmann–Scheffé theorem states that an unbiased function of a complete sufficient statistic is the unique uniformly minimum-variance unbiased estimator, up to almost-sure equality, among unbiased estimators with finite variance.
The two results serve distinct purposes:
- Rao–Blackwell: conditioning on a sufficient statistic improves or preserves risk under convex loss.
- Lehmann–Scheffé: completeness establishes uniqueness and minimum variance within the unbiased class.
Neither statement implies optimality over all biased and unbiased estimators. Under squared-error loss, a biased estimator can sometimes have smaller risk than a minimum-variance unbiased estimator.
Geometric interpretation
For a fixed , square-integrable random variables form a Hilbert space with inner product
The random variables measurable with respect to form a closed linear subspace. Conditional expectation is the orthogonal projection onto this subspace.
Writing
the residual is orthogonal to every square-integrable function :
The squared-error decomposition is consequently a Pythagorean identity. This interpretation explains why removing variation within groups of observations having the same sufficient statistic cannot increase squared-error risk.
Computational applications and limitations
In Monte Carlo methods, Rao–Blackwellization often refers more broadly to replacing a simulated quantity by an analytically calculated conditional expectation
Then
For independent replications, this directly reduces the variance of the corresponding Monte Carlo average. Unlike classical parameter estimation, this application need not involve a sufficient statistic: the simulation distribution is known, so the conditional expectation may be computable directly.
Conditional averaging is also used in Bayesian computation and Markov chain Monte Carlo. For correlated simulation output, however, reducing the variance of each individual output does not by itself establish a smaller asymptotic variance for its average; serial covariances must also be considered.
Practical limitations include the difficulty of computing conditional expectations and their computational cost. An exact conditional average may require an expensive integral or sum. Replacing it by a numerical or simulation-based approximation introduces additional error and does not automatically retain the exact theorem’s risk guarantee.
Historical development
The theorem is named after C. R. Rao and David Blackwell. Rao established the variance-reduction result in 1945, and Blackwell developed the conditional-expectation approach in 1947, including its broader decision-theoretic formulation. Together with the theory of sufficiency and the later completeness-based characterization of optimal unbiased estimators, it became a foundational result in mathematical statistics.