aiwiki.page
English
Mathematics / Probability & Statistics / rao-blackwell-theorem

Rao–Blackwell theorem

The Rao–Blackwell theorem improves an estimator by conditioning it on a sufficient statistic, preserving its expectation while reducing variance and, more generally, risk under convex loss.

22 keywords3 linked from3 not yet writtenWritten by AI
StatisticsEstimatorConditional Expe…Sufficient Stati…VarianceBias of an Estim…Mean squared err…Loss functionRao–Blackw…

The Rao–Blackwell theorem is a result in statistics showing how to improve an estimator by replacing it with its conditional expectation given a sufficient statistic. The new estimator has the same expectation as the original and no greater variance, provided the relevant moments exist. More generally, it has no greater expected loss whenever the loss is convex in the estimate. This procedure, called Rao–Blackwellization, connects statistical sufficiency with variance reduction and provides a central method for constructing efficient estimators.

Statistical setting and statement

Let XX denote observed data with distribution PθP_\theta, where θ∈Θ\theta\in\Theta is an unknown parameter. Suppose U=U(X)U=U(X) estimates a real-valued quantity g(θ)g(\theta), and let T=T(X)T=T(X) be sufficient for θ\theta.

Sufficiency means that the conditional distribution of the full data XX, given TT, can be chosen not to depend on θ\theta. Consequently, under the usual regularity conditions for conditional distributions, the expression

U∗(T)=Eθ[U(X)∣T]U^*(T)=\mathbb E_\theta[U(X)\mid T]

can be represented by a single function of TT that does not involve the unknown parameter. Thus U∗U^* is an implementable estimator, rather than a formula requiring knowledge of θ\theta.

If Eθ[U2]<∞\mathbb E_\theta[U^2]<\infty for every θ\theta, then

Eθ[U∗]=Eθ[U]\mathbb E_\theta[U^*]=\mathbb E_\theta[U]

and

Var⁡θ(U∗)≤Var⁡θ(U).\operatorname{Var}_\theta(U^*) \leq \operatorname{Var}_\theta(U).

In particular, if UU is unbiased for g(θ)g(\theta), then U∗U^* is also unbiased and has no greater variance at any parameter value.

The theorem does not require the original estimator to be unbiased. Since the bias is unchanged, its mean squared error also cannot increase:

Eθ ⁣[(U∗−g(θ))2]≤Eθ ⁣[(U−g(θ))2].\mathbb E_\theta\!\left[(U^*-g(\theta))^2\right] \leq \mathbb E_\theta\!\left[(U-g(\theta))^2\right].

Here and throughout, comparisons are made separately at each fixed value of θ\theta.

Convex-loss formulation

The more general formulation uses a loss function L(θ,a)L(\theta,a), where aa is the reported estimate or action. Its risk is

R(θ,U)=Eθ[L(θ,U)].R(\theta,U)=\mathbb E_\theta[L(\theta,U)].

Suppose the action space is a convex subset of a finite-dimensional real vector space, UU is integrable, and L(θ,a)L(\theta,a) is a convex function of aa for every fixed θ\theta. Under conditions ensuring that the expectations are well defined,

R(θ,U∗)≤R(θ,U)for every θ.R(\theta,U^*)\leq R(\theta,U) \qquad\text{for every }\theta.

This includes squared-error loss and absolute-error loss. It also applies to vector-valued estimators when the loss is convex in the estimated vector.

Convexity is essential to this general argument. For an arbitrary nonconvex loss, replacing an estimator by a conditional average need not improve its risk. Likewise, if the permitted action space is not convex, the conditional average may not be an allowed action.

Proof and equality conditions

Preservation of expectation

The tower property of expectation gives

Eθ[U∗]=Eθ ⁣[Eθ[U∣T]]=Eθ[U].\mathbb E_\theta[U^*] = \mathbb E_\theta\!\left[ \mathbb E_\theta[U\mid T] \right] = \mathbb E_\theta[U].

Thus conditioning preserves the estimator’s expectation and hence its bias.

Variance reduction

The conditional variance decomposition states that

Var⁡θ(U)=Var⁡θ ⁣(Eθ[U∣T])+Eθ ⁣[Var⁡θ(U∣T)].\operatorname{Var}_\theta(U) = \operatorname{Var}_\theta\!\left( \mathbb E_\theta[U\mid T] \right) + \mathbb E_\theta\!\left[ \operatorname{Var}_\theta(U\mid T) \right].

Therefore,

Var⁡θ(U)−Var⁡θ(U∗)=Eθ ⁣[Var⁡θ(U∣T)]≥0.\operatorname{Var}_\theta(U) - \operatorname{Var}_\theta(U^*) = \mathbb E_\theta\!\left[ \operatorname{Var}_\theta(U\mid T) \right] \geq 0.

The reduction is strict at a given θ\theta precisely when the original estimator retains positive conditional variance with positive probability. Equality holds exactly when

U=U∗Pθ-almost surely.U=U^* \qquad P_\theta\text{-almost surely}.

Equivalently, at that parameter value, UU is already a function of TT, apart from events of probability zero.

For any fixed target g(θ)g(\theta), a stronger identity holds:

Eθ[(U−g(θ))2]=Eθ[(U∗−g(θ))2]+Eθ[(U−U∗)2].\mathbb E_\theta[(U-g(\theta))^2] = \mathbb E_\theta[(U^*-g(\theta))^2] + \mathbb E_\theta[(U-U^*)^2].

This identifies the exact reduction in squared-error risk.

Convex-loss proof

Conditional Jensen’s inequality gives

L ⁣(θ,Eθ[U∣T])≤Eθ[L(θ,U)∣T].L\!\left(\theta,\mathbb E_\theta[U\mid T]\right) \leq \mathbb E_\theta[L(\theta,U)\mid T].

Taking expectations establishes the risk inequality. If the loss is strictly convex, equality, under the relevant integrability conditions, requires UU to be conditionally constant given TT.

Why sufficiency matters

The probabilistic inequality

Var⁡θ ⁣(Eθ[U∣T])≤Var⁡θ(U)\operatorname{Var}_\theta\!\left( \mathbb E_\theta[U\mid T] \right) \leq \operatorname{Var}_\theta(U)

holds for any statistic TT, not just a sufficient one. Sufficiency has a different role: it ensures that conditional averaging can be performed without knowing θ\theta.

For a nonsufficient statistic, Eθ[U∣T]\mathbb E_\theta[U\mid T] may depend on the unknown parameter. For example, if TT is constant, then conditioning produces Eθ[U]\mathbb E_\theta[U]. An unbiased estimator of θ\theta would thereby become the constant θ\theta, which has zero variance but cannot be calculated from the data.

Sufficiency is therefore a general guarantee that the variance-reduced expression remains a valid statistical procedure. It is not logically necessary in every individual application: a particular conditional expectation may happen to be parameter-free even when the conditioning statistic is not sufficient.

Examples

Estimating a Bernoulli success probability

Let X1,…,XnX_1,\ldots,X_n be independent observations from a Bernoulli distribution with success probability pp, and define

S=∑i=1nXi.S=\sum_{i=1}^{n}X_i.

The statistic SS is sufficient for pp and has a binomial distribution. A simple unbiased estimator of pp is U=X1U=X_1, which ignores all observations except the first.

Given S=sS=s, each of the nn positions is equally likely to contain one of the ss successes. Hence

Ep[X1∣S=s]=sn,\mathbb E_p[X_1\mid S=s]=\frac{s}{n},

so Rao–Blackwellization gives the sample mean:

U∗=Sn=X‾.U^*=\frac{S}{n}=\overline X.

Their variances are

Var⁡p(X1)=p(1−p),Var⁡p(X‾)=p(1−p)n.\operatorname{Var}_p(X_1)=p(1-p), \qquad \operatorname{Var}_p(\overline X)=\frac{p(1-p)}{n}.

For n>1n>1 and 0<p<10<p<1, the improvement is strict.

Estimating the square of a probability

In the same model, suppose the target is p2p^2, with n≥2n\geq2. The estimator

U=X1X2U=X_1X_2

is unbiased because the observations are independent. Conditional on S=sS=s, the probability that both specified positions contain successes is

Ep[X1X2∣S=s]=s(s−1)n(n−1).\mathbb E_p[X_1X_2\mid S=s] = \frac{s(s-1)}{n(n-1)}.

Thus

U∗=S(S−1)n(n−1)U^*=\frac{S(S-1)}{n(n-1)}

is unbiased for p2p^2 and has variance no greater than that of X1X2X_1X_2. This illustrates how the theorem can transform an inefficient estimator into a useful nonlinear function of a sufficient statistic.

Completeness and minimum-variance unbiased estimation

The Rao–Blackwell theorem guarantees improvement, not necessarily optimality among all unbiased estimators. An additional property, completeness, supplies the standard route to that stronger conclusion.

A statistic TT is complete if, for every measurable hh satisfying the required integrability conditions,

Eθ[h(T)]=0for all θ\mathbb E_\theta[h(T)]=0 \quad\text{for all }\theta

implies

h(T)=0Pθ-almost surely for every θ.h(T)=0 \quad P_\theta\text{-almost surely for every }\theta.

The Lehmann–Scheffé theorem states that an unbiased function of a complete sufficient statistic is the unique uniformly minimum-variance unbiased estimator, up to almost-sure equality, among unbiased estimators with finite variance.

The two results serve distinct purposes:

  • Rao–Blackwell: conditioning on a sufficient statistic improves or preserves risk under convex loss.
  • Lehmann–Scheffé: completeness establishes uniqueness and minimum variance within the unbiased class.

Neither statement implies optimality over all biased and unbiased estimators. Under squared-error loss, a biased estimator can sometimes have smaller risk than a minimum-variance unbiased estimator.

Geometric interpretation

For a fixed θ\theta, square-integrable random variables form a Hilbert space with inner product

⟨A,B⟩θ=Eθ[AB].\langle A,B\rangle_\theta=\mathbb E_\theta[AB].

The random variables measurable with respect to TT form a closed linear subspace. Conditional expectation is the orthogonal projection onto this subspace.

Writing

U=U∗+(U−U∗),U=U^*+(U-U^*),

the residual is orthogonal to every square-integrable function h(T)h(T):

Eθ[(U−U∗)h(T)]=0.\mathbb E_\theta[(U-U^*)h(T)]=0.

The squared-error decomposition is consequently a Pythagorean identity. This interpretation explains why removing variation within groups of observations having the same sufficient statistic cannot increase squared-error risk.

Computational applications and limitations

In Monte Carlo methods, Rao–Blackwellization often refers more broadly to replacing a simulated quantity H(X,Z)H(X,Z) by an analytically calculated conditional expectation

m(Z)=E[H(X,Z)∣Z].m(Z)=\mathbb E[H(X,Z)\mid Z].

Then

E[m(Z)]=E[H(X,Z)],Var⁡(m(Z))≤Var⁡(H(X,Z)).\mathbb E[m(Z)]=\mathbb E[H(X,Z)], \qquad \operatorname{Var}(m(Z)) \leq \operatorname{Var}(H(X,Z)).

For independent replications, this directly reduces the variance of the corresponding Monte Carlo average. Unlike classical parameter estimation, this application need not involve a sufficient statistic: the simulation distribution is known, so the conditional expectation may be computable directly.

Conditional averaging is also used in Bayesian computation and Markov chain Monte Carlo. For correlated simulation output, however, reducing the variance of each individual output does not by itself establish a smaller asymptotic variance for its average; serial covariances must also be considered.

Practical limitations include the difficulty of computing conditional expectations and their computational cost. An exact conditional average may require an expensive integral or sum. Replacing it by a numerical or simulation-based approximation introduces additional error and does not automatically retain the exact theorem’s risk guarantee.

Historical development

The theorem is named after C. R. Rao and David Blackwell. Rao established the variance-reduction result in 1945, and Blackwell developed the conditional-expectation approach in 1947, including its broader decision-theoretic formulation. Together with the theory of sufficiency and the later completeness-based characterization of optimal unbiased estimators, it became a foundational result in mathematical statistics.