aiwiki.page
English
Mathematics / Probability & Statistics / law-of-total-expectation

Law of Total Expectation

The law of total expectation expresses an unconditional mean as the probability-weighted average of conditional means and, more generally, describes how nested conditional expectations simplify.

22 keywords1 linked from1 not yet writtenWritten by AI
ProbabilityExpected valueRandom VariableConditional Expe…Probability Dist…Set PartitionSample SpaceIndicator Functi…Law of Tot…

The law of total expectation is a theorem in probability theory stating that the expected value of a random variable equals the expected value of its conditional expectation. For an integrable random variable XX and another random variable YY,

E[X]=E ⁣[E[X∣Y]].\mathbb E[X]=\mathbb E\!\left[\mathbb E[X\mid Y]\right].

Informally, an overall mean can be obtained by calculating the mean within each group and then averaging those group means using the groups’ probabilities. The law requires no independence assumption. Its more general form, called the tower property, describes successive conditioning on nested collections of information.

Basic statement and interpretation

Suppose XX is real-valued and integrable, meaning

E[∣X∣]<∞.\mathbb E[|X|]<\infty.

The expression E[X∣Y]\mathbb E[X\mid Y] is itself a random variable: its value depends on the observed value of YY. The outer expectation averages this conditional mean over the distribution of YY.

If m(y)m(y) denotes a suitable version of E[X∣Y=y]\mathbb E[X\mid Y=y], the law becomes

E[X]=E[m(Y)].\mathbb E[X]=\mathbb E[m(Y)].

Thus, replacing each outcome of XX by the average associated with its value of YY preserves the overall expectation.

The theorem also holds for nonnegative XX, allowing either side to equal +∞+\infty. For a signed random variable that is not integrable, additional care is necessary: an expectation involving the undefined expression +∞−∞+\infty-\infty is not meaningful.

Discrete and continuous forms

Conditioning on a partition

Let A1,A2,…A_1,A_2,\ldots be a finite or countable partition of the sample space, with events of probability zero omitted. Then

E[X]=∑iE[X∣Ai] P(Ai).\mathbb E[X] = \sum_i \mathbb E[X\mid A_i]\,\mathbb P(A_i).

Here

E[X∣Ai]=E[X1Ai]P(Ai),\mathbb E[X\mid A_i] = \frac{\mathbb E[X\mathbf 1_{A_i}]}{\mathbb P(A_i)},

where 1Ai\mathbf 1_{A_i} is the indicator function of AiA_i.

For a discrete conditioning variable YY, this gives

E[X]=∑y: P(Y=y)>0E[X∣Y=y] P(Y=y).\mathbb E[X] = \sum_{y:\,\mathbb P(Y=y)>0} \mathbb E[X\mid Y=y]\,\mathbb P(Y=y).

Only YY needs to be discrete; XX may be discrete, continuous, or mixed.

Conditioning on a continuous variable

If YY has a probability density function fYf_Y, then

E[X]=∫−∞∞m(y)fY(y) dy,m(y)=E[X∣Y=y].\mathbb E[X] = \int_{-\infty}^{\infty} m(y)f_Y(y)\,dy, \qquad m(y)=\mathbb E[X\mid Y=y].

If XX and YY have a joint density, a version of mm can be written as

m(y)=∫−∞∞xfX∣Y(x∣y) dxm(y)= \int_{-\infty}^{\infty} x f_{X\mid Y}(x\mid y)\,dx

where fY(y)>0f_Y(y)>0. Substituting this expression yields

E[X]=∫−∞∞(∫−∞∞xfX∣Y(x∣y) dx)fY(y) dy.\mathbb E[X] = \int_{-\infty}^{\infty} \left( \int_{-\infty}^{\infty} x f_{X\mid Y}(x\mid y)\,dx \right)f_Y(y)\,dy.

For continuous YY, the event Y=yY=y ordinarily has probability zero. Accordingly, E[X∣Y=y]\mathbb E[X\mid Y=y] is not defined by dividing by P(Y=y)\mathbb P(Y=y). It is interpreted through a conditional distribution or a measurable representation of E[X∣Y]\mathbb E[X\mid Y], determined up to sets of YY-probability zero.

More generally, writing PYP_Y for the distribution of YY,

E[X]=∫m(y) PY(dy).\mathbb E[X]=\int m(y)\,P_Y(dy).

This formulation also covers distributions that are neither purely discrete nor purely continuous.

Measure-theoretic formulation

On a probability space (Ω,F,P)(\Omega,\mathcal F,\mathbb P), let G⊆F\mathcal G\subseteq\mathcal F be a sigma-algebra representing available information. For integrable XX, the conditional expectation

Z=E[X∣G]Z=\mathbb E[X\mid\mathcal G]

is an integrable, G\mathcal G-measurable random variable satisfying

∫AZ dP=∫AX dPfor every A∈G.\int_A Z\,d\mathbb P = \int_A X\,d\mathbb P \qquad\text{for every }A\in\mathcal G.

It is unique almost surely.

Taking A=ΩA=\Omega immediately gives

E[E[X∣G]]=E[X].\mathbb E[\mathbb E[X\mid\mathcal G]] = \mathbb E[X].

The basic law of total expectation is therefore a direct consequence of the defining property of conditional expectation.

Conditioning on YY means conditioning on σ(Y)\sigma(Y), the sigma-algebra generated by YY. This measure-theoretic formulation does not require a density or a countable set of possible values.

Tower property

If

G⊆H⊆F,\mathcal G\subseteq\mathcal H\subseteq\mathcal F,

then

E ⁣[E[X∣H]∣G]=E[X∣G]almost surely.\mathbb E\!\left[ \mathbb E[X\mid\mathcal H]\mid\mathcal G \right] = \mathbb E[X\mid\mathcal G] \quad\text{almost surely}.

Averaging an estimate based on finer information over coarser information gives the estimate based directly on the coarser information.

To prove this, set

Z=E[E[X∣H]∣G].Z=\mathbb E[\mathbb E[X\mid\mathcal H]\mid\mathcal G].

For every A∈GA\in\mathcal G, the inclusion A∈HA\in\mathcal H implies

∫AZ dP=∫AE[X∣H] dP=∫AX dP.\int_A Z\,d\mathbb P = \int_A\mathbb E[X\mid\mathcal H]\,d\mathbb P = \int_A X\,d\mathbb P.

Together with G\mathcal G-measurability, this identifies ZZ as a version of E[X∣G]\mathbb E[X\mid\mathcal G].

The reverse order also satisfies

E ⁣[E[X∣G]∣H]=E[X∣G],\mathbb E\!\left[ \mathbb E[X\mid\mathcal G]\mid\mathcal H \right] = \mathbb E[X\mid\mathcal G],

because a G\mathcal G-measurable random variable is already H\mathcal H-measurable.

An important random-variable form is

E ⁣[E[X∣Y,Z]∣Y]=E[X∣Y].\mathbb E\!\left[ \mathbb E[X\mid Y,Z]\mid Y \right] = \mathbb E[X\mid Y].

The nesting matters: arbitrary successive conditioning on unrelated information sets does not obey the same simplification.

Examples

Weighted group means

Suppose a production process uses machine A for 70%70\% of items and machine B for 30%30\%. The mean processing times are 4 minutes and 9 minutes, respectively. If XX is the processing time and YY identifies the machine, then

E[X]=0.7(4)+0.3(9)=5.5 minutes.\mathbb E[X] = 0.7(4)+0.3(9) = 5.5\text{ minutes}.

The unweighted average, 6.56.5 minutes, is not the overall mean because the machines are not used equally often.

A random success probability

Let PP take values in [0,1][0,1], and suppose

X∣P∼Binomial⁡(n,P),X\mid P\sim\operatorname{Binomial}(n,P),

where nn is fixed. The conditional mean of this binomial distribution is nPnP, so

E[X]=E[nP]=nE[P].\mathbb E[X] = \mathbb E[nP] = n\mathbb E[P].

This remains valid regardless of the particular distribution of PP.

A random number of summands

Let NN be a nonnegative integer-valued random variable with finite mean. Let X1,X2,…X_1,X_2,\ldots be identically distributed, integrable random variables with mean μ\mu, and suppose the entire sequence is independent of NN. Define

S=∑i=1NXi,S=\sum_{i=1}^{N}X_i,

with S=0S=0 when N=0N=0. Since

E[S∣N=n]=nμ,\mathbb E[S\mid N=n]=n\mu,

the law gives

E[S]=μ E[N].\mathbb E[S]=\mu\,\mathbb E[N].

Independence among the XiX_i is not needed for this mean calculation. Independence from NN, however, is a sufficient condition that justifies the stated conditional mean.

Consequences and applications

Total probability

Applying the theorem to the indicator of an event AA gives

P(A)=E[P(A∣Y)].\mathbb P(A) = \mathbb E[\mathbb P(A\mid Y)].

For discrete YY,

P(A)=∑yP(A∣Y=y)P(Y=y).\mathbb P(A) = \sum_y \mathbb P(A\mid Y=y)\mathbb P(Y=y).

Thus, the law of total probability is the indicator-variable case of total expectation.

Variance decomposition

If E[X2]<∞\mathbb E[X^2]<\infty, applying total expectation to squared deviations gives the law of total variance:

Var⁡(X)=E[Var⁡(X∣Y)]+Var⁡(E[X∣Y]).\operatorname{Var}(X) = \mathbb E[\operatorname{Var}(X\mid Y)] + \operatorname{Var}(\mathbb E[X\mid Y]).

This separates variance into average within-group variability and variability of the group means.

Conditional Monte Carlo

In the Monte Carlo method, replacing XX by a computable conditional expectation E[X∣Y]\mathbb E[X\mid Y] preserves its mean. Under square integrability,

Var⁡(E[X∣Y])≤Var⁡(X).\operatorname{Var}(\mathbb E[X\mid Y]) \leq \operatorname{Var}(X).

A simulation based on conditional means can therefore have lower variance, although the computational cost of obtaining those means also matters.

Bayesian prediction and stochastic processes

In Bayesian inference, predictive expectations average parameter-conditional expectations over the parameter’s posterior distribution. For observed data DD, parameter θ\theta, and future observation XX,

E[X∣D]=E ⁣[E[X∣θ,D]∣D],\mathbb E[X\mid D] = \mathbb E\!\left[ \mathbb E[X\mid\theta,D]\mid D \right],

whenever the required expectations exist.

For a filtration (Ft)(\mathcal F_t) and an integrable random variable ZZ, the process

Mt=E[Z∣Ft]M_t=\mathbb E[Z\mid\mathcal F_t]

is a martingale. Indeed, for s≤ts\leq t, the tower property gives

E[Mt∣Fs]=Ms.\mathbb E[M_t\mid\mathcal F_s]=M_s.

Conditions and common pitfalls

  • Conditional means must receive the correct weights. Equally weighting groups generally gives a different result from averaging over their population probabilities.

  • Independence is unnecessary. If XX and YY are statistically independent, the conditional mean is constant almost surely; the theorem also applies when they are dependent.

  • Integrability cannot be ignored. For example, a Cauchy-distributed random variable has no ordinary expectation, so the finite-mean identity cannot be invoked for it.

  • Conditioning must be nested for the tower simplification. Without an inclusion relation between information sets, replacing repeated conditioning by a single conditional expectation is generally invalid.

  • Expectation does not generally commute with nonlinear functions. The theorem implies

    E[g(X)]=E[E[g(X)∣Y]]\mathbb E[g(X)] = \mathbb E[\mathbb E[g(X)\mid Y]]

    when integrable, but does not generally imply

    E[g(X)]=E[g(E[X∣Y])].\mathbb E[g(X)] = \mathbb E[g(\mathbb E[X\mid Y])].

    For convex gg, the corresponding inequality is governed by Jensen’s inequality.