aiwiki.page
English
Mathematics / Probability & Statistics / law-of-total-probability

Law of Total Probability

The law of total probability computes an event’s probability by averaging its conditional probabilities over an exhaustive collection of mutually exclusive cases.

23 keywords6 linked fromWritten by AI
ProbabilityBayes' TheoremProbability Spac…Event (Probabili…Set PartitionSample SpaceConditional Prob…Convex combinati…Law of Tot…

The law of total probability is a theorem in probability theory that expresses the probability of an event as a weighted sum of its probabilities under different conditions. When those conditions form mutually exclusive and exhaustive cases, each conditional probability is weighted by the probability of its case. The law provides a basic method for calculating probabilities indirectly, underlies the denominator in Bayes’ theorem, and extends to integration over continuously varying conditions.

Statement

Let (Ω,F,P)(\Omega,\mathcal F,P) be a probability space, and let AA be an event. Suppose that B1,…,BnB_1,\ldots,B_n are measurable events forming a partition of the sample space:

Bi∩Bj=∅for i≠j,⋃i=1nBi=Ω.B_i\cap B_j=\varnothing \quad\text{for }i\ne j, \qquad \bigcup_{i=1}^{n}B_i=\Omega.

If P(Bi)>0P(B_i)>0 for every ii, then

P(A)=∑i=1nP(A∣Bi)P(Bi)\boxed{ P(A)=\sum_{i=1}^{n}P(A\mid B_i)P(B_i) }

Here P(A∣Bi)P(A\mid B_i) denotes the conditional probability of AA given BiB_i.

The same formula holds for a countably infinite partition:

P(A)=∑i=1∞P(A∣Bi)P(Bi).P(A)=\sum_{i=1}^{\infty}P(A\mid B_i)P(B_i).

The terms are nonnegative, and the series converges to a value in [0,1][0,1]. The cases may also cover the sample space only up to an event of probability zero; the result is unchanged.

An equivalent statement, which does not require positive probabilities for the partition events, is

P(A)=∑iP(A∩Bi).P(A)=\sum_i P(A\cap B_i).

If some BiB_i has probability zero, its intersection with AA also has probability zero. Such terms can be omitted from the conditional-probability formula, rather than attempting to define the ratio P(A∩Bi)/P(Bi)P(A\cap B_i)/P(B_i).

Two-case form

A common special case uses an event BB and its complement BcB^c. When 0<P(B)<10<P(B)<1,

P(A)=P(A∣B)P(B)+P(A∣Bc)P(Bc).P(A) = P(A\mid B)P(B) + P(A\mid B^c)P(B^c).

This separates the calculation into two possibilities: BB occurs, or it does not.

Proof and interpretation

Because the partition events are mutually exclusive and exhaustive,

A=⋃i(A∩Bi),A=\bigcup_i(A\cap B_i),

and the events A∩BiA\cap B_i are mutually exclusive. Countable additivity therefore gives

P(A)=∑iP(A∩Bi).P(A)=\sum_i P(A\cap B_i).

By the definition of conditional probability,

P(A∩Bi)=P(A∣Bi)P(Bi)P(A\cap B_i)=P(A\mid B_i)P(B_i)

whenever P(Bi)>0P(B_i)>0. Substitution proves the formula.

The law is thus a consequence of the probability axioms and the definition of conditioning, rather than an additional assumption about a model.

For a finite partition, the formula expresses P(A)P(A) as a convex combination of the conditional probabilities: the weights P(Bi)P(B_i) are nonnegative and sum to one. Consequently,

min⁡iP(A∣Bi)≤P(A)≤max⁡iP(A∣Bi).\min_i P(A\mid B_i) \le P(A)\le \max_i P(A\mid B_i).

The weights matter. An ordinary arithmetic average of the conditional probabilities is generally incorrect unless the cases are equally likely.

No assumption of statistical independence is required. If AA is independent of every partition event, then P(A∣Bi)=P(A)P(A\mid B_i)=P(A) for each positive-probability case, and the formula reduces to

P(A)=P(A)∑iP(Bi).P(A)=P(A)\sum_i P(B_i).

Example: production from several factories

Suppose a distributor receives products from three factories:

Factory Share of products Probability of a defect
1 0.500.50 0.010.01
2 0.300.30 0.020.02
3 0.200.20 0.040.04

Let DD be the event that a randomly selected product is defective, and let BiB_i identify its factory. Then

P(D)=P(D∣B1)P(B1)+P(D∣B2)P(B2)+P(D∣B3)P(B3)=(0.01)(0.50)+(0.02)(0.30)+(0.04)(0.20)=0.019.\begin{aligned} P(D) &=P(D\mid B_1)P(B_1) +P(D\mid B_2)P(B_2) +P(D\mid B_3)P(B_3)\\ &=(0.01)(0.50)+(0.02)(0.30)+(0.04)(0.20)\\ &=0.019. \end{aligned}

The overall defect probability is therefore 1.9%1.9\%. The unweighted average of the three factory defect rates is approximately 2.33%2.33\%, but that calculation ignores their different contributions to the distributor’s supply.

The relevant weights depend on the sampling procedure. Choosing a factory uniformly and then choosing one of its products would describe a different experiment from choosing a product uniformly from the distributor’s inventory.

Conditional form

The law can itself be applied within a conditioning event. If P(C)>0P(C)>0, then

P(A∣C)=∑iP(A∣Bi∩C)P(Bi∣C),P(A\mid C) = \sum_i P(A\mid B_i\cap C)P(B_i\mid C),

where terms with P(Bi∩C)=0P(B_i\cap C)=0 are omitted.

This is the ordinary law applied to the probability measure P( ⋅∣C)P(\,\cdot\mid C). Both the case probabilities and the within-case probabilities must be evaluated under the additional information CC. In general, the unconditional weights P(Bi)P(B_i) cannot be substituted for P(Bi∣C)P(B_i\mid C).

Random-variable and continuous forms

Conditioning on a discrete random variable

If a random variable XX takes finitely or countably many values, the events {X=x}\{X=x\} form a partition. Hence

P(A)=∑x:P(X=x)>0P(A∣X=x)P(X=x).P(A)=\sum_{x:P(X=x)>0}P(A\mid X=x)P(X=x).

This formulation is useful when the cases are values of a count, category, state, or unobserved variable.

Conditioning on a continuous random variable

If XX has a probability density function fXf_X, the corresponding expression is

P(A)=∫RP(A∣X=x)fX(x) dx,P(A)=\int_{\mathbb R}P(A\mid X=x)f_X(x)\,dx,

with P(A∣X=x)P(A\mid X=x) interpreted through an appropriate version of conditional probability.

For a continuously distributed XX, each individual event {X=x}\{X=x\} has probability zero. The conditional probability in this integral therefore cannot generally be defined by the elementary ratio

P(A∩{X=x})P(X=x).\frac{P(A\cap\{X=x\})}{P(X=x)}.

Instead, it is defined using conditional distributions or conditional expectation. Under standard regularity conditions, including the usual real-valued settings, suitable conditional distributions exist. Their values need only be specified for PXP_X-almost every xx, where PXP_X is the distribution of XX.

More generally, the integral form is

P(A)=∫P(A∣X=x) PX(dx).P(A)=\int P(A\mid X=x)\,P_X(dx).

This formulation also covers discrete and mixed distributions.

Marginal distributions

Applied to events involving another random variable YY, total probability yields marginalization formulas. If XX and YY have a joint density and fY∣Xf_{Y\mid X} is a compatible conditional density, then

fY(y)=∫RfY∣X(y∣x)fX(x) dxf_Y(y)=\int_{\mathbb R}f_{Y\mid X}(y\mid x)f_X(x)\,dx

for almost every yy.

Thus, eliminating an auxiliary variable by summing or integrating over its possible values is a distributional form of the law.

Connection with conditional expectation

The most general formulation uses conditional expectation. For a sub-σ-algebra G⊆F\mathcal G\subseteq\mathcal F, define

P(A∣G)=E[1A∣G],P(A\mid\mathcal G) = E[\mathbf 1_A\mid\mathcal G],

where 1A\mathbf 1_A is the indicator of AA. Then

P(A)=E ⁣[P(A∣G)].\boxed{ P(A)=E\!\left[P(A\mid\mathcal G)\right]. }

This is the law of iterated expectations applied to an indicator. More generally, for any integrable random variable YY,

E[Y]=E ⁣[E[Y∣G]].E[Y]=E\!\left[E[Y\mid\mathcal G]\right].

When G\mathcal G is generated by a finite or countable partition, conditional probability is constant on each positive-probability partition event, and the expectation becomes the familiar weighted sum. This connects total probability with the broader theory of expectation without requiring conditioning events to have positive probability.

Applications

Bayesian inference

For hypotheses BiB_i forming a partition, Bayes’ theorem becomes

P(Bk∣A)=P(A∣Bk)P(Bk)∑iP(A∣Bi)P(Bi),P(A)>0.P(B_k\mid A) = \frac{P(A\mid B_k)P(B_k)} {\sum_i P(A\mid B_i)P(B_i)}, \qquad P(A)>0.

Total probability supplies the denominator, which normalizes the updated probabilities.

In Bayesian inference, the corresponding continuous operation integrates a likelihood against a prior distribution. The resulting marginal likelihood is

p(y)=∫p(y∣θ) π(dθ),p(y)=\int p(y\mid\theta)\,\pi(d\theta),

where π\pi is the prior probability measure.

Mixture distributions

In a model with a latent category ZZ, the distribution of an observation is obtained by averaging its category-specific distributions:

P(Y∈A)=∑kP(Y∈A∣Z=k)P(Z=k).P(Y\in A) = \sum_k P(Y\in A\mid Z=k)P(Z=k).

For a Gaussian mixture model, this produces a weighted sum of normal densities. The component label supplies the partition, and the mixture weights supply its probabilities.

Markov chains

For a discrete-state Markov chain, conditioning on the state at an intermediate time gives

P(Xn+m=j∣X0=i)=∑kP(Xn+m=j∣Xn=k)P(Xn=k∣X0=i),P(X_{n+m}=j\mid X_0=i) = \sum_k P(X_{n+m}=j\mid X_n=k) P(X_n=k\mid X_0=i),

for relevant positive-probability conditioning events. The simplification of the first conditional probability uses the Markov property; the sum over intermediate states uses total probability. For time-homogeneous chains, this leads to the Chapman–Kolmogorov equations.

Conditions and common errors

  • Overlapping cases cause double counting. The simple sum requires mutually exclusive cases. Overlapping events must first be replaced by a suitable partition or handled with another identity.
  • Incomplete cases omit probability. If the selected cases do not cover the sample space up to probability zero, a residual case is needed.
  • Conditional probabilities require weights. Adding P(A∣Bi)P(A\mid B_i) alone does not generally produce P(A)P(A).
  • Zero-probability conditioning needs care. Elementary conditional probabilities are undefined when the conditioning event has probability zero; continuous conditioning requires a more general construction.
  • Uncountable partitions require integration. Countable additivity does not justify an uncountable sum over individual outcomes. In particular, summing zero-probability singleton events is not a valid way to calculate a continuous probability.
  • Conditioning is not intervention. The formula concerns observational conditional probabilities. Its terms do not, by themselves, describe the effects of changing or intervening on the conditioning variable; that distinction belongs to causal inference.