Conditional Independence

Conditional independence is a relation in probability theory and statistics describing variables whose dependence disappears once specified information is known. Two random variables $X$ and $Y$ are conditionally independent given $Z$ when their conditional joint distribution factors into their conditional marginal distributions. Written $X\perp\!\!\!\perp Y\mid Z$ , this means that, after accounting for $Z$ , observing $Y$ does not change the conditional distribution of $X$ , and conversely. Unlike ordinary statistical independence, the relation explicitly depends on the information being conditioned on. (stats.ox.ac.uk)

Mathematical definition

For discrete variables, conditional independence requires

P(X=x,Y=y\mid Z=z) = P(X=x\mid Z=z)P(Y=y\mid Z=z)

for every $x,y$ and every $z$ with $P(Z=z)>0$ . Equivalently,

P(X=x\mid Y=y,Z=z)=P(X=x\mid Z=z)

whenever the conditioning event has positive probability. Thus, the conditional joint distribution is a product distribution within each relevant stratum of $Z$ . For variables with suitable densities, the analogous factorization uses a conditional probability density:

f_{X,Y\mid Z}(x,y\mid z) =f_{X\mid Z}(x\mid z)f_{Y\mid Z}(y\mid z).

These identities need hold only almost everywhere, rather than at arbitrarily chosen points where conditional distributions may be undefined. (stats.ox.ac.uk)

A formulation using conditional expectation avoids requiring densities. For all bounded measurable functions $u,v$ ,

E[u(X)v(Y)\mid Z] = E[u(X)\mid Z]E[v(Y)\mid Z] \quad\text{almost surely}.

In measure theory, conditioning on $Z$ means conditioning on the sigma-algebra generated by $Z$ . The same definition extends to independence conditional on a general information sigma-algebra. (stats.ox.ac.uk)

Conditioning can remove or create dependence

Marginal independence and conditional independence do not imply one another. A shared underlying variable can make two observations dependent before conditioning, even when they are independent given that variable. Conversely, conditioning on information jointly determined by two independent variables can make them dependent. These patterns correspond to common-cause and common-effect structures in graphical models. (cs.cmu.edu)

As an explicit illustration, let $X,Y$ be independent fair binary variables and set $Z=X+Y$ . Conditional on $Z=1$ , only $(X,Y)=(0,1)$ and $(1,0)$ are possible, each with probability $1/2$ . Consequently,

P(X=1,Y=1\mid Z=1)=0,

whereas the product of the two conditional marginal probabilities is $1/4$ . Knowing the sum therefore destroys their independence.

Conditional independence is also stronger than zero conditional covariance or absence of conditional linear correlation. When the necessary moments exist, it implies

E[XY\mid Z]=E[X\mid Z]E[Y\mid Z],

but this single moment identity does not generally establish independence of the full conditional distributions. (arxiv.org)

Logical properties

Conditional independence obeys four fundamental inference rules, often called the semigraphoid properties. Here $X,Y,W,Z$ may represent collections of variables:

Symmetry: $X\perp Y\mid Z$ implies $Y\perp X\mid Z$ .
Decomposition: $X\perp(Y,W)\mid Z$ implies $X\perp Y\mid Z$ and $X\perp W\mid Z$ .
Weak union: $X\perp(Y,W)\mid Z$ implies $X\perp Y\mid(Z,W)$ .
Contraction: $X\perp Y\mid Z$ together with $X\perp W\mid(Y,Z)$ implies $X\perp(Y,W)\mid Z$ . (stats.ox.ac.uk)

A further rule, intersection, holds under suitable strict-positivity assumptions:

X\perp Y\mid(Z,W),\qquad X\perp W\mid(Z,Y) \quad\Longrightarrow\quad X\perp(Y,W)\mid Z.

Without those assumptions it can fail. Weak union does not license adding arbitrary conditioning variables: its premise requires independence from the entire pair $(Y,W)$ . (stats.ox.ac.uk)

Graphical representation

A probabilistic graphical model encodes conditional independence through graph structure. In a Bayesian network, a directed acyclic graph supports the factorization

p(x_1,\ldots,x_n) =\prod_{i=1}^{n}p(x_i\mid x_{\operatorname{pa}(i)}),

where $\operatorname{pa}(i)$ denotes the parents of node $i$ . Each variable is conditionally independent of its nondescendants given its parents. (cs.cmu.edu)

The d-separation criterion identifies further independences implied by this factorization. Conditioning on the middle node blocks a simple chain $X\to Z\to Y$ or fork $X\leftarrow Z\to Y$ . In a collider $X\to Z\leftarrow Y$ , however, conditioning on $Z$ , or a descendant of $Z$ , can open a previously blocked path. (cs.cmu.edu)

D-separation guarantees independence for distributions that factorize according to the graph; failure of d-separation does not guarantee dependence in every particular distribution. Special parameter choices may produce additional independences. Graphical dependence also does not by itself establish causation: causal inference requires additional assumptions about causal structure. (cs.cmu.edu)

Statistical applications and testing

In machine learning, the naive Bayes classifier assumes mutual conditional independence of features given a class variable $C$ :

p(x_1,\ldots,x_n\mid C) =\prod_i p(x_i\mid C).

Combined with Bayes’ theorem, this yields a compact classification model. The assumption concerns independence within classes, not independence across the pooled population. (cs.cmu.edu)

Conditional independence also expresses the role of a sufficient statistic: under an appropriate Bayesian interpretation with random parameter $\Theta$ , sufficiency can be expressed as $X\perp\Theta\mid T(X)$ . It formalizes the idea that the statistic retains the sample’s information about the parameter. (stats.ox.ac.uk)

Empirically assessing the relation is a problem in statistical hypothesis testing. Categorical variables permit contingency-table approaches, while continuously valued conditioning variables make distribution-free testing substantially harder. General impossibility results show that useful tests require restrictions on the distributional class or other assumptions; a test of conditional covariance alone cannot generally certify conditional independence. (stat.cmu.edu)