Joint entropy is the entropy of two or more random variables considered as a single random object. In information theory, it measures the average uncertainty about their combined outcome. Joint entropy depends on the variables’ full joint probability distribution, not just their individual distributions: independent variables contribute separate uncertainty, while dependence can make their combined outcome more predictable.
Definition
For discrete random variables and , with joint probability mass function
their joint entropy is
The sum ranges over all possible pairs of values, with the convention . The logarithm base determines the unit:
- Base gives bits.
- Base gives nats.
- Base gives hartleys.
Unless otherwise specified, information-theoretic calculations commonly use base .
Equivalently,
Thus, joint entropy is the expected value of the self-information of the observed pair. It is a property of the distribution, rather than of any particular realized pair.
For discrete variables,
These definitions apply to finite or countably infinite alphabets. For countably infinite alphabets, joint entropy may be infinite; identities involving subtraction require care in that case.
Conditional entropy and the chain rule
Conditional entropy measures the uncertainty remaining about one variable when another is known:
Values of with zero probability contribute nothing, so the conditional probability only needs to be specified where .
The factorization
gives the entropy chain rule:
Likewise,
This expresses joint uncertainty as the uncertainty about one variable plus the uncertainty remaining about the other after the first is known. Although the two decompositions use different conditional entropies, joint entropy itself is symmetric:
For multiple variables, the chain rule becomes
where the first term is . Any ordering gives the same total joint entropy, although individual conditional terms may change.
Bounds and equality conditions
For discrete variables with finite entropies,
The lower bound follows because discrete conditional entropy is nonnegative. Learning the pair cannot involve less uncertainty than learning either component alone.
The upper bound is called subadditivity. Equality holds exactly when and exhibit statistical independence:
At the other extreme,
exactly when . For discrete variables, this means that is a deterministic function of with probability one. In particular,
Repeating a variable adds no uncertainty.
If the joint distribution has exactly possible outcomes with positive probability, then
Equality holds precisely when those outcomes are equally likely. Consequently, if has at most values and at most values,
Joint entropy is also a concave function of the joint probability mass function: mixing distributions cannot produce an entropy smaller than the corresponding weighted average of their entropies.
Relationship to mutual information
Mutual information quantifies the departure of joint entropy from the sum of the marginal entropies:
Equivalently,
Thus, joint entropy measures total uncertainty in the pair, while mutual information measures how much knowing one variable reduces uncertainty about the other. The quantities answer different questions: two independent, highly unpredictable variables can have large joint entropy but zero mutual information.
Mutual information can also be written using Kullback–Leibler divergence:
Its nonnegativity yields subadditivity of joint entropy.
For more than two variables, the analogous difference
is called total correlation. It is nonnegative and, for finite entropies, vanishes exactly when the variables are mutually independent.
Examples
Two independent fair bits
Let and be independent binary variables, each equally likely to be or . All four pairs have probability , so
Each variable contributes one bit of uncertainty.
Two identical fair bits
If is a fair bit and , only and occur, each with probability . Hence
Both marginal entropies are still one bit, but the second variable adds no uncertainty once the first is known.
A noisy copy
Let be a fair bit, let be an independent binary noise variable with , and define
where denotes exclusive OR. Given , the uncertainty about is exactly the uncertainty about the noise:
where
is the binary entropy function.
At , the variables are identical; at , one is the deterministic complement of the other. Both cases give one bit of joint entropy. At , the variables are independent and the joint entropy is two bits.
Pairwise independence is not enough
Let and be independent fair bits and set
Every pair among is independent, but the three variables are not mutually independent: any two determine the third. Therefore,
although
This illustrates why pairwise dependence measures do not fully describe multivariable uncertainty.
Continuous variables
For jointly continuous variables with probability density function , the corresponding quantity is joint differential entropy:
when the integral is well defined.
Differential entropy differs fundamentally from discrete entropy:
- It can be negative.
- It changes under rescaling and other coordinate transformations.
- Joint differential entropy need not be at least as large as either marginal differential entropy.
Chain-rule and mutual-information identities continue to hold under suitable finiteness conditions, but discrete entropy bounds cannot all be transferred unchanged.
For a -dimensional multivariate normal distribution with positive-definite covariance matrix ,
The determinant reflects the spread of the distribution across all dimensions. If is singular, the distribution has no density with respect to -dimensional Lebesgue measure, so the usual full-dimensional differential-entropy formula does not apply.
Coding and stochastic processes
Joint entropy belongs to the framework developed by Claude Shannon for communication and information measurement. Its operational interpretation comes from lossless coding.
For a finite joint alphabet, the minimum expected length of a binary prefix code for the pair satisfies
Through coding long independent blocks, the average number of bits per pair can approach , as described by the source coding theorem. Joint coding can therefore exploit dependence that separate coding ignores.
For a stochastic process , block entropy is
The entropy rate, when the limit exists, is
It measures uncertainty per observation after accounting for temporal dependence. For a stationary finite-alphabet process, this limit exists. For a stationary first-order Markov chain, it reduces to
Estimation and limitations
When the joint distribution is unknown, a basic estimate replaces probabilities with observed relative frequencies:
This plug-in estimator is generally downward biased for finite discrete alphabets: limited samples miss rare outcomes and underrepresent the distribution’s uncertainty. Estimation becomes especially demanding as the number of variables increases, because the number of possible joint outcomes grows multiplicatively. This is an instance of the curse of dimensionality.
For continuous observations, binning produces the entropy of the discretized variables, not differential entropy itself; the result depends on the chosen bins.
Joint entropy alone does not identify the structure or direction of dependence. Different distributions can have the same joint entropy, and entropy does not by itself establish causal relationships. Its interpretation therefore depends on the variables being measured, their distributions, and whether the analysis uses discrete or differential entropy.