Total variation distance is a measure of discrepancy between two probability distributions on the same measurable space. It equals the largest absolute difference between the probabilities they assign to a single event. Its values range from zero, for identical distributions, to one, for distributions concentrated on disjoint measurable sets. Total variation has equivalent interpretations in terms of density differences, optimal coupling, and the ability to distinguish distributions through statistical testing.
Definition
Let and be probability measures on a measurable space , where is a sigma-algebra of measurable events. Their total variation distance is
Thus, means that every measurable event has probabilities under and differing by at most .
Because , taking complements reverses the sign of the difference:
Consequently,
The absolute-value and one-sided definitions therefore agree for probability measures.
Normalization conventions
In measure theory, the total variation norm of the signed measure is commonly written
where denotes its variation measure. With this convention,
Some texts instead use “total variation norm” or “total variation distance” for a differently normalized quantity. Formulas involving this terminology must therefore be checked for a possible factor of two.
Density and mass-function formulas
Suppose and have densities and with respect to a common dominating measure . Then
Here “density” is understood in the general sense of a Radon–Nikodym derivative. A common dominating measure always exists: is sufficient. The formula therefore applies even when neither distribution has a probability density function with respect to Lebesgue measure.
To see why the factor appears, note that
The positive and negative parts of consequently have equal integrals. The event
attains the supremum, and
An equivalent overlap formula is
The integral of is the probability mass shared by the distributions.
Discrete distributions
For distributions on a finite or countably infinite set , with probability mass functions and ,
For example, let
Then
The event consisting of the first outcome achieves this difference.
For Bernoulli distributions with success probabilities and ,
Metric and expectation properties
Total variation is a metric on the probability measures over a fixed measurable space. In particular,
with exactly when . It is symmetric and satisfies the triangle inequality:
Moreover, exactly when and are mutually singular: there is a measurable set such that and .
Dual characterization
Total variation controls differences in expected values of bounded measurable functions:
Equivalently,
The suprema are over measurable functions satisfying the stated bounds. In particular, if , then
No corresponding bound for arbitrary unbounded functions follows from total variation distance alone.
Contraction under processing
If is a measurable function, let and denote the distributions obtained by applying to observations drawn from and . Then
More generally, if is a Markov kernel—a rule assigning an output distribution to each input—then
Thus, applying the same deterministic transformation or randomized procedure to both distributions cannot increase their total variation distance.
Coupling interpretation
A coupling of and is a pair of random variables , defined jointly, whose marginal distributions are and .
For any coupling for which the equality event is measurable,
Indeed, for every measurable ,
On standard Borel spaces, including finite or countable spaces and Euclidean spaces with their Borel sets, this bound is attainable:
A coupling attaining the minimum is called a maximal coupling. It arranges for and to agree with probability equal to the overlap mass,
This interpretation makes total variation useful for proving distributional approximation bounds: constructing a coupling with a small mismatch probability immediately gives a small total variation distance.
Statistical testing interpretation
Consider statistical hypothesis testing between the two simple hypotheses
A randomized test is a measurable function , where is the probability of choosing after observing . Its type I and type II error probabilities are
The smallest possible sum of errors is
If the hypotheses have equal prior probability, the optimal probability of correct classification is therefore
For densities and , an optimal rule chooses where , with arbitrary treatment of ties. Total variation thus has an operational meaning: it measures the distinguishability of two distributions from a single observation.
Convergence and comparison with other distances
A sequence converges to in total variation when
This is uniform convergence of event probabilities over all measurable events. Equivalently, expectations converge uniformly over all measurable functions taking values in .
On metric spaces equipped with their Borel sigma-algebras, total variation convergence implies weak convergence, but the converse need not hold. For example, point masses satisfy
while
for every . Total variation distinguishes exact locations of point masses without considering how close those locations are.
Similarly, for distributions on the real line, total variation bounds the discrepancy between their cumulative distribution functions:
The left-hand side considers only half-lines, whereas total variation considers every measurable set.
Relation to relative entropy
Kullback–Leibler divergence controls total variation through Pinsker’s inequality:
This form uses natural logarithms. Consequently, convergence of KL divergence to zero implies total variation convergence.
The reverse implication does not generally hold. A distribution can assign very little probability to a region where the other assigns none, producing a small total variation distance but infinite KL divergence.
Product distributions
For finite products,
In particular,
These bounds help quantify how discrepancies can accumulate across independent observations.
Applications and limitations
Markov-chain convergence
For a Markov chain with transition kernel and stationary distribution , a standard measure of convergence is
The mixing time at tolerance is
Coupling methods often bound this quantity by constructing two chains, one started from a specified state and one from stationarity, that eventually meet and then move together. This approach is important in the analysis of Markov chain Monte Carlo.
Distributional approximation
Total variation is widely used when approximating discrete distributions, including approximations of sums of rare-event indicators by a Poisson distribution. A single total variation bound controls the error for every event involving the approximated random quantity.
Its strength can also make it unsuitable for some approximations. A discrete distribution and a distribution having a density with respect to Lebesgue measure are mutually singular, so their total variation distance is one. Thus, a discrete standardized sum can converge weakly to a normal distribution, as in the central limit theorem, without converging in total variation.
Geometry and estimation
Total variation depends on measurable sets, not on a distance between individual outcomes. Moving all probability mass by an arbitrarily small geometric displacement can leave total variation equal to one. Metrics that incorporate the underlying geometry, such as Wasserstein distances, answer a different question.
This distinction also affects empirical approximation. If is a nonatomic probability measure and is the empirical distribution of finitely many observations, then
The finite set of sampled points has empirical probability one but probability zero under . Consequently, convergence of empirical averages or weak convergence of empirical distributions does not by itself imply total variation convergence; additional structure, such as a finite state space or suitable smoothing, is needed.