{
  "id": "total-variation-distance",
  "title": "Total Variation Distance",
  "lang": "en",
  "source_lang": "en",
  "summary": "Total variation distance measures the largest difference between the probabilities that two distributions assign to the same measurable event.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "TV distance",
    "Variational distance"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/total-variation-distance/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/total-variation-distance/",
      "json": "https://aiwiki.page/zh/total-variation-distance.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Total variation distance is a measure of discrepancy between two probability distributions on the same measurable space. It equals the largest absolute difference between the probabilities they assign to a single event. Its values range from zero, for identical distributions, to one, for distributions concentrated on disjoint measurable sets. Total variation has equivalent interpretations in terms of density differences, optimal coupling, and the ability to distinguish distributions through statistical testing."
    },
    {
      "heading": "Definition",
      "text": "Let \\(P\\) and \\(Q\\) be probability measures on a measurable space \\((\\Omega,\\mathcal F)\\), where \\(\\mathcal F\\) is a sigma-algebra of measurable events. Their total variation distance is\n\n\\[d_{\\mathrm{TV}}(P,Q) = \\sup_{A\\in\\mathcal F}|P(A)-Q(A)|.\\]\n\nThus, \\(d_{\\mathrm{TV}}(P,Q)\\leq\\varepsilon\\) means that every measurable event has probabilities under \\(P\\) and \\(Q\\) differing by at most \\(\\varepsilon\\).\n\nBecause \\(P(\\Omega)=Q(\\Omega)=1\\), taking complements reverses the sign of the difference:\n\n\\[P(A^c)-Q(A^c)=-(P(A)-Q(A)).\\]\n\nConsequently,\n\n\\[d_{\\mathrm{TV}}(P,Q) = \\sup_{A\\in\\mathcal F}\\bigl(P(A)-Q(A)\\bigr).\\]\n\nThe absolute-value and one-sided definitions therefore agree for probability measures.\n\nNormalization conventions\n\nIn measure theory, the total variation norm of the signed measure \\(P-Q\\) is commonly written\n\n\\[\\|P-Q\\|_{\\mathrm{var}} = |P-Q|(\\Omega),\\]\n\nwhere \\(|P-Q|\\) denotes its variation measure. With this convention,\n\n\\[d_{\\mathrm{TV}}(P,Q) = \\frac12\\|P-Q\\|_{\\mathrm{var}}.\\]\n\nSome texts instead use “total variation norm” or “total variation distance” for a differently normalized quantity. Formulas involving this terminology must therefore be checked for a possible factor of two."
    },
    {
      "heading": "Density and mass-function formulas",
      "text": "Suppose \\(P\\) and \\(Q\\) have densities \\(p\\) and \\(q\\) with respect to a common dominating measure \\(\\mu\\). Then\n\n\\[d_{\\mathrm{TV}}(P,Q) = \\frac12\\int_\\Omega |p-q|\\,d\\mu.\\]\n\nHere “density” is understood in the general sense of a Radon–Nikodym derivative. A common dominating measure always exists: \\(\\mu=P+Q\\) is sufficient. The formula therefore applies even when neither distribution has a probability density function with respect to Lebesgue measure.\n\nTo see why the factor \\(1/2\\) appears, note that\n\n\\[\\int_\\Omega(p-q)\\,d\\mu=0.\\]\n\nThe positive and negative parts of \\(p-q\\) consequently have equal integrals. The event\n\n\\[A_*=\\{x:p(x)>q(x)\\}\\]\n\nattains the supremum, and\n\n\\[P(A_*)-Q(A_*) = \\int_\\Omega(p-q)_+\\,d\\mu = \\frac12\\int_\\Omega|p-q|\\,d\\mu.\\]\n\nAn equivalent overlap formula is\n\n\\[d_{\\mathrm{TV}}(P,Q) = 1-\\int_\\Omega\\min(p,q)\\,d\\mu.\\]\n\nThe integral of \\(\\min(p,q)\\) is the probability mass shared by the distributions.\n\nDiscrete distributions\n\nFor distributions on a finite or countably infinite set \\(S\\), with probability mass functions \\(p(x)\\) and \\(q(x)\\),\n\n\\[d_{\\mathrm{TV}}(P,Q) = \\frac12\\sum_{x\\in S}|p(x)-q(x)|.\\]\n\nFor example, let\n\n\\[P=(0.5,0.3,0.2), \\qquad Q=(0.2,0.4,0.4).\\]\n\nThen\n\n\\[d_{\\mathrm{TV}}(P,Q) = \\frac12(0.3+0.1+0.2) = 0.3.\\]\n\nThe event consisting of the first outcome achieves this difference.\n\nFor Bernoulli distributions with success probabilities \\(p\\) and \\(q\\),\n\n\\[d_{\\mathrm{TV}}\\bigl(\\operatorname{Bern}(p), \\operatorname{Bern}(q)\\bigr) = |p-q|.\\]"
    },
    {
      "heading": "Metric and expectation properties",
      "text": "Total variation is a metric on the probability measures over a fixed measurable space. In particular,\n\n\\[0\\leq d_{\\mathrm{TV}}(P,Q)\\leq1,\\]\n\nwith \\(d_{\\mathrm{TV}}(P,Q)=0\\) exactly when \\(P=Q\\). It is symmetric and satisfies the triangle inequality:\n\n\\[d_{\\mathrm{TV}}(P,R) \\leq d_{\\mathrm{TV}}(P,Q)+d_{\\mathrm{TV}}(Q,R).\\]\n\nMoreover, \\(d_{\\mathrm{TV}}(P,Q)=1\\) exactly when \\(P\\) and \\(Q\\) are mutually singular: there is a measurable set \\(A\\) such that \\(P(A)=1\\) and \\(Q(A)=0\\).\n\nDual characterization\n\nTotal variation controls differences in expected values of bounded measurable functions:\n\n\\[d_{\\mathrm{TV}}(P,Q) = \\sup_{0\\leq f\\leq1} \\left| \\int f\\,dP-\\int f\\,dQ \\right|.\\]\n\nEquivalently,\n\n\\[d_{\\mathrm{TV}}(P,Q) = \\frac12 \\sup_{\\|f\\|_\\infty\\leq1} \\left| \\int f\\,dP-\\int f\\,dQ \\right|.\\]\n\nThe suprema are over measurable functions satisfying the stated bounds. In particular, if \\(a\\leq f\\leq b\\), then\n\n\\[|\\mathbb E_P f-\\mathbb E_Q f| \\leq (b-a)d_{\\mathrm{TV}}(P,Q).\\]\n\nNo corresponding bound for arbitrary unbounded functions follows from total variation distance alone.\n\nContraction under processing\n\nIf \\(T\\) is a measurable function, let \\(P_T\\) and \\(Q_T\\) denote the distributions obtained by applying \\(T\\) to observations drawn from \\(P\\) and \\(Q\\). Then\n\n\\[d_{\\mathrm{TV}}(P_T,Q_T) \\leq d_{\\mathrm{TV}}(P,Q).\\]\n\nMore generally, if \\(K\\) is a Markov kernel—a rule assigning an output distribution to each input—then\n\n\\[d_{\\mathrm{TV}}(PK,QK) \\leq d_{\\mathrm{TV}}(P,Q).\\]\n\nThus, applying the same deterministic transformation or randomized procedure to both distributions cannot increase their total variation distance."
    },
    {
      "heading": "Coupling interpretation",
      "text": "A coupling of \\(P\\) and \\(Q\\) is a pair of random variables \\((X,Y)\\), defined jointly, whose marginal distributions are \\(P\\) and \\(Q\\).\n\nFor any coupling for which the equality event is measurable,\n\n\\[d_{\\mathrm{TV}}(P,Q) \\leq \\Pr(X\\ne Y).\\]\n\nIndeed, for every measurable \\(A\\),\n\n\\[|P(A)-Q(A)| = |\\Pr(X\\in A)-\\Pr(Y\\in A)| \\leq \\Pr(X\\ne Y).\\]\n\nOn standard Borel spaces, including finite or countable spaces and Euclidean spaces with their Borel sets, this bound is attainable:\n\n\\[d_{\\mathrm{TV}}(P,Q) = \\min_{\\text{couplings }(X,Y)} \\Pr(X\\ne Y).\\]\n\nA coupling attaining the minimum is called a maximal coupling. It arranges for \\(X\\) and \\(Y\\) to agree with probability equal to the overlap mass,\n\n\\[\\Pr(X=Y)=\\int\\min(p,q)\\,d\\mu.\\]\n\nThis interpretation makes total variation useful for proving distributional approximation bounds: constructing a coupling with a small mismatch probability immediately gives a small total variation distance."
    },
    {
      "heading": "Statistical testing interpretation",
      "text": "Consider statistical hypothesis testing between the two simple hypotheses\n\n\\[H_0:Z\\sim P, \\qquad H_1:Z\\sim Q.\\]\n\nA randomized test is a measurable function \\(\\varphi:\\Omega\\to[0,1]\\), where \\(\\varphi(z)\\) is the probability of choosing \\(H_1\\) after observing \\(z\\). Its type I and type II error probabilities are\n\n\\[\\alpha=\\mathbb E_P[\\varphi], \\qquad \\beta=\\mathbb E_Q[1-\\varphi].\\]\n\nThe smallest possible sum of errors is\n\n\\[\\inf_\\varphi(\\alpha+\\beta) = 1-d_{\\mathrm{TV}}(P,Q).\\]\n\nIf the hypotheses have equal prior probability, the optimal probability of correct classification is therefore\n\n\\[\\frac{1+d_{\\mathrm{TV}}(P,Q)}{2}.\\]\n\nFor densities \\(p\\) and \\(q\\), an optimal rule chooses \\(H_1\\) where \\(q>p\\), with arbitrary treatment of ties. Total variation thus has an operational meaning: it measures the distinguishability of two distributions from a single observation."
    },
    {
      "heading": "Convergence and comparison with other distances",
      "text": "A sequence \\(P_n\\) converges to \\(P\\) in total variation when\n\n\\[d_{\\mathrm{TV}}(P_n,P)\\longrightarrow0.\\]\n\nThis is uniform convergence of event probabilities over all measurable events. Equivalently, expectations converge uniformly over all measurable functions taking values in \\([0,1]\\).\n\nOn metric spaces equipped with their Borel sigma-algebras, total variation convergence implies weak convergence, but the converse need not hold. For example, point masses satisfy\n\n\\[\\delta_{1/n}\\Rightarrow\\delta_0,\\]\n\nwhile\n\n\\[d_{\\mathrm{TV}}(\\delta_{1/n},\\delta_0)=1\\]\n\nfor every \\(n\\). Total variation distinguishes exact locations of point masses without considering how close those locations are.\n\nSimilarly, for distributions on the real line, total variation bounds the discrepancy between their cumulative distribution functions:\n\n\\[\\sup_{t\\in\\mathbb R}|F_P(t)-F_Q(t)| \\leq d_{\\mathrm{TV}}(P,Q).\\]\n\nThe left-hand side considers only half-lines, whereas total variation considers every measurable set.\n\nRelation to relative entropy\n\nKullback–Leibler divergence controls total variation through Pinsker’s inequality:\n\n\\[d_{\\mathrm{TV}}(P,Q) \\leq \\sqrt{\\frac12D_{\\mathrm{KL}}(P\\|Q)}.\\]\n\nThis form uses natural logarithms. Consequently, convergence of KL divergence to zero implies total variation convergence.\n\nThe reverse implication does not generally hold. A distribution can assign very little probability to a region where the other assigns none, producing a small total variation distance but infinite KL divergence.\n\nProduct distributions\n\nFor finite products,\n\n\\[d_{\\mathrm{TV}} \\left( \\bigotimes_{i=1}^{n}P_i, \\bigotimes_{i=1}^{n}Q_i \\right) \\leq 1-\\prod_{i=1}^{n} \\left(1-d_{\\mathrm{TV}}(P_i,Q_i)\\right) \\leq \\sum_{i=1}^{n}d_{\\mathrm{TV}}(P_i,Q_i).\\]\n\nIn particular,\n\n\\[d_{\\mathrm{TV}}(P^{\\otimes n},Q^{\\otimes n}) \\leq \\min\\{1,n\\,d_{\\mathrm{TV}}(P,Q)\\}.\\]\n\nThese bounds help quantify how discrepancies can accumulate across independent observations."
    },
    {
      "heading": "Applications and limitations",
      "text": "Markov-chain convergence\n\nFor a Markov chain with transition kernel \\(K\\) and stationary distribution \\(\\pi\\), a standard measure of convergence is\n\n\\[d(t)= \\sup_x d_{\\mathrm{TV}}\\bigl(K^t(x,\\cdot),\\pi\\bigr).\\]\n\nThe mixing time at tolerance \\(\\varepsilon\\) is\n\n\\[t_{\\mathrm{mix}}(\\varepsilon) = \\inf\\{t:d(t)\\leq\\varepsilon\\}.\\]\n\nCoupling methods often bound this quantity by constructing two chains, one started from a specified state and one from stationarity, that eventually meet and then move together. This approach is important in the analysis of Markov chain Monte Carlo.\n\nDistributional approximation\n\nTotal variation is widely used when approximating discrete distributions, including approximations of sums of rare-event indicators by a Poisson distribution. A single total variation bound controls the error for every event involving the approximated random quantity.\n\nIts strength can also make it unsuitable for some approximations. A discrete distribution and a distribution having a density with respect to Lebesgue measure are mutually singular, so their total variation distance is one. Thus, a discrete standardized sum can converge weakly to a normal distribution, as in the central limit theorem, without converging in total variation.\n\nGeometry and estimation\n\nTotal variation depends on measurable sets, not on a distance between individual outcomes. Moving all probability mass by an arbitrarily small geometric displacement can leave total variation equal to one. Metrics that incorporate the underlying geometry, such as Wasserstein distances, answer a different question.\n\nThis distinction also affects empirical approximation. If \\(P\\) is a nonatomic probability measure and \\(\\widehat P_n\\) is the empirical distribution of finitely many observations, then\n\n\\[d_{\\mathrm{TV}}(\\widehat P_n,P)=1.\\]\n\nThe finite set of sampled points has empirical probability one but probability zero under \\(P\\). Consequently, convergence of empirical averages or weak convergence of empirical distributions does not by itself imply total variation convergence; additional structure, such as a finite state space or suitable smoothing, is needed."
    }
  ],
  "links": [
    "probability-distribution",
    "measurable-space",
    "sigma-algebra",
    "measure-theory",
    "probability-density-function",
    "probability-mass-function",
    "bernoulli-distribution",
    "metric-space",
    "triangle-inequality",
    "expected-value",
    "measurable-function",
    "random-variable",
    "statistical-hypothesis-testing",
    "type-i-and-type-ii-errors",
    "cumulative-distribution-function",
    "kullback-leibler-divergence",
    "markov-chain",
    "stationary-distribution",
    "markov-chain-monte-carlo",
    "poisson-distribution",
    "normal-distribution",
    "central-limit-theorem"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}