{
  "id": "jensen-shannon-divergence",
  "title": "Jensen–Shannon Divergence",
  "lang": "en",
  "source_lang": "en",
  "summary": "The Jensen–Shannon divergence is a symmetric, bounded measure of difference between probability distributions, whose square root is a metric.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "Jensen–Shannon information radius",
    "JSD",
    "JS divergence"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/jensen-shannon-divergence/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/jensen-shannon-divergence/",
      "json": "https://aiwiki.page/zh/jensen-shannon-divergence.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The Jensen–Shannon divergence is a measure of the difference between two probability distributions. It averages the Kullback–Leibler divergence from each distribution to their common mixture. Unlike the Kullback–Leibler divergence, it is symmetric and always finite, even when the distributions assign positive probability to entirely different outcomes. Its square root satisfies the axioms of a distance, making it useful in information theory, statistical comparison, and machine learning."
    },
    {
      "heading": "Definition",
      "text": "For two probability distributions \\(P\\) and \\(Q\\) on the same measurable space, define their equally weighted mixture by\n\n\\[M=\\frac{P+Q}{2}.\\]\n\nTheir Jensen–Shannon divergence is\n\n\\[\\operatorname{JS}(P,Q) = \\frac12 D_{\\mathrm{KL}}(P\\|M) + \\frac12 D_{\\mathrm{KL}}(Q\\|M).\\]\n\nFor discrete distributions with probability masses \\(p_i\\) and \\(q_i\\), this becomes\n\n\\[\\operatorname{JS}(P,Q) = \\frac12\\sum_i p_i\\log\\frac{2p_i}{p_i+q_i} + \\frac12\\sum_i q_i\\log\\frac{2q_i}{p_i+q_i}.\\]\n\nA term with zero numerator is defined to be zero. In particular, an outcome with \\(p_i=q_i=0\\) contributes nothing.\n\nFor distributions with probability densities \\(p\\) and \\(q\\) relative to a common dominating measure \\(\\mu\\), the corresponding expression is\n\n\\[\\operatorname{JS}(P,Q) = \\frac12\\int p\\log\\frac{2p}{p+q}\\,d\\mu + \\frac12\\int q\\log\\frac{2q}{p+q}\\,d\\mu.\\]\n\nMore generally, the definition uses Radon–Nikodym derivatives. Both \\(P\\) and \\(Q\\) are absolutely continuous with respect to \\(M\\), whether or not they are absolutely continuous with respect to each other.\n\nLogarithm convention\n\nThe base of the logarithm determines the numerical scale:\n\n- Base \\(2\\) gives values in bits and a maximum of \\(1\\).\n- The natural logarithm gives values in nats and a maximum of \\(\\ln 2\\).\n\nUnless otherwise specified, theoretical formulas often use natural logarithms. Changing the base multiplies the divergence by a constant; it does not change which pairs of distributions are more or less dissimilar."
    },
    {
      "heading": "Entropy representation",
      "text": "For finite discrete distributions, the divergence can also be written using Shannon entropy:\n\n\\[\\operatorname{JS}(P,Q) = H\\!\\left(\\frac{P+Q}{2}\\right) -\\frac12 H(P)-\\frac12 H(Q),\\]\n\nwhere\n\n\\[H(P)=-\\sum_i p_i\\log p_i.\\]\n\nThus, the divergence is the increase in entropy caused by mixing the distributions, beyond their average individual entropy. The concavity of entropy, expressed through Jensen’s inequality, makes this quantity nonnegative.\n\nThis entropy-difference formula requires care for infinite alphabets or continuous distributions. If its terms are infinite or otherwise undefined, subtracting them is not valid. The definition through Kullback–Leibler divergences remains well-defined. An analogous expression using differential entropy is valid when the relevant terms are finite."
    },
    {
      "heading": "Fundamental properties",
      "text": "Nonnegativity and symmetry\n\nThe divergence satisfies\n\n\\[\\operatorname{JS}(P,Q)\\geq 0,\\]\n\nwith equality exactly when \\(P=Q\\) as probability measures. It is also symmetric:\n\n\\[\\operatorname{JS}(P,Q)=\\operatorname{JS}(Q,P).\\]\n\nSymmetry distinguishes it from the generally asymmetric Kullback–Leibler divergence.\n\nBoundedness and support mismatch\n\nFor natural logarithms,\n\n\\[0\\leq \\operatorname{JS}(P,Q)\\leq \\ln 2.\\]\n\nThe upper bound follows because \\(M\\geq P/2\\) and \\(M\\geq Q/2\\), so the density ratios appearing in the two Kullback–Leibler terms cannot exceed \\(2\\).\n\nThe upper bound is attained exactly when \\(P\\) and \\(Q\\) are mutually singular: there is a measurable set to which one distribution assigns probability \\(1\\) and the other probability \\(0\\). For discrete distributions, this means that their positive-probability supports are disjoint.\n\nConsequently, a zero probability in one distribution does not make the Jensen–Shannon divergence infinite. No smoothing is needed merely to make the population quantity finite.\n\nThe square root is a metric\n\nThe Jensen–Shannon distance is\n\n\\[d_{\\mathrm{JS}}(P,Q)=\\sqrt{\\operatorname{JS}(P,Q)}.\\]\n\nIt satisfies the triangle inequality:\n\n\\[d_{\\mathrm{JS}}(P,R) \\leq d_{\\mathrm{JS}}(P,Q)+d_{\\mathrm{JS}}(Q,R).\\]\n\nTogether with symmetry and identity of indiscernibles, this makes it a metric on probability distributions.\n\nThe divergence itself generally does not satisfy the triangle inequality. This distinction matters because some software functions named “Jensen–Shannon” return the square root rather than the divergence.\n\nFor example, with base-\\(2\\) logarithms and\n\n\\[P=(1,0),\\qquad Q=(1/2,1/2),\\qquad R=(0,1),\\]\n\nthe values are\n\n\\[\\operatorname{JS}(P,R)=1,\\]\n\nand\n\n\\[\\operatorname{JS}(P,Q) = \\operatorname{JS}(Q,R) \\approx 0.311278.\\]\n\nThus, the two shorter divergences sum to less than the third, whereas their square roots obey the triangle inequality."
    },
    {
      "heading": "Interpretation as mutual information",
      "text": "The divergence has a direct interpretation in terms of mutual information.\n\nLet \\(Z\\) be a binary random variable selecting one of two sources with equal probability. Conditional on \\(Z=0\\), draw an observation \\(X\\) from \\(P\\); conditional on \\(Z=1\\), draw it from \\(Q\\). The marginal distribution of \\(X\\) is \\(M\\), and\n\n\\[\\operatorname{JS}(P,Q)=I(Z;X).\\]\n\nThe divergence therefore measures how much observing a sample reveals about which source generated it.\n\n- If \\(P=Q\\), the observation gives no information about the source.\n- If the distributions are mutually singular, the observation determines the source with probability \\(1\\).\n- Intermediate values quantify partial information about source identity.\n\nThis also explains the upper bound: the observation cannot reveal more information about the source than the source label’s own entropy, which is one bit in the equal-weight binary case."
    },
    {
      "heading": "Weighted and multiple-distribution forms",
      "text": "For a weight \\(0<\\alpha<1\\), define\n\n\\[M_\\alpha=\\alpha P+(1-\\alpha)Q\\]\n\nand\n\n\\[\\operatorname{JS}_{\\alpha}(P,Q) = \\alpha D_{\\mathrm{KL}}(P\\|M_\\alpha) + (1-\\alpha)D_{\\mathrm{KL}}(Q\\|M_\\alpha).\\]\n\nIts upper bound is the binary entropy of the mixing weight:\n\n\\[\\operatorname{JS}_{\\alpha}(P,Q) \\leq -\\alpha\\log\\alpha-(1-\\alpha)\\log(1-\\alpha).\\]\n\nFor fixed \\(\\alpha\\neq 1/2\\), this expression is not generally symmetric in \\(P\\) and \\(Q\\). It satisfies instead\n\n\\[\\operatorname{JS}_{\\alpha}(P,Q) = \\operatorname{JS}_{1-\\alpha}(Q,P).\\]\n\nFor distributions \\(P_1,\\ldots,P_k\\) with nonnegative weights \\(w_i\\) summing to \\(1\\), let\n\n\\[M=\\sum_{i=1}^{k}w_iP_i.\\]\n\nThe generalized divergence is\n\n\\[\\operatorname{JS}_{w}(P_1,\\ldots,P_k) = \\sum_{i=1}^{k}w_iD_{\\mathrm{KL}}(P_i\\|M).\\]\n\nWhen the entropy terms are finite, this equals\n\n\\[H(M)-\\sum_{i=1}^{k}w_iH(P_i).\\]\n\nIt is the mutual information between the component label and an observation drawn from that component, and is bounded above by the entropy of the weights."
    },
    {
      "heading": "Local behavior and statistical geometry",
      "text": "For nearby distributions, Jensen–Shannon divergence behaves approximately like a quadratic distance.\n\nOn a fixed finite alphabet, suppose \\(p_i>0\\) and \\(q_i=p_i+\\delta_i\\), with \\(\\sum_i\\delta_i=0\\). Using natural logarithms, a Taylor expansion gives\n\n\\[\\operatorname{JS}(P,Q) = \\frac18\\sum_i\\frac{\\delta_i^2}{p_i} + O(\\|\\delta\\|^3).\\]\n\nFor a sufficiently regular parametric family \\(P_\\theta\\), this becomes\n\n\\[\\operatorname{JS}(P_\\theta,P_{\\theta+\\Delta\\theta}) = \\frac18\\Delta\\theta^{\\mathsf T} I(\\theta)\\Delta\\theta + o(\\|\\Delta\\theta\\|^2),\\]\n\nwhere \\(I(\\theta)\\) is the Fisher information matrix.\n\nThe same local quadratic structure appears in several information-theoretic divergences, although their global properties differ. These expansions require regularity and do not apply indiscriminately at boundaries where probabilities vanish."
    },
    {
      "heading": "Applications",
      "text": "Comparing empirical distributions\n\nJensen–Shannon divergence is used to compare normalized histograms, categorical frequencies, word distributions, and other probability representations. Its symmetry and bounded range support interpretable comparisons, while its square-root metric can be used in clustering and distance-based visualization.\n\nThe distributions must refer to a common outcome space. For histograms, bin definitions must be aligned; for text, vocabulary categories must be matched. Results can depend substantially on these choices.\n\nGenerative adversarial learning\n\nIn the original generative adversarial network formulation, the discriminator maximizes\n\n\\[V(D,G) = \\mathbb{E}_{X\\sim P}[\\log D(X)] + \\mathbb{E}_{X\\sim Q_G}[\\log(1-D(X))],\\]\n\nwhere \\(P\\) is the data distribution and \\(Q_G\\) is the generator distribution.\n\nFor a fixed generator and an unrestricted optimal discriminator,\n\n\\[\\max_D V(D,G) = -\\ln 4+2\\operatorname{JS}(P,Q_G).\\]\n\nThis identity connects the idealized minimax objective to divergence minimization. It does not imply that practical training computes the divergence exactly: discriminators have limited capacity, optimization is imperfect, and commonly used generator objectives differ from the original minimax loss."
    },
    {
      "heading": "Estimation, computation, and limitations",
      "text": "For finite probability vectors, direct evaluation takes time proportional to the number of categories. Reliable implementations enforce nonnegative entries, check normalization, and handle zero-probability terms explicitly. Logarithmic arithmetic and specialized zero-safe functions can improve numerical stability.\n\nSeveral limitations remain important:\n\n- Sampling uncertainty: A divergence between empirical distributions is not the same as the divergence between their generating distributions. Sparse samples and many categories can substantially affect estimates.\n- Dependence on representation: Binning, category aggregation, and density estimation change the distributions being compared. Applying the same measurable transformation to both distributions cannot increase their Jensen–Shannon divergence.\n- No built-in geometry of outcomes: Distinct point masses have maximal divergence regardless of whether their locations are close or far apart. The divergence does not incorporate an underlying physical or semantic distance.\n- Saturation for singular distributions: Mutually singular distributions all attain the maximum. This can limit the usefulness of the divergence as an optimization objective when models place mass on disjoint sets.\n- No automatic significance interpretation: A numerical value is neither a probability that distributions differ nor a p-value. Statistical inference requires an additional sampling model or testing procedure."
    },
    {
      "heading": "Historical development",
      "text": "Jianhua Lin introduced the Jensen–Shannon divergence under this name in 1991, developing entropy-based measures related to earlier information-radius constructions. Its name reflects both Jensen’s inequality and Shannon entropy. The metric property of its square root was established in work published in 2003 by Dominik Endres and Johannes Schindelin."
    }
  ],
  "links": [
    "probability-distribution",
    "kullback-leibler-divergence",
    "information-theory",
    "machine-learning",
    "probability-density-function",
    "logarithm",
    "natural-logarithm",
    "information-entropy",
    "jensens-inequality",
    "differential-entropy",
    "triangle-inequality",
    "metric-space",
    "mutual-information",
    "random-variable",
    "taylor-series",
    "fisher-information",
    "cluster-analysis",
    "generative-adversarial-network",
    "numerical-stability",
    "density-estimation",
    "objective-function",
    "p-value"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}