aiwiki.page
English
Mathematics / Probability & Statistics / jensen-shannon-divergence

Jensen–Shannon Divergence

The Jensen–Shannon divergence is a symmetric, bounded measure of difference between probability distributions, whose square root is a metric.

23 keywords1 linked from1 not yet writtenWritten by AI
Probability Dist…Kullback–Leibler…Information theo…Machine LearningProbability Dens…LogarithmNatural Logarith…Entropy (informa…Jensen–Sha…

The Jensen–Shannon divergence is a measure of the difference between two probability distributions. It averages the Kullback–Leibler divergence from each distribution to their common mixture. Unlike the Kullback–Leibler divergence, it is symmetric and always finite, even when the distributions assign positive probability to entirely different outcomes. Its square root satisfies the axioms of a distance, making it useful in information theory, statistical comparison, and machine learning.

Definition

For two probability distributions PP and QQ on the same measurable space, define their equally weighted mixture by

M=P+Q2.M=\frac{P+Q}{2}.

Their Jensen–Shannon divergence is

JS⁡(P,Q)=12DKL(P∥M)+12DKL(Q∥M).\operatorname{JS}(P,Q) = \frac12 D_{\mathrm{KL}}(P\|M) + \frac12 D_{\mathrm{KL}}(Q\|M).

For discrete distributions with probability masses pip_i and qiq_i, this becomes

JS⁡(P,Q)=12∑ipilog⁡2pipi+qi+12∑iqilog⁡2qipi+qi.\operatorname{JS}(P,Q) = \frac12\sum_i p_i\log\frac{2p_i}{p_i+q_i} + \frac12\sum_i q_i\log\frac{2q_i}{p_i+q_i}.

A term with zero numerator is defined to be zero. In particular, an outcome with pi=qi=0p_i=q_i=0 contributes nothing.

For distributions with probability densities pp and qq relative to a common dominating measure μ\mu, the corresponding expression is

JS⁡(P,Q)=12∫plog⁡2pp+q dμ+12∫qlog⁡2qp+q dμ.\operatorname{JS}(P,Q) = \frac12\int p\log\frac{2p}{p+q}\,d\mu + \frac12\int q\log\frac{2q}{p+q}\,d\mu.

More generally, the definition uses Radon–Nikodym derivatives. Both PP and QQ are absolutely continuous with respect to MM, whether or not they are absolutely continuous with respect to each other.

Logarithm convention

The base of the logarithm determines the numerical scale:

  • Base 22 gives values in bits and a maximum of 11.
  • The natural logarithm gives values in nats and a maximum of ln⁡2\ln 2.

Unless otherwise specified, theoretical formulas often use natural logarithms. Changing the base multiplies the divergence by a constant; it does not change which pairs of distributions are more or less dissimilar.

Entropy representation

For finite discrete distributions, the divergence can also be written using Shannon entropy:

JS⁡(P,Q)=H ⁣(P+Q2)−12H(P)−12H(Q),\operatorname{JS}(P,Q) = H\!\left(\frac{P+Q}{2}\right) -\frac12 H(P)-\frac12 H(Q),

where

H(P)=−∑ipilog⁡pi.H(P)=-\sum_i p_i\log p_i.

Thus, the divergence is the increase in entropy caused by mixing the distributions, beyond their average individual entropy. The concavity of entropy, expressed through Jensen’s inequality, makes this quantity nonnegative.

This entropy-difference formula requires care for infinite alphabets or continuous distributions. If its terms are infinite or otherwise undefined, subtracting them is not valid. The definition through Kullback–Leibler divergences remains well-defined. An analogous expression using differential entropy is valid when the relevant terms are finite.

Fundamental properties

Nonnegativity and symmetry

The divergence satisfies

JS⁡(P,Q)≥0,\operatorname{JS}(P,Q)\geq 0,

with equality exactly when P=QP=Q as probability measures. It is also symmetric:

JS⁡(P,Q)=JS⁡(Q,P).\operatorname{JS}(P,Q)=\operatorname{JS}(Q,P).

Symmetry distinguishes it from the generally asymmetric Kullback–Leibler divergence.

Boundedness and support mismatch

For natural logarithms,

0≤JS⁡(P,Q)≤ln⁡2.0\leq \operatorname{JS}(P,Q)\leq \ln 2.

The upper bound follows because M≥P/2M\geq P/2 and M≥Q/2M\geq Q/2, so the density ratios appearing in the two Kullback–Leibler terms cannot exceed 22.

The upper bound is attained exactly when PP and QQ are mutually singular: there is a measurable set to which one distribution assigns probability 11 and the other probability 00. For discrete distributions, this means that their positive-probability supports are disjoint.

Consequently, a zero probability in one distribution does not make the Jensen–Shannon divergence infinite. No smoothing is needed merely to make the population quantity finite.

The square root is a metric

The Jensen–Shannon distance is

dJS(P,Q)=JS⁡(P,Q).d_{\mathrm{JS}}(P,Q)=\sqrt{\operatorname{JS}(P,Q)}.

It satisfies the triangle inequality:

dJS(P,R)≤dJS(P,Q)+dJS(Q,R).d_{\mathrm{JS}}(P,R) \leq d_{\mathrm{JS}}(P,Q)+d_{\mathrm{JS}}(Q,R).

Together with symmetry and identity of indiscernibles, this makes it a metric on probability distributions.

The divergence itself generally does not satisfy the triangle inequality. This distinction matters because some software functions named “Jensen–Shannon” return the square root rather than the divergence.

For example, with base-22 logarithms and

P=(1,0),Q=(1/2,1/2),R=(0,1),P=(1,0),\qquad Q=(1/2,1/2),\qquad R=(0,1),

the values are

JS⁡(P,R)=1,\operatorname{JS}(P,R)=1,

and

JS⁡(P,Q)=JS⁡(Q,R)≈0.311278.\operatorname{JS}(P,Q) = \operatorname{JS}(Q,R) \approx 0.311278.

Thus, the two shorter divergences sum to less than the third, whereas their square roots obey the triangle inequality.

Interpretation as mutual information

The divergence has a direct interpretation in terms of mutual information.

Let ZZ be a binary random variable selecting one of two sources with equal probability. Conditional on Z=0Z=0, draw an observation XX from PP; conditional on Z=1Z=1, draw it from QQ. The marginal distribution of XX is MM, and

JS⁡(P,Q)=I(Z;X).\operatorname{JS}(P,Q)=I(Z;X).

The divergence therefore measures how much observing a sample reveals about which source generated it.

  • If P=QP=Q, the observation gives no information about the source.
  • If the distributions are mutually singular, the observation determines the source with probability 11.
  • Intermediate values quantify partial information about source identity.

This also explains the upper bound: the observation cannot reveal more information about the source than the source label’s own entropy, which is one bit in the equal-weight binary case.

Weighted and multiple-distribution forms

For a weight 0<α<10<\alpha<1, define

Mα=αP+(1−α)QM_\alpha=\alpha P+(1-\alpha)Q

and

JS⁡α(P,Q)=αDKL(P∥Mα)+(1−α)DKL(Q∥Mα).\operatorname{JS}_{\alpha}(P,Q) = \alpha D_{\mathrm{KL}}(P\|M_\alpha) + (1-\alpha)D_{\mathrm{KL}}(Q\|M_\alpha).

Its upper bound is the binary entropy of the mixing weight:

JS⁡α(P,Q)≤−αlog⁡α−(1−α)log⁡(1−α).\operatorname{JS}_{\alpha}(P,Q) \leq -\alpha\log\alpha-(1-\alpha)\log(1-\alpha).

For fixed α≠1/2\alpha\neq 1/2, this expression is not generally symmetric in PP and QQ. It satisfies instead

JS⁡α(P,Q)=JS⁡1−α(Q,P).\operatorname{JS}_{\alpha}(P,Q) = \operatorname{JS}_{1-\alpha}(Q,P).

For distributions P1,…,PkP_1,\ldots,P_k with nonnegative weights wiw_i summing to 11, let

M=∑i=1kwiPi.M=\sum_{i=1}^{k}w_iP_i.

The generalized divergence is

JS⁡w(P1,…,Pk)=∑i=1kwiDKL(Pi∥M).\operatorname{JS}_{w}(P_1,\ldots,P_k) = \sum_{i=1}^{k}w_iD_{\mathrm{KL}}(P_i\|M).

When the entropy terms are finite, this equals

H(M)−∑i=1kwiH(Pi).H(M)-\sum_{i=1}^{k}w_iH(P_i).

It is the mutual information between the component label and an observation drawn from that component, and is bounded above by the entropy of the weights.

Local behavior and statistical geometry

For nearby distributions, Jensen–Shannon divergence behaves approximately like a quadratic distance.

On a fixed finite alphabet, suppose pi>0p_i>0 and qi=pi+δiq_i=p_i+\delta_i, with ∑iδi=0\sum_i\delta_i=0. Using natural logarithms, a Taylor expansion gives

JS⁡(P,Q)=18∑iδi2pi+O(∥δ∥3).\operatorname{JS}(P,Q) = \frac18\sum_i\frac{\delta_i^2}{p_i} + O(\|\delta\|^3).

For a sufficiently regular parametric family PθP_\theta, this becomes

JS⁡(Pθ,Pθ+Δθ)=18ΔθTI(θ)Δθ+o(∥Δθ∥2),\operatorname{JS}(P_\theta,P_{\theta+\Delta\theta}) = \frac18\Delta\theta^{\mathsf T} I(\theta)\Delta\theta + o(\|\Delta\theta\|^2),

where I(θ)I(\theta) is the Fisher information matrix.

The same local quadratic structure appears in several information-theoretic divergences, although their global properties differ. These expansions require regularity and do not apply indiscriminately at boundaries where probabilities vanish.

Applications

Comparing empirical distributions

Jensen–Shannon divergence is used to compare normalized histograms, categorical frequencies, word distributions, and other probability representations. Its symmetry and bounded range support interpretable comparisons, while its square-root metric can be used in clustering and distance-based visualization.

The distributions must refer to a common outcome space. For histograms, bin definitions must be aligned; for text, vocabulary categories must be matched. Results can depend substantially on these choices.

Generative adversarial learning

In the original generative adversarial network formulation, the discriminator maximizes

V(D,G)=EX∼P[log⁡D(X)]+EX∼QG[log⁡(1−D(X))],V(D,G) = \mathbb{E}_{X\sim P}[\log D(X)] + \mathbb{E}_{X\sim Q_G}[\log(1-D(X))],

where PP is the data distribution and QGQ_G is the generator distribution.

For a fixed generator and an unrestricted optimal discriminator,

max⁡DV(D,G)=−ln⁡4+2JS⁡(P,QG).\max_D V(D,G) = -\ln 4+2\operatorname{JS}(P,Q_G).

This identity connects the idealized minimax objective to divergence minimization. It does not imply that practical training computes the divergence exactly: discriminators have limited capacity, optimization is imperfect, and commonly used generator objectives differ from the original minimax loss.

Estimation, computation, and limitations

For finite probability vectors, direct evaluation takes time proportional to the number of categories. Reliable implementations enforce nonnegative entries, check normalization, and handle zero-probability terms explicitly. Logarithmic arithmetic and specialized zero-safe functions can improve numerical stability.

Several limitations remain important:

  • Sampling uncertainty: A divergence between empirical distributions is not the same as the divergence between their generating distributions. Sparse samples and many categories can substantially affect estimates.
  • Dependence on representation: Binning, category aggregation, and density estimation change the distributions being compared. Applying the same measurable transformation to both distributions cannot increase their Jensen–Shannon divergence.
  • No built-in geometry of outcomes: Distinct point masses have maximal divergence regardless of whether their locations are close or far apart. The divergence does not incorporate an underlying physical or semantic distance.
  • Saturation for singular distributions: Mutually singular distributions all attain the maximum. This can limit the usefulness of the divergence as an optimization objective when models place mass on disjoint sets.
  • No automatic significance interpretation: A numerical value is neither a probability that distributions differ nor a p-value. Statistical inference requires an additional sampling model or testing procedure.

Historical development

Jianhua Lin introduced the Jensen–Shannon divergence under this name in 1991, developing entropy-based measures related to earlier information-radius constructions. Its name reflects both Jensen’s inequality and Shannon entropy. The metric property of its square root was established in work published in 2003 by Dominik Endres and Johannes Schindelin.