{
  "id": "joint-entropy",
  "title": "Joint entropy",
  "lang": "en",
  "source_lang": "en",
  "summary": "Joint entropy measures the uncertainty in several random variables considered together, accounting for their statistical dependence.",
  "category": "mathematics",
  "subcategory": "applied",
  "type": "concept",
  "aliases": [
    "Joint Shannon entropy"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/joint-entropy/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/joint-entropy/",
      "json": "https://aiwiki.page/zh/joint-entropy.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Joint entropy is the entropy of two or more random variables considered as a single random object. In information theory, it measures the average uncertainty about their combined outcome. Joint entropy depends on the variables’ full joint probability distribution, not just their individual distributions: independent variables contribute separate uncertainty, while dependence can make their combined outcome more predictable."
    },
    {
      "heading": "Definition",
      "text": "For discrete random variables \\(X\\) and \\(Y\\), with joint probability mass function\n\n\\[p(x,y)=\\Pr(X=x,Y=y),\\]\n\ntheir joint entropy is\n\n\\[H(X,Y) = -\\sum_x\\sum_y p(x,y)\\log_b p(x,y).\\]\n\nThe sum ranges over all possible pairs of values, with the convention \\(0\\log_b 0=0\\). The logarithm base \\(b>1\\) determines the unit:\n\n- Base \\(2\\) gives bits.\n- Base \\(e\\) gives nats.\n- Base \\(10\\) gives hartleys.\n\nUnless otherwise specified, information-theoretic calculations commonly use base \\(2\\).\n\nEquivalently,\n\n\\[H(X,Y) = \\mathbb{E}\\!\\left[-\\log_b p(X,Y)\\right].\\]\n\nThus, joint entropy is the expected value of the self-information of the observed pair. It is a property of the distribution, rather than of any particular realized pair.\n\nFor \\(n\\) discrete variables,\n\n\\[H(X_1,\\ldots,X_n) = -\\sum_{x_1,\\ldots,x_n} p(x_1,\\ldots,x_n) \\log_b p(x_1,\\ldots,x_n).\\]\n\nThese definitions apply to finite or countably infinite alphabets. For countably infinite alphabets, joint entropy may be infinite; identities involving subtraction require care in that case."
    },
    {
      "heading": "Conditional entropy and the chain rule",
      "text": "Conditional entropy measures the uncertainty remaining about one variable when another is known:\n\n\\[H(Y\\mid X) = -\\sum_{x,y}p(x,y)\\log_b p(y\\mid x).\\]\n\nValues of \\(x\\) with zero probability contribute nothing, so the conditional probability only needs to be specified where \\(p(x)>0\\).\n\nThe factorization\n\n\\[p(x,y)=p(x)p(y\\mid x)\\]\n\ngives the entropy chain rule:\n\n\\[H(X,Y)=H(X)+H(Y\\mid X).\\]\n\nLikewise,\n\n\\[H(X,Y)=H(Y)+H(X\\mid Y).\\]\n\nThis expresses joint uncertainty as the uncertainty about one variable plus the uncertainty remaining about the other after the first is known. Although the two decompositions use different conditional entropies, joint entropy itself is symmetric:\n\n\\[H(X,Y)=H(Y,X).\\]\n\nFor multiple variables, the chain rule becomes\n\n\\[H(X_1,\\ldots,X_n) = \\sum_{i=1}^{n} H(X_i\\mid X_1,\\ldots,X_{i-1}),\\]\n\nwhere the first term is \\(H(X_1)\\). Any ordering gives the same total joint entropy, although individual conditional terms may change."
    },
    {
      "heading": "Bounds and equality conditions",
      "text": "For discrete variables with finite entropies,\n\n\\[\\max\\{H(X),H(Y)\\} \\leq H(X,Y) \\leq H(X)+H(Y).\\]\n\nThe lower bound follows because discrete conditional entropy is nonnegative. Learning the pair cannot involve less uncertainty than learning either component alone.\n\nThe upper bound is called subadditivity. Equality holds exactly when \\(X\\) and \\(Y\\) exhibit statistical independence:\n\n\\[H(X,Y)=H(X)+H(Y) \\quad\\Longleftrightarrow\\quad p(x,y)=p(x)p(y).\\]\n\nAt the other extreme,\n\n\\[H(X,Y)=H(X)\\]\n\nexactly when \\(H(Y\\mid X)=0\\). For discrete variables, this means that \\(Y\\) is a deterministic function of \\(X\\) with probability one. In particular,\n\n\\[H(X,X)=H(X).\\]\n\nRepeating a variable adds no uncertainty.\n\nIf the joint distribution has exactly \\(M\\) possible outcomes with positive probability, then\n\n\\[H(X,Y)\\leq \\log_b M.\\]\n\nEquality holds precisely when those outcomes are equally likely. Consequently, if \\(X\\) has at most \\(m\\) values and \\(Y\\) at most \\(n\\) values,\n\n\\[H(X,Y)\\leq \\log_b(mn).\\]\n\nJoint entropy is also a concave function of the joint probability mass function: mixing distributions cannot produce an entropy smaller than the corresponding weighted average of their entropies."
    },
    {
      "heading": "Relationship to mutual information",
      "text": "Mutual information quantifies the departure of joint entropy from the sum of the marginal entropies:\n\n\\[I(X;Y)=H(X)+H(Y)-H(X,Y).\\]\n\nEquivalently,\n\n\\[I(X;Y)=H(X)-H(X\\mid Y).\\]\n\nThus, joint entropy measures total uncertainty in the pair, while mutual information measures how much knowing one variable reduces uncertainty about the other. The quantities answer different questions: two independent, highly unpredictable variables can have large joint entropy but zero mutual information.\n\nMutual information can also be written using Kullback–Leibler divergence:\n\n\\[I(X;Y) = D_{\\mathrm{KL}}\\!\\left(p_{XY}\\,\\middle\\|\\,p_Xp_Y\\right).\\]\n\nIts nonnegativity yields subadditivity of joint entropy.\n\nFor more than two variables, the analogous difference\n\n\\[\\sum_{i=1}^{n}H(X_i)-H(X_1,\\ldots,X_n)\\]\n\nis called total correlation. It is nonnegative and, for finite entropies, vanishes exactly when the variables are mutually independent."
    },
    {
      "heading": "Examples",
      "text": "Two independent fair bits\n\nLet \\(X\\) and \\(Y\\) be independent binary variables, each equally likely to be \\(0\\) or \\(1\\). All four pairs have probability \\(1/4\\), so\n\n\\[H(X,Y) = -4\\left(\\frac14\\log_2\\frac14\\right) = 2\\text{ bits}.\\]\n\nEach variable contributes one bit of uncertainty.\n\nTwo identical fair bits\n\nIf \\(X\\) is a fair bit and \\(Y=X\\), only \\((0,0)\\) and \\((1,1)\\) occur, each with probability \\(1/2\\). Hence\n\n\\[H(X,Y)=1\\text{ bit}.\\]\n\nBoth marginal entropies are still one bit, but the second variable adds no uncertainty once the first is known.\n\nA noisy copy\n\nLet \\(X\\) be a fair bit, let \\(N\\) be an independent binary noise variable with \\(\\Pr(N=1)=q\\), and define\n\n\\[Y=X\\oplus N,\\]\n\nwhere \\(\\oplus\\) denotes exclusive OR. Given \\(X\\), the uncertainty about \\(Y\\) is exactly the uncertainty about the noise:\n\n\\[H(X,Y)=1+h_2(q),\\]\n\nwhere\n\n\\[h_2(q) = -q\\log_2 q-(1-q)\\log_2(1-q)\\]\n\nis the binary entropy function.\n\nAt \\(q=0\\), the variables are identical; at \\(q=1\\), one is the deterministic complement of the other. Both cases give one bit of joint entropy. At \\(q=1/2\\), the variables are independent and the joint entropy is two bits.\n\nPairwise independence is not enough\n\nLet \\(X\\) and \\(Y\\) be independent fair bits and set\n\n\\[Z=X\\oplus Y.\\]\n\nEvery pair among \\(X,Y,Z\\) is independent, but the three variables are not mutually independent: any two determine the third. Therefore,\n\n\\[H(X,Y,Z)=2\\text{ bits},\\]\n\nalthough\n\n\\[H(X)+H(Y)+H(Z)=3\\text{ bits}.\\]\n\nThis illustrates why pairwise dependence measures do not fully describe multivariable uncertainty."
    },
    {
      "heading": "Continuous variables",
      "text": "For jointly continuous variables with probability density function \\(f_{XY}\\), the corresponding quantity is joint differential entropy:\n\n\\[h(X,Y) = -\\int_{\\mathbb R^2} f_{XY}(x,y)\\log_b f_{XY}(x,y)\\,dx\\,dy,\\]\n\nwhen the integral is well defined.\n\nDifferential entropy differs fundamentally from discrete entropy:\n\n- It can be negative.\n- It changes under rescaling and other coordinate transformations.\n- Joint differential entropy need not be at least as large as either marginal differential entropy.\n\nChain-rule and mutual-information identities continue to hold under suitable finiteness conditions, but discrete entropy bounds cannot all be transferred unchanged.\n\nFor a \\(d\\)-dimensional multivariate normal distribution with positive-definite covariance matrix \\(\\Sigma\\),\n\n\\[h(X_1,\\ldots,X_d) = \\frac12 \\log_b\\!\\left((2\\pi e)^d\\det\\Sigma\\right).\\]\n\nThe determinant reflects the spread of the distribution across all dimensions. If \\(\\Sigma\\) is singular, the distribution has no density with respect to \\(d\\)-dimensional Lebesgue measure, so the usual full-dimensional differential-entropy formula does not apply."
    },
    {
      "heading": "Coding and stochastic processes",
      "text": "Joint entropy belongs to the framework developed by Claude Shannon for communication and information measurement. Its operational interpretation comes from lossless coding.\n\nFor a finite joint alphabet, the minimum expected length \\(L^*\\) of a binary prefix code for the pair satisfies\n\n\\[H(X,Y)\\leq L^*<H(X,Y)+1.\\]\n\nThrough coding long independent blocks, the average number of bits per pair can approach \\(H(X,Y)\\), as described by the source coding theorem. Joint coding can therefore exploit dependence that separate coding ignores.\n\nFor a stochastic process \\(X_1,X_2,\\ldots\\), block entropy is\n\n\\[H(X_1,\\ldots,X_n).\\]\n\nThe entropy rate, when the limit exists, is\n\n\\[\\overline H = \\lim_{n\\to\\infty} \\frac{H(X_1,\\ldots,X_n)}{n}.\\]\n\nIt measures uncertainty per observation after accounting for temporal dependence. For a stationary finite-alphabet process, this limit exists. For a stationary first-order Markov chain, it reduces to\n\n\\[\\overline H=H(X_2\\mid X_1).\\]"
    },
    {
      "heading": "Estimation and limitations",
      "text": "When the joint distribution is unknown, a basic estimate replaces probabilities with observed relative frequencies:\n\n\\[\\widehat H(X,Y) = -\\sum_{x,y}\\widehat p(x,y)\\log_b\\widehat p(x,y).\\]\n\nThis plug-in estimator is generally downward biased for finite discrete alphabets: limited samples miss rare outcomes and underrepresent the distribution’s uncertainty. Estimation becomes especially demanding as the number of variables increases, because the number of possible joint outcomes grows multiplicatively. This is an instance of the curse of dimensionality.\n\nFor continuous observations, binning produces the entropy of the discretized variables, not differential entropy itself; the result depends on the chosen bins.\n\nJoint entropy alone does not identify the structure or direction of dependence. Different distributions can have the same joint entropy, and entropy does not by itself establish causal relationships. Its interpretation therefore depends on the variables being measured, their distributions, and whether the analysis uses discrete or differential entropy."
    }
  ],
  "links": [
    "information-entropy",
    "random-variable",
    "information-theory",
    "joint-probability-distribution",
    "probability-mass-function",
    "expected-value",
    "self-information",
    "conditional-entropy",
    "conditional-probability",
    "statistical-independence",
    "mutual-information",
    "kullback-leibler-divergence",
    "exclusive-or",
    "probability-density-function",
    "differential-entropy",
    "multivariate-normal-distribution",
    "covariance-matrix",
    "determinant",
    "claude-shannon",
    "source-coding-theorem",
    "stochastic-process",
    "markov-chain",
    "curse-of-dimensionality",
    "causal-inference"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}