{
  "id": "conditional-entropy",
  "title": "Conditional entropy",
  "lang": "en",
  "source_lang": "en",
  "summary": "Conditional entropy measures the average uncertainty remaining about a random variable when another variable is known.",
  "category": "mathematics",
  "subcategory": "applied",
  "type": "concept",
  "aliases": [
    "Conditional Shannon entropy"
  ],
  "generated_at": "2026-10-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/conditional-entropy/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/conditional-entropy/",
      "json": "https://aiwiki.page/zh/conditional-entropy.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Conditional entropy is a quantity in information theory that measures the average uncertainty remaining about a random variable after another variable is observed. Written \\(H(X\\mid Y)\\), it averages the Shannon entropy of the conditional distributions of \\(X\\) over possible observations of \\(Y\\). It describes uncertainty under a specified probabilistic model, rather than uncertainty about any particular observed outcome. (math.mit.edu)"
    },
    {
      "heading": "Definition and interpretation",
      "text": "For discrete variables \\(X\\) and \\(Y\\), let \\(p(x,y)\\) denote their joint probability distribution and \\(p(x\\mid y)\\) their conditional probability. Conditional entropy is defined by\n\n\\[H(X\\mid Y) =-\\sum_{y:p(y)>0}\\sum_x p(x,y)\\log_b p(x\\mid y).\\]\n\nEquivalently,\n\n\\[H(X\\mid Y)=\\sum_{y:p(y)>0}p(y)H(X\\mid Y=y).\\]\n\nTerms with zero joint probability contribute zero. Observations having \\(p(y)=0\\) are omitted because their conditional distributions do not affect the average. (math.mit.edu)\n\nThe logarithm’s base determines the unit: base two gives bits, while the natural logarithm gives nats. In expected-value notation,\n\n\\[H(X\\mid Y)=\\mathbb E[-\\log_b p(X\\mid Y)].\\]\n\nThus it is the average conditional self-information of \\(X\\). Unlike \\(H(X\\mid Y=y)\\), which concerns one observation, \\(H(X\\mid Y)\\) averages over the entire probability distribution of observations. A discrete \\(X\\) can also be conditioned on a continuous \\(Y\\), using an expectation rather than a discrete sum over \\(y\\). (people.lids.mit.edu)"
    },
    {
      "heading": "Fundamental identities and inequalities",
      "text": "The entropy chain rule expresses joint uncertainty as uncertainty about one variable plus the uncertainty remaining about the other:\n\n\\[H(X,Y)=H(Y)+H(X\\mid Y).\\]\n\nFor a finite sequence, its extension is\n\n\\[H(X_1,\\ldots,X_n) =\\sum_{i=1}^{n}H(X_i\\mid X_1,\\ldots,X_{i-1}),\\]\n\nwhere the first term is \\(H(X_1)\\). These identities follow by factoring joint probabilities into conditional probabilities. When entropies are finite, the first identity also gives \\(H(X\\mid Y)=H(X,Y)-H(Y)\\). Direct definitions remain important when subtraction would produce the undefined expression \\(\\infty-\\infty\\). (people.csail.mit.edu)\n\nFor finite-valued variables,\n\n\\[0\\le H(X\\mid Y)\\le H(X).\\]\n\nThe upper equality holds precisely when \\(X\\) and \\(Y\\) exhibit statistical independence. The lower equality holds precisely when \\(X\\) is a deterministic function of \\(Y\\), except on events of probability zero. Conditional entropy is generally asymmetric: knowing \\(Y\\) may determine \\(X\\), without knowing \\(X\\) determining \\(Y\\). (ocw.mit.edu)\n\nAdditional conditioning cannot increase average discrete entropy:\n\n\\[H(X\\mid Y,Z)\\le H(X\\mid Y).\\]\n\nThe difference is conditional mutual information, \\(I(X;Z\\mid Y)\\), which is nonnegative. For finite-valued variables, equality holds exactly when \\(X\\) and \\(Z\\) have conditional independence given \\(Y\\). Likewise, mutual information satisfies\n\n\\[I(X;Y)=H(X)-H(X\\mid Y),\\]\n\nquantifying how much observing \\(Y\\) reduces uncertainty about \\(X\\) on average. (ocw.mit.edu)"
    },
    {
      "heading": "Examples and the role of averaging",
      "text": "Consider a fair binary variable \\(X\\), and let \\(Y\\) equal \\(X\\) with probability \\(1-\\varepsilon\\), otherwise taking the opposite value, independently of \\(X\\). Applying the definition gives\n\n\\[H(X\\mid Y)=h_2(\\varepsilon), \\qquad h_2(t)=-t\\log_2t-(1-t)\\log_2(1-t).\\]\n\nAt \\(\\varepsilon=0\\), the observation identifies \\(X\\). At \\(\\varepsilon=\\tfrac12\\), it provides no information, leaving one bit of uncertainty. At \\(\\varepsilon=1\\), it again identifies \\(X\\), because the observed value is always inverted. These are direct consequences of the conditional distributions. (math.mit.edu)\n\n“Conditioning reduces entropy” is an average statement, not a guarantee for every observation. For example, let \\(U\\) and \\(Y\\) be independent fair binary variables, and set \\(X=U\\lor Y\\). Then \\(H(X)=h_2(1/4)\\), approximately \\(0.811\\) bits. Observing \\(Y=0\\) leaves \\(X=U\\), raising uncertainty to one bit; observing \\(Y=1\\) determines \\(X\\). Their average is nevertheless \\(H(X\\mid Y)=0.5\\) bits, below the unconditional entropy. (ocw.mit.edu)"
    },
    {
      "heading": "Coding and statistical learning",
      "text": "In lossless data compression with side information, conditional entropy determines an asymptotic coding threshold. For independent, identically distributed finite-alphabet pairs \\((X_i,Y_i)\\), rates above \\(H(X\\mid Y)\\) permit reconstruction of \\(X\\)-blocks with error probability tending to zero when the decoder knows the corresponding \\(Y\\)-blocks. The Slepian–Wolf theorem establishes that this threshold remains achievable even when the encoder does not know those observations. (people.lids.mit.edu)\n\nConditional entropy also describes uncertainty in dependent sequences. For a stationary finite-alphabet stochastic process, the uncertainty about the next symbol conditioned on increasingly long histories converges to the entropy rate. In a stationary first-order Markov chain, this rate is \\(H(X_{t+1}\\mid X_t)\\). (stanforddatacompressionclass.github.io)\n\nIn machine learning, \\(H(Y\\mid X)\\) measures uncertainty about a label \\(Y\\) given features \\(X\\). It is distinct from the conditional cross-entropy of a model \\(q(y\\mid x)\\). Their difference is the expected Kullback–Leibler divergence between the true conditional distribution and the model. Consequently, under unrestricted prediction, conditional entropy is the minimum achievable population logarithmic loss, rather than necessarily the loss attained by a trained model. (s3-us-west-2.amazonaws.com)"
    },
    {
      "heading": "Continuous variables",
      "text": "For continuous variables admitting a joint probability density function, conditional differential entropy is\n\n\\[h(X\\mid Y) =-\\int\\!\\!\\int f_{X,Y}(x,y)\\log f_{X\\mid Y}(x\\mid y)\\,dx\\,dy.\\]\n\nWhen the relevant quantities are finite, \\(h(X\\mid Y)=h(X,Y)-h(Y)\\). Unlike discrete conditional entropy, it can be negative and changes when the scale of \\(X\\) changes. It is therefore not directly interpretable as a nonnegative number of bits required for exact representation of a continuous observation. Mutual information remains nonnegative and equals \\(h(X)-h(X\\mid Y)\\) when that difference is well defined. (s3-us-west-2.amazonaws.com)"
    }
  ],
  "links": [
    "information-theory",
    "random-variable",
    "information-entropy",
    "joint-probability-distribution",
    "conditional-probability",
    "bit",
    "expected-value",
    "self-information",
    "probability-distribution",
    "statistical-independence",
    "mathematical-function",
    "conditional-independence",
    "mutual-information",
    "lossless-data-compression",
    "stochastic-process",
    "entropy-rate",
    "markov-chain",
    "machine-learning",
    "cross-entropy",
    "kullback-leibler-divergence",
    "loss-function",
    "probability-density-function",
    "differential-entropy"
  ],
  "sources": [
    {
      "title": "600: Lecture 33 — Entropy",
      "url": "https://math.mit.edu/~sheffield/2017600/Lecture33.pdf"
    },
    {
      "title": "Information Theory — Polyanskiy and Wu",
      "url": "https://people.lids.mit.edu/yp/homepage/data/itbook-export.pdf"
    },
    {
      "title": "441S16: Course Notes",
      "url": "https://ocw.mit.edu/courses/6-441-information-theory-spring-2016/5d8f16adc3385c9ff2975b121bd620e4_MIT6_441S16_course_notes.pdf"
    },
    {
      "title": "ST06 — Lecture 3",
      "url": "https://people.csail.mit.edu/madhu/ST06/mynotes/lect03.pdf"
    },
    {
      "title": "Non IID Sources and Entropy Rate",
      "url": "https://stanforddatacompressionclass.github.io/notes/lossless_iid/non_iid_sources.html"
    },
    {
      "title": "11. Information Theory — Dive into Deep Learning",
      "url": "https://s3-us-west-2.amazonaws.com/classic.d2l.ai/chapter_appendix-mathematics-for-deep-learning/information-theory.html"
    }
  ],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}