{
  "id": "information-entropy",
  "title": "Entropy (information theory)",
  "lang": "en",
  "source_lang": "en",
  "summary": "Information entropy measures the average uncertainty of a probability distribution and establishes fundamental limits on lossless data compression.",
  "category": "mathematics",
  "subcategory": "applied",
  "type": "concept",
  "aliases": [
    "Information entropy",
    "Shannon entropy",
    "Shannon information entropy"
  ],
  "generated_at": "2026-10-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/information-entropy/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/information-entropy/",
      "json": "https://aiwiki.page/zh/information-entropy.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Information entropy is a numerical measure of the uncertainty associated with a random variable. In information theory, it represents the average information gained when an outcome is observed, relative to a specified probability distribution. Claude Shannon introduced its foundational formulation in his 1948 paper A Mathematical Theory of Communication. Entropy concerns the statistical predictability of outcomes, not their meaning, truth, or practical importance. (cs.yale.edu)"
    },
    {
      "heading": "Definition and units",
      "text": "For a discrete random variable \\(X\\), with possible outcomes \\(x\\) and probabilities \\(p(x)\\), entropy is\n\n\\[H(X)=-\\sum_x p(x)\\log_b p(x).\\]\n\nThe convention \\(0\\log_b0=0\\) assigns zero contribution to impossible outcomes. The logarithm’s base determines the unit: base two gives bits, base \\(e\\) gives nats, and base ten gives hartleys. Changing the base rescales the quantity without changing comparisons between distributions. (web.mit.edu)\n\nAn outcome’s self-information, or surprisal, is \\(I(x)=-\\log_b p(x)\\). Entropy is therefore its expected value:\n\n\\[H(X)=\\mathbb{E}[I(X)].\\]\n\nRare outcomes convey more information when observed than common ones. This does not mean that a rare outcome dominates entropy: each surprisal is weighted by its probability. Entropy belongs to the distribution as a whole, whereas surprisal belongs to an individual outcome under that distribution. (web.mit.edu)\n\nFor a binary variable with a Bernoulli distribution,\n\n\\[H(X)=-p\\log_2p-(1-p)\\log_2(1-p).\\]\n\nA fair coin has entropy one bit; a coin that always produces the same result has entropy zero. A coin with head probability \\(0.9\\) has approximately \\(0.469\\) bits of entropy per toss. These values describe average uncertainty, not the information of every individual toss. (web.mit.edu)"
    },
    {
      "heading": "Mathematical properties",
      "text": "For a finite alphabet of \\(n\\) outcomes,\n\n\\[0\\leq H(X)\\leq\\log_b n.\\]\n\nThe lower bound is attained when one outcome has probability one. The upper bound is attained by the uniform distribution. Thus, increasing the number of available outcomes does not necessarily increase entropy: their probabilities also matter. Countably infinite distributions may have infinite entropy. (cs.yale.edu)\n\nEntropy is a concave function of the probability distribution. Consequently, mixing distributions cannot produce less entropy than their weighted average entropies. It is also unchanged by a one-to-one relabeling of outcomes. A deterministic transformation can only preserve or reduce discrete entropy, because merging outcomes discards distinctions. These properties distinguish entropy from numerical dispersion measures such as variance, which depend on the values assigned to outcomes. (arxiv.org)\n\nShannon motivated the logarithmic formula through requirements including continuity, increasing uncertainty for increasing numbers of equally likely choices, and consistency when a choice is decomposed into successive stages. Under these requirements, the entropy formula is determined up to a positive multiplicative constant. (cs.yale.edu)"
    },
    {
      "heading": "Joint entropy, conditioning, and shared information",
      "text": "Joint entropy \\(H(X,Y)\\) measures uncertainty about a pair of variables. Conditional entropy averages the uncertainty remaining about \\(X\\) after observing \\(Y\\):\n\n\\[H(X\\mid Y)=\\sum_y p(y)H(X\\mid Y=y).\\]\n\nThe chain rule states\n\n\\[H(X,Y)=H(Y)+H(X\\mid Y).\\]\n\nFor independent variables, joint entropy equals the sum of their individual entropies. More generally, \\(H(X,Y)\\leq H(X)+H(Y)\\). (ocw.mit.edu)\n\nMutual information quantifies the reduction in uncertainty supplied by another variable:\n\n\\[I(X;Y)=H(X)-H(X\\mid Y).\\]\n\nIt is symmetric, nonnegative, and zero exactly when the variables are independent. Conditioning cannot increase discrete entropy on average, although a particular observed value of \\(Y\\) may yield a conditional distribution with greater entropy than the original distribution. These statements concern statistical dependence, not necessarily causal relationships. (ocw.mit.edu)"
    },
    {
      "heading": "Compression and sequences",
      "text": "Entropy has an operational interpretation in lossless data compression. For a finite discrete source, any binary uniquely decodable code has expected codeword length at least \\(H(X)\\). A suitable prefix code achieves an expected length \\(L\\) satisfying\n\n\\[H(X)\\leq L<H(X)+1.\\]\n\nCoding independent source symbols in blocks makes the overhead per symbol arbitrarily small. Huffman coding constructs an optimal prefix code for a specified finite distribution. These are average-length bounds, not guarantees that every message becomes shorter. (people.csail.mit.edu)\n\nFor a stochastic process, successive symbols may be dependent. Its entropy rate, when the limit exists, is\n\n\\[h=\\lim_{n\\to\\infty}\\frac{H(X_1,\\ldots,X_n)}{n}.\\]\n\nFor independent, identically distributed symbols, this equals the single-symbol entropy. Dependencies can lower the rate by making future symbols predictable from preceding ones. This distinction matters for text and other structured sequences, where symbol frequencies alone do not capture all available compression. (people.csail.mit.edu)"
    },
    {
      "heading": "Statistical and machine-learning applications",
      "text": "Cross-entropy distinguishes the true distribution \\(p\\) from an assumed distribution \\(q\\):\n\n\\[H(p,q)=-\\sum_xp(x)\\log q(x) =H(p)+D_{\\mathrm{KL}}(p\\|q).\\]\n\nThe additional term is the Kullback–Leibler divergence. It measures the mismatch penalty in expected logarithmic prediction loss; it vanishes when the distributions coincide. (ocw.mit.edu)\n\nIn machine learning, decision-tree learning can use entropy to measure uncertainty in class labels. Information gain compares a parent node’s entropy with the weighted average entropy after a split. In language models, average negative log-probability provides a cross-entropy-based evaluation measure; perplexity is its exponential, using the corresponding logarithm base. Cross-entropy is also used as a loss function for probabilistic predictions. (cs.cmu.edu)"
    },
    {
      "heading": "Continuous variables and physical entropy",
      "text": "For a continuous variable with density \\(f(x)\\), differential entropy is\n\n\\[h(X)=-\\int f(x)\\log f(x)\\,dx.\\]\n\nUnlike discrete entropy, it can be negative and depends on the coordinate scale: \\(h(aX)=h(X)+\\log|a|\\) for nonzero \\(a\\). It is therefore not directly interchangeable with discrete entropy or a finite-resolution coding requirement. (people.csail.mit.edu)\n\nInformation entropy shares its mathematical form with physical entropy in statistical mechanics, where probabilities describe physical microstates and Boltzmann’s constant supplies physical units. Applying this connection requires a physical model; statistical unpredictability alone does not specify thermodynamic entropy or heat transfer. (cs.yale.edu)"
    }
  ],
  "links": [
    "random-variable",
    "information-theory",
    "probability",
    "claude-shannon",
    "self-information",
    "expected-value",
    "bernoulli-distribution",
    "variance",
    "joint-entropy",
    "conditional-entropy",
    "mutual-information",
    "lossless-data-compression",
    "prefix-code",
    "huffman-coding",
    "stochastic-process",
    "entropy-rate",
    "cross-entropy",
    "kullback-leibler-divergence",
    "machine-learning",
    "decision-tree-learning",
    "information-gain",
    "language-model",
    "perplexity",
    "loss-function",
    "differential-entropy",
    "entropy",
    "statistical-mechanics"
  ],
  "sources": [
    {
      "title": "cs.yale.edu/homes/yry/readings/general/shannon1948.pdf",
      "url": "https://www.cs.yale.edu/homes/yry/readings/general/shannon1948.pdf"
    },
    {
      "title": "web.mit.edu/6.02/www/f2011/handouts/2.pdf",
      "url": "https://web.mit.edu/6.02/www/f2011/handouts/2.pdf"
    },
    {
      "title": "arxiv.org/abs/1806.07249",
      "url": "https://arxiv.org/abs/1806.07249"
    },
    {
      "title": "ocw.mit.edu/courses/18-440-probability-and-random-variables-spring-2014/ed1f77e…",
      "url": "https://ocw.mit.edu/courses/18-440-probability-and-random-variables-spring-2014/ed1f77ec68cbaa6a415bdab0be8adeee_MIT18_440S14_Lecture34.pdf"
    },
    {
      "title": "people.csail.mit.edu/madhu/ST06/scribe/all.pdf",
      "url": "https://people.csail.mit.edu/madhu/ST06/scribe/all.pdf"
    },
    {
      "title": "ocw.mit.edu/courses/6-441-information-theory-spring-2016/5d8f16adc3385c9ff2975b…",
      "url": "https://ocw.mit.edu/courses/6-441-information-theory-spring-2016/5d8f16adc3385c9ff2975b121bd620e4_MIT6_441S16_course_notes.pdf"
    },
    {
      "title": "cs.cmu.edu/~guestrin/Class/10701-S07/Slides/decisiontrees-annotated.pdf",
      "url": "https://www.cs.cmu.edu/~guestrin/Class/10701-S07/Slides/decisiontrees-annotated.pdf"
    }
  ],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}