{
  "id": "mutual-information",
  "title": "Mutual information",
  "lang": "en",
  "source_lang": "en",
  "summary": "Mutual information measures statistical dependence by quantifying how much observing one random variable reduces uncertainty about another.",
  "category": "mathematics",
  "subcategory": "applied",
  "type": "concept",
  "aliases": [
    "MI",
    "Shannon mutual information"
  ],
  "generated_at": "2026-10-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/mutual-information/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/mutual-information/",
      "json": "https://aiwiki.page/zh/mutual-information.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Mutual information is a quantity in information theory that measures dependence between two random variables. It compares their actual joint distribution with the distribution they would have if they were independent. For discrete variables, it also expresses the average reduction in uncertainty about one variable obtained by observing the other. Unlike measures restricted to linear relationships, mutual information can detect nonlinear dependence. (stat.cmu.edu)"
    },
    {
      "heading": "Definition",
      "text": "For discrete variables \\(X\\) and \\(Y\\), with joint probability distribution \\(p(x,y)\\) and marginal distributions \\(p_X(x)\\) and \\(p_Y(y)\\), mutual information is\n\n\\[I(X;Y)= \\sum_{x,y}p(x,y) \\log\\frac{p(x,y)}{p_X(x)p_Y(y)}.\\]\n\nTerms with \\(p(x,y)=0\\) contribute zero. Base-two logarithms express information in bits, while natural logarithms express it in nats. The quantity depends on the entire probability distribution, rather than a particular observed pair. (people.csail.mit.edu)\n\nEquivalently,\n\n\\[I(X;Y)=D_{\\mathrm{KL}} \\!\\left(P_{XY}\\,\\middle\\|\\,P_X\\otimes P_Y\\right),\\]\n\nwhere \\(D_{\\mathrm{KL}}\\) is Kullback–Leibler divergence and \\(P_X\\otimes P_Y\\) is the product of the marginal probability measures. This definition extends to continuous variables and mixed discrete–continuous distributions. Mutual information can be infinite when the joint measure is not absolutely continuous with respect to that product. (web.stanford.edu)\n\nThe logarithmic ratio inside the discrete sum is pointwise mutual information. It may be negative for particular outcomes, whereas its expected value, mutual information, is always nonnegative. (stat.cmu.edu)"
    },
    {
      "heading": "Entropy interpretation and properties",
      "text": "For discrete variables with finite information entropies,\n\n\\[\\begin{aligned} I(X;Y) &=H(X)-H(X\\mid Y)\\\\ &=H(Y)-H(Y\\mid X)\\\\ &=H(X)+H(Y)-H(X,Y). \\end{aligned}\\]\n\nHere conditional entropy \\(H(X\\mid Y)\\) measures the uncertainty remaining about \\(X\\) after \\(Y\\) is observed. These identities explain the interpretation of mutual information as shared information, although it is a distributional quantity rather than a count of literally shared symbols. (people.csail.mit.edu)\n\nMutual information is symmetric: \\(I(X;Y)=I(Y;X)\\). It equals zero exactly when statistical independence holds. For finite-entropy discrete variables,\n\n\\[0\\leq I(X;Y)\\leq \\min\\{H(X),H(Y)\\}.\\]\n\nIf \\(Y\\) is a deterministic function of \\(X\\), then \\(I(X;Y)=H(Y)\\); in particular, \\(I(X;X)=H(X)\\). Nonnegativity follows from the corresponding property of divergence. (people.lids.mit.edu)"
    },
    {
      "heading": "Examples and comparison with correlation",
      "text": "As an elementary calculated example, let \\(X\\) be an equally likely binary value. If \\(Y=X\\), observing \\(Y\\) completely determines \\(X\\), giving one bit of mutual information. If \\(Y\\) is an independent equally likely binary value, the mutual information is zero. These results follow directly by substituting the joint probabilities into the definition. (people.csail.mit.edu)\n\nCorrelation and mutual information describe different aspects of dependence. Pearson correlation measures linear association, whereas mutual information detects any departure from independence. For a nonsingular bivariate normal distribution with correlation coefficient \\(\\rho\\),\n\n\\[I(X;Y)=-\\frac12\\log(1-\\rho^2),\\]\n\nin units determined by the logarithm. Outside jointly Gaussian models, correlation generally does not determine mutual information, and zero correlation need not imply independence. (web.stanford.edu)"
    },
    {
      "heading": "Continuous variables",
      "text": "For variables with a joint probability density function, the definition becomes an integral:\n\n\\[I(X;Y)= \\int\\!\\!\\int f_{XY}(x,y) \\log\\frac{f_{XY}(x,y)}{f_X(x)f_Y(y)} \\,dx\\,dy.\\]\n\nWhen the relevant quantities are finite, this equals \\(h(X)+h(Y)-h(X,Y)\\), where \\(h\\) denotes differential entropy. Unlike differential entropy, mutual information remains nonnegative and is invariant under suitably measurable invertible transformations applied separately to the variables. Exact self-observation of a non-atomic continuous variable has infinite mutual information; finite measurement resolution or added noise changes that model. (people.lids.mit.edu)"
    },
    {
      "heading": "Conditioning and information processing",
      "text": "Conditional mutual information measures dependence remaining after another variable \\(Z\\) is observed:\n\n\\[I(X;Y\\mid Z) = \\mathbb{E}_Z\\! \\left[ D_{\\mathrm{KL}} \\left(P_{XY\\mid Z}\\| P_{X\\mid Z}\\otimes P_{Y\\mid Z}\\right) \\right].\\]\n\nIt is nonnegative and vanishes exactly when conditional independence holds, up to probability-zero conditioning events. Its chain rule is\n\n\\[I(X;Y,Z)=I(X;Z)+I(X;Y\\mid Z).\\]\n\nConditioning can increase or decrease mutual information; it is not simply a subtraction of a fixed amount of dependence. (people.lids.mit.edu)\n\nThe data processing inequality states that if \\(X\\to Y\\to Z\\) forms a Markov chain, then\n\n\\[I(X;Z)\\leq I(X;Y).\\]\n\nProcessing \\(Y\\) without additional access to \\(X\\) cannot create information about \\(X\\). Equality can occur when the processing preserves a sufficient statistic. (people.lids.mit.edu)"
    },
    {
      "heading": "Applications and estimation",
      "text": "In communication theory, channel capacity for a discrete memoryless channel is the maximum of \\(I(X;Y)\\) over input distributions. Mutual information therefore connects probabilistic dependence with achievable reliable transmission rates. (cioffi-group.stanford.edu)\n\nIn machine learning, feature selection methods use it to measure relevance to a target. Conditional criteria account for information already supplied by selected features, helping distinguish additional relevance from redundancy. (jmlr.org)\n\nEstimating mutual information from samples is a separate statistical problem. Discrete plug-in estimates substitute observed frequencies into the formula and commonly exhibit positive finite-sample bias. Continuous estimators include binning, density estimation, and nearest-neighbor approaches. Kraskov–Stögbauer–Grassberger estimators use local neighbor distances rather than fixed bins; their accuracy depends on sample size, tuning choices, and distributional structure. (stat.cmu.edu)"
    }
  ],
  "links": [
    "information-theory",
    "random-variable",
    "joint-probability-distribution",
    "bit",
    "probability-distribution",
    "kullback-leibler-divergence",
    "expected-value",
    "information-entropy",
    "conditional-entropy",
    "statistical-independence",
    "correlation",
    "normal-distribution",
    "probability-density-function",
    "integral",
    "differential-entropy",
    "conditional-independence",
    "markov-chain",
    "sufficient-statistic",
    "channel-capacity",
    "machine-learning",
    "feature-selection",
    "statistics",
    "estimator-bias",
    "density-estimation"
  ],
  "sources": [
    {
      "title": "Information Theory I — Scene Setting and Statistical Applications (Lecture 9)",
      "url": "https://www.stat.cmu.edu/~cshalizi/dm/20/lectures/09/lecture-09.html"
    },
    {
      "title": "Lecture Notes on Statistics and Information Theory — John Duchi",
      "url": "https://web.stanford.edu/class/stats311/lecture-notes.pdf"
    },
    {
      "title": "441 Transmission of Information — Lecture Notes",
      "url": "https://people.csail.mit.edu/madhu/ST06/scribe/all.pdf"
    },
    {
      "title": "Information Theory — Chapter 3: Mutual Information",
      "url": "https://people.lids.mit.edu/yp/homepage/data/itbook-2022.pdf"
    },
    {
      "title": "Information Theory — Book Manuscript",
      "url": "https://people.lids.mit.edu/yp/homepage/data/itbook-export.pdf"
    },
    {
      "title": "Mutual Information and Channel Capacity",
      "url": "https://cioffi-group.stanford.edu/doc/book/chap2.pdf"
    },
    {
      "title": "Estimating mutual information",
      "url": "https://doi.org/10.1103/PHYSREVE.69.066138"
    },
    {
      "title": "Fast Binary Feature Selection with Conditional Mutual Information",
      "url": "https://www.jmlr.org/papers/v5/fleuret04a.html"
    },
    {
      "title": "Conditional Likelihood Maximisation: A Unifying Framework for Information Theoretic Feature Selection",
      "url": "https://jmlr.org/papers/v13/brown12a.html"
    }
  ],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}