{
  "id": "information-gain",
  "title": "Information Gain",
  "lang": "en",
  "source_lang": "en",
  "summary": "Information gain measures how much an observation or data split reduces uncertainty, commonly using entropy reduction or divergence between prior and posterior probability distributions.",
  "category": "computer-science",
  "subcategory": "algorithms-theory",
  "type": "concept",
  "aliases": [
    "Entropy reduction"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/information-gain/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/information-gain/",
      "json": "https://aiwiki.page/zh/information-gain.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Information gain is a measure of the information obtained about an uncertain quantity through an observation, experiment, or partition of data. In information theory, expected information gain is commonly expressed as a reduction in entropy and is equivalent to mutual information. In machine learning, it is widely used to select splits in decision trees and to evaluate predictive features. In Bayesian inference, information gain from a particular observation often means the Kullback–Leibler divergence from a prior distribution to a posterior distribution. These formulations are closely related, but they are not interchangeable in every context."
    },
    {
      "heading": "Entropy and uncertainty reduction",
      "text": "Shannon entropy measures the uncertainty of a discrete random variable. For a variable \\(Y\\) with probability distribution \\(p(y)\\), its entropy is\n\n\\[H(Y)=-\\sum_y p(y)\\log_2 p(y),\\]\n\nwhere \\(0\\log_2 0\\) is defined as zero. Base-two logarithms express information in bits; natural logarithms express it in nats.\n\nEntropy is zero when the outcome is certain. For a variable with \\(k\\) possible values, it is at most \\(\\log_2 k\\), attained when all values are equally likely.\n\nIf another variable \\(X\\) is observed, the average uncertainty remaining about \\(Y\\) is the conditional entropy\n\n\\[H(Y\\mid X) = \\sum_x p(x)H(Y\\mid X=x).\\]\n\nThe expected information gain about \\(Y\\) from observing \\(X\\) is\n\n\\[\\operatorname{IG}(Y;X) = H(Y)-H(Y\\mid X).\\]\n\nThis quantity is exactly mutual information, usually written \\(I(Y;X)\\). For discrete variables with finite entropies,\n\n\\[0\\leq I(Y;X)\\leq \\min\\{H(Y),H(X)\\}.\\]\n\nIt is zero precisely when \\(X\\) and \\(Y\\) are statistically independent. If \\(X\\) completely determines \\(Y\\), the gain is \\(H(Y)\\): all uncertainty about the target is removed.\n\nAlthough prediction gives \\(X\\) and \\(Y\\) different roles, mutual information is symmetric:\n\n\\[I(Y;X)=I(X;Y).\\]\n\nThis symmetry concerns their statistical dependence; it does not establish causation or imply that the two prediction tasks are equally useful."
    },
    {
      "heading": "Information gain in decision trees",
      "text": "In decision-tree learning, a classification tree repeatedly divides a set of examples into subsets. Information gain measures how much a candidate split reduces uncertainty about class labels.\n\nLet a node contain a dataset \\(D\\) with class proportions \\(p_1,\\ldots,p_k\\). Its empirical class entropy is\n\n\\[H(D)=-\\sum_{c=1}^{k}p_c\\log_2 p_c.\\]\n\nA candidate split \\(S\\) partitions \\(D\\) into nonempty subsets \\(D_1,\\ldots,D_m\\). Its information gain is\n\n\\[\\operatorname{IG}(D,S) = H(D) - \\sum_{j=1}^{m} \\frac{|D_j|}{|D|}H(D_j).\\]\n\nThe weighting by subset size is essential: a small pure child should not count as much as a large pure child. With weighted examples, sample counts are replaced by sums of weights.\n\nThe split variable may represent categories of a categorical feature, a threshold on a numerical feature, or a more complex test. For a numerical feature \\(A\\), a binary split might use \\(A\\leq t\\) and \\(A>t\\). Its gain then concerns the binary branch indicator, not necessarily all the information contained in the original numerical value.\n\nA tree-building algorithm can evaluate candidate splits and select the one with the largest gain, subject to restrictions such as minimum leaf size or maximum depth. This is a greedy algorithm: the best immediate reduction does not necessarily produce the best complete tree.\n\nExample\n\nSuppose a node contains ten examples: five positive and five negative. Its entropy is\n\n\\[H(D)=1\\text{ bit}.\\]\n\nA split produces two children:\n\n- The first contains four positive examples and one negative example.\n- The second contains one positive example and four negative examples.\n\nEach child has entropy\n\n\\[-\\frac45\\log_2\\frac45-\\frac15\\log_2\\frac15 \\approx 0.722.\\]\n\nBecause the children have equal size, their weighted average entropy is also approximately \\(0.722\\) bits. The information gain is therefore\n\n\\[\\operatorname{IG}(D,S)\\approx 1-0.722=0.278\\text{ bits}.\\]\n\nA split producing two pure children would gain one bit. A split leaving both children equally divided between the classes would gain zero bits.\n\nRelationship to classification loss\n\nEntropy also has a predictive interpretation. If a leaf predicts its empirical class proportions, its average training logarithmic loss equals its class entropy, using the same logarithm base. Information gain therefore measures the decrease in this fitted loss obtained by replacing one set of class probabilities with separate probabilities for each child.\n\nThis equivalence concerns probabilities fitted to the data at the node. It does not guarantee the same improvement on unseen data."
    },
    {
      "heading": "Bayesian information gain",
      "text": "In Bayesian inference, uncertainty about an unknown parameter \\(\\Theta\\) is represented by a prior distribution \\(p(\\theta)\\). After observing data \\(x\\), the distribution becomes the posterior \\(p(\\theta\\mid x)\\).\n\nA common definition of the information gained from that particular observation is\n\n\\[\\operatorname{IG}(x) = D_{\\mathrm{KL}} \\bigl(p(\\theta\\mid x)\\,\\|\\,p(\\theta)\\bigr),\\]\n\nwhere Kullback–Leibler divergence is\n\n\\[D_{\\mathrm{KL}}(p\\|q) = \\int p(\\theta) \\log\\frac{p(\\theta)}{q(\\theta)} \\,d\\theta\\]\n\nfor continuous distributions, with a sum replacing the integral in the discrete case.\n\nThis divergence is nonnegative but generally asymmetric. It measures how much the posterior differs from the prior, rather than merely how much the distribution has narrowed.\n\nAveraging over possible observations under the prior predictive distribution gives expected information gain:\n\n\\[\\mathbb{E}_{X} \\left[ D_{\\mathrm{KL}} \\bigl(p(\\theta\\mid X)\\,\\|\\,p(\\theta)\\bigr) \\right] = I(\\Theta;X).\\]\n\nFor discrete variables with finite entropies, this also equals\n\n\\[H(\\Theta)-\\mathbb{E}_X[H(\\Theta\\mid X)].\\]\n\nAn important distinction concerns a particular observation. The entropy difference\n\n\\[H(\\Theta)-H(\\Theta\\mid X=x)\\]\n\ncan be negative: an unexpected observation may leave the posterior more uncertain than the prior. The posterior-to-prior divergence remains nonnegative. The two measures agree after the appropriate averaging, not generally observation by observation.\n\nFor continuous parameters, differential entropy does not have all the properties of discrete entropy and depends on the coordinate system. Mutual information and Kullback–Leibler divergence are invariant under suitable invertible transformations, making them useful formulations of information gain in continuous models."
    },
    {
      "heading": "Applications",
      "text": "Feature selection\n\nInformation gain can rank candidate inputs by their relationship with a target in feature selection. Unlike simple linear correlation, mutual information can detect nonlinear statistical dependence.\n\nRanking features individually has limitations. Several high-scoring features may contain nearly identical information, while individually uninformative features may become useful jointly. For example, if two independent, uniformly random binary inputs determine a target through exclusive OR, each input individually has zero mutual information with the target, but the pair determines it completely.\n\nConditional mutual information addresses the additional information a variable contributes after other variables are known:\n\n\\[I(Y;X\\mid Z)=H(Y\\mid Z)-H(Y\\mid X,Z).\\]\n\nExperimental design and active learning\n\nIn Bayesian experimental design, an experiment can be selected to maximize expected information gain about unknown quantities. The expectation accounts for possible outcomes before the experiment is performed.\n\nIn active learning, a model may select examples whose labels are expected to reveal the most about its uncertain parameters or predictions. High uncertainty about an individual label is not always equivalent to high information gain: the uncertainty may reflect irreducible randomness rather than uncertainty that further observations can resolve.\n\nText classification\n\nIn natural language processing, the presence or absence of a term can be treated as a feature, and its information gain can measure its association with document classes. Such scores depend on the dataset’s class proportions and term frequencies; they do not measure a word’s intrinsic linguistic importance."
    },
    {
      "heading": "Limitations and alternative criteria",
      "text": "Bias toward many-valued features\n\nEmpirical information gain tends to favor splits with many branches. A unique identifier can create one pure child per training example, achieving maximal training gain without providing useful prediction for new cases. More generally, selecting among many candidate splits can exploit chance patterns in training data.\n\nThe gain ratio, used in C4.5, normalizes gain by the entropy of the branch assignment:\n\n\\[\\operatorname{GainRatio}(D,S) = \\frac{\\operatorname{IG}(D,S)} {-\\sum_j \\frac{|D_j|}{|D|} \\log_2\\frac{|D_j|}{|D|}}.\\]\n\nIt is undefined when the denominator is zero. Normalization can introduce other preferences, including sensitivity to very unbalanced splits, so gain ratio is typically used with additional selection rules rather than as an unrestricted replacement.\n\nEstimation and generalization\n\nInformation gain computed from finite data is an estimate of population dependence. Sparse categories, small samples, and extensive split searching can exaggerate apparent associations. Continuous-variable estimates also depend on methods such as discretization or density estimation.\n\nA large training gain does not establish generalization. Tree pruning, sample-size restrictions, and regularization can limit overfitting. Feature selection must also remain within the training portion of an evaluation procedure; using held-out labels to select features causes data leakage.\n\nOther split objectives\n\nClassification trees may instead use Gini impurity or other objectives. These criteria can favor similar splits but are not identical to Shannon information gain. Regression trees commonly minimize squared error or reduce variance; the term “gain” in such systems may refer to objective improvement rather than an information-theoretic quantity.\n\nInformation gain is also distinct from practical value. A highly informative observation may be costly, unavailable at prediction time, or irrelevant to the decision being made."
    },
    {
      "heading": "Historical development",
      "text": "The mathematical basis comes from Claude Shannon’s 1948 formulation of information theory, which established entropy as a measure of uncertainty. Information-based splitting became prominent in machine learning through J. Ross Quinlan’s ID3 decision-tree algorithm, described in his 1986 paper Induction of Decision Trees. His later C4.5 system used gain ratio to address the preference of unnormalized information gain for attributes with many outcomes.\n\nBayesian uses connect the concept to a separate tradition of statistical experimental design, in which observations are evaluated by how much they are expected to change knowledge about unknown quantities."
    }
  ],
  "links": [
    "information-theory",
    "machine-learning",
    "information-entropy",
    "random-variable",
    "probability-distribution",
    "bit",
    "mutual-information",
    "causation",
    "decision-tree-learning",
    "algorithm",
    "greedy-algorithm",
    "loss-function",
    "bayesian-inference",
    "kullback-leibler-divergence",
    "feature-selection",
    "active-learning",
    "natural-language-processing",
    "training-data",
    "generalization",
    "regularization",
    "overfitting",
    "data-leakage"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}