{
  "id": "self-information",
  "title": "Self-information",
  "lang": "en",
  "source_lang": "en",
  "summary": "Self-information is the negative logarithm of an event’s probability, measuring how surprising its occurrence is under a specified probability model.",
  "category": "mathematics",
  "subcategory": "applied",
  "type": "concept",
  "aliases": [
    "Surprisal",
    "Information content",
    "Shannon information"
  ],
  "generated_at": "2026-10-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/self-information/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/self-information/",
      "json": "https://aiwiki.page/zh/self-information.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Self-information is a quantity in information theory that measures the information associated with learning that a particular event has occurred. It is defined as the negative logarithm of the event’s probability: less probable events have greater self-information, while a certain event has none. Also called surprisal, it describes individual outcomes, whereas Shannon entropy describes their average self-information under a probability distribution. (ocw.mit.edu)"
    },
    {
      "heading": "Definition and units",
      "text": "For an event \\(A\\) with probability \\(P(A)>0\\), its self-information is\n\n\\[I_b(A)=-\\log_b P(A) =\\log_b\\frac{1}{P(A)}, \\qquad b>1.\\]\n\nFor a discrete random variable \\(X\\) with probability mass function \\(p_X(x)\\), the same definition gives\n\n\\[I_X(x)=-\\log_b p_X(x).\\]\n\nThe logarithm’s base determines the unit:\n\n- Base \\(2\\): bits.\n- Base \\(e\\): nats.\n- Base \\(10\\): hartleys.\n\nChanging the base changes only the scale, not the ordering of outcomes by self-information. In particular, one bit equals \\(\\ln 2\\) nats. (web.mit.edu)\n\nBecause probabilities cannot exceed one, self-information is nonnegative. An event with probability one has \\(I(A)=0\\), and \\(I(A)\\) grows without bound as \\(P(A)\\) approaches zero. The extension \\(I(A)=+\\infty\\) for a zero-probability event is therefore understood as a limiting convention. (mtlsites.mit.edu)\n\nThe quantity is relative to the assigned probability model. Different models can assign different self-information to the same observation; it is not a measure of the observation’s semantic importance or practical value. (stat.cmu.edu)"
    },
    {
      "heading": "Why the definition is logarithmic",
      "text": "A central motivation is additivity. If events \\(A\\) and \\(B\\) exhibit statistical independence, then\n\n\\[P(A\\cap B)=P(A)P(B),\\]\n\nso\n\n\\[I(A\\cap B)=I(A)+I(B).\\]\n\nThus, learning two independent outcomes contributes the sum of their individual self-informations. The logarithm converts multiplication of probabilities into addition of information quantities. (sites.stat.columbia.edu)\n\nMore formally, suppose an information measure \\(f(p)\\) depends only on probability, is continuous, decreases as probability increases, and satisfies\n\n\\[f(pq)=f(p)+f(q).\\]\n\nThese requirements yield \\(f(p)=-k\\ln p\\), with \\(k>0\\). Choosing a unit fixes the constant \\(k\\). (web.mit.edu)"
    },
    {
      "heading": "Examples",
      "text": "Direct substitution into the base-2 definition gives:\n\nEvent probability | Self-information\n\\(1\\) | \\(0\\) bits\n\\(1/2\\) | \\(1\\) bit\n\\(1/4\\) | \\(2\\) bits\n\\(1/8\\) | \\(3\\) bits\n\\(1/100\\) | Approximately \\(6.644\\) bits\n\nFor a fair coin, either outcome has one bit of self-information. For a biased coin with \\(P(\\text{heads})=0.9\\), heads has approximately \\(0.152\\) bits, while tails has approximately \\(3.322\\) bits. These values follow from the same probability-based definition: the less likely outcome is more surprising under the model. (ocw.mit.edu)"
    },
    {
      "heading": "Relationship to entropy and conditional information",
      "text": "The expected value of self-information is Shannon entropy:\n\n\\[H_b(X) =\\mathbb E[I_X(X)] =-\\sum_x p_X(x)\\log_b p_X(x).\\]\n\nSelf-information is therefore an outcome-level quantity; entropy is a distribution-level average. Zero-probability terms in the entropy sum are assigned the value zero, using the limit \\(p\\log p\\to0\\) as \\(p\\to0^+\\). (stanford.edu)\n\nGiven an observed value \\(Y=y\\), conditional self-information uses conditional probability:\n\n\\[I(x\\mid y)=-\\log_b P(X=x\\mid Y=y).\\]\n\nIts average over the joint distribution is conditional entropy. The probability product rule also gives the pointwise chain rule\n\n\\[I(x,y)=I(y)+I(x\\mid y).\\]\n\nIndependence reduces this to ordinary additivity. (stanford.edu)"
    },
    {
      "heading": "Coding and statistical modeling",
      "text": "In lossless data compression, self-information represents an idealized codeword length. Actual binary codewords have integer lengths, so an outcome generally cannot be assigned a codeword of exactly \\(-\\log_2 p(x)\\) bits. However, a prefix code can use lengths\n\n\\[\\ell(x)=\\left\\lceil-\\log_2 p(x)\\right\\rceil.\\]\n\nFor a finite alphabet, the minimum expected length \\(L^*\\) of a uniquely decodable binary symbol code satisfies\n\n\\[H_2(X)\\le L^*<H_2(X)+1.\\]\n\nCoding blocks of independent symbols makes the overhead per symbol arbitrarily small, connecting self-information with the source coding theorem. (stanford.edu)\n\nIf observations follow distribution \\(p\\), but their self-information is evaluated under a model \\(q\\), the average is cross-entropy:\n\n\\[H(p,q)=\\mathbb E_{X\\sim p}[-\\log_b q(X)].\\]\n\nWhen the relevant quantities are finite,\n\n\\[H(p,q)=H(p)+D_{\\mathrm{KL}}(p\\|q),\\]\n\nwhere \\(D_{\\mathrm{KL}}\\) is Kullback–Leibler divergence. Thus, evaluating observations under an incorrect model introduces an average excess surprisal. Minimizing summed model surprisal is equivalent to maximum likelihood estimation. (stat.cmu.edu)"
    },
    {
      "heading": "Continuous variables",
      "text": "For a variable with a probability density function \\(f(x)\\), an exact point generally has probability zero. Consequently, \\(-\\log f(x)\\) is not the self-information of the point event \\(X=x\\): density is not probability.\n\nThe expectation of the density-based quantity is differential entropy,\n\n\\[h(X)=-\\int f(x)\\log f(x)\\,dx.\\]\n\nUnlike discrete self-information, \\(-\\log f(x)\\) can be negative and depends on the coordinate scale. A finite-resolution observation instead has self-information determined by the probability of its measurement interval or region, preserving the original event-based definition. (stat.cmu.edu)"
    }
  ],
  "links": [
    "information-theory",
    "probability",
    "information-entropy",
    "probability-distribution",
    "random-variable",
    "probability-mass-function",
    "bit",
    "statistical-independence",
    "expected-value",
    "conditional-probability",
    "conditional-entropy",
    "lossless-data-compression",
    "prefix-code",
    "source-coding-theorem",
    "cross-entropy",
    "kullback-leibler-divergence",
    "maximum-likelihood-estimation",
    "probability-density-function",
    "differential-entropy"
  ],
  "sources": [
    {
      "title": "Lecture 4: Language as Communication",
      "url": "https://ocw.mit.edu/courses/9-59j-lab-in-psycholinguistics-spring-2017/c50ae829cf6d02ad14c4a50b05dd467d_MIT9_59jS17_lec4.pdf"
    },
    {
      "title": "MIT 6.02 DRAFT Lecture Notes: Information, Entropy, and the Motivation for Source Codes",
      "url": "https://web.mit.edu/6.02/www/f2011/handouts/2.pdf"
    },
    {
      "title": "Chapter 5: Probability",
      "url": "https://mtlsites.mit.edu/Courses/6.050/2014/notes/chapter5.pdf"
    },
    {
      "title": "Information Theory, Inference, and Learning Algorithms",
      "url": "https://sites.stat.columbia.edu/gelman/bayescomputation/mackay.pdf"
    },
    {
      "title": "1 Annotated Slides: Computation Structures",
      "url": "https://ocw.mit.edu/courses/6-004-computation-structures-spring-2017/pages/c1/c1s1/"
    },
    {
      "title": "Lecture Notes on Statistics and Information Theory",
      "url": "https://stanford.edu/class/stats311/lecture-notes.pdf"
    },
    {
      "title": "Statistics and Information Theory",
      "url": "https://stanford.edu/class/ee377/lecture-notes.pdf"
    },
    {
      "title": "Information Theory I — Scene Setting and Statistical Applications (Lecture 9)",
      "url": "https://www.stat.cmu.edu/~cshalizi/dm/20/lectures/09/lecture-09.html"
    },
    {
      "title": "Information Theory",
      "url": "https://www.stat.cmu.edu/~cshalizi/2010-06-15-1-CSSS.pdf"
    }
  ],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}