Self-information · aiwiki.page

Self-information is a quantity in information theory that measures the information associated with learning that a particular event has occurred. It is defined as the negative logarithm of the event’s probability: less probable events have greater self-information, while a certain event has none. Also called surprisal, it describes individual outcomes, whereas Shannon entropy describes their average self-information under a probability distribution. (ocw.mit.edu)

Definition and units

For an event $A$ with probability $P(A)>0$ , its self-information is

I_b(A)=-\log_b P(A) =\log_b\frac{1}{P(A)}, \qquad b>1.

For a discrete random variable $X$ with probability mass function $p_X(x)$ , the same definition gives

I_X(x)=-\log_b p_X(x).

The logarithm’s base determines the unit:

Base $2$ : bits.
Base $e$ : nats.
Base $10$ : hartleys.

Changing the base changes only the scale, not the ordering of outcomes by self-information. In particular, one bit equals $\ln 2$ nats. (web.mit.edu)

Because probabilities cannot exceed one, self-information is nonnegative. An event with probability one has $I(A)=0$ , and $I(A)$ grows without bound as $P(A)$ approaches zero. The extension $I(A)=+\infty$ for a zero-probability event is therefore understood as a limiting convention. (mtlsites.mit.edu)

The quantity is relative to the assigned probability model. Different models can assign different self-information to the same observation; it is not a measure of the observation’s semantic importance or practical value. (stat.cmu.edu)

Why the definition is logarithmic

A central motivation is additivity. If events $A$ and $B$ exhibit statistical independence, then

P(A\cap B)=P(A)P(B),

I(A\cap B)=I(A)+I(B).

Thus, learning two independent outcomes contributes the sum of their individual self-informations. The logarithm converts multiplication of probabilities into addition of information quantities. (sites.stat.columbia.edu)

More formally, suppose an information measure $f(p)$ depends only on probability, is continuous, decreases as probability increases, and satisfies

f(pq)=f(p)+f(q).

These requirements yield $f(p)=-k\ln p$ , with $k>0$ . Choosing a unit fixes the constant $k$ . (web.mit.edu)

Examples

Direct substitution into the base-2 definition gives:

Event probability	Self-information
$1$	$0$ bits
$1/2$	$1$ bit
$1/4$	$2$ bits
$1/8$	$3$ bits
$1/100$	Approximately $6.644$ bits

For a fair coin, either outcome has one bit of self-information. For a biased coin with $P(\text{heads})=0.9$ , heads has approximately $0.152$ bits, while tails has approximately $3.322$ bits. These values follow from the same probability-based definition: the less likely outcome is more surprising under the model. (ocw.mit.edu)

Relationship to entropy and conditional information

The expected value of self-information is Shannon entropy:

H_b(X) =\mathbb E[I_X(X)] =-\sum_x p_X(x)\log_b p_X(x).

Self-information is therefore an outcome-level quantity; entropy is a distribution-level average. Zero-probability terms in the entropy sum are assigned the value zero, using the limit $p\log p\to0$ as $p\to0^+$ . (stanford.edu)

Given an observed value $Y=y$ , conditional self-information uses conditional probability:

I(x\mid y)=-\log_b P(X=x\mid Y=y).

Its average over the joint distribution is conditional entropy. The probability product rule also gives the pointwise chain rule

I(x,y)=I(y)+I(x\mid y).

Independence reduces this to ordinary additivity. (stanford.edu)

Coding and statistical modeling

In lossless data compression, self-information represents an idealized codeword length. Actual binary codewords have integer lengths, so an outcome generally cannot be assigned a codeword of exactly $-\log_2 p(x)$ bits. However, a prefix code can use lengths

\ell(x)=\left\lceil-\log_2 p(x)\right\rceil.

For a finite alphabet, the minimum expected length $L^*$ of a uniquely decodable binary symbol code satisfies

H_2(X)\le L^*<H_2(X)+1.

Coding blocks of independent symbols makes the overhead per symbol arbitrarily small, connecting self-information with the source coding theorem. (stanford.edu)

If observations follow distribution $p$ , but their self-information is evaluated under a model $q$ , the average is cross-entropy:

H(p,q)=\mathbb E_{X\sim p}[-\log_b q(X)].

When the relevant quantities are finite,

H(p,q)=H(p)+D_{\mathrm{KL}}(p\|q),

where $D_{\mathrm{KL}}$ is Kullback–Leibler divergence. Thus, evaluating observations under an incorrect model introduces an average excess surprisal. Minimizing summed model surprisal is equivalent to maximum likelihood estimation. (stat.cmu.edu)

Continuous variables

For a variable with a probability density function $f(x)$ , an exact point generally has probability zero. Consequently, $-\log f(x)$ is not the self-information of the point event $X=x$ : density is not probability.

The expectation of the density-based quantity is differential entropy,

h(X)=-\int f(x)\log f(x)\,dx.

Unlike discrete self-information, $-\log f(x)$ can be negative and depends on the coordinate scale. A finite-resolution observation instead has self-information determined by the probability of its measurement interval or region, preserving the original event-based definition. (stat.cmu.edu)

References

Lecture 4: Language as Communicationocw.mit.edu
MIT 6.02 DRAFT Lecture Notes: Information, Entropy, and the Motivation for Source Codesweb.mit.edu
Chapter 5: Probabilitymtlsites.mit.edu
Information Theory, Inference, and Learning Algorithmssites.stat.columbia.edu
1 Annotated Slides: Computation Structuresocw.mit.edu
Lecture Notes on Statistics and Information Theorystanford.edu
Statistics and Information Theorystanford.edu
Information Theory I — Scene Setting and Statistical Applications (Lecture 9)stat.cmu.edu
Information Theorystat.cmu.edu