Channel capacity

Channel capacity is the fundamental limit on reliable information transmission through a communication channel. In information theory, it is defined operationally as the supremum of rates achievable by coding schemes whose decoding error probability approaches zero as the number of channel uses increases. Capacity depends on the channel’s statistical behavior and any restrictions on its inputs. It is usually measured in bits per channel use, or in bits per second for continuous-time systems. Claude Shannon established the mathematical framework for this concept in his 1948 paper A Mathematical Theory of Communication. (ocw.mit.edu)

Mathematical definition

A discrete memoryless channel has input alphabet $\mathcal X$, output alphabet $\mathcal Y$, and transition probabilities $W(y\mid x)$. These give the conditional probability of observing output $y$ when input $x$ is transmitted. “Memoryless” means that, conditional on the input sequence, outputs follow the product law

For finite alphabets, its capacity is

where $X$ and $Y$ are input and output random variables, $p_X$ is the input probability distribution, and $I(X;Y)$ is their mutual information. The channel law remains fixed while the input distribution is varied. With base-two logarithms,

Thus capacity measures the greatest information about the input that the output can convey per use. (ocw.mit.edu)

Equivalently,

where $H(Y)$ is information entropy and $H(Y\mid X)$ is conditional entropy. The first term measures output uncertainty; the second measures uncertainty remaining when the input is known. Maximizing output entropy alone is therefore generally insufficient. For continuous alphabets or constrained inputs, the maximum is commonly replaced by a supremum over admissible distributions. (ocw.mit.edu)

Operational meaning and coding theorem

A block code assigns each of $M$ possible messages an input sequence of length $n$. A decoder estimates the message from the received sequence. Its information rate is

The noisy-channel coding theorem establishes that, for a finite discrete memoryless channel, every rate strictly below $C$ can be approached by sequences of codes with error probability tending to zero. Conversely, reliable transmission at a fixed rate above $C$ is impossible. Capacity is a supremum: the theorem does not generally guarantee vanishing error at exactly $R=C$. (ocw.mit.edu)

An error-correcting code introduces structured redundancy so that different messages remain distinguishable despite noise. The achievable rate counts message information, not all transmitted coded symbols. Reliability is asymptotic and does not mean that a finite transmission is guaranteed to contain no errors. Finite block length, decoding complexity, and delay impose additional constraints beyond the capacity value. (ocw.mit.edu)

Elementary channel models

For a noiseless channel with $q$ distinct input symbols reproduced exactly at the output,

A uniform input distribution achieves this value. A noiseless binary channel consequently carries one bit per use. If the output distribution is identical for every input, input and output are independent and capacity is zero. (ocw.mit.edu)

A binary symmetric channel independently flips each transmitted bit with probability $p$. Its capacity is

Equiprobable inputs achieve capacity. At $p=1/2$, the output conveys no information about the input. At $p=1$, every bit is flipped deterministically, so inversion recovers the input and capacity is again one bit per use. (ocw.mit.edu)

A binary erasure channel reproduces an input bit correctly with probability $1-\varepsilon$, and otherwise produces a distinct erasure symbol. Its capacity is

Unlike an undetected bit flip, an erasure identifies which received position lacks information. This distinction produces different capacity formulas even when the two channels have equal probabilities of a corrupted output. (ocw.mit.edu)

Gaussian channels and physical constraints

For a real additive white Gaussian noise channel,

where the noise $Z$ is independent of the input and has a zero-mean normal distribution with variance $\sigma^2$. Under the average input constraint $\mathbb E[X^2]\le P$, expressed through expected value, capacity is

bits per real-valued channel use. A zero-mean Gaussian input with variance $P$ achieves the supremum. Input constraints are essential: this idealized channel has unbounded capacity if input power is unrestricted. (ocw.mit.edu)

For an ideal continuous-time band-limited Gaussian channel, the corresponding expression is

bits per second. Here $B$ is bandwidth in hertz, $P$ is average received signal power, and $N$ is noise power within that bandwidth. Their ratio is the signal-to-noise ratio. This formula assumes the specified Gaussian noise model and power constraint; it is not a universal formula for every physical communication medium. (ocw.mit.edu)

Computation and extensions

For a fixed finite channel, mutual information is concave in the input distribution, making capacity evaluation a convex optimization problem. Symmetry sometimes yields an analytical solution. Otherwise, the Blahut–Arimoto algorithm iteratively improves an input distribution and computes capacity numerically. (ocw.mit.edu)

Capacity must also specify the communication setting. Channels with memory need not obey the single-use formula above. Noiseless causal feedback does not increase the capacity of a discrete memoryless channel, although it can improve other coding properties. Zero-error capacity imposes the stronger requirement that decoding errors be impossible, rather than merely increasingly unlikely. Multiuser channels are described by achievable regions of rate combinations instead of a single scalar limit. (ocw.mit.edu)

References

1. A Mathematical Theory of Communicationprinceton.edu
2. MIT6_441S10_lec08.pdf | Information Theory | MIT OpenCourseWareocw.mit.edu
3. 441 Information Theory, Lecture 9ocw.mit.edu
4. Information Theory: Lecture Notes | MIT OpenCourseWareocw.mit.edu