{
  "id": "noisy-channel-coding-theorem",
  "title": "Noisy-channel coding theorem",
  "lang": "en",
  "source_lang": "en",
  "summary": "The noisy-channel coding theorem identifies channel capacity as the limiting rate for communication with arbitrarily small decoding error.",
  "category": "mathematics",
  "subcategory": "applied",
  "type": "concept",
  "aliases": [
    "Shannon’s channel coding theorem",
    "Shannon’s noisy-channel coding theorem",
    "Shannon’s second theorem"
  ],
  "generated_at": "2026-10-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/noisy-channel-coding-theorem/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/noisy-channel-coding-theorem/",
      "json": "https://aiwiki.page/zh/noisy-channel-coding-theorem.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The noisy-channel coding theorem is a fundamental result in information theory establishing the maximum rate at which information can be transmitted reliably through a noisy communication channel. For a discrete memoryless channel, every rate strictly below its channel capacity permits codes whose decoding error probability tends to zero as block length increases; rates strictly above capacity do not. Claude Shannon introduced the result in his 1948 paper A Mathematical Theory of Communication. It establishes the possibility of reliable communication despite noise, without requiring the noise itself to disappear. (web.njit.edu)"
    },
    {
      "heading": "Channel model and coding",
      "text": "The standard formulation concerns a discrete memoryless channel with finite input and output alphabets \\(\\mathcal X\\) and \\(\\mathcal Y\\). Its behavior is specified by the conditional probabilities \\(W(y\\mid x)\\). “Memoryless” means that, conditional on the transmitted sequence, successive outputs are independent and each depends only on the corresponding input:\n\n\\[W^n(y^n\\mid x^n)=\\prod_{i=1}^{n}W(y_i\\mid x_i).\\]\n\nThe same transition law applies at every use. This assumption does not require the symbols within a codeword to be independent. (ocw.mit.edu)\n\nAn \\((n,M)\\) code consists of an encoder assigning each of \\(M\\) messages a length-\\(n\\) input sequence, called a codeword, and a decoder assigning an estimated message to each received sequence. Its rate is\n\n\\[R_n=\\frac{\\log_2 M}{n},\\]\n\nmeasured in bits per channel use. With a uniformly selected message \\(U\\), the average block error probability is \\(P_e^{(n)}=\\Pr(\\widehat U\\ne U)\\). Block error concerns the entire decoded message, rather than the proportion of incorrectly decoded bits. An error-correcting code sacrifices some potential message rate to make different messages distinguishable after transmission. (ocw.mit.edu)"
    },
    {
      "heading": "Mathematical statement",
      "text": "For the finite-alphabet channel described above, capacity is\n\n\\[C=\\max_{P_X} I(X;Y),\\]\n\nwhere the maximization ranges over input probability distributions, and the input and output random variables have joint law \\(P_X(x)W(y\\mid x)\\). Their mutual information is\n\n\\[I(X;Y)= \\sum_{x,y}P_X(x)W(y\\mid x) \\log_2 \\frac{W(y\\mid x)} {\\sum_{x'}P_X(x')W(y\\mid x')}.\\]\n\nEquivalently, \\(I(X;Y)=H(X)-H(X\\mid Y)\\): the input’s entropy minus the conditional entropy remaining after the output is observed. (ocw.mit.edu)\n\nThe theorem has two parts:\n\n- Achievability: For every \\(0<R<C\\), there is a sequence of codes with limiting rate at least \\(R\\) and \\(P_e^{(n)}\\to0\\).\n- Converse: Any sequence of codes with vanishing error probability must have \\(\\limsup_{n\\to\\infty}R_n\\le C\\).\n\nThus capacity is a supremum of reliably achievable rates. The basic statement does not assert that a fixed positive-length code is error-free, nor that operation at exactly \\(C\\) always has vanishing error. For finite-alphabet discrete memoryless channels, the stronger converse states that error tends to one when rates remain bounded strictly above capacity. (ocw.mit.edu)"
    },
    {
      "heading": "Proof principles",
      "text": "Achievability can be established through random coding. Choose an input distribution with \\(I(X;Y)>R\\), then independently generate approximately \\(2^{nR}\\) codewords according to its product distribution. A decoder seeks the unique codeword statistically compatible with the received sequence. The law of large numbers makes the transmitted input-output pair typically compatible, while the chance of an unrelated codeword appearing compatible decreases approximately as \\(2^{-nI(X;Y)}\\). Because \\(R<I(X;Y)\\), the ensemble-average error tends to zero. Consequently, at least one deterministic codebook has small error; fresh random codebooks need not be generated during operation. (ocw.mit.edu)\n\nThe converse uses Fano’s inequality and the data-processing inequality. For a uniformly distributed message,\n\n\\[H(U\\mid Y^n)\\le 1+P_e^{(n)}\\log_2 M.\\]\n\nProcessing through the encoder and channel gives \\(I(U;Y^n)\\le I(X^n;Y^n)\\le nC\\). Combining these bounds yields\n\n\\[(1-P_e^{(n)})R_n\\le C+\\frac1n.\\]\n\nIf error vanishes, the limiting rate cannot exceed capacity. This argument proves the weak converse, not the strong converse by itself. (ocw.mit.edu)"
    },
    {
      "heading": "Representative channels",
      "text": "A binary symmetric channel flips each transmitted bit independently with probability \\(p\\). Its capacity is\n\n\\[C=1-h_2(p),\\qquad h_2(p)=-p\\log_2p-(1-p)\\log_2(1-p).\\]\n\nAt \\(p=1/2\\), output is independent of input and capacity is zero. At \\(p=0\\), capacity is one bit per use. A binary erasure channel, which replaces each bit by a recognizable erasure with probability \\(\\epsilon\\), has capacity \\(1-\\epsilon\\). Unlike an unknown bit flip, an erasure reveals where information was lost. (ocw.mit.edu)\n\nFor an ideal band-limited additive white Gaussian noise channel, the related Shannon–Hartley theorem gives\n\n\\[C=B\\log_2(1+P/N)\\]\n\nbits per second, where \\(B\\) is bandwidth, \\(P\\) is average signal power, and \\(N\\) is noise power within that bandwidth. Its continuous-channel assumptions and power constraint distinguish it from the finite-alphabet formulation. (web.njit.edu)"
    },
    {
      "heading": "Scope and limitations",
      "text": "The theorem is an asymptotic existence result, not an efficient encoding or decoding algorithm. It does not specify the block length needed for a target error probability, decoding cost, or acceptable delay. Finite-blocklength theory studies these additional rate–reliability trade-offs; channel dispersion quantifies an important second-order departure from capacity. (ocw.mit.edu)\n\nTogether with the source coding theorem, channel coding supports source–channel separation under standard point-to-point memoryless assumptions: lossless compression removes source redundancy, while channel coding introduces redundancy for protection. When source entropy per channel use is strictly below capacity, the two stages can provide asymptotically reliable transmission. This separation need not be optimal under short-delay constraints or in general multiuser settings. (ocw.mit.edu)"
    }
  ],
  "links": [
    "information-theory",
    "channel-capacity",
    "claude-shannon",
    "conditional-probability",
    "bit",
    "error-correcting-code",
    "probability-distribution",
    "random-variable",
    "mutual-information",
    "information-entropy",
    "conditional-entropy",
    "law-of-large-numbers",
    "additive-white-gaussian-noise",
    "shannon-hartley-theorem",
    "algorithm",
    "source-coding-theorem",
    "lossless-data-compression"
  ],
  "sources": [
    {
      "title": "A Mathematical Theory of Communication",
      "url": "https://web.njit.edu/~jkliewer/wp/paper/Shannon_1948.pdf"
    },
    {
      "title": "Lecture Notes — Information Theory, Spring 2016",
      "url": "https://www.ocw.mit.edu/courses/6-441-information-theory-spring-2016/pages/lecture-notes/"
    },
    {
      "title": "Information Theory Review",
      "url": "https://web.mit.edu/6.976/www/handout/math_inforev.pdf"
    }
  ],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}