{
  "id": "byte-pair-encoding",
  "title": "Byte-pair encoding",
  "lang": "en",
  "source_lang": "en",
  "summary": "Byte-pair encoding is a repeated pair-substitution algorithm used for lossless compression and, in adapted form, for learning subword vocabularies in language processing.",
  "category": "computer-science",
  "subcategory": "algorithms-theory",
  "type": "concept",
  "aliases": [
    "BPE",
    "Byte pair encoding"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/byte-pair-encoding/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/byte-pair-encoding/",
      "json": "https://aiwiki.page/zh/byte-pair-encoding.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Byte-pair encoding (BPE) is an algorithm that repeatedly replaces a frequent pair of adjacent symbols with a new symbol representing that pair. Originally developed for lossless data compression, it was later adapted to learn vocabularies for tokenization in natural language processing. In this adapted form, BPE represents text as a sequence of variable-length units, often called subwords, rather than relying exclusively on whole words or individual characters. It is widely used in machine translation and large language models.\n\nDespite its name, a BPE tokenizer need not operate directly on bytes: its initial symbols may be characters, bytes, or another predefined alphabet. The compression algorithm and the language-processing adaptation share the same pair-merging principle but use the learned replacements differently."
    },
    {
      "heading": "Origins and development",
      "text": "Philip Gage described byte-pair encoding in the 1994 article “A New Algorithm for Data Compression.” The compression method replaces frequently occurring byte pairs with otherwise unused byte values and records the substitutions so that the original data can be reconstructed.\n\nRico Sennrich, Barry Haddow, and Alexandra Birch adapted BPE to neural machine translation in work published in 2016 under the title “Neural Machine Translation of Rare Words with Subword Units.” Their approach addressed a limitation of fixed word vocabularies: uncommon words, names, and newly formed words could otherwise be replaced with an unknown-word symbol. BPE instead allowed such words to be represented as sequences of smaller, reusable units.\n\nSubsequent language-model tokenizers adopted related methods. GPT-2, introduced in 2019, used byte-level BPE, combining an alphabet covering every possible byte value with learned merges. BPE consequently became an important component of many generative pre-trained transformer systems, although language models also use other tokenization algorithms."
    },
    {
      "heading": "Basic principle",
      "text": "BPE begins with a sequence, or collection of sequences, expressed in an initial alphabet. It then repeatedly:\n\n- Counts adjacent symbol pairs.\n- Selects a most frequent eligible pair.\n- Creates a symbol representing the concatenation of that pair.\n- Replaces non-overlapping occurrences of the pair.\n- Updates the pair counts and continues until a stopping condition is reached.\n\nA newly created symbol may itself participate in later merges. Thus, the algorithm builds larger units from smaller ones.\n\nFor a tokenization corpus, let \\(f_t(a,b)\\) denote the frequency of adjacent symbols \\(a,b\\) after \\(t\\) merge steps. A standard frequency-based rule selects\n\n\\[(a_t,b_t)\\in\\operatorname*{arg\\,max}_{(a,b)} f_t(a,b).\\]\n\nThe selected pair becomes a new symbol \\(c_t=a_tb_t\\), where juxtaposition denotes concatenation. Tie-breaking rules must be fixed if training is to be reproducible.\n\nPair counting and replacement require some care. In a sequence such as aaa, the pair aa occurs at two overlapping positions, but both occurrences cannot be replaced simultaneously. Implementations therefore need consistent rules for counting overlaps and selecting non-overlapping replacements.\n\nIllustrative example\n\nConsider the following corpus, with each word initially split into characters:\n\nlow   → l o w\nlow   → l o w\nlower → l o w e r\n\nThe pairs l o and o w each occur three times. If the tie-breaking rule selects l o, the sequences become:\n\nlo w\nlo w\nlo w e r\n\nThe next most frequent pair is lo w, producing:\n\nlow\nlow\nlow e r\n\nThe token low is now available both as a complete word and as part of a longer word. Further merges depend on the rest of the corpus and the stopping condition.\n\nThis example omits word-boundary markers and other preprocessing conventions. Practical tokenizers often include such conventions, which affect the learned vocabulary."
    },
    {
      "heading": "BPE for data compression",
      "text": "In its compression role, BPE operates on a block of data. A frequent pair of bytes is replaced by an available byte value that serves as a reference to the pair. The encoder records a substitution table alongside the transformed data.\n\nFor example, if an unused symbol X is assigned to the pair ab, then:\n\nababab → XXX\n\nA later replacement may refer to symbols introduced by earlier replacements. Decoding expands these references in the appropriate order until the original bytes have been recovered.\n\nThe method is lossless because the substitution table preserves the information needed to reverse each replacement. However, a shorter substituted sequence does not automatically mean a smaller compressed file: the substitution table and other bookkeeping also occupy space.\n\nThe finite byte alphabet constrains how many replacement symbols can be introduced within a block. Block organization and symbol availability therefore matter in the original compression setting.\n\nBPE differs from Huffman coding and arithmetic coding. Those methods assign representations according to symbol probabilities, whereas BPE primarily constructs new symbols for repeated adjacent sequences. These ideas can be combined within broader data compression systems."
    },
    {
      "heading": "BPE for text tokenization",
      "text": "In language processing, BPE is generally used to learn a reusable vocabulary from training data. The resulting tokenizer is then kept fixed while a model processes text.\n\nUnlike the original byte-oriented compressor, a tokenizer typically assigns integer identifiers to vocabulary entries. It is not restricted to encoding every newly introduced symbol as a single byte.\n\nLearning a vocabulary\n\nTraining usually involves:\n\n- Preprocessing: optionally normalizing text, separating it into preliminary pieces, or marking boundaries.\n- Initializing the alphabet: representing each piece as characters, bytes, or other basic symbols.\n- Learning merges: repeatedly selecting and merging frequent adjacent pairs.\n- Saving the tokenizer: preserving the vocabulary, ordered merge rules, and preprocessing configuration.\n\nThe target vocabulary size is a hyperparameter. If the initial alphabet contains \\(|V_0|\\) symbols and each of \\(M\\) merges introduces a distinct entry, the resulting vocabulary contains\n\n\\[|V|=|V_0|+M\\]\n\nentries before separately accounting for special tokens.\n\nPre-tokenization constrains which units can be learned. If input is split at whitespace and merges cannot cross those boundaries, BPE cannot create tokens spanning multiple words. Other systems attach spaces to neighboring text or permit broader sequences.\n\nEncoding new text\n\nAt inference time, standard BPE applies the learned merge rules in their recorded priority order. An equivalent implementation can repeatedly apply the highest-priority eligible merge currently present.\n\nThis is not generally the same as choosing the longest matching vocabulary entry from left to right. The merge history determines how competing segmentations are resolved; the vocabulary alone may not fully specify the encoder.\n\nThe final pieces are mapped to token identifiers, which are used to retrieve embedding vectors for a language model.\n\nDecoding\n\nFor ordinary text tokens, decoding maps identifiers back to their associated strings or byte sequences and concatenates them. Boundary markers and byte representations are reversed according to the tokenizer’s conventions.\n\nExact round-trip recovery depends on the complete pipeline, not merely on BPE. Lowercasing, lossy normalization, or discarded whitespace can prevent recovery of the original input even when the pair-merging stage itself is reversible."
    },
    {
      "heading": "Character-level and byte-level BPE",
      "text": "Character-based BPE\n\nCharacter-based BPE starts from an alphabet of text characters, commonly Unicode code points. Frequent character sequences become tokens, while less common words remain expressible as smaller units if all their constituent characters are supported.\n\nAn unseen character may require an unknown token or a fallback mechanism. Consequently, character-based BPE does not automatically guarantee coverage of arbitrary text.\n\nHere, “character” need not mean a complete user-perceived written symbol. A displayed symbol may consist of multiple Unicode code points.\n\nByte-level BPE\n\nByte-level BPE first converts text to bytes, commonly using UTF-8, and learns merges over that representation. With all 256 byte values available as base symbols, any byte sequence can be represented without an unknown-text token.\n\nThis provides broad coverage across languages, symbols, and unusual strings. However:\n\n- A token may contain only part of a multibyte character.\n- An individual token’s bytes may not independently decode as valid UTF-8.\n- Languages and writing systems can require different numbers of tokens for comparable content.\n\nSome implementations map bytes to visible Unicode symbols internally. Those symbols are a representation convention, not evidence that merging begins from ordinary written characters."
    },
    {
      "heading": "Applications and trade-offs",
      "text": "BPE provides a compromise between word-level and character-level representation.\n\nA whole-word vocabulary can represent frequent words compactly but grows large and handles unfamiliar words poorly. A character or byte vocabulary is small and broadly reusable but produces longer sequences. BPE allocates vocabulary entries to frequent sequences while retaining smaller units for less familiar text.\n\nIts main applications include machine translation, text generation, and other deep learning systems that process discrete text sequences.\n\nVocabulary size affects both the tokenizer and the model:\n\n- Larger vocabularies often shorten sequences but enlarge embedding and output layers.\n- Smaller vocabularies reduce vocabulary-dependent parameters but usually produce longer sequences.\n- Corpus composition influences which languages, domains, and spelling patterns receive compact representations.\n\nFor models using a transformer architecture, sequence length affects computation and memory use. Tokenization also determines how much text fits within a fixed context window. A token budget therefore does not correspond to a fixed number of words or characters."
    },
    {
      "heading": "Limitations",
      "text": "Frequency is not linguistic structure\n\nBPE learns frequent adjacent sequences, not grammatical or semantic units. A token may coincide with a word, stem, or suffix, but it can also be an arbitrary fragment. Segmentation need not follow pronunciation, morphology, or meaning.\n\nUneven language and domain coverage\n\nA vocabulary learned mainly from one language or domain may fragment other material more heavily. Specialized terminology, unfamiliar scripts, and source code can require more tokens than equally long text resembling the tokenizer’s training corpus.\n\nSuch differences can affect processing cost and usable context length. They reflect both the algorithm and the data and preprocessing used to train it.\n\nSensitivity to surface form\n\nWhitespace, capitalization, spelling, and Unicode representation can change segmentation. Visually similar strings may receive different token sequences, while a small edit can alter several nearby token boundaries.\n\nThe tokenizer may also become less efficient under distribution shift, even when it retains complete byte-level coverage.\n\nGreedy local decisions\n\nBPE selects merges according to local pair frequencies. It does not generally find a globally optimal vocabulary or minimize a downstream model’s prediction loss. Vocabulary size, merge count, and corpus frequency are useful design controls, not guarantees of the best possible representation.\n\nImplementation cost\n\nA naive trainer repeatedly rescans the corpus after each merge, making training expensive at large scale. More efficient implementations maintain pair counts and occurrence indexes in specialized data structures, updating affected regions after replacements. Their time complexity and memory requirements depend on these implementation choices."
    },
    {
      "heading": "Related tokenization methods",
      "text": "WordPiece also builds a subword vocabulary but differs in how candidate units are selected and how text is commonly segmented. WordPiece tokenizers are associated with models such as BERT; BPE merge ranks and WordPiece vocabulary matching should not be treated as interchangeable.\n\nUnigram tokenization assigns probabilities to candidate pieces and selects segmentations under a probabilistic model. A common training approach starts with a large candidate vocabulary and prunes it, rather than constructing the vocabulary solely through successive pair merges.\n\nSentencePiece is a tokenization toolkit supporting both BPE and unigram models. It is not itself a synonym for either algorithm and can learn tokenizers without requiring a separate word-segmentation stage.\n\nBPE-dropout is a stochastic variation that skips some otherwise applicable merges during segmentation. It can expose a model to alternative segmentations of the same text during training rather than always using one deterministic BPE sequence."
    }
  ],
  "links": [
    "algorithm",
    "lossless-data-compression",
    "tokenization",
    "natural-language-processing",
    "machine-translation",
    "large-language-model",
    "generative-pre-trained-transformer",
    "byte",
    "huffman-coding",
    "arithmetic-coding",
    "data-compression",
    "training-data",
    "hyperparameter",
    "word-embedding",
    "language-model",
    "unicode",
    "utf-eight",
    "deep-learning",
    "transformer-architecture",
    "context-window",
    "source-code",
    "distribution-shift",
    "data-structure",
    "time-complexity",
    "bert",
    "sentencepiece"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}