{
  "id": "sentencepiece",
  "title": "SentencePiece",
  "lang": "en",
  "source_lang": "en",
  "summary": "SentencePiece is an open-source text tokenization toolkit that learns subword vocabularies directly from text and converts text into token sequences for neural language systems.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "product",
  "aliases": [
    "SentencePiece tokenizer"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/sentencepiece/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/sentencepiece/",
      "json": "https://aiwiki.page/zh/sentencepiece.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "SentencePiece is an open-source toolkit for training and applying text tokenizers, particularly subword tokenizers used in natural language processing. It learns a vocabulary from text and uses that vocabulary to convert strings into pieces or integer token identifiers. Unlike pipelines that require text to be split into words before subword learning, SentencePiece can train directly on sentences without language-specific word segmentation. It supports both byte-pair encoding (BPE) and unigram language-model tokenization, as well as character and word modes."
    },
    {
      "heading": "Purpose and design",
      "text": "Tokenization determines the units of text processed by a language system. Whole-word vocabularies can become very large and cannot directly represent every unfamiliar word, spelling variation, or productive word form. Character-level vocabularies are smaller but generally produce longer sequences. Subword tokenization provides an intermediate representation: common strings can receive their own tokens, while less frequent strings can be decomposed into smaller units.\n\nSentencePiece treats text as a sequence of Unicode characters rather than assuming that spaces already identify words. This is useful for languages whose writing systems do not consistently separate words with spaces, including Chinese and Japanese. It also reduces dependence on external, language-specific preprocessing.\n\nIts principal design features are:\n\n- Training from sentence text: a separate word-segmentation stage is not required.\n- Configurable vocabulary size: training aims to construct a vocabulary with a specified number of entries, subject to model type, input data, and configuration.\n- Explicit representation of spaces: a visible marker allows spaces to participate in piece construction.\n- Integrated normalization: normalization rules are packaged with the tokenizer model.\n- Multiple segmentation algorithms: the same toolkit supports different ways of learning and applying a vocabulary.\n- Sampling support: compatible configurations can generate alternative segmentations of the same text.\n\n“Language-independent” in this context describes the absence of a required language-specific word splitter. It does not mean that a trained vocabulary performs equally well across all languages."
    },
    {
      "heading": "Text representation and normalization",
      "text": "The whitespace marker\n\nSentencePiece commonly represents a space with ▁, the Unicode character U+2581, named LOWER ONE EIGHTH BLOCK. This marker is part of the tokenizer’s internal text representation and may appear within learned pieces.\n\nFor example, an illustrative segmentation of:\n\nHello world\n\ncould be:\n\n▁Hello  ▁world\n\nThe spaces separating the displayed pieces are presentation delimiters, not necessarily characters in the underlying text. A piece such as ▁world combines a space marker with the following letters.\n\nBy default, SentencePiece also adds a dummy prefix space before processing a sentence. Consequently, the first word can receive the same space-prefixed representation that it would have elsewhere in a sentence. The exact output depends on the trained vocabulary and normalization settings.\n\nThis representation enables decoding without guessing where word boundaries belong: the decoder reconstructs the text using the pieces and their space markers.\n\nNormalization and reversibility\n\nBefore segmentation, SentencePiece can apply text normalization. Its standard default uses an NFKC-based normalization scheme, which maps many compatibility characters and character sequences to standardized forms. Normalization rules can be changed or replaced, including with an identity normalizer.\n\nDefault settings also remove leading and trailing whitespace and collapse repeated whitespace. Therefore, encoding followed by decoding does not necessarily reproduce the original input exactly.\n\nThe distinction is important:\n\n- Segmentation can preserve the normalized text when its characters are representable.\n- Normalization can discard distinctions present in the original text.\n- Unknown-character handling and special-token behavior can introduce additional differences.\n\nSentencePiece’s reversible space representation should therefore not be confused with unconditional byte-for-byte preservation of arbitrary input."
    },
    {
      "heading": "Supported tokenization models",
      "text": "Unigram language-model tokenization\n\nThe unigram tokenizer associates each ordinary vocabulary piece with a probability. A candidate segmentation is scored using the product of its piece probabilities.\n\nFor a normalized string \\(x\\) and a segmentation \\(z=(z_1,\\ldots,z_m)\\),\n\n\\[P(z)=\\prod_{i=1}^{m}p(z_i).\\]\n\nThe probability assigned to the string sums over its valid segmentations:\n\n\\[P(x)=\\sum_{z\\in\\mathcal{S}(x)}\\prod_{i=1}^{|z|}p(z_i),\\]\n\nwhere \\(\\mathcal{S}(x)\\) is the set of segmentations allowed by the vocabulary.\n\nTraining begins with a large candidate vocabulary, estimates piece probabilities, and progressively removes pieces according to their contribution to the training objective. This process approaches the requested vocabulary size while retaining necessary characters and reserved entries.\n\nAt encoding time, the tokenizer can select the highest-scoring segmentation using dynamic programming. It can also sample alternative segmentations rather than always selecting a single best result.\n\nThe unigram model here is a model over token pieces. It is distinct from the much larger neural language model that may subsequently consume the resulting token identifiers.\n\nByte-pair encoding\n\nBPE starts from small units and repeatedly merges frequent adjacent pairs. Its learned merge structure determines how text is segmented.\n\nIn SentencePiece’s BPE implementation, spaces participate through the whitespace marker, although configuration options can constrain which boundaries pieces may cross. BPE and unigram tokenization can therefore learn different vocabularies and produce different segmentations from the same corpus.\n\nThe name “byte-pair encoding” does not mean that every SentencePiece BPE model operates directly over raw bytes. Its ordinary text representation is Unicode-based; byte fallback is a separate option.\n\nCharacter and word modes\n\nSentencePiece also supports:\n\n- Character mode, which uses individual characters as ordinary pieces.\n- Word mode, which uses word units and expects appropriately pretokenized input.\n\nThese modes provide simpler alternatives but do not offer the same learned subword decomposition as BPE or unigram tokenization."
    },
    {
      "heading": "Training and model artifacts",
      "text": "A tokenizer is trained on a corpus of training data. Important settings include vocabulary size, model type, character coverage, normalization rules, special symbols, and restrictions on piece construction.\n\nCharacter coverage controls which characters from the training corpus are retained as required characters. It is particularly relevant for writing systems with large character inventories. A tokenizer with limited coverage may encounter characters it cannot represent directly.\n\nTraining ordinarily produces two main artifacts:\n\n- A model file, containing the vocabulary, piece scores or model information, normalization specifications, and configuration needed by the processor.\n- A vocabulary file, providing a human-readable list of pieces and associated scores.\n\nThe model file is the operational artifact used for encoding and decoding; the text vocabulary alone does not capture the complete tokenizer.\n\nSentencePiece provides command-line programs and programming interfaces, including a C++ implementation and Python bindings. The processor can return either piece strings or integer identifiers. These identifiers commonly select rows in a neural network’s embedding table.\n\nA tokenizer and the neural model trained with it are tightly coupled. Replacing the tokenizer, changing the mapping from pieces to identifiers, or modifying normalization generally changes the meaning of the model’s inputs, even if the vocabulary size stays the same."
    },
    {
      "heading": "Special symbols and unseen characters",
      "text": "SentencePiece distinguishes ordinary pieces from symbols with special processing rules.\n\nAn unknown-token entry represents input that cannot otherwise be encoded. Beginning-of-sequence, end-of-sequence, and padding entries can be enabled and assigned identifiers. Control symbols are handled specially and do not behave like ordinary text pieces.\n\nUser-defined symbols serve a different purpose: specified strings can be retained as indivisible pieces when they occur in input. Defining such symbols is useful when a system needs stable representations for designated markers.\n\nAn optional byte fallback mechanism represents otherwise unencodable characters through the bytes of their UTF-8 encoding. This reduces reliance on an unknown token, but may require several tokens for a single character. It does not undo information lost during normalization."
    },
    {
      "heading": "Sampling and subword regularization",
      "text": "Deterministic tokenization gives a text one fixed segmentation. Subword regularization instead exposes a downstream model to multiple segmentations during training.\n\nFor unigram tokenization, SentencePiece can sample segmentations according to their model scores, with parameters controlling the sampling behavior. It also supports BPE-dropout, which introduces randomness by skipping some merge operations.\n\nThese methods act as data augmentation: the visible text remains substantially the same, but its token sequence varies. They can reduce dependence on one particular segmentation and improve robustness in some settings. Their effects depend on the task, corpus, tokenizer, and training procedure; improvements are not universal."
    },
    {
      "heading": "History and applications",
      "text": "Taku Kudo and John Richardson described SentencePiece in a 2018 paper titled SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. Its development was closely associated with neural machine translation and the need to integrate vocabulary learning with reproducible text preprocessing.\n\nThe toolkit is distributed under the Apache License 2.0. It has been used in machine translation, multilingual text processing, and large language models. T5 and the original LLaMA models are examples of systems associated with SentencePiece tokenization.\n\nSentencePiece is not a neural network architecture and does not itself generate translations or text. It supplies the representation layer that converts text into discrete units for a downstream model, including models based on the Transformer architecture."
    },
    {
      "heading": "Limitations and distinctions",
      "text": "Vocabulary quality depends on the corpus. A tokenizer trained predominantly on one language, domain, or writing system may split other text into much longer sequences. Under distribution shift, names, technical notation, informal spelling, and unfamiliar scripts may be represented inefficiently.\n\nToken counts affect computation. Longer token sequences occupy more of a model’s context window and can increase processing costs. Equal amounts of text in different languages need not produce equal numbers of tokens.\n\nPieces are not necessarily linguistic units. A token may be a word, part of a word, punctuation, a space-prefixed fragment, or another frequent string. Its boundary need not coincide with a morpheme or meaningful expression.\n\nReproducibility requires the complete configuration. Specifying only “SentencePiece” is insufficient to reproduce a tokenizer. The corpus, model type, vocabulary, normalization rules, special-symbol definitions, sampling settings, and implementation version can all matter.\n\nThe toolkit is distinct from its algorithms. SentencePiece is not a synonym for BPE, and not every SentencePiece model uses the unigram algorithm. Likewise, tokenizers implemented in other libraries may use similar algorithms without reproducing SentencePiece’s normalization, space representation, or identifier assignments."
    }
  ],
  "links": [
    "open-source-software",
    "natural-language-processing",
    "byte-pair-encoding",
    "tokenization",
    "unicode",
    "dynamic-programming",
    "language-model",
    "training-data",
    "word-embedding",
    "utf-eight",
    "data-augmentation",
    "machine-translation",
    "apache-license",
    "large-language-model",
    "transformer-architecture",
    "distribution-shift",
    "context-window"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}