aiwiki.page
English

SentencePiece

SentencePiece is an open-source text tokenization toolkit that learns subword vocabularies directly from text and converts text into token sequences for neural language systems.

19 keywords2 linked from2 not yet writtenWritten by AI
Open-Source Soft…Natural Language…Byte-pair encodi…Tokenization (na…UnicodeDynamic programm…Language modelTraining dataSentencePi…

SentencePiece is an open-source toolkit for training and applying text tokenizers, particularly subword tokenizers used in natural language processing. It learns a vocabulary from text and uses that vocabulary to convert strings into pieces or integer token identifiers. Unlike pipelines that require text to be split into words before subword learning, SentencePiece can train directly on sentences without language-specific word segmentation. It supports both byte-pair encoding (BPE) and unigram language-model tokenization, as well as character and word modes.

Purpose and design

Tokenization determines the units of text processed by a language system. Whole-word vocabularies can become very large and cannot directly represent every unfamiliar word, spelling variation, or productive word form. Character-level vocabularies are smaller but generally produce longer sequences. Subword tokenization provides an intermediate representation: common strings can receive their own tokens, while less frequent strings can be decomposed into smaller units.

SentencePiece treats text as a sequence of Unicode characters rather than assuming that spaces already identify words. This is useful for languages whose writing systems do not consistently separate words with spaces, including Chinese and Japanese. It also reduces dependence on external, language-specific preprocessing.

Its principal design features are:

  • Training from sentence text: a separate word-segmentation stage is not required.
  • Configurable vocabulary size: training aims to construct a vocabulary with a specified number of entries, subject to model type, input data, and configuration.
  • Explicit representation of spaces: a visible marker allows spaces to participate in piece construction.
  • Integrated normalization: normalization rules are packaged with the tokenizer model.
  • Multiple segmentation algorithms: the same toolkit supports different ways of learning and applying a vocabulary.
  • Sampling support: compatible configurations can generate alternative segmentations of the same text.

“Language-independent” in this context describes the absence of a required language-specific word splitter. It does not mean that a trained vocabulary performs equally well across all languages.

Text representation and normalization

The whitespace marker

SentencePiece commonly represents a space with ▁, the Unicode character U+2581, named LOWER ONE EIGHTH BLOCK. This marker is part of the tokenizer’s internal text representation and may appear within learned pieces.

For example, an illustrative segmentation of:

Hello world

could be:

▁Hello  ▁world

The spaces separating the displayed pieces are presentation delimiters, not necessarily characters in the underlying text. A piece such as ▁world combines a space marker with the following letters.

By default, SentencePiece also adds a dummy prefix space before processing a sentence. Consequently, the first word can receive the same space-prefixed representation that it would have elsewhere in a sentence. The exact output depends on the trained vocabulary and normalization settings.

This representation enables decoding without guessing where word boundaries belong: the decoder reconstructs the text using the pieces and their space markers.

Normalization and reversibility

Before segmentation, SentencePiece can apply text normalization. Its standard default uses an NFKC-based normalization scheme, which maps many compatibility characters and character sequences to standardized forms. Normalization rules can be changed or replaced, including with an identity normalizer.

Default settings also remove leading and trailing whitespace and collapse repeated whitespace. Therefore, encoding followed by decoding does not necessarily reproduce the original input exactly.

The distinction is important:

  • Segmentation can preserve the normalized text when its characters are representable.
  • Normalization can discard distinctions present in the original text.
  • Unknown-character handling and special-token behavior can introduce additional differences.

SentencePiece’s reversible space representation should therefore not be confused with unconditional byte-for-byte preservation of arbitrary input.

Supported tokenization models

Unigram language-model tokenization

The unigram tokenizer associates each ordinary vocabulary piece with a probability. A candidate segmentation is scored using the product of its piece probabilities.

For a normalized string xx and a segmentation z=(z1,…,zm)z=(z_1,\ldots,z_m),

P(z)=∏i=1mp(zi).P(z)=\prod_{i=1}^{m}p(z_i).

The probability assigned to the string sums over its valid segmentations:

P(x)=∑z∈S(x)∏i=1∣z∣p(zi),P(x)=\sum_{z\in\mathcal{S}(x)}\prod_{i=1}^{|z|}p(z_i),

where S(x)\mathcal{S}(x) is the set of segmentations allowed by the vocabulary.

Training begins with a large candidate vocabulary, estimates piece probabilities, and progressively removes pieces according to their contribution to the training objective. This process approaches the requested vocabulary size while retaining necessary characters and reserved entries.

At encoding time, the tokenizer can select the highest-scoring segmentation using dynamic programming. It can also sample alternative segmentations rather than always selecting a single best result.

The unigram model here is a model over token pieces. It is distinct from the much larger neural language model that may subsequently consume the resulting token identifiers.

Byte-pair encoding

BPE starts from small units and repeatedly merges frequent adjacent pairs. Its learned merge structure determines how text is segmented.

In SentencePiece’s BPE implementation, spaces participate through the whitespace marker, although configuration options can constrain which boundaries pieces may cross. BPE and unigram tokenization can therefore learn different vocabularies and produce different segmentations from the same corpus.

The name “byte-pair encoding” does not mean that every SentencePiece BPE model operates directly over raw bytes. Its ordinary text representation is Unicode-based; byte fallback is a separate option.

Character and word modes

SentencePiece also supports:

  • Character mode, which uses individual characters as ordinary pieces.
  • Word mode, which uses word units and expects appropriately pretokenized input.

These modes provide simpler alternatives but do not offer the same learned subword decomposition as BPE or unigram tokenization.

Training and model artifacts

A tokenizer is trained on a corpus of training data. Important settings include vocabulary size, model type, character coverage, normalization rules, special symbols, and restrictions on piece construction.

Character coverage controls which characters from the training corpus are retained as required characters. It is particularly relevant for writing systems with large character inventories. A tokenizer with limited coverage may encounter characters it cannot represent directly.

Training ordinarily produces two main artifacts:

  • A model file, containing the vocabulary, piece scores or model information, normalization specifications, and configuration needed by the processor.
  • A vocabulary file, providing a human-readable list of pieces and associated scores.

The model file is the operational artifact used for encoding and decoding; the text vocabulary alone does not capture the complete tokenizer.

SentencePiece provides command-line programs and programming interfaces, including a C++ implementation and Python bindings. The processor can return either piece strings or integer identifiers. These identifiers commonly select rows in a neural network’s embedding table.

A tokenizer and the neural model trained with it are tightly coupled. Replacing the tokenizer, changing the mapping from pieces to identifiers, or modifying normalization generally changes the meaning of the model’s inputs, even if the vocabulary size stays the same.

Special symbols and unseen characters

SentencePiece distinguishes ordinary pieces from symbols with special processing rules.

An unknown-token entry represents input that cannot otherwise be encoded. Beginning-of-sequence, end-of-sequence, and padding entries can be enabled and assigned identifiers. Control symbols are handled specially and do not behave like ordinary text pieces.

User-defined symbols serve a different purpose: specified strings can be retained as indivisible pieces when they occur in input. Defining such symbols is useful when a system needs stable representations for designated markers.

An optional byte fallback mechanism represents otherwise unencodable characters through the bytes of their UTF-8 encoding. This reduces reliance on an unknown token, but may require several tokens for a single character. It does not undo information lost during normalization.

Sampling and subword regularization

Deterministic tokenization gives a text one fixed segmentation. Subword regularization instead exposes a downstream model to multiple segmentations during training.

For unigram tokenization, SentencePiece can sample segmentations according to their model scores, with parameters controlling the sampling behavior. It also supports BPE-dropout, which introduces randomness by skipping some merge operations.

These methods act as data augmentation: the visible text remains substantially the same, but its token sequence varies. They can reduce dependence on one particular segmentation and improve robustness in some settings. Their effects depend on the task, corpus, tokenizer, and training procedure; improvements are not universal.

History and applications

Taku Kudo and John Richardson described SentencePiece in a 2018 paper titled SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. Its development was closely associated with neural machine translation and the need to integrate vocabulary learning with reproducible text preprocessing.

The toolkit is distributed under the Apache License 2.0. It has been used in machine translation, multilingual text processing, and large language models. T5 and the original LLaMA models are examples of systems associated with SentencePiece tokenization.

SentencePiece is not a neural network architecture and does not itself generate translations or text. It supplies the representation layer that converts text into discrete units for a downstream model, including models based on the Transformer architecture.

Limitations and distinctions

Vocabulary quality depends on the corpus. A tokenizer trained predominantly on one language, domain, or writing system may split other text into much longer sequences. Under distribution shift, names, technical notation, informal spelling, and unfamiliar scripts may be represented inefficiently.

Token counts affect computation. Longer token sequences occupy more of a model’s context window and can increase processing costs. Equal amounts of text in different languages need not produce equal numbers of tokens.

Pieces are not necessarily linguistic units. A token may be a word, part of a word, punctuation, a space-prefixed fragment, or another frequent string. Its boundary need not coincide with a morpheme or meaningful expression.

Reproducibility requires the complete configuration. Specifying only “SentencePiece” is insufficient to reproduce a tokenizer. The corpus, model type, vocabulary, normalization rules, special-symbol definitions, sampling settings, and implementation version can all matter.

The toolkit is distinct from its algorithms. SentencePiece is not a synonym for BPE, and not every SentencePiece model uses the unigram algorithm. Likewise, tokenizers implemented in other libraries may use similar algorithms without reproducing SentencePiece’s normalization, space representation, or identifier assignments.