FastText, usually stylized as fastText, is an open-source library and family of models for learning word embeddings and performing text classification. Developed at Facebook AI Research, it combines computationally inexpensive models with features derived from words and their constituent character sequences. Its best-known contribution is a subword-based representation that allows related word forms to share information and enables vectors to be constructed for words absent from the training vocabulary. FastText also provides a separate, efficient supervised learning system for assigning labels to text.
Origins and scope
FastText emerged from research on efficient natural language processing at Facebook. The 2016 paper Bag of Tricks for Efficient Text Classification, by Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov, described a simple text classifier designed to achieve strong performance at relatively low computational cost. The subword embedding method was presented by Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov in Enriching Word Vectors with Subword Information, released as a preprint in 2016 and published in 2017.
A related line of work, FastText.zip: Compressing Text Classification Models, developed methods for reducing the storage requirements of classifiers. Later work extended the availability of pretrained word vectors to many languages.
The name can therefore refer to several closely related things:
- The software library and command-line tools.
- The subword-based word embedding method.
- The lightweight text classification architecture.
- Pretrained embedding or classification models distributed through the project.
These uses share implementation techniques but are not identical. In particular, a FastText classifier does not necessarily use character-level subword features: these are configurable and are disabled by default in the standard supervised configuration.
Subword word embeddings
Relationship to Word2Vec
FastText extends the general approach of Word2Vec, which learns word vectors from patterns of word co-occurrence. It supports two familiar training architectures:
- Skip-gram, which predicts nearby words from a target word.
- Continuous bag of words, or CBOW, which predicts a target word from its surrounding context.
In ordinary word-level models, each vocabulary item has an independently learned representation. This makes it difficult to learn useful vectors for rare forms and leaves words outside the vocabulary without a directly available representation.
FastText addresses this by representing a word using both a whole-word component and components associated with character n-grams.
Character n-grams
A character n-gram is a consecutive sequence of characters of a specified length. FastText adds boundary markers to distinguish sequences at the beginning and end of a word from otherwise identical internal sequences.
For example, the boundary-marked form <cat> contains character trigrams such as <ca, cat, and at>. In the usual word-vector training configuration, FastText uses character n-grams of lengths three through six, although these limits are adjustable.
A simplified representation is
where contains the subword features associated with word , together with its whole-word feature when available, and is the learned vector for feature . Depending on the operation, the software averages these contributions; the central property is that words share parameters through overlapping character sequences.
For skip-gram training, a target word and context word can be assigned the score
where is an output-side context vector. Training makes observed target–context pairs more compatible than sampled alternatives.
This sharing is particularly useful for inflected words, derived forms, and uncommon vocabulary. Related forms can benefit from overlapping spelling patterns even when some of them occur infrequently in the training data.
Out-of-vocabulary words
FastText can construct a vector for an unseen word by combining its character n-gram vectors. No separately trained whole-word vector is required.
This capability does not guarantee that an unseen word receives an accurate meaning. The quality of its representation depends on whether its subword features received useful training and whether orthographic similarity reflects semantic similarity. Unrelated words can share character sequences, while semantically related words can have very different spellings.
FastText subwords are also not necessarily linguistic morphemes. Unlike byte-pair encoding, the method does not learn a vocabulary of variable-length units through repeated merging. It extracts overlapping character sequences within specified length limits.
Text classification
FastText's supervised classifier maps an input text to one or more labels. Applications include topic classification, sentiment classification, and language identification.
Its basic architecture resembles a bag-of-words model with learned dense feature vectors:
- Extract words and, optionally, word n-grams and character n-grams.
- Look up a vector for each feature.
- Average the feature vectors into a text representation.
- Apply a linear output layer to score labels.
If is the set or sequence of features extracted from text , its hidden representation can be written as
For a softmax classifier, the probability of label is
The feature embeddings and output parameters are learned jointly. This is a shallow architecture rather than a deep learning system with many stacked layers.
Word n-grams preserve limited local ordering information. For example, a bigram feature can distinguish “not good” from the independent occurrence of “not” and “good.” However, averaging does not explicitly model arbitrary long-range relationships or full sentence structure.
Different output objectives support different classification settings. Softmax is suited to mutually exclusive labels, while a one-versus-all objective supports classification settings in which several labels may apply to the same text. Hierarchical softmax is another option, particularly when the label inventory is large.
Efficiency and compression
Feature hashing
The number of possible subword features is extremely large. FastText maps many of them into a fixed number of buckets using a hash function, avoiding an explicit dictionary entry for every observed character sequence.
This bounds the size of the feature table, but distinct features can collide and share a vector. The bucket count therefore affects memory use and the amount of unintended parameter sharing.
Training and output computation
FastText uses simple vector operations and multithreaded implementations that run efficiently on a CPU. It does not require a GPU.
For word embeddings, negative sampling avoids computing scores for the entire output vocabulary at every training step. Hierarchical softmax instead organizes output choices in a tree, replacing a flat prediction problem with a sequence of decisions along a path.
These techniques make FastText attractive when training or prediction must be economical. Nevertheless, embedding dimension, vocabulary size, subword bucket count, and number of output labels can still produce substantial memory requirements.
Quantization
FastText supports compact classifiers through quantization, including product quantization of parameter vectors. Rather than storing every vector in full precision, the compressed model represents portions of vectors using learned codebooks. Feature pruning can further reduce model size.
Compression introduces a trade-off between storage, prediction cost, and accuracy. Its effect depends on the data and compression settings; smaller models are not guaranteed to preserve all of the original model's performance.
Software and pretrained resources
The main implementation is written in C++ and provides command-line tools and Python bindings. Common operations include training embeddings, training a classifier, evaluating predictions, finding nearby words, and obtaining vectors for individual words.
Supervised training data conventionally places labels and text on the same line, using a label prefix such as __label__.
FastText distinguishes between a full saved model and an exported vector table:
- A binary model preserves learned parameters and feature information needed for operations such as constructing vectors for unseen words.
- A text vector file normally contains explicit vectors for listed vocabulary items. That file alone does not retain the complete subword model.
The project has distributed pretrained vectors for many languages, including widely used 300-dimensional vectors trained on Wikipedia and Common Crawl text. It has also distributed language-identification models.
Multilingual availability should not be confused with cross-lingual alignment. Independently trained vectors for different languages do not automatically share a coordinate system in which translation equivalents are close together.
Applications and limitations
FastText embeddings can serve as features for information retrieval, document analysis, and other language-processing tasks. Its classifier is useful for high-throughput labeling and as a baseline against which more elaborate models can be evaluated.
Its main strengths are efficient computation, parameter sharing across word forms, and the ability to represent unseen words. These strengths are particularly relevant to morphologically rich languages and sparse vocabularies.
Important limitations include:
- Static representations. In its ordinary use, a word receives the same vector regardless of the sentence in which it appears. Distinct meanings of an ambiguous word are therefore conflated.
- Limited compositional modeling. Averaged features and short word n-grams do not directly capture complex syntax, extended negation, or long-distance dependencies.
- Dependence on preprocessing. Tokenization, normalization, and handling of punctuation affect the features the model can learn. Languages without whitespace-delimited words require particular attention to segmentation.
- Imperfect subword sharing. Spelling overlap can transfer useful information, but it can also connect unrelated forms. Orthographic units need not correspond to meaningful linguistic units.
- Corpus dependence. Rare usage, poorly represented languages, biased associations, and distribution shift can reduce reliability.
Unlike contextual models based on the Transformer architecture, FastText does not generate a context-dependent representation of every token through attention over a sequence. It is not a large language model or a general-purpose text generator. Its primary role is efficient representation learning and text classification, with quality determined by the task, training corpus, feature configuration, and evaluation conditions.