Skip-gram is a machine-learning model that learns numerical representations of words by using each word to predict words occurring near it in a text corpus. Introduced in 2013 by Tomas Mikolov and colleagues as part of word2vec, it produces dense vectors called word embeddings. Words that occur in similar contexts tend to acquire similar representations. The term skip-gram also has an older, broader meaning in text analysis: a sequence of tokens that need not be adjacent. That feature-extraction concept is related to, but distinct from, the predictive embedding model.
Principle and training examples
Skip-gram draws on the distributional hypothesis, according to which a word’s linguistic properties can be inferred partly from the contexts in which it occurs. Rather than encoding a dictionary definition, the model learns patterns of local co-occurrence.
Consider the sentence:
The small cat slept quietly.
With cat as the center word and a context window extending two positions in each direction, the model receives four training pairs:
- (cat, the)
- (cat, small)
- (cat, slept)
- (cat, quietly)
The first element is the input word; the second is a context word to be predicted. Moving the window through the corpus generates many such pairs.
In the standard formulation, left and right context words share the same output vocabulary, and their precise positions are not represented. Consequently, the model learns local association without explicitly modeling word order or syntax. Variants can encode direction, distance, or grammatical relations.
Skip-gram reverses the prediction direction used by continuous bag-of-words (CBOW). CBOW combines surrounding word representations to predict the center word; skip-gram uses the center word to predict surrounding words separately.
Model architecture and objective
Let a corpus contain tokens , and let be the context-window radius. A basic skip-gram objective maximizes
Each observed center–context pair contributes a conditional log-probability. Context predictions are trained separately rather than through a joint model of the entire surrounding sequence.
Two embedding tables
For a vocabulary of size and embedding dimension , the model typically contains two learned tables:
- an input embedding for each center word;
- an output embedding for each context word.
The compatibility of a center word and a context word is measured by the dot product . In the full softmax formulation,
Skip-gram can be represented as a shallow neural network with a linear projection layer. A one-hot input effectively selects a row of the input embedding table; no nonlinear hidden transformation is required.
After training, the input vectors are commonly used as word representations. The output vectors, or combinations of the two tables, can also be used, but they are not interchangeable by definition.
Computational cost
Full softmax requires a sum over the vocabulary for every training pair. Its computational cost grows roughly as per pair, which becomes expensive for large vocabularies.
Practical implementations therefore commonly use hierarchical softmax or negative sampling.
Efficient training methods
Hierarchical softmax
Hierarchical softmax places vocabulary items at the leaves of a binary tree. Predicting a word becomes a sequence of binary decisions along the path from the root to its leaf. The product of the decision probabilities defines the word’s probability.
A balanced tree gives paths of length approximately . Word2vec implementations commonly use a Huffman tree, assigning shorter paths to frequent words. Unlike negative sampling, hierarchical softmax defines a normalized probability distribution over the vocabulary through its tree structure.
Negative sampling
Negative sampling replaces full-vocabulary prediction with a binary discrimination task. For each observed center–context pair , the model learns to distinguish that pair from sampled noise pairs.
With negative samples, an expected objective for one positive pair is
where
and is the noise distribution. In implementation, the expectation is estimated using sampled words.
The positive term increases the compatibility of observed pairs; the negative terms reduce compatibility for sampled noise pairs. A widely used noise distribution is proportional to the unigram frequency raised to the three-quarter power:
This transformation makes the distribution less concentrated on very frequent words than raw unigram sampling.
Skip-gram with negative sampling, usually abbreviated SGNS, does not directly optimize the normalized conditional probabilities of the full-softmax model. It is primarily a representation-learning objective; its scores are not automatically calibrated word probabilities.
Context selection and preprocessing
The training procedure depends on more than the prediction objective.
Window size. Narrow windows emphasize immediate grammatical and lexical relationships. Wider windows often capture broader topical associations. This is a tendency, not a strict division between semantics and grammar.
Dynamic windows. Some implementations randomly vary the effective radius up to a maximum. Nearby words then appear as training contexts more often than distant words.
Frequent-word subsampling. Very frequent tokens may be probabilistically discarded before training. This reduces computation and changes which co-occurrences dominate the learned representation. Removing tokens before windows are formed can also change effective context distances.
Vocabulary thresholds. Words occurring fewer than a specified number of times may be excluded. This reduces memory use but prevents those words from receiving ordinary learned vectors.
Tokenization. Decisions about punctuation, capitalization, sentence boundaries, and word segmentation determine the units and contexts the model encounters. Phrase detection can make recurring multiword expressions single vocabulary entries.
Embedding dimension, the number of negative samples, and the amount of training also affect the results. Larger vectors are not necessarily better when the corpus is too small to estimate them reliably.
Geometry and interpretation
The learned representations occupy a vector space. A common measure of similarity is cosine similarity:
Words with similar contexts often become neighbors, but contextual similarity is not equivalent to synonymy. Antonyms can occur in similar constructions, while words associated with the same topic may be close despite having different meanings.
Some relationships can be approximated by vector offsets. A conventional analogy query searches for a vector near
Such patterns helped popularize word embeddings, but their reliability depends on the relation, corpus, vocabulary, and evaluation procedure. They do not establish that the model performs general symbolic reasoning.
Connection to co-occurrence matrices
SGNS also has a connection to matrix factorization. Under an idealized analysis in which each center–context score can be optimized independently, its optimum satisfies
If the noise distribution equals the observed context-word marginal, this becomes
where pointwise mutual information measures how strongly two events co-occur relative to independence.
This explains why SGNS is often described as implicitly factorizing a shifted PMI matrix. The equivalence has qualifications: finite-dimensional vectors constrain the scores jointly, empirical pair counts depend on preprocessing, and the common three-quarter-power noise distribution introduces a context-dependent adjustment rather than only a constant shift.
Historical development and applications
Mikolov and colleagues introduced skip-gram and CBOW in work on efficient word-representation learning in 2013. Further work that year developed negative sampling and frequent-word subsampling, making training practical on large corpora.
The model became influential because useful word vectors could be learned from unannotated text with comparatively simple computation. These vectors were used as features or initial representations for part-of-speech tagging, document classification, information retrieval, and other language-processing tasks.
Related methods have extended the training principle beyond ordinary word windows:
- Subword models incorporate character sequences, allowing words to share information through spelling components.
- Dependency-based embeddings use grammatical relations rather than only linear proximity.
- Graph embeddings treat nodes encountered on random walks as contexts.
- Cross-lingual methods align or jointly learn representations across languages.
These methods adapt the center–context learning framework; their results depend on how the underlying entities and contexts are defined.
Limitations
Standard skip-gram assigns one vector to each vocabulary item. It therefore combines different senses of a word into a single representation. For a word such as bank, financial and geographical uses are not assigned separate token-specific embeddings. This limits its treatment of polysemy.
Rare words provide few training examples, making their representations less stable. A basic word-level model also cannot directly produce a trained vector for an out-of-vocabulary word. Subword extensions mitigate this limitation but change the representation scheme.
Because skip-gram learns corpus associations, its vectors can reproduce social stereotypes and other imbalances in the training data. Similarity scores describe learned distributional structure, not verified facts about the world.
Results can vary with random initialization, corpus composition, preprocessing, and training settings. Apparent changes between embeddings trained on different corpora therefore require careful interpretation.
Finally, standard skip-gram is neither a complete model of sentence meaning nor an autoregressive language model. Its usual symmetric context window includes both preceding and following words, and SGNS does not supply normalized sequence probabilities. Contextual representation models such as ELMo and large language models instead compute representations that vary with the surrounding text.