{
  "id": "skip-gram",
  "title": "Skip-gram",
  "lang": "en",
  "source_lang": "en",
  "summary": "Skip-gram is a word-embedding model that learns vector representations by predicting surrounding words from a target word.",
  "category": "language",
  "subcategory": "",
  "type": "concept",
  "aliases": [
    "Skip-gram model",
    "continuous skip-gram model"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/skip-gram/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/skip-gram/",
      "json": "https://aiwiki.page/zh/skip-gram.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Skip-gram is a machine-learning model that learns numerical representations of words by using each word to predict words occurring near it in a text corpus. Introduced in 2013 by Tomas Mikolov and colleagues as part of word2vec, it produces dense vectors called word embeddings. Words that occur in similar contexts tend to acquire similar representations. The term skip-gram also has an older, broader meaning in text analysis: a sequence of tokens that need not be adjacent. That feature-extraction concept is related to, but distinct from, the predictive embedding model."
    },
    {
      "heading": "Principle and training examples",
      "text": "Skip-gram draws on the distributional hypothesis, according to which a word’s linguistic properties can be inferred partly from the contexts in which it occurs. Rather than encoding a dictionary definition, the model learns patterns of local co-occurrence.\n\nConsider the sentence:\n\nThe small cat slept quietly.\n\nWith cat as the center word and a context window extending two positions in each direction, the model receives four training pairs:\n\n- (cat, the)\n- (cat, small)\n- (cat, slept)\n- (cat, quietly)\n\nThe first element is the input word; the second is a context word to be predicted. Moving the window through the corpus generates many such pairs.\n\nIn the standard formulation, left and right context words share the same output vocabulary, and their precise positions are not represented. Consequently, the model learns local association without explicitly modeling word order or syntax. Variants can encode direction, distance, or grammatical relations.\n\nSkip-gram reverses the prediction direction used by continuous bag-of-words (CBOW). CBOW combines surrounding word representations to predict the center word; skip-gram uses the center word to predict surrounding words separately."
    },
    {
      "heading": "Model architecture and objective",
      "text": "Let a corpus contain tokens \\(w_1,\\ldots,w_T\\), and let \\(c\\) be the context-window radius. A basic skip-gram objective maximizes\n\n\\[\\mathcal{L} = \\sum_{t=1}^{T} \\sum_{\\substack{-c\\leq j\\leq c\\\\j\\neq 0\\\\1\\leq t+j\\leq T}} \\log P(w_{t+j}\\mid w_t).\\]\n\nEach observed center–context pair contributes a conditional log-probability. Context predictions are trained separately rather than through a joint model of the entire surrounding sequence.\n\nTwo embedding tables\n\nFor a vocabulary of size \\(|V|\\) and embedding dimension \\(d\\), the model typically contains two learned tables:\n\n- an input embedding \\(v_w\\in\\mathbb{R}^{d}\\) for each center word;\n- an output embedding \\(u_w\\in\\mathbb{R}^{d}\\) for each context word.\n\nThe compatibility of a center word \\(w\\) and a context word \\(o\\) is measured by the dot product \\(u_o^\\top v_w\\). In the full softmax formulation,\n\n\\[P(o\\mid w) = \\frac{\\exp(u_o^\\top v_w)} {\\sum_{x\\in V}\\exp(u_x^\\top v_w)}.\\]\n\nSkip-gram can be represented as a shallow neural network with a linear projection layer. A one-hot input effectively selects a row of the input embedding table; no nonlinear hidden transformation is required.\n\nAfter training, the input vectors are commonly used as word representations. The output vectors, or combinations of the two tables, can also be used, but they are not interchangeable by definition.\n\nComputational cost\n\nFull softmax requires a sum over the vocabulary for every training pair. Its computational cost grows roughly as \\(O(|V|d)\\) per pair, which becomes expensive for large vocabularies.\n\nPractical implementations therefore commonly use hierarchical softmax or negative sampling."
    },
    {
      "heading": "Efficient training methods",
      "text": "Hierarchical softmax\n\nHierarchical softmax places vocabulary items at the leaves of a binary tree. Predicting a word becomes a sequence of binary decisions along the path from the root to its leaf. The product of the decision probabilities defines the word’s probability.\n\nA balanced tree gives paths of length approximately \\(O(\\log |V|)\\). Word2vec implementations commonly use a Huffman tree, assigning shorter paths to frequent words. Unlike negative sampling, hierarchical softmax defines a normalized probability distribution over the vocabulary through its tree structure.\n\nNegative sampling\n\nNegative sampling replaces full-vocabulary prediction with a binary discrimination task. For each observed center–context pair \\((w,o)\\), the model learns to distinguish that pair from sampled noise pairs.\n\nWith \\(k\\) negative samples, an expected objective for one positive pair is\n\n\\[\\ell(w,o) = \\log \\sigma(u_o^\\top v_w) + k\\,\\mathbb{E}_{n\\sim P_n} \\left[ \\log \\sigma(-u_n^\\top v_w) \\right],\\]\n\nwhere\n\n\\[\\sigma(x)=\\frac{1}{1+\\exp(-x)}\\]\n\nand \\(P_n\\) is the noise distribution. In implementation, the expectation is estimated using sampled words.\n\nThe positive term increases the compatibility of observed pairs; the negative terms reduce compatibility for sampled noise pairs. A widely used noise distribution is proportional to the unigram frequency raised to the three-quarter power:\n\n\\[P_n(w) = \\frac{f(w)^{3/4}} {\\sum_{x\\in V}f(x)^{3/4}}.\\]\n\nThis transformation makes the distribution less concentrated on very frequent words than raw unigram sampling.\n\nSkip-gram with negative sampling, usually abbreviated SGNS, does not directly optimize the normalized conditional probabilities of the full-softmax model. It is primarily a representation-learning objective; its scores are not automatically calibrated word probabilities."
    },
    {
      "heading": "Context selection and preprocessing",
      "text": "The training procedure depends on more than the prediction objective.\n\nWindow size. Narrow windows emphasize immediate grammatical and lexical relationships. Wider windows often capture broader topical associations. This is a tendency, not a strict division between semantics and grammar.\n\nDynamic windows. Some implementations randomly vary the effective radius up to a maximum. Nearby words then appear as training contexts more often than distant words.\n\nFrequent-word subsampling. Very frequent tokens may be probabilistically discarded before training. This reduces computation and changes which co-occurrences dominate the learned representation. Removing tokens before windows are formed can also change effective context distances.\n\nVocabulary thresholds. Words occurring fewer than a specified number of times may be excluded. This reduces memory use but prevents those words from receiving ordinary learned vectors.\n\nTokenization. Decisions about punctuation, capitalization, sentence boundaries, and word segmentation determine the units and contexts the model encounters. Phrase detection can make recurring multiword expressions single vocabulary entries.\n\nEmbedding dimension, the number of negative samples, and the amount of training also affect the results. Larger vectors are not necessarily better when the corpus is too small to estimate them reliably."
    },
    {
      "heading": "Geometry and interpretation",
      "text": "The learned representations occupy a vector space. A common measure of similarity is cosine similarity:\n\n\\[\\operatorname{sim}(a,b) = \\frac{v_a^\\top v_b} {\\|v_a\\|\\,\\|v_b\\|}.\\]\n\nWords with similar contexts often become neighbors, but contextual similarity is not equivalent to synonymy. Antonyms can occur in similar constructions, while words associated with the same topic may be close despite having different meanings.\n\nSome relationships can be approximated by vector offsets. A conventional analogy query searches for a vector near\n\n\\[v_b-v_a+v_c.\\]\n\nSuch patterns helped popularize word embeddings, but their reliability depends on the relation, corpus, vocabulary, and evaluation procedure. They do not establish that the model performs general symbolic reasoning.\n\nConnection to co-occurrence matrices\n\nSGNS also has a connection to matrix factorization. Under an idealized analysis in which each center–context score can be optimized independently, its optimum satisfies\n\n\\[u_o^\\top v_w = \\log \\frac{P_{\\mathrm{data}}(o\\mid w)} {kP_n(o)}.\\]\n\nIf the noise distribution equals the observed context-word marginal, this becomes\n\n\\[u_o^\\top v_w = \\operatorname{PMI}(w,o)-\\log k,\\]\n\nwhere pointwise mutual information measures how strongly two events co-occur relative to independence.\n\nThis explains why SGNS is often described as implicitly factorizing a shifted PMI matrix. The equivalence has qualifications: finite-dimensional vectors constrain the scores jointly, empirical pair counts depend on preprocessing, and the common three-quarter-power noise distribution introduces a context-dependent adjustment rather than only a constant shift."
    },
    {
      "heading": "Historical development and applications",
      "text": "Mikolov and colleagues introduced skip-gram and CBOW in work on efficient word-representation learning in 2013. Further work that year developed negative sampling and frequent-word subsampling, making training practical on large corpora.\n\nThe model became influential because useful word vectors could be learned from unannotated text with comparatively simple computation. These vectors were used as features or initial representations for part-of-speech tagging, document classification, information retrieval, and other language-processing tasks.\n\nRelated methods have extended the training principle beyond ordinary word windows:\n\n- Subword models incorporate character sequences, allowing words to share information through spelling components.\n- Dependency-based embeddings use grammatical relations rather than only linear proximity.\n- Graph embeddings treat nodes encountered on random walks as contexts.\n- Cross-lingual methods align or jointly learn representations across languages.\n\nThese methods adapt the center–context learning framework; their results depend on how the underlying entities and contexts are defined."
    },
    {
      "heading": "Limitations",
      "text": "Standard skip-gram assigns one vector to each vocabulary item. It therefore combines different senses of a word into a single representation. For a word such as bank, financial and geographical uses are not assigned separate token-specific embeddings. This limits its treatment of polysemy.\n\nRare words provide few training examples, making their representations less stable. A basic word-level model also cannot directly produce a trained vector for an out-of-vocabulary word. Subword extensions mitigate this limitation but change the representation scheme.\n\nBecause skip-gram learns corpus associations, its vectors can reproduce social stereotypes and other imbalances in the training data. Similarity scores describe learned distributional structure, not verified facts about the world.\n\nResults can vary with random initialization, corpus composition, preprocessing, and training settings. Apparent changes between embeddings trained on different corpora therefore require careful interpretation.\n\nFinally, standard skip-gram is neither a complete model of sentence meaning nor an autoregressive language model. Its usual symmetric context window includes both preceding and following words, and SGNS does not supply normalized sequence probabilities. Contextual representation models such as ELMo and large language models instead compute representations that vary with the surrounding text."
    }
  ],
  "links": [
    "machine-learning",
    "text-corpus",
    "word2vec",
    "word-embedding",
    "word-order",
    "syntax",
    "softmax-function",
    "artificial-neural-network",
    "semantics",
    "chinese-word-segmentation",
    "vector-space",
    "cosine-similarity",
    "matrix",
    "part-of-speech-tagging",
    "polysemy",
    "training-data",
    "n-gram-language-model",
    "large-language-model"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}