{
  "id": "glove",
  "title": "GloVe",
  "lang": "en",
  "source_lang": "en",
  "summary": "GloVe is a word-embedding model that learns vector representations of words by fitting weighted relationships in global word–word co-occurrence statistics.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Global Vectors for Word Representation",
    "Global Vectors"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/glove/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/glove/",
      "json": "https://aiwiki.page/zh/glove.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "GloVe, short for Global Vectors for Word Representation, is a word-embedding model that represents words as dense numerical vectors learned from their co-occurrence patterns in a text corpus. Introduced in 2014 by Jeffrey Pennington, Richard Socher, and Christopher D. Manning, it combines corpus-wide statistical information with a training objective defined over individual word–context pairs. Words that occur in similar linguistic environments tend to receive related representations, making the resulting vectors useful for natural language processing tasks."
    },
    {
      "heading": "Background and motivation",
      "text": "GloVe belongs to distributional semantics, which studies meaning through patterns of linguistic use. Its underlying premise is that the contexts in which a word appears provide information about its meaning. For example, words associated with refrigeration are likely to occur near terms concerning cold temperatures, whereas words associated with combustion are more likely to occur near terms concerning heat.\n\nEarlier approaches represented words using large count tables and compressed those tables through matrix factorization. Predictive approaches, notably word2vec, instead learned representations through objectives involving words and their local contexts. GloVe was developed to combine useful aspects of both traditions: explicit global co-occurrence statistics and an objective that learns compact vectors from word–context relationships.\n\nHere, global refers to the aggregation of co-occurrence information across the training corpus. It does not mean that every word in a document is treated as the context of every other word, nor that the model directly represents complete documents.\n\nGloVe is generally classified as unsupervised learning or, more broadly, self-supervised learning, because its training signal is constructed from text rather than supplied as manually annotated semantic labels."
    },
    {
      "heading": "Co-occurrence statistics",
      "text": "Training begins by constructing a word–word co-occurrence matrix. Let \\(X_{ij}\\) denote how often context word \\(j\\) occurs near target word \\(i\\). A context is commonly defined by a fixed-size window around each word occurrence.\n\nCounts need not be integers. A preprocessing scheme may assign nearby words more weight than distant words, for example by weighting a context occurrence inversely by its distance from the target. Symmetric windows collect context on both sides; other choices can preserve directional information.\n\nDefine\n\n\\[X_i=\\sum_k X_{ik}, \\qquad P_{ij}=\\frac{X_{ij}}{X_i}.\\]\n\nThen \\(P_{ij}\\) is the empirical probability of seeing context word \\(j\\) around target word \\(i\\), under the chosen counting procedure.\n\nA central motivation for GloVe is that ratios of co-occurrence probabilities can reveal distinctions between words. Consider two target words, such as “ice” and “steam,” and a context word \\(k\\):\n\n\\[\\frac{P_{ik}}{P_{jk}}.\\]\n\nA context such as “solid” should favor one target, while “gas” should favor the other. A context associated with both targets, or with neither, may be less informative for distinguishing them. This motivates representing relationships through vector differences and dot products that approximate logarithms of co-occurrence probabilities.\n\nThe probability-ratio argument motivates the model; the actual training objective operates on weighted logarithms of co-occurrence counts."
    },
    {
      "heading": "Model and objective function",
      "text": "For each vocabulary item, GloVe learns a target vector \\(w_i\\), a context vector \\(\\tilde w_i\\), and corresponding scalar biases \\(b_i\\) and \\(\\tilde b_i\\). The vectors usually have far fewer dimensions than the vocabulary contains words.\n\nThe loss function is\n\n\\[J= \\sum_{i,j:\\,X_{ij}>0} f(X_{ij}) \\left( w_i^\\top \\tilde w_j+b_i+\\tilde b_j-\\log X_{ij} \\right)^2.\\]\n\nThe dot product and bias terms approximate the logarithm of an observed co-occurrence count:\n\n\\[w_i^\\top \\tilde w_j+b_i+\\tilde b_j \\approx \\log X_{ij}.\\]\n\nThe biases absorb effects associated with overall word frequencies, so the vectors do not have to encode those effects entirely through their dot products.\n\nThe standard weighting function is\n\n\\[f(x)= \\begin{cases} (x/x_{\\max})^\\alpha, & 0<x<x_{\\max},\\\\ 1, & x\\geq x_{\\max}. \\end{cases}\\]\n\nThe original paper used \\(\\alpha=0.75\\) and \\(x_{\\max}=100\\) in its reported experiments. These are hyperparameters, not universal requirements.\n\nThe weighting function reduces the influence of rare co-occurrences, whose counts may be unreliable, while capping the weight assigned to very frequent pairs. The logarithm also compresses the large differences between raw counts.\n\nUnobserved pairs\n\nPairs with \\(X_{ij}=0\\) are omitted from the objective. This avoids taking the logarithm of zero and makes sparse computation possible. It also means that the standard objective does not explicitly train every unobserved word pair to have a low score.\n\nConsequently, GloVe is not simply a squared-error fit to a dense matrix in which missing co-occurrences are treated as numerical zeros. The treatment of unobserved pairs is an important difference from some other embedding objectives.\n\nFinal word vectors\n\nTraining produces two vector tables: one for words as targets and one for words as contexts. A common final representation is their sum,\n\n\\[v_i=w_i+\\tilde w_i.\\]\n\nUsing one table alone is also possible. The appropriate choice depends on the implementation and evaluation setting."
    },
    {
      "heading": "Training and computational properties",
      "text": "A typical training pipeline consists of:\n\n- Applying tokenization and text normalization.\n- Building a vocabulary, often with a minimum frequency threshold.\n- Accumulating weighted co-occurrence counts.\n- Optimizing the vector and bias parameters over observed pairs.\n- Exporting the word vectors for later use.\n\nThe original implementation uses AdaGrad, an adaptive gradient-based optimization method. Training does not require a deep neural network: the core parameters are vector tables and biases fitted through a weighted least-squares objective.\n\nIf the vocabulary size is \\(V\\), the embedding dimension is \\(d\\), and the number of observed word–context pairs is \\(N_{\\mathrm{nz}}\\), the vector and bias parameters require \\(O(Vd)\\) storage. Storing the sparse co-occurrence data requires space proportional to \\(N_{\\mathrm{nz}}\\). A training pass over these pairs takes approximately \\(O(N_{\\mathrm{nz}}d)\\) arithmetic work.\n\nThese estimates exclude preprocessing, sorting, and implementation overhead. Building the co-occurrence statistics can be a substantial part of the total cost, particularly for large corpora.\n\nResults depend on corpus composition, context-window size, vocabulary filtering, vector dimension, optimization settings, and training duration. Capitalization and punctuation handling can also change which words share a representation and which co-occurrences are counted."
    },
    {
      "heading": "Properties and uses",
      "text": "GloVe places words in a learned vector space. Similarity is commonly measured using cosine similarity:\n\n\\[\\operatorname{sim}(u,v)= \\frac{u^\\top v}{\\lVert u\\rVert\\lVert v\\rVert}.\\]\n\nNearby vectors often correspond to words with related meanings or usage patterns. However, distributional similarity is not identical to synonymy: antonyms can occur in similar contexts and therefore receive similar vectors.\n\nSome relations can be approximated by vector offsets. Word-analogy evaluations, for example, test whether a relation expressed by one pair of words can be transferred to another through vector arithmetic. Such patterns are empirical properties of a representation, not guaranteed rules of language or evidence of general reasoning ability.\n\nCommon applications include:\n\n- Input representations for neural models. Pretrained vectors can initialize a word-embedding layer, which may remain fixed or undergo fine-tuning during task-specific training.\n- Text classification. Word vectors can be averaged or processed by models such as a convolutional neural network or recurrent neural network.\n- Similarity and retrieval. Vector comparisons can support lexical matching, clustering, and components of information retrieval systems.\n- Linguistic analysis. Researchers can examine associations encoded by a corpus, subject to the limitations of the data and model.\n\nPretrained GloVe releases have included vectors learned from Wikipedia and Gigaword, Common Crawl, and Twitter text. These are distinct resources with different vocabularies, preprocessing choices, dimensions, and corpus characteristics, rather than a single interchangeable model."
    },
    {
      "heading": "Comparison with other representations",
      "text": "Word2vec\n\nGloVe and word2vec both commonly produce static word vectors, but their training procedures differ. GloVe explicitly constructs co-occurrence statistics and fits weighted log-count relationships. Word2vec’s continuous bag-of-words and skip-gram architectures learn through predictive objectives involving local contexts.\n\nThe distinction is not simply “global information versus local information.” Aggregated local examples also allow word2vec to capture corpus-wide regularities, and some predictive embedding objectives have mathematical connections to matrix factorization.\n\nCount-based representations\n\nCompared with raw count vectors or TF–IDF representations, GloVe produces compact dense vectors that can share statistical information across related words. The cost is reduced direct interpretability: individual dimensions generally do not correspond to named context words or clearly defined semantic attributes.\n\nContextual representations\n\nGloVe normally assigns one vector to each vocabulary entry, regardless of the sentence in which it occurs. The same representation of “bank” is therefore used for both a financial institution and a riverbank.\n\nContextual models such as BERT generate representations that depend on surrounding text. These models can distinguish uses of the same token more directly, though doing so generally requires more computation than looking up a fixed GloVe vector. GloVe is a lexical representation model, not a text-generating language model."
    },
    {
      "heading": "Limitations and evaluation",
      "text": "Polysemy and context. A single vector blends different senses and uses of a word. GloVe does not directly model sentence-level composition, grammatical structure, negation, or discourse.\n\nVocabulary coverage. Standard GloVe has no built-in mechanism for constructing vectors for unseen words from their spelling or subword components. Rare words may have poorly estimated vectors.\n\nCorpus dependence. Representations reflect their training data. Performance can change under distribution shift, such as moving from general web text to specialist terminology or informal conversation.\n\nSocial associations. Co-occurrence statistics can encode stereotypes and other systematic associations present in the corpus. Vector proximity or analogy results should not be interpreted as factual claims about people or social groups.\n\nEvaluation limits. Similarity and analogy benchmarks measure selected properties of embeddings, not their usefulness for every downstream task. Strong performance on one benchmark does not establish superior performance in classification, retrieval, or sequence labeling.\n\nGeometric interpretation. The learned coordinates are not unique. Transformations of the target and context spaces can preserve the fitted dot products, so individual dimensions should not be treated as intrinsic semantic quantities. Comparisons between separately trained spaces require care, even when the corpora and hyperparameters are similar."
    }
  ],
  "links": [
    "word-embedding",
    "natural-language-processing",
    "matrix-factorization",
    "word2vec",
    "unsupervised-learning",
    "self-supervised-learning",
    "matrix",
    "loss-function",
    "hyperparameter",
    "tokenization",
    "mathematical-optimization",
    "artificial-neural-network",
    "vector-space",
    "cosine-similarity",
    "fine-tuning",
    "convolutional-neural-network",
    "recurrent-neural-network",
    "information-retrieval",
    "tf-idf",
    "bert",
    "language-model",
    "training-data",
    "distribution-shift"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}