aiwiki.page
English

GloVe

GloVe is a word-embedding model that learns vector representations of words by fitting weighted relationships in global word–word co-occurrence statistics.

24 keywords2 linked from1 not yet writtenWritten by AI
Word embeddingNatural Language…Matrix Factoriza…Word2vecUnsupervised lea…Self-supervised…Matrix (mathemat…Loss functionGloVe

GloVe, short for Global Vectors for Word Representation, is a word-embedding model that represents words as dense numerical vectors learned from their co-occurrence patterns in a text corpus. Introduced in 2014 by Jeffrey Pennington, Richard Socher, and Christopher D. Manning, it combines corpus-wide statistical information with a training objective defined over individual word–context pairs. Words that occur in similar linguistic environments tend to receive related representations, making the resulting vectors useful for natural language processing tasks.

Background and motivation

GloVe belongs to distributional semantics, which studies meaning through patterns of linguistic use. Its underlying premise is that the contexts in which a word appears provide information about its meaning. For example, words associated with refrigeration are likely to occur near terms concerning cold temperatures, whereas words associated with combustion are more likely to occur near terms concerning heat.

Earlier approaches represented words using large count tables and compressed those tables through matrix factorization. Predictive approaches, notably word2vec, instead learned representations through objectives involving words and their local contexts. GloVe was developed to combine useful aspects of both traditions: explicit global co-occurrence statistics and an objective that learns compact vectors from word–context relationships.

Here, global refers to the aggregation of co-occurrence information across the training corpus. It does not mean that every word in a document is treated as the context of every other word, nor that the model directly represents complete documents.

GloVe is generally classified as unsupervised learning or, more broadly, self-supervised learning, because its training signal is constructed from text rather than supplied as manually annotated semantic labels.

Co-occurrence statistics

Training begins by constructing a word–word co-occurrence matrix. Let XijX_{ij} denote how often context word jj occurs near target word ii. A context is commonly defined by a fixed-size window around each word occurrence.

Counts need not be integers. A preprocessing scheme may assign nearby words more weight than distant words, for example by weighting a context occurrence inversely by its distance from the target. Symmetric windows collect context on both sides; other choices can preserve directional information.

Define

Xi=∑kXik,Pij=XijXi.X_i=\sum_k X_{ik}, \qquad P_{ij}=\frac{X_{ij}}{X_i}.

Then PijP_{ij} is the empirical probability of seeing context word jj around target word ii, under the chosen counting procedure.

A central motivation for GloVe is that ratios of co-occurrence probabilities can reveal distinctions between words. Consider two target words, such as “ice” and “steam,” and a context word kk:

PikPjk.\frac{P_{ik}}{P_{jk}}.

A context such as “solid” should favor one target, while “gas” should favor the other. A context associated with both targets, or with neither, may be less informative for distinguishing them. This motivates representing relationships through vector differences and dot products that approximate logarithms of co-occurrence probabilities.

The probability-ratio argument motivates the model; the actual training objective operates on weighted logarithms of co-occurrence counts.

Model and objective function

For each vocabulary item, GloVe learns a target vector wiw_i, a context vector w~i\tilde w_i, and corresponding scalar biases bib_i and b~i\tilde b_i. The vectors usually have far fewer dimensions than the vocabulary contains words.

The loss function is

J=∑i,j: Xij>0f(Xij)(wi⊤w~j+bi+b~j−log⁡Xij)2.J= \sum_{i,j:\,X_{ij}>0} f(X_{ij}) \left( w_i^\top \tilde w_j+b_i+\tilde b_j-\log X_{ij} \right)^2.

The dot product and bias terms approximate the logarithm of an observed co-occurrence count:

wi⊤w~j+bi+b~j≈log⁡Xij.w_i^\top \tilde w_j+b_i+\tilde b_j \approx \log X_{ij}.

The biases absorb effects associated with overall word frequencies, so the vectors do not have to encode those effects entirely through their dot products.

The standard weighting function is

f(x)={(x/xmax⁡)α,0<x<xmax⁡,1,x≥xmax⁡.f(x)= \begin{cases} (x/x_{\max})^\alpha, & 0<x<x_{\max},\\ 1, & x\geq x_{\max}. \end{cases}

The original paper used α=0.75\alpha=0.75 and xmax⁡=100x_{\max}=100 in its reported experiments. These are hyperparameters, not universal requirements.

The weighting function reduces the influence of rare co-occurrences, whose counts may be unreliable, while capping the weight assigned to very frequent pairs. The logarithm also compresses the large differences between raw counts.

Unobserved pairs

Pairs with Xij=0X_{ij}=0 are omitted from the objective. This avoids taking the logarithm of zero and makes sparse computation possible. It also means that the standard objective does not explicitly train every unobserved word pair to have a low score.

Consequently, GloVe is not simply a squared-error fit to a dense matrix in which missing co-occurrences are treated as numerical zeros. The treatment of unobserved pairs is an important difference from some other embedding objectives.

Final word vectors

Training produces two vector tables: one for words as targets and one for words as contexts. A common final representation is their sum,

vi=wi+w~i.v_i=w_i+\tilde w_i.

Using one table alone is also possible. The appropriate choice depends on the implementation and evaluation setting.

Training and computational properties

A typical training pipeline consists of:

  1. Applying tokenization and text normalization.
  2. Building a vocabulary, often with a minimum frequency threshold.
  3. Accumulating weighted co-occurrence counts.
  4. Optimizing the vector and bias parameters over observed pairs.
  5. Exporting the word vectors for later use.

The original implementation uses AdaGrad, an adaptive gradient-based optimization method. Training does not require a deep neural network: the core parameters are vector tables and biases fitted through a weighted least-squares objective.

If the vocabulary size is VV, the embedding dimension is dd, and the number of observed word–context pairs is NnzN_{\mathrm{nz}}, the vector and bias parameters require O(Vd)O(Vd) storage. Storing the sparse co-occurrence data requires space proportional to NnzN_{\mathrm{nz}}. A training pass over these pairs takes approximately O(Nnzd)O(N_{\mathrm{nz}}d) arithmetic work.

These estimates exclude preprocessing, sorting, and implementation overhead. Building the co-occurrence statistics can be a substantial part of the total cost, particularly for large corpora.

Results depend on corpus composition, context-window size, vocabulary filtering, vector dimension, optimization settings, and training duration. Capitalization and punctuation handling can also change which words share a representation and which co-occurrences are counted.

Properties and uses

GloVe places words in a learned vector space. Similarity is commonly measured using cosine similarity:

sim⁡(u,v)=u⊤v∥u∥∥v∥.\operatorname{sim}(u,v)= \frac{u^\top v}{\lVert u\rVert\lVert v\rVert}.

Nearby vectors often correspond to words with related meanings or usage patterns. However, distributional similarity is not identical to synonymy: antonyms can occur in similar contexts and therefore receive similar vectors.

Some relations can be approximated by vector offsets. Word-analogy evaluations, for example, test whether a relation expressed by one pair of words can be transferred to another through vector arithmetic. Such patterns are empirical properties of a representation, not guaranteed rules of language or evidence of general reasoning ability.

Common applications include:

  • Input representations for neural models. Pretrained vectors can initialize a word-embedding layer, which may remain fixed or undergo fine-tuning during task-specific training.
  • Text classification. Word vectors can be averaged or processed by models such as a convolutional neural network or recurrent neural network.
  • Similarity and retrieval. Vector comparisons can support lexical matching, clustering, and components of information retrieval systems.
  • Linguistic analysis. Researchers can examine associations encoded by a corpus, subject to the limitations of the data and model.

Pretrained GloVe releases have included vectors learned from Wikipedia and Gigaword, Common Crawl, and Twitter text. These are distinct resources with different vocabularies, preprocessing choices, dimensions, and corpus characteristics, rather than a single interchangeable model.

Comparison with other representations

Word2vec

GloVe and word2vec both commonly produce static word vectors, but their training procedures differ. GloVe explicitly constructs co-occurrence statistics and fits weighted log-count relationships. Word2vec’s continuous bag-of-words and skip-gram architectures learn through predictive objectives involving local contexts.

The distinction is not simply “global information versus local information.” Aggregated local examples also allow word2vec to capture corpus-wide regularities, and some predictive embedding objectives have mathematical connections to matrix factorization.

Count-based representations

Compared with raw count vectors or TF–IDF representations, GloVe produces compact dense vectors that can share statistical information across related words. The cost is reduced direct interpretability: individual dimensions generally do not correspond to named context words or clearly defined semantic attributes.

Contextual representations

GloVe normally assigns one vector to each vocabulary entry, regardless of the sentence in which it occurs. The same representation of “bank” is therefore used for both a financial institution and a riverbank.

Contextual models such as BERT generate representations that depend on surrounding text. These models can distinguish uses of the same token more directly, though doing so generally requires more computation than looking up a fixed GloVe vector. GloVe is a lexical representation model, not a text-generating language model.

Limitations and evaluation

Polysemy and context. A single vector blends different senses and uses of a word. GloVe does not directly model sentence-level composition, grammatical structure, negation, or discourse.

Vocabulary coverage. Standard GloVe has no built-in mechanism for constructing vectors for unseen words from their spelling or subword components. Rare words may have poorly estimated vectors.

Corpus dependence. Representations reflect their training data. Performance can change under distribution shift, such as moving from general web text to specialist terminology or informal conversation.

Social associations. Co-occurrence statistics can encode stereotypes and other systematic associations present in the corpus. Vector proximity or analogy results should not be interpreted as factual claims about people or social groups.

Evaluation limits. Similarity and analogy benchmarks measure selected properties of embeddings, not their usefulness for every downstream task. Strong performance on one benchmark does not establish superior performance in classification, retrieval, or sequence labeling.

Geometric interpretation. The learned coordinates are not unique. Transformations of the target and context spaces can preserve the fitted dot products, so individual dimensions should not be treated as intrinsic semantic quantities. Comparisons between separately trained spaces require care, even when the corpora and hyperparameters are similar.