{
  "id": "n-gram",
  "title": "N-gram",
  "lang": "en",
  "source_lang": "en",
  "summary": "An n-gram is a sequence of consecutive units used to represent local patterns and estimate probabilities in language and other sequential data.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "n-gram",
    "ngram",
    "n-gram sequence"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/n-gram/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/n-gram/",
      "json": "https://aiwiki.page/zh/n-gram.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "An n-gram is a contiguous sequence of \\(n\\) units drawn from a longer sequence. The units may be words, characters, bytes, speech sounds, or other discrete symbols. A unigram contains one unit, a bigram two, and a trigram three. In natural language processing, n-grams provide a simple way to represent local patterns in text; in an n-gram language model, their observed frequencies are used to estimate the probability of a unit given a limited preceding context.\n\nN-grams connect the statistical description of sequences with practical tasks such as speech recognition, information retrieval, spelling correction, text classification, and machine translation. Their appeal lies in their explicit structure: occurrences can be counted directly, features can be inspected, and probabilities can be computed efficiently. Their central limitation is equally direct: longer patterns become increasingly sparse, while fixed-length contexts cannot capture every dependency in a sequence.\n\nThe term therefore denotes both a kind of sequence fragment and, by extension, several methods built around such fragments. An n-gram feature representation need not be a probabilistic model, and a probabilistic n-gram model need not operate on words. The choice of units, treatment of boundaries, statistical estimation method, and intended application determine what an n-gram system actually represents."
    },
    {
      "heading": "Units, boundaries, and basic definitions",
      "text": "For a sequence \\(x_1,x_2,\\ldots,x_T\\), an n-gram beginning at position \\(i\\) is the ordered tuple\n\n\\[(x_i,x_{i+1},\\ldots,x_{i+n-1}), \\qquad 1\\leq i\\leq T-n+1.\\]\n\nWithout padding, a sequence of length \\(T\\) contains \\(\\max(T-n+1,0)\\) n-gram occurrences. These occurrences normally overlap. The word sequence “the cat sat down,” for example, contains the bigrams “the cat,” “cat sat,” and “sat down,” and the trigrams “the cat sat” and “cat sat down.” The number of occurrences differs from the number of distinct types: “a a a” has two bigram occurrences but only one distinct bigram, “a a.”\n\nThe units are determined by tokenization, not by the n-gram definition itself. A word tokenizer might treat “don't” as one token or divide it into several components. It might retain punctuation, separate it from words, or discard it. Consequently, two systems described as word-trigram models can count different events in the same document. Lowercasing similarly merges distinctions between forms such as “Apple” and “apple,” while replacing numbers with a common symbol reduces vocabulary size but removes numerical detail.\n\nCharacter n-grams represent shorter orthographic patterns and can be extracted without a word dictionary. The character trigrams of “cat,” without padding, consist only of “cat”; with boundary markers, additional patterns can encode the beginning and end of the word. Character units themselves require definition. A Unicode code point is not necessarily a complete user-perceived character, and a visible accented letter may have more than one underlying representation. Byte n-grams operate instead on bytes, making them independent of linguistic segmentation but sensitive to character encoding. In UTF-8, a byte n-gram can cut through the multibyte encoding of a character.\n\nBoundary handling determines which transitions are considered meaningful. Sentence-based language models usually introduce start and end symbols and avoid counting ordinary word transitions across sentence boundaries. An order-\\(n\\) model may prepend \\(n-1\\) start symbols so that even the first word has a full-length history. End symbols allow the model to assign probability to stopping, rather than only to the next lexical token. Document and word boundaries can likewise be retained for particular applications.\n\nOrdinary n-grams are contiguous and ordered. A skip-gram permits intervening units, while an unordered collection of words discards their relative positions. These are related representations, but not identical ones. In particular, the skip-gram training objective used by word2vec should not be confused with a conventional n-gram count table: it learns predictive representations from nearby words rather than simply storing the frequencies of contiguous sequences."
    },
    {
      "heading": "Probabilistic language modeling",
      "text": "A probabilistic language model assigns probabilities to sequences. By the chain rule of probability, the probability of a token sequence can be written exactly as\n\n\\[P(w_1,\\ldots,w_T) = \\prod_{t=1}^{T}P(w_t\\mid w_1,\\ldots,w_{t-1}).\\]\n\nAn n-gram language model approximates each conditional distribution using only the previous \\(n-1\\) tokens:\n\n\\[P(w_t\\mid w_1,\\ldots,w_{t-1}) \\approx P(w_t\\mid w_{t-n+1},\\ldots,w_{t-1}).\\]\n\nFor positions near the beginning, the history is shortened or supplied with boundary symbols. A unigram model ignores the preceding tokens altogether; a bigram model conditions on one preceding token; a trigram model conditions on two. If complete sentence probabilities are required, the prediction of an end-of-sentence symbol is included as well.\n\nThis approximation is a finite-memory assumption closely related to a Markov chain. An order-\\(n\\) n-gram model uses a Markov history of length \\(n-1\\), which explains a potentially confusing difference in terminology: a bigram language model has a first-order Markov dependency. The state may be represented as the current suffix of up to \\(n-1\\) tokens. Generating a token updates that state by dropping the oldest token and appending the new one.\n\nThe simplest estimate comes from maximum likelihood estimation. If \\(h\\) denotes a history and \\(C(h,w)\\) counts how often token \\(w\\) follows it in the training data, then\n\n\\[\\widehat P_{\\mathrm{ML}}(w\\mid h) = \\frac{C(h,w)}{\\sum_{v\\in V}C(h,v)},\\]\n\nwhere \\(V\\) is the prediction vocabulary. The denominator counts eligible continuations of the history. For a history followed by “cat” 30 times, “dog” 10 times, and no other token, the estimated conditional probabilities are \\(0.75\\) and \\(0.25\\). With consistent boundary conventions, the denominator is often abbreviated as \\(C(h)\\).\n\nA model can score a sequence by multiplying its conditional probabilities, or generate a sequence by repeatedly sampling from them. In implementations, scoring normally uses sums of logarithms because products of many small probabilities can underflow:\n\n\\[\\log P(w_1,\\ldots,w_T) \\approx \\sum_{t=1}^{T}\\log P(w_t\\mid h_t).\\]\n\nThis local decomposition also makes n-gram models suitable for dynamic programming. Candidate sequences that share the same relevant suffix can reuse future calculations. Nevertheless, a conditional next-token model is not automatically a well-behaved distribution over all finite sentences: termination requires appropriate treatment of the end symbol and reachable states. These details matter when the model is used as a generator rather than merely as a scoring function."
    },
    {
      "heading": "Data sparsity and probability smoothing",
      "text": "The main statistical obstacle is the rapid growth of the space of possible patterns. A vocabulary of size \\(V\\) has \\(V^n\\) possible ordered n-grams before structural restrictions are imposed. With 50,000 word types, there are \\(1.25\\times10^{14}\\) possible trigrams. Actual language occupies only part of this space, but even a large corpus leaves many legitimate combinations unobserved. Increasing \\(n\\) gives a more specific context while reducing the evidence available for each context.\n\nMaximum likelihood assigns zero probability to every unseen continuation. A single such event makes the probability of an entire test sequence zero, regardless of how plausible its other parts are. Probability smoothing reallocates some probability mass from observed events to unseen or poorly supported ones. It is not simply a numerical workaround: it expresses the statistical expectation that a finite sample does not contain every event the underlying process can produce.\n\nAdditive smoothing replaces raw counts with counts plus a positive constant:\n\n\\[P_{\\alpha}(w\\mid h) = \\frac{C(h,w)+\\alpha} {\\sum_{v\\in V}C(h,v)+\\alpha |V|}.\\]\n\nSetting \\(\\alpha=1\\) gives add-one, or Laplace, smoothing. This guarantees positive probabilities for every vocabulary item, but can reserve too much mass for unseen events when the vocabulary is large. Good–Turing estimation instead uses the frequencies of frequencies. If \\(N_r\\) is the number of event types observed exactly \\(r\\) times, its classical adjusted count is\n\n\\[r^*=(r+1)\\frac{N_{r+1}}{N_r}.\\]\n\nPractical implementations smooth or otherwise stabilize these frequency-of-frequency estimates, particularly where counts are sparse.\n\nBackoff and interpolation combine evidence from different history lengths. Backoff generally uses a shorter history when a longer-history event is absent, with weights that preserve normalization. Interpolation combines several orders even when the longest n-gram has been observed. A general interpolated form is\n\n\\[P(w\\mid h) = \\lambda(h)P_{\\text{high}}(w\\mid h) + \\bigl(1-\\lambda(h)\\bigr)P_{\\text{low}}(w\\mid h'),\\]\n\nwhere \\(h'\\) is a shorter suffix and \\(0\\leq\\lambda(h)\\leq1\\). These weights can reflect the amount and diversity of evidence supporting the longer history.\n\nKneser–Ney smoothing makes a particularly important distinction between frequency and contextual diversity. Its lower-order distribution favors words that appear after many different histories, rather than merely words with high total counts. A word occurring frequently inside one fixed expression should not necessarily receive a high probability in unrelated contexts. In a common interpolated bigram formulation,\n\n\\[P_{\\mathrm{KN}}(w\\mid h) = \\frac{\\max(C(h,w)-D,0)}{C(h)} + \\lambda(h)P_{\\mathrm{cont}}(w),\\]\n\nwhere \\(P_{\\mathrm{cont}}(w)\\) depends on the number of distinct preceding words. Modified Kneser–Ney uses different discounts for different count ranges. Smoothing does not remove the finite-context assumption, but it substantially improves estimates within that assumption."
    },
    {
      "heading": "Counting, storage, and computational organization",
      "text": "Extracting n-grams begins with a sliding window over a sequence. For a single fixed order \\(n\\), a stream of \\(T\\) tokens yields a number of windows proportional to \\(T\\). Actual cost depends on representation: copying an entire length-\\(n\\) tuple for every occurrence takes more work than updating a rolling representation or storing references. Counting all orders from one through \\(N\\) produces approximately \\(NT\\) occurrences when \\(T\\) is large relative to \\(N\\), before repeated types are aggregated.\n\nA hash table can map each distinct n-gram to a count. This is straightforward, but storing token strings repeatedly is expensive. Implementations commonly map tokens to integer identifiers and represent n-grams as integer sequences. Alternatively, a prefix-oriented tree structure stores shared beginnings once. Reversing histories or organizing entries by their suffixes can make shorter-context lookups efficient. The best layout depends on whether the dominant operation is counting, conditional-probability lookup, sequential scanning, or model construction.\n\nLarge corpora often require external-memory or distributed processing. One approach emits n-gram records, sorts them, and aggregates adjacent identical records. A MapReduce implementation can distribute extraction and combine partial counts before final aggregation. Partitioning by history can place all continuations needed for a conditional distribution together. Such organization becomes important during smoothing, because constructing a model requires more than independent frequency totals: continuation counts, count-of-count statistics, and normalization information must also be computed consistently.\n\nThe distinction between possible and observed n-grams is crucial for space complexity. An implementation need not allocate the full \\(V^n\\) space; it usually stores only observed entries and enough information to assign probabilities to other events. Even then, rare high-order patterns may dominate storage. Pruning removes selected entries, sometimes by a frequency threshold and sometimes by an estimate of their effect on the model's distribution. Frequency pruning is simple, but a rare pattern can still be important for a specialized name or a highly discriminative expression.\n\nPrediction speed depends on memory organization as well as arithmetic. Compact arrays, quantized probability values, shared prefixes, and CPU cache locality can matter more than the nominal number of probability operations. Backoff models may require several lookups when a long context is absent. A commonly used interchange format, ARPA, organizes probability entries by order and can include backoff weights; optimized runtime formats often transform those entries into more compact structures.\n\nApproximate counting and hashed feature representations introduce additional trade-offs. Hash collisions merge distinct n-grams unless keys are checked, while retaining full keys preserves exact identity at a storage cost. Model construction must also guard against counter overflow, inconsistent token identifiers, and accidental cross-document windows. These engineering choices can alter observed counts or predictions even when the abstract n-gram definition remains unchanged."
    },
    {
      "heading": "Evaluation, model selection, and corpus effects",
      "text": "A language model is commonly evaluated by its predictive performance on a held-out test set. For a test sequence containing \\(T\\) scored tokens, the average negative log probability in base two is\n\n\\[H = -\\frac{1}{T} \\sum_{t=1}^{T}\\log_2P(w_t\\mid h_t).\\]\n\nThe corresponding perplexity is\n\n\\[\\operatorname{PP}=2^H.\\]\n\nA lower value means that the model assigns higher geometric-mean probability to the observed tokens. For example, \\(H=5\\) bits per token corresponds to perplexity \\(32\\). Perplexity can be interpreted as an effective average branching factor, but it is not a direct count of vocabulary items or a measure of how many distinct sentences the model can produce.\n\nComparisons require the same prediction units and compatible evaluation conventions. Word-level perplexity is not directly comparable with character-level perplexity, because one word typically contains several characters. Even two word models may differ in vocabulary, sentence segmentation, or the handling of unknown words. Replacing many rare words with one unknown-token symbol simplifies the prediction task and can lower reported perplexity without improving the ability to distinguish those words. Whether boundary symbols contribute to \\(T\\) also needs to be consistent.\n\nModel order, discount values, interpolation weights, vocabulary thresholds, and pruning settings are hyperparameters. They are normally selected using a validation set rather than the final test set. Increasing order can improve prediction when enough repeated contexts exist, but it also increases sparsity and storage. Strong training performance paired with weaker held-out performance indicates overfitting. Smoothing and order selection address related aspects of this problem, but neither guarantees that a model will generalize to a substantially different domain.\n\nCorpus composition determines which regularities are learnable. A model trained on newspaper articles may assign poor probabilities to conversational transcripts, technical documentation, or text containing recognition errors. This is a form of distribution shift. Adaptation can combine general and domain-specific models, adjust interpolation weights, or update counts using appropriate new data. The effects depend on whether the new material changes vocabulary, topic frequency, grammatical style, or all three.\n\nEvaluation also depends on how data are separated. Randomly dividing overlapping windows from the same documents can create substantial data leakage: near-identical contexts may appear in training and testing. Document-level or time-based separation can provide a more demanding assessment when it matches the intended deployment setting. For downstream tasks, predictive quality must also be measured at the task level. A reduction in perplexity does not guarantee a proportional reduction in speech-recognition errors, and useful classification features need not define a good language model at all."
    },
    {
      "heading": "N-grams as features and retrieval units",
      "text": "An n-gram representation can describe a document without assigning probabilities to complete sequences. Each distinct n-gram becomes a feature, and a document is represented by feature presence, raw counts, normalized frequencies, or weighted values. A bag-of-words model uses unigram word features; adding word bigrams or trigrams preserves limited local order. Thus “not useful” can be represented as a distinct feature rather than being reduced entirely to separate occurrences of “not” and “useful.”\n\nThis representation is sparse: a document normally contains only a small fraction of the n-grams observed across a collection. Linear classifiers, including a support vector machine, can operate effectively on such sparse vectors. A naive Bayes classifier can also use n-gram-derived features, although adjacent overlapping n-grams are plainly not statistically independent. Their usefulness as predictive features does not depend on that independence assumption being a literal description of language.\n\nTerm frequency–inverse document frequency weighting can reduce the influence of n-grams that appear in many documents while emphasizing patterns concentrated in a smaller subset. This is often useful for topic classification and retrieval. However, frequent expressions are not always uninformative: function-word sequences may help identify writing style, and common syntactic patterns may distinguish genres. Feature weighting and feature selection therefore depend on the task rather than on a universal rule that rare patterns are better.\n\nCharacter n-grams are particularly useful when exact word matching is brittle. Related inflected forms can share many character fragments even when their complete word forms differ. Misspellings, transliteration variants, and previously unseen names may retain partial overlap with known forms. This supports language identification, approximate string matching, and classification in settings where word segmentation is difficult. The same robustness can introduce noise, since unrelated strings may share short fragments, and very short n-grams often provide little discrimination.\n\nIn retrieval systems, a database index can associate each n-gram with the documents or strings containing it. Character trigrams, for example, can generate candidate matches for a query before a more expensive edit-distance calculation verifies them. For document comparison, sets of word shingles—often another name for word n-grams—can be compared by overlap. Their Jaccard similarity is\n\n\\[J(A,B)=\\frac{|A\\cap B|}{|A\\cup B|}.\\]\n\nThis set-based measure ignores multiplicity, whereas count-vector comparisons retain it.\n\nFeature extraction must be fitted and evaluated carefully. Selecting discriminative n-grams using labels from the full dataset leaks information into evaluation, even if the final classifier is trained only on a designated training portion. Large n-gram vocabularies can also encode dataset-specific artifacts, such as headers or source identifiers, rather than the intended concept. Inspectability makes these artifacts easier to detect, but does not prevent their inclusion."
    },
    {
      "heading": "Sequence decoding, compression, and nonlinguistic applications",
      "text": "In speech recognition, an n-gram language model supplies a prior preference among candidate word sequences. An acoustic model evaluates how well a candidate explains the sound, while the language model evaluates its linguistic plausibility. In a simplified probabilistic formulation,\n\n\\[\\widehat W = \\arg\\max_W P(X\\mid W)P(W),\\]\n\nwhere \\(X\\) is the acoustic observation and \\(W\\) a word sequence. Practical decoders often use a weighted combination of acoustic and language-model scores, along with insertion penalties. The finite n-gram history allows hypotheses with compatible states to be managed efficiently during search.\n\nTraditional statistical machine translation also used n-gram language models to favor fluent target-language sequences. A translation model might propose several word choices and reorderings; the target-language model helped distinguish locally natural combinations from awkward ones. Its contribution was fluency rather than a direct guarantee of semantic fidelity to the source. A high-probability target phrase could still express the wrong meaning, and a long-distance agreement relation could remain outside the model's context.\n\nSpelling correction similarly combines evidence about likely errors with evidence about intended text. A candidate that is close to a misspelled string may become more or less plausible in its surrounding word context. Word n-grams can distinguish alternatives such as homophones, while character n-grams can score orthographic patterns within candidate words. Contextual correction requires more than choosing the most frequent candidate: frequency must be balanced against the probability of the observed error and the relevant neighboring units.\n\nN-gram probabilities also connect prediction to lossless data compression. An idealized code for an event of probability \\(p\\) requires approximately \\(-\\log_2p\\) bits. A context model that assigns high probability to the actual next symbol therefore supports shorter encodings through methods such as arithmetic coding. Practical compressors may use variable-length contexts, adaptive updates, or mixtures of several predictors rather than a single fixed-order table. The conceptual relationship remains: better calibrated sequential predictions can reduce coding length, subject to model overhead and coding details.\n\nBeyond natural language, n-grams apply whenever observations have a meaningful discrete order. In DNA analysis, contiguous sequences of \\(k\\) bases are usually called k-mers. Counts of these patterns support sequence comparison and other computational analyses, although biological interpretation depends on strand orientation, sequencing errors, and the analysis being performed. In cybersecurity, byte or instruction n-grams can serve as features for identifying file types or classifying malware, while sequences of system events can support anomaly detection.\n\nThese applications share a representational principle, not a guarantee that the underlying process has short memory. A byte pattern may reflect a file format rather than program behavior; a biological k-mer may occur in unrelated genomic contexts. Choosing \\(n\\) balances specificity against coverage, and useful local correlations must be distinguished from a complete explanation of the sequence-generating mechanism."
    },
    {
      "heading": "Historical development and relation to neural models",
      "text": "The statistical study of symbol sequences predates modern digital language processing. Andrey Markov's early twentieth-century work on dependent sequences included an analysis of vowel and consonant patterns in Russian literary text. These studies did not constitute contemporary word n-gram systems, but helped establish the mathematical framework for finite-memory dependencies. Claude Shannon's 1948 work on communication theory used successive approximations to English based on statistical dependencies, and his 1951 study examined the prediction and redundancy of printed English.\n\nWith the development of computational speech recognition, n-gram language models became a central engineering tool. During the 1970s and 1980s, research systems combined acoustic evidence with statistical word-sequence models, and larger machine-readable corpora made increasingly detailed counts possible. Frederick Jelinek and colleagues at IBM were influential in this statistical approach. The practical strength of the method came from decomposing a difficult sequence problem into locally estimated probabilities that could be integrated into efficient decoding algorithms.\n\nSeveral developments improved the treatment of sparse evidence. I. J. Good's 1953 publication presented the frequency-estimation ideas associated with Alan Turing. Slava Katz introduced an influential backoff language-modeling method in 1987. Reinhard Kneser and Hermann Ney introduced Kneser–Ney smoothing in 1995, and Stanley Chen and Joshua Goodman examined smoothing methods extensively in work published during the late 1990s. These contributions addressed a central empirical fact: increasing corpus size alone does not eliminate the need to estimate probabilities for unseen combinations.\n\nNeural probabilistic language models offered a different way to share statistical strength. Yoshua Bengio and colleagues' 2003 model learned distributed word representations so that evidence about one word sequence could inform predictions for related sequences. Rather than treating every history as an unrelated table key, such models learned continuous parameters shared across contexts. Recurrent neural networks subsequently provided mechanisms for carrying information across variable-length histories, while the Transformer architecture, introduced in 2017, used attention to make predictions from representations of a bounded input context.\n\nA large language model is therefore not simply a high-order n-gram table. Both may estimate next-token probabilities, but a neural model composes learned representations and shares parameters across many different contexts. This permits forms of generalization unavailable to a literal lookup table. Conversely, having a long context window does not guarantee correct use of every dependency within it, and neural models can still learn strong local statistical regularities.\n\nN-gram methods remain useful where compactness, transparent counts, predictable lookup behavior, or small-data baselines matter. They also expose a clear boundary between local sequence statistics and broader linguistic structure. A fixed-order model assigns the same next-token distribution to any two histories with the same relevant suffix, regardless of earlier discourse, factual context, or syntactic commitments. Adding more orders postpones this boundary but does not remove it; character and subword units reduce some vocabulary problems while leaving the finite-memory restriction intact."
    }
  ],
  "links": [
    "natural-language-processing",
    "language-model",
    "speech-recognition",
    "information-retrieval",
    "machine-translation",
    "tokenization",
    "unicode",
    "byte",
    "character-encoding",
    "utf-eight",
    "word2vec",
    "probability",
    "markov-chain",
    "maximum-likelihood-estimation",
    "training-data",
    "dynamic-programming",
    "hash-table",
    "tree-data-structure",
    "mapreduce",
    "space-complexity",
    "cpu-cache",
    "test-set",
    "perplexity",
    "hyperparameter",
    "validation-set",
    "overfitting",
    "distribution-shift",
    "data-leakage",
    "bag-of-words-model",
    "support-vector-machine",
    "naive-bayes-classifier",
    "tf-idf",
    "feature-selection",
    "database-index",
    "edit-distance",
    "lossless-data-compression",
    "arithmetic-coding",
    "dna",
    "cybersecurity",
    "malware",
    "anomaly-detection",
    "ibm",
    "recurrent-neural-network",
    "transformer-architecture",
    "large-language-model",
    "generalization",
    "context-window"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}