{
  "id": "bleu",
  "title": "BLEU",
  "lang": "en",
  "source_lang": "en",
  "summary": "BLEU is an automatic translation evaluation metric that combines clipped n-gram precision with a penalty for translations shorter than their references.",
  "category": "language",
  "subcategory": "",
  "type": "concept",
  "aliases": [
    "Bilingual Evaluation Understudy",
    "BLEU score"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/bleu/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/bleu/",
      "json": "https://aiwiki.page/zh/bleu.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "BLEU (Bilingual Evaluation Understudy) is an automatic metric for evaluating machine translation by comparing system-generated text with one or more human-written reference translations. It measures overlap between sequences of tokens, limits credit for repeated matches, and penalizes outputs that are too short. Introduced in 2002, BLEU became a standard tool for comparing translation systems because it is inexpensive to compute, requires no trained evaluation model, and can summarize performance over a large test set. It measures resemblance to references rather than directly determining whether a translation preserves the source meaning."
    },
    {
      "heading": "Origin and purpose",
      "text": "BLEU was introduced by Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu in a 2002 paper titled BLEU: a Method for Automatic Evaluation of Machine Translation. Its motivation was to provide a fast, repeatable alternative to conducting human assessments for every iteration of system development.\n\nThe central assumption is that, across a sufficiently large collection of examples, better translations will tend to share more words and longer word sequences with professional reference translations. Several references can accommodate alternative acceptable word choices and constructions, although no finite reference set can represent every valid translation.\n\nBLEU was designed primarily for corpus-level evaluation: comparing systems over an entire test corpus. Its original evaluation demonstrated correspondence with human judgments in the settings studied, not a universal relationship between a particular BLEU score and an absolute level of translation quality."
    },
    {
      "heading": "Calculation",
      "text": "BLEU combines modified n-gram precision with a brevity penalty. An n-gram is a contiguous sequence of \\(n\\) tokens: a unigram contains one token, a bigram two, and so forth. Standard BLEU commonly uses orders one through four, with equal weights.\n\nModified n-gram precision\n\nOrdinary precision would count the proportion of candidate n-grams that occur in a reference. Without a restriction on repeated matches, however, a candidate could obtain excessive credit by repeating a common word.\n\nBLEU therefore clips each candidate n-gram count to the largest count found in any individual reference for the same segment. For a candidate segment \\(h_i\\), reference set \\(R_i\\), and n-gram \\(g\\), the clipped count is\n\n\\[m_i(g)= \\min\\left( \\operatorname{count}_{h_i}(g), \\max_{r\\in R_i}\\operatorname{count}_{r}(g) \\right).\\]\n\nIf a candidate contains a word seven times but no reference contains it more than twice, at most two occurrences receive unigram credit. With multiple references, the limit is the maximum count in one reference, not the sum of counts across references.\n\nCorpus-level modified precision for order \\(n\\) is\n\n\\[p_n= \\frac{ \\displaystyle\\sum_i\\sum_{g\\in G_n(h_i)}m_i(g) }{ \\displaystyle\\sum_i\\sum_{g\\in G_n(h_i)} \\operatorname{count}_{h_i}(g) },\\]\n\nwhere \\(G_n(h_i)\\) is the set of distinct n-grams in candidate segment \\(i\\).\n\nMatches are counted within corresponding candidate–reference segments; an n-gram in one sentence does not receive credit merely because it appears in a reference for another sentence. N-grams are not formed across sentence boundaries.\n\nBrevity penalty\n\nPrecision alone can reward an incomplete translation that contains only a few correct words. BLEU addresses this problem with a brevity penalty:\n\n\\[\\mathrm{BP}= \\begin{cases} 1, & c\\ge r,\\\\[4pt] \\exp(1-r/c), & 0<c<r, \\end{cases}\\]\n\nwhere \\(c\\) is the total candidate length and \\(r\\) is the effective reference length.\n\nFor multiple references, standard BLEU selects, for each segment, the reference length closest to the candidate length. A tie is resolved in favor of the shorter reference. The selected lengths are then summed to obtain \\(r\\).\n\nThe penalty is applied once at corpus level. A candidate longer than its effective reference receives no explicit length penalty, although unsupported additional words can reduce its modified precision. An empty candidate output receives a score of zero.\n\nCombined score\n\nThe BLEU score is\n\n\\[\\mathrm{BLEU} = \\mathrm{BP} \\exp\\left( \\sum_{n=1}^{N}w_n\\log p_n \\right), \\qquad \\sum_{n=1}^{N}w_n=1.\\]\n\nFor the usual four-order version, often called BLEU-4, \\(N=4\\) and \\(w_n=1/4\\). Thus, BLEU multiplies the geometric mean of the modified precisions by the brevity penalty.\n\nThe mathematical score lies between 0 and 1, but results are commonly multiplied by 100. A reported score of 30 therefore normally means 0.30, not that 30% of sentences are correct or that the translation has 30% of human quality."
    },
    {
      "heading": "Worked example",
      "text": "Consider a single reference and candidate, tokenized by spaces:\n\n- Reference: the cat is on the mat\n- Candidate: the cat is on mat\n\nAll five candidate words receive unigram credit. Of the four candidate bigrams, three occur in the reference. Two of its three trigrams match, and one of its two four-grams matches:\n\n\\[p_1=1,\\qquad p_2=\\frac34,\\qquad p_3=\\frac23,\\qquad p_4=\\frac12.\\]\n\nThe candidate has five tokens and the reference six, so\n\n\\[\\mathrm{BP}=\\exp(1-6/5)=\\exp(-0.2).\\]\n\nWith equal weights,\n\n\\[\\mathrm{BLEU} = \\exp(-0.2) \\left( 1\\cdot\\frac34\\cdot\\frac23\\cdot\\frac12 \\right)^{1/4} \\approx 0.579.\\]\n\nOn the 0–100 reporting scale, this is approximately 57.9. The example illustrates the calculation, not a general interpretation of translation quality from a single sentence."
    },
    {
      "heading": "Corpus BLEU, sentence BLEU, and smoothing",
      "text": "Corpus BLEU is not the arithmetic mean of sentence BLEU scores. It first aggregates matching counts, candidate counts, and lengths across the test set, then computes the score. Averaging separately calculated sentence scores generally gives a different result.\n\nThis distinction matters because short segments contain few higher-order n-grams. In unsmoothed BLEU, if any positively weighted precision \\(p_n\\) is zero, the geometric mean is zero. A sentence can therefore receive zero despite sharing many words with its reference. Segments shorter than the maximum n-gram order also require an implementation convention for orders with no candidate n-grams.\n\nSentence-level variants often use smoothing, such as adding small counts or assigning a nonzero estimate to otherwise zero precisions. Some implementations also offer an effective-order option that uses only available n-gram orders. These modifications make short-text scores more usable, but their choices affect results and must be specified."
    },
    {
      "heading": "Tokenization and reproducibility",
      "text": "BLEU operates on tokens, so tokenization is part of the evaluation procedure. Splitting punctuation from words, changing case, or applying different normalization rules can alter both matching counts and sequence lengths.\n\nThis is especially important for languages whose conventional writing does not consistently separate words with spaces. Different methods of Chinese word segmentation, for example, can produce different BLEU scores for the same underlying output. A word-tokenized score is also not directly interchangeable with one calculated over characters or subword units such as WordPiece tokens.\n\nSacreBLEU is a widely used implementation designed to improve reproducibility through standardized processing, support for common test sets, and a configuration signature. Its signatures record relevant settings, such as tokenization, case handling, reference count, smoothing, effective-order behavior, and software version.\n\nMeaningful comparisons require the same test data, references, preprocessing, and scoring configuration. Labels such as “BLEU,” “tokenized BLEU,” or “case-insensitive BLEU” alone may not identify the procedure sufficiently."
    },
    {
      "heading": "Uses and interpretation",
      "text": "BLEU is commonly used to:\n\n- Compare translation systems on a fixed test set.\n- Track changes during model development.\n- Select checkpoints or tune system settings.\n- Reproduce historical translation benchmarks.\n\nIts practical advantages include speed, transparency, and independence from a learned evaluator. Provided the evaluation setup is fixed, its component counts and penalties can be inspected directly.\n\nThere is no universal threshold separating good from bad translation. Score distributions vary with language pair, domain, reference style, number of references, and tokenization. Adding references cannot reduce the clipped matching counts for a fixed candidate, but it can also change the effective reference length; scores obtained with different reference sets should not be treated as equivalent measurements.\n\nSmall differences should be interpreted in light of test-set variability. Methods from statistics, including paired bootstrap resampling, can help assess whether an observed difference is stable under resampling. Statistical evidence of a difference does not by itself establish its practical importance."
    },
    {
      "heading": "Limitations",
      "text": "Surface overlap is not meaning equivalence\n\nBLEU does not explicitly model meaning, factual consistency, or translation equivalence. It ordinarily compares the candidate with target-language references without examining the source.\n\nA valid paraphrase may receive a low score because its wording differs from the reference. Conversely, an output with substantial lexical overlap can receive considerable credit despite a consequential error, such as an incorrect name, number, or negation.\n\nLimited treatment of order and structure\n\nHigher-order n-grams capture local word order, but BLEU does not directly analyze sentence structure, long-distance dependencies, or document coherence. Its aggregated matches need not correspond to one coherent alignment with a single reference.\n\nInflectional alternatives and different grammatical constructions can also reduce overlap even when they preserve meaning. BLEU is therefore sensitive to linguistic form in ways that do not always match human judgments.\n\nDependence on references and evaluation conditions\n\nA reference is an example of an acceptable translation, not an exhaustive definition of correctness. References may prefer particular vocabulary, registers, or constructions. A system that resembles those choices may score better than an equally acceptable system using different ones.\n\nConsequently, BLEU scores are unsuitable as stand-alone measures of absolute quality across unrelated datasets or languages. Repeated optimization against the same evaluation material can also encourage improvements specific to that material rather than general translation ability.\n\nIncomplete coverage of human evaluation criteria\n\nHuman assessors may consider adequacy, fluency, terminology, omissions, discourse consistency, and the severity of individual errors. BLEU compresses surface overlap and length into one number and does not separately measure these dimensions.\n\nOther metrics address different aspects of evaluation. chrF uses character n-gram precision and recall, making it sensitive to partial word overlap. COMET uses learned representations and, in reference-based versions, information from the source, candidate, and reference. These methods have their own dependencies and limitations; neither they nor BLEU are interchangeable with a detailed human error analysis."
    }
  ],
  "links": [
    "machine-translation",
    "translation",
    "text-corpus",
    "n-gram",
    "tokenization",
    "capitalization",
    "chinese-word-segmentation",
    "statistics",
    "semantics",
    "word-order"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}