Word error rate (WER) is a measure of the difference between a reference transcript and a candidate transcript, most commonly used to evaluate automatic speech recognition. It is the minimum number of word substitutions, deletions, and insertions needed to transform one transcript into the other, divided by the number of words in the reference. Lower values indicate closer agreement with the reference under the chosen transcription and scoring conventions. WER measures word-sequence agreement, not necessarily preservation of meaning or practical usefulness.
Definition
Given a reference transcript containing words, word error rate is
where:
- is the number of substitutions: a reference word is replaced by a different word.
- is the number of deletions: a reference word is missing from the candidate.
- is the number of insertions: an extra word appears in the candidate.
The counts come from an alignment that minimizes the total number of these operations, with each operation normally assigned a cost of one. WER is therefore a normalized form of word-level Levenshtein edit distance.
WER is often reported as a percentage. A value of means exact agreement after any preprocessing used by the evaluator. It is not bounded above by : sufficiently many inserted words can produce a larger value. For example, a one-word reference and a completely different three-word candidate require one substitution and two insertions, giving a WER of .
The normalization makes WER asymmetric. Exchanging the reference and candidate preserves the unnormalized edit distance but generally changes the denominator and therefore the score.
Example
Consider the following transcripts:
| Transcript | Words |
|---|---|
| Reference | the cat sat on the mat |
| Candidate | the dog sat the mat today |
An optimal alignment is:
Reference: the cat sat on the mat —
Candidate: the dog sat — the mat today
This alignment contains:
- One substitution: “cat” becomes “dog”.
- One deletion: “on” is omitted.
- One insertion: “today” is added.
There are six reference words, so
The alignment matters because words are not compared only at matching positions. Without alignment, an early insertion or deletion could make many later words appear incorrect even when their sequence is otherwise preserved.
Computation
WER is usually computed with dynamic programming. Let the reference be , the candidate be , and be the minimum edit cost between their first and words. The recurrence is
with boundary conditions
The indicator is zero for matching words and one otherwise. The final distance is , and
The standard algorithm has time complexity . Storing the complete table also takes space; computing only the distance can use less memory by retaining a limited number of rows.
Tracing back through the table produces an alignment and separate substitution, deletion, and insertion counts. Multiple optimal alignments can exist. They have the same total edit cost but can differ in how they divide that cost among error types, so tie-breaking conventions matter when analyzing the individual counts.
Tokenization and normalization
WER is defined over words, but what counts as a word depends on the scoring procedure. Tokenization and text normalization are therefore part of the metric’s operational definition.
Common scoring decisions include:
- Whether letter case is ignored.
- Whether punctuation is removed or scored as separate tokens.
- Whether numerals and written-out numbers are treated as equivalent.
- Whether contractions are split or expanded.
- Whether filled pauses, repetitions, and incomplete words are included.
- How hyphenated words, abbreviations, and compound expressions are represented.
For example, “21” and “twenty one” may express the same content but have different token sequences. Similarly, “don't” and “do not” can produce errors unless the evaluation convention normalizes them consistently.
These choices are especially important for languages whose writing systems do not consistently separate words with spaces. Different segmentation rules can produce different WER values for the same pair of transcripts.
The scoring tokens also need not be the internal tokens used by a language model. A recognizer may generate subword units, but standard WER evaluation generally reconstructs text and applies the specified word-level scoring procedure.
Consequently, WER values are directly comparable only when reference conventions, normalization, segmentation, and scoring rules are sufficiently consistent.
Evaluation over a dataset
For a test set containing multiple utterances, corpus-level WER is normally computed by pooling error counts:
For utterances with nonempty references, this is a reference-length-weighted average of their individual WER values. It is generally not equal to the unweighted mean of utterance-level WERs. An unweighted mean gives a short utterance the same influence as a much longer one.
A reference with zero words requires special treatment because its individual WER has a zero denominator. Insertions from such an utterance can still contribute to a corpus-level score when the total number of reference words across the corpus is positive. Software packages may use different conventions for displaying empty-reference cases.
Transcripts are commonly aligned separately for each utterance or recording segment before counts are summed. Aligning all transcripts as one unrestricted sequence can allow matches across boundaries and produce different results.
Two useful ways to describe improvement are:
- Absolute reduction: a change from to is a reduction of 5 percentage points.
- Relative reduction: the same change is a reduction of .
These quantities should not be confused. A relative reduction from a zero baseline is undefined.
Applications
WER is widely used to compare speech-recognition systems, evaluate model updates, and measure robustness across recording conditions. It can reveal changes associated with background noise, speaking style, vocabulary, language variety, or a distribution shift between development data and deployment data.
Separate substitution, deletion, and insertion counts provide additional diagnostic information. For example, a lower total WER can conceal an increase in insertions offset by fewer deletions.
WER can also evaluate other systems that produce text intended to reproduce a fixed reference. Its interpretation is less straightforward when many substantially different outputs are equally valid. In machine translation or automatic summarization, alternative wording can preserve meaning while receiving a high word-level edit score.
Limitations
Errors have equal cost, not equal importance
Standard WER assigns the same cost to every substitution, deletion, and insertion. It does not distinguish a minor grammatical error from an incorrect name, number, or negation. A transcript with lower WER can therefore be less useful for a particular task than one with higher WER.
Weighted edit measures and task-specific metrics can distinguish such errors, but they are not interchangeable with standard WER.
Agreement is not semantic correctness
WER evaluates surface word sequences. It does not directly measure meaning, readability, factual correctness, or whether a user can complete a task using the transcript.
Reference quality also matters. Incorrect reference words, inconsistent conventions, or mismatches between verbatim and edited transcription styles can increase the measured error rate without representing a corresponding recognition failure.
Aggregate scores conceal variation
A single corpus-level value can hide differences among speakers, acoustic environments, language varieties, and subject domains. Longer recordings also contribute more heavily to the pooled score.
Observed differences between systems depend on the test sample. Statistical comparisons should account for the fact that competing systems are usually evaluated on the same recordings and that errors within a speaker or recording may be correlated.
Standard WER excludes other aspects of transcription
Unless explicitly incorporated into the scoring protocol, WER does not evaluate timestamps, speaker attribution, punctuation quality, or capitalization. Two outputs can have identical WER while differing substantially in these properties.
Related measures
Character error rate (CER) applies the same edit-distance principle to characters rather than words. It is useful when word segmentation is ambiguous or when character-level differences are of interest. Its value still depends on conventions for whitespace, punctuation, and character normalization.
Sentence error rate (SER) is the proportion of scored sentences or utterances containing at least one error. It measures how often an entire unit is transcribed perfectly but does not distinguish a unit with one error from one with many errors.
Word accuracy is sometimes defined as . Because WER can exceed one, this quantity can be negative and should not automatically be interpreted as the proportion of reference words correctly recognized.