Part-of-speech tagging is the assignment of grammatical category labels, such as noun, verb, adjective, or determiner, to words or other tokens in a text. It is a basic task in natural language processing and a common stage in the annotation of linguistic corpora. Unlike a dictionary entry, which may list several possible categories for a word, a tagger normally selects the category appropriate to a particular occurrence. Its output depends both on the surrounding context and on the annotation scheme being used.
Linguistic basis
A part of speech, or word class, groups words according to their grammatical behavior. Relevant evidence includes their distribution in sentences, their patterns of inflection, and their relationships with other words. Categories are therefore not determined by meaning alone: nouns do not exclusively name physical things, and verbs do not exclusively denote actions.
Many words can belong to different categories in different contexts. In English, book is a noun in “a useful book” but a verb in “book a ticket.” Likewise, can is an auxiliary in “they can swim” and a noun in “an empty can.” Tagging resolves this kind of contextual ambiguity, though some cases remain uncertain or depend on theoretical choices.
Part-of-speech labels describe lexical or grammatical categories, not complete sentence structure. A noun is a category; subject and object are syntactic functions. Identifying how words combine and which words depend on others belongs to syntactic parsing. Tagging provides useful evidence for parsing, but it does not by itself recover the structure of a sentence.
The boundary between tagging and morphological analysis varies. A coarse annotation may identify a word simply as a verb. A richer scheme may also record tense, number, person, mood, or other properties. Some systems encode these distinctions in a single tag; others keep the part-of-speech label separate from morphological features.
Tokens and tagsets
The unit being labeled
Tagging generally operates on tokens produced by tokenization, rather than on an intuitively fixed notion of “word.” Tokens may include punctuation marks, numerals, symbols, or components of a written word.
Different tokenization conventions can produce different analyses of the same text. English don't, for example, can be split into do and n't. In languages where spaces do not regularly mark word boundaries, segmentation is a substantial part of the processing pipeline. Chinese word segmentation and part-of-speech tagging may be performed sequentially or jointly, because the choice of boundaries can affect the grammatical analysis.
An orthographic unit may also contain multiple syntactic words. Annotation frameworks can distinguish the surface form from the units that receive grammatical labels. Consequently, token counts and tagging results are not directly comparable unless tokenization conventions are aligned.
Tagset design
A tagset is the inventory of labels available to annotators or a tagging system. Its granularity reflects the intended application and linguistic assumptions.
The Penn Treebank tagset distinguishes several forms of English nouns and verbs. Its noun tags include NN for a singular or mass noun and NNS for a plural noun; its verb tags include VB for a base form and VBD for a past-tense form. Under this scheme, a sentence might be tagged as follows:
| Token | Tag | Interpretation |
|---|---|---|
| The | DT | Determiner |
| dogs | NNS | Plural noun |
| barked | VBD | Past-tense verb |
| loudly | RB | Adverb |
| . | . | Sentence-final punctuation |
Universal Dependencies instead uses 17 universal part-of-speech categories, including NOUN, PROPN, VERB, AUX, ADJ, and ADV, alongside labels for punctuation, symbols, and other material. Morphological information is represented separately. This supports cross-linguistic consistency while allowing language-specific features and annotation conventions.
Mapping between tagsets is not always lossless. A fine-grained scheme may encode distinctions absent from a coarse one, while two schemes may draw category boundaries differently. Reported results therefore depend partly on what counts as a correct label.
Computational formulation
Given a sequence of tokens , a probabilistic tagger seeks a sequence of tags that maximizes a conditional probability:
This formulation treats tagging as a sequence-labeling problem. The best label for one token may depend on neighboring words, neighboring tags, or a representation of the whole sentence.
A simple baseline assigns each known word the tag it most frequently received in the training data. Such a method exploits lexical regularities but cannot adequately resolve contextual ambiguity. It also needs a fallback strategy for unseen words. More sophisticated systems combine lexical evidence with contextual and morphological information.
Some taggers choose labels independently after computing contextual representations. Others explicitly model dependencies between output labels and search for the best complete sequence. Thus, using sentence context and performing structured sequence decoding are related but distinct design choices.
Main approaches
Rule-based methods
Rule-based taggers combine lexical information with contextual rules. A lexicon may supply candidate categories, while rules select or eliminate candidates according to grammatical patterns.
Rules can encode facts about suffixes, capitalization, nearby function words, or permitted category sequences. They are interpretable and can incorporate linguistic expertise directly, but building and maintaining broad-coverage rule systems is labor-intensive. Rules may also depend strongly on the language, genre, and annotation conventions.
Transformation-based tagging combines an initial analysis with learned correction rules. A well-known example is the Brill tagger, which learns ordered transformations that improve an initial tagging on annotated examples.
Statistical sequence models
A hidden Markov model treats tags as hidden states and words as observed outputs. In a first-order version, the joint probability is approximated by
This model combines tag-transition probabilities with word-emission probabilities. Higher-order versions can condition a tag on more preceding tags, and dynamic programming can recover the highest-probability sequence efficiently.
Discriminative methods learn how features distinguish tags without requiring the same generative assumptions. Maximum-entropy taggers use features such as word identity, affixes, and neighboring tokens. A conditional random field models the conditional probability of an entire label sequence and can incorporate overlapping contextual features.
These approaches made tagging a prominent application of machine learning in language technology.
Neural methods
Taggers based on artificial neural networks learn representations of words and their contexts. Character-level representations can capture spelling patterns and morphological regularities, helping with rare and unseen forms. Recurrent models can encode information from both preceding and following tokens.
Pretrained contextual language models provide representations that vary with the sentence in which a token occurs. A tagging classifier can be trained on top of these representations, sometimes together with a structured decoding layer.
Subword-based models introduce an alignment issue: the units processed internally may not correspond to the words that require tags. A system must therefore specify how subword representations and word-level labels are connected. Modern pipelines may also predict parts of speech jointly with morphological features or syntactic dependencies.
Historical development and annotated resources
Computational tagging developed alongside corpus linguistics and the creation of machine-readable text collections. The Brown Corpus became an important resource for early English grammatical annotation and automatic tagging.
During the late twentieth century, rule-based systems, probabilistic models, and transformation-based learning established tagging as a widely studied computational task. The Penn Treebank supplied influential annotation conventions and benchmark data, while the growing availability of treebanks connected tagging more closely with parsing research.
Subsequent work expanded beyond well-resourced written languages toward multilingual processing, morphologically rich languages, historical texts, conversational language, and informal online writing. Universal Dependencies has provided a shared framework for part-of-speech, morphological, and dependency annotation across many languages.
Training and evaluation
Supervised taggers learn from text whose tokens have been manually labeled or automatically labeled and then checked. Annotation guidelines define the intended interpretation of difficult constructions. Agreement between annotators helps reveal both guideline problems and genuine analytical ambiguity.
The usual evaluation measure is token accuracy:
This measure is useful but incomplete. Frequent categories can dominate the result, masking poor performance on rare categories or unfamiliar vocabulary. Evaluation may therefore also report per-category scores, confusion matrices, sentence-level accuracy, or separate results for out-of-vocabulary words.
Reliable comparisons require compatible tagsets, tokenization, data splits, and treatment of punctuation. If systems also perform segmentation, evaluation must account for incorrect boundaries rather than considering only labels on already correct tokens.
Performance on a familiar benchmark does not guarantee comparable performance elsewhere. Differences in genre, period, spelling conventions, or subject matter create domain shift. A tagger trained on edited news text may behave differently on poetry, transcripts, or social-media posts.
Applications
Part-of-speech annotation supports several kinds of analysis and processing:
- Corpus research: measuring grammatical patterns, retrieving constructions, and comparing varieties or historical periods.
- Syntactic analysis: supplying category information to parsers, although some parsers infer it internally.
- Lexicography: identifying uses of words and helping organize evidence for dictionary entries.
- Information extraction and retrieval: supporting phrase patterns, terminology extraction, and other structured searches.
- Language technology: contributing to text-to-speech processing, grammatical error analysis, and some translation systems.
Not every modern application requires a separate tagging stage. End-to-end models can learn representations useful for these tasks without explicitly producing part-of-speech labels. Explicit tags nevertheless remain valuable when systems require interpretable linguistic annotations or searchable grammatical structure.
Limitations and cross-linguistic issues
Part-of-speech categories are not identically organized across languages. A distinction straightforward in one language may be less clearly applicable in another. Cross-linguistic tagsets therefore function as operational annotation frameworks, not as proof that every language has the same grammatical categories.
Ambiguity can also exceed what local context resolves. A construction may permit multiple grammatical analyses, and a required single label may conceal that uncertainty. Annotation guidelines often select one consistent analysis for practical reasons.
Errors commonly involve unfamiliar names, abbreviations, spelling variation, nominalized forms, and words used in unusual constructions. Code-switching introduces additional difficulties because category assignments and grammatical patterns may depend on more than one language.
For low-resource languages, the main constraint may be limited annotated data, incomplete lexical resources, or unsettled annotation conventions rather than the choice of model. Multilingual transfer can reduce data requirements, but its effectiveness depends on linguistic similarity, writing systems, training coverage, and the consistency of the labels being transferred.