Multiclass classification is a supervised learning task in which a model assigns an input to exactly one of three or more predefined categories. Examples include recognizing a handwritten digit as one of ten digits, assigning a document to a single topic, and identifying the main object in an image. It extends binary classification, which distinguishes between only two classes, and differs from multilabel classification, in which an input may receive several labels simultaneously.
Problem formulation
A multiclass classification problem has an input space and a finite label set , with . Given training data
the objective is to learn a classifier
that predicts labels accurately for previously unseen inputs.
The labels represent categories, not numerical quantities. Encoding three animal species as 1, 2, and 3 does not imply that the third species is greater than the second or that the distance between species is meaningful. One-hot encoding represents each class by a vector whose corresponding entry is 1 and whose other entries are 0, although many implementations accept integer class indices directly.
A probabilistic classifier estimates conditional class probabilities,
When every incorrect classification has the same cost, the standard prediction rule selects the class with the largest estimated probability:
With the true conditional probabilities, this rule minimizes the probability of misclassification. If different errors have different costs, the appropriate rule instead minimizes expected cost.
Related classification settings
Several neighboring problems have different output assumptions:
- Multilabel classification: each input can belong to several classes, such as a photograph labeled both “beach” and “sunset.”
- Ordinal classification: classes have an inherent order, such as low, medium, and high severity.
- Hierarchical classification: classes occupy a taxonomy, and predictions may be made at different levels.
- Open-set recognition: inputs may belong to classes absent from training, rather than necessarily belonging to one of the known classes.
Standard multiclass classification is usually closed-set: the available classes are assumed to cover all admissible inputs. A classifier can nevertheless produce a label for an input outside this intended domain, so producing an answer does not establish that the input belongs to a known class.
Modeling approaches
Multiclass problems can be handled either by models that represent all classes jointly or by combinations of binary classifiers.
Joint multiclass models
Several machine-learning methods naturally support more than two classes:
- Decision trees partition the input space and assign class labels or class distributions to their leaves.
- Random forests combine predictions from multiple trees.
- A naive Bayes classifier combines class priors with class-conditional feature likelihoods under a conditional-independence assumption.
- The k-nearest neighbors algorithm predicts from the labels of nearby training examples.
- Artificial neural networks can produce a score for each class and learn all scores jointly.
Multinomial logistic regression uses a linear score for each class followed by a softmax transformation. Neural classifiers generalize this arrangement by learning nonlinear representations before the final class-scoring layer.
One-versus-rest
The one-versus-rest strategy trains binary classifiers. Classifier distinguishes class from all other classes. At prediction time, the class with the highest decision score is typically selected.
This approach allows binary learning methods to be reused, but each binary problem may be imbalanced, especially when is large. Independent binary predictions may also select several positive classes or none at all; a score-based selection rule resolves the final single-label prediction.
Scores from separately trained classifiers are not automatically comparable as probabilities. Their interpretation depends on the underlying model and any subsequent calibration.
One-versus-one
The one-versus-one strategy trains a classifier for every pair of classes, giving
binary classifiers. Each classifier is trained using examples from only its two classes. Predictions are combined through voting or another aggregation procedure.
Although this produces many classifiers, each uses a smaller training subset. It has been widely used with support vector machines. Its practical cost depends on the learning algorithm, class sizes, and prediction procedure, not just the number of classifiers.
Error-correcting output codes
Error-correcting output codes assign each class a binary or ternary codeword. Binary classifiers predict code components, and a decoding rule chooses the closest class codeword.
Inspired by error-correcting codes, this approach can provide redundancy: some component predictions may be wrong without changing the decoded class. Its effectiveness depends on code design, classifier accuracy, and diversity among component errors.
Softmax and training objectives
A common multiclass model produces real-valued scores, called logits, . The softmax function converts these into a normalized probability vector:
For numerical stability, implementations commonly subtract the largest logit before exponentiation. This leaves the probabilities unchanged.
The standard loss function for probabilistic multiclass classification is categorical cross-entropy. For a training example whose correct class is ,
For one-hot targets , the equivalent expression is
Minimizing average cross-entropy corresponds to maximizing the conditional likelihood of the observed labels. Neural networks commonly optimize this objective using backpropagation and gradient-based methods.
Cross-entropy is not the only option. Multiclass margin losses, including hinge-style objectives, encourage the correct class score to exceed competing scores by a specified margin. Class weighting changes the relative influence of examples from different categories. Label smoothing replaces hard one-hot targets with softened targets and can affect both learning and confidence estimates.
Regularization constrains model fitting and can reduce overfitting. A softmax output alone does not ensure that reported probabilities accurately reflect real-world frequencies.
Evaluation
Evaluation requires data separate from model fitting. A validation set supports model selection, while a held-out test set estimates performance after those choices have been made. Duplicated or related observations crossing these boundaries can cause data leakage and inflate measured performance.
Confusion matrix and accuracy
A confusion matrix records the number of examples for every true-class and predicted-class pair. It reveals which categories are confused, although the convention for placing true labels on rows or columns must be specified.
Accuracy is the fraction of correctly classified examples:
It is useful when classes and errors have comparable importance, but it can conceal poor performance on rare categories. A classifier that almost always predicts the largest class may achieve high accuracy without usefully distinguishing the others.
Precision, recall, and F-scores
For each class, all other classes can be treated as the negative category:
The per-class F1 score is
Results can be aggregated in several ways:
- Macro averaging gives every class equal weight.
- Weighted averaging weights class scores by the number of true examples in each class.
- Micro averaging pools counts across classes before computing a score.
In ordinary single-label multiclass evaluation, when all classes are included and every example receives exactly one prediction, micro-averaged precision, recall, and F1 all equal accuracy. Macro F1 is normally the mean of per-class F1 scores, not the F1 computed from macro-averaged precision and recall. Implementations must define how to handle zero denominators.
Ranked predictions and probability quality
Top- accuracy counts a prediction as correct when the true class appears among the highest-scoring classes. It is useful when a system presents several candidates, but depends strongly on the number and difficulty of the available classes.
Log loss and multiclass Brier scores evaluate predicted probabilities rather than only the winning class. Probability calibration concerns whether stated confidence corresponds to observed correctness. A model can classify accurately while being systematically overconfident.
Applications
Multiclass classification appears across computer vision, natural language processing, and speech recognition.
In image classification, a model may assign an image to one object category. In document processing, it may select a topic, language, or routing destination. Character recognition treats individual symbols as classes. In semantic segmentation, each pixel receives a class label, making multiclass prediction a component of a larger structured-output task.
The task depends on the labeling scheme rather than the data type alone. Assigning one topic to an article is multiclass classification; assigning every applicable topic is multilabel classification. Predicting a numerical property from the same article is regression.
Limitations and practical challenges
Class imbalance and scarce examples. Rare classes may contribute little to the overall training objective and have uncertain evaluation estimates. Weighting and resampling alter the learning problem, but their effects must be assessed against the intended deployment distribution.
Ambiguous or inconsistent labels. A single-label formulation may force a choice between overlapping categories. Annotation errors and unclear class definitions can limit achievable agreement independently of model capacity.
Distribution changes. Distribution shift can change input characteristics, class frequencies, or relationships between inputs and labels. Performance on a fixed test set does not guarantee equivalent performance after such changes.
Unknown inputs and confidence. Closed-set models normally select a known class even for unfamiliar inputs. Rejection mechanisms and open-set methods address this separately; high softmax confidence is not reliable proof that an input is familiar.
Large label spaces. As class counts grow, output-layer computation, label storage, and data requirements can become substantial. Hierarchical prediction, candidate retrieval, and approximate normalization can reduce some costs, but introduce additional design choices and possible errors.
Unequal error costs. Ordinary accuracy treats every mistake alike. Where confusing one pair of classes has different consequences from confusing another, evaluation and decision rules must reflect those costs rather than relying only on the most probable label.