A confusion matrix is a table that compares the class labels assigned by a classification system with reference labels for the same observations. Its entries record how often each reference class is predicted as each possible class. Used in statistics and machine learning, it distinguishes correct predictions from particular kinds of mistakes rather than reducing performance to a single score. The reference labels are often called actual, true, or ground-truth labels, although they may themselves contain errors.
Structure and mathematical definition
For a classification task with mutually exclusive classes, a confusion matrix is a matrix. A common convention places reference classes in rows and predicted classes in columns:
where is the number of evaluated observations, is the reference label, is the predicted label, and is an indicator function.
Under this convention:
- counts correct predictions for class .
- , for , counts observations of class incorrectly assigned to class .
- The row sum is the number of reference observations in class , often called its support.
- The column sum is the number of predictions assigned to class .
- The sum of all entries is .
Some publications and software packages reverse the axes. Consequently, row and column labels must be checked before interpreting a table; the phrase “confusion matrix” alone does not establish its orientation.
A weighted confusion matrix replaces each unit count with an observation weight :
Weighted entries need not be integers. They may represent sampling weights, exposure, or another explicitly defined weighting scheme.
Binary classification
For two classes, one is conventionally designated positive and the other negative. These labels identify the outcome of interest; “positive” does not necessarily mean desirable.
With reference labels in rows, the matrix is:
| Reference class | Predicted positive | Predicted negative |
|---|---|---|
| Positive | True positive, | False negative, |
| Negative | False positive, | True negative, |
A false positive is a negative observation classified as positive. A false negative is a positive observation classified as negative. The distinction matters whenever the two errors have different consequences, such as in spam filtering, fault detection, or document retrieval.
These errors are related to Type I and Type II errors in hypothesis testing when a positive decision corresponds to rejecting a negative null hypothesis. The correspondence depends on that setup; classification terminology does not itself specify a hypothesis test.
Metrics derived from the matrix
Many classification metrics are ratios of these four counts.
| Metric | Formula | Interpretation |
|---|---|---|
| Accuracy | Fraction of all predictions that are correct | |
| Error rate | Fraction of all predictions that are incorrect | |
| Precision | Fraction of positive predictions that are reference-positive | |
| Recall, sensitivity, or true positive rate | Fraction of reference-positive observations detected | |
| Specificity or true negative rate | Fraction of reference-negative observations correctly rejected | |
| False positive rate | Fraction of reference-negative observations incorrectly flagged | |
| False negative rate | Fraction of reference-positive observations missed | |
| Negative predictive value | Fraction of negative predictions that are reference-negative |
Precision and recall condition on different groups: precision concerns predicted positives, whereas recall concerns reference positives. Neither determines the other.
The F1 score is their harmonic mean:
It does not directly include true negatives. Binary balanced accuracy gives equal weight to the recall of each reference class:
A metric with a zero denominator is mathematically undefined. Software may return a missing value, issue a warning, or substitute a specified value. Such conventions can affect comparisons, especially when a class is absent or never predicted.
Worked example
Suppose a classifier is evaluated on 1,000 observations:
| Reference class | Predicted positive | Predicted negative | Total |
|---|---|---|---|
| Positive | 80 | 20 | 100 |
| Negative | 90 | 810 | 900 |
| Total | 170 | 830 | 1,000 |
Its accuracy is
Its recall is , its specificity is , and its precision is approximately . Its F1 score is approximately , while balanced accuracy is .
A system that always predicts negative would achieve 90% accuracy on these observations, but zero recall for the positive class. This illustrates why accuracy alone can obscure performance when class frequencies are unequal. Whether either system is preferable depends on the purpose of classification and the relative consequences of its errors.
Counts, normalization, and probability
The raw matrix preserves the number of observations behind each result. Normalized matrices emphasize proportions instead.
Overall normalization
Dividing every entry by produces
These values form the empirical joint probability distribution of reference and predicted labels. They sum to one across the whole matrix.
Row normalization
Dividing each entry by its row total gives
For nonempty rows, this estimates the conditional probability
Each such row sums to one. Diagonal entries are class-specific recalls, making this display useful for comparing how reliably different reference classes are recognized.
Column normalization
Dividing by column totals gives
which estimates
Each nonempty column sums to one, and diagonal entries are class-specific precisions.
Row- and column-normalized matrices answer different questions. A normalized display should identify its denominator and, where possible, retain class counts so that a proportion based on a few observations is distinguishable from one based on thousands.
Multiclass and multilabel classification
Multiclass classification
In a single-label multiclass task, every observation belongs to exactly one of classes and receives exactly one predicted class. The off-diagonal entries reveal which classes are confused with which others.
Each class can also be evaluated against all remaining classes:
These quantities yield per-class precision, recall, and F-scores. Common summaries include:
- Macro averaging: calculate a metric for each class and take the unweighted mean.
- Support-weighted averaging: weight each class’s metric by its reference support.
- Micro averaging: sum the relevant counts across classes before calculating the metric.
In ordinary single-label multiclass evaluation over all classes, micro-averaged precision, recall, and F1 all equal accuracy. Macro-averaged recall gives each class equal weight and is commonly called multiclass balanced accuracy.
The averaging rule matters. Macro F1, for example, usually means the mean of the class-specific F1 scores, not the harmonic mean of macro precision and macro recall.
Multilabel classification
In multilabel classification, an observation may have several labels simultaneously. A single table generally cannot represent all label-presence decisions.
A common approach constructs a separate binary confusion matrix for each label, treating that label as present or absent. These matrices support per-label and aggregated metrics, but do not fully capture dependencies among labels or the accuracy of complete predicted label sets.
Thresholds and decision costs
Many classifiers, including logistic regression, produce scores or estimated probabilities before assigning labels. In binary classification, a threshold may define the decision rule:
For fixed scores and observations, lowering the threshold cannot decrease either the true positive count or the false positive count. A confusion matrix therefore describes performance at a particular decision rule, rather than every possible operating point. Receiver operating characteristic curves and precision–recall curves summarize performance across thresholds.
Error costs can be incorporated using a loss function. If is the cost of predicting class when the reference class is , the average observed cost is
When correct decisions have zero cost and every incorrect decision has unit cost, this reduces to the error rate. Different costs can favor different classifiers or thresholds even when their accuracies are similar, connecting confusion-matrix analysis with decision theory.
Applications and limitations
Confusion matrices are used to compare classifiers, inspect error patterns, identify poorly recognized classes, and investigate whether a model systematically substitutes one label for another. Applications include text classification, image recognition, remote sensing, speech recognition, and industrial inspection.
Their interpretation has several important limits.
Class frequencies affect the results. Precision depends on the proportion of positive cases in the evaluated population. If that proportion is , sensitivity is , and the false positive rate is , then Bayes’ theorem gives
provided the denominator is nonzero. Precision can therefore change across populations even if sensitivity and specificity remain unchanged.
Evaluation data must represent the intended setting. Results on training data generally do not establish performance on unseen observations. Held-out evaluation and cross-validation address aspects of this problem, but distribution changes, nonrepresentative sampling, and information leakage can still undermine interpretation.
Reference labels may be imperfect. Disagreement with a reference label does not establish that the classifier is wrong in an absolute sense. Ambiguous definitions, annotation errors, and disagreements among annotators can contribute to off-diagonal counts.
Counts have sampling uncertainty. Rates calculated from finite samples vary with the observations selected. Confidence intervals or bootstrap methods can quantify aspects of that uncertainty. Dependence among observations, such as repeated records from the same person, must be considered when estimating it.
Hard labels discard score information. Two models can have identical confusion matrices while assigning very different probabilities. The matrix alone cannot establish probability calibration or determine score-based losses such as cross-entropy.
Aggregation hides structure. A matrix does not preserve individual observations, subgroup differences, time ordering, or the circumstances of each error. Additional analyses are needed to determine where errors occur and whether performance differs across relevant groups.
For structured prediction tasks such as object detection, a confusion matrix also depends on how predictions are matched to reference objects. Rejected predictions, unknown classes, unmatched detections, or missing reference objects require explicit conventions; otherwise, apparently similar matrices may count different events.