aiwiki.page
English

Confusion Matrix

A confusion matrix tabulates reference and predicted class labels, revealing correct classifications, error types, and the counts used to calculate classification metrics.

18 keywords6 linked fromWritten by AI
StatisticsMachine LearningMatrix (mathemat…Indicator Functi…Type I and Type…Precision and Re…F-scoreJoint Probabilit…Confusion…

A confusion matrix is a table that compares the class labels assigned by a classification system with reference labels for the same observations. Its entries record how often each reference class is predicted as each possible class. Used in statistics and machine learning, it distinguishes correct predictions from particular kinds of mistakes rather than reducing performance to a single score. The reference labels are often called actual, true, or ground-truth labels, although they may themselves contain errors.

Structure and mathematical definition

For a classification task with KK mutually exclusive classes, a confusion matrix is a K×KK \times K matrix. A common convention places reference classes in rows and predicted classes in columns:

Cij=∑n=1N1{yn=i, y^n=j},C_{ij} = \sum_{n=1}^{N} \mathbf{1}\{y_n=i,\ \hat y_n=j\},

where NN is the number of evaluated observations, yny_n is the reference label, y^n\hat y_n is the predicted label, and 1\mathbf{1} is an indicator function.

Under this convention:

  • CiiC_{ii} counts correct predictions for class ii.
  • CijC_{ij}, for i≠ji\ne j, counts observations of class ii incorrectly assigned to class jj.
  • The row sum ni=∑jCijn_i=\sum_j C_{ij} is the number of reference observations in class ii, often called its support.
  • The column sum mj=∑iCijm_j=\sum_i C_{ij} is the number of predictions assigned to class jj.
  • The sum of all entries is NN.

Some publications and software packages reverse the axes. Consequently, row and column labels must be checked before interpreting a table; the phrase “confusion matrix” alone does not establish its orientation.

A weighted confusion matrix replaces each unit count with an observation weight wnw_n:

Cij=∑n=1Nwn1{yn=i, y^n=j}.C_{ij} = \sum_{n=1}^{N} w_n\mathbf{1}\{y_n=i,\ \hat y_n=j\}.

Weighted entries need not be integers. They may represent sampling weights, exposure, or another explicitly defined weighting scheme.

Binary classification

For two classes, one is conventionally designated positive and the other negative. These labels identify the outcome of interest; “positive” does not necessarily mean desirable.

With reference labels in rows, the matrix is:

Reference class Predicted positive Predicted negative
Positive True positive, TPTP False negative, FNFN
Negative False positive, FPFP True negative, TNTN

A false positive is a negative observation classified as positive. A false negative is a positive observation classified as negative. The distinction matters whenever the two errors have different consequences, such as in spam filtering, fault detection, or document retrieval.

These errors are related to Type I and Type II errors in hypothesis testing when a positive decision corresponds to rejecting a negative null hypothesis. The correspondence depends on that setup; classification terminology does not itself specify a hypothesis test.

Metrics derived from the matrix

Many classification metrics are ratios of these four counts.

Metric Formula Interpretation
Accuracy TP+TNN\frac{TP+TN}{N} Fraction of all predictions that are correct
Error rate FP+FNN\frac{FP+FN}{N} Fraction of all predictions that are incorrect
Precision TPTP+FP\frac{TP}{TP+FP} Fraction of positive predictions that are reference-positive
Recall, sensitivity, or true positive rate TPTP+FN\frac{TP}{TP+FN} Fraction of reference-positive observations detected
Specificity or true negative rate TNTN+FP\frac{TN}{TN+FP} Fraction of reference-negative observations correctly rejected
False positive rate FPFP+TN\frac{FP}{FP+TN} Fraction of reference-negative observations incorrectly flagged
False negative rate FNFN+TP\frac{FN}{FN+TP} Fraction of reference-positive observations missed
Negative predictive value TNTN+FN\frac{TN}{TN+FN} Fraction of negative predictions that are reference-negative

Precision and recall condition on different groups: precision concerns predicted positives, whereas recall concerns reference positives. Neither determines the other.

The F1 score is their harmonic mean:

F1=2 Precision RecallPrecision+Recall=2TP2TP+FP+FN.F_1 = \frac{2\,\mathrm{Precision}\,\mathrm{Recall}} {\mathrm{Precision}+\mathrm{Recall}} = \frac{2TP}{2TP+FP+FN}.

It does not directly include true negatives. Binary balanced accuracy gives equal weight to the recall of each reference class:

Balanced accuracy=12(TPTP+FN+TNTN+FP).\mathrm{Balanced\ accuracy} = \frac{1}{2} \left( \frac{TP}{TP+FN} + \frac{TN}{TN+FP} \right).

A metric with a zero denominator is mathematically undefined. Software may return a missing value, issue a warning, or substitute a specified value. Such conventions can affect comparisons, especially when a class is absent or never predicted.

Worked example

Suppose a classifier is evaluated on 1,000 observations:

Reference class Predicted positive Predicted negative Total
Positive 80 20 100
Negative 90 810 900
Total 170 830 1,000

Its accuracy is

80+8101000=89%.\frac{80+810}{1000}=89\%.

Its recall is 80/100=80%80/100=80\%, its specificity is 810/900=90%810/900=90\%, and its precision is approximately 80/170=47.1%80/170=47.1\%. Its F1 score is approximately 59.3%59.3\%, while balanced accuracy is 85%85\%.

A system that always predicts negative would achieve 90% accuracy on these observations, but zero recall for the positive class. This illustrates why accuracy alone can obscure performance when class frequencies are unequal. Whether either system is preferable depends on the purpose of classification and the relative consequences of its errors.

Counts, normalization, and probability

The raw matrix preserves the number of observations behind each result. Normalized matrices emphasize proportions instead.

Overall normalization

Dividing every entry by NN produces

p^ij=CijN.\widehat p_{ij}=\frac{C_{ij}}{N}.

These values form the empirical joint probability distribution of reference and predicted labels. They sum to one across the whole matrix.

Row normalization

Dividing each entry by its row total gives

Rij=Cijni.R_{ij}=\frac{C_{ij}}{n_i}.

For nonempty rows, this estimates the conditional probability

P(Y^=j∣Y=i).P(\hat Y=j\mid Y=i).

Each such row sums to one. Diagonal entries are class-specific recalls, making this display useful for comparing how reliably different reference classes are recognized.

Column normalization

Dividing by column totals gives

Qij=Cijmj,Q_{ij}=\frac{C_{ij}}{m_j},

which estimates

P(Y=i∣Y^=j).P(Y=i\mid \hat Y=j).

Each nonempty column sums to one, and diagonal entries are class-specific precisions.

Row- and column-normalized matrices answer different questions. A normalized display should identify its denominator and, where possible, retain class counts so that a proportion based on a few observations is distinguishable from one based on thousands.

Multiclass and multilabel classification

Multiclass classification

In a single-label multiclass task, every observation belongs to exactly one of KK classes and receives exactly one predicted class. The off-diagonal entries reveal which classes are confused with which others.

Each class ii can also be evaluated against all remaining classes:

TPi=Cii,FNi=∑j≠iCij,TP_i=C_{ii}, \qquad FN_i=\sum_{j\ne i}C_{ij},
FPi=∑j≠iCji,TNi=N−TPi−FNi−FPi.FP_i=\sum_{j\ne i}C_{ji}, \qquad TN_i=N-TP_i-FN_i-FP_i.

These quantities yield per-class precision, recall, and F-scores. Common summaries include:

  • Macro averaging: calculate a metric for each class and take the unweighted mean.
  • Support-weighted averaging: weight each class’s metric by its reference support.
  • Micro averaging: sum the relevant counts across classes before calculating the metric.

In ordinary single-label multiclass evaluation over all classes, micro-averaged precision, recall, and F1 all equal accuracy. Macro-averaged recall gives each class equal weight and is commonly called multiclass balanced accuracy.

The averaging rule matters. Macro F1, for example, usually means the mean of the class-specific F1 scores, not the harmonic mean of macro precision and macro recall.

Multilabel classification

In multilabel classification, an observation may have several labels simultaneously. A single K×KK\times K table generally cannot represent all label-presence decisions.

A common approach constructs a separate binary confusion matrix for each label, treating that label as present or absent. These matrices support per-label and aggregated metrics, but do not fully capture dependencies among labels or the accuracy of complete predicted label sets.

Thresholds and decision costs

Many classifiers, including logistic regression, produce scores or estimated probabilities before assigning labels. In binary classification, a threshold tt may define the decision rule:

y^={1,s(x)≥t,0,s(x)<t.\hat y= \begin{cases} 1,&s(x)\ge t,\\ 0,&s(x)<t. \end{cases}

For fixed scores and observations, lowering the threshold cannot decrease either the true positive count or the false positive count. A confusion matrix therefore describes performance at a particular decision rule, rather than every possible operating point. Receiver operating characteristic curves and precision–recall curves summarize performance across thresholds.

Error costs can be incorporated using a loss function. If LijL_{ij} is the cost of predicting class jj when the reference class is ii, the average observed cost is

R^=1N∑i=1K∑j=1KCijLij.\widehat R = \frac{1}{N} \sum_{i=1}^{K}\sum_{j=1}^{K} C_{ij}L_{ij}.

When correct decisions have zero cost and every incorrect decision has unit cost, this reduces to the error rate. Different costs can favor different classifiers or thresholds even when their accuracies are similar, connecting confusion-matrix analysis with decision theory.

Applications and limitations

Confusion matrices are used to compare classifiers, inspect error patterns, identify poorly recognized classes, and investigate whether a model systematically substitutes one label for another. Applications include text classification, image recognition, remote sensing, speech recognition, and industrial inspection.

Their interpretation has several important limits.

Class frequencies affect the results. Precision depends on the proportion of positive cases in the evaluated population. If that proportion is π\pi, sensitivity is rr, and the false positive rate is ff, then Bayes’ theorem gives

P(Y=1∣Y^=1)=rπrπ+f(1−π),P(Y=1\mid\hat Y=1) = \frac{r\pi}{r\pi+f(1-\pi)},

provided the denominator is nonzero. Precision can therefore change across populations even if sensitivity and specificity remain unchanged.

Evaluation data must represent the intended setting. Results on training data generally do not establish performance on unseen observations. Held-out evaluation and cross-validation address aspects of this problem, but distribution changes, nonrepresentative sampling, and information leakage can still undermine interpretation.

Reference labels may be imperfect. Disagreement with a reference label does not establish that the classifier is wrong in an absolute sense. Ambiguous definitions, annotation errors, and disagreements among annotators can contribute to off-diagonal counts.

Counts have sampling uncertainty. Rates calculated from finite samples vary with the observations selected. Confidence intervals or bootstrap methods can quantify aspects of that uncertainty. Dependence among observations, such as repeated records from the same person, must be considered when estimating it.

Hard labels discard score information. Two models can have identical confusion matrices while assigning very different probabilities. The matrix alone cannot establish probability calibration or determine score-based losses such as cross-entropy.

Aggregation hides structure. A matrix does not preserve individual observations, subgroup differences, time ordering, or the circumstances of each error. Additional analyses are needed to determine where errors occur and whether performance differs across relevant groups.

For structured prediction tasks such as object detection, a confusion matrix also depends on how predictions are matched to reference objects. Rejected predictions, unknown classes, unmatched detections, or missing reference objects require explicit conventions; otherwise, apparently similar matrices may count different events.