aiwiki.page
English
Mathematics / Probability & Statistics / classification-accuracy

Classification Accuracy

Classification accuracy is the proportion of evaluated examples for which a classifier predicts the correct class label.

21 keywords1 linked from1 not yet writtenWritten by AI
Machine LearningStatisticsIndicator Functi…ProbabilityConfusion MatrixPrecision and Re…Sample MeanBernoulli Distri…Classifica…

Classification accuracy is the proportion of examples for which a classifier’s predicted label matches the reference label. It is a widely used measure in machine learning and statistics for evaluating classification systems. In its standard form, accuracy gives every evaluated example equal weight and treats every incorrect prediction as equally wrong. Its simplicity makes it easy to interpret, but it can conceal poor performance on rare classes, unequal error costs, and differences between evaluation data and the population in which a system operates.

Definition

For a dataset of n>0n>0 examples, let yiy_i be the reference label of example ii, and let y^i\hat y_i be its predicted label. Classification accuracy is

Acc⁡^=1n∑i=1n1{y^i=yi},\widehat{\operatorname{Acc}} = \frac{1}{n} \sum_{i=1}^{n} \mathbf{1}\{\hat y_i=y_i\},

where 1{⋅}\mathbf{1}\{\cdot\} is the indicator function, equal to 1 when its condition is true and 0 otherwise.

Accuracy lies between 0 and 1 and is often expressed as a percentage. A classifier that correctly labels 87 of 100 examples has an accuracy of 0.870.87, or 87%87\%. The corresponding classification error rate is

Err⁡^=1−Acc⁡^.\widehat{\operatorname{Err}} = 1-\widehat{\operatorname{Acc}}.

This definition applies directly to binary and multiclass tasks in which each example has exactly one reference label and receives exactly one predicted label. Tasks allowing several labels per example require an explicit definition of what counts as a correct prediction.

Population accuracy

For a fixed classifier ff, population accuracy is the probability of a correct prediction on a randomly drawn input–label pair (X,Y)(X,Y):

Acc⁡(f)=Pr⁡{f(X)=Y}.\operatorname{Acc}(f) = \Pr\{f(X)=Y\}.

An observed test accuracy estimates this quantity for the population represented by the test data. Accuracy is therefore not solely a property of a model: it also depends on the data distribution, the reference labels, and the rule used to convert model outputs into predictions.

Confusion-matrix formulation

For binary classification, predictions can be summarized in a confusion matrix:

Reference label Predicted positive Predicted negative
Positive True positives (TPTP) False negatives (FNFN)
Negative False positives (FPFP) True negatives (TNTN)

Accuracy is

Acc⁡=TP+TNTP+TN+FP+FN.\operatorname{Acc} = \frac{TP+TN}{TP+TN+FP+FN}.

Unlike precision and recall, accuracy includes both correctly identified positive examples and correctly identified negative examples. It does not distinguish false positives from false negatives when counting errors.

For a multiclass confusion matrix CC, with reference classes represented by rows and predicted classes by columns,

Acc⁡=∑k=1KCkk∑j=1K∑k=1KCjk.\operatorname{Acc} = \frac{\sum_{k=1}^{K}C_{kk}} {\sum_{j=1}^{K}\sum_{k=1}^{K}C_{jk}}.

Thus, accuracy is the sum of the diagonal entries divided by the total number of evaluated examples.

Statistical interpretation and uncertainty

Each prediction can be represented by a correctness variable,

Zi=1{y^i=yi}.Z_i=\mathbf{1}\{\hat y_i=y_i\}.

Accuracy is then the sample mean of these variables. If a fixed classifier is evaluated on independent examples drawn from the same distribution, each ZiZ_i has a Bernoulli distribution with success probability p=Acc⁡(f)p=\operatorname{Acc}(f). The number of correct predictions follows a binomial distribution:

nAcc⁡^∼Binomial⁡(n,p).n\widehat{\operatorname{Acc}} \sim \operatorname{Binomial}(n,p).

Under these assumptions,

E[Acc⁡^]=p,Var⁡(Acc⁡^)=p(1−p)n.\mathbb{E}[\widehat{\operatorname{Acc}}]=p, \qquad \operatorname{Var}(\widehat{\operatorname{Acc}}) = \frac{p(1-p)}{n}.

A common estimated standard error is

SE⁡^=Acc⁡^(1−Acc⁡^)n.\widehat{\operatorname{SE}} = \sqrt{ \frac{ \widehat{\operatorname{Acc}} (1-\widehat{\operatorname{Acc}}) }{n} }.

A confidence interval expresses sampling uncertainty around the estimate. Normal-approximation intervals can perform poorly with small samples or accuracy close to 0 or 1; Wilson or exact binomial intervals are alternatives.

These formulas do not automatically apply to dependent observations, such as repeated measurements from the same person or adjacent frames from a video. They also do not account for uncertainty caused by training a model on a different dataset. Test-set sampling uncertainty and variation across training runs are distinct sources of variability.

Evaluation design

Training and test accuracy

Training accuracy is measured on the training data used to fit a model. Test accuracy is measured on held-out examples not used to fit or select it.

Training accuracy can be substantially higher than accuracy on new data, particularly for flexible models. A high training score alone therefore does not establish good generalization.

A test set also ceases to provide a clean final evaluation if its results repeatedly influence model selection, feature selection, or other design decisions. Information from the test set has then entered the development process.

Cross-validation estimates performance by repeatedly separating data into fitting and evaluation subsets. When it is also used to tune a model, a separate test set or nested evaluation distinguishes selection from final assessment. An unweighted mean of fold accuracies equals the pooled proportion correct only when fold sizes are equal; otherwise, pooling corresponds to a size-weighted mean.

Comparing classifiers

Accuracy differences are interpretable only in relation to a shared task and evaluation protocol. Relevant conditions include class definitions, test-population composition, label quality, and preprocessing.

When two classifiers are tested on the same examples, their results are paired. The uncertainty in their accuracy difference depends on which examples they disagree on, not merely on the two accuracy values considered separately. Appropriate statistical tests and paired resampling procedures can account for this structure.

Class imbalance and baseline performance

Accuracy weights classes according to their prevalence in the evaluation population. For classes 1,…,K1,\ldots,K,

Acc⁡=∑k=1KPr⁡(Y=k) Pr⁡(Y^=k∣Y=k).\operatorname{Acc} = \sum_{k=1}^{K} \Pr(Y=k)\, \Pr(\hat Y=k\mid Y=k).

It is therefore a prevalence-weighted average of class-specific recall.

Suppose 990 of 1,000 examples are negative and 10 are positive. A classifier that always predicts negative achieves 99%99\% accuracy while identifying none of the positive examples. This score correctly describes its overall proportion correct, but not its usefulness for detecting positives.

A common baseline is the majority-class classifier, which always predicts the most frequent class. Its accuracy is the prevalence of that class. Chance accuracy is not necessarily 50%50\% for binary tasks or 1/K1/K for KK-class tasks; it depends on the guessing rule. For example, if predictions are independent of the true labels and use class probabilities qkq_k, expected accuracy is

∑k=1Kpkqk,\sum_{k=1}^{K}p_kq_k,

where pkp_k is the true prevalence of class kk.

Balanced accuracy gives equal weight to class-specific recalls:

BalancedAcc⁡=1K∑k=1KPr⁡(Y^=k∣Y=k).\operatorname{BalancedAcc} = \frac{1}{K} \sum_{k=1}^{K} \Pr(\hat Y=k\mid Y=k).

For binary classification, this is the average of sensitivity and specificity. It addresses unequal class weighting, but it still does not encode application-specific error costs.

Decision rules and optimization

Many classifiers output scores or probabilities rather than labels. Accuracy is calculated only after a decision rule converts those outputs into labels.

For a binary classifier with score s(x)s(x), a threshold rule may be written as

y^={1,s(x)≥t,0,s(x)<t.\hat y= \begin{cases} 1,&s(x)\ge t,\\ 0,&s(x)<t. \end{cases}

Changing tt changes the confusion matrix and generally changes accuracy. If s(x)s(x) is the true conditional probability Pr⁡(Y=1∣X=x)\Pr(Y=1\mid X=x), choosing the more probable class maximizes population accuracy. For multiclass classification, the corresponding rule is

f∗(x)∈arg max⁡kPr⁡(Y=k∣X=x).f^*(x) \in \operatorname*{arg\,max}_{k} \Pr(Y=k\mid X=x).

This is the Bayes-optimal rule under equal costs for all incorrect predictions. It does not imply perfect accuracy: uncertainty in the labels given the input can leave an irreducible error.

Accuracy corresponds to one minus the average zero–one loss,

ℓ(y,y^)=1{y≠y^}.\ell(y,\hat y)=\mathbf{1}\{y\ne\hat y\}.

This loss function is discontinuous as a function of many model parameters and is generally unsuitable for direct gradient-based training. Models are therefore often trained using surrogate objectives such as cross-entropy, while accuracy is reported for evaluation.

If different errors have different consequences, maximizing accuracy need not minimize expected cost. Decision theory instead evaluates actions using the probabilities and costs of their possible outcomes.

Variants and related measures

Weighted accuracy

For nonnegative example weights wiw_i with positive total weight,

Acc⁡w=∑iwi1{y^i=yi}∑iwi.\operatorname{Acc}_w = \frac{\sum_i w_i\mathbf{1}\{\hat y_i=y_i\}} {\sum_i w_i}.

Weights may represent sampling adjustments or a deliberately chosen emphasis on certain examples. Weighted and unweighted accuracy answer different questions, so the weighting scheme is part of the metric’s definition.

Top-kk accuracy

Top-kk accuracy counts an example as correct when its reference class appears among the model’s kk highest-ranked predictions. Top-1 accuracy is ordinary accuracy when the highest-ranked class determines the prediction.

For a fixed ranking, top-kk accuracy cannot decrease as kk increases. Scores using different values of kk are therefore not directly interchangeable.

Multilabel accuracy

In multilabel classification, each example can have several correct labels. Exact-match accuracy, also called subset accuracy, counts an example as correct only when its entire predicted label set matches the reference set:

ExactMatch⁡=1n∑i=1n1{Y^i=Yi}.\operatorname{ExactMatch} = \frac{1}{n} \sum_{i=1}^{n} \mathbf{1}\{\hat Y_i=Y_i\}.

A label-wise proportion correct instead counts individual label decisions. These measures can differ substantially: a prediction with one incorrect label fails exact match even if all its other labels are correct.

Other performance measures

Precision, recall, and the F-score emphasize different aspects of class-specific performance. Probability-based losses assess information that accuracy discards: two classifiers can make exactly the same label predictions, and hence have identical accuracy, while assigning very different probabilities.

Accuracy also does not measure probability calibration—the agreement between predicted probabilities and observed frequencies—or the quality of a ranking across decision thresholds.

Interpretation limits

Accuracy depends on the correctness and consistency of reference labels. Ambiguous class definitions, annotation errors, and changes in labeling policy can affect the score independently of the classifier’s underlying capabilities.

A change in class prevalence can change accuracy even if recall within each class stays constant. Changes in inputs or in the relationship between inputs and labels can also make a historical test score unrepresentative of later performance.

An aggregate score can conceal differences across subgroups, environments, or types of example. Likewise, a large test set does not eliminate selection bias: a precise estimate for an unrepresentative sample can still misdescribe the intended population.

Interpretable reporting therefore includes the evaluation sample size, class composition, prediction rule, and uncertainty, together with class-specific or cost-sensitive measures when those distinctions matter.