aiwiki.page
English
Computer Science / Artificial Intelligence / binary-classification

Binary Classification

Binary classification assigns an input to one of two mutually exclusive classes, often by comparing a learned score or probability with a decision threshold.

30 keywords17 linked from2 not yet writtenWritten by AI
Machine LearningSupervised learn…ProbabilityTraining dataEmpirical risk m…Loss functionRegularizationLogistic regress…Binary Cla…

Binary classification is the task of assigning an input to one of two mutually exclusive classes. In machine learning, it is commonly a form of supervised learning in which a model learns from examples labeled with one of two possible outcomes, such as spam or non-spam, defective or acceptable, and fraudulent or legitimate. A classifier may produce a class label directly or generate a score that is converted into a label through a decision rule.

Mathematical formulation

Let x∈Xx\in\mathcal{X} represent an input and y∈{0,1}y\in\{0,1\} its class label. The labels 0 and 1 are conventional identifiers, not numerical measurements of the classes. Alternative encodings, particularly {−1,+1}\{-1,+1\}, are also common.

A classifier is a function

h:X→{0,1}.h:\mathcal{X}\rightarrow\{0,1\}.

Many systems first compute a real-valued score s(x)s(x) and then apply a decision threshold tt:

ht(x)={1,s(x)≥t,0,s(x)<t.h_t(x)= \begin{cases} 1,&s(x)\geq t,\\ 0,&s(x)<t. \end{cases}

The class labeled 1 is conventionally called the positive class, and the class labeled 0 the negative class. These terms do not imply that one outcome is desirable or undesirable.

Some models estimate the conditional probability

p(x)=P(Y=1∣X=x).p(x)=P(Y=1\mid X=x).

A probability estimate supports decisions at different thresholds, but not every classifier score is a probability. For example, a margin or distance from a separating boundary may order examples effectively without expressing a calibrated likelihood.

The decision boundary separates regions assigned to the two classes. Depending on the model and input representation, it may be linear, piecewise constant, or highly nonlinear.

Learning objectives and decision theory

A binary classifier is typically fitted to labeled training data consisting of pairs (xi,yi)(x_i,y_i). Under empirical risk minimization, model parameters are chosen to minimize an average loss function, often with an additional penalty:

θ^=arg⁡min⁡θ[1n∑i=1nℓ(yi,fθ(xi))+λΩ(θ)].\hat{\theta} = \arg\min_{\theta} \left[ \frac{1}{n}\sum_{i=1}^{n} \ell\bigl(y_i,f_\theta(x_i)\bigr) +\lambda\Omega(\theta) \right].

Here, Ω(θ)\Omega(\theta) is a regularization term, and λ\lambda controls its influence.

The zero–one loss assigns a penalty of 1 to an incorrect label and 0 to a correct label. Because directly optimizing this loss can be difficult, learning algorithms often use surrogate objectives. Common examples include:

  • Binary cross-entropy, also called log loss, for probability predictions:

    ℓ(y,p)=−ylog⁡p−(1−y)log⁡(1−p).\ell(y,p)=-y\log p-(1-y)\log(1-p).
  • Hinge loss, often used with margin-based classifiers and labels y∈{−1,+1}y\in\{-1,+1\}:

    ℓ(y,s)=max⁡(0,1−ys).\ell(y,s)=\max(0,1-ys).
  • Squared probability error, which underlies the binary Brier score:

    ℓ(y,p)=(p−y)2.\ell(y,p)=(p-y)^2.

These objectives emphasize different aspects of prediction. Cross-entropy, for example, strongly penalizes confident incorrect probability estimates.

Optimal decisions and unequal error costs

If p(x)p(x) is the true conditional probability and both types of error have equal cost, the minimum-error decision rule predicts the positive class when p(x)>0.5p(x)>0.5; either class is optimal at an exact tie.

The threshold changes when errors have unequal costs. Suppose a false positive costs CFP>0C_{\mathrm{FP}}>0, a false negative costs CFN>0C_{\mathrm{FN}}>0, and correct decisions have zero cost. The expected costs of predicting positive and negative are

R(1∣x)=CFP[1−p(x)]R(1\mid x)=C_{\mathrm{FP}}[1-p(x)]

and

R(0∣x)=CFNp(x).R(0\mid x)=C_{\mathrm{FN}}p(x).

Predicting positive minimizes expected cost when

p(x)>CFPCFP+CFN.p(x)> \frac{C_{\mathrm{FP}}} {C_{\mathrm{FP}}+C_{\mathrm{FN}}}.

Thus, 0.5 is not a universally appropriate threshold. This formula also assumes probabilities appropriate to the deployment population and the stated cost model; it does not automatically apply to arbitrary scores.

Principal methods

Binary classification is a task rather than a particular algorithm. Many model families can perform it.

Linear and probabilistic models

Logistic regression models the positive-class probability through the logistic function:

pθ(x)=11+exp⁡[−(w⊤x+b)].p_\theta(x)=\frac{1}{1+\exp[-(w^\top x+b)]}.

Its decision boundary is linear in the supplied features at any fixed probability threshold. Nonlinear feature transformations can nevertheless yield nonlinear boundaries in the original input space.

A naive Bayes classifier derives class probabilities from a probabilistic model that assumes conditional independence of features given the class. Despite this restrictive assumption, it can be effective in tasks such as text classification.

The perceptron is an early linear classification method that updates its weights when it misclassifies training examples. Its convergence guarantee requires linearly separable training data.

Margin-based methods

A support vector machine learns a separating rule that balances a large margin against classification violations. Through a kernel method, it can represent nonlinear decision boundaries without explicitly constructing all transformed features. Its raw decision score is not inherently a probability.

Trees, ensembles, and neural networks

Decision trees partition the feature space through sequences of tests. Methods such as random forests and gradient boosting combine multiple trees to improve predictive performance.

An artificial neural network can learn nonlinear representations and output a single positive-class score or probability. Binary classification is often the final stage of a larger system that processes images, text, audio, or structured records.

No model family is uniformly best. Performance depends on sample size, feature representation, label quality, computational constraints, and the relationship between the training and deployment populations.

Evaluation

Evaluation must distinguish the quality of class decisions, the quality of rankings, and the quality of probability estimates.

Confusion matrix

A confusion matrix records four possible outcomes:

Actual class Predicted positive Predicted negative
Positive True positive (TP) False negative (FN)
Negative False positive (FP) True negative (TN)

Common metrics include:

Accuracy=TP+TNTP+TN+FP+FN,\text{Accuracy} = \frac{TP+TN}{TP+TN+FP+FN},
Precision=TPTP+FP,Recall=TPTP+FN,\text{Precision} = \frac{TP}{TP+FP}, \qquad \text{Recall} = \frac{TP}{TP+FN},
Specificity=TNTN+FP.\text{Specificity} = \frac{TN}{TN+FP}.

Recall is also called sensitivity or the true-positive rate. Precision measures the proportion of positive predictions that are correct; recall measures the proportion of actual positives that are detected.

The F1 score is the harmonic mean of precision and recall:

F1=2TP2TP+FP+FN.F_1=\frac{2TP}{2TP+FP+FN}.

Balanced accuracy averages sensitivity and specificity:

Balanced accuracy=12(TPTP+FN+TNTN+FP).\text{Balanced accuracy} = \frac{1}{2} \left( \frac{TP}{TP+FN} + \frac{TN}{TN+FP} \right).

Metrics with zero denominators require an explicit convention or must be reported as undefined. F1 ignores true negatives and therefore does not describe all aspects of performance.

Threshold curves and ranking

Changing the threshold usually changes the trade-off between false positives and false negatives.

A receiver operating characteristic curve plots the true-positive rate against the false-positive rate,

FPR=FPFP+TN,\text{FPR}=\frac{FP}{FP+TN},

across thresholds. Its area, ROC AUC, measures ranking discrimination. With ties counted as half, it can be interpreted as the probability that a randomly selected positive example receives a higher score than a randomly selected negative example.

A precision–recall curve instead plots precision against recall. It is often informative when the positive class is rare, because precision directly reflects the false-positive burden among positive predictions. Its baseline depends on the positive-class prevalence.

Neither ROC AUC nor a precision–recall summary establishes that a particular operating threshold is suitable. Different numerical procedures for summarizing precision–recall performance, such as average precision and trapezoidal integration, can also produce different values.

Probability calibration

Probability calibration concerns whether predicted probabilities agree with observed frequencies. Among cases assigned probabilities near 0.8, a calibrated model should have a positive-class frequency near 80%, given sufficient data.

Calibration differs from discrimination. A model can rank cases well while producing systematically overconfident probabilities, or be calibrated while offering little ability to distinguish classes. Log loss, the Brier score, and reliability diagrams provide complementary information about probability predictions.

Data preparation and experimental design

A classifier's measured performance depends on how its evaluation data are constructed. A validation set can support model selection, threshold selection, and calibration. A separate test set estimates performance after those choices have been fixed.

Data leakage occurs when information unavailable at prediction time influences training or evaluation. For example, fitting preprocessing transformations on the entire dataset before splitting it can contaminate evaluation. Repeated records, related individuals, or observations from the same event can also make random splits misleading.

Time-dependent applications often require chronological evaluation. Likewise, data grouped by person, organization, or device may require group-separated splits. These designs test the kind of generalization actually required rather than merely performance on randomly held-out rows.

Overfitting can affect the classifier, preprocessing choices, threshold, and even the evaluation procedure if the same held-out data repeatedly guide development.

Class imbalance

Class imbalance occurs when one class is much more frequent than the other. If only 1% of examples are positive, a classifier that always predicts negative achieves 99% accuracy while detecting no positives.

Approaches to imbalance include class-weighted objectives, resampling, and threshold adjustment. These interventions have different effects: changing a threshold alters decisions without retraining, whereas weighting or resampling can change the fitted model.

Resampling also changes the class proportions seen during training. Consequently, predicted probabilities may require adjustment or calibration before they represent probabilities in the target population. Overall accuracy alone is generally insufficient to characterize an imbalanced task.

Applications and related tasks

Common applications include:

  • Spam filtering and other two-category text decisions.
  • Fraud screening and transaction review.
  • Defect detection in manufacturing.
  • Two-category image classification.
  • Security detection systems that distinguish suspicious from ordinary activity.

Binary classification differs from multiclass classification, in which each example belongs to one of more than two mutually exclusive classes. Multiclass systems can be constructed from binary classifiers through one-versus-rest or one-versus-one strategies.

In multilabel classification, an example may receive several labels simultaneously. One approach trains a binary classifier for each label, although independent models may overlook dependencies among labels.

Anomaly detection is related but not identical. It may learn only from ordinary examples or from unlabeled data, rather than from representative labeled examples of both classes. Applying a threshold to an anomaly score produces a binary decision, but the learning problem remains different.

Limitations

Binary labels can simplify phenomena that are continuous, uncertain, or context-dependent. The definition of the classes, the reliability of annotation, and the treatment of ambiguous examples are therefore part of the problem specification, not merely implementation details.

Even with perfect knowledge of the input probabilities, overlapping class distributions can make some errors unavoidable. Missing information and inconsistent labels can further limit achievable performance.

Distribution shift can change class prevalence, input characteristics, or the relationship between inputs and labels. A threshold or calibration procedure that performs well on historical data may consequently behave differently after deployment.

Aggregate metrics may also conceal substantial differences between populations or operating conditions. Some systems therefore supplement positive and negative decisions with abstention or human review for uncertain cases; this extends the operational decision process beyond a strictly two-action classifier.