Intersection over Union (IoU) is a measure of similarity between two sets, defined as the size of their intersection divided by the size of their union. In computer vision, it measures how closely a predicted region overlaps a reference region, such as an annotated bounding box or segmentation mask. IoU ranges from 0 for non-overlapping regions to 1 for identical regions, provided the union has nonzero size. It is widely used in evaluating object detection, semantic segmentation, and instance segmentation, and also serves as a component of training objectives and prediction-filtering procedures.
Definition and mathematical properties
For two finite sets and , IoU is
Here, denotes the number of elements. For geometric regions, it instead denotes an appropriate measure, such as area in two dimensions or volume in three dimensions. The union counts every element belonging to either set exactly once.
For example, if a predicted mask contains 80 pixels, a reference mask contains 100 pixels, and their intersection contains 60 pixels, then
Important properties include:
- Symmetry: exchanging the prediction and reference does not change the result.
- Boundedness: .
- Exact agreement: a value of 1 means the sets agree, apart from measure-zero differences when continuous regions are used.
- Scale invariance: applying the same uniform scaling to both geometric regions leaves their IoU unchanged.
The set-based measure is also known as the Jaccard index, originating in early twentieth-century work on comparing species distributions. Its complement,
is the Jaccard distance, which is a metric on finite sets when the empty-set case is defined appropriately.
If both sets are empty, the displayed ratio is and is undefined without an additional convention. Implementations may assign a value of 1, omit the case from an average, or adopt another explicitly documented rule. If exactly one set is empty, IoU is 0.
Bounding-box computation
For axis-aligned rectangular boxes, IoU can be computed directly from their coordinates. Let box have corners and , with coordinates ordered from minimum to maximum, and define box similarly.
The intersection width and height are
The intersection area is , giving
Coordinate conventions matter. Continuous-coordinate boxes normally use side lengths calculated by subtraction. Some older image-processing conventions treat integer endpoints as inclusive and add 1 to the corresponding differences. Mixing these conventions produces inconsistent scores, especially for small boxes.
Rotated rectangles and more general shapes require geometric intersection calculations, often involving polygon clipping from computational geometry. Three-dimensional boxes use intersection and union volumes. For binary segmentation masks, the intersection and union can be obtained through elementwise logical operations on pixels or voxels.
Use in object detection
In object detection, IoU primarily measures localization accuracy. A predicted box is compared with a reference box, and an overlap threshold helps determine whether the localization is sufficient to count as a correct detection.
A threshold of 0.5 is common, but it is not part of the definition of IoU. Different benchmarks and applications use different thresholds. A detection also generally needs the correct class label, and predictions must be matched to reference objects under the evaluation protocol.
Matching rules prevent several predictions for the same object from all receiving credit. In typical evaluation procedures, predictions are processed by confidence score and matched to eligible reference objects. Unmatched predictions count as false positives, while unmatched reference objects count as false negatives. Special rules may apply to ignored regions or groups of objects.
IoU is therefore distinct from average precision (AP):
- IoU describes overlap between a pair of regions.
- AP summarizes detection performance across confidence thresholds, using matches established at a specified overlap threshold.
The COCO evaluation protocol reports a principal AP measure averaged over IoU thresholds from 0.50 to 0.95 in increments of 0.05, as well as separate measures at particular thresholds. Higher overlap thresholds place greater emphasis on precise localization.
IoU also appears in non-maximum suppression (NMS). Standard NMS retains a high-scoring detection and suppresses lower-scoring detections that overlap it beyond a chosen threshold. This reduces duplicate predictions, but can mistakenly suppress distinct objects that are close together or heavily overlapping. The NMS overlap threshold serves a different purpose from the threshold used to evaluate detection correctness.
Use in segmentation
For a single foreground class in image segmentation, IoU can be written using pixel-level classification counts:
where , , and are true-positive, false-positive, and false-negative pixel counts. True-negative pixels are excluded. Consequently, a large correctly predicted background cannot directly inflate the foreground IoU, as it can inflate overall pixel accuracy.
For semantic segmentation with evaluated classes, mean Intersection over Union, usually written mIoU, is commonly defined as
Each class receives equal weight in this average, regardless of how frequently it appears. Other aggregations include frequency-weighted IoU and pooled ratios formed from accumulated counts.
Aggregation choices materially affect results. Computing each class's intersection and union over an entire dataset before division is generally not equivalent to averaging per-image IoUs. Results also depend on whether background is included, how ignored labels are handled, and what happens when a class is absent from both prediction and reference.
For instance segmentation, IoU compares individual object masks rather than merely the combined pixels of a semantic class. A separate matching procedure determines which predicted instance corresponds to which reference instance.
Relationship to other overlap measures
The Dice coefficient, also called the Sørensen–Dice coefficient, is
For the same two sets, with nonzero denominators, Dice and IoU are related by
Thus, they rank individual region pairs in the same order, although their numerical values differ. Averaging their scores across multiple cases can produce different rankings because the conversion is nonlinear.
IoU also differs from precision and recall. Precision measures the fraction of predicted-positive elements that are correct; recall measures the fraction of reference-positive elements recovered. IoU penalizes both extra predicted area and missing reference area in one ratio.
Training objectives and extensions
In deep learning, an IoU-based loss function can align training more directly with an overlap-based evaluation criterion. A basic form is
Ordinary IoU nevertheless has limitations as an optimization objective. For disjoint bounding boxes, intersection area remains zero over a range of coordinates, so the loss may provide no useful local gradient directing the boxes toward one another. Its piecewise geometric definition also introduces nondifferentiable boundaries.
One extension is generalized IoU (GIoU):
where is a specified enclosing region, commonly the smallest axis-aligned box containing both input boxes. The added term penalizes unused space in that enclosure and distinguishes many non-overlapping configurations. Unlike ordinary IoU, GIoU can be negative.
Distance-IoU adds a penalty based on the distance between box centers, while Complete-IoU additionally accounts for aspect-ratio differences. These objectives incorporate geometric information not represented by overlap alone.
For segmentation, soft IoU replaces binary membership with continuous prediction scores. With scores and binary labels , one common form is
A small smoothing constant may be added for numerical stability. Soft IoU is a training surrogate, not necessarily the score obtained after thresholding predictions into a binary mask. Its behavior also depends on whether terms are accumulated per image, per class, or over a training batch.
Limitations and interpretation
IoU summarizes overlap, not every aspect of prediction quality.
- Small objects are sensitive to displacement. The same absolute coordinate error can cause a much larger IoU reduction for a small object than for a large one.
- Boundary quality is only indirectly represented. Two predictions can have similar IoU while differing substantially in contour accuracy, connectivity, or topology.
- Zero overlap hides distance. Ordinary IoU gives the same score of 0 to a near miss and a prediction far from the target.
- Region representation matters. Bounding-box IoU measures overlap of enclosing rectangles, not necessarily overlap of the objects' actual shapes.
- Reference annotations affect scores. Ambiguous boundaries, occlusion conventions, and annotation errors can change measured overlap without a corresponding change in the prediction.
- Thresholds create discontinuities. In detection evaluation, a small overlap change can move a prediction from one side of a matching threshold to the other.
Meaningful comparisons therefore require consistent region representations, coordinate conventions, matching rules, and aggregation procedures. Depending on the task, IoU is reported alongside detection precision and recall, boundary measures, or object-level error counts.