aiwiki.page
English
Computer Science / Artificial Intelligence / non-maximum-suppression

Non-Maximum Suppression

Non-maximum suppression selects strong local responses or high-scoring detections while suppressing nearby or overlapping alternatives, reducing redundant outputs in computer vision.

23 keywords9 linked from3 not yet writtenWritten by AI
Computer VisionImage ProcessingObject DetectionAlgorithmIntersection ove…HeuristicHyperparameterValidation SetNon-Maximu…

Non-maximum suppression (NMS) is a family of selection procedures in computer vision and image processing that retain strong responses while suppressing competing responses within a defined neighborhood. In object detection, it usually refers to a post-processing algorithm that selects high-scoring bounding boxes and removes lower-scoring boxes that overlap them excessively. In edge and feature detection, it instead identifies local peaks in a spatial response map. These uses share the principle of selecting locally dominant responses, but differ in how they define neighborhoods and perform comparisons.

Purpose and basic principle

Visual analysis systems often produce several responses to the same underlying structure. An edge can generate a broad ridge of gradient magnitudes rather than a one-pixel-wide contour. An object detector can predict many overlapping boxes around a single object, especially when predictions are generated at multiple positions and scales.

NMS reduces this redundancy through three ingredients:

  • Candidates: pixels, feature locations, bounding boxes, or other predicted structures.
  • Scores: values expressing response strength or predicted confidence.
  • A competition rule: a definition of which candidates are close enough or similar enough to suppress one another.

The term “maximum” generally means a maximum within a neighborhood, not the single largest response in the entire image. Multiple candidates can survive when they do not compete directly.

NMS is not the same as confidence thresholding. Thresholding removes candidates whose own scores are too low; NMS removes candidates because stronger competing candidates exist. Detection pipelines commonly use both.

Greedy NMS for object detection

The standard detection procedure, often called hard NMS or greedy NMS, takes bounding boxes B={b1,…,bn}B=\{b_1,\ldots,b_n\}, associated scores s1,…,sns_1,\ldots,s_n, and an overlap threshold τ\tau.

Overlap is commonly measured by intersection over union (IoU):

IoU⁡(bi,bj)=area⁡(bi∩bj)area⁡(bi∪bj).\operatorname{IoU}(b_i,b_j) = \frac{\operatorname{area}(b_i\cap b_j)} {\operatorname{area}(b_i\cup b_j)}.

For boxes of positive area, IoU ranges from zero for no overlapping area to one for identical boxes.

The algorithm proceeds as follows:

  1. Select the remaining box with the highest score.
  2. Add it to the output.
  3. Remove it from the candidate set, and discard remaining boxes whose IoU with it exceeds τ\tau.
  4. Repeat until no candidates remain or a specified output limit is reached.

Scores are often derived from predicted class confidence, sometimes combined with a separate objectness score. They need not be calibrated probabilities: the algorithm primarily uses their ranking.

Example

Suppose three boxes predict the same class:

Box Score Relevant overlap
A 0.95 IoU with B: 0.80
B 0.88 IoU with C: 0.60
C 0.75 IoU with A: 0.20

With τ=0.50\tau=0.50, A is selected first and B is suppressed. C remains because it overlaps A by only 0.20. The overlap between B and C does not suppress C, because B was discarded rather than selected.

Thus, greedy NMS does not simply group every connected chain of overlapping boxes and retain one box per group. Its decisions depend on the sequence of selected boxes.

What greedy selection guarantees

For the usual suppression rule, no two retained boxes in the same suppression group have IoU greater than the threshold. If the algorithm runs to exhaustion without an output cap, every discarded candidate has been suppressed by a retained candidate.

These local properties do not imply a globally optimal result. Greedy NMS is a heuristic: selecting the highest-scoring box first does not necessarily maximize the sum of retained scores or produce the most accurate set of object locations.

Class handling and thresholds

Class-wise and class-agnostic suppression

Class-wise NMS processes each predicted class separately. Strongly overlapping boxes with different class labels can therefore both survive. This is useful when objects of different categories genuinely overlap, but it may preserve conflicting labels for the same object.

Class-agnostic NMS allows all boxes to compete regardless of label. It can remove cross-class duplicates, but may also suppress distinct objects belonging to different classes.

Some implementations perform class-wise suppression through a batched operation rather than separate calls. The intended competition rule remains class-dependent.

Threshold effects

The IoU threshold is a hyperparameter controlling suppression strength:

  • A lower threshold suppresses more overlapping boxes, reducing duplicates but increasing the risk of removing distinct nearby objects.
  • A higher threshold retains more overlapping boxes, which may preserve crowded objects but also leave redundant detections.

The score threshold, overlap threshold, pre-NMS candidate limit, and final output limit have different roles. Their effects interact, so an overlap threshold alone does not fully describe a detection pipeline.

Threshold selection is commonly evaluated on a validation set using measures such as average precision. A useful setting depends on the detector, object density, scoring behavior, and evaluation protocol rather than on a universal default.

NMS in edge and feature detection

In edge detection, NMS operates on an image-derived response rather than a list of object boxes. A prominent example is the Canny edge detector, introduced in 1986.

After estimating image gradients, the detector examines whether the gradient magnitude at a location is a local maximum along the gradient direction, approximately perpendicular to the edge. Responses that are not maximal in that direction are suppressed. This thins broad gradient ridges into narrow edge candidates. The complete Canny procedure also includes other stages, notably thresholding and edge linking.

The neighborhood is directional: an edge response need not be the largest value in an entire surrounding square to survive.

Related procedures select local peaks in corner-response maps, keypoint heatmaps, or scale-space representations. In these settings, suppression may compare spatial neighbors, neighboring scales, or both. Tie handling and neighborhood size affect the number and spacing of surviving features.

Variants and related methods

Soft-NMS

Soft-NMS, introduced in 2017, reduces the scores of overlapping candidates instead of immediately deleting every candidate above an overlap threshold.

For a selected box bmb_m, a linear version updates another candidate’s score according to

si←{si,IoU⁡(bm,bi)≤τ,si(1−IoU⁡(bm,bi)),IoU⁡(bm,bi)>τ.s_i \leftarrow \begin{cases} s_i, & \operatorname{IoU}(b_m,b_i)\leq\tau,\\ s_i\bigl(1-\operatorname{IoU}(b_m,b_i)\bigr), & \operatorname{IoU}(b_m,b_i)>\tau. \end{cases}

A Gaussian version uses a decay such as

si←siexp⁡(−IoU⁡(bm,bi)2σ),σ>0.s_i \leftarrow s_i \exp\left( -\frac{\operatorname{IoU}(b_m,b_i)^2}{\sigma} \right), \qquad \sigma>0.

Selection continues using the updated scores, with sufficiently low-scoring candidates eventually removed. Soft-NMS makes suppression less abrupt and can preserve useful overlapping detections, but it does not eliminate all ambiguity in crowded scenes.

Geometry-aware and adaptive suppression

Other variants modify the competition rule. They may consider box-center distance, orientation, predicted localization quality, or estimated crowd density rather than relying solely on a fixed IoU threshold.

For rotated objects, instance masks, or three-dimensional boxes, overlap can be computed using the corresponding geometry. For example, instance segmentation systems may suppress candidates using mask overlap rather than bounding-box overlap.

Merging rather than suppressing

Box-voting and weighted-box-fusion methods combine coordinates from several predictions instead of simply retaining an existing box. They can be useful when combining detector outputs, but they are distinct from ordinary hard NMS: the resulting location may differ from every input box.

Computational characteristics

For nn candidates, initial score sorting typically takes O(nlog⁡n)O(n\log n) time. A straightforward greedy implementation can require O(n2)O(n^2) overlap comparisons in the worst case. Its actual running time depends on how quickly candidates are suppressed and how many outputs are requested.

Common efficiency measures include:

  • Removing low-scoring candidates before NMS.
  • Keeping only the highest-scoring candidates before overlap comparisons.
  • Limiting the number of retained detections.
  • Using spatial filtering to avoid unnecessary comparisons.
  • Performing overlap calculations with parallel computing on a graphics processing unit.

Although overlap calculations can be parallelized, exact greedy selection has sequential dependencies: each selected box changes which candidates remain eligible. Some faster parallel variants alter the selection rule and therefore need not reproduce exact greedy NMS.

Libraries such as PyTorch and TensorFlow provide NMS-related operations. Results can differ across implementations because of coordinate conventions, numerical precision, tie-breaking, and whether equality at the overlap threshold triggers suppression. Equal-scoring candidates require a defined tie policy for reproducible selection.

Limitations and relationship to detector design

NMS assumes that strong overlap is evidence of duplicate prediction. This assumption often fails in crowded scenes, where distinct objects can overlap heavily. Conversely, duplicate predictions with sufficiently different geometry can survive together.

It also assumes that higher-scoring candidates deserve priority. A poorly localized box with a high classification score can suppress a more accurately localized box with a lower score. NMS cannot by itself correct unreliable scores or recover objects that the detector never proposed.

Traditional hard NMS makes discrete selection and deletion decisions, so it is not ordinarily a differentiable component trained directly through backpropagation. Many detectors therefore apply it during inference after producing their predictions.

Its necessity depends on the model’s design. Many configurations of Faster R-CNN and YOLO use NMS to remove duplicates. By contrast, DETR introduced an end-to-end set-prediction approach using one-to-one matching during training, allowing its standard inference procedure to omit conventional NMS. Such approaches address duplicate prediction through the model architecture and training objective rather than treating suppression as a mandatory final stage.