{
  "id": "detr",
  "title": "DETR",
  "lang": "en",
  "source_lang": "en",
  "summary": "DETR is a Transformer-based object detector that predicts objects as an unordered set, using bipartite matching to train without hand-designed anchors or non-maximum suppression.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Detection Transformer",
    "DEtection TRansformer"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/detr/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/detr/",
      "json": "https://aiwiki.page/zh/detr.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "DETR (short for DEtection TRansformer) is a deep learning architecture for object detection that combines image features with a Transformer encoder–decoder and predicts a fixed-size set of object candidates. Introduced in 2020 in the paper End-to-End Object Detection with Transformers, it reformulated detection as direct set prediction: each detected object receives a category and a bounding box, while unused predictions receive a “no object” label. Its distinguishing features are learned object queries and a one-to-one matching procedure that trains different predictions to represent different objects, eliminating the need for hand-designed anchor boxes and conventional duplicate-removal post-processing."
    },
    {
      "heading": "Background and motivation",
      "text": "Object detection requires both identifying objects and locating them within an image. Unlike image classification, it must accommodate an unknown number of outputs. It also has no natural output order: detecting a person before a bicycle should be equivalent to detecting the bicycle before the person.\n\nBefore DETR, many successful detectors generated numerous candidate boxes associated with image locations, anchor boxes, or proposed regions. Multiple candidates could describe the same object, so detectors commonly used non-maximum suppression (NMS) to discard overlapping, lower-scoring predictions. Training also required rules for assigning candidate boxes to annotated objects.\n\nDETR replaced much of this task-specific machinery with two connected ideas:\n\n- Set prediction: the output represents a collection of objects rather than a sequence with a prescribed ordering.\n- One-to-one assignment: each annotated object is matched to exactly one prediction during training.\n\nThese choices allow a single network to learn both object localization and the separation of distinct detections. “End-to-end” does not mean that DETR has no preprocessing, decoding, or confidence filtering; it means that its core detection pipeline can be jointly trained without a separate proposal-and-suppression procedure."
    },
    {
      "heading": "Architecture",
      "text": "The original DETR has three main components: an image backbone, a Transformer encoder–decoder, and prediction heads.\n\nImage backbone and positional information\n\nA convolutional neural network, such as a ResNet, extracts a spatial feature map from the input image. A projection maps the feature channels to the Transformer’s hidden dimension, and the spatial grid is flattened into a sequence.\n\nFlattening alone does not tell an attention layer where a feature lies in the image. DETR therefore introduces positional encodings to retain spatial information.\n\nThe original architecture is consequently a hybrid convolutional–Transformer model, not a detector that applies a Transformer directly to raw pixels. Its set-prediction formulation is distinct from the patch-based image representation associated with the Vision Transformer.\n\nTransformer encoder\n\nThe encoder applies self-attention and feed-forward layers to image features. Each spatial feature can incorporate information from other locations, providing a representation with image-wide context.\n\nThis global interaction is useful when an object’s interpretation depends on surrounding objects or widely separated image regions. It also introduces a computational cost: for \\(S\\) spatial tokens, a standard dense attention layer forms interactions proportional to \\(S^2\\).\n\nObject queries and Transformer decoder\n\nThe decoder receives a fixed number \\(N\\) of learned object queries. These are trainable embeddings that distinguish output slots; they are not text prompts, predefined object categories, or manually placed bounding boxes.\n\nThrough cross-attention, the queries retrieve information from the encoded image. Decoder self-attention also allows the output slots to interact, helping the model coordinate which objects they represent. In the original DETR, all slots are processed in parallel rather than generated autoregressively.\n\nAn object query should not be interpreted as permanently assigned to a specific category or image location. Its behavior is learned, and its prediction depends on the input image. Queries may nevertheless develop statistical preferences for certain spatial regions or object scales.\n\nPrediction heads\n\nEach final decoder output is passed to:\n\n- a classification head predicting probabilities over object categories and an additional no-object category;\n- a small multilayer perceptron predicting a bounding box.\n\nBoxes are typically expressed as normalized center coordinates, width, and height:\n\n\\[b=(c_x,c_y,w,h).\\]\n\nThe original model commonly used 100 query slots. This is an architectural setting, not an estimate that every image contains 100 objects. Unused slots should predict no object, and the number of slots imposes an upper bound on the number of objects that one forward pass can return."
    },
    {
      "heading": "Bipartite matching and training objective",
      "text": "DETR is trained using supervised learning from images annotated with object categories and bounding boxes. Its central training problem is deciding which prediction should learn each annotated object.\n\nOptimal assignment\n\nSuppose an image contains \\(M\\) annotated objects and the detector produces \\(N\\) predictions, with \\(M\\leq N\\). Ground-truth object \\(i\\) has class \\(c_i\\) and box \\(b_i\\); prediction \\(j\\) provides class probabilities \\(\\hat p_j\\) and box \\(\\hat b_j\\).\n\nTraining first finds an injective assignment \\(\\sigma\\) from ground-truth objects to prediction slots:\n\n\\[\\hat{\\sigma} = \\underset{\\sigma}{\\operatorname{arg\\,min}} \\sum_{i=1}^{M} C(i,\\sigma(i)).\\]\n\nA representative matching cost is\n\n\\[C(i,j) = -\\hat p_j(c_i) + \\lambda_1\\lVert b_i-\\hat b_j\\rVert_1 + \\lambda_{\\mathrm{G}} \\left[1-\\operatorname{GIoU}(b_i,\\hat b_j)\\right],\\]\n\nwhere the coefficients balance category and localization terms. The assignment is computed using the Hungarian algorithm.\n\nThe one-to-one constraint is essential. Two predictions cannot both be assigned to the same ground-truth object, and a prediction cannot be assigned to two different objects. Reordering the annotations does not change the underlying set-matching problem.\n\nClassification and box losses\n\nOnce the assignment is fixed for a training step, matched predictions receive classification and localization losses. Unmatched predictions are trained to predict no object. A simplified objective is\n\n\\[\\mathcal{L} = \\sum_{i=1}^{M} \\left[ -\\log \\hat p_{\\hat{\\sigma}(i)}(c_i) + \\lambda_1\\lVert b_i-\\hat b_{\\hat{\\sigma}(i)}\\rVert_1 + \\lambda_{\\mathrm{G}} \\left(1-\\operatorname{GIoU} (b_i,\\hat b_{\\hat{\\sigma}(i)})\\right) \\right] + \\lambda_{\\varnothing} \\sum_{j\\notin\\operatorname{Im}(\\hat{\\sigma})} -\\log \\hat p_j(\\varnothing).\\]\n\nHere, \\(\\varnothing\\) denotes the no-object category. Its contribution is down-weighted because most slots may be unmatched.\n\nThe localization loss combines an \\(L_1\\) coordinate loss with generalized intersection over union (GIoU). GIoU provides a geometric penalty even for boxes that do not overlap.\n\nThe discrete assignment is not differentiated through. Instead, it selects the prediction–target pairs whose losses are differentiated using backpropagation. The original model also applies auxiliary prediction losses to intermediate decoder layers to support optimization.\n\nWhy duplicate suppression is usually unnecessary\n\nIf several slots predict the same object, only one can be matched to it. The others must match different objects or learn the no-object label. Duplicate avoidance is therefore encouraged during training rather than imposed by a separate NMS stage.\n\nThis is a learned behavior, not a mathematical guarantee that duplicates can never occur. At inference, implementations still decode boxes, remove no-object predictions, and may apply score thresholds or output-count limits."
    },
    {
      "heading": "Original results and practical characteristics",
      "text": "The original DETR was presented at the 2020 European Conference on Computer Vision. On the COCO detection benchmark, it achieved performance competitive with established detectors while using a simpler detection-specific pipeline.\n\nIts results were not uniformly better across all conditions. The original model was particularly effective on large objects but weaker on small objects than strong contemporary baselines. It also required a long training schedule: the widely used original configuration trained for 500 epochs.\n\nSeveral factors help explain these characteristics:\n\n- Low-resolution features: the final backbone feature map can lose fine details needed for small-object localization.\n- Global dense attention: increasing feature-map resolution raises encoder attention costs sharply.\n- Difficult early optimization: initially uninformative queries must learn both where to attend and which objects to represent.\n- A single feature scale: the original architecture does not directly exploit a hierarchy of feature resolutions in the way many detection systems do.\n\nIts conceptual simplicity therefore does not imply lower training cost or faster inference in every setting. Comparisons with detector families such as YOLO depend on the specific models, input resolutions, hardware, and accuracy requirements."
    },
    {
      "heading": "Subsequent developments",
      "text": "DETR established a broad design pattern rather than a single unchanged architecture. Later detectors retained set prediction and matching while altering attention, queries, feature representations, or training.\n\nDeformable attention and multi-scale features\n\nDeformable DETR replaces dense attention over all image locations with attention to a small collection of learned sampling points around reference locations. It also combines features at multiple resolutions.\n\nThis reduces the cost of attending to high-resolution representations and improves convergence and small-object detection. Its sampling mechanism differs from the original DETR’s dense global attention, while preserving the overall set-prediction approach.\n\nSpatially structured queries\n\nLater designs gave queries more explicit geometric structure. Instead of relying only on unconstrained embeddings, queries can incorporate reference points or dynamically refined boxes.\n\nDAB-DETR, for example, represents queries using dynamic anchor boxes. The word “anchor” here does not mean that it simply restores the original dense, hand-designed anchor pipeline: query geometry is part of the Transformer’s learned detection process.\n\nDenoising-based training\n\nDN-DETR introduced an auxiliary training task that reconstructs objects from deliberately corrupted labels and boxes. This gives the decoder useful learning signals without requiring every training example to depend solely on an initially unstable assignment.\n\nDINO further developed denoising-based training, query initialization, and box refinement. Such systems remain DETR-family models but can differ substantially from the original architecture and training schedule. Results reported for a later family member should not be attributed to the 2020 model."
    },
    {
      "heading": "Applications and extensions",
      "text": "DETR is used as a general detection architecture, a research baseline, and a component in larger computer vision systems.\n\nIts query-based representation can also support tasks beyond category-and-box prediction:\n\n- Instance segmentation: object queries can be associated with masks representing individual objects.\n- Panoptic segmentation: a mask-producing model can combine countable object instances with background categories.\n- Tracking and video analysis: query representations can be extended to associate objects across frames.\n- Open-vocabulary object detection: query-based detectors can incorporate language-derived category representations instead of being restricted to a fixed classification head.\n\nThese applications require additional architectural components, training objectives, or data. They are extensions of the framework, not capabilities automatically provided by the original detector."
    },
    {
      "heading": "Limitations and distinctions",
      "text": "Set prediction does not remove the need for task-specific choices. Query count, backbone resolution, matching costs, loss weights, confidence thresholds, and annotation conventions remain important.\n\nThe fixed query budget limits output capacity. Extremely crowded images may require more queries, higher-resolution features, or specialized methods. Increasing the query count also changes computational and training demands.\n\nOne-to-one matching can produce sparse supervision. Each object supplies a positive detection target to only one slot in the main matching loss. Auxiliary tasks, denoising, and alternative assignment strategies are among the ways later systems address this issue.\n\nGlobal context does not eliminate recognition errors. Occlusion, tiny objects, ambiguous categories, missing annotations, and distribution shift remain sources of failure.\n\n“DETR” and “Transformer-based detector” are not synonyms. Some detectors use Transformers while retaining dense predictions or NMS. Conversely, DETR-derived systems may replace substantial parts of the original Transformer design. The defining contribution is the combination of learned object queries, direct set prediction, and one-to-one assignment, rather than attention alone."
    }
  ],
  "links": [
    "deep-learning",
    "object-detection",
    "transformer-architecture",
    "image-classification",
    "non-maximum-suppression",
    "convolutional-neural-network",
    "residual-neural-network",
    "positional-encoding",
    "vision-transformer",
    "self-attention",
    "cross-attention",
    "multilayer-perceptron",
    "supervised-learning",
    "loss-function",
    "backpropagation",
    "coco-dataset",
    "yolo",
    "computer-vision",
    "instance-segmentation",
    "open-vocabulary-object-detection",
    "distribution-shift"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}