The COCO dataset, short for Common Objects in Context, is a large-scale collection of images and annotations used in computer vision. Introduced in 2014, it emphasizes everyday scenes containing multiple objects in their natural surroundings, rather than isolated objects. Its annotations support object detection, instance segmentation, image captioning, and human keypoint estimation, with additional annotation projects supporting other tasks. COCO serves both as training data for learning visual representations and as a benchmark for comparing model performance under standardized evaluation procedures.
Origins and design
COCO was introduced in the paper Microsoft COCO: Common Objects in Context, presented at the 2014 European Conference on Computer Vision. Its design addressed a limitation of image collections centered on identifying a prominent object: recognizing objects in realistic scenes also requires handling clutter, partial occlusion, small objects, and relationships among objects.
The images were collected from Flickr and selected to represent a range of everyday situations. Many contain several object categories and multiple instances of the same category. A street scene, for example, may contain pedestrians, bicycles, cars, and traffic lights, with objects overlapping or appearing at markedly different scales.
A central feature was the provision of instance-level segmentation annotations. Instead of assigning only a class label to an image or drawing a rectangle around an object, annotators outlined the visible regions of individual objects. These annotations made it possible to evaluate both object recognition and precise localization.
COCO complements datasets such as ImageNet. ImageNet is especially associated with large-scale image classification, whereas COCO is especially associated with locating and distinguishing multiple objects within an image.
Images, categories, and dataset splits
COCO comprises several releases and task-specific annotation sets, rather than a single unchanging package. Its headline scale is approximately 330,000 images, with more than 200,000 labeled images and around 1.5 million object instances. These figures describe the broader collection; the number of images available for a particular experiment depends on the release and task.
The standard object detection and instance segmentation benchmark uses 80 object categories, including people, vehicles, animals, household objects, and food. The original dataset description discussed 91 candidate object categories, but the widely used benchmark contains annotations for 80. Category identifiers are not consecutive integers from 1 to 80, so software often maps the published identifiers to a contiguous internal label space.
The principal 2017 image splits are:
| Split | Number of images | Main purpose |
|---|---|---|
| Training | 118,287 | Learning model parameters |
| Validation | 5,000 | Evaluating models during development |
| Test | 40,670 | Benchmark evaluation using withheld annotations |
The 2014 release used different training and validation partitions. Consequently, an experimental result described only as “COCO performance” is incomplete without specifying the release, split, task, and metric.
The distinction between training, validation, and test data matters for interpreting results. Validation annotations are publicly available, whereas official test annotations are withheld for benchmark evaluation. Repeatedly adapting a model to a particular evaluation set can reduce the independence of that evaluation.
Main annotation tasks
Object detection
Object detection annotations identify an object's category and its bounding box. A bounding box is an axis-aligned rectangle enclosing the object, commonly represented in COCO files as an upper-left coordinate, a width, and a height.
A detection model predicts categories, boxes, and confidence scores. Evaluation measures whether the predictions identify the correct categories and localize objects accurately, while accounting for missed objects and false detections.
Instance segmentation
Instance segmentation assigns a separate mask to each annotated object instance. Two people standing next to one another therefore receive distinct masks, even though both belong to the same category.
This differs from semantic segmentation, which assigns category labels to image regions without necessarily distinguishing separate instances. COCO masks generally describe visible object regions, not the complete shape of an object hidden behind an obstruction.
Image captioning
The captioning annotations provide five independently written textual descriptions for images in the standard caption annotation sets. Captions may describe objects, actions, attributes, and relationships.
Image captioning connects visual understanding with natural language processing and supports research in multimodal learning. Because several different descriptions can accurately characterize one image, agreement with reference captions is an imperfect measure of descriptive quality.
Human keypoint estimation
The person keypoint task annotates 17 anatomical keypoints, including the eyes, ears, shoulders, elbows, wrists, hips, knees, and ankles, together with the nose. Annotations distinguish whether a keypoint is visible, labeled but not visible, or not labeled.
This task supports human pose estimation: predicting the spatial arrangement of a person's body from an image. It is distinct from detecting a person as a single object.
Stuff and panoptic annotations
COCO-Stuff adds labels for background and material-like regions, such as sky, grass, and road, alongside object categories.
COCO's panoptic segmentation annotations combine semantic labeling with instance separation. Every labeled pixel receives a category, while countable objects also receive instance identities. The standard COCO panoptic category system contains 80 “thing” categories and 53 “stuff” categories.
These annotation sets build on COCO images but have their own task definitions and evaluation procedures. They should not be treated as interchangeable with the original instance annotations.
Annotation format and software
COCO annotations are commonly distributed as JSON files. For instance annotations, the principal collections are:
- Images, containing image identifiers, file names, and dimensions.
- Categories, defining category identifiers and names.
- Annotations, connecting individual labeled objects to images and categories.
An instance annotation can include an annotation identifier, image identifier, category identifier, bounding box, segmentation, area, and an iscrowd flag. Segmentation data may be represented as polygons or run-length encoding. Crowd annotations provide special handling for groups of objects that are difficult to separate into individual instances.
The official COCO application programming interface, commonly used through pycocotools, provides tools for loading annotations, manipulating masks, visualizing labels, and evaluating predictions.
The phrase COCO format is also used for independently created datasets that adopt this file structure. Such a dataset is not necessarily part of COCO: it may contain different images, categories, annotation rules, and licensing conditions.
Evaluation metrics
Bounding-box and mask average precision
COCO's principal detection metric is average precision (AP), averaged over multiple localization thresholds and categories.
Localization is assessed using intersection over union (IoU). For a predicted region and a ground-truth region ,
For bounding-box evaluation, these regions are rectangles; for instance segmentation, they are masks.
Standard COCO AP averages results across ten IoU thresholds from to , in increments of . The evaluation uses interpolated precision at 101 recall levels. The principal metric therefore rewards accurate localization more strongly than a metric evaluated only at IoU .
Frequently reported measures include:
- AP: average precision across the standard IoU thresholds.
- AP50 and AP75: average precision at IoU thresholds of and .
- AP for small, medium, and large objects: performance separated by annotated object area.
- Average recall: recall summarized under specified limits on the number of detections per image.
The standard headline detection AP uses at most 100 detections per image. Crowd regions and ignored annotations receive special treatment, making faithful use of the evaluation implementation important for comparable results.
Other task metrics
Human keypoint evaluation uses object keypoint similarity rather than box or mask IoU. It measures keypoint localization error relative to person scale and keypoint-specific tolerance parameters.
Captioning evaluation uses text-comparison metrics, including BLEU, METEOR, ROUGE-L, and CIDEr. Panoptic segmentation uses panoptic quality, combining recognition and segmentation quality. Scores from different tasks measure different capabilities and are not directly comparable.
Research uses and limitations
COCO is widely used to develop and compare deep learning systems, including detector families such as YOLO and instance segmentation architectures. COCO-trained models also serve as starting points for transfer learning and fine-tuning on more specialized datasets.
Its standardized annotations and evaluation tools have made it a common experimental reference. Nevertheless, benchmark performance has several limitations:
- Restricted categories: the standard 80-category detection task covers only a small subset of real-world objects.
- Sampling bias: Flickr photographs do not represent all environments, cultures, activities, or imaging conditions equally.
- Annotation limitations: ambiguous boundaries, occlusion, crowd regions, and annotation errors affect the interpretation of results.
- Distribution differences: performance on COCO does not establish reliability on medical images, satellite imagery, industrial scenes, or other substantially different domains. Such changes are examples of distribution shift.
- Benchmark dependence: extensive tuning on familiar splits can encourage overfitting to benchmark-specific characteristics.
- Data overlap: larger training collections may contain COCO images or related material, creating possible data leakage when evaluation independence is not established.
Licensing also requires distinguishing the annotation files from the photographs. COCO's principal annotations are released under the Creative Commons Attribution 4.0 license, while the underlying images retain their individual licensing terms. Availability through the dataset does not place all photographs under one uniform license.