{
  "id": "instance-segmentation",
  "title": "Instance Segmentation",
  "lang": "en",
  "source_lang": "en",
  "summary": "Instance segmentation identifies individual objects in images and assigns a separate pixel-level mask to each detected object.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Object instance segmentation",
    "instance-level segmentation"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/instance-segmentation/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/instance-segmentation/",
      "json": "https://aiwiki.page/zh/instance-segmentation.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Instance segmentation is a task in computer vision that identifies individual objects in an image and delineates the pixels belonging to each one. Its output typically consists of a collection of object masks, category labels, and confidence scores. Unlike object detection, which usually locates objects with bounding boxes, instance segmentation describes their visible shapes. Unlike semantic segmentation, it distinguishes separate objects belonging to the same category: three adjacent people should receive three instance identities rather than a single undifferentiated “person” region.\n\nThe task combines recognition, localization, and fine-grained spatial assignment. It supports object counting, robotic manipulation, biological image analysis, scene understanding, and image editing, where knowing an object's category or approximate position is insufficient. Its central difficulties include separating touching objects, recognizing partly occluded instances, preserving thin structures, and deciding which image regions constitute distinct objects.\n\nModern systems are predominantly based on deep learning. Their designs range from detecting candidate objects and predicting a mask for each candidate to directly predicting sets of masks from image features. Understanding their behavior requires distinguishing the task definition from its annotation conventions, model architecture, training procedure, and evaluation protocol."
    },
    {
      "heading": "Task definition and output representations",
      "text": "For an image defined on a pixel domain \\(\\Omega\\), a conventional instance segmentation prediction can be represented as an unordered set\n\n\\[\\mathcal{S}=\\{(M_i,c_i,s_i)\\}_{i=1}^{N},\\]\n\nwhere \\(N\\) is the predicted number of objects, \\(M_i\\subseteq\\Omega\\) is the mask of instance \\(i\\), \\(c_i\\) is its category, and \\(s_i\\) is a confidence score. A model commonly produces a soft mask \\(P_i(x,y)\\in[0,1]\\), which is converted into a binary mask using a threshold. Instance identifiers are arbitrary: exchanging the identifiers of two objects does not change the segmentation, provided their masks and associated labels remain consistent.\n\nThis output differs from a semantic label map, which assigns a category to each pixel without necessarily recording object identity. Two touching cars may form one connected semantic region but remain two instances. Conversely, one partly occluded object may appear in several disconnected visible regions that all belong to the same instance. Connected-component analysis alone therefore cannot generally convert semantic segmentation into correct instance segmentation.\n\nMost general-purpose benchmarks emphasize countable object categories, often called “things,” such as people, vehicles, or animals. Regions such as sky, road, and grass are commonly called “stuff,” because their division into individual objects is not usually defined. Panoptic segmentation, introduced as a unified task in 2018, combines semantic labels for all pixels with instance identities for thing categories. Conventional instance segmentation, by contrast, need not label every pixel and may leave background regions unassigned.\n\nThe meaning of a mask depends on the annotation protocol. Visible, or modal, segmentation labels only the observable part of an object. Amodal segmentation estimates the object's full projected extent, including regions hidden behind other objects. In visible segmentation, an occluded person's arm and torso may be separated in the image but share an identity; amodal segmentation additionally requires an estimate of the hidden body outline. These are different prediction targets rather than alternative scoring rules for the same target.\n\nMasks may be stored as raster arrays, polygons, or run-length encodings. Raster masks represent irregular boundaries directly but can require substantial storage. Polygons are compact for simple outlines, although holes and intricate contours require additional conventions. Run-length encoding compresses sequences of foreground and background pixels without approximating the boundary. These choices affect annotation tools, memory consumption, and processing speed, but do not by themselves determine a model's recognition quality."
    },
    {
      "heading": "Development from region proposals to mask prediction",
      "text": "Instance segmentation developed at the intersection of image segmentation and category-level object recognition. Earlier segmentation methods divided images into regions using color, texture, edges, or graph-based criteria. Such regions did not necessarily correspond to complete objects: a patterned animal could be divided into several pieces, while adjacent objects with similar appearance could be merged. Recognition systems supplied category information, but category recognition alone did not determine precise object boundaries.\n\nThe success of convolutional neural networks in image recognition accelerated the combination of these capabilities. R-CNN, presented in 2014 by Ross Girshick and colleagues, classified candidate image regions using convolutional features. Also in 2014, Bharath Hariharan and colleagues introduced simultaneous detection and segmentation, commonly abbreviated SDS, which scored region candidates for both object detection and segmentation. This work helped establish instance-level mask prediction as a distinct benchmarked problem rather than simply a refinement of semantic labeling.\n\nFast R-CNN in 2015 reduced repeated computation by extracting shared image features before processing candidate regions. Faster R-CNN, also introduced in 2015, added a learned region proposal network. Although these systems were principally object detectors, their shared-feature and proposal-based designs supplied a practical foundation for instance segmentation. DeepMask, introduced by Pedro Pinheiro and colleagues in 2015, explored learned object mask proposals, while subsequent systems improved proposal refinement and joint prediction.\n\nA major milestone was Mask R-CNN, introduced in 2017 by Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. It extended Faster R-CNN with a parallel branch that predicted an object mask for each selected region. Its importance lay in a relatively simple, trainable integration of detection and segmentation, together with a carefully aligned feature-sampling operation. The method became a widely used baseline and a foundation for many domain-specific systems.\n\nLater work challenged the assumption that instance segmentation must begin with bounding-box proposals. YOLACT in 2019 combined shared mask prototypes with instance-specific coefficients; SOLO, first presented in 2019 and published in 2020, assigned instances to spatial grid locations. CondInst in 2020 generated instance-specific convolutional parameters. Transformer-based methods subsequently represented predictions as a set of mask-producing queries, with MaskFormer in 2021 and Mask2Former in 2022 demonstrating architectures applicable across several segmentation tasks.\n\nThese developments did not replace one another in a simple sequence. Proposal-based systems remain useful where mature detection pipelines and per-object processing are advantageous. Dense prediction systems emphasize throughput and shared computation, while query-based systems offer a common formulation for multiple segmentation tasks. Comparisons depend on image resolution, training data, feature extractors, and evaluation settings as much as on the architectural family."
    },
    {
      "heading": "Proposal-based architectures and spatial alignment",
      "text": "A proposal-based system first identifies a manageable set of likely object regions and then predicts a mask within each region. A typical architecture contains a feature-extraction backbone, a region proposal network, and separate heads for category classification, bounding-box refinement, and mask prediction. A residual neural network is a common backbone, while a feature pyramid network combines feature maps at several spatial scales to support objects of different sizes.\n\nThe proposal network predicts candidate boxes and objectness scores from shared features. During training, candidates are associated with annotated objects according to overlap rules. During prediction, candidates are filtered and refined. Non-maximum suppression commonly removes lower-scoring boxes that overlap strongly with higher-scoring ones, reducing duplicate detections. This step introduces a trade-off in crowded scenes: a rule intended to suppress duplicates may also suppress a nearby, genuinely distinct object.\n\nEach retained region must be converted into a fixed-size feature representation. Earlier region-of-interest pooling operations quantized coordinates when mapping a box onto the feature map and dividing it into bins. Such rounding can shift features relative to the original image. A small shift may have little effect on category recognition but can noticeably degrade a predicted boundary.\n\nMask R-CNN addressed this issue with RoIAlign. Rather than rounding the region coordinates to discrete feature-map positions, RoIAlign samples features at fractional positions using bilinear interpolation and aggregates those samples into output bins. The resulting region features retain more accurate spatial correspondence with the image. This is particularly important when a low-resolution feature grid must support pixel-level predictions.\n\nIn the original Mask R-CNN formulation, the mask branch predicts one small binary mask for each object category within a region. During training, the mask loss is applied to the channel corresponding to the region's ground-truth category. The categories do not compete through a per-pixel softmax in the mask branch; classification is handled separately. At inference, the predicted category selects the relevant mask, which is resized and placed into image coordinates. The standard architecture commonly used \\(28\\times28\\) mask outputs, although this resolution is not an inherent requirement of the task.\n\nProposal-based designs isolate each object in a normalized coordinate frame, allowing a mask head to focus on local shape. Their limitations follow from the same decomposition. A missed proposal cannot be recovered by the mask branch, inaccurate boxes can truncate masks, and repeated region processing adds computation as object counts rise. Higher-resolution mask heads and refinement stages can improve boundaries, but increase memory use and runtime."
    },
    {
      "heading": "Dense, prototype-based, and dynamically generated masks",
      "text": "An alternative to region-by-region processing is to make dense predictions over shared image features. These methods differ in how they assign predictions to objects and how they construct the final mask. “One-stage” is a broad architectural description rather than a guarantee that a system contains no intermediate operations: candidate filtering, mask assembly, and duplicate removal may still be necessary.\n\nYOLACT, introduced by Daniel Bolya and colleagues in 2019, separates mask construction into shared prototypes and instance-specific coefficients. A fully convolutional branch generates \\(K\\) prototype maps, while a detection branch predicts coefficients for each candidate object. An instance mask can be formed as\n\n\\[P_i(x,y)=\\sigma\\left(\\sum_{k=1}^{K}a_{ik}B_k(x,y)\\right),\\]\n\nwhere \\(B_k\\) is prototype \\(k\\), \\(a_{ik}\\) is an instance-specific coefficient, and \\(\\sigma\\) is the sigmoid function. The assembled mask is usually restricted to the predicted object box. The prototypes need not correspond to named object parts; training discovers spatial patterns useful for reconstruction.\n\nThis factorization shares most mask computation across objects. Its cost depends on the number and resolution of the prototypes as well as the number of retained detections. A small prototype basis can constrain the representation of complicated overlapping shapes, while increasing its capacity raises computation and memory requirements. The detection branch also remains important because it determines which coefficient vectors survive filtering and where masks are cropped.\n\nSOLO takes a different approach by assigning objects to locations in a spatial grid according to their centers and handling object sizes across feature levels. Predictions at those locations encode categories and masks. SOLOv2, introduced in 2020, separates mask features from dynamically predicted mask kernels. This avoids treating every spatial location as an entirely independent full-resolution mask generator and makes mask production more efficient.\n\nCondInst likewise uses dynamic computation, but generates parameters for a small mask-producing network conditioned on each instance. Shared image features are combined with relative position information, enabling a compact instance-specific predictor to separate one object from others. This differs from predicting an ordinary static mask head once and applying identical parameters to every object: the mask-generating operation itself changes with the candidate instance.\n\nAnother family predicts pixel embeddings, object centers, boundaries, or displacement vectors, then groups pixels into instances. For example, pixels may predict offsets toward their object's center, after which a grouping procedure assembles the masks. Such methods can be attractive for densely packed objects, but introduce sensitivity to center detection, clustering thresholds, or fragmented boundaries. Their behavior illustrates a general design choice: object identity can be represented explicitly by candidate objects or recovered from relationships among pixel-level predictions."
    },
    {
      "heading": "Query-based segmentation and promptable models",
      "text": "Query-based models formulate instance segmentation as prediction of an unordered set of objects. Instead of attaching every prediction to a predefined box or grid cell, they maintain a fixed collection of learned or image-conditioned query vectors. A decoder updates these queries using image features, and each query predicts a category and a mask. Queries that do not correspond to objects receive a “no object” outcome.\n\nThe set-prediction formulation is closely related to DETR, introduced for object detection in 2020. It uses a one-to-one assignment between predictions and annotated objects during training. A matching cost can combine category agreement with mask or box similarity. The Hungarian algorithm is commonly used to find a minimum-cost assignment. Because each annotated object is assigned to only one query, training discourages multiple queries from producing duplicate predictions for the same object.\n\nA transformer architecture can update queries through cross-attention to image features and through interactions among the queries themselves. The output is permutation-invariant at the set level even though the model internally maintains indexed query slots. The number of slots usually places an upper bound on how many objects can be returned in one prediction pass, making query capacity relevant in images containing hundreds of instances.\n\nMaskFormer, introduced by Bowen Cheng and colleagues in 2021, treated segmentation as mask classification: predict binary masks and assign a category to each mask. Mask2Former, introduced in 2022, extended this framework with masked attention that restricts attention using predicted mask regions. These systems demonstrated how related architectures could support semantic, instance, and panoptic segmentation. Their tasks remain different, however, because the training annotations and rules for interpreting outputs determine whether masks represent categories, individual objects, or a complete scene partition.\n\nPromptable segmentation addresses a related but distinct problem. The Segment Anything Model, introduced by Alexander Kirillov and colleagues in 2023, accepts prompts such as points or boxes and predicts masks. A prompt can identify the intended object without requiring a category label from a fixed recognition vocabulary. Its image encoder can compute reusable features, while a prompt-conditioned decoder produces masks for subsequent interactions.\n\nPromptable mask generation should not be equated automatically with category-aware instance segmentation. A mask generator may outline an object accurately without naming it, deciding which objects should be counted, or resolving all duplicate masks. Automatic pipelines may therefore combine it with detectors, text-conditioned models, or additional filtering. Similarly, open-vocabulary object detection and language-guided segmentation broaden the category interface, but still require a defined policy for object selection, confidence scoring, and instance identity."
    },
    {
      "heading": "Annotation, supervision, and optimization",
      "text": "Instance segmentation usually relies on supervised learning with an individual mask and category label for each annotated object. Creating these annotations is more demanding than drawing boxes because annotators must trace boundaries, separate touching objects, and apply consistent rules to occlusion, holes, reflections, and truncated objects. Annotation guidelines also determine whether an object held by a person is a separate instance and how to handle dense groups whose members cannot be delineated reliably.\n\nThe COCO dataset, introduced in 2014, became a central benchmark for common-object instance segmentation. Its widely used instance annotations cover 80 object categories. Cityscapes, introduced in 2016, provides urban street scenes with fine pixel annotations for 5,000 images and instance distinctions for selected object categories. LVIS, introduced in 2019, expands the category vocabulary to more than a thousand classes in commonly used versions and emphasizes the challenge of long-tailed category frequencies.\n\nA training objective usually combines several terms. A proposal-based model may use\n\n\\[\\mathcal{L} = \\mathcal{L}_{\\mathrm{cls}} +\\lambda_{\\mathrm{box}}\\mathcal{L}_{\\mathrm{box}} +\\lambda_{\\mathrm{mask}}\\mathcal{L}_{\\mathrm{mask}} +\\mathcal{L}_{\\mathrm{proposal}}.\\]\n\nThe terms supervise category prediction, box coordinates, masks, and proposal generation. The coefficients determine their relative influence. A loss function applied to masks may use binary cross-entropy, Dice loss, focal loss, or a combination, depending on the architecture and class imbalance.\n\nFor soft predictions \\(p_j\\) and binary targets \\(y_j\\), a common Dice-style loss is\n\n\\[\\mathcal{L}_{\\mathrm{Dice}} = 1-\\frac{2\\sum_jp_jy_j+\\epsilon} {\\sum_jp_j+\\sum_jy_j+\\epsilon}.\\]\n\nIt measures overlap at the region level rather than treating every pixel as an independent classification example. Implementations differ in smoothing constants, averaging conventions, and whether squared terms appear in the denominator. These details matter when reproducing results, especially for small objects.\n\nData augmentation can include resizing, cropping, flipping, and color transformations. Geometric transformations must be applied consistently to images, masks, and boxes. Cropping may remove an object's center or most of its visible area, so training pipelines need explicit rules for retaining or discarding affected annotations. Copying segmented objects into other images can increase exposure to rare categories, but also changes occlusion patterns and can create unrealistic boundaries.\n\nWeakly supervised methods reduce annotation requirements by learning from boxes, points, image labels, or incomplete masks. Pseudo-labeling can expand a labeled dataset using model predictions, although systematic mistakes may be reinforced. Transfer learning from pretrained image encoders is common, and self-supervised learning can supply representations without manually labeled pretraining images. Regardless of supervision type, separating training and evaluation data by independent scenes or acquisition sources helps prevent near-duplicate images from inflating measured performance."
    },
    {
      "heading": "Evaluation and benchmark interpretation",
      "text": "The basic overlap measure for an individual mask is intersection over union, or IoU:\n\n\\[\\operatorname{IoU}(M,G)=\\frac{|M\\cap G|}{|M\\cup G|},\\]\n\nwhere \\(M\\) is a predicted mask and \\(G\\) is an annotated mask. An IoU of 1 indicates exact agreement and 0 indicates no overlap. IoU alone does not evaluate a complete system because predictions also have categories, confidence scores, missing objects, and duplicates.\n\nAverage precision combines detection confidence with matching quality. Predictions are ranked by score, and a prediction is treated as a true positive when it has the correct category and matches an eligible, previously unmatched annotation at the required IoU threshold. Extra predictions and duplicates generally count as false positives; unmatched annotated objects contribute to false negatives. Precision measures the fraction of detections that are correct, while recall measures the fraction of annotated objects recovered.\n\nCOCO's principal mask AP averages results over ten IoU thresholds from 0.50 to 0.95 in increments of 0.05, as well as over categories. It therefore rewards both object recognition and precise localization. AP at 0.50 is more tolerant of imperfect boundaries than AP at 0.75 or the higher thresholds. COCO also reports results by object size and average recall under specified detection limits. Crowd annotations and ignored regions receive special treatment in the official evaluation procedure.\n\nMask AP is not interchangeable with bounding-box AP. A system may detect nearly every object with accurate boxes but produce poor contours, or delineate selected objects well while missing many instances. Mean semantic IoU is also insufficient: it can reward correct category coverage while failing to penalize the merger of adjacent same-category objects in the way instance evaluation requires.\n\nDifferent applications require complementary measurements. Boundary-based scores can reveal errors in thin structures that have little effect on total mask area. Counting error emphasizes missed and duplicated instances. Biological imaging may use object-level precision, recall, split-and-merge statistics, or aggregated overlap metrics. Panoptic segmentation commonly uses panoptic quality, which combines matched-region overlap with recognition of individual segments; it measures a related but distinct output protocol.\n\nBenchmark scores must also be interpreted alongside computational conditions. Image size, backbone capacity, test-time augmentation, external training data, maximum detections, and category coverage can materially affect results. Runtime comparisons depend on hardware, batch size, numerical precision, preprocessing, and postprocessing. A high frames-per-second figure on reduced-resolution images with few objects does not establish equivalent throughput on high-resolution crowded scenes. Reproducible comparisons specify these conditions and evaluate on an independent test set."
    },
    {
      "heading": "Applications and extensions beyond still images",
      "text": "In robotics, instance masks identify image regions belonging to separate objects and help associate those regions with depth measurements. A robot sorting items on a conveyor belt may need to distinguish touching packages before selecting a grasp target. Segmentation provides object support in the image, but grasp planning additionally requires geometry, pose, material properties, and collision constraints. A correct two-dimensional mask does not by itself establish an object's three-dimensional shape or whether it can be grasped safely.\n\nMicroscopy uses instance segmentation to separate cells, nuclei, or other structures for counting and quantitative analysis. This setting differs from common-object photography: objects can be numerous, visually similar, and tightly packed, and the boundary between touching objects may be weak. Measurements of area, shape, or intensity depend directly on the mask. A split error can count one structure twice, while a merge error can combine several structures into one measurement. Annotated targets and validation protocols must therefore match the biological structure being measured.\n\nRemote sensing applies instance-level delineation to buildings, individual tree crowns, vehicles, and other spatial objects. Large images are often processed as overlapping tiles because they exceed model input or memory limits. This creates an additional reconciliation problem: an object crossing a tile boundary may be truncated, predicted twice, or missed. Image scale, viewing angle, seasonal changes, and differences between sensors also affect how well a model transfers between geographic regions.\n\nImage editing uses masks to isolate objects for selection, recoloring, compositing, or replacement. Here the visible boundary can be more important than category recognition. Nevertheless, a binary mask is not equivalent to image matting, which estimates fractional opacity at mixed pixels such as hair, motion blur, or translucent material. Matting requires an image-formation model and may need information beyond an object's hard foreground region.\n\nVideo instance segmentation, established as a benchmarked task with YouTube-VIS in 2019, extends the problem across frames. The system must not only produce masks but also preserve object identity over time. Occlusion, rapid motion, appearance changes, and re-entry into the scene make temporal association difficult. Independent frame-by-frame segmentation can produce accurate masks yet repeatedly change identifiers, so video evaluation considers trajectories rather than isolated detections.\n\nThree-dimensional instance segmentation assigns identities to points or voxels in spatial data, such as scans from depth cameras or lidar. It can support scene reconstruction and object-level mapping, including integration with simultaneous localization and mapping. Unlike a single image mask, a three-dimensional instance may combine observations from several viewpoints. Its representation must handle irregular sampling, incomplete surfaces, and inconsistent visibility, while distinguishing persistent objects from moving objects and transient sensor artifacts."
    },
    {
      "heading": "Failure modes and unresolved research problems",
      "text": "Occlusion and crowding remain central difficulties. Visible evidence may support several plausible interpretations: two touching animals can resemble one connected shape, while an occluded object can appear as disconnected fragments. Models use category and contextual information to resolve these cases, but can also impose an incorrect familiar shape on ambiguous evidence. Instance separation therefore depends on both local boundaries and global object-level reasoning.\n\nSmall objects and fine structures expose the limits of spatial resolution. Feature extraction often reduces image dimensions substantially, and resizing a compact mask back to the original image cannot recover information that was never preserved. A thin bicycle spoke or plant stem may occupy only a fraction of a feature-map cell. Increasing input resolution, incorporating high-resolution features, or refining boundaries can help, but increases computation and still cannot overcome insufficient sensor detail.\n\nRecognition coverage is another limitation. Fixed-vocabulary systems cannot assign an unseen category its correct label unless that category is represented in their output space. Open-vocabulary methods use language-associated representations to broaden recognition, yet uncommon names, visually similar categories, and ambiguous descriptions remain difficult. Class-agnostic mask prediction avoids some category restrictions but does not resolve the question of which regions count as objects or which level of a part–whole hierarchy should be selected.\n\nDistribution shift can arise from new cameras, illumination, weather, locations, or imaging modalities. A model trained on street photographs may fail on aerial views even when the nominal categories overlap. Long-tailed datasets create an additional imbalance: frequent categories provide abundant examples, while rare categories may have little shape variation in training. Aggregate scores can conceal these disparities, making performance by category, size, and acquisition condition important for understanding a model's operating range.\n\nConfidence estimation is also difficult because category correctness and mask quality are not identical. A model may confidently recognize a person while producing an inaccurate outline, or generate a good boundary with an uncertain category. Some architectures explicitly estimate mask quality to improve ranking. Even then, score calibration can change under distribution shift, and disagreement between annotators may reflect genuine ambiguity rather than a simple modeling error.\n\nScalability remains constrained by image resolution and object count. Explicitly storing \\(N\\) masks of height \\(H\\) and width \\(W\\) requires storage proportional to \\(NHW\\) before compression or alternative representations. Prototype factorization, sparse sampling, tiled processing, and compact mask heads reduce parts of this cost but introduce their own trade-offs. In dense scenes, limits on proposals, queries, or retained detections can cap recall independently of boundary quality. Research consequently addresses not only more accurate masks, but also how to represent large numbers of uncertain, overlapping objects without prohibitive memory and computation."
    }
  ],
  "links": [
    "computer-vision",
    "object-detection",
    "semantic-segmentation",
    "deep-learning",
    "pixel",
    "panoptic-segmentation",
    "image-segmentation",
    "convolutional-neural-network",
    "residual-neural-network",
    "non-maximum-suppression",
    "transformer-architecture",
    "cross-attention",
    "open-vocabulary-object-detection",
    "supervised-learning",
    "coco-dataset",
    "loss-function",
    "data-augmentation",
    "transfer-learning",
    "self-supervised-learning",
    "intersection-over-union",
    "average-precision",
    "training-data",
    "test-set",
    "simultaneous-localization-and-mapping",
    "distribution-shift"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}