Image segmentation is the process of dividing a digital image into regions that correspond to objects, object parts, materials, or other meaningful structures. Unlike image classification, which assigns a label to an entire image, or object detection, which usually locates objects with bounding boxes, segmentation specifies their spatial extent at the level of individual pixels. Its output may be a foreground mask, a map of semantic categories, a collection of object-instance masks, or an unlabeled partition based on visual similarity.
Segmentation is a central problem in computer vision because many forms of image understanding require knowing not only what is present but exactly where it is. It supports measurements of organs and cells, mapping of roads and land cover, robotic scene interpretation, industrial inspection, and selective image editing. The problem is difficult because visible boundaries do not always coincide with object boundaries: shadows can divide one surface, while two touching objects can have almost identical appearance.
Methods range from intensity thresholds and geometric models to large deep learning systems trained on annotated images. Across these approaches, the essential questions remain how to define a meaningful region, how to combine local evidence with broader context, and how to evaluate the resulting boundaries and labels.
Problem formulation and principal tasks
For a two-dimensional image, let be the set of pixel locations and let represent its measured channels, such as three color components. A segmentation can be expressed as a labeling function
Each label identifies a region or category. In a strict partition, the resulting regions are disjoint and cover the entire image. Practical systems may instead leave uncertain pixels unlabeled, produce overlapping candidate masks, or assign probabilities rather than final labels. For three-dimensional images, the same formulation applies to volume elements, or voxels; for video, labels additionally vary with time.
Semantic segmentation assigns a category to each pixel, such as road, building, vegetation, or person. Separate objects of the same category ordinarily receive the same label. Instance segmentation distinguishes individual objects: two adjacent people receive different instance identifiers even though both belong to the person category. These identifiers are arbitrary; exchanging the numerical identifiers of two objects does not change the segmentation. This makes instance prediction partly a grouping problem rather than simply pixelwise classification.
Panoptic segmentation combines semantic coverage with instance separation. It distinguishes countable “things,” such as cars and people, from amorphous “stuff,” such as sky and grass. Every labeled pixel receives a semantic category, and pixels belonging to things also receive an instance identifier. A binary foreground–background mask is a simpler task, but its meaning still depends on the application: foreground might mean a selected person, all moving objects, or a particular tissue type.
Not every segmentation seeks semantic categories. A superpixel method groups nearby pixels into small, visually coherent regions that can simplify subsequent processing. Conversely, part segmentation can divide a recognized object into components, such as a vehicle’s wheels and body. Amodal segmentation estimates an object’s complete extent, including occluded portions, whereas ordinary visible-mask segmentation describes only the observed surface.
The desired answer therefore depends on conventions as well as image evidence. A transparent container, a reflection, an object seen through a fence, and a partly hidden animal all require explicit labeling rules. Segmentation also differs from image matting: matting estimates fractional opacity at boundaries such as hair or smoke, while a hard segmentation assigns discrete membership. A soft class probability expresses uncertainty about a label and is not necessarily a physical opacity.
Thresholds, regions, and morphological methods
Many early and still widely used methods treat segmentation as the identification of regions with similar measured properties. Thresholding is the simplest example. For a grayscale image, a binary mask may be defined by when , and otherwise. This works well when foreground and background occupy distinct intensity ranges, as can occur in controlled industrial imaging or images of stained specimens. It becomes unreliable when illumination varies or the intensity distributions overlap.
The Otsu method, introduced by Nobuyuki Otsu in 1979, selects a threshold by maximizing the between-class variance of two intensity groups. If their probabilities are and their mean intensities are , the criterion is
It requires no manually selected intensity threshold, but its statistical criterion does not establish that the resulting classes correspond to meaningful objects. Adaptive thresholding instead computes thresholds from local neighborhoods, making it more suitable for uneven illumination, such as a photographed document.
Region-growing methods begin with seed pixels and add neighboring pixels that satisfy a similarity rule. Region splitting and merging work in the opposite or a combined direction, dividing heterogeneous areas and joining compatible neighbors. Their results depend on the similarity measure, neighborhood definition, processing order, and stopping criterion. K-means clustering can group pixels by color or other features, but without spatial constraints it can assign distant, disconnected pixels to the same cluster. Including position in the feature vector encourages compact regions but introduces a scale parameter balancing appearance against distance.
Watershed segmentation interprets an image-derived scalar field, often gradient magnitude, as a topographic surface. Catchment basins form regions, and the ridges separating them form boundaries. Small fluctuations can produce severe oversegmentation. Marker-controlled watershed limits the process to selected foreground and background markers, making it useful for separating touching objects when reliable interior markers are available. For approximately round objects, peaks in a distance transform can provide such markers.
Mathematical morphology modifies masks using a structuring element. Erosion shrinks foreground sets, dilation expands them, opening can remove small protrusions or isolated objects, and closing can bridge narrow gaps. Connected-component analysis then identifies individual connected regions under a chosen neighborhood rule. These operations are valuable components of practical image processing pipelines, but they encode geometric assumptions. A structuring element that removes noise can also erase a genuine thin vessel, and a closing operation can incorrectly join adjacent objects.
Graph-based and variational formulations
Graph-based methods represent image elements as nodes connected by weighted edges. Nodes may correspond to pixels or superpixels, while edge weights encode similarity, spatial adjacency, or boundary evidence. Segmentation becomes a problem of assigning labels or partitioning this graph. The formulation permits local measurements to interact with global constraints, rather than requiring each pixel to be classified independently.
A common objective has the form
where measures the cost of assigning label to pixel , specifies neighboring pairs, and controls the penalty for placing a boundary between them. Strong penalties between visually similar neighbors encourage coherent regions; weaker penalties across sharp image changes allow boundaries. The parameter balances agreement with the image against spatial smoothness.
For certain binary objectives, a minimum graph cut gives an exact global optimum through a minimum-cut/maximum-flow construction. More general label sets and interactions require additional conditions or approximate optimization. GrabCut, introduced by Carsten Rother, Vladimir Kolmogorov, and Andrew Blake in 2004, combined iterative color modeling with graph-cut optimization for interactive foreground extraction. A user-provided rectangle established an initial foreground region, and additional interaction could correct mistakes. The example illustrates how limited human input can resolve ambiguity that appearance alone cannot.
Normalized cuts, developed by Jianbo Shi and Jitendra Malik and published in 2000, approach segmentation as graph partitioning. The objective penalizes separation between groups while normalizing by their connections to the overall graph, discouraging trivial cuts that isolate tiny sets. Approximate solutions use spectral properties of graph matrices. The method helped establish a connection between image grouping, clustering, and global optimization, although constructing and solving large graph problems can be computationally demanding.
Variational approaches optimize a boundary or continuous field. The Mumford–Shah model, proposed in 1989, seeks a piecewise-smooth image approximation while penalizing discontinuity length. Active contours, introduced as “snakes” by Michael Kass, Andrew Witkin, and Demetri Terzopoulos in 1988, balance internal curve regularity against image-derived forces. Level-set formulations represent contours implicitly, allowing regions to split and merge. Region-based active contours, such as the Chan–Vese model of 2001, can operate without requiring a strong image gradient at every boundary.
Probabilistic graphical models provide another interpretation of spatial coherence. A conditional random field models label dependencies conditioned on image observations. Such models have been used both as primary segmentation systems and to refine neural-network predictions. Their central limitation is also their strength: smoothness, connectivity, and appearance assumptions are explicit, so errors often arise when real scenes violate those assumptions.
Deep networks for dense semantic prediction
The rise of convolutional neural networks changed segmentation from a pipeline dominated by manually designed features into one that learns visual representations and dense prediction jointly. Classification networks typically reduce spatial resolution to obtain increasingly abstract features. Segmentation requires recovering detailed spatial output from those features without losing the contextual information needed to distinguish visually similar categories.
The fully convolutional network work of Jonathan Long, Evan Shelhamer, and Trevor Darrell, published in 2015, demonstrated how classification networks could be adapted for end-to-end dense labeling. Replacing fixed-size classification layers with convolutional operations allowed spatial score maps. Upsampling restored output resolution, while skip connections combined coarse semantic features with finer features from earlier layers. The approach avoided separately classifying a large overlapping patch around every pixel.
An encoder–decoder architecture makes this division explicit. The encoder extracts progressively more contextual features; the decoder reconstructs a spatially detailed prediction. U-Net, introduced by Olaf Ronneberger, Philipp Fischer, and Thomas Brox in 2015, used an approximately symmetric contracting and expanding architecture with skip connections between corresponding resolutions. Its biomedical demonstrations showed the value of combining precise localization with learned context, particularly when annotated data were limited and augmentation was extensive. Three-dimensional variants apply related principles to volumetric data.
Other architectures emphasize the size and diversity of the receptive field. Dilated, or atrous, convolution inserts spaces between kernel samples, enlarging the receptive field without a corresponding increase in learned kernel parameters. DeepLab systems combined this idea with multiscale processing; atrous spatial pyramid pooling uses several dilation rates to capture context at different scales. Pyramid pooling similarly aggregates features over regions of different sizes. Such mechanisms help distinguish, for example, a small patch of road from a visually similar roof by incorporating surrounding structure.
More recent systems use the transformer architecture, including vision transformers, to model longer-range relationships. Self-attention allows features at one location to interact with features elsewhere, while hierarchical and windowed designs control computational cost. Some models predict a set of masks and associated classes rather than treating every pixel solely as an independent classification output. These designs also create architectural links between semantic, instance, and panoptic segmentation.
A network usually produces a score or probability distribution for each output location, followed by upsampling and label selection. High accuracy does not imply exact boundaries: downsampling, interpolation, class imbalance, and limited training examples can all suppress thin structures or small objects. Architecture, training objective, and image resolution therefore jointly determine what spatial detail a model can retain.
Separating object instances and unifying scene labels
Instance segmentation must solve two coupled questions: which pixels belong to an object, and which object each pixel belongs to. Pixelwise semantic classification alone cannot distinguish touching objects of the same class. A successful system needs some representation of object identity, whether through proposals, centers, embeddings, queries, or explicit grouping rules.
Proposal-based systems first locate candidate objects and then predict a mask for each candidate. Mask R-CNN, introduced by Kaiming He and colleagues in 2017, extended a region-based detection system with a parallel mask-prediction branch. Its RoIAlign operation sampled region features without the coarse coordinate quantization used by earlier pooling methods, improving alignment between image locations and mask predictions. The mask branch predicted spatial foreground membership within each proposed region, while separate branches handled object classification and bounding-box refinement.
This decomposition provides a clear association between each detection and its mask, but it inherits detection errors. A missed proposal cannot yield a correct instance mask, and highly overlapping objects can be difficult to separate. Computational work can also increase with the number of proposals. Alternative systems predict masks more directly, generate instance-specific filters, or group pixels using predicted centers and offsets. Embedding-based methods learn feature vectors that bring pixels of the same instance together while separating pixels from different instances.
Set-prediction approaches maintain a collection of object or mask queries. During training, predicted outputs are matched to annotated targets, often through a bipartite assignment procedure. This addresses the fact that instance identifiers have no inherent ordering: the first predicted mask need not correspond to the first annotated object. Unmatched predictions can be trained as “no object.” Mask-based transformer architectures have extended this formulation across several segmentation tasks rather than requiring entirely separate output designs.
Panoptic segmentation adds the requirement that the final visible scene labeling be consistent. An ordinary instance system may output overlapping masks, whereas a standard panoptic map assigns each pixel to only one final segment. Systems must therefore resolve overlaps, combine thing and stuff predictions, and handle categories differently when instances are meaningful. The panoptic formulation introduced by Alexander Kirillov and colleagues brought these requirements into a shared task and evaluation framework.
Crowded scenes expose the limits of visible evidence. A mask may be geometrically accurate but assigned to the wrong instance, while a semantically correct region may merge several people. Occlusion conventions matter: visible-instance masks exclude hidden surfaces, while amodal methods infer them. Video adds persistent identity across frames, turning individual image masks into trajectories and requiring systems to avoid identity switches when objects disappear and reappear.
Annotation, supervision, and training objectives
Dense annotation is expensive because a single image may contain many objects and intricate boundaries. Annotators use polygons, brush tools, interactive models, or three-dimensional contouring interfaces. The resulting labels depend on rules concerning occlusion, reflections, narrow gaps, uncertain edges, and minimum object size. Training data therefore encode both visual examples and operational definitions of the task. Two well-annotated datasets can disagree because their labeling policies differ.
With supervised learning, a common objective is pixelwise cross-entropy. If indicates the reference class and is the predicted probability for class at pixel , then
The loss treats errors at different pixels as separable unless additional terms introduce spatial structure. Class weighting can reduce domination by common categories. Focal loss downweights easily classified examples, while overlap-based objectives such as soft Dice loss directly emphasize region agreement. Boundary and topology-aware losses target properties that ordinary pixelwise objectives may not preserve.
Data augmentation exposes models to transformations such as cropping, reflection, changes in brightness, and geometric deformation. The validity of a transformation depends on the task: a rotation may preserve cell identity but alter the meaning of text orientation. Images and masks must undergo corresponding geometric transformations, and interpolation rules must respect categorical labels. In three-dimensional imaging, the physical spacing between slices also affects whether a deformation is plausible.
Weak supervision reduces annotation requirements by using image-level labels, bounding boxes, points, or scribbles. These provide less information about exact boundaries, so methods rely more heavily on learned priors and propagation from sparse constraints. Semi-supervised learning combines labeled and unlabeled images, often using pseudo-labels or consistency under perturbations. Pseudo-labeling can reinforce systematic mistakes if confident predictions are treated as reliable regardless of their actual correctness.
Self-supervised learning and transfer learning can provide useful representations before dense labels are available. Large pretrained models may then be adapted through fine-tuning or lightweight task-specific components. Nevertheless, pretraining does not eliminate differences in image formation, category definitions, or scale. Features learned from everyday photographs may require substantial adaptation for electron microscopy, thermal images, or multispectral satellite data.
Data separation is particularly important because neighboring pixels, adjacent video frames, or image slices from the same subject are highly correlated. Randomly distributing these across training and evaluation sets can cause data leakage. Patient-level, scene-level, geographic, or temporal separation is often necessary to measure generalization to genuinely independent cases.
Interactive, promptable, and video segmentation
Interactive segmentation treats user input as part of the specification. A positive click can identify an intended object, a negative click can exclude an unwanted region, and a box can establish a spatial constraint. Classical graph-based tools and modern neural models share this principle: limited interaction supplies information that may be impossible to infer from the image alone. The objective is not merely automatic accuracy but accurate masks with little corrective effort.
Promptable models generalize interaction across many image categories. The Segment Anything Model, introduced by Alexander Kirillov and colleagues in 2023, combined an image encoder, a prompt encoder, and a mask decoder. Its associated SA-1B dataset contained approximately 11 million images and 1.1 billion masks. An expensive image representation could be computed once and reused for multiple prompts, while the mask decoder produced candidate masks more quickly. Multiple outputs helped represent ambiguity when a click could reasonably refer to a part, an entire object, or a larger grouping.
A promptable mask generator is not automatically a semantic recognition system. It may delineate an object without naming it, and a spatial prompt does not guarantee that a mask follows the category definition required by a specialized application. Text-conditioned systems add a connection to natural language processing and multimodal learning, allowing requests such as a named category or a referring expression. These systems must resolve both linguistic reference and spatial boundaries; “the cup beside the plate” demands relational understanding beyond recognizing cups.
Video segmentation introduces temporal continuity. Some tasks begin with a mask in the first frame and propagate it; others detect and segment objects automatically throughout the sequence. Optical flow can transfer information by estimating image motion, but occlusion, fast motion, motion blur, and newly appearing surfaces complicate propagation. Memory-based models store features from earlier frames and retrieve them when processing later frames. Segment Anything 2, introduced in 2024, extended promptable segmentation to images and video with a streaming-memory architecture.
Temporal consistency is not simply frame-by-frame accuracy. A model can produce good masks in individual frames while changing object identity, flickering at boundaries, or drifting onto a nearby object. Long-term tracking also requires distinguishing disappearance from failure: an occluded object should not necessarily be assigned to the visible background or another similar object.
Interactive evaluation therefore measures more than overlap after a single prediction. Relevant quantities include the number of clicks needed to reach a target accuracy, response latency, recovery after corrective prompts, and performance over long sequences. Evaluation protocols must specify how prompts are generated, because carefully chosen human prompts and automatically simulated clicks can produce different results.
Evaluation metrics and benchmark design
Segmentation quality has several dimensions: correct category assignment, region overlap, boundary accuracy, instance separation, and, where relevant, temporal identity. No single metric captures all of them. Pixel accuracy is easy to compute but can be misleading when one category dominates. If only 1 percent of an image belongs to the target, predicting background everywhere gives 99 percent pixel accuracy while detecting none of the target.
The most widely used overlap measure is intersection over union (IoU), also called the Jaccard index. For a predicted region and reference region ,
Semantic benchmarks commonly report mean IoU across categories. Results depend on which classes are included, whether unlabeled pixels are ignored, and how counts are aggregated. Computing IoU from totals over a dataset is not equivalent to averaging per-image IoU, particularly when objects vary greatly in size.
The Dice coefficient is
For nonempty binary regions, Dice and IoU are related by , so they order predictions identically for a fixed comparison. Their numerical values are nevertheless different. Both require an explicit convention when prediction and reference are empty. In volumetric analysis, an overlap score alone may conceal a substantial local boundary error.
Boundary metrics address this limitation. Boundary precision and recall compare contours within a specified tolerance. The Hausdorff distance measures the greatest nearest-boundary discrepancy and is sensitive to isolated outliers; percentile variants reduce that sensitivity. Average surface distance measures a different aspect of geometric agreement. Distances expressed in physical units are important when pixels or voxels have unequal spacing.
Instance benchmarks often use average precision based on mask IoU and confidence-ranked predictions. The COCO evaluation averages results over IoU thresholds from 0.50 to 0.95 in steps of 0.05, rewarding both detection and precise mask placement. Panoptic quality combines matched-segment overlap with a recognition term that penalizes missed and spurious segments. Video benchmarks additionally measure temporal association.
Benchmark datasets also define what is being tested. PASCAL VOC helped standardize semantic segmentation of common object categories; Cityscapes emphasizes urban street scenes; COCO supports instance and panoptic tasks; ADE20K includes diverse scenes and many semantic categories. Scores are not directly comparable across these settings. Independent test sets, fixed preprocessing, explicit category mappings, and reporting by object size or scene condition are essential for interpreting differences between methods.
Scientific, industrial, and creative applications
In biomedical imaging, segmentation converts image intensities into structures that can be measured. Masks can support organ-volume estimation, delineation of anatomical regions, or quantification of cells and tissue components. Magnetic resonance imaging illustrates why the task is not equivalent to ordinary photographic recognition: contrast depends on acquisition settings, slices can have anisotropic spacing, and a boundary may be poorly defined by intensity alone. Three-dimensional context and consistent annotation protocols are therefore important.
Microscopy presents a related but distinct set of problems. Closely packed cells must often be separated into instances, while nuclei may serve as markers for a watershed or a learned instance model. Measurements can include cell counts, area distributions, shape, or fluorescence intensity within masks. Errors propagate into these measurements: merging two cells reduces the count and inflates estimated size, whereas splitting one cell has the opposite effect. A high pixel overlap score does not necessarily guarantee accurate biological quantification.
In remote sensing, segmentation maps land cover, buildings, roads, water, and other geographic features. Images may contain spectral bands outside visible light and may vary substantially in ground sampling distance. A road only a few pixels wide can be important for network connectivity even though it contributes little to total pixel count. Seasonal changes, cloud cover, different sensors, and regional variation create substantial distribution shift. Mapping changes over time also requires distinguishing real changes from differences in registration or acquisition.
Robotics and automated driving use segmentation to identify traversable surfaces, obstacles, and scene structure. Semantic labels can be combined with depth measurements and geometric maps, while instance masks help separate nearby objects. Performance requirements extend beyond benchmark accuracy to latency, stability, and behavior under poor visibility. Segmentation is generally one component of a larger perception system rather than a complete basis for navigation or decision-making.
Industrial systems use masks to measure manufactured parts, locate contamination, and identify surface defects. Under controlled lighting, simple thresholding can outperform a more complex system in predictability and cost. Learning-based methods become useful when product appearance varies or defects have complicated structure. Since defective examples may be rare, segmentation can be combined with anomaly detection rather than trained only as a conventional category-labeling task.
In photography, film, and computer graphics, segmentation supports background replacement, selective color adjustment, object removal, and compositing. Document analysis uses region masks to separate text, tables, illustrations, and page structure. These uses differ in boundary requirements: a rough object mask may be sufficient for retrieval, whereas convincing compositing around hair often requires matting in addition to segmentation.
Reliability, computational constraints, and unresolved problems
Segmentation remains sensitive to the relationship between training conditions and deployment conditions. Changes in camera response, illumination, weather, image compression, or acquisition equipment can alter predictions without changing the underlying scene. Models may exploit contextual correlations, such as an object’s usual surroundings, instead of its defining appearance. Consequently, strong performance on a familiar benchmark does not establish reliability for a different site or sensor.
Ambiguity is sometimes irreducible at the image level. A blurred edge may support several plausible contours, and an object category may have no universally agreed boundary. Annotation disagreement can therefore reflect genuine uncertainty rather than careless labeling. Probabilistic outputs, multiple candidate masks, and ensemble learning can represent aspects of uncertainty, but confidence scores are not automatically calibrated probabilities. A system can be highly confident in an incorrect mask under unfamiliar conditions.
Global structure is another persistent difficulty. Pixelwise losses and local overlap measures do not directly ensure that a road network stays connected, a vessel remains unbroken, or two adjacent instances remain separate. Explicit shape models, graph constraints, and topology-aware objectives attempt to encode such properties. However, a strong prior can suppress unusual but genuine structures. Balancing flexible appearance recognition with reliable geometric constraints remains an active research problem.
Computation limits what can be processed at full resolution. An image with dense class scores requires output storage, apart from intermediate features. For global attention over image tokens, a conventional attention matrix contains entries. Three-dimensional volumes magnify memory demands, motivating patches, sliding-window inference, hierarchical models, and windowed attention. Tiling can introduce seams or remove context unless overlapping predictions and border handling are carefully designed.
Deployment often uses graphics processing units, reduced-precision arithmetic, knowledge distillation, or neural network quantization to reduce latency and memory use. These transformations can affect small objects and boundaries differently from large regions. End-to-end timing must include image loading, preprocessing, transfers, postprocessing, and mask assembly, not only the neural network’s central computation.
Large foundation models broaden the range of objects and prompts that a single system can handle, but they do not remove the need for task-specific evaluation. Open-ended recognition, unfamiliar scientific imagery, persistent video identity, and accurate treatment of transparent or heavily occluded objects remain difficult. Reliability also depends on traceable data preparation and model versions: a small change in resampling, label conversion, or threshold selection can change the final mask even when the underlying network is unchanged.