aiwiki.page
English

Object Tracking

Object tracking estimates the changing positions and persistent identities of objects across sequences of images or other sensor measurements.

34 keywords1 linked from10 not yet writtenWritten by AI
Computer VisionObject DetectionInstance Segment…Kalman FilterOptical FlowDeep LearningConvolutional ne…Transformer Arch…Object Tra…

Object tracking is the computational task of estimating the state of one or more objects over time from a sequence of observations. The state may include position, velocity, size, orientation, shape, and an identity that persists between observations. In computer vision, observations usually come from video, but tracking also uses radar, lidar, depth sensors, and combinations of sensing systems. A tracker produces trajectories rather than merely independent descriptions of individual frames.

Tracking differs from object detection, which locates objects within an observation without necessarily establishing their correspondence over time. It also differs from recognition: maintaining the identifier “track 17” does not imply knowing a person's name or an object's real-world identity. These distinctions matter in applications ranging from biological microscopy and industrial inspection to traffic analysis, interactive media, and robotic navigation.

The central difficulty is that observations are incomplete and ambiguous. Objects disappear behind obstacles, change appearance, move unpredictably, and resemble other objects; sensors introduce noise, distortion, and missed measurements. Tracking systems address these difficulties through motion prediction, visual representation, association between observations, and explicit management of uncertainty. Their design depends on whether the task concerns one designated target, an unknown number of objects, or the coordinated interpretation of several sensors.

Task definitions and representations

In single-object tracking, a system is typically given the location of a particular target in the first frame and must follow that target thereafter. The target may be specified by a rectangular bounding box, a segmentation mask, or a set of points. Unlike a conventional category detector, the tracker may have to follow an object whose category was not represented during training. A cardboard package, an unfamiliar animal, or a particular region of machinery can therefore be valid targets. The essential requirement is continuity of the designated instance, not recognition of its category.

Multiple-object tracking estimates several trajectories simultaneously. Objects may enter and leave the field of view, so the number of active tracks changes over time. In category-specific tracking, the system follows instances of selected classes, such as pedestrians or vehicles. In class-agnostic tracking, the eligible objects are defined more broadly. Multi-object tracking and segmentation extends the output from boxes to per-object masks, bringing tracking into close contact with instance segmentation. Point tracking instead follows selected image locations, which need not correspond to independently moving objects.

Representations determine both the information retained and the ambiguities introduced. A two-dimensional axis-aligned bounding box is compact but includes background and poorly describes articulated or rotated objects. A rotated box adds orientation; a mask describes visible image support more precisely. Three-dimensional tracking may represent an object by a center, dimensions, orientation, and velocity in a world or vehicle coordinate system. Tracking a person's joint positions requires a structured representation whose components are related by anatomical constraints rather than a single rigid box.

A further distinction concerns the information available when producing an estimate. An online tracker processes observations in temporal order without consulting future frames. An offline tracker can revise earlier associations after seeing an entire recording. Fixed-lag systems occupy an intermediate position: they delay output by a limited interval to resolve ambiguities. Online does not necessarily mean real-time. An algorithm can operate causally but still take longer to process a frame than the interval between frames.

Short-term tracking usually assumes that a target remains at least partly observable and that its location can be inferred from nearby frames. Long-term tracking must also detect disappearance and recover the target after an extended absence. In either setting, the persistent track identifier is an algorithmic hypothesis. When two visually indistinguishable objects pass behind an opaque obstacle, their subsequent identities may be impossible to determine from the available images alone.

State estimation and motion models

Tracking can be formulated as inference about a hidden dynamical state. Let xtx_t describe an object's state at time tt, and let ztz_t denote the corresponding observation. A transition model describes how the state evolves, while an observation model connects states to measurements:

xt=f(xt−1)+wt,zt=h(xt)+vt,x_t=f(x_{t-1})+w_t,\qquad z_t=h(x_t)+v_t,

where wtw_t and vtv_t represent process and measurement noise. The distinction is important: process noise represents departures from the assumed motion model, whereas measurement noise represents uncertainty in what the sensor reports.

Bayesian filtering alternates prediction and correction. The prediction step propagates the previous state distribution through the motion model:

p(xt∣z1:t−1)=∫p(xt∣xt−1)p(xt−1∣z1:t−1) dxt−1.p(x_t\mid z_{1:t-1})= \int p(x_t\mid x_{t-1})p(x_{t-1}\mid z_{1:t-1})\,dx_{t-1}.

The correction step incorporates the new measurement through its likelihood:

p(xt∣z1:t)∝p(zt∣xt)p(xt∣z1:t−1).p(x_t\mid z_{1:t})\propto p(z_t\mid x_t)p(x_t\mid z_{1:t-1}).

This framework separates assumptions about motion from assumptions about sensing. It also makes clear that a predicted location during an occlusion is not equivalent to an observed location.

The Kalman filter, introduced by Rudolf E. Kálmán in 1960, gives a recursive solution for linear systems with Gaussian uncertainty. A common image-plane state contains box-center coordinates, velocity, and parameters describing box size. Under a constant-velocity model, the predicted position advances by velocity multiplied by elapsed time. The covariance matrix grows during prediction and contracts when informative measurements arrive. The Kalman gain determines how strongly the estimate is corrected, balancing predicted uncertainty against measurement uncertainty. Constant velocity is a useful local approximation, not a claim that objects never accelerate.

Nonlinear observation geometry or non-Gaussian uncertainty requires other methods. Extended Kalman filters linearize nonlinear functions near the current estimate; unscented Kalman filters propagate selected representative points. A particle filter represents uncertainty with weighted samples and can retain several competing location hypotheses. Michael Isard and Andrew Blake's CONDENSATION work, published in 1998, demonstrated this approach for visual contour tracking in clutter. Particle methods can model multimodal uncertainty, but their computational requirements increase when many state dimensions must be explored.

Motion models also constrain which measurements are plausible. A statistical gate can reject detections too far from a predicted state relative to its covariance, reducing impossible associations before more expensive comparisons. Camera motion complicates this process because image displacement combines object motion with viewpoint change. A pedestrian standing still in the world may move rapidly across the image of a turning vehicle. Accurate tracking therefore depends on choosing an appropriate coordinate frame and distinguishing uncertainty in the object's behavior from uncertainty in the observing platform.

Visual appearance and single-target tracking

A visual tracker needs a way to distinguish its target from the surrounding image. Early approaches used intensity templates, edges, color histograms, and local features. Template matching searches for an image patch resembling a stored example, but direct pixel comparison is sensitive to lighting changes, rotation, deformation, and partial occlusion. Histogram-based methods tolerate some shape changes but discard spatial arrangement: two different objects can have similar color distributions. These trade-offs motivated representations that combine local structure with robustness to nuisance variation.

Optical flow estimates image motion between frames and can propagate target points or regions. The Lucas–Kanade method, introduced in 1981, estimates local displacement under assumptions including approximate brightness constancy and sufficiently small motion. Tracking several reliable features within an object can be more stable than matching an entire rectangular patch. However, features may disappear, move onto the background, or belong to different parts of a deformable object. Optical flow supplies local correspondence evidence; it does not by itself establish that a set of points belongs to one persistent object.

Discriminative trackers formulate localization as distinguishing the target from nearby background. The MOSSE tracker, published in 2010, helped establish efficient adaptive correlation-filter tracking. A correlation filter learns a response that peaks near the target location, and frequency-domain computation can make localization inexpensive. Later methods incorporated richer features and improved treatment of scale and boundaries. Their efficiency made them useful on limited hardware, although the assumptions used to construct training examples could introduce background artifacts and constrain the search region.

Deep learning changed appearance matching by learning representations from large collections of image pairs and videos. Siamese trackers compare an exemplar of the target with candidate regions in a search image using shared feature extractors. SiamFC, presented in 2016, used a fully convolutional similarity formulation. Subsequent systems added bounding-box regression, region-proposal mechanisms, or more elaborate interaction between template and search features. Convolutional neural networks provided the dominant early backbone, while later trackers increasingly used the transformer architecture to exchange information across image regions.

Updating the appearance model creates a fundamental stability–adaptation trade-off. A fixed template may fail after the target changes pose or illumination. Updating too aggressively can incorporate an occluding object or background region, causing drift: the tracker confidently follows the wrong content and reinforces its mistake with further updates. Systems counter this with confidence thresholds, multiple templates, protected initial exemplars, and separate local tracking and global recovery mechanisms. Long-term tracking also needs an explicit absent state, because selecting the best candidate in every frame produces a location even when the target is no longer visible.

Multiple objects and data association

In multiple-object tracking, the central combinatorial problem is data association: deciding which observations correspond to which existing objects. The widely used tracking-by-detection architecture first runs a detector on each frame and then links its outputs over time. This separates object discovery from temporal reasoning and allows a tracker to benefit from improvements in detectors such as Faster R-CNN or YOLO. It also exposes a dependency: objects that the detector systematically misses are difficult for the association stage to maintain reliably.

Association usually combines several sources of evidence. Predicted geometry measures whether a detection lies near the expected location. Intersection over union measures bounding-box overlap. Appearance embeddings measure visual similarity, while category labels, size constraints, and scene geometry rule out unlikely matches. A typical pairwise cost is

Cij=λmdmotion(i,j)+λadappearance(i,j)+λgdgeometry(i,j),C_{ij}= \lambda_m d_{\mathrm{motion}}(i,j) +\lambda_a d_{\mathrm{appearance}}(i,j) +\lambda_g d_{\mathrm{geometry}}(i,j),

where track ii is compared with detection jj. The terms require compatible scaling and appropriate gating; a large appearance similarity should not automatically justify physically impossible motion.

For frame-to-frame matching, the Hungarian algorithm or related linear-assignment solvers find a minimum-cost one-to-one assignment. Practical systems must also allow unmatched tracks and unmatched detections. An unmatched detection may start a tentative track; repeated support can confirm it. An unmatched track may remain alive temporarily under motion prediction and later be deleted. Confirmation thresholds suppress short-lived false detections, while deletion thresholds determine how long the system tolerates missing observations. These policies strongly affect fragmentation and false-track persistence.

SORT, introduced by Alex Bewley and colleagues in 2016, showed the effectiveness of a comparatively simple combination of detection, Kalman filtering, and assignment based on box overlap. Deep SORT, introduced by Nicolai Wojke and colleagues in 2017, added learned appearance information to improve association through occlusion. In pedestrian applications, these descriptors are related to person re-identification, although maintaining local video identity and recognizing a person across cameras are distinct tasks. ByteTrack, published in 2022, emphasized associating lower-confidence detections with existing tracks rather than discarding all of them immediately.

Greedy local matching is not the only approach. Offline systems can link short trajectory fragments through a graph and optimize associations over a longer interval. Multiple hypothesis tracking retains alternative explanations until later evidence distinguishes them, at a potentially substantial computational cost. Joint probabilistic data association instead distributes support among plausible associations. In crowded scenes, these methods address the same underlying ambiguity: two nearby detections can both be plausible continuations of one track, while a missed detection can make a correct trajectory appear discontinuous.

Joint detection, representation, and temporal learning

The boundary between detection and tracking is architectural rather than absolute. A detector normally processes an image to identify objects, while a tracker imposes temporal continuity. Joint models share features or optimization objectives between these tasks. This can reduce repeated computation and allow temporal evidence to improve weak detections, but it also couples errors: a representation optimized for category recognition may suppress the instance-specific differences needed to distinguish two objects of the same class.

Joint detection-and-embedding systems produce locations and appearance descriptors from a shared network. FairMOT, published in 2021, examined how to balance these objectives in a unified multi-object tracker. Its motivation illustrates a broader design problem: object localization and identity discrimination may favor different spatial features and training signals. A detection branch benefits from invariance to within-class appearance, whereas an association branch must preserve precisely those details. Shared computation is therefore not automatically equivalent to compatible objectives.

Other models propagate information directly between frames. CenterTrack, published in 2020, combined object-center detection with displacement estimation using consecutive images and previous detection information. This formulation makes association part of the network's prediction rather than relying only on a separately calculated motion model. It works particularly naturally when displacement is modest and objects remain visible, but abrupt camera motion, long gaps, and large displacements still require sufficient spatial context or additional recovery mechanisms.

Transformer-based systems use attention mechanisms to exchange information between object representations and image features. Following developments in DETR, some trackers maintain persistent track queries that carry information about previously observed objects. TrackFormer, published in 2021, and MOTR, published in 2022, are examples of this direction. Queries can encode temporal identity and retrieve evidence from later frames, while additional queries discover new objects. Such models may still use explicit assignment during training, so an end-to-end prediction architecture does not necessarily eliminate correspondence optimization from the learning process.

Training temporal representations requires more than presenting unrelated still images. Video supervision can specify which instances persist across frames, allowing identity-sensitive objectives and losses for motion or trajectory continuity. Contrastive learning can encourage observations of the same object to have similar representations while separating different objects. However, visually identical instances cannot always be separated from appearance alone. Longer temporal memory also has costs: storing many image features increases memory use, and irrelevant historical evidence can interfere with current localization. Practical architectures therefore restrict memory, summarize past observations, or select reference frames according to visibility and confidence rather than retaining every frame equally.

Three-dimensional, multisensor, and multicamera tracking

Two-dimensional image coordinates are useful for video analysis but do not directly specify physical motion. A box moving downward in an image may indicate an approaching vehicle, a camera tilt, or an object descending in the scene. Three-dimensional tracking estimates position and often orientation in a metric coordinate system. Camera calibration supplies the geometric relationship between scene points and image measurements, including focal parameters, lens distortion, and, when available, the camera's position and orientation relative to a reference frame.

Different sensors contribute different kinds of evidence. A camera provides rich appearance and category information but does not uniquely determine depth from a single image. Lidar measures ranges along sampled directions and supports three-dimensional localization, although point density varies with distance and sensing configuration. Radar can measure range and radial velocity through Doppler information and can function under some conditions that impair optical imaging. Radar measurements have their own ambiguities and resolution limits; they are not direct substitutes for detailed visual appearance.

Sensor fusion can occur before detection, at the feature level, or after each sensor has produced objects or tracks. Early fusion seeks to exploit complementary raw evidence but requires careful alignment of different data representations. Late fusion combines detections or track estimates and can preserve modularity. In either case, timestamps and coordinate transformations are fundamental. A camera and lidar observing a moving vehicle at slightly different times can produce apparently inconsistent positions even when both measurements are individually accurate. Uncertainty must include calibration and synchronization errors, not just detector confidence.

Tracking from a moving platform also requires an estimate of the platform's own motion. Simultaneous localization and mapping estimates sensor motion and environmental structure, whereas object tracking estimates the changing state of scene entities. The tasks interact: moving objects can corrupt assumptions of a static map, and inaccurate platform motion can create false object velocities. Systems may first compensate for estimated ego-motion and then model residual motion, or jointly estimate static structure, camera pose, and dynamic objects.

Multicamera tracking adds correspondence between viewpoints. When camera fields of view overlap, calibrated geometry and simultaneous observations can support cross-view matching. With non-overlapping cameras, association relies more heavily on appearance, plausible travel times, and scene topology. Similar clothing, lighting differences, and long gaps make such associations uncertain. It is also necessary to distinguish relative identity continuity from actual personal identity: linking two anonymous trajectory segments does not require identifying a person by name. Depending on the application, outputs may be local camera tracks, a shared ground-plane trajectory, or a cross-camera sequence with explicit gaps where no observation exists.

Training data, benchmarks, and generalization

Tracking datasets provide not only images but temporal annotations. A single-object dataset typically specifies the designated target in each frame, sometimes including attributes such as occlusion, scale variation, or camera motion. A multiple-object dataset supplies per-frame locations and persistent instance identifiers. Three-dimensional datasets add coordinate frames, calibration information, and spatial boxes or point annotations. The annotation convention defines what the tracker is being asked to infer; differences between visible extent and estimated full-object extent can materially change both training targets and evaluation results.

The OTB evaluations introduced in 2013 and expanded in 2015 helped standardize single-object tracking comparisons across common visual challenges. The Visual Object Tracking challenge, initiated in 2013, developed protocols emphasizing accuracy and failure behavior. GOT-10k, presented in 2019, emphasized broad object diversity and a one-shot evaluation protocol designed to test transfer to object classes not shared with its training split. LaSOT and TrackingNet supplied larger-scale video resources that supported the shift toward data-intensive learned trackers. These benchmarks differ in sequence length, category coverage, annotation density, and evaluation rules.

For multiple objects, the MOTChallenge benchmark series standardized pedestrian tracking tasks and evaluation. MOT17 offers multiple sets of public detector outputs for the same sequences, while MOT20 emphasizes crowded pedestrian scenes. Such datasets make identity continuity measurable under occlusion and close interactions. Autonomous-driving resources, including KITTI, nuScenes, and Waymo Open Dataset, extend evaluation to moving platforms and three-dimensional environments. Strong results on pedestrian video do not directly establish competence at tracking vehicles from sparse point clouds.

Dataset construction creates several sources of bias. Frequently observed object classes receive more training support than rare ones; urban daytime recordings do not represent every weather condition or lighting environment. Identity annotations can be uncertain during complete occlusion, and ambiguous objects may be ignored according to dataset-specific rules. If frames or near-duplicate sequences appear across training and evaluation partitions, measured performance can overstate generalization. Preventing data leakage requires splitting at a level appropriate to the task, such as source video, scene, subject, or recording session rather than isolated frames.

Data augmentation can simulate scale variation, blur, color changes, occlusion, and missing observations. Synthetic sequences provide exact motion and identity labels, but their appearance and dynamics may differ from real recordings. Self-supervised learning can exploit temporal consistency without exhaustive manual identities, although the inferred correspondences can themselves be wrong. Evaluation under distribution shift is therefore important: a model may perform well on a familiar benchmark yet fail when frame rate changes, a new camera viewpoint appears, or the visual characteristics of the target population differ.

Evaluation metrics and experimental protocols

Tracking evaluation measures several different properties: localization, detection coverage, identity continuity, failure recovery, and computational cost. No single score fully captures all of them. A tracker can produce accurate boxes while repeatedly changing object identifiers, or preserve identifiers while missing many objects. Protocol details also matter. Whether a tracker is restarted after failure, whether future frames are permitted, and whether detector outputs are supplied can change the meaning of a reported result.

Single-object tracking commonly uses center-error precision and overlap-based success. Center-error precision measures the fraction of frames whose predicted target center lies within a specified distance of the reference center. Pixel thresholds depend on image resolution and object scale, which motivates normalized alternatives. Success evaluation measures the fraction of frames for which predicted and reference regions exceed an overlap threshold; averaging across thresholds gives a success-curve score. A tracker that always predicts an oversized box may keep the correct center while obtaining poor overlap, showing why center accuracy alone is incomplete.

Multiple Object Tracking Accuracy, or MOTA, combines missed objects, false positives, and identity switches:

MOTA=1−FN+FP+IDSWGT,\mathrm{MOTA} = 1-\frac{\mathrm{FN}+\mathrm{FP}+\mathrm{IDSW}}{\mathrm{GT}},

where GT\mathrm{GT} is the total number of reference object instances across frames. MOTA can be negative when the counted errors exceed that total. Because missed detections and false positives often dominate numerically, a high MOTA does not necessarily imply strong identity continuity. Multiple Object Tracking Precision, historically paired with MOTA in the CLEAR MOT framework, measures localization quality of matched hypotheses under the benchmark's chosen distance convention.

Identity-based metrics ask how consistently a trajectory is assigned to the correct reference identity. IDF1 is the harmonic mean of identity precision and identity recall, calculated after identity-aware matching between predicted and reference trajectories. HOTA, introduced in work published in 2021, balances detection and association. At a localization threshold α\alpha, its core form is

HOTAα=DetAα AssAα,\mathrm{HOTA}_{\alpha} = \sqrt{\mathrm{DetA}_{\alpha}\,\mathrm{AssA}_{\alpha}},

with results aggregated across localization thresholds. DetA measures detection accuracy, while AssA measures association accuracy for matched detections. Separate components help distinguish a detector improvement from a genuine improvement in temporal identity handling.

Runtime comparisons need similar care. Frames per second depends on hardware, input resolution, object count, detector inclusion, and whether processing is batched. Throughput is not latency: a system processing a batch efficiently may still deliver each result too late for an interactive application. Reported experiments should distinguish network execution from image decoding, sensor transfer, association, and output handling. Evaluation of an operational system may additionally measure track-initiation delay, recovery time after occlusion, and uncertainty quality, since these properties are not fully represented by aggregate benchmark scores.

Deployment constraints, applications, and limits

Practical tracking systems operate inside larger decision processes. In industrial inspection, trajectories can connect observations of the same item across production stages. In microscopy, cell and particle tracks support measurements of movement, division, and interaction. Sports analysis uses player and ball trajectories to describe movement patterns, while film production and augmented displays use tracked points or objects to attach graphics consistently to moving scenes. Each setting changes the acceptable representation and error: a temporary identifier swap may be tolerable for aggregate traffic flow but destructive for reconstructing an individual cell lineage.

Resource constraints influence architecture. A detector may run less frequently while a cheaper motion or appearance tracker fills intervening frames. This reduces computation but makes sudden scene changes harder to handle. Processing on a graphics processing unit can improve throughput, yet transfer overhead and shared workloads affect end-to-end latency. Neural-network quantization reduces numerical precision to lower storage or computation costs, but its effect on localization and association must be measured rather than assumed. Variable numbers of objects also produce variable association and memory costs, especially in dense scenes.

Confidence is difficult to interpret. A detector score, an appearance similarity, and a state covariance quantify different things and should not be treated as interchangeable probabilities. A track can have a highly certain predicted position under its motion model while belonging to the wrong physical object. Systems therefore distinguish measurement confidence, localization uncertainty, association ambiguity, and the probability that a target remains present. During a long occlusion, returning an uncertain or absent result may be more faithful to the evidence than extrapolating a precise trajectory indefinitely.

Tracking people and vehicles raises issues of data privacy because trajectories can reveal routines, relationships, and visits to sensitive locations even without explicit names. Cross-camera linkage increases the amount of behavior that can be reconstructed. Removing an obvious personal identifier does not necessarily make a detailed movement history anonymous. These considerations concern the data lifecycle as well as the tracking algorithm: capture scope, retention, access, linkage, and the distinction between aggregate measurements and persistent individual records all affect what information a deployed system contains.

Some limitations are intrinsic to observation. Complete occlusion removes visual evidence; identical objects can exchange positions in ways that cannot be resolved from a recording; low sampling rates can make several trajectories equally plausible. More powerful models may exploit useful contextual regularities, but they cannot turn unobserved events into direct measurements. A technically meaningful output can therefore include competing hypotheses, explicit uncertainty, and intervals with no asserted correspondence. In scientific and operational uses, preserving that distinction prevents a continuous-looking track from being mistaken for a fully observed history.