{
  "id": "change-data-capture",
  "title": "Change Data Capture",
  "lang": "en",
  "source_lang": "en",
  "summary": "Change data capture identifies and records database changes so other systems can incrementally update their own data or respond to those changes.",
  "category": "computer-science",
  "subcategory": "software",
  "type": "concept",
  "aliases": [
    "CDC"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/change-data-capture/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/change-data-capture/",
      "json": "https://aiwiki.page/zh/change-data-capture.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Change data capture (CDC) is a set of techniques for identifying and recording changes to data in a database and making those changes available to other systems. Typically, it captures insertions, updates, and deletions, allowing downstream applications to process changes incrementally rather than repeatedly read an entire dataset. CDC is used in data integration, database replication, analytics, and event-driven applications. It describes a capability rather than a single protocol or implementation: different systems capture changes through transaction logs, database triggers, queries, or database-provided change feeds."
    },
    {
      "heading": "Purpose and basic model",
      "text": "An operational database stores the current state of an application, but other systems often need copies, derived representations, or notifications of changes. A data warehouse may require updated sales records, a search engine may need to refresh indexed documents, and an application cache may need to invalidate obsolete entries.\n\nA full extract reads all relevant data on each run. CDC instead identifies what has changed since a previously recorded position or observation. This can reduce repeated scanning and data transfer, especially when the changed portion is small relative to the total dataset. The benefit depends on workload: a bulk update affecting most rows may still produce a large change stream.\n\nA typical CDC pipeline has four functions:\n\n- Capture: detect changes at the source.\n- Representation: encode changes as records or events.\n- Transport: deliver those records to consumers, directly or through an intermediate service.\n- Application: update a destination or perform downstream computation.\n\nThese functions may be combined in one product or distributed across several components. CDC does not inherently mean continuous streaming. Changes may be delivered with low latency, in small batches, or through periodically executed queries."
    },
    {
      "heading": "Capture techniques",
      "text": "Transaction-log-based capture\n\nMany database systems maintain logs to support recovery and replication. Log-based CDC reads these records, or a database-supported logical representation of them, to obtain changes without repeatedly scanning application tables. Such logs are often associated with write-ahead logging, although their formats and exposed interfaces vary.\n\nPhysical log records describe low-level storage changes. Logical decoding or equivalent mechanisms translate relevant records into operations on tables and rows. Depending on the database and configuration, output can include row identifiers, changed values, transaction boundaries, and commit positions.\n\nA CDC reader must distinguish committed work from changes belonging to aborted or incomplete transactions. Implementations may buffer records until commit, emit transaction markers for downstream interpretation, or use another database-specific mechanism. A reliable stream intended to represent committed state must not expose rolled-back changes as durable updates.\n\nLog-based capture often imposes less query overhead than repeated table scans, but it is not free. It can require additional logging, replication privileges, retained log storage, and resources for decoding or buffering large transactions. Available row images and supported operations depend on database settings.\n\nTrigger-based capture\n\nA database trigger can write a record to a change table whenever an application modifies a monitored table. If the trigger executes within the modifying transaction, the change record can commit or roll back with the original operation.\n\nTriggers can capture custom information and work where suitable log interfaces are unavailable. Their main disadvantages are additional work on the write path, extra storage and maintenance, and the need to manage trigger definitions as tables evolve. Bulk-loading mechanisms, disabled triggers, or operations outside their configured scope can create gaps.\n\nQuery-based capture\n\nQuery-based CDC periodically examines source tables for evidence of change. Common approaches use a modification timestamp, a monotonically increasing version, or a database-maintained change-tracking value.\n\nThis approach can be relatively simple, but its correctness depends on the meaning of the chosen marker. A timestamp assigned before transaction commit can cause missed records if a later polling boundary advances past a transaction that has not yet become visible. Timestamp precision, clock behavior, and multiple rows sharing the same marker also require attention.\n\nOrdinary polling of current rows cannot detect physical deletions unless a separate deletion record, soft-delete flag, or database-supported tracking mechanism exists. It also generally cannot reconstruct intermediate updates that occurred between polls.\n\nSnapshot comparison and native change feeds\n\nAnother approach compares successive snapshots using keys, row values, or hashes. It can identify differences between observed states, but not every intermediate operation. A row inserted and deleted between snapshots may leave no trace.\n\nSome databases expose native change tables, change streams, or change-feed interfaces. These can simplify access, but their guarantees differ. A feed may contain every supported row-level operation, only the latest state for each changed key, or a restricted history available for a limited retention period."
    },
    {
      "heading": "Change-event representation",
      "text": "A change record commonly contains:\n\n- The source database, table, or collection.\n- An operation type, such as insert, update, or delete.\n- A row identifier, often a primary key.\n- New values and, where available, previous values.\n- A transaction identifier and source-log position.\n- Timing information and other metadata.\n- A schema identifier or version.\n\nAn update may contain a complete row image, only modified fields, or both “before” and “after” values. These formats are not interchangeable. A partial update requires the consumer to interpret absent fields correctly; absence does not necessarily mean a null value.\n\nDelete records often contain only a key, although some configurations provide previous column values. In certain messaging systems, a separate tombstone record requests removal of a key from compacted storage. A tombstone is a transport or storage convention, not a universal synonym for a database deletion.\n\nSource timestamps also require interpretation. They may represent transaction start, commit, capture, or publication time. A source position is usually more suitable than an ordinary wall-clock timestamp for resuming a particular log stream."
    },
    {
      "heading": "Initial snapshots and continuous capture",
      "text": "A newly created destination generally needs the existing dataset as well as future changes. CDC pipelines therefore commonly combine an initial snapshot with ongoing change capture.\n\nThe difficult part is establishing a consistent boundary between the snapshot and the stream. Reading tables first and starting capture afterward can miss changes made during the read. Starting capture first without coordinating the snapshot can produce duplicates or apply changes in the wrong relationship to snapshot rows.\n\nImplementations address this through mechanisms such as:\n\n- Taking a transactionally consistent snapshot associated with a known log position.\n- Retaining and buffering changes while the snapshot is read.\n- Coordinating incremental snapshot chunks with recorded stream boundaries.\n\nThe required guarantees depend on whether consistency is needed for individual rows, entire tables, or multiple related tables. A snapshot that is valid for each table independently may not represent a single consistent database state across all tables.\n\nIncremental snapshotting can reduce the need for a long-running database snapshot, but it introduces reconciliation work between scanned rows and concurrent changes. Long-running snapshots or retained logs can also consume substantial source resources."
    },
    {
      "heading": "Delivery and consistency guarantees",
      "text": "CDC pipelines are distributed systems: source databases, capture processes, transport services, and destinations can fail independently. Correctness therefore depends on more than detecting changes.\n\nOrdering and transaction boundaries\n\nA source log may provide an ordered sequence within one database or log partition. That does not establish a global order across independent databases, shards, or transport partitions.\n\nPer-key ordering may be enough to maintain the latest value of each record. Other applications require transaction-level consistency. If a source transaction changes several rows, applying those events separately can temporarily expose a destination state that never existed as a committed state at the source.\n\nPreserving source ACID properties at the destination requires appropriate transaction metadata and destination-side application logic; a sequence of row events alone does not provide this automatically.\n\nDuplicates and replay\n\nMany pipelines use at-least-once delivery: after a failure, some records may be delivered again rather than risk losing them. Consumers can handle this through idempotent operations, deduplication, or source-position checks.\n\nFor example, assigning an absolute account balance can be idempotent, whereas incrementing a balance by an amount is not. Even an idempotent assignment can produce the wrong result if an older event overwrites a newer value, so replay handling also needs suitable ordering or version checks.\n\nEnd-to-end exactly-once effects require coordination between consuming an event, changing destination state, and recording progress. A guarantee provided by an intermediate transport does not automatically cover arbitrary external side effects.\n\nCheckpoints and recovery\n\nA checkpoint records a position from which processing can resume. Recording progress before destination changes are durable can cause data loss; recording it afterward without atomic coordination can cause duplicates after a crash.\n\nRecovery also depends on source retention. If the source removes log records before a stalled consumer reads them, resuming from the last checkpoint may no longer be possible. Reinitialization or another recovery procedure may then be necessary.\n\nAsynchronous CDC usually creates a period of eventual consistency between source and destination. Low latency does not mean that every source commit is immediately visible everywhere."
    },
    {
      "heading": "Applications and related concepts",
      "text": "CDC is widely used in data engineering to feed warehouses, data lakes, and other analytical stores. It can support incremental extraction and loading while leaving transformation to downstream systems. In stream processing, change events become inputs to continuously maintained aggregates, joins, and alerts.\n\nOther applications include maintaining search indexes, refreshing materialized views, propagating selected changes between services, and supporting database migration. During migration, an initial copy followed by CDC can keep a new system close to the source before a coordinated cutover.\n\nCDC and replication overlap but are not identical. Replication aims to maintain copies of data, while CDC identifies and exposes changes. Replication may use CDC as an input, and a CDC stream may instead feed transformations or notifications that do not reproduce the source.\n\nCDC also differs from event sourcing. In event sourcing, an event sequence is the authoritative record from which application state is derived. CDC typically observes changes to an already authoritative database. A row update may show that an order status changed without revealing the full business intent, command, or causal context.\n\nThe transactional outbox pattern connects database changes with application-level messaging. An application writes business data and an outbox record in the same transaction; CDC can then publish the committed outbox records. This avoids an uncoordinated dual write to a database and a message system, but downstream delivery still needs its own duplicate-handling and recovery semantics."
    },
    {
      "heading": "Operational limitations and security",
      "text": "Schema evolution and unsupported operations\n\nSchema changes can alter the interpretation of existing streams. Adding or renaming columns, changing data types, modifying keys, and dropping tables may require coordinated changes to event schemas and consumers. Support for data-definition operations, table truncation, large objects, and specialized data types varies by implementation.\n\nTables without stable keys are harder to synchronize reliably. A destination may be unable to identify the intended row for an update or deletion without additional identity information.\n\nLoad, latency, and retention\n\nA pipeline's sustainable throughput must accommodate change volume and bursts. Large transactions may require substantial buffering before they can be emitted or applied. Slow consumers create backlog, potentially increasing transport storage or forcing the source to retain logs.\n\nOperational measurements commonly include source-to-capture lag, end-to-end application lag, backlog size, retained-log volume, error rates, and checkpoint age. These measure different parts of the pipeline; a responsive capture process does not establish that a destination is current.\n\nPrivacy, access, and auditability\n\nCDC can move sensitive information beyond the source database's original security boundary. Previous row images may retain values that have already been changed or removed at the source. Destination deletion, retention, and privacy obligations therefore need separate treatment.\n\nSecurity controls include restricted access, encryption, field filtering, and appropriate retention policies. Operational lineage metadata can improve data lineage and troubleshooting.\n\nA CDC stream is not automatically a complete audit trail or a backup. It may omit earlier history, unsupported operations, actor identity, queries that read data, or the context needed to explain a business action. Its suitability for reconstruction or audit depends on explicitly retained information and verified capture coverage."
    }
  ],
  "links": [
    "database",
    "database-replication",
    "data-warehouse",
    "search-engine",
    "write-ahead-logging",
    "database-transaction",
    "primary-key",
    "metadata",
    "database-schema",
    "distributed-computing",
    "acid-properties",
    "idempotence",
    "eventual-consistency",
    "data-engineering",
    "data-lake",
    "stream-processing",
    "data-privacy",
    "access-control",
    "encryption",
    "data-lineage",
    "data-backup"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}