{
  "id": "stream-processing",
  "title": "Stream Processing",
  "lang": "en",
  "source_lang": "en",
  "summary": "Stream processing continuously computes over arriving data, using time, state, and recovery mechanisms to produce timely results from potentially unbounded inputs.",
  "category": "computer-science",
  "subcategory": "software",
  "type": "concept",
  "aliases": [
    "Data stream processing",
    "Streaming data processing"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/stream-processing/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/stream-processing/",
      "json": "https://aiwiki.page/zh/stream-processing.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Stream processing is a computing approach in which data is processed incrementally as it becomes available, rather than only after it has been collected into a complete dataset. Its inputs commonly represent events: financial transactions, sensor measurements, website interactions, application logs, or changes to database records. A stream-processing application may transform individual events, maintain continuously updated aggregates, correlate multiple sources, or detect patterns extending across time.\n\nThe approach is important wherever the usefulness of a result depends on how quickly it follows the underlying activity. Examples include operational monitoring, fraud detection, live analytics, and the continuous maintenance of derived data. Although stream processing is often associated with low latency and big data, neither extremely high volume nor instantaneous response is a defining requirement. Its central characteristic is sustained computation over arriving data, often without a predetermined endpoint.\n\nModern systems combine ideas from databases, distributed computing, and incremental algorithms. Their central difficulties arise from the interaction of time, mutable state, parallel execution, and failure: events may arrive late or out of order, machines may fail after partially completing work, and an apparently final result may need revision when additional information arrives."
    },
    {
      "heading": "Computational model and boundaries",
      "text": "A stream can be modeled as a sequence of records, each containing a payload and, commonly, a key, timestamp, and source position. The payload describes the observation or change; the key identifies an entity such as an account or device; the timestamp places the event in a temporal context; and the source position identifies its location in an input sequence. These fields serve different purposes. A source offset can support recovery without accurately describing when an event occurred, while an event timestamp can support temporal analysis without uniquely identifying the record.\n\nThe principal distinction from batch processing concerns the treatment of the input. A batch job operates on a bounded collection whose extent is known, such as all files assigned to a daily report. A streaming computation is designed to continue accepting input, potentially indefinitely. Boundedness is not the same as size: a trillion-record archive is bounded, whereas a slowly reporting temperature sensor may produce an unbounded stream. Many modern engines implement both forms using related execution machinery.\n\nA stream-processing program is often expressed as a graph of operators. A filter selects records, a map transforms their contents, a grouping operation routes records with the same key together, and an aggregation maintains a value such as a running count. Unlike a one-time query over a static table, a continuous query remains active and produces outputs as its inputs evolve. Some outputs are append-only facts; others are updates or retractions of earlier results. This distinction becomes essential when later arrivals change an aggregate or invalidate a previously reported condition.\n\nStream processing overlaps with, but is not identical to, several neighboring approaches. Event-driven architecture concerns how software components communicate and react to events; an event-triggered service need not perform continuous analytics. Complex event processing emphasizes the identification of meaningful event patterns, often using temporal constraints. Reactive programming provides abstractions for propagating changes and coordinating asynchronous work, but it can operate entirely within one process. Likewise, streaming audio or video delivery primarily concerns continuous media transport and playback rather than analytical computation over event records.\n\n“Real time” requires additional qualification. A streaming system may usually respond within milliseconds but still provide no guaranteed maximum delay. Hard real-time systems require deadlines whose violation constitutes failure, a stronger property associated with carefully bounded scheduling and resource behavior. Distributed stream-processing services generally emphasize measured latency, availability, and recovery rather than hard timing guarantees."
    },
    {
      "heading": "Historical development",
      "text": "Stream processing developed from several traditions rather than from a single invention. Telecommunications systems processed call records and network measurements incrementally, industrial systems monitored continuously changing signals, and financial applications maintained rapidly updated views of market activity. Database research supplied the idea of a standing query that remains registered and processes subsequent updates. Work on active databases and event-condition-action rules also explored computations initiated by changes rather than by repeated explicit queries.\n\nDuring the late 1990s and early 2000s, research on data-stream management systems formalized the problems created by unbounded input and limited memory. The Stanford STREAM project studied continuous queries and developed the Continuous Query Language, commonly called CQL. Its formulation distinguished streams from time-varying relations and described operators that translate between them. Aurora, introduced in research publications in 2002, emphasized dataflow networks, operator scheduling, and quality-of-service requirements. Its successor Borealis investigated distributed operation and the processing of revisions, among other issues.\n\nLarge-scale data infrastructure subsequently changed the engineering context. MapReduce, described by Jeffrey Dean and Sanjay Ghemawat in 2004, popularized fault-tolerant parallel processing of bounded datasets. However, repeatedly executing large batch jobs was poorly suited to applications needing frequent updates. Distributed event-processing systems such as Apache Storm, publicly released in 2011, provided long-running topologies that processed individual events and recovered work after failures. Durable messaging systems, notably Apache Kafka, helped separate event production from consumption and made replay an ordinary operational capability.\n\nAnother influential direction used small batches as the unit of streaming execution. Spark Streaming, introduced in a 2013 research paper on discretized streams, represented a stream as a sequence of short, fault-tolerant batches. This approach reused ideas from Apache Spark and made it easier to combine streaming with existing distributed data-processing operations. It also illustrated that continuous service behavior does not require every record to be scheduled independently.\n\nResearch on time and correctness helped unify these approaches. Google’s MillWheel paper, published in 2013, described fault-tolerant stream processing with persistent state and low-watermark mechanisms. The 2015 Dataflow Model paper distinguished event time from processing time and articulated trade-offs among correctness, latency, and cost. Apache Beam subsequently provided a programming model influenced by this work, while Apache Flink became a prominent example of an engine combining stateful streaming, event-time processing, and checkpoint-based recovery. These developments shifted attention from merely processing events quickly toward specifying what results mean under disorder, replay, and failure."
    },
    {
      "heading": "Execution architecture and event transport",
      "text": "A typical deployment consists of sources, a processing graph, and sinks. Sources acquire records from durable logs, message brokers, network connections, files, or database change feeds. Processing operators transform and combine those records. Sinks write results to databases, analytical stores, object storage, monitoring systems, or further event streams. The separation permits independent scaling and maintenance, but each boundary introduces questions about buffering, ordering, authentication, and failure handling.\n\nA durable event log differs from a transient communication channel because it preserves records for subsequent reading. Consumers maintain positions within the log and can resume from those positions after restarting. Several applications may independently process the same history, and a new application may reconstruct its state by replaying retained events. Retention policies still impose limits: a log that removes old records after seven days cannot by itself support reconstruction requiring a month of history. Compaction, which retains selected records by key, provides different historical guarantees from ordinary time-based retention.\n\nParallelism usually depends on partitioning. An input topic or stream is divided into partitions, and processing tasks consume different partitions concurrently. A key-based shuffle routes all events for a particular key to the same logical operator instance, allowing that instance to maintain a coherent account balance, device status, or session history. Ordering is ordinarily local to a partition or key rather than global across an entire distributed system. Creating a single total order can introduce a bottleneck and is unnecessary for many applications.\n\nExecution engines differ in scheduling granularity. Record-at-a-time engines pass events through continuously running operators, whereas micro-batch engines periodically schedule work over newly available data. Operator chaining can place adjacent transformations in the same task, avoiding serialization and network transfer. Exchanges between tasks provide opportunities for parallelism but require buffers, communication, and coordination. Consequently, increasing the number of workers does not automatically reduce latency: overhead can outweigh the benefits when each record requires little computation.\n\nA further architectural concern is backpressure, the propagation of downstream capacity limits toward upstream producers or operators. If a sink can accept only 20,000 records per second while upstream tasks produce 30,000, the excess must accumulate, be discarded, or cause upstream work to slow. Bounded queues and flow-control protocols prevent unlimited memory growth, while durable logs can absorb temporary imbalances. They cannot eliminate a sustained capacity deficit. Stream processing therefore depends not only on computation speed but also on coordinated control of the flow of work through the entire pipeline."
    },
    {
      "heading": "Time, disorder, and window semantics",
      "text": "A stream-processing system commonly distinguishes three notions of time. Event time describes when an event occurred according to its source or assigned business meaning. Ingestion time describes when infrastructure first accepted it. Processing time is the clock time at which an operator handles it. A mobile device might record a purchase at 10:00, upload it at 10:20 after reconnecting, and have it processed at 10:21. A report of purchases during the 10:00 hour generally requires event time; a measurement of processing-system activity may instead use processing time.\n\nBecause an unbounded input cannot generally be aggregated to completion, many computations use windows. A tumbling window partitions time into nonoverlapping intervals, such as successive five-minute periods. A sliding window may cover the preceding hour and advance every minute, placing a record in multiple windows. A session window groups activity separated by gaps no larger than a specified threshold; a newly arrived event may bridge two previously separate sessions. Windows can also be based on record counts or application-specific boundaries rather than clock time.\n\nThe challenge is determining when sufficient input has arrived. A watermark is a statement about progress in event time. Informally, a watermark at time \\(w\\) indicates that the system does not expect ordinary future input with timestamps earlier than \\(w\\). Whether that statement is a strict guarantee or a heuristic depends on the source and implementation. A watermark is therefore not simply the current wall-clock time. In multi-input operators, progress often depends on the slowest relevant input, with explicit handling needed for idle partitions.\n\nTriggers determine when a window emits results. An application may produce an early estimate every second, emit another result when a watermark passes the window boundary, and issue corrections for late records during an additional retention period. Allowed lateness controls how long relevant window state remains available for such corrections. Discarding very late data, routing it to a separate output, or revising historical results are distinct policies with different meanings. None follows automatically from the use of event-time timestamps.\n\nFor example, a five-minute sales window may initially report 100 transactions and later change to 103 when delayed devices reconnect. An append-only output containing both numbers is ambiguous unless consumers know whether the second replaces the first. Systems therefore distinguish accumulating outputs, incremental updates, and retractions. Exact results, low delay, and finite resource use cannot all be obtained without assumptions about arrivals: an arbitrarily late record can always challenge a previously closed event-time result."
    },
    {
      "heading": "Stateful computation, joins, and incremental views",
      "text": "A stateless operation can process each record independently. A stateful operation depends on information retained from previous records: a running total, the last observed device condition, an incomplete event pattern, or records waiting for a matching join input. State may be local to an operator or partitioned by key. Its lifetime and size are central design properties. A count per active account can remain modest, while retaining every historical event for every account grows without bound unless the application defines an expiration or compaction rule.\n\nIncremental aggregation avoids recomputing a result from all prior data. For a running sum, an arriving value \\(x\\) changes state from \\(s\\) to \\(s+x\\). An average requires both a sum and a count, not merely the previous average. Aggregates differ in how easily they support combination and removal. Counts and sums can be updated when a record enters or leaves a sliding window; exact medians require richer structures. Associative aggregation enables partial results to be computed independently and then combined, an important foundation for parallel computing.\n\nJoins require especially careful state management. A stream–table join enriches each event using a stored relation, such as adding customer information to an order. A stream–stream join buffers records from both inputs until potential matches arrive. If transactions and confirmations can match only within ten minutes, time bounds and watermarks can eventually justify deleting unmatched records. An unrestricted join between two unbounded streams may require retaining arbitrarily large histories because any old record might match a future one.\n\nTemporal joins introduce a further distinction: matching against the latest known dimension value is not equivalent to matching against the value valid when the event occurred. An order placed before a customer changed address may need the earlier address even if processing occurs afterward. Supporting such semantics requires versioned state and a defined relationship between event timestamps, update validity, and retained history. This is one reason streaming joins cannot be understood solely as ordinary joins executed more frequently.\n\nContinuous queries are closely related to materialized views. A materialized view stores a query result and updates it as underlying data changes. In a streaming representation, insertions, deletions, and updates may be encoded as a changelog. A grouped total then changes through deltas rather than full recomputation. Connecting a database through change data capture extends this model beyond append-only events: record updates and deletions become first-class inputs. Correct maintenance depends on interpreting those changes consistently, including primary keys and transaction boundaries where they matter."
    },
    {
      "heading": "Failure recovery and processing guarantees",
      "text": "Failure recovery must coordinate two forms of progress: which input records have been consumed and what state those records have produced. Recording only input positions can lose state changes; recording only operator state can cause input to be applied again. A coherent recovery point associates source positions with a consistent snapshot of the computation. Checkpointing periodically creates such recovery points, allowing execution to restart without replaying the entire available history.\n\nDistributed snapshots require coordination because records may be in transit between operators. Some engines inject barriers into input streams to mark checkpoint boundaries. An operator receiving multiple inputs may align their barriers so that its snapshot reflects a consistent cut through the computation. Other designs capture in-flight data or use alternative coordination strategies. Alignment can increase delay under backpressure, whereas capturing in-flight buffers can increase checkpoint size. The right approach depends on throughput, state size, network behavior, and recovery objectives.\n\nProcessing guarantees describe behavior under retries and failures, but their scope matters. At-most-once handling permits records to be lost rather than retried. At-least-once handling retries records when completion is uncertain, so duplicates are possible. Exactly-once processing generally means that each logical input has one effective contribution to the specified state or output, even if the underlying computation executes more than once. It does not mean that networks never duplicate messages or that a processor physically touches each record only once.\n\nExactly-once state recovery is easier to establish than exactly-once effects across arbitrary external systems. If an operator updates its checkpointed state and separately sends an email, a crash between those actions creates uncertainty: repeating the input may send the email twice, while skipping it may omit the message. Coordinated transactions, transactional sinks, idempotent writes, or deduplication using stable event identifiers can address particular cases. An idempotent operation has the same effect when repeated, such as setting a row to a specified versioned value rather than blindly incrementing it.\n\nGuarantees also depend on the source. If a device emits a measurement twice without a stable identifier, a processor may have no reliable way to distinguish a duplicate from two genuine measurements. Conversely, replay from a durable source with stable positions permits precise recovery even when workers fail repeatedly. Operational descriptions therefore need to identify the boundary of the guarantee: internal operator state, a particular sink, or an entire application. Recovery correctness must also account for nondeterministic code, changing external lookups, and software upgrades, all of which can make replay differ from the original execution."
    },
    {
      "heading": "Programming models and query languages",
      "text": "Stream-processing interfaces range from low-level event handlers to declarative queries. At the low level, a program can receive each event, read and modify keyed state, register timers, and emit outputs. This gives detailed control over temporal logic and application-specific behavior, but it places more responsibility on the programmer to bound state, manage event ordering, and avoid incorrect interactions with recovery. Higher-level interfaces provide operators for grouping, windowing, joining, and pattern recognition.\n\nA dataflow application programming interface represents computation as connected transformations. The engine can inspect this graph to determine where repartitioning is required, which operations can be chained, and where state must be managed. User-defined functions supply application logic inside a framework-controlled execution model. This differs from a collection of independently written services whose coordination exists mainly in message formats and deployment configuration. The explicit graph gives an optimizer or runtime more information about the whole computation.\n\nSQL offers a declarative alternative. A streaming SQL query can continuously group incoming purchases by region or join transactions with a reference table. However, ordinary SQL syntax does not by itself define all streaming semantics. Implementations must specify time attributes, window definitions, update behavior, and the treatment of unbounded joins or ordering. A query asking for every record sorted by timestamp cannot emit a final, globally ordered unbounded result unless additional assumptions or boundaries make progress possible.\n\nThe relationship between streams and tables provides a useful semantic foundation. An append-only stream can describe insertions into a relation; a changelog can describe an evolving relation with updates and deletions. Conversely, changes in a continuously maintained table can be emitted as a stream. This allows relational reasoning to extend to continuous applications, but it also exposes differences among output modes. A query producing only new rows is easier to send to an append-only file sink than one that repeatedly revises previous rows.\n\nPortability frameworks such as Apache Beam separate the application model from the execution engine, often called a runner. Portability does not imply identical performance or universal support for every feature. Timer behavior, state facilities, connectors, and operational tooling may differ. Other systems favor close integration with one runtime. Across these approaches, static checking and query planning can identify some invalid operations, but important correctness properties remain semantic: whether a timestamp represents the intended event, whether a join interval matches the business rule, and whether downstream consumers interpret revisions correctly."
    },
    {
      "heading": "Streaming algorithms and resource bounds",
      "text": "Stream processing is both a systems discipline and an area of algorithm design. The theoretical data-stream model asks what can be computed in one or a few passes using memory much smaller than the input. This setting is stricter than merely running a conventional algorithm continuously. An indefinitely growing input cannot generally be stored in full, and some exact answers require memory proportional to the number of distinct values or records encountered.\n\nApproximate summaries address this limitation. A Count–Min Sketch, introduced by Graham Cormode and S. Muthukrishnan in 2005, uses several hash functions and arrays of counters to estimate item frequencies. In its standard nonnegative-update setting, collisions cause overestimation rather than underestimation, with probabilistic bounds controlled by the sketch dimensions. HyperLogLog, published in 2007, estimates the number of distinct elements using compact register summaries derived from hashed values. Such structures are valuable when exact per-item state would be prohibitively large.\n\nReservoir sampling maintains a uniform sample of a stream when its final length is unknown. For a reservoir of size \\(k\\), a standard algorithm retains the first \\(k\\) records and, for record \\(n>k\\), admits it with probability \\(k/n\\), replacing a uniformly selected reservoir entry if admitted. After \\(n\\) records, each record has probability \\(k/n\\) of being included. The procedure illustrates how a carefully designed incremental algorithm can preserve a statistical property without retaining the full input.\n\nResource planning also relies on basic queueing relationships. If records arrive at average rate \\(\\lambda\\) and remain in a stable system for average duration \\(W\\), Little’s law gives average occupancy \\(L=\\lambda W\\). Thus, at 100,000 records per second and an average residence time of two seconds, roughly 200,000 records are in the system on average. This does not specify peak memory, because record sizes, burstiness, retained state, and serialization overhead also matter. It does show why additional latency often implies additional buffered data.\n\nKey skew can dominate performance even when aggregate capacity appears sufficient. If one customer or device accounts for a large fraction of traffic, routing all records for that key to one task creates a hotspot. Partial aggregation, specialized handling of heavy keys, or splitting combinable computations can help, but arbitrary stateful logic cannot always be divided safely. Similarly, a bounded event rate does not guarantee bounded state: a million new keys per hour can steadily increase memory use even if each key appears only once. Expiration policies and approximate representations therefore belong to the computational semantics, not merely to infrastructure tuning."
    },
    {
      "heading": "Applications and integration patterns",
      "text": "Operational monitoring is a common application because incoming measurements are useful before a complete historical dataset exists. A pipeline can calculate error rates over recent intervals, correlate failures with deployments, and detect changes in service latency. The choice of clock matters: processing-time metrics describe what the monitoring infrastructure is observing now, while event-time metrics can reconstruct when the underlying failures occurred. Delayed telemetry may make those views differ substantially.\n\nIn financial and commercial systems, streaming computations maintain running features, identify unusual sequences, and correlate transactions with account history. A payment-risk application might combine the current transaction with recent counts by account and device. Anomaly detection can then use rules, statistical thresholds, or a trained model. The stream-processing engine is responsible for timely feature updates and reproducible event handling; it does not by itself establish whether a model is accurate or whether an alert represents actual misconduct.\n\nIndustrial and sensor applications use streaming to combine high-frequency measurements, detect threshold crossings, and calculate rolling statistics. Processing near the data source can reduce communication requirements, while central processing can correlate activity across sites. Disconnected devices and unreliable links make delayed and out-of-order arrival normal. A local control loop with strict deadlines is generally a different engineering problem from a distributed analytical pipeline, even when both consume measurements from the same equipment.\n\nContinuous data integration is another major use. Database changes can update search indexes, analytical tables, caches, and downstream services without repeated full extraction. A transaction log may become the source for several independently maintained representations. Correctness then depends on handling deletions, schema changes, and initial snapshots consistently with subsequent changes. If a snapshot and a live feed overlap, the pipeline needs a defined handover procedure to avoid missing or double-applying records.\n\nTwo architectural patterns describe broader combinations of historical and continuous computation. The Lambda architecture maintains separate batch and speed-processing paths, with the batch path providing comprehensive recomputation and the speed path supplying recent results. Its cost is duplicated logic and the need to reconcile outputs. The Kappa architecture emphasizes a streaming path whose state can be rebuilt by replaying retained input. Replay-based designs reduce one form of duplication but require sufficient history, compatible schemas, and enough processing capacity to catch up. Neither pattern removes the need to define event semantics, correction policies, and operational ownership."
    },
    {
      "heading": "Operations, evolution, and practical limits",
      "text": "A production stream-processing service must be observed as a continuously changing system rather than merely as a job that eventually finishes. Important measurements include input rate, output rate, processing latency, event-time lag, consumer backlog, checkpoint duration, checkpoint failure rate, and state size. These are not interchangeable. A service may process each newly received event quickly while remaining hours behind a durable input log, or it may be current in processing time while event-time progress is stalled by one source.\n\nLatency measurements require explicit endpoints. Producer-to-sink latency includes delays outside the processing engine, whereas operator execution time covers only part of the path. Percentiles reveal behavior hidden by averages: a low mean can coexist with severe tail delays during checkpointing, garbage collection, or bursts of input. Watermark lag is also distinct from record latency because it measures progress assumptions rather than the completion time of any one event.\n\nScaling stateful applications is more complex than adding stateless workers. When parallelism changes, keyed state must move to new owners, and in-flight records must reach the correct tasks. A large state transfer can temporarily increase latency or lengthen recovery. Checkpoint intervals create another trade-off: frequent snapshots consume storage and network resources but can reduce the amount of work replayed after failure. Incremental snapshots can reduce repeated transfer of unchanged state, although their effectiveness depends on the state backend and workload.\n\nSoftware and schema evolution introduce long-lived compatibility obligations. Events retained for months may have been written using older field definitions, while a restored checkpoint may contain an earlier state representation. Renaming an operator, changing key selection, or modifying a serialized data type can affect recovery even if the new application compiles successfully. Testing consequently needs histories containing duplicates, delayed arrivals, reordered events, source interruptions, and failures near commit boundaries. Replay testing is particularly useful for comparing old and new logic, provided external inputs are controlled.\n\nRetention and replication also have governance consequences. Event logs, checkpoints, and derived state may preserve personal or sensitive information in several places. Access control, encryption, deletion procedures, and data privacy requirements must apply across those copies rather than only to the primary database. A deletion in a source table does not automatically remove an earlier event from a retained log or historical checkpoint. At the same time, retaining less history can restrict auditability and reconstruction. These constraints directly shape the available recovery strategy, state model, and permissible replay behavior."
    }
  ],
  "links": [
    "big-data",
    "database",
    "distributed-computing",
    "event-driven-architecture",
    "mapreduce",
    "apache-kafka",
    "apache-spark",
    "message-broker",
    "parallel-computing",
    "change-data-capture",
    "database-transaction",
    "sql",
    "algorithm",
    "anomaly-detection",
    "access-control",
    "data-privacy"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}