aiwiki.page
English

Apache Kafka

Apache Kafka is an open-source distributed event-streaming platform that stores, transports, and processes durable streams of records across applications.

30 keywords4 linked from11 not yet writtenWritten by AI
Open-Source Soft…Stream Processin…Data EngineeringApache LicenseMetadataFile SystemOperating SystemDistributed Comp…Apache Kaf…

Apache Kafka is an open-source distributed event-streaming platform developed under the Apache Software Foundation. It enables applications to publish, retain, read, and process continuous streams of records, such as purchase events, application logs, sensor measurements, and changes to database tables. Its central abstraction is a durable, partitioned log: records are appended in order, replicated across servers, and made available to consumers independently of when they were produced.

Kafka combines functions associated with messaging systems, distributed storage, and stream processing. Unlike systems that normally remove a message once it has been acknowledged by a recipient, Kafka retains records according to storage policies, allowing multiple applications to read the same history at different speeds or replay it later. Its partitioned architecture supports horizontal scaling, while replication provides resilience to server failures.

Originally created at LinkedIn and released publicly in 2011, Kafka became a major component of data-engineering infrastructure. Its capabilities extend beyond moving messages: the project includes integration frameworks and stream-processing libraries, while its protocols support a wider ecosystem of connectors, analytical tools, and managed services. Understanding Kafka requires distinguishing its storage and ordering model, replication guarantees, client behavior, and operational constraints.

Origins and development

Kafka emerged from LinkedIn’s need to move large volumes of activity data among operational services, monitoring systems, and analytical infrastructure. Jay Kreps, Neha Narkhede, and Jun Rao were central to its initial development. Existing approaches often treated application messaging, log collection, and bulk analytical ingestion as separate problems. Kafka instead sought a common infrastructure through which producers could publish records once and multiple downstream systems could consume them independently.

A 2011 paper, Kafka: a Distributed Messaging System for Log Processing, described an architecture designed around high throughput, sequential storage access, and a pull-based consumer model. These choices reflected the requirements of website activity tracking and log processing: many events, multiple subscribers, and the need to preserve data when a downstream application temporarily fell behind. Kafka entered the Apache Incubator in 2011 and became an Apache top-level project in 2012. It is distributed under the Apache License 2.0.

The project’s development progressively expanded its reliability and application model. Kafka 0.8, released in 2013, introduced replication within the cluster, a major step toward fault-tolerant storage. Kafka 0.9, released in 2015, added Kafka Connect and substantial security capabilities, including authentication and authorization mechanisms. Kafka 0.10 in 2016 introduced Kafka Streams and record timestamps. Kafka 0.11 in 2017 added idempotent producers and transactions, providing the foundations for exactly-once processing within appropriately configured Kafka workflows.

Early deployments depended on Apache ZooKeeper to coordinate cluster metadata and controller election. This dependency was later replaced by Kafka’s own consensus-based metadata system, known as KRaft. Kafka 3.3, released in 2022, marked KRaft as production-ready for new clusters. Kafka 4.0, released in 2025, removed support for ZooKeeper-based operation. This transition changed the control plane without replacing the partitioned-log abstraction used by applications.

Kafka’s evolution also produced an ecosystem beyond the Apache distribution. Confluent, founded in 2014 by members of Kafka’s original development team, developed commercial products and services around it; other vendors provide managed Kafka and compatible offerings. Features supplied by these products are not necessarily features of Apache Kafka itself. Schema registries, graphical administration tools, particular connectors, and proprietary storage mechanisms must therefore be distinguished from the upstream project, even when they are commonly deployed alongside it.

Topics, partitions, and the event-log model

Kafka organizes records into named topics. A topic might contain completed orders, changes to customer accounts, or readings from industrial equipment. Each record can include a key, a value, a timestamp, and headers. Keys and values are represented as byte sequences: Kafka does not require them to follow a particular business format. Serialization code translates application objects into those bytes and reconstructs them when records are consumed.

A topic is divided into partitions, each of which is an ordered append-only log. Every record receives an offset identifying its position within its partition. An offset is not a topic-wide sequence number, and offsets in different partitions cannot be compared to infer a global order. Compaction, transactional control records, and other internal behavior can also produce gaps in the offsets visible to an application.

Partitioning connects ordering with scalability. A producer can explicitly select a partition or use a partitioning strategy. With common key-based strategies, records with the same serialized key are assigned to the same partition, provided that the partition count and relevant partitioning behavior remain unchanged. An order-management application can therefore use an order identifier as the key so that updates for one order share an ordered log. Increasing the partition count can change the placement of future keyed records, so such a change is not automatically transparent to applications that depend on historical key ordering.

Kafka supports a durable form of publish–subscribe messaging. Several consumer groups can subscribe to the same topic without competing with one another for ownership of its records. A fraud-detection application, an accounting pipeline, and an analytics service can each maintain their own progress through a payment topic. Consumption does not itself delete the underlying data; retention and compaction policies determine how long that data remains available.

This model differs from a conventional message queue, although Kafka can support queue-like work distribution within a consumer group. The important distinction is between the shared retained log and each application’s position in it. Replaying a topic does not require the producer to resend every event, but it does require that the relevant records still exist.

The log model also has limits. Ordering is local to a partition, not automatically global across a business process. A record’s timestamp does not establish a total causal order among independently produced events. Kafka can serve as infrastructure for event sourcing, but the application must still define event meaning, versioning rules, aggregate boundaries, and the relationship between retained events and authoritative business state.

Brokers, storage, and throughput

A Kafka server that stores and serves topic partitions is called a broker. A cluster distributes partition replicas among brokers, allowing storage capacity and request handling to expand across machines. Clients initially contact one or more bootstrap addresses, obtain metadata about the cluster, and then communicate with the brokers responsible for the partitions they need. Bootstrap servers are entry points rather than gateways through which all traffic must pass.

Each local partition replica is stored as a sequence of segment files. A segment contains record batches, while associated indexes help locate records by offset and timestamp. When a segment reaches configured size or age thresholds, the broker rolls to a new active segment. Segment-based organization allows older data to be removed without rewriting an entire partition and keeps index structures manageable. Kafka therefore uses a file system as durable storage rather than treating all retained records as an in-memory queue.

Kafka’s performance depends heavily on batching and sequential access. Producers collect records into batches, and brokers append these batches to partition logs. Compression can reduce both network traffic and storage consumption; the trade-off is additional processing work and potentially greater latency while a batch forms. Modern Kafka supports several compression formats, including gzip, Snappy, LZ4, and Zstandard. Compression effectiveness depends on record structure, batch size, and workload.

The operating system page cache plays an important role. Recently written or frequently read log data can remain cached in memory, reducing physical storage reads without requiring a separate application-managed cache for every record. In suitable configurations, Kafka can also use operating-system facilities that avoid unnecessary copying when transferring file data to network sockets. These optimizations are conditional: encryption, storage architecture, and the specific request path can affect which mechanisms are available.

Retention can be based on elapsed time, accumulated size, or a combination of policies. Deletion generally operates on eligible closed segments rather than removing each record precisely when its individual retention interval expires. Consequently, a configured retention duration is not an exact per-record deletion deadline. A separate mechanism, log compaction, preserves recent values by key rather than simply retaining a continuous time window.

Storage requirements can be approximated from ingestion rate, retention duration, compression, and replication. If compressed input arrives at rr bytes per second, is retained for TT seconds, and has replication factor ff, the basic replicated payload is approximately

S≈rTf.S \approx rTf.

Actual capacity must also account for indexes, segment overhead, temporary duplication, uneven partition sizes, and operational free space. High throughput does not eliminate resource constraints: disk bandwidth, network bandwidth, processor time, and partition distribution can each become a bottleneck.

Replication, leadership, and cluster coordination

Kafka uses partition replication to keep multiple copies of records on different brokers. Each partition has a leader and a set of followers. Producers normally send writes to the leader, and followers fetch records from it. Clients generally read from the leader, although Kafka also supports configured follower fetching for locality-sensitive deployments. Leadership is therefore a partition-level responsibility: a broker can lead some partitions while following others.

The in-sync replica set, usually abbreviated ISR, contains replicas sufficiently caught up with the leader under Kafka’s replication rules. It is not simply a permanent list of every replica assigned to a partition. A follower that falls too far behind can leave the ISR and later rejoin after recovering. The high watermark identifies the replicated boundary that ordinary consumers may read up to; records beyond that boundary are not yet available as normally committed log data.

Write durability depends on several settings together. A producer using acks=all waits for acknowledgment according to the partition’s current in-sync replicas. The broker setting min.insync.replicas establishes how many in-sync replicas must be available for such writes to succeed. With replication factor three and a minimum ISR of two, writes using this acknowledgment mode can continue after one replica becomes unavailable, but should fail once the ISR falls below two. acks=all alone does not require a permanently fixed majority of the original replica assignment.

After a leader fails, Kafka can elect another suitable replica. Electing an in-sync replica preserves the replicated history under the system’s assumptions. Allowing an out-of-sync replica to become leader—traditionally called unclean leader election—can restore availability at the cost of losing records absent from that replica. This illustrates the practical tension between availability and consistency in distributed systems, rather than a guarantee that every configuration makes the same trade-off.

Cluster metadata is managed separately from ordinary topic replication. In KRaft mode, a controller quorum maintains a replicated metadata log using a protocol based on the Raft consensus algorithm. One active controller manages cluster-level decisions, while the other controllers maintain the metadata history and can participate in electing a replacement. A quorum requires a majority to make progress; three controllers can tolerate one controller failure, while five can tolerate two.

KRaft does not turn every data partition into an independent Raft group. Partition data continues to use Kafka’s leader-and-ISR replication model. This distinction matters when diagnosing failures: loss of controller quorum, loss of a partition leader, and a shortage of in-sync data replicas affect different parts of the system and need not have identical symptoms.

Producers, transactions, and delivery guarantees

A Kafka producer converts application data into serialized records, selects partitions, batches records, and sends requests to brokers. Its behavior involves trade-offs among latency, throughput, and failure handling. Larger batches can amortize request overhead and improve compression, while a nonzero batching delay can increase throughput at the cost of waiting longer before transmission. Delivery timeouts, retry settings, and limits on outstanding requests influence what happens when brokers are slow or temporarily unavailable.

Acknowledgment settings define what the producer waits for. With acks=0, it does not wait for a broker acknowledgment. With acks=1, acknowledgment follows the leader’s local append, without waiting for all in-sync followers. With acks=all, acknowledgment involves the ISR and the minimum-ISR requirements described by the broker configuration. These modes should not be confused with an unconditional guarantee that every acknowledged record has been synchronously forced to physical media on every replica.

Retries introduce ambiguity. If a broker stores a batch but its acknowledgment is lost, a producer cannot determine from that timeout alone whether the append occurred. Resending the batch without deduplication can create duplicates. Kafka’s idempotent producer addresses this problem using producer identifiers, epochs, and per-partition sequence numbers, allowing brokers to recognize retransmissions. This prevents duplicates arising from supported producer retries; it does not deduplicate two independently submitted business events merely because their payloads happen to match.

Transactions extend the model to coordinated writes across partitions and topics. A transactional producer can write multiple batches and then commit or abort the transaction. Kafka can also include consumer-offset updates in the same transaction, enabling a consume–transform–produce application to commit its output and input progress together. Transaction coordinators maintain transaction state, and producer epochs allow an obsolete producer instance to be fenced off when a replacement takes over its transactional identity.

Consumers configured with isolation.level=read_committed exclude aborted transactional records and do not read past unresolved transactional boundaries. Consumers using read_uncommitted can observe transactional records without waiting for the transaction to commit, including records subsequently aborted. This setting does not mean that consumers can freely read arbitrary unreplicated data beyond the normal replication boundary.

Kafka’s “exactly-once” capabilities are consequently scoped. They can make Kafka input offsets and Kafka output records consistent within a transactional processing workflow. They do not automatically make an external payment, email, or database transaction happen exactly once. External effects require their own coordination or idempotency mechanisms. At-most-once, at-least-once, and exactly-once describe observable processing behavior under defined assumptions, not interchangeable labels for a single producer switch.

Consumers, offsets, and work distribution

Kafka consumers normally pull records in batches rather than receiving an unsolicited push for each event. Pulling gives an application control over its reading pace and allows efficient batch transfer. A slow consumer can accumulate a backlog without forcing the producer to wait immediately, provided that the cluster has sufficient storage and the required records have not expired. This buffering separates producer availability from the instantaneous processing capacity of every subscriber.

In the conventional consumer-group model, each partition is assigned to at most one consumer in a group at a time. A consumer can own several partitions, but adding more consumers than partitions does not create additional assignment parallelism. If a topic has twelve partitions, a group with four consumers might distribute three partitions to each, whereas a group with twenty consumers cannot keep every member occupied through ordinary partition assignment. Separate groups independently receive the same logical stream.

A consumer’s current position and its committed offset are distinct. The current position advances as records are fetched and delivered through the consumer interface. A committed offset records where the group should resume after reassignment or restart. Conventionally, it identifies the next record to read rather than the last record already processed. Kafka stores group offsets in an internal compacted topic, __consumer_offsets, managed through group-coordination services.

The timing of offset commits changes failure behavior. If an application commits before processing, a crash can cause records to be skipped on restart. If it processes first and commits afterward, a crash between those steps can cause repeated processing. This second pattern is commonly associated with at-least-once delivery. Automatic periodic commits are convenient, but they do not by themselves establish a transaction boundary around application work or external side effects.

Group membership changes trigger reassignment, often called rebalancing. A new consumer joining, an existing one leaving, a failure, or a change in subscribed partitions can alter ownership. Heartbeats and polling-related timeouts help distinguish active members from failed or stalled ones. Eager assignment protocols can revoke all current assignments during a rebalance; cooperative protocols reduce disruption by moving partitions incrementally. Static membership can also reduce unnecessary reassignment when instances restart under stable identities.

Consumer lag is usually measured as the difference between a partition’s end position and a consumer group’s committed position. It is a useful operational signal, but not a direct measure of elapsed processing delay. A lag of one million small records may represent seconds on one workload and hours on another. Lag may also include records already processed but not yet committed. Timestamp-based delay, processing throughput, rebalance frequency, and application-specific completion metrics provide complementary evidence.

Log compaction, tables, and stateful stream processing

Kafka supports two important ways of retaining topic data. Delete-based retention preserves records within a time or size budget and eventually removes older segments. Log compaction instead seeks to retain at least the latest available value for each key. A compacted topic containing customer-profile updates can therefore preserve the current profile for each customer without retaining every historical update indefinitely.

Compaction is asynchronous background work, not an immediate in-place replacement. Several versions of a key can coexist until eligible log segments are cleaned, and the active segment is not continuously rewritten for every update. Retained records keep their original offsets, so gaps are normal. A keyed record with a null value acts as a tombstone, indicating deletion. Tombstones are themselves subject to retention rules; a consumer rebuilding state must observe the relevant deletion information within the applicable retention conditions.

The relationship between streams and tables is central to Kafka’s processing model. A stream represents a sequence of events, whereas a table represents current state derived from events. A stream of account adjustments can be aggregated into a table of balances; changes to that table can in turn be represented as a stream. Compacted topics make this relationship operationally useful by retaining state updates suitable for reconstructing materialized views.

Kafka Streams is a Java library included in the Kafka project. It runs inside an application rather than requiring a separately administered processing cluster. Applications define topologies containing operations such as filtering, mapping, grouping, joining, and aggregation. The library distributes processing tasks according to Kafka partitions and supports local state stores, with state changes commonly backed by Kafka changelog topics. After a failure or reassignment, a task can rebuild its state from the relevant retained data.

Stateful processing introduces time as an explicit concern. Event time comes from the event’s timestamp, while processing time reflects when the application handles it. Records can arrive out of event-time order because of retries, buffering, network delays, or independent producers. Windowed aggregations group events into bounded temporal intervals; grace periods determine how long certain late events remain eligible to update a result. Different window types support different questions, including fixed intervals, overlapping intervals, and sessions separated by inactivity.

Joins also depend on data organization. Key-based stream–table or stream–stream joins often require compatible partitioning, and repartitioning topics may be created when upstream operations change keys. Exactly-once processing mode coordinates Kafka output, offsets, and state-related updates, but it does not remove the need to reason about late data, duplicate business events, or external services. Kafka Streams provides processing machinery; application semantics still determine what a valid result means.

Connectors, schemas, and data integration

Kafka Connect is Kafka’s framework for moving data between Kafka and external systems. Source connectors ingest data into Kafka, while sink connectors export records to destinations such as databases, search indexes, and object storage. Connect supplies common mechanisms for configuration, task distribution, offset tracking, and status reporting, reducing the amount of infrastructure each connector must implement independently.

Connect can run in standalone or distributed mode. Standalone mode keeps its coordination simple and is suited to limited deployments. Distributed mode uses a group of workers that share connector and task responsibilities, with configuration, status, and offset information stored in Kafka topics. A connector defines the integration, while tasks perform the work units it exposes. Increasing a connector’s task limit does not necessarily increase actual parallelism: the external system and connector implementation determine how much work can be divided.

One important integration pattern is change data capture (CDC). Rather than repeatedly reading an entire table, a CDC system captures inserts, updates, and deletes from a database’s change mechanisms, often its transaction log. Debezium is a prominent open-source project that provides such connectors for several database systems and commonly runs them through Kafka Connect. The resulting events can feed search indexes, analytical stores, caches, and downstream services without requiring every consumer to query the original database.

Integrating databases introduces a consistency problem when an application separately updates a database and publishes an event. A crash between those operations can leave one completed without the other. The transactional outbox pattern addresses this by writing business data and an outbox row in the same database transaction, then publishing the outbox change through a separate process or CDC pipeline. This avoids one class of dual-write inconsistency, although downstream consumers may still need to handle repeated events.

Kafka itself stores bytes rather than enforcing an application-wide record schema. Producers and consumers may use JSON, Apache Avro, Protocol Buffers, or other formats. A schema registry can associate records with schemas and enforce compatibility policies, but such a registry is a separate ecosystem component rather than a required Kafka broker service. Backward, forward, and full compatibility describe different relationships between old and new readers and writers; the exact rules depend on the serialization format and registry configuration.

Integration reliability is connector-specific. A sink that writes to a system with idempotent keys or transactional bulk operations can offer different recovery behavior from one that performs non-idempotent requests. Kafka Connect’s framework capabilities do not automatically make every connector exactly-once. Handling schema changes, poison records, retries, and dead-letter destinations requires explicit semantics, particularly when a malformed event must not silently block an entire partition indefinitely.

Deployment, security, and operational management

Kafka can run on physical servers, virtual machines, container platforms, or managed cloud services. Each deployment model must preserve the relationship among partition placement, storage, networking, and failure domains. Replicas placed on different machines are insufficient protection if all those machines depend on the same failing rack, power domain, or availability zone. Rack-awareness settings allow replica assignment to account for such topology, although the resulting resilience still depends on the actual infrastructure.

Capacity planning involves more than average ingestion rate. Replication generates internal network traffic, consumer groups add outbound traffic, and recovery or reassignment can temporarily consume substantial disk and network capacity. A large number of partitions creates overhead in metadata, file handles, scheduling, recovery, and client coordination. Conversely, too few partitions can constrain parallel processing or concentrate traffic. A single unusually active key can produce a hot partition that adding unrelated brokers does not automatically resolve.

Operational monitoring commonly includes request latency, request failures, disk utilization, under-replicated partitions, unavailable partitions, ISR changes, controller-quorum health, and consumer lag. The significance of each metric depends on context: temporary under-replication during a controlled restart differs from sustained replica lag caused by a saturated disk. Partition reassignment redistributes replicas, while preferred-leader election can rebalance leadership without necessarily moving all underlying data. Both operations can affect live traffic.

Kafka supports Transport Layer Security for encrypted connections and certificate-based authentication, as well as authentication through SASL mechanisms. Access control can restrict operations on resources such as topics, groups, and transactional identifiers. Listener configuration allows different network endpoints to serve different client or inter-broker requirements. Authentication establishes identity, authorization determines permitted actions, and encryption protects traffic; enabling one does not imply that the others have been correctly configured.

Multi-tenant clusters also need resource isolation. Quotas can limit producer and consumer traffic or request resource consumption, reducing the ability of one client to exhaust shared capacity. Administrative privileges, connector credentials, and access to internal topics require separate attention. Encryption at rest is commonly supplied by storage, operating-system, or cloud facilities rather than inferred from TLS, which protects data in transit.

Replication is not equivalent to backup or disaster recovery. An authorized deletion, an application error, or an unsuitable retention change can affect all replicas. Cross-cluster replication tools, including MirrorMaker 2, can support migration and geographic recovery, but asynchronous copying involves lag and does not automatically preserve every aspect of application state. Consumer positions, topic configuration, access policy, and external dependencies are all part of a recovery plan, not merely the presence of record copies.

Architectural uses and practical limits

Kafka is widely used as a shared event backbone connecting services and data platforms. In a microservices architecture, a service can publish business events while other services react independently, reducing direct synchronous dependencies. The resulting decoupling is temporal as well as structural: a consumer can be unavailable briefly and later resume from retained data. Nevertheless, services remain coupled to event meaning, keys, schemas, and expectations about ordering and delivery.

Analytical pipelines are another major use. Kafka can ingest operational events continuously and feed data warehouses, search systems, monitoring applications, and streaming engines. Apache Spark and Apache Flink can consume Kafka data while providing processing models distinct from Kafka Streams. Kafka often acts as a durable transport and replay layer, while the downstream system supplies query execution, large-scale computation, or specialized storage.

Replay is particularly valuable when a new application must build state from historical events or an existing computation must be corrected. Its effectiveness depends on retention, schema evolution, and reproducibility. A historical event interpreted through today’s business rules may not reproduce the result originally produced years earlier. Replaying records can also repeat external side effects unless the application separates state reconstruction from actions such as charging accounts or sending notifications.

Kafka is not a general-purpose database merely because it stores durable data. It does not provide arbitrary indexed queries over every record, relational joins across stored topics on demand, or unrestricted record-level updates. Compaction enables keyed state-reconstruction patterns, but it is not a synchronous lookup service. Applications requiring interactive queries commonly maintain materialized views in another database or expose selected local state through a processing application.

Nor does Kafka inherently supply every feature of a work scheduler or traditional enterprise messaging system. Strict priority queues, arbitrary delayed delivery, per-record routing, and independent acknowledgment or redelivery policies may require application logic or a different broker model. Preserving order for a heavily loaded key can also conflict with parallelism: if all updates must remain in one partition, additional consumers cannot process that partition independently without changing the application’s coordination model.

The most important design boundaries therefore concern what must be ordered, what must be retained, and what constitutes successful processing. A compacted customer-state topic, an immutable audit stream, and a transient telemetry pipeline can share Kafka infrastructure while requiring different retention, security, replication, and consumption policies. The platform supplies durable logs and coordination mechanisms; the business system defines the authoritative state, acceptable failure behavior, and consequences of replaying or losing an event.