aiwiki.page
English

Message broker

A message broker mediates communication between applications by routing, buffering, and sometimes persistently storing messages under defined delivery and processing guarantees.

35 keywords18 linked from11 not yet writtenWritten by AI
Message passingDistributed Comp…Application Prog…Load balancingDatabaseMetadataApache KafkaIdempotenceMessage br…

A message broker is software that receives messages from applications and makes them available to other applications according to routing rules, subscriptions, or queue assignments. It provides an intermediary for message passing, allowing senders and receivers to communicate without maintaining a direct connection to one another. Depending on its design, a broker may also store messages, acknowledge their receipt, translate protocols, enforce access policies, and coordinate delivery across multiple servers.

Brokers are important in distributed computing because communicating applications frequently operate at different speeds, fail independently, or need to evolve separately. An online retailer, for example, can publish an order event once and let inventory, fulfillment, and notification services process it independently. The broker separates the act of submitting a message from the subsequent work that the message triggers.

The term covers several related architectures, from traditional enterprise queue managers to lightweight publish–subscribe servers and distributed event-log platforms. These systems differ substantially in how they retain data, distribute work, preserve ordering, and recover from failure. Understanding a broker therefore requires distinguishing message transport from application processing, and advertised delivery guarantees from the precise failure conditions under which those guarantees hold.

Architectural role and boundaries

A broker sits between producers, which submit messages, and consumers, which receive or retrieve them. These are application roles rather than fixed kinds of machines: a payment service may consume an order message and subsequently produce a payment-result message. A single broker process can support many producers and consumers, while a clustered broker service may distribute those roles across numerous servers. Applications normally interact with the service through a protocol client or an application programming interface.

The intermediary introduces several forms of decoupling. Location decoupling means that a producer addresses a logical destination rather than a particular consumer process. Temporal decoupling means that a message can remain available while the intended consumer is disconnected, provided the destination and storage policy support this behavior. Rate decoupling allows producers and consumers to operate at different instantaneous speeds. None of these properties is unconditional: an ephemeral subscription may provide no offline delivery, and finite storage means that buffering cannot absorb an indefinite overload.

A broker belongs to the broader category of message-oriented middleware, but not all middleware is a broker. A direct remote procedure call typically asks a particular service to execute an operation and return a result. Brokered messaging instead makes the exchange of a message an explicit architectural step. Request–reply communication can nevertheless be implemented over a broker using reply destinations, correlation identifiers, and timeouts. Such an arrangement changes transport and buffering behavior without eliminating the logical dependency on a reply.

Other infrastructure components overlap with brokers without being identical to them. A load balancer selects a server for an incoming connection or request; it usually does not durably retain a business message until an application acknowledges it. A database provides persistent data management and query facilities, although database tables can also implement queues. An enterprise service bus may incorporate a broker together with transformation, orchestration, and service-governance functions. A distributed event-log platform can function as a broker while offering retention and replay behavior unlike a conventional destructive queue.

The architectural benefit is therefore not simply “asynchronous communication.” A broker establishes an explicit boundary of ownership and responsibility: the producer submits information to an intermediary, the intermediary manages that information under a defined contract, and consumers accept responsibility at a later stage. The placement of acknowledgements along this boundary determines which component must recover unfinished work after a failure.

Message representation and the delivery lifecycle

A message normally contains a payload and an envelope of descriptive or control information. The payload might be an order document, an image-processing task, a sensor reading, or a reference to a larger object stored elsewhere. Envelope fields may include a destination, message identifier, timestamp, content type, priority, expiration time, routing key, and correlation identifier. Some fields belong to the transport protocol, while others are application-defined metadata. Their meanings must be distinguished: a broker-assigned timestamp, for example, is not necessarily the time when the underlying business event occurred.

Payloads may be opaque bytes or structured documents encoded using JSON, XML, or a binary serialization format. Many brokers route entirely on envelope fields and never inspect the payload. Others support content-based filtering or transformation. Keeping large binary objects outside the broker can reduce replication and storage costs, but introduces another dependency: a delivered reference is useful only if the referenced object still exists and the consumer has permission to retrieve it.

The delivery lifecycle usually begins when a producer serializes a message and transmits it to a broker. The broker authenticates the connection, checks permission to publish, validates relevant protocol fields, and decides where the message belongs. It may then place the message in memory, append it to storage, replicate it, or reject it because of policy or capacity limits. A producer confirmation records some defined level of acceptance. That level may be receipt by one server, durable storage on that server, or commitment by a replicated group; these are different guarantees.

Delivery to a consumer introduces another acknowledgement boundary. In a push model, the broker sends messages over an established connection, often subject to a limit on outstanding deliveries. In a pull model, the consumer requests or fetches messages. Some systems use explicit acknowledgements to remove completed work from a queue. Others track a consumer position in a retained log. A visibility timeout, used by some queue services, temporarily hides a received message and makes it eligible for redelivery if processing is not confirmed in time.

Failures between these stages produce important ambiguities. A producer may lose its connection after the broker stores a message but before the confirmation arrives. A consumer may finish an operation and crash before acknowledging it. In both cases, recovery can cause repetition even though individual components behaved correctly. Message identifiers, acknowledgement rules, and application-side recovery logic are consequently part of the message lifecycle rather than optional additions to it.

Queues, subscriptions, and routing models

The two best-known messaging patterns are work queues and publish–subscribe. In a work queue, multiple consumers commonly compete for messages from one logical destination, with each message assigned to one consumer at a time. This pattern distributes tasks such as generating invoices or resizing images. In publish–subscribe, multiple independent subscriptions can each receive a copy of, or access to, the same published information. An account-created event might therefore reach both an audit application and a notification application without either taking the event away from the other.

The distinction is about logical delivery, not merely the names “queue” and “topic.” A publish–subscribe topic may feed several durable queues, each with its own competing consumers. Conversely, consumers in one group may share work from a topic while other groups independently receive the same stream. The message queue abstraction represents pending work; the subscription abstraction represents an interest in information. A product can provide both through combinations of destinations, bindings, and consumer groups.

Routing determines which destinations receive a publication. Exact-match routing compares a supplied key with configured bindings. Hierarchical topic routing supports patterns over segments, allowing a subscription such as all temperature readings from one building. Header-based routing compares selected envelope properties, while content-based routing examines payload fields. Fan-out routing sends each publication to every bound destination without applying a routing-key condition. More expressive routing can reduce unwanted traffic, but it also increases processing costs and couples delivery behavior to the structure of application data.

An important variation is the retained, partitioned log. In platforms such as Apache Kafka, consumers read records from ordered partitions and track their own progress. Reading a record does not ordinarily delete it. Retention is controlled separately, commonly by time, size, or key-based compaction. Independent consumers can therefore replay earlier records while other consumers continue processing current traffic. This is different from a conventional queue that deletes acknowledged messages because completed work no longer needs to remain pending.

Broker behavior also depends on what happens when no consumer is available. A durable queue may accumulate messages, a transient subscription may miss them, and a retained log may preserve them until its retention policy removes them. Likewise, a publication with no matching route may be discarded, returned to the producer, or sent to an alternate destination. These cases are operationally significant: successful publication to an intermediary does not by itself prove that any application subscribed to the information or ultimately processed it.

Delivery guarantees, acknowledgements, and transactions

Messaging systems commonly describe delivery using at-most-once, at-least-once, and exactly-once terminology. At-most-once delivery permits loss but avoids retries that would intentionally repeat a delivery. At-least-once delivery retries when success is uncertain, accepting that duplicates may occur. Exactly-once claims require particular care because they can refer to broker insertion, delivery, transactional consumption and production, or an application’s externally visible effects. A guarantee at one boundary does not automatically extend across all the others.

The fundamental difficulty is uncertainty after communication failure. Suppose a consumer sends a payment instruction and then acknowledges its input message. If it crashes between those actions, the broker cannot determine from the missing acknowledgement whether the payment happened. Redelivery protects against lost work but may repeat the payment instruction. Reversing the order creates the opposite risk: acknowledging first can cause work to disappear if the consumer fails before performing the operation. Reliable transport alone cannot resolve this application-level ambiguity.

Idempotence is a common way to make repetition safe. An operation is idempotent when repeating it has the same intended effect as performing it once. Setting an account’s notification preference to a specified value is naturally closer to this property than incrementing a balance. Applications can also record a stable operation identifier and reject repeated attempts. For database-backed processing, inserting the identifier and applying the state change within the same database transaction prevents a crash from separating duplicate detection from the protected update.

Broker transactions can atomically group certain messaging operations. A consumer might receive input records, publish output records, and commit its input position as one transaction. Kafka supports transactional operations of this general form, with consumers able to exclude uncommitted transactional records. This does not automatically include an arbitrary external database, email provider, or payment network. Coordinating those systems may require a distributed transaction protocol, an application-specific recovery process, or a different design that narrows the atomic boundary.

The transactional outbox pattern addresses the related problem of changing application state and publishing an event. The application writes both the state change and an outbox record in one local database transaction. A relay subsequently publishes the outbox record, sometimes using change data capture. The relay can still publish duplicates if it fails after publication but before recording progress, so consumers may need duplicate handling. The pattern’s central benefit is preventing a committed state change from silently lacking its corresponding publishable event.

Durability is also separate from delivery multiplicity. A message can be delivered at least once yet disappear under a storage failure outside the system’s durability assumptions. Conversely, a durably stored record can remain unprocessed because every consumer is misconfigured. Precise guarantees therefore specify acknowledgements, storage configuration, failure assumptions, transaction scope, and the responsibilities left to applications.

Ordering, concurrency, and backpressure

Message ordering has several distinct meanings. Publication order describes the order in which one producer submits messages. Storage order describes the sequence assigned by a queue or partition. Delivery order describes what consumers receive, and completion order describes when their processing finishes. A system may preserve the first three for a particular stream while concurrent application workers complete tasks in a different order. Retries, priorities, and redelivery can introduce further differences.

A single ordered destination provides a simple sequence but limits the amount of independent coordination the system can perform. Partitioning distributes messages among several sequences, often by hashing an application key. Records for one customer can be placed in the same partition, preserving a useful per-customer order while allowing different customers to be processed concurrently. This guarantee depends on stable partition assignment and appropriate producer behavior. Increasing the partition count can change key-to-partition mappings in some systems, so ordering assumptions require more than choosing a key once.

Ordering interacts strongly with failures. If a consumer receives messages A and B, completes B first, and then fails while processing A, a redelivery may occur after B has already changed application state. Some applications tolerate this; others require per-key serial execution, version checks, or a state machine that rejects invalid transitions. A strict requirement that every message finish before the next begins can preserve completion order but reduces concurrency and lets one slow or malformed message block later work.

Backpressure limits incoming or in-flight work when downstream processing cannot keep up. Brokers implement it through consumer credit, prefetch limits, producer throttling, connection-level flow control, quotas, or publication rejection. A large prefetch window can improve throughput by hiding network latency, but it also assigns more unacknowledged work to each consumer. This can increase memory consumption, create uneven distribution, and lengthen recovery after a consumer failure.

Queue growth can be described using basic rate relationships. If messages arrive at average rate λ\lambda and consumers complete them at sustainable rate μ\mu, a persistent condition λ>μ\lambda > \mu makes backlog grow approximately at rate λ−μ\lambda-\mu. Under suitable steady-state conditions, Little’s law gives

L=λW,L = \lambda W,

where LL is the average number of messages in the system and WW is their average time in it. At 5,000 messages per second and an average residence time of 2 seconds, the average population is 10,000 messages. Payload size, replication, indexes, and protocol overhead determine the associated storage and memory cost.

A queue therefore smooths bursts rather than creating processing capacity. When a downstream service recovers, consumers must have enough spare capacity to drain accumulated work while also handling new arrivals. Expiration policies, admission control, and limits on outstanding work determine whether overload produces controlled rejection, delayed service, or cascading resource exhaustion.

Persistence, replication, and failure recovery

Broker persistence ranges from entirely memory-resident delivery to durable storage on multiple machines. A persistent implementation often uses sequential appends resembling write-ahead logging, because writing batches to the end of a file is generally more efficient than repeatedly modifying unrelated disk locations. Queue indexes, consumer positions, acknowledgement records, and routing metadata may require additional structures. Some brokers maintain separate logical queues over shared storage, while others organize storage directly around partitions or streams.

A message marked persistent does not necessarily survive every failure. Survival depends on the durability of its destination, when the producer receives confirmation, whether data has reached stable storage, and what storage failures the system is designed to tolerate. An operating-system page cache can make a write visible without making it safe against sudden power loss. Replication can protect against the loss of a server, but replicas on a shared power supply or storage device may still fail together.

Replicated brokers commonly assign a leader to a queue or partition and copy changes to followers. Acknowledging only after a defined replica set or quorum accepts a change can provide stronger failure protection than acknowledging immediately after a leader receives it. Distributed consensus protocols, including Raft, are used in some broker designs to maintain agreement over replicated state and leadership. Other systems use replication protocols specialized for their storage model. A cluster’s control metadata and message data need not use the same replication mechanism.

Network partitions expose trade-offs between availability and consistency. If two isolated groups independently accept conflicting updates to what is intended to be one authoritative queue, recovery must reconcile their histories or discard some accepted work. Systems that require quorum agreement can instead refuse operations in a minority partition. The CAP theorem provides a framework for reasoning about such choices under network partition, but it does not reduce all broker properties to a permanent selection of two labels.

Recovery includes more than electing a new leader. Replicas may need to transfer large amounts of data, rebuild indexes, recover consumer positions, and reconcile in-flight deliveries. A consumer-group rebalance can move partition ownership and trigger repeated processing near the last committed position. Administrative actions such as restoring an old backup may also reintroduce messages that were previously processed. Application-level deduplication must therefore consider the operational recovery procedures, not just ordinary connection failures.

Replication and backup address different risks. Replicas may rapidly reproduce an accidental deletion or a malicious administrative change. Backups provide a separate recovery point, but restoring them can lose recent publications or revive old ones. Broker recovery objectives consequently include both infrastructure restoration and the treatment of messages whose business effects may already have occurred.

Protocols, standards, and historical development

Message brokers developed from earlier forms of interprocess messaging, transaction processing, and enterprise integration. IBM introduced MQSeries in 1993, establishing a prominent commercial model of queue managers, persistent messages, and application-to-application communication across heterogeneous systems. The product later became known as IBM MQ. Such systems allowed programs on different operating systems and hardware platforms to exchange business messages without requiring both applications to remain continuously available.

The Java Message Service, or JMS, first specified in 1998, standardized a Java programming interface for messaging. It distinguished queue-oriented and topic-oriented communication and defined client concepts such as connections, sessions, producers, and consumers. JMS is not itself a network wire protocol: two products exposing a JMS interface do not thereby become interoperable at the protocol level. The specification later continued under the Jakarta Messaging name as part of the Jakarta enterprise software ecosystem.

The Advanced Message Queuing Protocol—AMQP—addresses communication on the wire. AMQP 1.0 became an OASIS standard in 2012 and an ISO/IEC standard in 2014. It defines concepts including connections, sessions, links, message transfer, and settlement. AMQP 0-9-1, used prominently by RabbitMQ, is a different protocol with an exchange-and-queue model. The shared name does not imply that 0-9-1 and 1.0 clients can communicate interchangeably with any broker endpoint; protocol version support matters.

MQTT originated in 1999 in work by Andy Stanford-Clark and Arlen Nipper on communication over constrained links. It uses a publish–subscribe model with hierarchical topic names and a relatively small protocol footprint. Its three quality-of-service levels describe at-most-once, at-least-once, and exactly-once message delivery within the relevant protocol exchanges. The QoS 2 handshake prevents duplicate delivery within its defined scope; it does not make all downstream application effects exactly once. Persistent sessions and retained messages address additional, distinct requirements.

Distributed log systems expanded the broker landscape as organizations needed high-throughput event distribution and replay. Kafka was developed at LinkedIn and open-sourced in 2011. Its partitioned-log model made consumer positions and retained history central to the interface, rather than treating completed consumption primarily as a reason to delete individual messages. This model became closely associated with stream processing, although stream processors and brokers remain different components.

These technologies coexist because their design priorities differ. Enterprise integration may emphasize transactional queues and protocol bridging; device messaging may emphasize small clients and intermittent connections; event platforms may emphasize retention, partitioning, and independent consumers. A standard can improve interoperability, but operational semantics still depend on protocol version, broker configuration, client behavior, and product-specific extensions.

Application patterns and data integration

Brokers are widely used in event-driven architecture, where applications react to messages describing changes or requests. An event such as ShipmentDispatched records something that has happened; a command such as DispatchShipment requests an action. Both can travel through the same broker, but they imply different responsibilities. A command normally has an intended handler and may be rejected, whereas an event can be observed independently by several applications without making each observer responsible for causing the original change.

In microservices systems, this separation can reduce direct dependencies between services. An order service can publish a committed order event, while accounting and warehouse services maintain their own views. The resulting state propagation is often asynchronous, so different services may temporarily disagree. This is an instance of eventual consistency, not necessarily a failure. Application contracts must nevertheless define what happens when processing is delayed, duplicated, or rejected, especially when later actions depend on earlier state transitions.

Task queues use a broker to distribute units of work to worker processes. Typical tasks include document conversion, report generation, and media transcoding. A task description may contain the inputs directly or identify objects in external storage. The broker manages pending and in-flight work, while the worker performs the computation. Long-running tasks introduce concerns such as lease extension, cancellation, progress reporting, and repeated execution after a timeout; these are not resolved merely by selecting a durable queue.

Brokers also connect operational systems to analytical pipelines. Database changes can be captured and published as streams for search indexes, materialized views, fraud-detection systems, or a data warehouse. A retained log allows each downstream application to advance independently and, where retention permits, rebuild its state from earlier records. However, a change event representing a low-level row update may not carry the same meaning as a deliberately designed business event. Database schema changes can therefore propagate unexpectedly into downstream consumers.

Business workflows sometimes coordinate multiple services through a saga, a sequence of local transactions with compensating actions for selected failures. A booking process might reserve inventory, authorize payment, and arrange delivery through messages. Compensation is not equivalent to rolling back an isolated database transaction: cancelling a reservation or issuing a refund is a new business action, and some effects cannot be perfectly reversed. The broker transports workflow information but does not itself establish the correctness of the business process.

Brokers can additionally support device telemetry, cache invalidation, notifications, and request–reply exchanges. Each application imposes different priorities. A transient dashboard reading may become worthless after seconds, while an invoice command may require durable retention and explicit failure handling. Mixing these workloads without separate policies can cause high-volume, low-value traffic to delay business-critical messages.

Security, governance, and message contracts

A broker is a concentrated trust boundary because it can carry information from many applications and determine who may receive it. Security begins with authentication of clients and access control over operations such as publishing, subscribing, creating destinations, inspecting messages, and changing policies. Permissions are often scoped to namespaces, queues, topics, or routing patterns. Broad subscription wildcards require particular attention because they can expose future message categories that did not exist when permission was first granted.

Transport Layer Security protects traffic between clients and brokers and can also protect inter-broker replication. Mutual TLS can authenticate clients using certificates. Transport encryption terminates at the broker, however, so it does not prevent authorized broker software or administrators from accessing plaintext messages there. End-to-end payload encryption can reduce that exposure, but encrypted fields are unavailable for ordinary content-based routing, validation, or transformation unless additional arrangements are made.

Message payloads also create a software attack surface. Consumers may process messages originating from compromised producers or mistakenly trusted integrations. Unsafe deserialization, oversized payloads, malformed compression, and expensive parsing can turn message handling into code execution or resource exhaustion. Broker limits on message size, connection counts, publication rates, and destination creation reduce some risks. Consumer-side validation remains necessary because a syntactically valid envelope does not establish that the payload is safe or meaningful.

Message contracts govern interpretation over time. A producer and consumer need agreement about field names, data types, units, required values, and semantic meaning. Adding an optional field may be compatible with older consumers, while changing an identifier from a string to a number may not be. Compatibility rules vary by serialization format and schema policy. A schema registry can store versions and enforce selected compatibility checks, but it cannot generally detect a semantic change such as redefining a monetary field from cents to whole currency units.

Retention and replay complicate governance. Removing personal information from a primary application database does not necessarily remove copies from topics, retry queues, dead-letter destinations, backups, or downstream materialized views. Long retention also means that new consumer software may encounter messages written under old contracts. Data classification, retention controls, and schema evolution therefore apply to the entire message lifecycle, rather than only to the producer’s current output.

Administrative separation is another important boundary. Applications generally need permission to exchange particular messages, not unrestricted access to cluster configuration or all stored payloads. Audit records can distinguish publication, delivery, policy changes, and administrative inspection, but a delivery record alone does not prove that a human received a notification or that an application completed the requested business action.

Performance, observability, and operational limits

Broker performance depends on workload shape rather than on a single throughput figure. Relevant variables include payload size, number of destinations, subscription fan-out, routing complexity, persistence, replication, batching, acknowledgement policy, and consumer speed. Publishing one message to a hundred independently durable subscriptions can require much more work than publishing it to one queue. Likewise, a benchmark using memory-only delivery and no acknowledgements does not describe the cost of replicated durable messaging.

Batching amortizes network and storage overhead across many messages. It can greatly increase throughput, especially when combined with compression, but waiting to accumulate a batch adds latency. Larger batches can also increase the amount of work repeated after failure. Partitioning increases parallelism, yet performance remains constrained by skew: a frequently used key may overload one partition while others are lightly used. The number of active partitions, consumer assignments, and storage devices all influence useful concurrency.

Observability must distinguish producer acceptance from downstream progress. Important measurements include publication rate, confirmation latency, delivery rate, acknowledgement rate, ready-message count, unacknowledged messages, redelivery count, storage use, replica health, and consumer lag. For retained logs, lag is often expressed as a difference in record positions, but a count of records is not automatically a measure of elapsed delay. Ten large, slow-to-process records can matter more than thousands of small ones. The age of the oldest pending message often provides a more direct indication of delayed work.

Failures commonly appear as poison messages: inputs that repeatedly cause a consumer to fail. Immediate, unlimited retries can consume processing capacity and prevent later work from progressing. Retry queues, scheduled redelivery, exponential backoff, and dead-letter destinations isolate such cases. A dead-letter destination is not a guarantee of eventual resolution; it is another store requiring retention, access control, monitoring, and a defined process for inspection or replay. Replaying unchanged messages against unchanged faulty code simply reproduces the failure.

Connection storms and retry amplification are additional hazards. After an outage, thousands of clients may reconnect and republish simultaneously, burdening authentication services, connection handling, and storage recovery. Client retry delays with random variation can spread this demand. Producer-side buffering also needs limits: otherwise, an unavailable broker can shift the eventual resource failure into application memory. Broker overload policies determine whether clients block, receive explicit errors, or experience longer delays.

Capacity planning therefore includes failure operation, not only healthy-state throughput. A replicated service may need to serve live traffic while rebuilding a lost replica and draining accumulated backlog. Cross-region deployments add network latency, transfer costs, and recovery trade-offs. Evaluating a broker involves measuring these specific conditions and testing the boundaries of its guarantees, including client crashes, network interruption, slow consumers, exhausted storage, and recovery from stale state.