Messaging and streaming

In synchronous microservice architectures, when service A calls service B directly over HTTP or gRPC, the calling thread blocks until the downstream service finishes processing. If service B experiences latency or goes offline, service A quickly exhausts its thread pool, triggering cascading outages across the entire application stack.

Introducing an asynchronous messaging layer decouples services in both time and space. The producer writes an event or task payload to a managed message broker and immediately resumes processing user traffic. Downstream consumers pull or receive messages at their own pace. If a consumer crashes or is taken down for deployment, messages accumulate safely within the broker until the service recovers.

Cloud providers offer three primary messaging abstractions: queues, pub/sub topics, and event streams.

Queues

A provides point-to-point task buffering where each message is delivered to and processed by exactly one worker. For example, when a user requests an export, the web API enqueues a task and returns an immediate confirmation. A background worker picks up the message, compiles the report, and acknowledges the job by deleting the message.

When a worker retrieves a message, the queue hides it from other consumers for a configurable duration known as a visibility timeout. If the worker completes its processing and deletes the message, the task is finished. If the worker crashes, runs out of memory, or encounters a fatal error, the visibility timeout expires and the message becomes visible again for another worker to process.

This mechanism guarantees . The broker ensures that no message is lost, but intermittent network partitions or worker crashes can result in duplicate deliveries. Consequently, queue consumers must be engineered to be idempotent: processing the same message payload multiple times must produce the identical system state as processing it once.

Queues also deliver essential load-leveling capabilities. A sudden surge of ten thousand requests is absorbed instantly by the queue, allowing downstream database workers to process tasks at a steady, sustainable rate without exhausting database connection pools.

Topics

While queues deliver messages to a single worker, a broadcasts each published event to multiple independent subscribers simultaneously. This architecture—known as (pub/sub)—allows systems to emit business state notifications without coupling the producer to downstream consumers.

When an order_placed event is published to a topic, it can be fanned out simultaneously to inventory, invoicing, email notification, and fraud detection services. Introducing a new consumer requires subscribing to the topic without modifying upstream code. In production systems, topics typically deliver into dedicated subscriber queues, providing each downstream service with its own isolated buffer and retry policies.

Streams

A organizes data into an append-only, distributed commit log. Unlike message queues that delete items upon consumption, event streams retain all published events across a configurable time window (ranging from days to weeks).

Multiple consumer groups maintain independent cursor positions (offsets) within the log, reading events sequentially at their own pace. Because data is immutable and persistent, consumers can replay historical event streams from any previous timestamp or offset. This makes event streaming the standard foundation for event-sourcing architectures, real-time analytics pipelines, and audit trails.

Streams guarantee total message ordering within individual partitions. Using a consistent partition key (such as an account ID or device UUID) ensures that all events for a given entity land on the same partition, preserving strict chronological ordering while scaling cluster throughput across multiple partitions in parallel.

Mechanism	Point-to-Point Queue	Pub/Sub Topic	Distributed Event Stream
Delivery topology	One-to-one: exactly one consumer processes each message	One-to-many: fan-out delivers each message to all active subscribers	Many-to-many: independent consumer groups read from shared partitions
Message ordering	Best-effort standard delivery; strict order in FIFO queues	Best-effort delivery order	Strict sequential ordering guaranteed within each partition
Storage & retention	Ephemeral: deleted immediately upon successful processing	Ephemeral: dropped after delivery to subscriber endpoints or queues	Persistent: retained across configurable time windows (days or weeks)
Historical replay	Unsupported; once consumed or expired, messages are gone	Unsupported; messages deliver to current active subscribers only	Supported; consumers can rewind offset cursors to replay historical events
Primary architectural role	Background task execution, rate leveling, service decoupling	Event broadcasting, asynchronous notifications, domain event fan-out	Event sourcing, continuous stream analytics, audit logging, data sync

Failure and dead-letter queues

When a message payload is corrupt, contains unexpected data, or triggers an unhandled application exception, automated retries will fail repeatedly. Without intervention, a failing message (a "poison pill") will loop indefinitely, consuming worker resources and blocking valid messages behind it.

To resolve this, queues and subscriptions route unprocessable messages to a (DLQ). When a message exceeds a configured maximum receive count, the broker automatically transfers it to the DLQ and continues normal processing. Operations teams configure automated alarms on DLQ depth to investigate underlying application bugs, release fixes, and redrive messages back into the primary queue.

Writing a queue consumer

Guarantee idempotency by validating unique message identifiers before executing side effects.

1. Retrieve the message payload and extract its unique message identifier or idempotency key.
2. Query the persistent store or cache to check whether this identifier has already been processed.
3. If the identifier exists, delete the message from the queue and exit immediately.
4. Execute the business logic, persisting the identifier and the result within the same transactional boundary if possible.
5. Acknowledge and delete the message from the queue upon successful completion.

When to use asynchronous messaging

Use direct synchronous calls when the caller requires immediate data to proceed (such as authenticating a user password or confirming live inventory during checkout).

Use asynchronous messaging when work can execute in the background, when operations take longer than several hundred milliseconds, or when multiple independent systems need to react to domain state changes. Relying on direct HTTP calls for multi-step workflows creates tight coupling that inevitably leads to cascading outages when any downstream dependency degrades.

Terms introduced

• Queue: a buffer that delivers each message to one consumer and redelivers it if the consumer fails.
• At-least-once delivery: the promise that no message is lost, at the cost that some arrive twice.
• Topic: a channel that delivers each message to every subscriber.
• Publish-subscribe: the pattern where producers send to a topic and never know who receives.
• Stream: an ordered, replayable log of events with a retention period, read by consumers at their own pace.
• Dead-letter queue: where a message goes after it has failed every retry, so it stops blocking the others.

---

How providers do it

Asynchronous messaging primitives—point-to-point queues, publish-subscribe topics, and distributed event streams—are supported across AWS, Azure, and Google Cloud. However, each provider structures its product portfolio differently, with Google Cloud unifying queuing and pub/sub into a single integrated architecture.

Concept	AWS	Azure	Google Cloud
Point-to-point queue	Amazon SQS	Azure Service Bus queues; Azure Queue Storage	Cloud Pub/Sub subscription
Pub/Sub topic	Amazon SNS	Azure Service Bus topics (with subscriptions)	Cloud Pub/Sub topic
Distributed event stream	Amazon Kinesis Data Streams, Amazon MSK	Azure Event Hubs (with Kafka endpoint)	Google Cloud Managed Service for Apache Kafka
Dead-letter queue (DLQ)	Secondary SQS queue configured via redrive policy	Built-in dead-letter sub-queue	Cloud Pub/Sub dead-letter topic
Processing lock timeout	Visibility Timeout	Message Lock Duration	Acknowledgment Deadline (ack deadline)
Message ordering guarantees	SQS FIFO queues (ordered by Message Group ID)	Service Bus Sessions	Cloud Pub/Sub ordering keys
Historical message replay	Not supported on SQS/SNS; Kinesis replays within retention	Not supported on Service Bus; Event Hubs replays within retention	Cloud Pub/Sub Seek (rewind subscription to timestamp/snapshot)
Managed event bus	Amazon EventBridge	Azure Event Grid	Google Cloud Eventarc

Every product name and technical mapping above is confirmed against provider documentation. Specific maximum payload limits and service tier lifecycles are marked unconfirmed in the provider tabs below.

A distinctive architectural model exists in Google Cloud Pub/Sub:

• In AWS, SQS (queuing) and SNS (pub/sub topics) are separate services; implementing a fan-out pattern requires explicitly provisioning an SNS topic and subscribing individual SQS queues to it.
• In Google Cloud Pub/Sub, the topic and subscription model is natively integrated: publishers always write to a topic, and consumers read from individual subscriptions. A single subscription behaves exactly like a point-to-point worker queue, while attaching multiple subscriptions provides fan-out broadcasting without managing separate queuing services.

AWS

What this maps to: Amazon SQS for queues and Amazon SNS for topics. Streams are Amazon Kinesis Data Streams or Amazon MSK (Managed Streaming for Apache Kafka). Amazon EventBridge is an event bus that routes between all of them.

Concept	On AWS	Status
Queue	SQS. Standard queues are at-least-once with best-effort order. FIFO queues are exactly-once within a deduplication window and ordered within a message group, at lower throughput	confirmed
Redelivery	A message read but not deleted reappears after the visibility timeout	confirmed
Dead-letter queue	Another SQS queue, named in the source queue's redrive policy with a maximum receive count	confirmed
Topic	SNS. Subscribers are SQS queues, Lambda functions, HTTP endpoints, email, or SMS. SNS to SQS is the usual fan-out	confirmed
Stream	Kinesis Data Streams, partitioned into shards, retained for a period you set. MSK for Kafka with its own vocabulary	confirmed
Event bus	EventBridge, with rules matching on event content and routing to targets. Also the source for scheduled events	confirmed
Function trigger	Lambda polls SQS and Kinesis through an event source mapping; SNS and EventBridge push	confirmed
Message size	A hard limit per message, with an extended client library that stores larger payloads in S3	unconfirmed; check current quotas

Their vocabulary

Standard term	Their term
Redelivery timeout	Visibility timeout
Move to the dead-letter queue after N failures	Redrive policy, maxReceiveCount
Partition	Shard (Kinesis), partition (MSK)

Where to look

The SQS console shows how many messages are available, in flight, and sitting in the dead-letter queue. For alarms, the CloudWatch metric you want is ApproximateAgeOfOldestMessage.

Last verified: never.

Azure

What this maps to: Azure Service Bus for queues and topics, Azure Event Hubs for streams, Azure Event Grid as the event bus, and Storage queues for a cheap simple queue inside a storage account.

Concept	On Azure	Status
Queue	A Service Bus queue. Ordered within a session if you use sessions, with duplicate detection as an option	confirmed
Redelivery	A message received in peek-lock mode returns to the queue if not completed before the lock duration expires	confirmed
Dead-letter queue	Built into every Service Bus queue and subscription as a dead-letter sub-queue, filled after the max delivery count	confirmed
Topic	A Service Bus topic with subscriptions, each of which can filter on message properties and behaves as a queue for its subscribers	confirmed
Stream	Event Hubs, partitioned, retained for a period you set, with a Kafka-compatible endpoint so Kafka clients connect unchanged	confirmed
Event bus	Event Grid, with topics, subscriptions, and filters, delivering to functions, webhooks, and Service Bus; also the source for Azure resource events	confirmed
Simple queue	Storage queues, inside a storage account, with fewer features and a lower cost	confirmed
Function trigger	Functions bind to Service Bus, Event Hubs, Event Grid, and Storage queues directly	confirmed
Tiers	Service Bus Basic, Standard, and Premium; Premium runs on dedicated capacity with predictable latency	confirmed

Their vocabulary

Standard term	Their term
Redelivery timeout	Lock duration
Move to the dead-letter queue after N failures	Max delivery count
Consumer	Receiver
Message group	Session

Where to look

The queue's Overview shows active, scheduled, and dead-lettered message counts. For alerts, use Azure Monitor's ActiveMessages and DeadletteredMessages metrics.

Last verified: never.

Google Cloud

What this maps to: Pub/Sub is one product that covers both queue and topic. For HTTP work with rate control there is Cloud Tasks, a queue. The stream is Managed Service for Apache Kafka.

Concept	On Google Cloud	Status
Topic	A Pub/Sub topic. Publishers write to it	confirmed
Queue	A Pub/Sub subscription on a topic. Each subscription gets every message. Subscribers on the same subscription split those messages between them, so a single subscription works as a queue. Add more subscriptions and you have fan-out	confirmed
Delivery	Pull subscriptions, where consumers ask, or push, where Pub/Sub calls an HTTPS endpoint such as a Cloud Run service	confirmed
Redelivery	A message not acknowledged within the ack deadline is redelivered	confirmed
Dead-letter queue	A dead-letter topic on the subscription, with a maximum delivery attempts count	confirmed
Order	Best effort unless messages carry an ordering key, which orders within the key	confirmed
Retention and replay	Messages are retained for a period you set. A subscription can seek back to a timestamp or snapshot, so Pub/Sub has some of a stream's replay	confirmed
Stream	Managed Service for Apache Kafka. Pub/Sub Lite was the earlier partitioned option	unconfirmed; check Pub/Sub Lite's current status
Event bus	Eventarc routes events from Google services and Pub/Sub to Cloud Run, GKE, and Workflows	confirmed
HTTP task queue	Cloud Tasks, with rate limits and scheduling per queue, for calling an endpoint later	confirmed

Their vocabulary

Standard term	Their term
Queue	Subscription
Redelivery timeout	Ack deadline
Dead-letter queue	Dead-letter topic
Consumer	Subscriber

Where to look

The subscription's metrics show unacked message count and oldest unacked message age. Alarm on the age.

Last verified: never.

---

Check your understanding

0 of 4 answered

1. A worker reads a message, charges a card, and crashes before deleting the message. What happens next?

   The message reappears after a timeout and another worker charges the card again, unless the consumer checks for work already done.The message is lost with the worker.The queue detects the crash and marks the message done.The card charge is rolled back automatically.
2. A new analytics consumer needs to see every event from the past week. Which shape allows that?

   A stream, which keeps an ordered log for a retention period and lets each consumer read from any position.A queue, if the messages have not been deleted.A topic, if the consumer subscribes with a start date.None. Messages are gone once delivered.
3. An order-placed event must reach the warehouse, the email service, and analytics. Which shape carries it?

   A topic, which delivers each message to every subscriber.A queue, which each service reads in turn.Three direct calls from the order service.A shared database table that each service polls.
4. What does a dead-letter queue prevent?

   One bad message being retried forever and blocking the messages behind it.Messages being delivered twice.Messages being delivered out of order.The queue filling up during a burst.