Reliability of Event-Driven & Streaming Systems

A request/response service fails loudly and immediately: the caller gets a timeout or a 500, right now, and everyone downstream knows something is wrong. A queue- or event-based system can fail quietly for hours before anyone notices, because the whole point of a queue is to absorb the gap between how fast work arrives and how fast it gets done. That absorption is a feature — until the gap never closes, and the "buffer" turns out to have been debt the whole time. This page is about the reliability failure modes that are specific to producers, brokers, and consumers: what actually happens when a consumer can't keep up, how at-least-once and exactly-once delivery trade real guarantees for real costs, what a poison message does to a queue that has no dead-letter path, and why the metric most teams watch — downstream processing latency — is often the last thing to move when the system is already in trouble.

Why event-driven systems fail on a different axis

☺ Like you're 10: A queue doesn't remove the mismatch between "how fast work arrives" and "how fast it gets done" — it just hides that mismatch somewhere you have to know to look.

A synchronous call has one failure signal: it either returns in time or it doesn't, and backpressure is immediate and visible — the caller blocks, times out, or gets an explicit rejection. An event-driven pipeline — a producer publishing to a broker (Kafka, Amazon SQS, RabbitMQ, Google Pub/Sub, NATS JetStream), and one or more consumers reading from it — decouples the producer from the consumer in time, not just in space. The producer doesn't wait for the consumer to be ready; it writes to the broker and moves on. That decoupling is precisely why queues exist: it lets a slow or temporarily-down consumer fail without taking the producer down with it, and it smooths bursty traffic that would otherwise overwhelm a fixed-capacity consumer. The cost of that benefit is that the mismatch between arrival rate and processing rate doesn't disappear — it accumulates somewhere, silently, as lag: the growing gap between the last message produced and the last message a consumer has finished processing.

This page works through four reliability problems that are specific to this shape of system and mostly absent from request/response services: what actually happens, mechanically, when a consumer falls behind a producer (backpressure); what a message that can never be successfully processed does to a queue with no escape hatch for it (poison messages and dead-letter queues); the real difference between "the message was delivered" and "the message was delivered exactly once," and why the second one is far harder to get than the phrase implies (delivery semantics); and why the single most useful early-warning signal in most of these systems isn't the metric most dashboards default to (queue depth as a leading SLI).

Backpressure: what happens when a consumer falls behind a producer

☺ Like you're 10: A bathtub with the tap running faster than the drain doesn't overflow the first second — the water level just climbs, quietly, until it does.

Backpressure is what a system does — deliberately or by accident — when downstream can't keep up with upstream. In a synchronous chain, backpressure is a first-class concept with a name at every layer: TCP's own flow-control window, a thread pool's bounded queue rejecting new work, a load balancer returning 503. In an event-driven pipeline, backpressure is easy to build in accidentally as an absence of a limit rather than a deliberate policy, because the broker sitting between producer and consumer is usually happy to keep accepting writes long after the consumer has stopped keeping up. Whether that's safe depends entirely on what the broker does with the messages piling up behind a slow consumer, and the honest answer is: it depends which broker, and it's rarely "nothing bad happens."

What "something runs out" means depends heavily on the broker, and treating them as interchangeable is a common source of surprise incidents:

Notice the pattern: only RabbitMQ's classic-queue behavior gives you anything resembling the synchronous world's instinct — a hard stop that protects the system by refusing new work. Kafka and SQS both choose availability for the producer over any signal that a consumer is drowning, which is usually the right default (a producer that can't write is often worse than a consumer that's behind) but means the "backpressure" isn't really backpressure at all — it's postponed data loss with a deadline attached, and the deadline is whatever the retention setting happens to be.

Building real backpressure back in

Because most managed brokers won't do it for you, a reliable event-driven system needs to choose deliberately how it responds when λ persistently exceeds μ, rather than discovering the answer at the retention boundary:

• Scale the consumer, not just watch it. Autoscale consumer replicas on a lag-based signal — KEDA's Kafka and SQS scalers, or a custom Horizontal Pod Autoscaler metric fed by consumer lag, are the standard mechanism on Kubernetes. This only helps up to the partition-count ceiling: a Kafka topic with 12 partitions cannot usefully run more than 12 consumers in one group, no matter how aggressively you scale, because a partition is only ever read by one consumer within a group at a time.
• Throttle the producer at the broker. Kafka supports first-class producer quotas (kafka-configs.sh --entity-type clients --entity-name checkout-service --alter --add-config 'producer_byte_rate=1048576') that cap a client's write rate and return backpressure (throttled responses) instead of silently accepting unlimited load.
• Shed load deliberately, by priority. If not every event is equally important, make that explicit — sample, drop, or downgrade low-priority event classes (verbose telemetry, best-effort analytics) before the queue depth threatens anything that actually matters, rather than treating every message as equally worth the retention budget.
• Alert on the backlog itself, before it becomes data loss. Section below is entirely about why this is the metric to watch, not consumer CPU or per-message latency.

Delivery semantics: at-least-once, at-most-once, and "exactly-once" with an asterisk

☺ Like you're 10: A mail carrier can promise "I'll never lose your letter" or "I'll never deliver it twice," but almost never both at once — and the version that sounds like it promises both usually has fine print.

The order of two operations — acknowledge the message and process the message — is the entire difference between the three delivery semantics a consumer can offer, and it's worth being precise about what each one actually guarantees rather than treating "exactly-once" as a checkbox to aim for by default.

That third row is the one worth dwelling on, because "exactly-once" is the phrase vendors reach for and the guarantee is almost always narrower than it sounds. Kafka's exactly-once semantics (EOS) rest on two real mechanisms: an idempotent producer (enable.idempotence=true), which tags each message with a producer ID and a per-partition sequence number so the broker can silently discard a duplicate caused by a producer-side retry after a network blip; and transactions (a transactional.id, with downstream consumers reading at isolation.level=read_committed), which let a consume-transform-produce step commit its output records and its input offset atomically — either both happen or neither does. That is a genuine, well-engineered guarantee, and it is real exactly-once processing — as long as every effect of processing a message stays inside Kafka. The instant your consumer's job is to write a row to Postgres, call a payments API, or send an email, that side effect sits outside the transaction boundary. A crash between "Kafka transaction committed" and "external write happened" (or the reverse order) can duplicate or lose that external effect regardless of how correctly Kafka's own internals behaved, because Kafka has no way to make an external system's write atomic with its own commit.

Making at-least-once safe: idempotency in practice

Three concrete techniques turn "the consumer might see this message twice" from a hazard into a non-event:

• Idempotency keys. Attach a unique, stable ID to each logical operation (not just each message — a retried message and its original should carry the same key) and check a dedup store before applying the effect. Stripe's public API popularizes this exact pattern with its Idempotency-Key header: the same key submitted twice returns the cached result of the first attempt instead of executing twice.
• Naturally idempotent operations. Prefer SET balance = 500 over UPDATE balance = balance + 50; prefer an upsert keyed on a natural ID over a blind insert. An operation that produces the same end state no matter how many times it runs doesn't need a dedup store at all.
• The transactional outbox pattern. The dual-write problem — "update the database AND publish an event, atomically" — has no clean answer if you literally do both as two separate writes, because a crash between them leaves one done and the other not. The outbox pattern instead writes the event to an outbox table in the same database transaction as the business write, then a separate process (commonly Debezium, reading the database's write-ahead log via change-data-capture) publishes rows from that table to Kafka asynchronously. The business write and the "I published this" fact are now atomic by construction, and publishing itself is free to be at-least-once, because the outbox rows make republishing idempotent to reason about.

-- Idempotent consumer, sketch:
BEGIN;
  SELECT 1 FROM processed_messages WHERE message_id = :id FOR UPDATE;
  -- if a row exists, COMMIT and skip — this message was already applied
  INSERT INTO processed_messages (message_id, processed_at) VALUES (:id, now());
  -- ... apply the actual business effect, in the SAME transaction ...
COMMIT;
-- A redelivered message with the same :id now short-circuits safely,
-- whether the redelivery was caused by a crash, a rebalance, or a retry.

Poison messages and dead-letter queues

☺ Like you're 10: One letter with a return address a mail carrier can't read shouldn't be able to jam the whole mail truck so nothing else gets delivered — but without a separate "can't deliver" bin, that's exactly what happens.

A poison message is one that a consumer can never successfully process, no matter how many times it's redelivered — a malformed payload, a schema violation from an upstream producer that shipped a breaking change, or a message whose specific content deterministically triggers a bug (a null field the handler doesn't expect, a numeric overflow, a foreign key that no longer exists). The defining property isn't that it's rare; it's that retrying it changes nothing. Without a deliberate escape hatch, at-least-once delivery's own retry logic turns a poison message into an infinite loop: the consumer fails, the message gets redelivered (on nack, or once a visibility timeout expires), it fails again, forever.

How bad that infinite loop is depends heavily on whether the broker enforces ordering. In SQS or RabbitMQ, a poison message is one bad object among many independent ones — annoying, wasteful of retries, but not fatal to throughput. In Kafka, it's much worse: a consumer that throws on a message and doesn't advance its committed offset for that partition blocks every subsequent message in that partition behind it — this is head-of-line blocking, and because ordering is only guaranteed within a partition, one bad message can stall an entire key's worth of traffic indefinitely while every other partition keeps flowing normally, making the failure look partial and confusing until someone notices lag is climbing on exactly one partition.

Dead-letter queue mechanics, broker by broker

A dead-letter queue (DLQ) is the deliberate off-ramp: after N failed delivery attempts, move the message somewhere else entirely, so the main queue keeps flowing and the poison message becomes a triage item instead of a blockage.

// SQS redrive policy, attached to the source queue —
// after 5 failed receives, the message moves to the DLQ automatically:
{
  "deadLetterTargetArn": "arn:aws:sqs:us-east-1:111122223333:orders-dlq",
  "maxReceiveCount": "5"
}

# RabbitMQ: declare the DLQ routing on the source queue's arguments —
# a message that's rejected (nack, requeue=false) or whose TTL expires
# gets republished through the dead-letter exchange automatically:
x-dead-letter-exchange:    "orders.dlx"
x-dead-letter-routing-key: "orders.dead"
x-message-ttl:              30000   # also used to build tiered retry delays

Kafka has no native DLQ concept — the log doesn't know what "failed" means, only what's been committed. The pattern is built by hand: catch the processing exception, produce the offending record to a separate <topic>.DLT topic (a convention popularized by Spring Kafka's DeadLetterPublishingRecoverer), then commit past it on the source topic so the partition keeps moving. Kafka Connect ships this as a configuration rather than code you write yourself:

errors.tolerance = all
errors.deadletterqueue.topic.name = connect-dlq
errors.deadletterqueue.topic.replication.factor = 3
errors.deadletterqueue.context.headers.enable = true   # keeps the failure reason attached

A more forgiving middle ground than "fail once, go straight to the DLQ" is a tiered retry topic pattern — used in production at Uber and documented publicly by Confluent — where a failure republishes to orders-retry-5s, then orders-retry-1m, then orders-retry-1h, each with exponential backoff and jitter, before finally landing in the DLQ only once every tier is exhausted. This absorbs the ordinary case (a downstream dependency was down for ninety seconds) without immediately quarantining a message that would have succeeded on the second try, while still protecting the main topic from being blocked while the retries play out.

Ordering, partitioning, and the rebalancing feedback loop

☺ Like you're 10: When one runner on a relay team stumbles and the whole team has to stop and re-hand off the baton, that pause is worse for everyone's time than the original stumble was.

Ordering guarantees in event-driven systems are almost always scoped, not global, and assuming otherwise causes quiet correctness bugs. Kafka guarantees order only within a partition — messages sharing a partition key (say, an order ID) arrive at their consumer in the order they were produced, but there is no ordering guarantee across partitions or across keys. A single RabbitMQ queue with exactly one consumer preserves strict order; the moment you add a second consumer for throughput, round-robin dispatch breaks that guarantee unless you deliberately use a consistent-hashing exchange or a single-active-consumer policy to keep it.

Consumer rebalancing — a Kafka consumer group reassigning partitions among its members — is where backpressure and poison messages compound each other into something worse than either alone. A rebalance triggers when a consumer joins, leaves, crashes, or fails to call poll() again within max.poll.interval.ms (the default is five minutes) — exactly the situation a consumer stuck on a poison message, or one buried under backpressure-induced processing delay, ends up in. Under the older "eager" rebalance protocol, every consumer in the group stops consuming for the duration of the rebalance — a stop-the-world pause that hits precisely when the group can least afford it, because lag was already climbing. Kafka's incremental cooperative rebalancing (CooperativeStickyAssignor, from KIP-429, and the newer client default) narrows this considerably: only the specific partitions actually being reassigned pause, and consumers that keep their existing assignment keep consuming through it.

Queue depth as a leading SLI

☺ Like you're 10: Timing how long it takes to answer one letter tells you nothing about whether the bin behind you is quietly filling up faster than you can empty it.

Here is the failure mode this whole page has been building toward, and the one most teams learn the hard way. Suppose a consumer processes every message it receives in a steady 50ms — its p99 processing latency SLI looks perfect, hour after hour, dashboard green throughout. Meanwhile the producer is emitting 120 messages/second and this consumer can only sustain 80/second. The backlog grows by 40 messages every single second, continuously, and nothing about the per-message latency metric moves at all, because that metric only measures how long it takes to handle a message once processing starts — it has no way to see how long a message waited before that, or how many more are piling up behind it. The first person to notice is usually a customer asking "why is this event three hours stale," or worse, nobody, until the broker's retention window quietly starts discarding the oldest unread messages.

Two better metrics, and how to read them

The fix isn't to abandon latency SLIs — it's to add a leading indicator that measures the backlog directly, and to prefer the version of it that's already normalized into time rather than a raw count:

Oldest-message age deserves the emphasis: it's the one signal on this list you can put directly into an SLO statement without any unit conversion, and it degrades gracefully into an intuitive story — "the oldest thing we haven't gotten to yet is now four minutes old" is meaningful to anyone, where "lag is 9,400 messages" requires knowing the topic's normal throughput to interpret at all.

Building it into an SLO and alerting policy

☺ Like you're 10: Six habits that turn everything above from "things to know about" into "things your pager actually catches before a customer does."

Everything above compresses into a short, concrete policy for any team operating a queue- or event-based pipeline in production.

1. Make oldest-message age (or lag-time) the primary SLI

State the SLO in the same units a human would use — "95% of the time, no unprocessed message is older than 5 minutes" — not "average lag stays under 10,000 messages." See SLIs, SLOs & error budgets for how to formalize this as a real, measured SLO with a compliance window, and keep the per-message latency SLI as a secondary signal rather than the primary one.

2. Alert on the growth rate, not just the absolute value

A backlog that's high but shrinking is a different situation from one that's lower but climbing. Apply the same multi-window, multi-burn-rate thinking used for error-budget alerting — a fast, short-window threshold for a sudden spike, a slower, longer-window threshold for a gradual leak — to lag growth instead of error rate; see multi-window, multi-burn-rate alerting for the underlying mechanism, which transfers directly.

3. Treat DLQ depth as its own zero-tolerance SLI

A DLQ receiving messages means something is actively broken right now — a schema mismatch, a downstream outage, a bug triggered by real production data. Alert on any sustained non-zero depth rather than waiting for a threshold, and route it to an owner, not a shared inbox.

4. Load-test consumer throughput headroom before production finds the ceiling

Verify sustained consumer throughput exceeds worst-case producer rate with real margin, using the same load-testing discipline as any other capacity plan — k6 or Locust driving synthetic load into a staging topic or queue, measured against the M/M/1 knee math in queueing theory for SRE and folded into the standard checklist in a production readiness review.

5. Chaos-test backpressure and poison-message handling deliberately

Don't wait for the first real poison message to find out whether your DLQ routing actually works. Deliberately publish a malformed message in staging, or pause a consumer group to force lag growth, using tools like Gremlin or Litmus, and confirm the whole chain — DLQ routing, autoscaling, lag alerts — fires the way this page describes it should. Chaos engineering covers the broader discipline this is one application of.

6. Design idempotent consumers by default, not as an incident follow-up

Build the dedup store or outbox infrastructure before you need it. Every consumer should be written assuming redelivery will happen — because, per the delivery semantics section above, it eventually will — rather than treating idempotency as a fix applied after the first duplicate-processing incident.