Reliability patterns

Every dependency your service calls will eventually be slow, unavailable, or wrong: a database will time out, a payment gateway will start returning 500s, a network partition will drop half your requests. This page catalogs the concrete patterns engineers use to keep a system serving traffic anyway — redundancy, circuit breakers, retries with backoff and jitter, timeouts, bulkheads, graceful degradation, and load shedding. Each one addresses a specific failure mode, and in production they are almost always combined rather than used alone.

Redundancy and replication: no single point of failure

Redundancy is the foundation every other pattern on this page sits on top of: run more than one copy of a critical component so that any single copy failing doesn't take the service down with it. A stateless application server behind a load balancer is the simple case — kill one instance and the load balancer routes around it. Stateful components (databases, caches, queues) need replication instead: multiple copies of the same data kept in sync, so a failed primary has a replica ready to take over.

The redundancy model determines how much failure you can absorb. N+1 redundancy keeps one spare beyond what peak load requires — enough to survive a single failure, but not two at once. N+2, or spreading instances across three availability zones instead of two, survives a second failure while the first is still being repaired, which matters because failures cluster: a bad deploy that takes down one instance often takes down its siblings too. Replication also forces a consistency trade-off — a quorum write across three replicas, requiring acknowledgment from two of three before a write is considered durable, tolerates one slow or down replica without blocking writes, but is slower than writing to a single node. Redundancy buys the physical possibility of continuing to serve traffic; it doesn't by itself detect a failure or decide what to do next — see monitoring and observability for how failures actually get noticed.

The circuit breaker pattern

A circuit breaker sits in front of a call to a downstream dependency and tracks its recent success and failure rate. It has three states, named after the electrical breaker it borrows from. Closed is normal operation: calls pass through, and the breaker counts failures over a sliding window — say, the last 20 requests. If the failure rate crosses a threshold (a common default is 50%), the breaker trips to open: for a fixed cooldown period, often 20 to 60 seconds, every call fails immediately without attempting the network request at all, returning an error or a fallback straight away.

The open state protects both sides of the call. It stops your service from piling up threads or connections waiting on a dependency that's already struggling, and it stops hammering that dependency with traffic while it's trying to recover — retries against an overloaded service are often exactly what turns a brief blip into a prolonged outage. After the cooldown, the breaker moves to half-open: it lets a small number of test requests through. If they succeed, the breaker closes and normal traffic resumes; if they fail, it reopens and the cooldown restarts. Resilience4j (Java), Polly (.NET), and Envoy's outlier detection all implement this state machine; Netflix Hystrix, now in maintenance mode, is the reference implementation that defined the pattern's vocabulary. A tripped breaker is a real, alertable signal — see incident management and on-call for how it fits into an on-call rotation's alert design.

Retries, exponential backoff, and jitter

A single failed call is often transient — a dropped packet, a momentary GC pause, a load balancer mid-reroute — and retrying it is usually the right instinct. Retrying naively is not: if every client hitting the same failure retries immediately, the resulting spike of duplicate traffic can be exactly what keeps a recovering service down. That's why retries are paired with exponential backoff: each retry waits longer than the last, typically doubling (100ms, 200ms, 400ms, 800ms…) up to a capped maximum, so retry traffic tapers off instead of piling on.

Backoff alone still isn't enough, because every client following the same schedule retries in synchronized waves — everyone's attempt two lands at the same moment, then everyone's attempt three, and so on. Jitter fixes this by adding randomness to the wait. The AWS Architecture Blog's 2015 analysis of the problem (Marc Brooker, "Exponential Backoff and Jitter") found that full jitter — picking a random wait uniformly between zero and the current backoff ceiling, rather than the ceiling plus a little noise on top — spreads retries out most effectively and does measurably less total work against a recovering service than backoff without jitter.

function call_with_retry(request, max_attempts=5):
    base_delay_ms = 100
    max_delay_ms  = 20000

    for attempt in 1..max_attempts:
        try:
            return call_downstream(request, timeout_ms=2000)
        catch TransientError:
            if attempt == max_attempts:
                raise

            # exponential backoff: doubles each attempt, then caps
            ceiling = min(max_delay_ms, base_delay_ms * 2^(attempt - 1))

            # full jitter: a random wait in [0, ceiling], not
            # "ceiling plus a little noise" — the whole wait is random
            wait_ms = random_uniform(0, ceiling)

            sleep(wait_ms)

# attempt 1 fails -> ceiling  200ms -> wait somewhere in 0-200ms
# attempt 2 fails -> ceiling  400ms -> wait somewhere in 0-400ms
# attempt 3 fails -> ceiling  800ms -> wait somewhere in 0-800ms
# attempt 4 fails -> ceiling 1600ms -> wait somewhere in 0-1600ms
# attempt 5: last try, re-raise on failure

Client libraries implement this exact loop rather than needing it hand-written: resilience4j's Retry module, Polly, and most gRPC and HTTP clients all expose base delay, cap, and jitter strategy as configuration. See the SRE toolchain for where these libraries sit in a service's dependency stack.

Timeouts and the bulkhead pattern

A call with no timeout is a call that can wait forever, and "forever" is exactly how one slow dependency becomes a full outage of the service calling it: threads or connections pile up waiting on a response that never arrives, the pool exhausts, and every other request — including ones with nothing to do with the failing dependency — starts failing too. Every network call needs an explicit timeout, set separately for connection establishment and for waiting on a response, and in a call chain the timeout budget should shrink as it propagates: if a user-facing request has a 2-second deadline, a service three hops downstream shouldn't still get the full 2 seconds after the first two hops already spent 1.5 of them. gRPC propagates a deadline through the call context for exactly this reason.

The bulkhead pattern addresses a related but distinct failure: even with timeouts set, a slow dependency can still exhaust resources — thread pool slots, connection pool capacity, semaphore permits — that every other dependency also needs, if all calls draw from one shared pool. Named after the watertight compartments that let a ship's hull take damage in one section without sinking the whole vessel, the pattern isolates resource pools per dependency, so a slow downstream can only exhaust its own allotment. Hystrix implemented this literally, with a dedicated thread pool per external call; the same idea applies to connection pools sized per downstream and to per-tenant rate limits. Sizing those pools correctly is itself a capacity planning exercise, not a one-time guess.

Graceful degradation and load shedding

Graceful degradation and load shedding both trade completeness for availability, but they trigger on different conditions. Graceful degradation applies when one specific, non-critical dependency is down: instead of failing the whole request, the service serves a reduced version that omits what that dependency provided. A product page whose recommendation service is unavailable can still render the product, price, and buy button — it just drops the "customers also bought" rail instead of returning an error. The design decision has to be made upfront: classify each dependency as critical (its failure should fail the request) or non-critical (its failure should degrade the response), and write an explicit fallback for every non-critical one, because the default behavior of most code is to let a failed call propagate as an exception all the way up.

Load shedding is a different lever, applied at the front door rather than at one dependency: under extreme load, the service deliberately rejects some requests outright — typically a 503 with a Retry-After header — to protect its ability to serve the rest with acceptable latency. The alternative, accepting every request and letting everyone get proportionally slower, tends to be worse: as latency climbs, upstream timeouts start firing and triggering retries, adding more load onto an already-overloaded system, a feedback loop that can turn a load spike into a full outage. Priority-aware shedding (drop low-priority batch traffic before user-facing traffic) and adaptive concurrency limits (Netflix's concurrency-limits library and Envoy's adaptive concurrency filter both estimate a safe in-flight request count from observed latency rather than using a fixed number) are the common production implementations. Both patterns are worth validating deliberately with fault injection — see chaos engineering — rather than discovering a broken fallback path during a real incident.