Engineering for Reliability · Multi-Region & Multi-AZ Architecture

Multi-Region & Multi-AZ Architecture

"Just deploy to two regions" is one of the most confidently-said, least-examined sentences in infrastructure planning. It sounds like a capacity decision — stand up a second copy of the compute, done — but it's actually a distributed-systems promise with a price tag attached: a promise about how much data you're willing to lose, how long you're willing to be down, and how you'll stop two regions from disagreeing about reality at the exact moment you need them to agree the most. This page takes that promise apart. We'll separate what multi-AZ actually buys you from what multi-region buys you, walk through active-passive and active-active architectures mechanically, quantify replication lag as the real source of data loss, explain split-brain and how quorum and fencing prevent it, and end with the part everyone glosses over: the literal DNS and traffic-shifting mechanics of a regional failover, second by second.

Availability zones and regions are not the same failure domain

☺ Like you're 10: An AZ is a different building on the same street; a region is a different city. Losing a building is common and cheap to survive. Losing a city is rare and expensive to survive.

Before comparing architectures, get the vocabulary exactly right, because the two terms protect against genuinely different failure domains and the confusion between them is where a lot of false confidence comes from. An Availability Zone (AZ) is one or more discrete datacenters within a region, each with independent power, cooling, and physical network, but connected to the other AZs in that region by dedicated, low-latency, high-bandwidth private fiber — typically sub-2ms round trip. A region is a fully separate geographic area — hundreds or thousands of miles from the next nearest region, on a different power grid, often in a different climate and seismic zone — connected to other regions only by the public internet or the provider's long-haul backbone, with round-trip latency typically in the tens to low hundreds of milliseconds depending on distance (roughly 60–70ms between two U.S. coasts, 70–90ms across the Atlantic; consult your provider's current network map for real numbers, since backbone routes change).

Multi-AZ protects against the failure domain that's actually common: a single datacenter losing power, a rack failing, a single AZ's network gear misbehaving, or a botched change that only reaches one AZ's infrastructure. It does not protect against anything that's regional in scope by construction: a region-wide control-plane service having an outage (the load balancer provisioning API, the DNS control plane, IAM), a regional network peering failure, a natural disaster affecting an entire metro area, or a bad global configuration change that a region-scoped rollout doesn't catch in time. Several of the highest-profile cloud outages of the last decade were exactly this shape — not a single AZ going dark, but a regional control-plane dependency taking every AZ in that region down together, which is precisely the failure mode multi-AZ cannot help with because all the AZs were never really independent of that one shared regional service in the first place.

Active-passive: the conservative failover architecture

☺ Like you're 10: One stand is open, the other is stocked and ready but closed — and "open the second stand" is a real decision someone (or something) has to make and execute.

In an active-passive (also called active/standby) architecture, one region — the primary — serves all live read and write traffic. The other region — the standby — continuously receives a replicated copy of the primary's data but does not serve production write traffic, and often not read traffic either, until it's explicitly promoted. The AWS Well-Architected Framework's disaster-recovery strategies map onto a spectrum of how "ready" that standby actually is, and the spectrum is really a cost/RTO tradeoff dial:

The mechanical failover step, regardless of posture, is promotion: the standby's replica is told to stop following the primary and start accepting writes as the new primary. Concretely this might mean promoting a PostgreSQL streaming-replication standby with pg_promote(), triggering a managed cross-region failover such as an Aurora Global Database's unplanned failover (which detaches the secondary cluster and promotes it to a standalone read/write cluster), or failing over a DNS/traffic layer to point at compute that was already running in warm/hot postures. Whatever the mechanism, promotion is a one-way door in the common case — the old primary, if it comes back, is now a second writer with data the new primary never saw, which is exactly the split-brain problem covered later on this page.

Active-passive's appeal is that it sidesteps the hardest problem in this whole topic: because only one region ever accepts writes at a time, there's no multi-writer conflict resolution to design. Its cost is nonzero RTO (Recovery Time Objective — how long you're down before the standby is serving) and nonzero RPO (Recovery Point Objective — how much recently-written data you lose, bounded by however far replication had gotten before the primary died). Both numbers are architecture decisions, not accidents; see Disaster Recovery & Business Continuity for how to set and test them formally as part of a DR plan.

Active-active: the harder, better promise

☺ Like you're 10: Both stands are open at once, selling lemonade to whoever's closest — which is great for customers, but now the two stands have to agree on prices in real time.

In an active-active architecture, two or more regions accept live read and write traffic concurrently, usually routed by proximity so each user hits their nearest healthy region. Done well, this gives you the two things active-passive can't: near-zero RTO for a regional failure (surviving regions were already serving real traffic, so there's no promotion step to wait on) and better latency for a geographically distributed user base. The price is that you now have a genuine distributed-writes problem, and there are really only two honest ways to solve it.

The first is ownership partitioning — sharding data (often by customer, tenant, or geography) so that any given key is only ever written in one region at a time, even though multiple regions are "active." There are no write conflicts because there's never more than one writer for a given piece of data; a region outage takes that region's shard offline for writes until it's failed over, but it doesn't take the whole system down. This is sometimes called a cell-based architecture, and it's the approach most large-scale active-active systems actually use, because it avoids conflict resolution entirely rather than solving it.

The second is true multi-master replication, where any region can accept a write to any key, and the system reconciles concurrent writes after the fact. DynamoDB Global Tables and Cosmos DB's multi-region-write mode both work this way by default, using last-writer-wins (LWW) conflict resolution keyed on a timestamp: if two regions write the same item within the replication window, the write with the later timestamp survives and the earlier one is silently discarded — not merged, not flagged, just gone. Cosmos DB additionally lets you register a custom conflict-resolution procedure instead of LWW, which is the honest way to handle domains where "just keep the newer one" is the wrong business rule (a shopping-cart merge and a bank-balance conflict need very different resolution logic, and LWW gets both of them wrong by default).

A third, more expensive option sits between these two: synchronously consistent multi-region databases like Google Cloud Spanner or CockroachDB, which use a consensus protocol (Spanner's TrueTime plus Paxos, CockroachDB's Raft) to make every write durable across a quorum of regions before it's acknowledged. This buys you real linearizability — no conflicts, no LWW data loss, no split-brain window — but every write pays the full cross-region round-trip as latency, because the write literally isn't done until a majority of regions have agreed to it. This is consistency bought with latency, not consistency for free; there is no version of this tradeoff that avoids the tradeoff.

Replication lag is the real number behind your RPO

☺ Like you're 10: "How much can we lose?" isn't a policy you write down — it's however far behind the copy actually was the instant the original stand caught fire.

Whatever architecture you pick, cross-region replication moves data over one of two disciplines, and the choice is a direct trade of latency against data loss. Synchronous replication holds the client's write unacknowledged until the data is durable in at least one (often a quorum of) remote regions — RPO effectively zero, because nothing is acknowledged as written until it's already safe elsewhere, at the cost of adding the full cross-region round trip to every write's latency, typically tens to well over a hundred milliseconds depending on distance. Asynchronous replication acknowledges the write locally and ships it to the remote region afterward — write latency stays low and local, but there is now a window, the replication lag, during which the remote copy is behind. If the primary dies inside that window, whatever hadn't shipped yet is gone. The replication lag is your RPO; it isn't a separate number you get to choose independently.

The actual mechanism varies by data store, and it's worth knowing what's really moving: PostgreSQL streaming replication ships the write-ahead log (WAL) to standbys, and you can watch the real lag directly — SELECT client_addr, state, sent_lsn, replay_lsn, replay_lag FROM pg_stat_replication; — where replay_lag is a live interval, not an estimate. MySQL ships the binary log. Change-data-capture pipelines (Debezium into Kafka, consumed by the remote region) generalize this pattern to heterogeneous systems. Managed cross-region databases often replicate below the SQL layer entirely — Aurora Global Database ships the storage-layer redo log directly to replica-region storage nodes, bypassing the compute tier, which is how it typically holds sub-second lag under normal load even across continents; that's an engineering choice, not a law of physics, and it degrades under write bursts or backpressure the same as any other pipeline.

Lag isn't a single number, either — it's a distribution, and the tail matters more than the median for RPO planning. A pipeline with 200ms p50 lag and 8-second p99 lag under normal load doesn't give you an RPO of 200ms; a failure that happens to land during a backlog spike gives you the tail, not the median, and a well-run team monitors and alerts on p99 lag specifically because that's the number that determines what a bad day actually costs. There's a second, quieter consequence of lag worth naming: a read that lands on a lagging replica can violate read-your-writes — a user who just wrote something and immediately reads it back can see the old value if the read is routed to a region that hasn't caught up yet, which is a correctness bug your users will report as "your app lost my data" even though it didn't. Database Reliability Engineering covers replication topologies and consistency guarantees in more depth; Distributed Systems Reliability Fundamentals covers the CAP-theorem framing underneath all of this.

Split-brain: when both regions think they're the primary

☺ Like you're 10: If both stands think they're the only one open and both start selling, you don't find out you have a mess until you try to count the money afterward.

Split-brain is what happens when a network partition — not necessarily a real regional outage — separates two regions from each other, and each side concludes, wrongly, that the other is down and it should become (or remain) the sole active primary. Both sides then accept writes independently, and the two datasets diverge for as long as the partition lasts. This is a partition problem, not an outage problem: Region A can be completely healthy and serving its own users just fine while Region B's failover automation, unable to reach Region A over the specific network path it's monitoring, concludes Region A is dead and promotes itself. Now there are two primaries, both correct from their own point of view, both wrong about the world.

The traditional on-prem answer to this is fencing, sometimes called STONITH ("shoot the other node in the head") in classic HA-cluster terminology: before a new primary is allowed to accept writes, the old primary must be forcibly and verifiably prevented from accepting any more — revoke its database credentials, pull it out of the load balancer pool, or have a proxy in front of it start rejecting writes outright — so that if it turns out to have been alive the whole time, it physically cannot re-accept a write once the new primary has taken over. Promote-then-fence is backwards and dangerous; fence-then-promote is the only safe order. The cloud-native answer to "was that node really down, or just unreachable from where I'm standing" is quorum: never let a single observer's opinion trigger a promotion. Require an odd number of independent voters (often three, sometimes five) spread so that no single regional failure can produce two separate majorities, and require a majority of them to agree "the primary is genuinely unreachable" before any promotion fires. This is exactly the discipline consensus protocols like Raft use for leader election inside a single system, applied one level up to the question of which region gets to be primary.

Not every system tries to prevent split-brain — some just let it happen and clean up afterward, and it's worth knowing which strategy a given system uses before you rely on it. DynamoDB Global Tables' last-writer-wins conflict resolution, covered above, is exactly this: instead of a quorum gate stopping concurrent writes from ever happening, both writes are simply accepted and the timestamp comparison discards one later. That's a legitimate design choice for data where losing the losing write silently is acceptable — but it is a categorically different guarantee than a fenced, quorum-gated promotion, and conflating the two is how teams discover data loss during an incident review instead of during the design review where it belonged.

Failover automation: what actually has to happen, in order

☺ Like you're 10: Deciding to open the second stand is only step one — someone still has to lock the first stand's register before anyone's allowed to sell from the second.

A production failover isn't a single action; it's an ordered sequence, and getting the order wrong is exactly how split-brain and data loss happen in practice rather than in theory. A defensible automated failover runbook does roughly this, in this order: (1) detect — confirm, via quorum across independent health-check vantage points, that the primary is genuinely unreachable or unhealthy, not just unreachable from one path; (2) fence — cut the old primary off from accepting further writes before anything else happens, so it can never become a second writer even if it turns out to be alive; (3) promote — instruct the standby (or the surviving active-active region) to take over as primary, including verifying how much of the replication stream it actually received before the primary went dark, since that gap is your realized RPO for this specific incident; (4) redirect — repoint traffic at the new primary, which is the DNS/traffic-shifting mechanics covered in the next section; (5) warm — if the standby was running at reduced capacity (pilot light or warm standby), scale it up and warm caches before it's expected to absorb full production load; (6) verify — watch error rates and latency as traffic actually arrives, because a region that passed a synthetic health check can still fail under the real traffic shape.

Whether steps 1–5 run fully automated or require a human to confirm the cutover is a real design decision with a real tradeoff on both sides, not a maturity ladder where "more automated" is unconditionally better. Fully automated failover minimizes downtime for genuine outages, but it also means a noisy or flapping health signal can trigger an unwanted cutover — and an unwanted cutover to a data-losing failover is its own incident, sometimes a worse one than the outage it was meant to fix. Human-in-the-loop failover avoids that risk but adds however long it takes to page someone, get them oriented, and get their confirmation — which is real added downtime, typically minutes, layered on top of detection time. Most mature setups split the difference: automate detection and even automate promotion for high-confidence, quorum-backed signals, but keep a human explicitly in the loop for the traffic-redirect step, or at minimum require a human acknowledgment within a short window before the redirect completes. See Incident Command for Large-Scale Incidents for how a regional failover gets run as a live incident once it's underway, and incident management & on-call for the paging mechanics Pip the Hummingbird owns elsewhere in this course.

The DNS and traffic-shifting mechanics of an actual failover

☺ Like you're 10: Flipping the sign on the door doesn't mean everyone who was already walking toward the old stand turns around instantly — some of them find out minutes later.

This is the part "just deploy to two regions" skips entirely, and it's where a design that looked fine on a whiteboard turns into a multi-minute real-world cutover. There are two fundamentally different mechanisms for redirecting traffic, and they have very different latency characteristics.

DNS-based failover — for example, Route 53 failover routing policy — pairs a primary and secondary record with a health check. A concrete, realistic configuration: a health checker probes an endpoint every 10 or 30 seconds, requires some number of consecutive failures (commonly 3) before marking the endpoint unhealthy, and the DNS record carries a TTL telling resolvers how long they're allowed to cache the answer. Add those up and you get a real detection-to-flip time budget that's already tens of seconds to a couple of minutes before Route 53 even changes what it's serving — and that's before any client actually sees the change. The record's TTL is the second half of the cost: a 60-second TTL means, in principle, that any resolver caching the old answer stops using it within 60 seconds of expiry — but real-world resolver behavior doesn't reliably honor that. Some ISP resolvers, corporate proxies, and OS-level stub resolvers cache longer than instructed — a well-documented behavior sometimes called TTL disrespect. In practice, long-tail clients can keep hitting the dead primary for minutes after the record officially changed, through no fault of your configuration.

Anycast-based failover — the approach behind AWS Global Accelerator, Cloudflare, and most large CDNs — sidesteps the caching problem entirely by not changing an address at all. The same IP is announced from multiple regions via BGP, and network routing (not DNS) decides which announcement a given client's traffic actually reaches; when a region goes unhealthy, its BGP announcement is withdrawn and traffic already routes to the next-nearest healthy region within the normal convergence time of the network, typically seconds, with no client-side cache to go stale because the address never changed. The tradeoff is operational: you're now running (or paying a provider to run) actual network-layer infrastructure, and "which region did this specific request actually land in" becomes a harder question to answer than reading a DNS record, which matters when you're debugging degraded performance rather than a hard failure.

A safer pattern than a big-bang 100% cutover, available with either mechanism through weighted routing, is to shift traffic gradually — 10%, then 50%, then 100% — watching error rates and latency at each step before continuing, exactly the canary discipline already covered in release engineering & progressive delivery applied to a region instead of a software version. It's slower than an instant flip, but it means a failover that turns out to be a bad idea — the "healthy" secondary is actually cold, under-provisioned, or has stale configuration — gets caught at 10% of traffic instead of discovered at 100%.

Rehearsing it: game days and the real cost of two regions

☺ Like you're 10: A fire drill you've never actually run isn't a plan — it's a guess, and the only way to know the guess is right is to set off the alarm on purpose.

A failover runbook nobody has executed against real traffic is a hypothesis, not a capability. Schemas drift between primary and standby regions in ways that only show up under real load; IAM policies, secrets, and third-party API allowlists sometimes exist only in the primary region because they were set up by hand once and never replicated into the runbook; hardcoded region identifiers show up in configuration nobody thought to check; and cache-warming or connection-pool ramp-up takes meaningfully longer under real production traffic than it did in the tabletop version of the plan. The only reliable way to find these gaps before an actual outage does is to schedule regular, real regional failover drills — quarterly is a common cadence for teams that take this seriously — and treat a drill that reveals a gap as a success, not a near-miss. This is the same discipline as chaos engineering generally, applied at the largest blast radius the organization is willing to test; tools purpose-built for exactly this, like AWS Fault Injection Service's region- and AZ-level disruption actions or Gremlin's infrastructure attacks, exist specifically because hand-rolling a regional outage safely is hard to get right. Chaos Engineering at Scale covers running these experiments as a standing program rather than one-off events.

None of this is free, and the cost is worth stating plainly rather than discovering on an invoice. Active-active roughly doubles steady-state infrastructure spend in the simplest case, and cross-region data transfer — billed per gigabyte by every major provider — is routinely the surprise line item once every write is also being replicated across a continent, especially for write-heavy workloads. The operational cost compounds the financial one: twice the surface area to patch, monitor, alert on, and reason about during an incident, and a failover runbook that itself needs to be maintained, versioned, and re-tested every time the architecture changes underneath it. Reliability Economics covers how to weigh this spend explicitly against the SLO it's meant to protect, rather than buying multi-region because it sounds safer without pricing what "safer" actually costs.

Check your answers

1. Multi-AZ protects against common, small-blast-radius failures — a single datacenter losing power, a rack failing, one AZ's network gear misbehaving. Multi-region protects against regional-scope events multi-AZ cannot touch by construction, because all AZs in a region share regional control-plane dependencies: a regional control-plane outage, a natural disaster across a whole metro area, or a regional network peering failure.
2. Promotion tells the standby to stop following the primary and start accepting writes as the new primary. It's a one-way door because the old primary, if it later comes back online, is now a second writer holding data the new primary never saw — continuing to write to it would create split-brain rather than undo the promotion.
3. Ownership partitioning (cell-based/sharded active-active), which avoids conflicts entirely because any given key is only ever written in one region, at the cost of that region's shard being unavailable for writes during its own outage; and true multi-master replication with conflict resolution (commonly last-writer-wins), which allows writes anywhere at the cost of silently discarding the losing concurrent write. A third, costlier option — synchronous consensus (Spanner, CockroachDB) — avoids both costs but adds a full cross-region round trip to every write's latency.
4. Replication lag is the window during which a remote copy is behind the primary; if the primary dies inside that window, whatever hadn't shipped yet is lost, so the lag distribution directly determines how much data a failover actually loses. The p99 matters more than the p50 because a real failure is as likely to land during a lag spike (network congestion, backpressure) as during typical conditions, and planning around the median understates the real worst case.
5. Split-brain is when a network partition causes two regions to each believe they are the sole active primary and both accept writes, producing diverged data. Fencing (cutting the old primary off from accepting writes before any promotion) is the traditional prevention; quorum (requiring a majority of independent observers to agree the primary is truly down before any promotion fires) is the consensus-based prevention — both are needed together for a safe failover.
6. The DNS record changing is only the first step: health checks need consecutive failures across independent checkers before the failover condition is even met (tens of seconds to roughly a minute), then the record flips — but resolvers and clients that already cached the old answer keep using it until their TTL expires, and real-world resolvers often don't honor the configured TTL exactly, so a meaningful tail of clients can keep hitting the dead region for minutes after the record itself changed.