Disaster Recovery & Business Continuity

Every SLO in this course quietly assumes the system survives — that the servers, the network, and the data center or cloud region hosting them still exist while you work an incident. Disaster recovery (DR) planning is for the case where that assumption fails outright: a whole cloud region goes dark, a data center loses power and doesn't come back, or someone runs terraform destroy against the wrong workspace. That's a different discipline from the on-call response in incident management & on-call and the per-request resilience in reliability patterns — those cover a dependency failing or a service degrading while the rest of the world stays put. DR planning covers the world itself disappearing out from under the service, and it is governed by exactly two numbers. This page defines both precisely, walks through the four standard DR strategy tiers that trade cost for how small you can make those numbers, and makes the case — with evidence, not assertion — for why a DR plan nobody has ever actually executed should be treated as broken until proven otherwise.

RTO and RPO: the two numbers that actually define a disaster-recovery plan

☺ Like you're 10: RPO is "how much recent stuff are we okay losing," measured backward from the disaster. RTO is "how long can we be down," measured forward from it. Every DR decision is really just picking those two numbers.

A disaster, for this page's purposes, is not the same thing as an incident. An incident is a service or dependency misbehaving while the infrastructure underneath it is still there to fix. A disaster is the loss of the infrastructure itself — a full availability zone, a full region, or a full data center becomes unreachable or is destroyed, and there is nothing left to SSH into. Recovering from that requires standing up service somewhere else entirely, and two numbers define what "recovering" is even supposed to mean:

• Recovery Point Objective (RPO) — the maximum acceptable amount of data loss, measured as a span of time. It answers "how far back does the data we recover from actually go?" An RPO of 15 minutes means that, in the worst case, the 15 minutes of writes right before the disaster are gone forever — never replicated, never backed up, unrecoverable by definition.
• Recovery Time Objective (RTO) — the maximum acceptable duration of the outage itself. It answers "how long are users looking at a broken or unavailable service?" An RTO of 1 hour means the clock from disaster to fully-restored service must not exceed 60 minutes, including detection, decision-making, failover execution, and validation — not just the mechanical part of bringing infrastructure up.

The two numbers measure in opposite directions from the same event. RPO looks backward from the disaster and bounds how much of the recent past is at risk; RTO looks forward from the disaster and bounds how long the present stays broken. A plan that only specifies one of them is incomplete — a system that restores instantly from a backup that's a week stale has a great RTO and a terrible RPO, and a system that never loses a single write but takes three days to bring back up has the opposite problem. Both numbers have to be chosen, and chosen deliberately, for every DR-relevant service.

Two details are easy to get wrong. First, RTO is not the time to bring infrastructure online — it's the time until the service is validated as genuinely serving correct traffic, which includes confirming data integrity, warming caches, and checking that dependent systems have reconnected. Teams that measure RTO as "servers respond to a health check" routinely discover their real RTO, the one users experience, is much longer. Second, RPO is a ceiling, not an average — a 15-minute RPO target means no disaster may ever lose more than 15 minutes of data, so the replication or backup mechanism backing it must guarantee that bound even in its worst case (a saturated replication link, a backup job that silently failed last night), not just on a typical day.

The cost curve: why driving both numbers toward zero gets expensive fast

☺ Like you're 10: Getting from "down for a day" to "down for an hour" is cheap. Getting from "down for a minute" to "down for a second" is wildly expensive — the last bit of improvement always costs the most.

RTO and RPO of zero — no downtime, no data loss, ever — is not a fantasy; it's achievable with synchronous, multi-region, always-on infrastructure. It is also, for almost every workload in existence, not worth what it costs. The relationship between "how close to zero" and "how much it costs" is not linear — it's closer to inverse-exponential, and understanding why is what keeps a DR plan grounded in business reality instead of engineering perfectionism.

Three cost drivers compound as the two numbers shrink. Idle capacity — the closer to zero RTO you want, the more fully-provisioned, already-running standby infrastructure you need sitting in a second region doing nothing most of the time, which is the opposite of how cloud economics normally reward you for scaling down what you're not using. Replication engineering — the closer to zero RPO you want, the tighter the coupling between regions has to be, moving from "nightly backup job" (cheap, simple, asynchronous) to "synchronous cross-region write acknowledgment" (expensive in both infrastructure and in the added write latency every single request now pays, everywhere, all the time, to protect against an event that might happen once a decade). Operational complexity — every additional region actively participating in production at least doubles the surface area a platform team has to keep configured identically: IAM policies, network rules, service quotas, secrets, deployment pipelines, all drifting independently unless actively held in sync, which is itself an ongoing cost that never shows up on a cloud bill. Reliability economics works through this cost-of-reliability curve in general terms; DR is simply its most extreme, most visible application, because the "next nine" here isn't an abstraction — it's a line item for an entire duplicate environment.

The four standard DR strategy tiers

☺ Like you're 10: Four ready-made plans, from "keep a backup in a drawer" to "run two full copies of everything, all the time" — each one buys you a faster recovery for a higher monthly bill.

AWS's Well-Architected disaster-recovery guidance names four standard tiers, and the same four show up, with different branding, in Azure's and Google Cloud's own DR documentation — this framework has become the industry's shared vocabulary for the point on the cost curve a given service occupies. Treat the specific numbers below as order-of-magnitude guidance rather than guaranteed figures; verify current numbers against your cloud provider's own DR documentation if you need to cite them precisely, since actual achievable RTO/RPO depends heavily on your specific architecture, data volume, and automation maturity.

Tier 1 — Backup and restore

The baseline tier: data is backed up on a schedule to a separate, durable location — ideally a different region from production — and no compute runs in the DR region at all until a disaster actually happens. Recovery means provisioning fresh infrastructure from code (Terraform, CloudFormation, or equivalent), restoring the most recent backup into it, and redeploying the application on top. Nothing is pre-warmed; everything is built from scratch, on demand, under pressure.

# The two mechanical pieces of a backup-and-restore posture, made concrete:

# 1. Continuous, cross-region backup of the data layer (RDS example — point-in-time
#    recovery plus automated snapshots replicated to a second region)
aws rds modify-db-instance \
  --db-instance-identifier orders-db \
  --backup-retention-period 7 \
  --apply-immediately
aws rds start-db-instance-automated-backups-replication \
  --source-db-instance-arn arn:aws:rds:us-east-1:111122223333:db:orders-db \
  --backup-retention-period 7 \
  --region us-west-2

# 2. The restore test — this is the command nobody runs often enough,
#    and the only way to actually measure this tier's real RTO:
aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier orders-db \
  --target-db-instance-identifier orders-db-dr-drill \
  --restore-time 2026-08-15T09:00:00Z \
  --region us-west-2
# then: terraform apply the app-tier stack against it, and TIME THE WHOLE THING

Backup and restore is cheap — you pay for storage and cross-region transfer, not idle compute — and its RPO can actually be quite good if you use continuous point-in-time recovery rather than nightly snapshots. Its RTO is the weak point: provisioning a full environment from infrastructure-as-code and restoring a potentially large dataset routinely takes hours, and that number is only a guess until it's been timed for real, which is exactly the theme this page closes on.

Tier 2 — Pilot light

A pilot light keeps the smallest possible core of the system continuously running and continuously in sync in the DR region — almost always just the data layer, kept current via near-real-time replication — while the compute layer sits fully defined (as an AMI, a container image, an autoscaling group set to zero desired capacity) but not actually running. The name is the metaphor: the pilot light in a furnace is a tiny flame that's always lit, ready to ignite the full burner the instant it's needed, rather than lighting a match from nothing.

# A pilot-light data layer: an Aurora Global Database keeps a secondary
# region's read replica continuously current with sub-second typical lag.
# The compute layer (not shown) is defined but scaled to zero until needed.

resource "aws_rds_global_cluster" "orders" {
  global_cluster_identifier = "orders-global"
  engine                    = "aurora-postgresql"
  engine_version             = "15.4"
}

resource "aws_rds_cluster" "orders_primary" {
  cluster_identifier        = "orders-us-east-1"
  global_cluster_identifier = aws_rds_global_cluster.orders.id
  # ... primary-region config
}

resource "aws_rds_cluster" "orders_dr" {
  provider                  = aws.us_west_2
  cluster_identifier        = "orders-us-west-2"
  global_cluster_identifier = aws_rds_global_cluster.orders.id
  # secondary — read-only until a failover PROMOTES it to a standalone writer
}

resource "aws_autoscaling_group" "app_dr" {
  provider         = aws.us_west_2
  desired_capacity = 0   # the unlit burner — capacity plan proven, not running
  min_size         = 0
  max_size         = 20  # sized to full prod capacity in advance

A pilot light buys a meaningfully better RPO than backup-and-restore — typically minutes, bounded by replication lag rather than backup cadence — at moderate cost, since the always-on piece is usually just a database, the cheapest thing to leave running relative to a full application fleet. RTO improves too, into the tens-of-minutes range, but it still requires scaling the compute layer from zero, deploying the application, and validating — none of which is instantaneous.

Tier 3 — Warm standby

A warm standby runs a scaled-down but fully functional copy of the entire stack continuously in the DR region — smaller instance counts and sizes than production, but a real, working, end-to-end request path that could serve live traffic today, right now, at reduced capacity. Failover means scaling that environment up to full production capacity and shifting traffic to it, rather than building anything from scratch. This is the tier where the DR region stops being purely passive infrastructure and starts being a genuine, if undersized, second production environment.

The operational payoff is that failover becomes a scaling and traffic-shifting problem instead of a provisioning problem, which is a categorically faster and more reliable class of operation — multi-region & multi-AZ architecture covers the infrastructure patterns (health-check-driven routing, cross-region data replication topologies, capacity headroom planning) that make a warm standby's failover safe and fast rather than just fast. RTO drops into single-digit minutes; RPO, driven by the same near-continuous replication as pilot light, sits in the seconds-to-low-minutes range. The cost jump from pilot light is real, though — you are now running a live, if downsized, copy of every service in the stack around the clock, not just a database.

Tier 4 — Multi-site active-active

The top of the ladder: full production capacity runs simultaneously in two or more regions, all of them actively serving live traffic all the time via global load balancing — not standing by, working. Each region can already absorb the other's traffic, because it's already doing real work at real scale; a "failover" is just a traffic-routing change, often completing in seconds because there's no cold start anywhere in the path. Multi-site active-active is how you get RTO close to zero.

RPO close to zero is the harder promise, and it's where this tier's real engineering difficulty lives. Reads are easy to serve from anywhere. Writes are not: a genuinely synchronous multi-region write (wait for acknowledgment from every region before confirming success to the caller) buys near-zero RPO at the cost of adding cross-region network latency to every single write in the system, permanently — often the dominant cost of running active-active at all. Most systems that call themselves active-active are actually active-active for reads with a single elected write region per key or per shard, or they accept eventual consistency and resolve conflicting concurrent writes after the fact with techniques like last-write-wins timestamps or CRDTs. Database reliability engineering and distributed systems reliability fundamentals cover this replication and consistency trade-off — the CAP-theorem tension between availability and consistency during a partition — in the depth it deserves; purpose-built platforms for this tier include DynamoDB Global Tables, Cosmos DB multi-region writes, Cloud Spanner, and CockroachDB.

*Synchronous writes buy near-zero RPO at the cost of added write latency everywhere, permanently; most production active-active systems trade some write RPO for lower latency via single-writer sharding or eventual consistency — see the tier-4 discussion above.

Matching a tier to the business: run a Business Impact Analysis, don't guess

☺ Like you're 10: Don't pick a DR tier because it sounds impressive — figure out what an hour of downtime actually costs the business, and buy exactly the amount of speed that's worth that cost.

The four tiers are a menu, not a recommendation, and the temptation to reach straight for warm standby or active-active "to be safe" is exactly the mistake the cost curve exists to prevent — most services genuinely do not need it, and paying for it anyway is money and complexity that bought nothing measurable. The correct process is a Business Impact Analysis (BIA): for each service, estimate the cost of an hour of downtime (lost revenue, contractual SLA penalties, regulatory exposure, support load, reputational damage) and the cost of an hour of lost data, then find the tier where the infrastructure cost stops being smaller than the risk it's buying down. This is the same falsifiable, numbers-first instinct behind an error budget — a DR tier, like an SLO, should be a deliberately chosen trade-off with a number attached, not a feeling.

Worked example — the checkout API's order-processing path:

  Average transaction volume:        $40,000 / hour during business hours
  Contractual SLA penalty exposure:  $15,000 flat, if downtime exceeds 2 hours
  Support + reputational estimate:   ~$5,000 / hour of user-visible outage

  At backup-and-restore's ~4-hour RTO:
    revenue-at-risk  ≈ $40,000 × 4          = $160,000
    + SLA penalty (crosses 2h threshold)    = $15,000
    + support/reputational  ≈ $5,000 × 4    = $20,000
    total exposure per incident            ≈ $195,000

  At warm standby's ~10-minute RTO:
    revenue-at-risk  ≈ $40,000 × (10/60)    ≈ $6,700
    + SLA penalty (never crosses threshold) = $0
    + support/reputational ≈ $5,000×(10/60) ≈ $830
    total exposure per incident            ≈ $7,500

  Difference per incident avoided: ~$187,500 — now compare that
  against warm standby's actual monthly infrastructure delta
  and multiply by your honestly-estimated disasters-per-year.
  THAT comparison, not intuition, is what justifies the tier.

Run this per service, not once for the whole company — a single enterprise-wide DR tier is almost always wrong in both directions, overpaying for services that could tolerate a slow restore and underprotecting the one payment path that can't. Most organizations that do this well end up with a small number of criticality classes (often three or four: revenue-critical and contractually-bound, important-but-tolerant-of-a-short-gap, and best-effort) and map each class to a DR tier, the same way capacity planning tiers infrastructure investment by service criticality rather than treating every service identically. Production readiness reviews is where this mapping should actually get enforced — a new service's DR tier is exactly the kind of decision that belongs in the launch checklist, made deliberately before the service has real traffic and real data at stake, not retrofitted after an outage makes the gap obvious.

The mechanics of failover: what actually happens when you flip the switch

☺ Like you're 10: Flipping a DR switch isn't one action — it's a whole ordered checklist: notice, decide, promote the data, redirect traffic, and check nothing's broken, all in the right order.

A DR tier describes an end state; failover is the sequence of operations that gets you there, and it has to happen in a specific order or it makes the disaster worse instead of better. A representative sequence for a database-backed service:

1. Detect and confirm. Health checks and monitoring (see monitoring & observability) flag the primary region as unreachable — but automated detection alone shouldn't trigger an irreversible failover for anything short of the fastest, most narrowly-scoped tiers, because a false positive that triggers a real cutover can be more disruptive than the outage it was meant to fix.
2. Declare the disaster. A human — typically an incident commander, per incident command for large-scale incidents — makes the call, because this decision trades a currently-degraded-but-recoverable state for a deliberate, harder-to-reverse cutover, and that trade deserves a decision-maker, not just a threshold.
3. Fence the old primary. Before promoting anything in the DR region, cut off the failed primary's ability to accept writes — revoke its credentials, remove it from the load balancer, or explicitly mark its data volume read-only. Skip this step and you risk two writable primaries diverging simultaneously, the exact split-brain failure mode multi-site architectures have to design around from day one.
4. Promote the data layer. The replica in the DR region is promoted to a standalone, writable primary. This is the moment RPO gets realized for real — whatever replication lag existed at the instant of the disaster is now permanent data loss, which is exactly why this step's safety depends entirely on the replication discipline chosen back in the DR-tier decision.
5. Redirect traffic. DNS-based failover (Route 53 health-check-triggered records) or anycast-based routing (AWS Global Accelerator, a CDN's origin failover) shifts traffic to the DR region. DNS TTLs matter here — a long TTL means clients and resolvers keep caching the old, dead endpoint well past the moment you've fixed everything else, which is why DR-critical DNS records are kept on short TTLs specifically to keep this step from becoming the failover's actual bottleneck.
6. Validate, don't assume. Confirm the promoted environment is serving correct traffic end to end — not just that health checks pass, but that a real transaction completes correctly — before declaring the incident resolved. This is the moment the achieved RTO gets recorded.

// Route 53 failover: a primary record with health-check-driven failover
// to the DR region's endpoint. TTL is deliberately short.
{
  "Comment": "Failover routing for checkout.acme.io",
  "Changes": [{
    "Action": "UPSERT",
    "ResourceRecordSet": {
      "Name": "checkout.acme.io",
      "Type": "A",
      "SetIdentifier": "primary-us-east-1",
      "Failover": "PRIMARY",
      "TTL": 30,
      "HealthCheckId": "abcd1234-health-check-id",
      "ResourceRecords": [{ "Value": "203.0.113.10" }]
    }
  }]
}
// A second UPSERT, SetIdentifier "secondary-us-west-2", Failover: "SECONDARY",
// pointing at the DR region's IP, completes the pair.

Why an untested DR plan should be assumed broken

☺ Like you're 10: A DR plan you've never actually run is a guess wearing a costume that makes it look like a fact — the only way to turn the guess into a fact is to actually set it off and time it.

Everything above this section describes what a good DR plan should say. None of it establishes that a specific organization's actual DR plan says anything true, because DR infrastructure rots in ways that are invisible until the day it's needed. A DR region's infrastructure-as-code drifts from what actually got clicked-through in the console during an emergency last year. IAM roles referenced in a runbook get renamed during a security audit and nobody updates the runbook. TLS certificates provisioned for the DR region's load balancer expire quietly because nothing in that region has served real traffic to notice. A new S3 bucket or a new microservice gets added to production and never gets added to the backup or replication scope, because "add it to DR" isn't a step in the normal service-launch checklist unless production readiness review explicitly enforces it. And one of the most common failures of all: the cloud account's service quotas in the DR region — EC2 instance limits, load balancer counts, IP address allocations — were never raised to match production's actual scale, so on the day of a real failover, autoscaling tries to scale up and simply can't, silently capped by a limit nobody remembered existed.

None of these failures show up by reading the runbook. They only show up by running it. This is precisely the job chaos engineering's game days already do — a scheduled, deliberately-injected failure that tests the people and the process, not just the automated failover code — and DR planning doesn't need a separate mechanism so much as it needs to insist that this specific mechanism gets pointed at the DR path specifically, on a real cadence, with a real success criterion: the achieved RTO and RPO, measured during the drill, must be logged and compared against the target. A drill that "worked" but took four hours against a one-hour target didn't validate the plan — it falsified it, and that's a genuinely useful, actionable result, not a failed exercise. Netflix's Chaos Kong, which simulates the loss of an entire AWS region, exists for exactly this reason: some failure modes are only real if you can point to the day you survived one on purpose.

The fix is not more documentation. It's a standing calendar entry: a real DR drill, at a cadence matched to the service's criticality tier (quarterly for the services a BIA marked revenue-critical, at minimum annually for everything else with a DR obligation at all), run against production or fully-representative traffic where the risk profile allows it, with a facilitator, a blast-radius scope, and a blameless debrief exactly like the one postmortems & blameless culture describes — because a DR drill that surfaces a broken assumption is a postmortem that got to happen on a Tuesday afternoon instead of during a real 3 a.m. disaster.

Where this fits

☺ Like you're 10: This page gave you the two numbers and the four price points; the pages around it show you how to build the infrastructure, run the drill, and prove the numbers are real.

Disaster recovery sits at the intersection of several disciplines this course treats separately for good reason. Multi-region & multi-AZ architecture is the infrastructure pattern layer underneath every tier above pilot light. Database reliability engineering and distributed systems reliability fundamentals go deep on the replication and consistency trade-offs tier 4 only sketched here. Chaos engineering and chaos engineering at scale are where the drill discipline this page insists on gets its full treatment, and Capstone Part 6 — chaos engineer it is where you can actually run one end to end. The 2017 AWS S3 outage is worth reading as a case study in how "regional" dependencies turn out to be more tightly coupled, and more widely relied upon, than almost anyone's architecture diagram admitted beforehand. And if a full production failover isn't yet on your team's calendar, the incident response tabletop drill is a lower-stakes rep of the same muscle: deciding, under time pressure, with incomplete information, before the real thing forces the decision on you.