Network Reliability Engineering

Every SLO, retry budget, and circuit breaker this course has covered so far assumes a request can actually reach your service. That assumption quietly rests on four layers most teams never write an error budget for: DNS translating a name into an address, BGP routing packets to that address across networks nobody on your team operates, a load balancer correctly deciding which backend is healthy enough to receive the request, and — for most production systems today — a CDN's edge network doing all of the above on your behalf, globally, before a request ever touches infrastructure you control. This page goes under the application layer into that stack: DNS as a single point of failure and the TTL tradeoff that governs how fast you can escape it, load-balancer health-check design and the specific thundering-herd failure a badly-built check creates, BGP's failure modes worked through Meta's six-hour outage in October 2021, and how CDN and edge architectures try to contain a failure's blast radius instead of amplifying it. None of the patterns from earlier in this course — retries, circuit breakers, redundant replicas — help you if a request can't find your service in the first place.

The stack beneath every SLO you've set

☺ Like you're 10: Most of the dashboards you've built so far only start watching once a request has already arrived — this page is about everything that has to go right before that moment.

Application-layer reliability engineering is thoroughly instrumented by the time most teams reach this page: the four golden signals — latency, traffic, errors, saturation — get dashboards, the SLO gets a burn-rate alert (see multi-window, multi-burn-rate alerting), and a circuit breaker trips when a downstream dependency misbehaves (see reliability patterns). Almost none of that instrumentation looks at DNS resolution time, BGP route stability, or CDN edge health directly, because those layers are usually invisible right up until the moment they fail — and when they fail, they tend to fail completely rather than gradually. A memory leak degrades a service over hours, giving you a rising latency graph and time to react. A withdrawn BGP route or a bad global CDN config push doesn't degrade — the destination is either reachable or it isn't, in the same way a light switch is either on or off. That binary character is what makes this layer worth its own page: the failure modes below aren't "slower," they're "gone," and the diagnostic instinct of watching a metric trend upward doesn't apply to a system that vanishes in one step.

DNS: the resolution layer everything depends on first

☺ Like you're 10: Before any of your other reliability engineering matters, a name has to turn into an address — and that one lookup is arguably the single highest-leverage point of failure on the entire internet.

Every request to your service starts the same way, whether the client is a browser, a mobile app, another one of your own microservices, or a monitoring probe checking whether you're up: a name has to resolve to an IP address. That resolution travels through a chain of caches. A client's stub resolver asks a recursive resolver (your ISP's, or a public one like 8.8.8.8 or 1.1.1.1); if that recursive resolver doesn't already have a cached answer, it walks the hierarchy itself — a root server, then a TLD server for .com, then finally your domain's authoritative nameserver, the one place that actually knows the current answer. Every hop in that chain can cache the result, and every one of those caches is governed by a single number attached to the DNS record: the TTL, time-to-live, in seconds.

The TTL tradeoff

TTL is the lever that trades failover speed against load and latency, and it's a lever you set explicitly on every record. A low TTL (say, 30–60 seconds) means caches expire quickly: if you need to redirect traffic away from a failed region or a compromised IP, clients pick up the new answer almost immediately once their cache lapses. The cost is that your authoritative nameservers see dramatically more query volume — every client re-asks far more often — and every cache miss adds a full resolution round trip to that client's request latency. A high TTL (hours, sometimes a full day) is nearly free for your authoritative infrastructure and shaves latency for clients who almost always hit a warm cache, but it means a DNS change you make today might not be visible to every client for as long as the TTL you set weeks ago — including, critically, during an incident where you're trying to redirect traffic away from something broken.

# The TTL is right there in a normal dig answer, in seconds:
dig +noall +answer api.example.com
#   api.example.com.  60   IN   A   203.0.113.44
#                      ^^ TTL: this answer is good for 60 more seconds
#                         in every cache that already has it

# Watch the TTL count down in real time against a specific resolver:
dig +noall +answer @1.1.1.1 api.example.com
# run it again a few seconds later — the number drops until it hits 0
# and the resolver is forced to re-ask your authoritative nameserver

The failover pattern most teams actually want — an automatic switch to a healthy region when the primary goes down — is usually built as a health-checked DNS answer: a DNS provider (Route 53's failover routing policy is the canonical example, though most major providers offer something equivalent) runs its own health checks against your endpoints and stops returning an unhealthy one from the pool of answers it serves. That mechanism is a genuinely good pattern, but it inherits the TTL tradeoff completely: you cannot fail over in DNS faster than your TTL, because a client already holding a cached answer has no way to know the provider changed its mind until that cache expires and the client asks again. A 300-second TTL puts a hard, unavoidable five-minute floor under your DNS-based failover time, no matter how fast the health check itself detects the failure.

DNS as a single point of failure

Because literally everything upstream of your application — the browser's request, the TLS handshake's SNI lookup, even your own synthetic monitoring checking whether you're up — starts with a name resolution, an authoritative DNS outage doesn't degrade your service, it erases it from the reachable internet for every client whose cache has expired. The clearest public example remains the October 2016 Mirai-botnet DDoS attack against Dyn, then a major managed DNS provider: because Dyn was the sole authoritative DNS provider for a number of large sites, the attack on Dyn's infrastructure made those sites unreachable for a large fraction of internet users for several hours, even though the sites' own origin servers were never touched — verify the exact scope and timeline against public reporting if you're citing it precisely, but the structural lesson holds regardless of the specific numbers: a single authoritative DNS provider is a single point of failure for every domain that depends on it exclusively, no matter how reliable that one provider's own infrastructure is.

The direct mitigation is multi-provider authoritative DNS: publish your zone through two unrelated DNS providers (for example, Route 53 alongside NS1 or Cloudflare) with both sets of nameservers listed as authoritative, so a single provider's outage — or a DDoS attack targeting that provider specifically — leaves the other still answering. The operational cost is real: you now need to keep zone data synchronized across two systems, typically via a DNS-as-code tool that treats zone records the same way you'd treat any other declarative config, rather than clicking updates into two consoles by hand and hoping they match.

Load-balancer health checks: the mechanism that decides who gets traffic

☺ Like you're 10: A health check sounds like the safe, boring part of the system — until a badly designed one turns a small, local problem into a fleet-wide outage all by itself.

Once a request has an address, a load balancer usually stands between it and your actual backends, and its most safety-critical job is deciding, continuously, which of those backends are healthy enough to receive traffic. Two independent axes describe how a health check works, and getting either one wrong is a common, avoidable source of self-inflicted outages.

Active vs. passive, shallow vs. deep

An active health check has the load balancer independently poll each backend on a fixed interval — an HTTP GET to /healthz, say — regardless of whether any real traffic is flowing. A passive check instead watches the outcomes of real requests and ejects a backend that's been returning errors, without a separate polling loop; this is usually called outlier detection. Most production load balancers and service meshes run both together: active checks catch a backend that's failed silently with no traffic yet flowing to notice, and passive checks catch a backend that fails only under real load patterns an active probe never reproduces.

Separately, a check can be shallow — "is the process alive and the port open?" — or deep — "can this instance actually reach its database, its cache, and every other dependency it needs to serve a real request?" An L4 (TCP) check is inherently shallow: it can only confirm a socket accepted a connection, which happily reports healthy for an application that's accepted the connection and then deadlocked, wedged on a downstream call, or is otherwise alive-but-useless. An L7 (HTTP) check that actually exercises application logic is more honest about true readiness — but a deep L7 check introduces the exact correlated-failure risk this section exists to warn you about.

The thundering herd a bad health check creates

Picture a fleet of API instances behind a load balancer, each with a readiness check that verifies connectivity to a shared primary database — a reasonable-sounding choice, since an instance that can't reach the database genuinely isn't ready to serve most requests. Now the database has a connection-pool exhaustion event, or a brief network blip on its side. Every single instance's health check depends on the exact same shared resource, so every single instance fails its check within roughly the same polling interval — not because anything is wrong with the instances themselves, but because the check measured a shared dependency instead of the instance's own, independent state. The load balancer now sees zero healthy backends behind a perfectly healthy fleet, and depending on its configuration, one of two things happens next, both bad in different ways: it can fail closed and return an error to 100% of traffic — an outage the health check caused, not the one it was trying to prevent — or it can behave unpredictably if nobody configured a defined fallback at all.

The direct mitigation is a panic threshold: configure the load balancer so that once more than some percentage of the fleet is simultaneously unhealthy — Envoy's default is 50%, and it's a sensible starting point for most fleets — the load balancer stops trusting the health check and load-balances across every host regardless of reported status, on the reasoning that if this many backends failed at once, the check itself (or a shared dependency the check measures) is far more likely to be the actual problem than every instance failing independently at the same moment. This is a deliberate, engineered fail-open, and it's the network-layer sibling of the same instinct behind a circuit breaker's half-open state in reliability patterns: past a certain point, trusting the signal you have does more damage than ignoring it.

# Envoy: outlier detection (passive) plus a panic threshold, on a cluster.
# Field names shift between Envoy releases — confirm against the current
# API reference before shipping — but the shape of the idea is stable.
clusters:
  - name: checkout_backend
    outlier_detection:
      consecutive_5xx: 5            # eject after 5 consecutive 5xxs
      interval: 10s
      base_ejection_time: 30s
      max_ejection_percent: 50      # never eject more than half the fleet
    common_lb_config:
      healthy_panic_threshold:
        value: 50.0                 # below 50% healthy, ignore health
                                     # status and load-balance across
                                     # every host anyway (fail open)

A second, quieter thundering-herd risk sits on the recovery side rather than the failure side. When a backend that was marked unhealthy — or a brand-new instance that just passed its first check — is added back into rotation, a naive load balancer routes it its full, steady-state share of traffic immediately. That instance's caches are cold, its connection pools aren't warmed up, and its JIT compiler (where applicable) hasn't optimized any hot paths yet, so the sudden full-weight traffic can push it straight back into failure — a self-inflicted flap. The mitigation is a gradual traffic ramp, usually called slow start: the load balancer sends a newly-healthy host a small fraction of its eventual share and increases it over a defined window, giving the instance time to warm up before it takes a full load. Kubernetes-native readers will recognize the liveness/readiness probe split covered in depth in Kubernetes reliability patterns as the same discipline one layer down — separating "should this be restarted" from "should this receive traffic right now" is exactly the shallow-vs-deep, correlated-vs-independent distinction this section is about.

BGP: the routing layer beneath the CDN and the load balancer

☺ Like you're 10: None of the layers above matter if the postal service's own map no longer shows a road to your town — and unlike a server crashing, that can happen to a perfectly healthy destination with a single bad command.

The Border Gateway Protocol is the mechanism that glues the internet's many independently-operated networks — tens of thousands of autonomous systems (ASes), each with its own AS number, from your cloud provider down to your own company if you run your own network edge — into one routable whole. Every AS tells its neighboring ASes which IP prefixes it can deliver traffic to, and those neighbors pass that reachability information on to their neighbors, hop by hop, until in principle every network on the internet knows a path — an "AS path" — toward every reachable prefix. BGP is a path-vector protocol, and critically, it was designed in an era of implicit trust: by default, nothing in the protocol cryptographically verifies that an AS announcing a prefix is actually authorized to do so.

Announcement, withdrawal, and why "withdrawn" looks identical to "never existed"

An AS announces a prefix to say "route traffic for this network through me," and withdraws it to retract that promise. A withdrawal propagates through BGP update messages to every network that had learned the route, typically within minutes for a well-connected prefix — and once it's propagated, the destination is simply gone from the global routing table. There is no error state, no degraded-but-reachable middle ground: a router with no path to a prefix drops packets addressed to it exactly as if the destination had never existed, regardless of whether the servers behind that prefix are running perfectly. This is the single fact that makes BGP failures so disorienting to debug from the application side — your own dashboards, your own health checks, everything running on the servers themselves can report fully healthy, because from inside those servers nothing is wrong. The problem is entirely in a layer those servers can't see or influence.

Before a withdrawal — routers worldwide hold a path:
  203.0.113.0/24  via AS64500  (3 hops away)   -> reachable

After AS64500 withdraws that prefix:
  203.0.113.0/24  -- no route --                -> unreachable
  (packets addressed here are simply dropped upstream;
   the origin server never even sees the request arrive)

A short taxonomy of BGP failure

Four distinct failure shapes get lumped together as "a BGP problem," and they call for different mitigations. A misconfigured withdrawal is self-inflicted: an operator or, as the case study below shows, an automated tool removes an announcement it shouldn't have, and every downstream network loses its route almost immediately. A route leak happens when an AS announces routes it learned from one peer onward to another peer it had no business re-announcing them to — often because a routing "optimizer" appliance misbehaves — which can pull traffic through an unintended, often under-provisioned path, causing congestion or an effective blackhole for the leaked prefixes; a widely-cited 2019 incident involving a small ISP's route-optimization equipment leaking routes it had learned from larger networks caused reachability problems for a number of major cloud and CDN providers for roughly two hours, and is worth reading about directly from the affected providers' own public writeups for the precise mechanism. A BGP hijack is an AS originating a prefix it doesn't actually own — sometimes a fat-fingered typo, sometimes deliberate — which can silently redirect traffic meant for the legitimate owner to the hijacking network instead; well-documented historical examples include a 2008 incident where a small ISP's attempt to locally blackhole a video-sharing site accidentally propagated globally and made that site unreachable for a large part of the internet for a couple of hours, and a 2018 hijack of a cloud DNS provider's prefix that briefly redirected a cryptocurrency wallet service's traffic to an attacker-controlled server. And route flapping — a route rapidly appearing and disappearing, often from a flaky physical link — can degrade performance network-wide as routers repeatedly recompute paths, which is why most networks apply route-flap dampening to suppress an unstable announcement rather than propagate every flicker.

The mitigations that matter most for a team operating a service, rather than operating a network, are three. Multi-homing — announcing your own prefixes through more than one upstream transit provider — means a single provider's failure or misconfiguration doesn't remove your only path to the internet. External route monitoring — a third-party service watching global BGP tables for unexpected changes to your own announced prefixes — matters because your own internal monitoring depends on the same network path that might just have vanished; a system can't reliably alert on its own unreachability using infrastructure that routes through the very path that broke. And change-control discipline for anything that touches route announcements — staged rollout, an automated guardrail that validates a proposed change before it goes live, a fast rollback path — deserves exactly the same rigor as a database migration or an API schema change, a lesson the next section makes uncomfortably concrete.

Case study: how one withdrawn route took Facebook offline for six hours

☺ Like you're 10: A routine maintenance command, checked by an automated tool that had a bug in exactly the wrong place, deleted the internet's only road to Facebook's own DNS servers — and then made it hard to even get inside the building to fix it.

On October 4, 2021, Meta's production network experienced roughly six hours of total unreachability across Facebook, Instagram, WhatsApp, and Workplace — a scale of outage that made it one of the most closely examined public incidents in this course's case-study set. The proximate trigger, according to Meta's own public postmortem, was routine maintenance intended to assess the capacity of the global backbone network connecting Meta's data centers. A command issued as part of that maintenance had the unintended effect of withdrawing the BGP announcements for the IP prefixes hosting Meta's own authoritative DNS servers — and the audit tooling meant to catch exactly this class of unsafe command had a bug that let it through.

The consequences cascaded in a specific, instructive order. Once those prefixes were withdrawn, Meta's authoritative DNS servers became unreachable from the rest of the internet — not down, not overloaded, simply gone from every router's table, exactly as the section above describes. Every recursive resolver worldwide whose cached answer for facebook.com, instagram.com, or whatsapp.com had expired had nowhere left to ask, and lookups began failing globally. This is the layered-failure lesson in its purest form: Meta's application servers were, by every account, healthy and idle the entire time — the outage was never a capacity, code, or database problem. It was a DNS problem caused by a BGP problem, two layers below the application, that made perfectly functioning servers completely unreachable.

The recovery was complicated by a second-order effect worth remembering on its own: Meta's own internal tools — including, per public reporting at the time, systems used for physical badge access at data centers — depended on the same internal network and DNS infrastructure that had just gone dark. Engineers reportedly had to gain physical, on-site access to reset the affected systems because the usual remote tooling was itself a casualty of the outage it was needed to fix. The full narrative, including Meta's own account of the audit-tool bug and the remediation steps taken afterward, is worth reading directly — see Meta's 2021 BGP outage for the complete case study.

CDN and edge: isolating blast radius at the perimeter

☺ Like you're 10: A CDN is built to make single-location failures invisible — which is exactly why the failures that do get through tend to be the kind that hit every location at once.

A content delivery network operates dozens to hundreds of points of presence (PoPs) around the world, using anycast addressing so the same public IP is announced via BGP from many physical locations at once — a client's traffic simply routes, at the BGP layer covered above, to whichever announcing PoP is topologically nearest and currently reachable. Behind the edge, a well-designed CDN also runs an origin shield: a smaller, intermediate caching tier that coalesces cache-miss requests arriving from many edge PoPs into far fewer requests actually reaching your origin, so a popular piece of content going cold across hundreds of PoPs simultaneously doesn't turn into a request stampede against your origin servers — the same protective instinct as the jittered health checks from the load-balancer section, one layer further out.

Single-PoP failure is usually the good case

Because anycast relies on the same BGP mechanics covered above, a single PoP going offline — hardware failure, a local power event, a fiber cut — is usually handled gracefully and automatically: that PoP simply stops announcing its routes, and traffic that would have gone there gets picked up by the next-nearest healthy PoP through completely ordinary BGP convergence, often invisibly to end users. This is the CDN inheriting the graceful, contained failure mode that BGP is capable of when it's a routine withdrawal rather than a mistaken one.

The real risk is a global config push

The failure mode that actually produces headline-scale CDN outages isn't hardware — it's configuration. A WAF rule, an edge-compute function, a cache-key policy, or a TLS termination change typically needs to be consistent across every PoP simultaneously, for the same reason DNS needs one authoritative answer: a request landing on PoP A and a request landing on PoP B should behave identically. That consistency requirement means config usually propagates to the entire global network in one push, nearly simultaneously — so unlike a hardware failure, a bad config doesn't degrade gracefully. It goes from working to broken everywhere, at once. Cloudflare's July 2, 2019 outage is a well-documented example: a change to a WAF rule containing an inefficient regular expression caused CPU exhaustion on machines processing HTTP/HTTPS traffic across Cloudflare's entire network within minutes of the rule going live globally. Fastly's June 2021 outage is another: a latent bug in a service-configuration code path, triggered when a customer made a legitimate configuration change, caused a large share of Fastly's network to begin returning errors within about an hour. Verify the precise timelines and root-cause details against each provider's own public postmortem before citing exact figures — both are published and worth reading in full — but the structural lesson is identical in both cases and is the mirror image of the DNS TTL problem from earlier in this page: the same "push it everywhere at once, consistently" design that makes a CDN reliable in the normal case is exactly what removes the blast-radius containment during a bad change.

Two further mitigations round out CDN-layer resilience. Stale-while-revalidate and stale-if-error cache-control directives (RFC 5861) let an edge PoP keep serving a stale but still-plausible cached response when it can't reach or trust your origin, instead of turning an origin problem into a hard error for the end user — a direct CDN-layer analogue of the graceful-degradation patterns in reliability patterns. And for organizations where a CDN-level outage genuinely can't be tolerated, multi-CDN architectures — steering traffic across two independent CDN providers, via DNS weighting or client-side selection — add a final layer of redundancy at real, ongoing operational cost: keeping two providers' configurations equivalent, and maintaining the steering logic itself, is nontrivial extra work that only pays for itself once the cost of a CDN-level outage clearly exceeds it. Cloudflare's own extensive public postmortem history is worth reading end to end as a case study in doing incident transparency well; see Cloudflare's public postmortem culture.

Cache-Control: max-age=300, stale-while-revalidate=86400, stale-if-error=604800

# max-age=300            fresh for 5 minutes under normal conditions
# stale-while-revalidate serve the stale copy immediately while a
#                         background fetch quietly refreshes it, for
#                         up to 1 day past max-age — the user never
#                         waits on the revalidation
# stale-if-error          if the origin errors or is unreachable, keep
#                         serving the stale copy for up to 7 days
#                         rather than propagating the origin's failure

Building resilience across the whole stack

☺ Like you're 10: Four different layers, four different failure shapes — but the same handful of habits protect against nearly all of them.

Pulled together, the four layers on this page share a small set of design principles worth carrying forward as a checklist rather than four separate lessons.

Two habits apply across all four rows. First, monitor from outside your own network: DNS, BGP, and CDN-edge failures are precisely the class of problem your own internal monitoring, which depends on that same network path, is structurally poorly positioned to detect — a synthetic check running from independent, third-party vantage points around the world can see that your service is unreachable from the outside even while every internal dashboard reports green, because internal dashboards are asking a question ("is the application healthy?") one layer removed from the one that's actually broken ("can anything reach the application at all?"). Second, practice these specific failures, not just application-layer ones: a chaos or game-day exercise that blackholes a route in a test environment, forces a DNS provider to look unreachable, or kills a CDN PoP builds exactly the muscle memory Meta's engineers needed under real pressure — see chaos engineering and chaos engineering at scale for how to run that safely, and production readiness reviews for where "what happens to this service if its DNS provider disappears for an hour" belongs as a standing question before launch. Finally, because these failures are step functions rather than gradual degradations, make sure your burn-rate alerting windows are actually fast enough to catch an instant, total failure and not just a slow leak — the short window in that pattern exists precisely for failures shaped like the ones on this page.