Meta's 2021 BGP Outage

On October 4, 2021, Facebook, Instagram, and WhatsApp disappeared from the internet at the same moment, for roughly six hours — not because any application server crashed, but because a routine network-maintenance command, and a bug in the tool meant to stop dangerous commands like it, withdrew the very routes that let the rest of the internet find Meta's own DNS servers. This page sticks to the mechanism Meta's own postmortem described and the sequence outside network observers watched unfold in real time, and it flags plainly where a detail comes from press or employee accounts Meta itself hasn't confirmed to the minute.

Six hours, three platforms, one command

☺ Like you're 10: One maintenance command near the core of Meta's own network took Facebook, Instagram, and WhatsApp offline together, for about six hours, on an ordinary Monday afternoon.

By outside accounts — network-monitoring firms and journalists tracking the event as it happened — Facebook, Instagram, WhatsApp, Messenger, and Oculus all became unreachable within minutes of each other starting in the mid-afternoon UTC on Monday, October 4, 2021, and stayed down for roughly six hours before service was fully restored that evening. Meta's own account, published the next day on its engineering blog under the title "More details about the October 4 outage" and attributed to Santosh Janardhan, then Meta's VP of Infrastructure, focuses less on to-the-minute timestamps and more on mechanism and sequence — so treat "roughly six hours" as the widely reported figure, and check Meta's own post if you need an authoritative number to cite. What made the incident unusual wasn't its length by itself; standalone six-hour outages happen elsewhere too. It was that three separate, massive consumer products with three separate engineering teams went dark at once, from a single change, and that Meta's own people spent a meaningful share of those six hours locked out of the tools they'd normally use to fix exactly this kind of thing.

The backbone, BGP, and an audit tool that didn't stop what it should have

☺ Like you're 10: Someone ran a routine check on how much spare network capacity Meta had — and a bug in the safety tool meant to catch mistakes let that check accidentally disconnect everything instead.

Meta's data centers talk to each other, and to the internet, over an internal backbone network — a set of routers and long-haul links that carry traffic between facilities at global scale. Per Meta's own account, an engineer was performing routine maintenance on that backbone, running a command intended to assess how much spare backbone capacity was available. Meta's infrastructure includes an audit tool specifically built to check commands like this one before they execute, and stop anything that could cause the kind of damage this one did. A bug in that audit tool meant it failed to do its job: instead of catching and blocking the command, it let it through, and the command took down every connection in Meta's global backbone at once, disconnecting Meta's data centers from the internet entirely.

That detail is worth sitting with on its own, before the DNS part of the story makes it worse: the audit tool existed. Meta hadn't skipped change control for backbone commands — it had built a specific safeguard for exactly this failure mode. The outage happened because that safeguard had a bug, not because nobody thought to build one. A change-safety mechanism that fails silently, once, at global scope, is functionally identical to having no change-safety mechanism at all for the one command that needed it most.

Why DNS died too: the safety mechanism that shared the outage's fate

☺ Like you're 10: Meta's own DNS servers are built to turn themselves off if they can't reach the rest of the network — which is a smart safety rule right up until the network itself is the thing that's broken, and turning off is the wrong move for everyone else on the internet.

Losing the backbone should, in principle, have been a Meta-internal problem — painful, but contained to Meta's own infrastructure. It didn't stay contained, and the reason is a genuinely elegant piece of engineering that turned into the visible failure mode. Some of Meta's facilities are responsible for answering DNS queries for facebook.com, instagram.com, and whatsapp.com — translating those names into IP addresses for the rest of the internet. Those authoritative name servers advertise their own reachability to the internet using BGP, the same routing protocol that connects the backbone. As a safety measure, Meta's DNS servers are designed to withdraw their own BGP advertisements if they can't reach Meta's data centers — the logic being that an unreachable DNS server shouldn't keep telling the internet "ask me," because it can't actually answer. When the backbone dropped, every one of those DNS locations correctly concluded it was unhealthy and withdrew its own BGP routes, exactly as designed. The result: Meta's DNS servers were still physically running, but nothing on the internet could find them anymore, because the routes that said "traffic for these IP addresses goes here" had just been pulled. Networking teams at other companies watched it happen in close to real time — Cloudflare's own engineering blog, published the same day, described watching Meta's BGP route announcements disappear from the global routing table and DNS query volume to Cloudflare's own public resolver spike as clients everywhere kept retrying lookups that could no longer succeed.

This is the detail that makes the incident worth a full page in an SRE course rather than a footnote: the mechanism that broke DNS wasn't a bug in DNS. It was a correctly functioning safety feature, responding correctly to a real problem, whose correct response happened to be exactly the wrong thing for the rest of the internet in this specific failure mode. See network reliability engineering for how BGP, DNS, and backbone design fit together in more depth, and dependency management & Hyrum's Law for how a "safety mechanism" quietly becomes a load-bearing dependency nobody mapped.

Locked out of the building: the blast radius nobody had mapped

☺ Like you're 10: The tools Meta's own engineers needed to fix the network, and even the door badges that let them physically walk in, turned out to depend on the exact same network that had just gone down.

The part of this incident that made it a genuinely hard six hours, rather than a fast rollback, is what happened once Meta's own responders tried to act. Per Meta's own postmortem, both the primary remote-access path engineers would normally use to reach backbone routers, and the independent out-of-band path built specifically as a fallback for this kind of scenario, were unavailable — because both, in different ways, still depended on infrastructure the backbone outage had just taken down. With no working remote path in, Meta sent engineers physically on-site to the affected data centers to debug the routers and restart systems by hand. Meta's own account is candid that this took real time, because those facilities are deliberately built with strong physical and system security — hard to enter, and with hardware deliberately hard to reconfigure even once you're standing in front of it — protections that exist for good reason on every other day and worked directly against the recovery on this one. Multiple outside reports from that week, including contemporaneous employee accounts reported in the press, described internal collaboration and communication tools also being affected, since those tools were themselves built on Meta's internal network and DNS infrastructure — complicating coordination during the same hours coordination mattered most. Treat that detail as widely reported rather than a line from Meta's own postmortem specifically, since Meta's own post focuses on the network mechanism rather than a tool-by-tool account of what employees experienced.

Recovery: slow, deliberate, and physically constrained

☺ Like you're 10: There was no button to press from a laptop at home — someone had to physically walk into a guarded building, reach the actual hardware, and bring the network back up carefully so the flood of everyone trying to reconnect at once didn't cause a second collapse.

Once engineers regained physical access and control of the backbone routers, restoring service wasn't a single switch flip. Bringing global backbone capacity back online had to be done carefully rather than all at once — reintroducing routes and capacity too quickly risks a fresh overload as every dependent system, plus every client device across the internet that had spent the last several hours retrying failed DNS lookups, tries to reconnect and resolve at the same instant. That kind of retry-storm dynamic is the same class of problem this course covers generally under thundering-herd and backoff behavior; recovering from a global backbone-and-DNS failure means recovering from the outage and from the surge of pent-up demand the outage itself created, without one causing a second incident on top of the first. Outside monitoring accounts describe DNS and BGP activity for Meta's domains beginning to stabilize in the evening UTC that day, consistent with the roughly six-hour figure widely reported for the outage's total duration.

What Meta said it changed

☺ Like you're 10: Meta said it would make the safety-check tool actually stop dangerous commands like this one, and build a way to fix the network in an emergency that doesn't depend on the network already working.

Meta's own postmortem described the response in terms of hardening the specific mechanisms that failed, rather than a single fix. It said the company was working to audit and strengthen the tooling and processes that validate changes before they're allowed to run against the backbone, specifically so a command with this kind of blast radius can't execute without being caught first — closing the gap the audit-tool bug opened rather than assuming a second bug like it won't happen. It also described investing in more resilient ways to recover from this class of event in the future, aimed squarely at the chicken-and-egg problem that stretched the incident to six hours: an emergency access path that only works when the thing it's meant to fix is already partly working isn't a real emergency path. Meta additionally stated that it had found no evidence user data was compromised during the outage — a distinct claim from the availability failure itself, and one worth separating clearly: this was a reachability incident, not reported as a data-security one.

The lesson for SRE: an emergency path that depends on what it's rescuing isn't one

☺ Like you're 10: If your plan for "the network is broken" secretly needs the network to be working, you don't actually have a plan — you have a wish.

Strip away the specific company and product names, and this incident is a clean illustration of two ideas this course treats as central. The first is blast radius: a change's real blast radius is set by everything that transitively depends on what it touches, not by what the change's author believed they were touching. The engineer running a capacity-assessment command almost certainly wasn't reasoning about DNS, internal tooling, or building badge readers at all — and didn't need to be, because the audit tool existed specifically so no single human had to hold that entire dependency graph in their head at the moment they typed a command. The second is independent recovery paths: a fallback, an out-of-band access mechanism, or a break-glass procedure is only as independent as its own dependency graph, and that graph needs to be tested against the specific failure it's meant to survive, not just documented and trusted. See disaster recovery & business continuity for how to design and actually test an out-of-band path that doesn't quietly share fate with production, and production readiness reviews for catching this exact class of hidden shared dependency before a system as consequential as an audit tool or a DNS server ever ships.

Applying the five whys (an illustrative reconstruction from the
public record, not a document Meta itself published this way):

Symptom: Facebook, Instagram, and WhatsApp were unreachable for
         roughly six hours, and Meta's own engineers struggled to
         intervene quickly.

Why?    Because facebook.com, instagram.com, and whatsapp.com
        stopped resolving anywhere on the internet.
Why?    Because Meta's authoritative DNS servers withdrew their own
        BGP route advertisements — a built-in safety response to
        losing reachability to the backbone.
Why?    Because a routine backbone-maintenance command withdrew
        every backbone connection at once, disconnecting Meta's
        data centers from the internet entirely.
Why?    Because the audit tool built specifically to catch and stop
        commands like this one had a bug that let it through.
Why?    Because recovery then required physical, on-site
        intervention — primary remote access, out-of-band access,
        and even building entry all depended, directly or
        indirectly, on the same infrastructure the command had
        just broken.

Root cause: not one bug, but a change-safety and recovery design
that assumed some path back in — remote access, DNS, a badge
reader — would survive any command the audit tool failed to catch.
None of them were actually independent of the system a single
command had just taken down.

What to steal for your own systems

☺ Like you're 10: You don't need Meta's scale to make the same mistake — so borrow the fixes at whatever scale you actually run at.

• Map your break-glass path's own dependencies, on paper, before you trust it. Ask specifically: does this emergency access mechanism route through, authenticate against, or resolve names using anything the failure it's meant to survive could also take down? If the answer is yes for any link in the chain, it isn't independent.
• Treat "internal" and "low blast radius" as two separate claims. An audit tool, a DNS server, or a config-validation service with zero external users can still be the single point of failure for everything that depends on it existing — including, in this case, the very tools meant to fix it.
• Test your out-of-band access path the way you'd test a failover, not just document it. A break-glass procedure nobody has exercised against a real failure of its primary path is a hope, not a control — the same discipline this course covers on chaos engineering applies directly to disaster-recovery tooling itself.
• Give change-safety tooling the same rigor as the systems it protects. An audit or validation tool that can wave through a command with this much blast radius needs its own production readiness review, its own testing, and ideally a second, independent check — not just trust that it will always catch what it's built to catch.
• Rehearse losing your own naming and routing layer specifically. Most game days assume DNS and internal routing still work while some other component fails. This incident is the reminder to occasionally assume they don't, and see what's actually left standing.

Honest caveats: Meta's account vs. outside reporting

☺ Like you're 10: Some of this story is Meta's own official explanation, and some of it is what outside reporters and network watchers pieced together — this page tries not to blur the two.

• The mechanism is Meta's own account. The backbone-audit-command sequence, the DNS self-withdrawal explanation, and the statement that no user data was found to be compromised all come from Meta's own published postmortem. That's the part of this page you can cite with confidence, and the part worth reading in Meta's original words if you're using this case in a talk or a write-up of your own.
• Exact timestamps and total duration vary slightly by source. Outside network-monitoring accounts and press coverage converge on "roughly six hours" and an afternoon-UTC start, but Meta's own post is not built around a minute-by-minute timeline. Don't cite a specific minute figure as Meta's own without checking the original post first.
• Financial-impact figures are outside estimates, not a Meta-disclosed number. Various analysts and outlets published same-day estimates of lost revenue in the tens of millions of dollars and noted a same-day drop in the company's stock price; Meta itself has not published an official damage estimate for the outage. Treat any specific dollar figure you see quoted as press estimate, not company data.
• Employee-experience details are widely reported, not all Meta-confirmed. Accounts of internal communication tools being affected, and of engineers needing extraordinary means to coordinate and gain building access, are drawn from contemporaneous press coverage and employee accounts from that week, not itemized in Meta's own postmortem. The broad shape — internal tooling shared the network dependency that took down the public products — is well corroborated; specific anecdotes should be attributed to their original reporting, not to Meta directly.

Where this connects in the course

☺ Like you're 10: This one incident touches several lessons — read whichever matches what you're building next.

The networking mechanism itself — BGP, DNS, and how a backbone actually connects data centers to the internet — is covered in full on network reliability engineering. The discipline of mapping a fallback's own dependencies before trusting it is disaster recovery & business continuity, and the broader habit of finding a hidden shared dependency before it becomes an incident is dependency management & Hyrum's Law and production readiness reviews. Coordinating a response at the scale this incident required — multiple products, multiple teams, hours of physical constraints — is incident command for large-scale incidents, building on this course's own incident management & on-call page. And deliberately rehearsing the loss of your own naming and routing layer, rather than assuming it as a constant, is a direct extension of chaos engineering and chaos engineering at scale. For how a different company handled — and openly published — its own infrastructure failures, see Cloudflare's public postmortem culture, whose engineering blog is itself one of the outside sources that documented this incident in real time.