The 2017 AWS S3 Outage

This page sticks closely to one primary source: Amazon's own "Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region," published at aws.amazon.com/message/41926/ shortly after the event. On February 28, 2017, a single incorrectly entered input to a routine, playbook-driven debugging command removed far more server capacity than intended from Amazon S3's US-EAST-1 region — and the two subsystems that lost that capacity happened to be the ones nearly every S3 request depends on. What followed is one of the clearest public examples of three separate SRE lessons landing in the same afternoon: how little margin a "routine" operational command can have without a guardrail, how long a large stateful system can take to come back once it actually has to restart from scratch, and what happens when the system you'd use to tell customers something is wrong depends on the very thing that's wrong. Figures on outside financial impact and affected-company counts in this page are industry estimates from third-party analysts, not numbers AWS has published or audited — this page says so plainly wherever they appear.

What actually happened, in one paragraph

☺ Like you're 10: Someone typed one number wrong while fixing something unrelated, and it accidentally cleared out way more than planned — including the two systems that basically everything else needed to work at all.

At 9:37 AM PST on February 28, 2017, an authorized Amazon S3 team member was debugging an issue that was causing the S3 billing system to process more slowly than expected. Using an established playbook, they executed a command intended to remove a small number of servers for one of the S3 subsystems used by the billing process. One of the inputs to that command was entered incorrectly, and a much larger set of servers was removed than intended. The servers that were mistakenly pulled out of service happened to support two other S3 subsystems, not the billing one the engineer was actually working on: the index subsystem, which manages the metadata and location information for every object stored in S3 in the region, and the placement subsystem, which manages allocating storage for new objects and depends on the index subsystem to do it correctly. Losing that much capacity all at once didn't just slow those two subsystems down — it forced both into a full, from-scratch restart, and during that restart S3 could not reliably serve requests in the region at all.

Why two subsystems, and why that combination was so damaging

☺ Like you're 10: One system is the "where is everything" map, and the other is "where do I put the next thing" — lose the map and you can't even use the second system correctly.

This is the part of the incident that turns a capacity mistake into a multi-hour outage rather than a brief blip: S3's own summary states plainly that these two particular subsystems had not needed a full restart in many years, and that S3's metadata volume had grown enormously in that time. Restarting them wasn't a matter of flipping the servers back on — it meant reloading and re-validating a much larger body of metadata than these subsystems had ever had to reload before, plus running the safety checks needed to confirm that metadata's integrity before trusting it again. AWS's own account is direct about the consequence: that combination of scale and infrequent restart meant the process "took longer than expected."

The blast radius: how far one region's trouble actually reached

☺ Like you're 10: Because so many other AWS tools and outside companies quietly depend on S3 without anyone thinking about it day to day, one region's bad afternoon became everybody's bad afternoon.

S3 outages don't stay contained to S3, because so much else is quietly built on top of it. AWS's own summary lists Amazon EC2's ability to launch new instances, Amazon EBS's ability to complete volume operations that needed data from S3-backed snapshots, and AWS Lambda as directly impacted in the region while S3 APIs were unavailable — none of those services are S3 themselves, but all three lean on it for specific operations. Outside AWS, a wide swath of the public web that stores images, static assets, or application data in S3 either broke outright or degraded for the roughly four hours the region was affected — contemporary reporting from that day named services including Quora, Medium, Slack, Trello, Docker Hub, Coursera, Expedia, and Business Insider among the sites and apps visibly affected, alongside numerous smart-home and IoT products that used S3 for configuration or control and stopped responding. Cyber-risk analysis firm Cyence estimated in the days after the outage that S&P 500 companies alone lost roughly $150 million from the disruption, with U.S. financial-services firms accounting for an estimated $160 million more — figures widely reported at the time as a combined estimate near $310 million. Treat those specific dollar figures as one analyst's modeled estimate, not an AWS-published or independently audited number; AWS's own summary does not put a financial figure on the incident at all.

The detail everyone remembers: AWS's own status page couldn't say AWS was down

☺ Like you're 10: The "everything's fine" sign was also plugged into the thing that broke, so for the first two hours the sign just... stayed stuck on "fine."

Per AWS's own summary, the Service Health Dashboard's administration console — the tool AWS engineers use to update the little status indicators customers check during an incident — itself depended on S3 for storing the images and other content it needed to render, and it was hosted in the very region that had just gone down. The result: AWS could not update the individual services' status on its own public dashboard from 9:37 AM PST, when the incident began, until 11:37 AM PST — two full hours into a roughly four-hour event. During that window, AWS communicated instead through Twitter and, once it could, a banner posted directly on the dashboard explaining that the lack of a status change was itself related to the S3 issue. For an industry whose default instinct during an outage is "check the status page," the status page saying nothing was wrong while a large fraction of the internet visibly was, was the detail that made this outage as memorable as it was.

The recovery, hour by hour

☺ Like you're 10: It wasn't instant and it wasn't all-at-once — different pieces came back at different times over roughly four hours.

The timeline below is drawn directly from AWS's own published summary. Treat the specific minute marks as AWS's own account, not an independently timestamped reconstruction — this is exactly the kind of figure the brief for this page asked to be verified against AWS's own words rather than secondhand retellings, and secondhand retellings of this incident have varied by a few minutes here and there over the years.

9:37 AM PST   Command executed with an incorrect input; capacity removal begins
9:37–11:37 AM Service Health Dashboard admin console can't update — it too
              depends on S3, hosted in the same affected region. AWS
              communicates via Twitter and a manual dashboard banner instead
12:26 PM PST  Index subsystem recovers enough to serve GET, LIST, and
              DELETE requests again
1:18 PM PST   Index subsystem fully recovered
1:54 PM PST   Placement subsystem recovers; S3 reports full operation
              region-wide — roughly 4 hours 17 minutes after 9:37 AM

What AWS said it changed afterward

☺ Like you're 10: AWS didn't just apologize — they changed the tool that let this happen, split the risky system into smaller pieces, and fixed the sign out front so it can't get stuck again.

AWS's own summary lists specific, concrete changes made in direct response to this incident, not general reassurances:

• Slower, safer capacity-removal tooling. The tool used to remove capacity was modified to remove resources more slowly, and to include safeguards that prevent capacity from being removed if doing so would take any subsystem below its minimum required capacity level — the specific guardrail that would have prevented this exact mistake from cascading, whatever the input error had been.
• An audit of other operational tools for the same gap. AWS stated it audited related operational tools to ensure similar safety checks — minimum-capacity floors, in effect — existed elsewhere, rather than assuming this was a one-off tool with a one-off problem.
• Smaller partitions to shrink future blast radius. AWS said it would accelerate existing plans to partition the S3 index subsystem into smaller cells, so that a future capacity event, whatever caused it, would affect a smaller portion of the system and both fail and recover faster.
• A status dashboard that no longer shares a fate with the thing it reports on. AWS committed to changing the Service Health Dashboard so it runs across multiple, separate AWS regions — directly targeting the failure mode described above, so a single region's outage can no longer take the status page down with it.

What to steal for your own systems

☺ Like you're 10: You don't need to run anything at AWS's scale to copy these habits — put a floor under your dangerous commands, and make sure your "everything's fine" sign doesn't share a fate with everything.

• Put a minimum-capacity floor on anything that removes resources. A command that can remove an unbounded amount of capacity from a critical subsystem, with no automated check on how far is too far, is one mistyped input away from this exact incident — regardless of how experienced the person running it is, or how well-established the playbook is. This is precisely the toil-to-automation gap covered on toil & automation: a manual, playbook-driven command with no guardrail coded into it is still, in an important sense, unreviewed.
• Test your restart time before you need it, not during the incident. The index subsystem's restart took longer than expected specifically because it hadn't been fully restarted in years while its data kept growing — nobody had current, tested evidence of how long that restart would actually take under real load. See capacity planning & performance and chaos engineering at scale for exercising recovery paths on a schedule, not by accident.
• Never let your status or paging system share a failure domain with what it watches. Host your status page's assets, your alerting pipeline, and your incident-communication tooling somewhere structurally independent from the production systems they report on — a different region, a different provider, or at minimum a genuinely separate dependency chain. This is the one line item from AWS's own remediation list worth copying verbatim.
• Design your blast radius on purpose. Partitioning the index subsystem into smaller cells after the fact is the same idea as choosing deliberate failure domains before the fact — see multi-region & multi-AZ architecture and distributed systems reliability fundamentals for how to decide, in advance, how much of your system a single mistake is allowed to take down.

Honest caveats: what doesn't transfer cleanly

☺ Like you're 10: AWS's own summary is real and specific, but it also leaves a few things out on purpose — this page doesn't fill those gaps with guesses.

• AWS never disclosed exactly how many servers were removed, or any detail about the engineer. The summary says "a larger set of servers was removed than intended" without a number, and identifies no individual — this page follows that same restraint rather than inventing a specific figure or a name that AWS chose not to publish.
• The financial-impact figures are third-party modeling, not AWS's own numbers. Cyence's roughly $150 million (S&P 500) and $160 million (financial services) estimates are widely cited in press coverage of this incident, but they're a risk-modeling firm's after-the-fact estimate, not a disclosed or audited AWS figure, and no comparably rigorous, independently verified total exists.
• The affected-company list is illustrative, not exhaustive. Any list of "who was down" compiled from contemporary reporting reflects what journalists happened to notice and confirm that day, not a complete inventory of every service that degraded — many smaller and internal-facing systems affected by the outage were never individually reported on at all.
• This was US-EAST-1 specifically, at 2017-era S3 architecture and scale. AWS has made the exact remediations described above since this incident — including the index-subsystem repartitioning and the multi-region status dashboard — so treat this case as a historical example of the failure mode, not a live description of what would happen if the same mistyped command were run today.

Where this connects in the course

☺ Like you're 10: This one afternoon touches several other lessons — follow whichever one matches what you're actually building.

The missing guardrail at the center of this incident — a manual, playbook-driven command with no automated floor on how much damage it could do — is exactly the territory of toil & automation. The shared-fate failure in AWS's own status dashboard is a direct, real-world instance of the design principles on monitoring & observability. How a region-wide event like this one should actually page someone, and how the response gets structured once it's declared, is incident management & on-call, and this course's own worked example of writing the resulting postmortem is postmortems & blameless culture and the course's own incident case study. The architectural lesson about not concentrating an unreasonable share of dependencies in one region lives on multi-region & multi-AZ architecture, disaster recovery & business continuity, and distributed systems reliability fundamentals. For two other real companies' outages with their own distinct lessons, see Slack's January 2021 outage and Meta's 2021 BGP outage; for how a company chose to talk about incidents like this one in public afterward, see Cloudflare's public postmortem culture. And for the operating-model view of the specific anti-pattern this incident illustrates — tooling with no blast-radius limit — see SRE anti-patterns & pitfalls.