Netflix & the Simian Army
This is a real, named company's story, and it sticks to what Netflix's own engineers have said about it in public — the Netflix TechBlog, conference talks, and Netflix's own retrospectives. The subject is Chaos Monkey and the menagerie of tools, the Simian Army, it grew into: not a one-time resilience audit or an annual disaster-recovery drill, but a continuous, scheduled habit of breaking production on purpose, during business hours, with no warning to the on-call engineer. The question this page is built around is the one that actually matters for your own systems: why did Netflix choose continuous chaos over a once-a-year fire drill, and what did that choice actually buy them the day a real AWS region had a bad day?
Imagine two schools preparing for a real fire. School A holds one fire drill a year — everyone knows it's coming, files out calmly, and forgets everything by month three. School B runs surprise drills constantly, all year, sometimes locking a random door or cutting the lights in one hallway on an ordinary Tuesday, just to see if people still know the way out. School B's drills are annoying and its teachers sometimes grumble. But when a real fire breaks out, School B's kids don't think, they just walk the route they've already walked fifty times — while School A's kids are still remembering where the exit sign was. Netflix is School B, and Chaos Monkey is the surprise drill it runs on its own computers, every single day.
The outage that made "hope" an unacceptable strategy
☺ Like you're 10: A real three-day breakdown convinced Netflix that just hoping its servers wouldn't fail wasn't good enough — so it decided to make them fail on purpose instead, while everyone was watching.
In August 2008, a database corruption incident took down the system that shipped physical DVDs — at the time, still Netflix's core business — for three days straight. Netflix engineers have retold this incident many times in public talks as the trigger for a hard conclusion: tightly coupled, vertically scaled infrastructure with no tolerance for a single bad component was a standing risk the company had been quietly carrying, not an acceptable cost of doing business. The response was to rebuild on Amazon Web Services, spread horizontally across redundant components instead of concentrated in one fragile chain. That decision fixed the coupling problem it was built to fix — and it created a new one nobody had to deal with running their own datacenter: on someone else's cloud, individual virtual instances disappear routinely and without warning, retired for maintenance, moved off a degrading host, lost to a transient fault, in a way a team owning a fixed rack of physical servers almost never experiences.
That second problem is the direct ancestor of everything on this page. Netflix could have responded to "instances now disappear unpredictably" the conventional way: write a runbook, document the expected behavior, and hope the redundancy held when it mattered. Instead, an internal team led by engineer Greg Orzell built a tool, starting around 2010, whose entire job was to make that unpredictable disappearance predictable — by causing it themselves, deliberately, before it could happen to them by accident.
Chaos Monkey: making the disappearance a habit, not an accident
☺ Like you're 10: Instead of waiting to lose a server by accident, Netflix built a robot whose entire job is to unplug one on purpose, during the day, so people are already watching when it happens.
Chaos Monkey's mechanism is almost aggressively simple: on a schedule, it picks a target at random from a pool of production instances and terminates it — not a simulated failure in a staging environment, a real kill of a real instance carrying real traffic. Netflix ran it internally for roughly two years before open-sourcing it on July 30, 2012, and the blog post announcing the release stated the underlying philosophy in one line that has outlived the tool's original Java codebase: the best defense against major unexpected failures is to fail often.
The logic behind that sentence is worth sitting with, because it's counterintuitive on first read. A team that loses an instance by accident once or twice a year treats that loss as an incident — rare enough that nobody's really sure the failover code still works, because it hasn't been exercised since the last person who wrote it left the team. A team that loses one every few days, on purpose, on a schedule, treats a lost instance as background noise its automation is expected to absorb without anyone noticing — and any code that can't survive that gets found and fixed within days, not discovered for the first time during a real outage at 3am. See the Chaos Monkey tool page for the actual termination mechanics, leashed/unleashed config, and how the tool evolved from a Java service riding alongside Asgard into a Go service integrated with Spinnaker.
Chaos Monkey doesn't test whether your application code can survive losing an instance — it tests whether your automation notices and replaces that instance correctly, every time, without a human in the loop. The instance dying is the boring, easy-to-simulate part. The actual experiment is whether the auto-scaling group's health check, launch template, and desired-capacity logic quietly do their job — or have silently rotted since the last time anyone actually watched them fire.
The Simian Army: a whole roster, not one tool
☺ Like you're 10: Netflix didn't stop at killing servers — they built a whole team of animal-named tools, each one hunting a different way the system could quietly go wrong.
Netflix's July 2011 TechBlog post "The Netflix Simian Army" laid out the broader suite Chaos Monkey had grown into by then — a family of tools, most running continuously rather than on Chaos Monkey's kill schedule, each aimed at a different failure mode or hygiene problem:
| "Monkey" | What it targets |
|---|---|
| Chaos Monkey | Randomly terminates individual production instances on a schedule — the founding member. |
| Latency Monkey | Injects artificial delay into service-to-service calls to see whether upstream callers degrade gracefully or fall over. |
| Conformity Monkey | Finds instances that don't follow best practice — not running inside an auto-scaling group, for example — and shuts them down to force a proper redeploy. |
| Doctor Monkey | Watches health checks and signals like CPU load, pulls unhealthy instances out of rotation, and eventually terminates the persistently sick ones. |
| Janitor Monkey | Finds and cleans up unused cloud resources quietly accumulating cost and clutter. |
| Security Monkey | Finds security-group misconfigurations and terminates the offending instances; also checks SSL and DRM certificates for validity. |
| 10-18 Monkey | Catches configuration and runtime problems in instances serving multiple locales — "10-18" is shorthand for internationalization (i18n) and localization (l10n). |
| Chaos Gorilla | Simulates the loss of an entire AWS availability zone, to confirm services rebalance automatically with no user-visible impact. |
Notice how little of that roster is actually about killing things. Most of the Simian Army is continuous hygiene and drift-detection — the same instinct that shows up in this course's toil & automation page, pointed at cloud configuration instead of manual runbooks. Chaos Monkey got the fame because "randomly terminates production servers" makes a better headline than "quietly checks SSL certificate expiry," but the roster as a whole is really one team's answer to a single question, asked continuously: what's quietly rotting in our fleet right now, that we won't find out about until it's an incident?
Escalating the blast radius: Gorilla, then Kong
☺ Like you're 10: First they practiced losing one computer. Once that felt boring, they practiced losing a whole building's worth. Once that felt boring, they practiced losing an entire city.
Chaos Gorilla, killing an availability zone, was already a meaningfully bigger step than Chaos Monkey, killing one instance — but Netflix escalated the idea once more with Chaos Kong, introduced in the TechBlog post "Chaos Engineering Upgraded." Rather than one instance or one availability zone, Chaos Kong simulates the loss of an entire AWS region: it evacuates live customer traffic away from the target region to Netflix's other regions and watches the aggregate metrics to confirm the failover is clean before reverting. This is the same blast-radius ladder taught in general form on chaos engineering — start at the smallest scope that teaches you something, escalate only once the smaller step is trusted — expressed here as three separate named tools, years apart, rather than one tool with a configurable severity dial.
Why continuous, not a once-a-year fire drill
☺ Like you're 10: A fire drill you only run once a year mostly tests whether people remember last year's drill — running it constantly, without warning, tests whether the exit route actually still works today.
This is the actual thesis Netflix's chaos-engineering story keeps proving, and it's easy to skim past in favor of the more quotable "they kill servers on purpose" headline. A traditional disaster-recovery test — the kind many companies still run — is scheduled, announced weeks in advance, staffed by people who know exactly what's about to happen, and performed maybe once a year. That has real value, but it tests one thing: whether the system, as it existed on the day of the drill, survives one scenario the facilitator chose in advance. Software doesn't hold still. Netflix ships changes constantly — new code, new dependency versions, new configuration, shifting traffic patterns — and every one of those changes can silently reintroduce a failure mode the last DR drill proved didn't exist eleven months ago. An annual test is stale again the moment the next deploy ships.
Chaos Monkey's answer was to make the test as continuous as the changes it was meant to catch. It runs constantly, on a schedule, unannounced to the specific engineer whose service gets hit that day, testing whatever code is running right now rather than whatever code was running on drill day. This is exactly the discipline later formalized as one of the four core tenets at principlesofchaos.org, the written framework Netflix's own chaos and traffic engineers — including Casey Rosenthal and Nora Jones — put together once the practice had matured: form a hypothesis around a system's normal ("steady-state") behavior, vary real-world events, run experiments in production, and — the tenet that matters most here — automate experiments to run continuously. Rosenthal and Jones later expanded the same discipline into the 2020 O'Reilly book Chaos Engineering. There's a cultural payoff too, distinct from the technical one: because failure is routine, scheduled, and blast-radius-limited rather than rare and catastrophic, it stops being a politically charged event nobody wants their name on and becomes an ordinary Tuesday — the same blameless instinct this course covers in full on postmortems & blameless culture. Netflix's own Chaos Automation Platform (ChAP), the successor to the original Chaos Monkey lineage, made the split explicit in practice: automated, continuous experiments as the daily default, with human-facilitated game days reserved for rehearsing organizational response rather than system behavior — the full distinction is covered in chaos engineering at scale.
The payoff: two real AWS regional dependency outages
☺ Like you're 10: Twice, a chunk of the internet Netflix depended on actually broke for real — and both times, Netflix mostly shrugged, because it had already rehearsed exactly that.
Rehearsal only matters if it changes what happens on the day it counts. Netflix's own public account credits its chaos practice with doing exactly that, twice, against two different real AWS regional failures years apart.
On April 21, 2011, a botched network configuration change during a routine upgrade inside a single US East availability zone triggered what AWS's own postmortem called a "re-mirroring storm" in its Elastic Block Store (EBS) service — a cascading failure that left a meaningful share of EBS volumes stuck and took down or degraded major sites, including Reddit, Quora, and Foursquare, for an extended stretch. Netflix's own TechBlog post on the incident, "Lessons Netflix Learned from the AWS Outage," said its systems felt comparatively little impact, and gave a specific architectural reason: Netflix avoided EBS as its primary data store, ran on Cassandra spread across multiple availability zones with no single point of failure, and — in the post's own words — was "designed explicitly for these sorts of failures," because that was precisely the class of failure the team had already spent time deliberately causing.
The second, more direct payoff came later. On September 20, 2015, Amazon's DynamoDB service suffered a multi-hour availability issue in US-EAST-1 that cascaded into more than twenty dependent AWS services and took down or degraded a long list of major sites for a six-to-eight-hour window. Netflix's own "Chaos Engineering Upgraded" post credits its regular, scheduled Chaos Kong exercises directly: because Netflix had been rehearsing full-region evacuation as a routine drill, not a hypothetical, the real outage produced only a brief availability blip rather than a customer-facing incident — a rehearsed failure mode meeting a real one and behaving exactly as practiced. Later TechBlog posts describe Netflix's traffic-engineering team continuing to invest in that same capability, cutting full regional-traffic evacuation time down from roughly an hour to a matter of minutes.
Contrast this with what happened to a much larger set of companies during the February 2017 AWS S3 outage in US-EAST-1 — covered in this course's own case study on that incident — where a single region's storage degradation took down status pages, dashboards, and entire products for hours, for teams that had never rehearsed losing that dependency. The difference between "brief blip" and "hours-long incident" wasn't luck on either side. It was almost entirely whether the failure had already been rehearsed on purpose before it happened for real.
The discipline that makes it safe, not reckless
☺ Like you're 10: "Randomly break production" sounds like the exciting part — the actual hard work nobody puts in the headline is making sure it never accidentally breaks something for real.
None of the above works, and none of it is responsible to copy, without the safety engineering underneath it — and this is the part that's genuinely least imitated when people retell the Chaos Monkey story. The principlesofchaos.org framework's other three tenets exist specifically to keep an experiment from becoming an accident: a falsifiable hypothesis stated before the experiment runs (not "let's see what happens," but a specific, measurable steady-state that should hold), variation limited to real-world events rather than arbitrary destruction, and — the discipline the framework itself calls out as advanced — deliberately engineering safer ways to run the experiment once a team has learned how much risk it actually carries. Blast-radius control did the rest of the work here: Netflix didn't wake up one day and unplug an entire AWS region. It spent years earning that step, one instance at a time, then one availability zone, before Chaos Kong was ever pointed at a full region — the exact progression covered in more depth on chaos engineering.
What to steal for your own systems
☺ Like you're 10: You don't need Netflix's size to copy the habit — start small, write the hypothesis down, and make it routine instead of an annual event.
- Make chaos routine, not an annual event. The entire lesson of this page collapses to one sentence: a resilience test that runs once a year tests last year's system. A test that runs continuously tests today's.
- Start at the smallest blast radius that teaches you something. One non-critical instance in a non-critical service, with an automated abort mechanism already tested — not an entire availability zone on day one. Chaos Monkey preceded Chaos Kong by years, not weeks.
- Write the hypothesis down before you break anything. "If this instance dies, the auto-scaling group should replace it within 90 seconds with no dropped requests" is falsifiable. "Let's see what happens" is not an experiment, it's an unscheduled incident with a witness.
- You don't have to build your own Chaos Monkey. The idea has since been productized: Gremlin and Litmus for Kubernetes-native fault injection, and AWS Fault Injection Service for AWS-managed experiments with the same blast-radius and abort-mechanism discipline built in. Try the whole loop yourself in this course's capstone chaos experiment or the shorter chaos-experiment-design drill.
- Rehearse the failure you're most afraid of, specifically because you're afraid of it. Region loss is rare. Netflix's regular Chaos Kong drills turned it into something the on-call team had already survived a dozen times in rehearsal — which is exactly why the real September 2015 outage was a non-event instead of a headline.
Honest caveats: what doesn't transfer
☺ Like you're 10: A twenty-year-old habit at one of the world's biggest engineering organizations doesn't fit in a weekend, and copying only the exciting part is the fastest way to hurt yourself.
- The published figures are self-reported, not independently audited. The "brief blip" characterization of the September 2015 incident and the region-evacuation timing figures come from Netflix's own engineering blog, not a third-party study. Treat them as "Netflix has said," not as a peer-reviewed result.
- Netflix's scale is genuinely close to unmatched. A dedicated chaos and traffic-engineering team, multi-region active-active infrastructure, and years of accumulated tooling sit underneath this story. A five-person startup adopting "randomly terminate production instances" as a Monday-morning practice, without the redundancy and automated failover Netflix built first, is copying the headline and skipping the foundation the headline depends on.
- Chaos engineering without the safety engineering is just an outage you scheduled yourself. The hypothesis discipline, the blast-radius limits, and the tested abort mechanism are the least headline-friendly part of this story and the part that actually made it survivable. Skip those and "we do chaos engineering now" means "we occasionally cause unplanned incidents on purpose."
- This is one company's public account of its own practice. Where Netflix's own retellings differ in small details across years of blog posts and talks, this page has favored the versions stated in Netflix's own TechBlog posts specifically about each incident, cited by name above.
Rocky the Raccoon: Netflix killed an entire AWS region on purpose. On a Tuesday. During business hours. That's the whole job I want.
Foxy: Why constantly, though? Most companies I've investigated run one disaster-recovery drill a year and call it done.
Professor Owl: Because the system running today isn't the system that existed on last year's drill day. Software changes every week — a test that runs once a year is stale again the moment the next deploy ships.
Timmy the Turtle: Rocky, before you touch anything — what's the hypothesis, and what's the abort mechanism if it's wrong?
Rocky the Raccoon: Fine. Kill one instance, checkout completion stays above 99%, rollback trips automatically if it doesn't. Start there, not with the whole region.
Sol the Sloth: And I already worked out how much error budget that one experiment is allowed to spend. It isn't much — which is exactly why you start small.
Where this connects in the course
☺ Like you're 10: This one company's story touches several other lessons — read whichever matches what you're building next.
The hypothesis-driven mechanics behind every experiment on this page — steady state, blast radius, abort mechanisms, the maturity path from staging to production — are covered in full on chaos engineering, with the automated-versus-game-day distinction ChAP made explicit on chaos engineering at scale. The tool itself, its config, and the roster it grew into live on the Chaos Monkey tool page. The multi-region architecture that made Chaos Kong's evacuation possible in the first place is multi-region & multi-AZ architecture, and the broader discipline of treating deliberate failure as a design input rather than a threat is anti-fragility & learning from failure. For a company that had the opposite experience during a real AWS regional dependency failure, see the 2017 AWS S3 outage.
1. What 2008 incident is publicly credited with pushing Netflix toward AWS, and what new problem did that migration introduce that Chaos Monkey was built to answer? 2. Name at least three members of the Simian Army besides Chaos Monkey, and what each one targets. 3. Put Chaos Monkey, Chaos Gorilla, and Chaos Kong in order of increasing blast radius, and say what each one destroys. 4. Why did Netflix run chaos experiments continuously rather than as an annual disaster-recovery drill — what's the core argument? 5. Name the two real AWS regional incidents this page credits Netflix's chaos practice with weathering well, and what Netflix's own account said made the difference each time.
Check your answers
- A three-day database-corruption outage that halted DVD shipping. Moving to AWS solved the tight-coupling problem behind that outage, but introduced a new one: on someone else's cloud, individual instances disappear unpredictably and without warning — the exact failure mode Chaos Monkey was built to force deliberately, on a schedule, instead of waiting for it to happen by accident.
- Any three of: Latency Monkey (injects artificial delay into service calls), Conformity Monkey (shuts down instances that don't follow best practice), Doctor Monkey (pulls unhealthy instances from rotation), Janitor Monkey (cleans up unused cloud resources), Security Monkey (finds security-group misconfigurations and expired certificates), 10-18 Monkey (catches internationalization/localization config problems), Chaos Gorilla (simulates loss of an entire availability zone).
- Smallest to largest: Chaos Monkey (one production instance) → Chaos Gorilla (one AWS availability zone) → Chaos Kong (an entire AWS region, with full customer traffic evacuated).
- Software changes constantly — new code, new config, new traffic patterns — so a resilience test performed once a year is already stale by the next deploy. Running the same test continuously, on whatever code is running today, is the only way the test stays true to the system that actually exists.
- The April 21, 2011 AWS US East EBS/"re-mirroring storm" outage, which Netflix's own blog post said caused comparatively little impact because its architecture avoided EBS as a primary store and was "designed explicitly for these sorts of failures"; and the September 20, 2015 AWS DynamoDB outage in US-EAST-1, which Netflix's "Chaos Engineering Upgraded" post credits its regular Chaos Kong region-evacuation drills with turning into a brief blip rather than a customer-facing incident.