Chaos Monkey
Chaos Monkey is the tool that gave chaos engineering its name: a Netflix-built service that randomly terminates production instances, on a schedule, with no warning to the on-call engineer. It doesn't simulate failure in a test environment and it doesn't wait for a rare event to happen on its own — it manufactures the exact failure every distributed system is supposed to already tolerate, continuously, in production, so that "the instance died" stops being a special case a team scrambles to handle and becomes an ordinary Tuesday the system was already built to survive. Everything on this page — the scheduler, the config, the Simian Army it grew into, and the tools that carried its idea into Kubernetes and beyond — is downstream of that one design choice.
A vaccine doesn't wait for you to catch the real disease. It gives your immune system a small, controlled, deliberately-triggered version of the threat, on a day when doctors are watching, so your body builds real defenses before the real disease ever shows up uninvited. Chaos Monkey does that for a computer system: instead of waiting for a server to die for real, at 3am, during the worst possible traffic spike, it kills one on purpose, during the day, while engineers are at their desks watching dashboards — so the system's "immune system," its automatic replacement and failover, gets exercised constantly instead of only when it's least convenient.
What Chaos Monkey is and the problem it solves
☺ Like you're 10: Netflix's own servers kept randomly disappearing once they moved to someone else's cloud, so instead of hoping that stopped, they started making it happen on purpose, during the day, so they'd find the weak spots themselves.
Chaos Monkey was born out of a specific, well-documented pain point in Netflix's history. In August 2008 a database corruption incident left Netflix unable to ship DVDs to members for three days — a failure traced back to tightly coupled, vertically scaled infrastructure with no tolerance for a single bad component. That incident is the reason Netflix committed to rebuilding on a horizontally scaled, redundant, cloud-based architecture on AWS, a migration that ran through the following several years. Cloud infrastructure solved the coupling problem, but it introduced a new one: on someone else's hardware, individual virtual instances disappear routinely and without warning — retired for maintenance, moved off a degrading host, lost to a transient fault — in a way a team renting a fixed rack of physical servers rarely experiences. Netflix's engineers reasoned, in a widely cited 2010 engineering-blog post on lessons learned running on AWS, that the correct response to "instances disappear unpredictably" wasn't better documentation about how the fleet should theoretically survive it — it was proof, and the proof had to come from continuously causing the exact failure being planned for.
Chaos Monkey is that proof, automated: a service that runs on a schedule, picks a target at random from a pool of production instances, and terminates it — not a simulation, not a staging environment, a real kill of a real instance serving real traffic. Netflix open-sourced it in 2012 as part of a broader release the engineering blog called the Simian Army, and the underlying philosophy is stated most cleanly in Netflix's own recurring line on the subject: "the best defense against major unexpected failures is to fail often." A team that only ever loses an instance by accident, once or twice a year, treats a lost instance as an incident. A team that loses one every few days on purpose treats a lost instance as background noise their automation is expected to absorb without anyone noticing — and any code that can't survive that gets found and fixed within days instead of surfacing for the first time during a real outage. See chaos engineering for the general discipline this tool founded, and Netflix & the Simian Army for the fuller organizational story of how it grew inside the company.
Architecture: two generations, one philosophy
☺ Like you're 10: The first version was a simpler helper bolted onto Netflix's own deploy tooling; the newer version is a proper service with its own database, wired into the same pipeline that ships Netflix's code.
Chaos Monkey exists as two architecturally distinct generations, and it's worth being able to tell them apart, because config and terminology from one doesn't map cleanly onto the other. The original, released in 2012 as part of Netflix/SimianArmy, is a Java/Groovy service that ran alongside Asgard, Netflix's deployment and cloud-management console of that era: it read a set of properties files, walked the AWS Auto Scaling Groups (ASGs) it had permission to see, and terminated one randomly chosen instance per eligible ASG per scheduled cycle. The current generation, a rewrite published at Netflix/chaosmonkey, is a Go service that instead integrates with Spinnaker — Netflix's own continuous-delivery platform, a different project from Argo CD though built for the same GitOps-adjacent delivery role — reading which applications have opted in through Spinnaker's application config and persisting its kill schedule in a MySQL database rather than local files. Both generations answer the same question the same way — pick a random instance from an eligible group, kill it, let the underlying scaling mechanism replace it — but the newer one is deeply coupled to running Spinnaker as your deployment platform, which is precisely why most organizations outside Netflix's own stack today reach for one of the tools in the comparison table near the bottom of this page instead of running either generation directly.
Chaos Monkey doesn't test whether your application can survive losing an instance — it tests whether your automation notices and replaces that instance correctly. The instance dying is the easy, uninteresting part; the actual experiment is whether the Auto Scaling Group's health check, launch template, and desired-capacity logic quietly do their job, or whether they've silently rotted since the last time anyone actually watched them fire.
The Simian Army: the roster it grew into
☺ Like you're 10: Netflix didn't stop at killing servers — they built a whole team of animal-named tools, each one hunting a different way the system could quietly rot.
Chaos Monkey was the first and most famous member of a broader suite Netflix called the Simian Army — announced together in 2011 on the Netflix engineering blog. Where Chaos Monkey's job is randomized termination, its siblings each target a different failure mode or hygiene problem, and several of them run continuously rather than on Chaos Monkey's kill schedule. Reading the full roster is the fastest way to understand how broad "chaos engineering" was meant to be from the start — it was never only about killing instances.
| Member | What it does |
|---|---|
| Chaos Monkey | Randomly terminates production instances on a schedule — the founding member and this page's subject. |
| Latency Monkey | Injects artificial delay into the RESTful calls between services, and can simulate a downstream dependency being fully down by returning errors instead of just being slow — testing timeout and fallback logic that instance termination alone never exercises. |
| Conformity Monkey | Finds instances that don't follow defined best practices — not part of an Auto Scaling Group, missing required tags, no defined health check — and shuts them down so the gap gets fixed rather than quietly tolerated. |
| Doctor Monkey | Taps into per-instance health checks and external signals like CPU load to find instances that are unhealthy but not yet dead, pulls them from service, and terminates them if the owning team doesn't act in time. |
| Janitor Monkey | Searches the environment for unused resources — orphaned volumes, stale snapshots, abandoned instances — and disposes of them, keeping cloud clutter and cost from silently accumulating. |
| Security Monkey | An extension of Conformity Monkey focused specifically on security posture — overly permissive security groups, misconfigured access rules, expiring SSL/TLS certificates — flagging or terminating the offending resource. |
| 10-18 Monkey | Short for Localization-Internationalization Monkey (from the abbreviations l10n and i18n) — detects configuration and runtime problems for instances serving customers across different regions, languages, and character sets. |
| Chaos Gorilla | Simulates the loss of an entire AWS Availability Zone, one level up in blast radius from a single instance. |
| Chaos Kong | Simulates the loss of an entire AWS region — the largest blast radius Netflix exercised, proving the multi-region failover story actually worked rather than just existing on a diagram. |
Notice the progression from Chaos Monkey through Chaos Gorilla to Chaos Kong: it's the same blast-radius ladder — instance, then availability zone, then region — expressed as three separate named tools rather than one tool with a configurable scope. That's a real architectural choice worth learning from even outside Netflix's own stack: making the blast radius part of the tool's identity, not just a config flag, makes it much harder for someone to accidentally point the AZ-killing tool at "just one instance" or vice versa.
The configuration you actually write
☺ Like you're 10: A handful of settings decide who gets picked on, how often, during which hours, and — the single most important one — whether it's pretend or real.
Both generations of the tool are configured, not scripted — you don't write chaos experiments as code the way you'd write a Terraform module; you set a handful of parameters and let the scheduler run unattended. The concepts below are stable across both generations even where the exact file format and key names differ.
# chaos.properties — the original Java/SimianArmy generation, representative shape simianarmy.chaos.enabled = true simianarmy.chaos.leashed = true # true = dry run: log only, never actually terminate simianarmy.chaos.ASG.enabled = true simianarmy.chaos.ASG.meanTimeBetweenKillsInWorkDays = 2 # ~ once every 2 business days, per eligible ASG # per-ASG opt-out — the default is "every eligible ASG is a target" simianarmy.chaos.ASG.checkout-prod.enabled = false
# chaosmonkey.toml — the current Go/Spinnaker generation, representative shape [chaosmonkey] enabled = true leashed = true # same dry-run concept, same name, carried over from the original schedule_enabled = true start_hour = 9 # business-hours window — no kills outside it end_hour = 15 time_zone = "America/Los_Angeles" [spinnaker] endpoint = "https://spinnaker.internal.example.com" [database] # the Go generation persists its kill schedule here, not in a local file name = "chaosmonkey"
Both examples above show the real, named concepts — leashed mode, mean-time-between-kills, a business-hours window, per-app opt-out — but the precise file format and key names have shifted across releases of both Netflix/SimianArmy and Netflix/chaosmonkey, and the original Java generation in particular has seen little maintenance in recent years. Treat the snippets above as illustrative of the shape, and check each project's current README on GitHub before configuring either one for real.
Three concepts do almost all the work, regardless of which generation you're reading about. Leashed mode is the dry-run switch — leashed, the tool only logs what it would have killed; unleashed, it makes the real API call. Mean time between kills is the randomization parameter — not a fixed daily kill, but a probability distribution that makes any given day's outcome genuinely unpredictable to the team on call, which is the entire point. And grouping — app, stack, or cluster — decides what counts as "one thing" for the purposes of a single day's kill: group at the cluster level and Chaos Monkey might terminate one instance from every cluster in a large application on the same day; group at the app level and it terminates at most one instance total for that whole application, regardless of how many clusters it has. Getting the grouping wrong is a quiet way to make your blast radius much bigger, or much smaller, than you intended.
Running it in practice
☺ Like you're 10: There's almost no button to press — you set the config, let it run in the background for weeks, and your job is mostly watching what it did rather than telling it what to do next.
Unlike a tool such as Helm or Terraform, Chaos Monkey has almost no day-to-day CLI surface, and that's deliberate: it's meant to run unattended, continuously, in the background, not be invoked interactively. The operational workflow in practice looks the same across both generations. First, opt an application in — through Spinnaker's application config for the current generation, or a properties file for the original — while leashed mode stays on, and watch the logs for a run or two to confirm the tool is correctly identifying the app, the grouping, and a plausible target instance without actually touching anything. Only once those dry-run logs look right do you flip leashed to false for that application and let it run for real, ideally starting with a single non-critical service before opting in anything that sits on the checkout path.
# the underlying AWS mechanism Chaos Monkey's own termination call is built on —
# useful to run by hand once, in a sandbox account, so the mechanic isn't a black box
$ aws autoscaling describe-auto-scaling-groups \
--auto-scaling-group-names checkout-prod-asg \
--query 'AutoScalingGroups[0].Instances[].InstanceId'
$ aws autoscaling terminate-instance-in-auto-scaling-group \
--instance-id i-0abc123def456789 \
--no-should-decrement-desired-capacity # false: the ASG must launch a replacement
# confirm the replacement actually landed
$ aws autoscaling describe-auto-scaling-groups \
--auto-scaling-group-names checkout-prod-asg \
--query 'AutoScalingGroups[0].[DesiredCapacity,Instances[].InstanceId]'That last flag is the mechanic worth internalizing even if you never touch Netflix's own code: --no-should-decrement-desired-capacity tells the Auto Scaling Group "one of your instances just died, replace it" rather than "reduce your target size by one." Get that flag backwards and Chaos Monkey silently shrinks the fleet every time it runs instead of testing whether the fleet heals itself — which is a good illustration of how a one-line config mistake can turn a chaos tool from a resilience test into the outage it was built to prevent.
Gotchas and failure modes
☺ Like you're 10: Most of the ways this goes wrong aren't the killing itself — they're a setting nobody double-checked, or a kind of failure the tool was never built to cause in the first place.
Leashed forever is a false sense of security
The single most common failure mode isn't a bad kill — it's a kill that never actually happens. A team enables Chaos Monkey, watches a few days of "would have terminated" log lines, feels good about the exercise, and never flips leashed to false. Months later everyone on the team believes their termination handling has been battle-tested, and nobody has actually verified it, because every single run was a dry run. Leashed mode is a staging step, not a destination — it belongs in the same category as an alert nobody ever actually got paged by: unproven right up until the moment it matters.
Business hours protect the team, not the coverage
Chaos Monkey's business-hours window exists so a human is awake, caffeinated, and at a keyboard when something goes wrong — a genuinely good safety practice. But it also means the tool never proves anything about instance loss at 2am, on a weekend, or during a holiday traffic spike, which is disproportionately when real unplanned failures actually happen. Confidence built entirely from business-hours experiments is confidence about business-hours resilience specifically; it says nothing about whether the on-call runbook and the automation still hold up when nobody's watching a dashboard live.
It only tests one failure shape
Chaos Monkey terminates a whole instance, cleanly. That's a narrow slice of how production actually fails: a disk filling up, a dependency responding slowly instead of erroring outright, a network partition that isolates one AZ without killing anything in it, memory pressure that degrades performance without crashing the process — none of those look anything like "the instance is suddenly gone." That gap is exactly why Netflix built Latency Monkey alongside it, and why every modern successor in the comparison table below ships a much broader fault catalog than clean termination.
It assumes an Auto Scaling Group–shaped world
Both generations of Chaos Monkey are built around the AWS ASG model: terminate an instance, let a scaling group notice and replace it. That maps naturally onto EC2 fleets and awkwardly onto everything else — a Kubernetes pod is not an ASG instance, a serverless function has no "instance" to terminate at all, and a stateful database node often shouldn't be killed the same casual way a stateless web server can be. Teams running mostly on Kubernetes reach for a Kubernetes-native tool instead, not because the philosophy doesn't apply, but because the termination mechanic literally doesn't exist in that world the way Chaos Monkey expects it to.
The original Java Netflix/SimianArmy generation has seen little active development in recent years, and the Go Netflix/chaosmonkey rewrite, while newer, is also not a fast-moving project — Netflix built both primarily to solve its own infrastructure problem, not to be a general-purpose product for other companies. Check each repository's recent commit history and open-issue activity before committing production infrastructure to either one; for most organizations today, one of the actively maintained tools in the comparison below is the more pragmatic starting point, with Chaos Monkey studied as the origin of the idea rather than run as the implementation.
In a disposable AWS sandbox account, create a tiny Auto Scaling Group with a minimum of 2 instances behind a load balancer health check. Run the describe-auto-scaling-groups and terminate-instance-in-auto-scaling-group commands from the section above by hand, exactly as shown, and watch the console: note the timestamp you killed the instance, and the timestamp a replacement's health check first passes. That gap — not the kill itself — is the real number a chaos experiment is trying to measure. Then run it again with --should-decrement-desired-capacity instead, on purpose, and watch the fleet shrink instead of heal. Tear the sandbox down when you're done.
Chaos Monkey vs. the modern chaos-engineering toolchain
☺ Like you're 10: Chaos Monkey started the idea, but most teams today use one of its more capable descendants to actually put it into practice.
Almost nobody outside Netflix runs Netflix's own Chaos Monkey code in production today. What nearly every serious chaos-engineering program does run is a tool built on the same philosophy, with a broader fault catalog, better guardrails, and active maintenance. Knowing the shape of that landscape matters more day to day than knowing Chaos Monkey's own config syntax.
| Tool | Model | Best when |
|---|---|---|
| Chaos Monkey | Random ASG-instance termination, scheduled, leashed/unleashed | Studying the origin of the discipline, or running an AWS/Spinnaker fleet where the simple original mechanic is genuinely enough |
| Gremlin | Commercial, agent-based "Failure as a Service" with a wide fault catalog (CPU, memory, disk, network black-hole/latency, process-kill, shutdown) and built-in game-day facilitation | An organization wants a managed, broad, guardrail-heavy platform and is willing to pay for it — Gremlin also runs its own chaos engineering certification |
| Litmus | CNCF, Kubernetes-native, ChaosEngine/ChaosExperiment CRDs, experiments published to a community ChaosHub | Kubernetes-first environments that want an open-source, declarative, CRD-driven experiment model — see Kubernetes reliability patterns |
| Chaos Mesh | CNCF sandbox, Kubernetes-native, CRDs for pod/network/IO/stress/DNS/time faults, with a web dashboard | Similar territory to Litmus with a different CRD design and fault taxonomy — the two are commonly compared head to head for a new Kubernetes chaos program |
| AWS Fault Injection Service | Fully managed AWS service, native to EC2/ECS/EKS/RDS and friends, with experiment templates and CloudWatch-alarm stop conditions built in | An AWS-native shop wants the safety rails (automated stop conditions) as a first-class managed feature rather than something the team has to build itself |
| kube-monkey | Open-source, direct spiritual successor to Chaos Monkey, purpose-built for Kubernetes — kills pods rather than EC2 instances, using the same opt-in-and-schedule philosophy | A team wants Chaos Monkey's exact original simplicity and randomization model, just retargeted at pods instead of instances |
The pattern across every row in that table is the same: each tool keeps Chaos Monkey's founding idea — inject real failure, continuously, so resilience is assumed rather than hoped for — and extends it along one axis Chaos Monkey itself doesn't cover: a broader catalog of fault types, a platform other than raw EC2 ASGs, or safety mechanisms (automated stop conditions, blast-radius limits) built into the tool instead of left to operator discipline. Chaos engineering at scale covers how a mature program actually chooses and layers these tools together rather than picking exactly one.
Where this fits in the SREF blueprint
☺ Like you're 10: The exam doesn't care whether you've run Netflix's own code — it wants you to recognize "this is the tool that started chaos engineering" and know what problem it was solving.
The DevOps Institute SRE Foundation (SREF) exam is closed-book and, as SRE Tools & Automation covers in full, tests tool categories and their originating philosophy rather than vendor-specific syntax. Chaos Monkey is the representative example for the chaos-engineering/fault-injection category, and its story is a direct, testable illustration of Anti-Fragility & Learning from Failure — the domain built around the idea that a system gets more reliable by being deliberately, safely broken, not by being protected from every possible failure. Know the origin story, the leashed/unleashed distinction as a stand-in for "dry run vs. real experiment," and the blast-radius ladder from instance to AZ to region that Chaos Monkey, Chaos Gorilla, and Chaos Kong embody as three separate tools — that's the transferable knowledge the exam is actually after.
Rocky the Raccoon: Checkout's been opted into Chaos Monkey for six weeks. Zero incidents. I'm calling it proven.
Timmy the Turtle: Zero incidents, or zero terminations? Those aren't the same sentence.
Rocky the Raccoon: ...let me check the config. ...It's still leashed. Every single one of those was a log line.
Foxy: So we've spent six weeks proving Chaos Monkey can identify a target correctly. We've proven nothing about whether checkout survives losing one.
Professor Owl: Then unleash it — on one instance, during business hours, with someone watching. That's the whole next step, not a bigger one.
Rocky the Raccoon: Fine. Flipping leashed = false for checkout-prod right now. If the ASG doesn't heal itself, we all find out together in about ninety seconds.
1. What specific 2008 incident set Netflix on the path that eventually produced Chaos Monkey, and what was the actual design response to it? 2. What is Chaos Monkey really testing when it terminates an instance — the application, or something else? 3. Name three other members of the Simian Army and what each one targets. 4. What does "leashed" mode do, and what's the most common way teams misuse it? 5. Why does an ASG-shaped tool like Chaos Monkey map awkwardly onto Kubernetes, and what's the direct spiritual successor built for that gap? 6. What's the honest reason most organizations today run a tool like Gremlin, Litmus, or AWS FIS instead of Netflix's own Chaos Monkey code?
Check your answers
- A database corruption incident left Netflix unable to ship DVDs for three days, traced back to tightly coupled, vertically scaled infrastructure. The response was migrating to a horizontally scaled, redundant architecture on AWS — which introduced routine, unpredictable instance loss as a new problem Chaos Monkey was built to force the team to confront continuously rather than occasionally.
- It's really testing the replacement automation — the Auto Scaling Group's health check, launch template, and desired-capacity logic — not just whether the application code can technically survive one fewer instance. The instance dying is the easy part; whether it gets correctly and automatically replaced is the actual experiment.
- Any three of: Latency Monkey (injects artificial delay or simulated outright failure into service calls), Conformity Monkey (finds and shuts down instances that don't meet best-practice standards), Doctor Monkey (finds unhealthy instances via health checks and load signals), Janitor Monkey (disposes of unused cloud resources), Security Monkey (finds security misconfigurations and expiring certificates), 10-18 Monkey (finds localization/internationalization problems), Chaos Gorilla (simulates an Availability Zone loss), Chaos Kong (simulates a full region loss).
- Leashed mode makes the tool log what it would have terminated without actually calling the termination API — a dry run. The most common misuse is leaving it leashed indefinitely: a team gets comfortable watching "would have terminated" log lines and never actually unleashes it, so nothing about real termination handling ever gets proven.
- Chaos Monkey's mechanic assumes an EC2 Auto Scaling Group — terminate an instance, let the ASG replace it — and Kubernetes pods aren't ASG instances, so that exact mechanic doesn't exist in a Kubernetes cluster. kube-monkey is the direct spiritual successor, built to kill pods using the same opt-in, scheduled, randomized philosophy.
- Netflix built both generations of Chaos Monkey to solve its own infrastructure problem, not as a maintained general-purpose product, and both — especially the original Java generation — have seen limited recent development. Tools like Gremlin, Litmus, Chaos Mesh, and AWS FIS are actively maintained, cover a much broader range of fault types than clean instance termination, and in several cases build safety mechanisms like automated stop conditions directly into the tool.