Chaos Engineering & Game Days

Chaos engineering is the practice of deliberately injecting failure into a running system — killing an instance, blackholing a network path, starving a dependency of CPU — to find out whether it actually recovers the way everyone assumes it does, before a real outage forces the question on its own schedule. It traces straight back to Netflix's Chaos Monkey, but the discipline that grew out of that one script is bigger than any single tool: a set of principles for designing a real experiment, a body of fault-injection tooling for actually causing the failure, and a structured practice — the game day — for rehearsing the whole response with real people under real conditions. This page goes past "break things on purpose" as a slogan and into the mechanics: the Simian Army's actual roster, the principles a real chaos experiment has to satisfy, the fault types and tools you reach for, how to run a game day without becoming the outage you were trying to prevent, and why none of this is optional if you want your DORA change-failure rate to mean what everyone assumes it means.

A change-failure rate you've never tested is a guess, not a measurement

☺ Like you're 10: A 100% on a test you wrote yourself, that nobody else ever checked, doesn't prove you know the material — it might just mean you never asked yourself the hard question.

Measuring success: the DORA metrics defines change failure rate as failed deploys divided by total deploys over a period, with Elite teams landing in the 0–15% band. That formula has a quiet assumption baked into it: it only counts deployments whose failure mode actually got exercised during the measurement window. It says nothing about the failure modes that never had a chance to fire — the dependency that's never once timed out mid-release, the Availability Zone nobody's canary has lost, the retry logic nobody wrote that's simply never been asked to matter yet. A team can carry a low change failure rate for two very different reasons: because it's genuinely resilient — dependencies fail, traffic spikes, infrastructure disappears, and the system degrades gracefully or gets caught fast — or because it's simply never been unlucky yet. From a dashboard, those two teams are indistinguishable. From a pager going off at 2 a.m. for the first time in company history, they are not.

Chaos engineering and its structured cousin, the game day, exist to collapse that ambiguity on your own terms. Instead of waiting for a real dependency failure, a real zone loss, or a real traffic spike to arrive on its own schedule and reveal which team you actually are, you manufacture the same conditions deliberately, at a scale and on a day you choose, and watch what actually happens against the steady-state signals covered in monitoring & observability.

This is also the sharpest possible defense against Goodhart's Law gaming a metric that's supposed to matter. A change failure rate can drift low not just because a team ships carefully, but because it quietly avoids the situations that would expose fragility — batching risky changes together and logging them as one deploy, rarely touching the brittle legacy path, leaning on auto-scaling to silently mask a capacity problem that never registers as a "failed" deployment. Fault injection is the check against exactly that: it forces the failure paths to run even when a real deploy never would have exercised them, so the resilience your change failure rate implies has actually been demonstrated — not merely inferred from an absence of bad luck.

The Chaos Monkey lineage: from a shell script to a discipline

☺ Like you're 10: One team got tired of surprise outages, so they built a robot whose entire job was to cause small, planned ones on purpose — and the idea spread until it became its own field.

Netflix's move off its own datacenters and onto AWS began in earnest after an August 2008 database corruption incident that took the DVD-shipping service down for three days — a single point of failure the company decided never to be at the mercy of again. As engineering rebuilt the service on cloud infrastructure over the following years, the team reasoned that in a cloud environment individual instances fail routinely and unpredictably as a matter of course, so a service that couldn't tolerate losing one instance wasn't actually production-ready — it just hadn't found that out yet. Their answer, built around 2010, was Chaos Monkey: a tool that randomly terminates instances inside a production Auto Scaling group during business hours, so any instance-level fragility gets discovered by an engineer at their desk on a Tuesday afternoon with the whole team watching, instead of by whoever happens to be paged at 2 a.m.

Chaos Monkey wasn't a one-off script — Netflix grew it into a full toolkit it publicly called the Simian Army, open-sourcing Chaos Monkey itself in 2012. Each member automated a different, narrow way of finding fragility before a customer did.

The Simian Army solved the "how do you break things" problem; it didn't yet answer "how do you know if what you broke matters, and how do you do this responsibly." Around 2015–2017, members of Netflix's chaos and traffic teams — most notably Casey Rosenthal and Nora Jones — distilled the practice into a short public document, the Principles of Chaos Engineering, and later expanded it into the O'Reilly book Chaos Engineering (2020). That document, not any single tool, is what actually defines the discipline today, and it's why a Kubernetes-native tool with no shared code and a Netflix EC2 script from 2010 can both honestly claim to practice chaos engineering — they satisfy the same principles against different substrates. Chaos Monkey's own active development at Netflix has wound down since; its most actively maintained descendant today ships as Chaos Monkey for Spinnaker, distributed through the open-source Spinnaker release-pipeline project.

The principles of chaos engineering

☺ Like you're 10: A real experiment needs a hypothesis, a fair test, and a size limit — "we broke something and hoped for the best" isn't science, it's just an outage with extra steps.

Netflix's chaos team distilled the practice into a short set of tenets, published as the Principles of Chaos Engineering, that separate a real experiment from simply causing damage. Four define the experiment itself; a fifth, added as the practice matured into running against production, defines how to do it safely.

• Build a hypothesis around steady-state behavior. Steady state is a measurable output of the system's business behavior — checkout success rate, order throughput, login latency — never an internal signal like CPU utilization on its own. Define exactly what "the system is behaving normally" looks like before you touch anything, and state your hypothesis as a prediction that it will hold throughout the experiment.
• Vary real-world events. Inject the failures a system actually sees in production — instance and process crashes, malformed responses, traffic spikes, added network latency, DNS resolution failures, clock skew — not contrived scenarios nobody expects to happen for real.
• Run experiments in production. Staging never carries real traffic shape, real data cardinality, or the full real dependency graph. A resilience claim you only trust in staging is a resilience claim you don't actually trust.
• Automate experiments to run continuously. An experiment run once, by hand, six months ago describes a system that no longer exists — the code, the traffic, and the dependency graph have all moved on. Automate it so it runs again every time the system changes underneath it.
• Minimize blast radius. An experiment whose worst case could genuinely take down the whole service isn't an experiment — it's just causing the outage yourself. Scope every run so its worst plausible outcome is something the team can absorb without customer-visible harm.

Fault-injection testing in practice

☺ Like you're 10: There isn't one way to "break" a system — you can pull the plug, slow the wires, fill up the disk, or make one machine tell every other machine a lie, and each one finds a different weak spot.

"Chaos engineering" names the discipline; fault injection is the mechanism that actually does the breaking. Faults generally fall into four categories, and a mature practice deliberately works through all four rather than only ever pulling the one lever that's easiest to script.

A handful of tools have become the default way teams actually cause these faults, and each one targets a different substrate:

A Kubernetes-native experiment usually reads as a custom resource. Chaos Mesh's Schedule object wraps a fault type — here, killing a quarter of a target service's pods every Monday afternoon — so the experiment runs on its own without a human triggering it by hand each time, in the spirit of "automate experiments to run continuously":

apiVersion: chaos-mesh.org/v1alpha1
kind: Schedule
metadata:
  name: checkout-pod-kill-drill
  namespace: chaos-testing
spec:
  schedule: "0 14 * * MON"        # every Monday at 14:00
  type: PodChaos
  historyLimit: 5
  concurrencyPolicy: Forbid       # never let two runs overlap
  podChaos:
    action: pod-kill
    mode: fixed-percent
    value: "25"                  # blast radius: at most a quarter of the fleet
    selector:
      namespaces: [checkout]
      labelSelectors: { app: checkout }

Against AWS infrastructure directly, an FIS experiment template names its targets (which resources, and how much of them), its actions (what happens to them), and its stop conditions — a CloudWatch alarm that, if it trips, halts the experiment automatically, the same alarm-driven abort pattern Resilient Cloud Solutions covers for failover in general:

{
  "description": "Stop half the checkout Auto Scaling group for 10 minutes",
  "targets": {
    "checkoutInstances": {
      "resourceType": "aws:ec2:instance",
      "resourceTags": { "Service": "checkout" },
      "selectionMode": "PERCENT(50)"
    }
  },
  "actions": {
    "stopHalf": {
      "actionId": "aws:ec2:stop-instances",
      "parameters": { "startInstancesAfterDuration": "PT10M" },
      "targets": { "Instances": "checkoutInstances" }
    }
  },
  "stopConditions": [
    { "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:us-east-1:111122223333:alarm:checkout-error-rate-high" }
  ],
  "roleArn": "arn:aws:iam::111122223333:role/fis-checkout-experiment-role"
}

Both examples are provisioned the same way as everything else in this course's infrastructure as code — checked into version control, applied through the same pipeline, never clicked together by hand.

Running a structured game day

☺ Like you're 10: A game day is a fire drill for your whole team — you set the fire on purpose, on a day you chose, so everyone practices finding the exit while it's still safe to get it wrong.

A single automated fault-injection run tells you whether a system holds up. A game day is the larger exercise built around that same fault, deliberately including the humans and the process that would respond to it for real. Amazon's Jesse Robbins — an operations engineer who also worked as a volunteer firefighter — is generally credited with popularizing the structured GameDay inside Amazon in the mid-2000s: deliberately triggering a real, large-scale failure in production, on a scheduled day, to rehearse the response the way firefighters run a live burn rather than only ever reading the manual. Google runs an equivalent, company-wide practice called DiRT (Disaster Recovery Testing) — multi-day exercises that intentionally take down real infrastructure to find gaps in the response, not just the architecture.

A well-run game day moves through the same phases every time:

• Define the hypothesis and scope. One system, one steady-state signal, one specific fault, written down in one sentence before anyone touches anything — the same discipline the first principle above demands.
• Set success and abort criteria in advance. The exact number that ends the game day the instant it's crossed, agreed before the fault is injected, so nobody has to make that call live, under stress, with an audience watching.
• Assign roles. A facilitator runs the clock and makes the go/no-go call; a single named chaos injector is the one hand on the actual switch, so there's never ambiguity about who caused what; a scribe timestamps everything as it happens, which becomes the postmortem's timeline for free; observers watch dashboards and stay quiet unless asked; and the actual on-call responders react exactly as they would to a real page — that reaction is the entire point of the exercise.
• Decide announced or unannounced. An announced game day tests whether the system and its runbook hold up. An unannounced one — a genuine surprise sprung on the live on-call rotation — additionally tests whether the humans and the process hold up under real uncertainty. Both are legitimate practice; a surprise drill should still have a facilitator and a known abort condition watched by a "safety" observer, even though the responders themselves don't know it's a drill.
• Execute, then do nothing but watch. Inject the fault, then observe and timestamp until either the steady-state hypothesis breaks or the timebox ends — resist the urge to intervene early "just in case," or the experiment never actually answers its own question.
• Debrief. Run the exact same blameless structure as a real incident review (see culture & collaboration and incident management): what did the hypothesis get right, what surprised everyone, and which findings became a tracked follow-up action with a named owner and a date — not a "we should probably look at that" that quietly never happens.

Blast radius, safety rails, and abort criteria

☺ Like you're 10: A good chaos experiment has a big red "stop" button that anyone in the room can hit — and if you didn't build the button first, you didn't build an experiment.

Minimizing blast radius is a design decision made before an experiment ever runs, not a hope held during it. In practice it means targeting a small slice of the fleet rather than all of it — FIS's selectionMode: PERCENT(50) and Chaos Mesh's mode: fixed-percent in the examples above are both that same instinct expressed in two different tools' syntax — and, where possible, aiming an experiment at a canary population that already receives a small, deliberately carved slice of real traffic, the same isolation idea behind canary deployments in deployment strategies. Automatic stop conditions matter as much as the scoping: wiring an experiment's abort trigger to the same alarm that would already page a human, as FIS does natively through stopConditions, means the experiment can't outrun anyone's ability to notice it going wrong.

Netflix's own tooling evolved past the Simian Army for exactly this reason. Its Chaos Automation Platform (ChAP) runs experiments continuously against a small, statistically matched canary population receiving a real slice of production traffic, using automated regression detection to decide — without a human babysitting every run — whether to keep escalating an experiment's exposure or abort it immediately. That's the practical answer to the fourth and fifth principles together: how to run experiments in production, continuously, without a human having to watch every single one by hand. The abort criteria live in the platform itself, not in a person's judgment call under pressure.

The safety rails extend to people, not just traffic. Don't schedule a game day at 2 a.m. to be "more realistic," and don't spring a surprise drill on someone already carrying a rough on-call week — On-Call Culture & Sustainable Operations covers why treating a responder's time in a drill as anything less than real work undermines the confidence the drill was supposed to build. And once a fault is actually running, Distributed Tracing & Telemetry is usually what turns "something degraded" into "here's exactly which hop absorbed it and which one didn't."

Chaos maturity: from ad hoc drills to continuous verification

☺ Like you're 10: The goal isn't one big scary drill a year — it's making the drill so routine it stops being scary, the same way a real fire drill eventually becomes boring instead of terrifying.

Teams don't arrive at continuous, production-grade chaos testing on day one, and they shouldn't try to. A practical maturity model runs through five stages:

This closes the loop this page opened with. At level 0, a team's change failure rate — however good it looks on the measuring success dashboard — describes a system that has gotten lucky, not one that has been tested. At level 2 and above, the identical number describes something categorically different: a system whose failure paths get exercised on a schedule the team controls, not just on whatever schedule reality happens to choose. A byte-for-byte identical "Elite, 0–15%" reading means the team actually knows what happens when a dependency times out — not merely that it hasn't happened yet during the measurement window. Wiring these experiments into the release pipeline itself is exactly the ground Testing in the Pipeline covers, and a deliberate chaos experiment's simulated impact is a legitimate, planned draw against the same budget SLOs, Error Budgets & Toil sets aside for exactly this kind of controlled risk.

You now have the full shape of the practice: a lineage running from one Netflix script to a documented discipline, principles that separate a real experiment from simply causing an outage, the fault-injection tooling that actually causes the failure, the structured game day that rehearses the response around it, and the safety rails that keep all of it survivable. None of it is decoration — the DORA change failure rate this course already taught you to read is only ever as honest as the failure paths that have actually been exercised behind it.