Engineering for Reliability · Chaos engineering

Chaos engineering

Every production system depends on assumptions about how its parts fail — assumptions nobody has actually tested. Chaos engineering is the discipline of testing them on purpose, by injecting real failure into a running system and observing whether it degrades the way you designed it to. By the end of this page you'll know where the practice came from, how to write a chaos experiment that actually proves something, and how to run one without causing the outage you're trying to prevent.

Origin: Chaos Monkey and the Simian Army

Chaos engineering as a named discipline traces to Netflix's migration to AWS around 2010. Running on someone else's infrastructure meant instances could and did disappear without warning — a host could be retired, an availability zone could degrade, a network partition could isolate a service from its dependencies — and Netflix's engineers realized that a system which merely could tolerate that was not the same as a system that had been shown to tolerate it. Their answer was Chaos Monkey: a tool that randomly terminates production instances during business hours, on a schedule, with no warning to the team on call.

The insight behind Chaos Monkey is a forcing function, not a stress test. If instance termination is rare and unpredictable, engineers write code that assumes instances live forever, and that assumption goes untested until the day a real termination cascades into an incident. If termination happens constantly and on purpose, resilience to it stops being optional — code that can't survive an instance disappearing gets found and fixed immediately, because it fails every few days instead of once a year. Netflix extended the idea into the "Simian Army": Latency Monkey injected artificial delay into service calls, Conformity Monkey found instances that didn't meet best-practice standards and shut them down, Chaos Gorilla killed an entire AWS availability zone, and Chaos Kong simulated the loss of a full AWS region. Netflix open-sourced the tooling (Chaos Monkey itself, and later the broader Chaos Monkey for Spinnaker), and the practice generalized into what the industry now calls chaos engineering, formalized in the 2020 O'Reilly book of the same name by Casey Rosenthal and Nora Jones, both former Netflix chaos engineers.

The core principle: hypothesis-driven experiments

The single most important discipline in chaos engineering is that an experiment starts with a specific, falsifiable hypothesis about steady-state behavior — not a vague intention to "see what breaks." A hypothesis names the failure you're injecting, the dependency or component it targets, and the exact user-facing behavior you expect to hold anyway. "If the recommendations service times out, checkout should still complete, degraded to a static fallback list" is a real hypothesis: it's specific, and it's falsifiable, because you can run the experiment and get a clear yes-or-no answer. "Let's see what happens if we kill the recommendations service" is not a hypothesis, it's an unplanned test with no defined pass condition — and without a stated pass condition, an engineer watching the aftermath has no way to distinguish a working fallback from a lucky escape.

A well-formed hypothesis has three parts: the fault (what you're injecting — a timeout, a dependency outage, added latency, a full instance kill), the scope (which service, which percentage of traffic, which environment), and the expected steady-state (the specific metric or user-visible behavior that should hold, stated as a testable threshold — checkout completion rate stays above 99%, p99 latency stays under 2 seconds — not just "things still work"). You then run the experiment against that hypothesis exactly like a scientific test: inject the fault, measure the steady-state metric, and record whether the hypothesis held. A failed hypothesis is not a failed experiment — it's the experiment doing its job, surfacing a real gap between the resilience you designed and the resilience you actually have, before a real failure finds the same gap for you. If every experiment a team runs confirms the hypothesis, that's usually a sign they're only testing failure modes already known to be handled; the useful experiments are the ones with real uncertainty attached, where nobody's sure the fallback actually works. This is the same falsifiability discipline that makes SLOs useful: a claim about the system only means something if there's a concrete way it could turn out to be false.

Blast radius control

An experiment that could take down the whole platform if the hypothesis is wrong is not a chaos experiment, it's a gamble. Blast radius control is the practice of bounding, in advance, exactly how much damage a failed hypothesis can do — and expanding that bound only after confidence has been earned at a smaller one. The progression is consistent across mature chaos programs:

• Non-production first. Staging or a dedicated chaos environment, ideally one that mirrors production topology closely enough that the results transfer. Nothing here has a real user attached to it, so a wrong hypothesis costs nothing but engineering time.
• One instance. A single host or pod in production, chosen so that load balancing and redundancy should absorb its loss without any visible effect. This is the smallest unit of real-world signal you can get.
• A small percentage of traffic. 1%, then 5%, then larger slices, usually gated behind the same traffic-shifting mechanism used for canary releases — the infrastructure that makes progressive delivery safe is the same infrastructure that makes progressive chaos safe.
• Full production traffic. The fault is injected platform-wide. By the time a team runs an experiment at this scope, the same hypothesis has already been confirmed at every smaller scope, and rollback has been exercised, not just planned.

Every step up this ladder should be gated on a real abort mechanism, not just intention: an automated rollback that trips the instant the steady-state metric breaches its threshold, a kill switch the operator running the experiment can hit by hand, and a monitoring dashboard watched live for the duration of the run. Tools built for this — Netflix's own Chaos Monkey and its successor ChAP (Chaos Automation Platform), Gremlin, and the open-source Chaos Mesh and Litmus for Kubernetes environments — all build blast-radius limiting and automatic abort into the tool itself, rather than leaving it to operator discipline alone, because operator discipline is exactly what erodes under incident-adjacent pressure.

Game days

A game day is a scheduled, planned exercise in which a team deliberately breaks something — often as a single large fault injected all at once, rather than the incremental faults of a routine chaos experiment — and practices the full incident response, not just the technical failover. Where an automated chaos experiment mainly tests the system, a game day mainly tests the team: does the right person get paged, does the runbook match reality, does the on-call engineer reach for the correct dashboard, does the incident commander role get filled without confusion. See incident management & on-call for the response mechanics a game day is meant to exercise.

A typical game day has a facilitator who knows what fault is about to be injected and a responding team that, ideally, does not — the closer the exercise is to a surprise, the more honestly it tests instinct rather than rehearsed steps. A game day the responding team knows about in full detail in advance mostly tests whether people can follow a script they just read: still useful for validating a runbook, but not the same as testing response readiness. The facilitator picks a specific scenario in advance (a database failover, a full AZ loss, an expired certificate on a load balancer), sets explicit start and stop conditions, and holds a debrief immediately afterward while the details are fresh — structured much like a blameless postmortem, because a game day surfaces the same kind of gaps a real incident does: a missing dashboard, a runbook step that's stale, an alert that never fired. The payoff is muscle memory. A team that has run the failover before, even in a controlled exercise, executes it faster and with less panic during the real thing, because the unfamiliar parts have already been made familiar once.

The maturity progression: staging to production

Chaos engineering programs mature along a predictable path, and skipping stages is the most common way teams sour on the practice. Early-stage programs run exclusively in staging or a chaos-specific pre-production environment, with automated rollback as the primary safety net — the fault injection tool itself watches the steady-state metric and reverts the instant it degrades past threshold, so a bad hypothesis costs minutes, not an incident. This stage is where a team builds its library of hypotheses, its tooling, and its confidence that the abort mechanism actually works, because an abort mechanism that's never been triggered is untested itself.

Graduating to production requires more than technical readiness — it requires stakeholder buy-in. That means the on-call team knows experiments are happening and how to distinguish them from a real incident, leadership has agreed to the acceptable risk (usually expressed as an SLO error-budget allowance the experiment is permitted to spend), and there's a clear communication channel for pausing or aborting an experiment on request from anyone watching it unfold. Production experiments start at the smallest blast radius described above and only widen once the same hypothesis has held cleanly, more than once, at the smaller scope. A program that jumps straight from "we tried this in staging once" to "let's kill an AZ in production" hasn't actually built the confidence the blast-radius ladder is supposed to establish — it's borrowed the ladder's name without doing the climb. The abort mechanism deserves the same rigor as the fault itself: before running anything against production traffic, the team should be able to name the exact metric and threshold that trigger rollback, and confirm that rollback path has actually been exercised recently, not just documented.

Check your answers

1. It randomly terminated production instances during business hours with no warning to the on-call team. The randomness and lack of warning were the point — a rare, scheduled, announced termination lets engineers keep writing code that assumes instances live forever, while a constant, unpredictable one forces that assumption to be fixed immediately because it's tested every few days.
2. The fault (what's being injected), the scope (which service/environment/percentage), and the expected steady-state (a specific, measurable threshold that should hold). "Let's see what happens" has no stated pass condition, so there's no way to tell a working fallback from a lucky escape — it isn't falsifiable.
3. Non-production/staging, then one production instance, then a small percentage of production traffic, then full production traffic. Gating mechanisms include: no real users at risk (staging), redundancy expected to fully absorb a single loss (one instance), traffic-shifting infrastructure shared with canary releases plus a tested automated rollback (small %), and a proven rollback path plus stakeholder buy-in (full production).
4. It tests the team and the response process — paging, runbook accuracy, incident-commander handoff, communication — rather than just whether the system's automated failover code works, which is the main thing a routine automated chaos experiment verifies.