Security Chaos Engineering

A SAST gate can be green, a purple-team session can score every technique tested green on the ATT&CK Navigator, and the on-call security engineer can still sleep through a real incident three weeks later — because nobody ever actually pulled the trigger on the exact control that was supposed to catch it, on the exact day it mattered. Security chaos engineering borrows a discipline built for a different problem — Netflix's chaos engineering, invented to prove a distributed system survives a server dying mid-request — and points it at a different question: not "does the system survive," but "does anyone notice." You revoke a real credential and watch whether anything pages. You plant a fake leaked secret somewhere a real one would leak and watch whether it's caught before it's ever used. You fire one small, deliberately reversible attack pattern against a live account and watch whether the detection you believe exists actually exists. This page covers the method borrowed from resilience chaos engineering, one fully worked credential-revocation experiment including the gotcha it was designed to surface, the tooling that runs these experiments safely, and what it takes to run one without the drill itself becoming the incident.

What security chaos engineering actually tests

☺ Like you're 10: It's not enough to believe your smoke alarm has a battery in it. Make some smoke, on purpose, in a safe spot, and watch whether it actually goes off.

Chaos engineering as a discipline started at Netflix around 2010 with Chaos Monkey — a tool, later open-sourced as part of the "Simian Army," that randomly killed production instances to prove the streaming service could survive a server dying mid-request. Casey Rosenthal and collaborators later codified the practice into a shared set of principles, published at principlesofchaos.org, aimed squarely at resilience: does the system keep working when something breaks. Around the mid-to-late 2010s, Aaron Rinehart — then Chief Security Architect at UnitedHealth Group, later a founder of the chaos-engineering company Verica — and Kelly Shortridge argued the same method answers a security question just as well as a resilience one, work that became the O'Reilly book Security Chaos Engineering: Sustaining Resilience in Software and Systems (check the current edition for exact publication details — it went through early-release chapters before a full print run). The idea: instead of asking "will checkout survive a dead node," ask "will anyone notice a stolen credential."

It's worth being precise about how this differs from work this course already covers. Purple teaming is a scheduled, human-facilitated session — a red operator and a detection engineer sit down together, run a handful of techniques, and compare notes in real time. Atomic Red Team tests are one-off, run manually against a lab host to earn a single ATT&CK Navigator score. Security chaos engineering is the discipline that turns that kind of test into a routine: hypothesis-driven, run against production or as close to it as is responsible, and — the part most teams skip — automated to run again and again on a schedule, not once before an audit. The nearest commercial category is breach and attack simulation (BAS), a market of vendor platforms that continuously fire known attack techniques against your environment; the tools later on this page overlap with what BAS products do, but the discipline itself — a stated hypothesis, a controlled blast radius, a real production target — is what chaos engineering adds on top of "run more simulated attacks."

The method, borrowed: steady state, hypothesis, blast radius

☺ Like you're 10: Say out loud exactly what you expect to happen before you do the risky thing — otherwise you can't tell whether the result surprised you or you just weren't paying attention.

Netflix's chaos engineering principles translate onto a security experiment almost line for line, with one inversion that changes everything about how you read the result:

• Define a steady-state hypothesis. Not "the system is secure" — that's not falsifiable — but a specific, measurable claim: "an AWS access key used from an IP outside our CI runner allow-list generates a GuardDuty finding routed to the on-call Slack channel within 5 minutes."
• Vary real-world events. Here that means real classes of security failure: a credential gets revoked, a secret leaks, a bucket policy gets loosened, an attacker runs one specific technique — the same categories a real incident would actually involve, not a synthetic test case invented for convenience.
• Run it in production — or the smallest slice of production you can responsibly touch. Staging never carries the real log volume, the real noisy baseline, or the real field-name drift that determines whether a detection rule survives contact with actual data; a rule that only ever fired in a clean lab has not actually been tested yet.
• Automate the experiment to run continuously rather than once before an audit. This is the difference between a purple-team session and a chaos program, and it's the subject of the coverage-decay section further down this page.
• Minimize blast radius. Choose reversible actions over destructive ones, scope the experiment as narrowly as the hypothesis allows, and always have a rollback ready before you start — the discipline that makes doing any of this in production defensible at all, covered in full near the end of the page.

Here's the inversion. In a classic resilience experiment, success looks like nothing changing — you kill a node, and checkout keeps working exactly as it did before, which confirms the system is as resilient as you hoped. In a detection-focused security experiment, success looks like something changing — you revoke a credential, and the quiet steady state of "no open incident" should stop being true within your stated window. If you inject a real failure and the dashboard stays exactly as calm as it was five minutes earlier, that calm isn't good news. It's the blind spot the whole exercise exists to surface.

A full worked experiment: revoking a production credential

☺ Like you're 10: Turning off a stolen key stops someone from using it again. It doesn't automatically kick out someone who's already inside, using a copy they made five minutes earlier.

Say a legacy deploy job — not everything has migrated to OIDC-federated workload identity yet; a third-party integration, a break-glass admin path, or a cron job nobody's touched in a year is a completely realistic gap even in a shop that's otherwise done the migration — still authenticates with a static IAM user access key, svc-deploy-legacy. The hypothesis: "if this key is compromised and someone deactivates it, (1) no further calls succeed using it, and (2) security on-call is paged within 5 minutes of the deactivation itself, because that's exactly the kind of event a real incident response would need to happen fast."

# 1) INJECT — deliberately reversible: Inactive, not Delete, so one command undoes it
aws iam update-access-key --user-name svc-deploy-legacy \
  --access-key-id AKIAEXAMPLE111 --status Inactive

# rollback ready before the drill starts:
#   aws iam update-access-key --user-name svc-deploy-legacy \
#     --access-key-id AKIAEXAMPLE111 --status Active

# 2) OBSERVE — did the pipeline this rule depends on even exist?
# an EventBridge rule that should be watching for exactly this:
{
  "source": ["aws.iam"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": { "eventName": ["CreateAccessKey", "DeleteAccessKey", "UpdateAccessKey"] }
}

# 3) THE SURPRISE — a session already assumed using this key keeps working:
aws sts get-caller-identity   # run from the CI job's already-cached session — still succeeds

# 4) THE ACTUAL FIX the drill was missing from the runbook —
# revoke the ROLE's active sessions, not just the key used to assume it
# (this is the same mechanism behind the IAM console's own "Revoke active sessions" button)
aws iam put-role-policy --role-name ci-deploy-prod \
  --policy-name RevokeSessionsBefore2026-08-16T14-00 \
  --policy-document file://deny-before-revocation-time.json

deny-before-revocation-time.json is a short inline policy keyed on aws:TokenIssueTime, denying every action for any credential minted before the moment you set:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Deny",
    "Action": "*",
    "Resource": "*",
    "Condition": { "DateLessThan": { "aws:TokenIssueTime": "2026-08-16T14:00:00Z" } }
  }]
}

Neither finding here is a hypothetical stretch — it's exactly how AWS session credentials work: an STS session issued via sts:AssumeRole is a self-contained, signed credential that doesn't re-check the originating key's status on every call, so it stays valid until its own expiration (up to the role's configured MaxSessionDuration) regardless of what happens to the key that was used to obtain it. A team that "knows" this in the abstract and a team that's actually watched a live session survive a deliberate key revocation are not in the same place operationally — one has a belief, the other has a corrected runbook and a rule that now watches UpdateAccessKey specifically, not just CreateAccessKey and DeleteAccessKey.

The experiment catalog: beyond one credential

☺ Like you're 10: Different drills catch different sleeping guards — a fire drill doesn't tell you whether the burglar alarm works, so you need more than one kind.

Credential revocation is one instance of a general pattern: pick something a real incident would involve, do the smallest reversible version of it against something real, and watch whether the control that's supposed to react actually does.

Tooling for injection, and what counts as "observing"

☺ Like you're 10: The tool that pretends to be a burglar is only half the job — you also need someone actually watching the cameras while it happens.

Stratus Red Team, an open-source CLI, packages individual cloud attack techniques — organized by provider and MITRE ATT&CK tactic — as small, self-contained, reversible units, closely mirroring Atomic Red Team's approach but purpose-built for cloud control-plane actions instead of host-level ones:

# list techniques mapped to one ATT&CK tactic
stratus list --mitre-attack-tactic exfiltration

# warmup provisions any prerequisite scratch infrastructure the technique needs
stratus warmup aws.exfiltration.ec2-share-ebs-snapshot

# detonate performs the technique for real, against your real account
stratus detonate aws.exfiltration.ec2-share-ebs-snapshot
#   -> now go check: did a GuardDuty finding land? did the mapped Sigma/CloudTrail rule fire?

# revert undoes the technique's effect; cleanup tears the warmup infra back down
stratus revert aws.exfiltration.ec2-share-ebs-snapshot
stratus cleanup aws.exfiltration.ec2-share-ebs-snapshot

Thinkst Canarytokens (a free service at canarytokens.org, also self-hostable) generates a realistic-looking but entirely inert credential — an AWS key pair, an Azure app ID, a Kubernetes kubeconfig, a Word document, a DNS hostname — that does exactly one thing: it phones home the moment anyone tries to use it. Plant it somewhere a real leaked secret would realistically end up, not somewhere only your own team would ever look — a public GitHub gist named like a leftover debug dump, an internal wiki page (to test insider or compromised-account detection specifically, as a separate experiment from the public-leak case), a low-privilege EC2 instance's user-data script. The token is the detector:

// illustrative — Canarytokens' exact webhook payload shape can change; treat as representative
{
  "canarytoken": "a1b2c3d4e5f6...",
  "channel": "AWS API Key Token",
  "time": "2026-08-16T14:02:11Z",
  "src_ip": "203.0.113.44",
  "additional_data": { "event_name": "GetCallerIdentity", "user_agent": "aws-cli/2.x" }
}

Two more tools round out the toolbox. MITRE Caldera, a server-and-agent adversary emulation platform built by the same organization behind ATT&CK, runs multi-step "operations" autonomously against a fleet of agents rather than one command at a time — the closest open-source path to the "automate to run continuously" principle for host-level techniques. Infection Monkey (open-sourced by Guardicore, now maintained under Akamai after its 2021 acquisition) does something similar for internal network lateral-movement testing, deliberately built to be safe-by-design and to end in a report rather than real damage. And on the cheap-but-shallow end, AWS's own guardduty create-sample-findings API generates a sample finding of a chosen type with no real attack behind it at all — useful for proving your GuardDuty-to-Slack plumbing works, but it tests the pipe, not the detection logic, and shouldn't be mistaken for the real thing.

For orchestrating the whole loop as code rather than a checklist, Chaos Toolkit (an open-source, provider-agnostic experiment runner) expresses a security experiment in the same declarative shape a resilience one would use — a steady-state hypothesis with probes, a method of actions, and rollbacks:

{
  "version": "1.0.0",
  "title": "Revoking svc-deploy-legacy's key pages security on-call within 5 minutes",
  "steady-state-hypothesis": {
    "title": "No open security incident right now",
    "probes": [{
      "type": "probe", "name": "no-open-pagerduty-incident", "tolerance": 0,
      "provider": { "type": "python", "module": "chaos_pagerduty.probes",
                     "func": "count_open_incidents", "arguments": { "service": "security-oncall" } }
    }]
  },
  "method": [{
    "type": "action", "name": "deactivate-ci-access-key",
    "provider": { "type": "process", "path": "aws",
      "arguments": "iam update-access-key --user-name svc-deploy-legacy --access-key-id AKIAEXAMPLE111 --status Inactive" },
    "pauses": { "after": 300 }
  }],
  "rollbacks": [{
    "type": "action", "name": "reactivate-ci-access-key",
    "provider": { "type": "process", "path": "aws",
      "arguments": "iam update-access-key --user-name svc-deploy-legacy --access-key-id AKIAEXAMPLE111 --status Active" }
  }]
}

Notice the assertion this experiment is actually built to make: after the 5-minute pause, re-running the same probe should now find at least one open incident — the steady state was supposed to break. A framework built for resilience testing (where the probe should still pass afterward) needs that inversion spelled out explicitly, or the experiment silently tests the wrong thing.

Detection coverage decays — why a green cell needs a re-test date

☺ Like you're 10: A smoke alarm you tested once, two years ago, isn't the same as a smoke alarm you know works today — batteries die quietly, and so do detection rules.

The ATT&CK Navigator scoring this course already covers treats a score of 3 — "a rule exists and has been tested" — as an earned, trustworthy state. It is, on the day it's earned. What it isn't is permanent. Log pipelines get refactored. A service migrates to a new logging library and quietly renames a field a Sigma rule's logsource pipeline depended on. An EDR agent ships a version bump that reshapes its event schema. Someone "cleans up" an EventBridge rule during an unrelated audit and narrows its eventName filter without realizing a downstream detection depended on the wider match. None of these changes announce themselves, and the absence of an alert looks identical whether nothing bad happened or the rule quietly stopped working — that's detection drift, and it's invisible by construction until something tests it again.

The fix follows directly from the same "security as code" argument this course has made since What is DevSecOps?: don't just write and test a detection once — schedule the experiment that proved it works to run again, automatically, and treat a failure to re-fire as a regression, not a curiosity.

# .ci/chaos-regression.yml — re-run the SAME experiment that earned a Navigator score of 3,
# on a schedule, against a scoped account, and fail loudly if the paired rule stops firing
schedule:
  - cron: "0 3 * * *"        # nightly, off-peak, against a non-prod or tightly scoped account

steps:
  - name: detonate-known-technique
    run: stratus detonate aws.exfiltration.ec2-share-ebs-snapshot

  - name: wait-for-pipeline
    run: sleep 300            # give collection, normalization, and correlation time to catch up

  - name: assert-alert-fired
    run: python verify_alert_fired.py --rule-id T1537-ebs-share --window 10m
    # non-zero exit = a rule "tested and validated" months ago has silently regressed —
    # open a ticket against the rule owner; don't just log it and move on

  - name: revert
    if: always()
    run: stratus revert aws.exfiltration.ec2-share-ebs-snapshot

Running it safely: blast radius, deconfliction, and the game day

☺ Like you're 10: A fire drill only teaches anyone anything if the building doesn't actually burn down in the process — plan the exit before you pull the alarm.

Everything on this page targets production, or something close enough to it that the result means something — which raises the stakes past what a scoped penetration test against a signed rules-of-engagement document already requires, not below it. A few practices make that responsible rather than reckless:

• Authorization in writing, every time. Exact scope, exact systems, an exec sponsor who knows it's happening, and a named person who can call it off immediately — the same non-negotiable this course already argued for offensive testing applies at least as strongly here, because the target is live.
• Prefer reversible actions. Inactive over Delete, a feature flag over a code change, a single canary account over the whole fleet — every experiment on this page was chosen because one command undoes it.
• Time-box and automate the abort. Set a hard stop before you start, and wire an automatic rollback to a condition, not just a person's judgment — "if error rate exceeds X, revert immediately" beats "someone will notice and revert."
• Deconflict with the people watching. Keep a dated, visible registry of what's running right now, so anyone who notices anomalous activity mid-drill can check "is this a scheduled experiment" in seconds before escalating to a customer-facing incident, a third-party notification, or a call to an on-call engineer's actual phone at 2am for nothing.

How much the responding team knows in advance is a real spectrum, not a binary. At the fully announced end sits something close to Etsy's own GameDay tradition under John Allspaw — collaborative, blameless, everyone in the room learning together, closer to a purple-team session than a covert test. At the unannounced end sits something like Google's internally documented DiRT (Disaster Recovery Testing) program, which has publicly discussed running realistic, unannounced failure scenarios — including security ones — specifically because a defender who knows a drill is coming behaves differently than one who doesn't. Neither end is more "correct"; an announced game day teaches a team to build and tune a control together, while an unannounced one tells you what actually happens the one time nobody's braced for it. Mature programs run both, at different cadences, for different questions.

Whichever end of the spectrum a given drill sits on, the debrief matters as much as the injection did: a blameless write-up of exactly what fired, what didn't, and why — fed into a backlog with an actual SLA, the same discipline best practices and the operating model argues for everywhere else in this course, and the opposite of the pattern DevSecOps anti-patterns calls out when a finding just becomes a slide nobody revisits.