AWS Fault Injection Service

AWS Fault Injection Service (FIS) is AWS's own managed chaos-experiment service: you describe a fault, the AWS resources it should hit, and the condition that halts it, and FIS runs that experiment as ordinary, IAM-governed AWS API calls against your own account — no third-party agent fleet, no separate vendor to onboard, no control plane living outside AWS. It's scoped deliberately narrow: FIS only understands native AWS resources — EC2, ECS, EKS, RDS, and a growing list of others — and it makes no attempt to be a cross-cloud or on-prem chaos tool. That narrowness is the whole pitch. For a workload that lives entirely inside AWS, FIS is the fault-injection tool that's already covered by the IAM policies, CloudTrail audit trail, and CloudWatch alarms your team runs anyway, which is why it's the natural default rather than one option among several.

What AWS FIS is and the problem it solves

☺ Like you're 10: It's AWS's own tool for breaking your AWS stuff on purpose, using the same permission system and the same audit log you already have — not a new company you have to trust with a copy of your infrastructure.

AWS Fault Injection Service reached general availability in March 2021 under the name AWS Fault Injection Simulator; AWS later renamed it AWS Fault Injection Service while keeping the FIS abbreviation, which is why the acronym and the current product name don't quite match if you read them literally — worth knowing so old blog posts and re:Invent talks that say "Fault Injection Simulator" aren't describing a different product. It exists to solve a specific problem with running chaos experiments on AWS: every other option for injecting a fault into an EC2 instance, an ECS task, or an RDS cluster either means writing bespoke scripts against the AWS SDK yourself — no experiment template, no built-in stop condition, no audit trail beyond whatever you remember to log — or bringing in a third-party SaaS chaos platform that needs its own agent installed, its own credentials issued, and its own security review before anyone will let it touch production.

FIS closes that gap by being, structurally, just another AWS service: it authenticates through IAM, it's invoked through the same CLI and SDKs as every other AWS API, every StartExperiment and StopExperiment call lands in CloudTrail automatically, and the resources it touches are resources you already manage with the same tags, the same Config rules, and the same Organizations SCPs as everything else in the account. The tradeoff for that convenience is scope: FIS's action catalog only covers AWS-native resource types, so a fault you want to inject into an on-prem host, a non-AWS SaaS dependency, or infrastructure running in another cloud is out of reach, and that's exactly the boundary Gremlin and LitmusChaos exist to cross — see the comparison later on this page for when that boundary actually matters to you.

Architecture: the control plane, the SSM Agent, and the EKS extension

☺ Like you're 10: For most faults, AWS just quietly makes an API call it was already allowed to make — nothing new to install. For faults that happen inside an operating system or a Kubernetes pod, a small helper has to already be running there first.

FIS itself has no infrastructure for you to run — the "control plane" is entirely AWS's, the same way EC2's or IAM's is. What varies is how a given action actually reaches its target, and that split matters for how much you need to prepare before an experiment can succeed.

For control-plane-level actions — stopping or terminating an EC2 instance, failing over an RDS cluster, stopping an ECS task — FIS assumes the experiment's IAM role and simply calls the same underlying AWS API (ec2:StopInstances, rds:FailoverDBCluster, and so on) that a human or a script would call directly. Nothing runs on the target resource itself; the "agent" is the AWS control plane, which every AWS resource already talks to. For OS-level actions — stressing CPU or memory, injecting network latency or packet loss, killing a specific process inside an EC2 instance — FIS routes the fault through the SSM Agent (part of AWS Systems Manager) that must already be installed, running, and registered as managed on that instance; FIS calls ssm:SendCommand against one of AWS's predefined AWSFIS-Run-* documents (AWSFIS-Run-CPU-Stress, AWSFIS-Run-Memory-Stress, AWSFIS-Run-Network-Latency, and several more — check the current action catalog for the full, evolving list), and the SSM Agent executes it locally. For Kubernetes-level actions against EKS — deleting a pod, terminating a node — FIS requires a separate in-cluster component, the FIS extension for EKS, installed and kept upgraded like any other cluster workload, with its own IAM-to-Kubernetes-RBAC mapping so the FIS experiment role can actually act inside the cluster.

That distinction is the single most useful mental model for planning an experiment: AWS-boundary faults are agentless by default, OS-boundary faults need the SSM Agent already healthy, and Kubernetes-boundary faults need a real piece of software you now operate inside the cluster. Every experiment, regardless of which layer it targets, is still described the same way — as an experiment template naming its actions, targets, and stop conditions — which is the artifact covered next.

The experiment template you actually write

☺ Like you're 10: One file names three things — what to break, exactly which of your things to break, and the exact alarm that means "stop right now" — and that file is what gets reviewed before anyone runs it.

An experiment template is the single artifact FIS runs: it names one or more targets (which resources), one or more actions (what fault, and which target it applies to), one or more stop conditions (what halts the whole experiment immediately), and the IAM role FIS assumes to actually do any of it. You create it as plain JSON through the CLI, as a CloudFormation resource, or — the version most teams standardize on for code review — as Terraform.

// checkout-stop-template.json — stop 10% of checkout's prod instances,
// auto-restart after 5 minutes, abort immediately if the fast-burn alarm fires
{
  "description": "Stop 10% of checkout prod instances; confirm the ALB drains them cleanly",
  "targets": {
    "checkout-instances": {
      "resourceType": "aws:ec2:instance",
      "resourceTags": { "Environment": "prod", "Service": "checkout" },
      "filters": [ { "path": "State.Name", "values": ["running"] } ],
      "selectionMode": "PERCENT(10)"
    }
  },
  "actions": {
    "stop-some-instances": {
      "actionId": "aws:ec2:stop-instances",
      "parameters": { "startInstancesAfterDuration": "PT5M" },
      "targets": { "Instances": "checkout-instances" }
    }
  },
  "stopConditions": [
    { "source": "aws:cloudwatch:alarm", "value": "arn:aws:cloudwatch:us-east-1:111122223333:alarm:checkout-fast-burn" }
  ],
  "roleArn": "arn:aws:iam::111122223333:role/fis-checkout-experiment-role",
  "experimentOptions": { "emptyTargetResolutionMode": "fail" },
  "tags": { "Name": "checkout-instance-stop", "Owner": "sre-team" }
}

$ aws fis create-experiment-template --cli-input-json file://checkout-stop-template.json
$ aws fis start-experiment --experiment-template-id EXT12345678abcdef0

Most teams manage this as infrastructure, not as a one-off CLI call, which is where a versioned, code-reviewed template earns the same trust an application manifest does:

resource "aws_fis_experiment_template" "checkout_cpu_stress" {
  description = "Stress CPU on one checkout instance for 5 minutes"
  role_arn    = aws_iam_role.fis_checkout.arn

  target {
    name           = "checkout-instance"
    resource_type  = "aws:ec2:instance"
    selection_mode = "COUNT(1)"          # ONE instance — smallest possible blast radius

    resource_tag { key = "Service", value = "checkout" }
    filter        { path = "State.Name", values = ["running"] }
  }

  action {
    name      = "cpu-stress"
    action_id = "aws:ssm:send-command"   # routed through the SSM Agent — see architecture above

    target   { key = "Instances", value = "checkout-instance" }
    parameter { key = "documentArn",        value = "arn:aws:ssm:us-east-1::document/AWSFIS-Run-CPU-Stress" }
    parameter { key = "documentParameters",
                value = jsonencode({ DurationSeconds = "300", InstallDependencies = "True" }) }
  }

  stop_condition {
    source = "aws:cloudwatch:alarm"
    value  = aws_cloudwatch_metric_alarm.checkout_fast_burn.arn   # same alarm your SLO burn-rate alert uses
  }

  tags = { Owner = "sre-team" }
}

Actions and targets: the fault catalog and where it points

☺ Like you're 10: "Action" is the specific thing that goes wrong; "target" is which of your resources it goes wrong to; and you can dial the target down from "everything matching this tag" to "exactly one thing."

A target is defined by a resourceType (aws:ec2:instance, aws:ecs:task, aws:rds:cluster, and others) plus a selection method — resource ARNs listed explicitly, or the far more common pattern of resourceTags plus optional filters (like the running-state filter above) — and a selectionMode that bounds how many of the matched resources actually get hit: ALL, COUNT(n) for an exact number, or PERCENT(n) for a proportion. That selection-mode field is FIS's built-in version of the blast-radius ladder from chaos engineering — COUNT(1) is the smallest possible real-world signal, and widening to a larger PERCENT is a deliberate, reviewable change to the template rather than an accident of how a tag happened to match.

An action names the fault itself, drawn from FIS's growing action library, and points at one of the targets defined above. Representative actions, grouped by the layer they hit — treat this as illustrative of the shape, not an exhaustive or permanently current list, since AWS adds actions and resource types release over release:

Multiple actions in one template can be chained with startAfter, so a single experiment can, for example, disrupt network connectivity to one AZ and only then, thirty seconds later, terminate an instance in a neighboring AZ — testing a compound failure instead of a single one. Each target's emptyTargetResolutionMode defaults to fail: if your tag filter matches zero resources, the experiment fails outright rather than quietly running against nothing, which is a safer default than it might first look but does surprise teams used to a tool that skips silently on an empty match.

Stop conditions: the built-in abort switch

☺ Like you're 10: Every experiment carries its own smoke detector, wired in from the start — if a named alarm ever goes off, FIS stops the experiment itself, without anyone needing to be watching a screen.

Every experiment template must declare at least one stop condition, and in production that condition should always be source: aws:cloudwatch:alarm pointing at a real CloudWatch alarm ARN. For the entire lifetime of the experiment — regardless of what triggered it, whether a human clicked "start" or an EventBridge Scheduler rule fired it on a cron — FIS evaluates every declared stop condition independently, and the instant any one of the referenced alarms enters ALARM state, FIS halts the experiment and moves it to a terminal stopped status. This is the mechanism the platform-wide discussion in chaos engineering at scale points to directly: FIS's stop condition is the concrete implementation of the "guardrail controller" idea, built into the tool itself rather than assembled separately.

Two details matter more than they look. First, a stop-condition alarm that's already in ALARM state when you try to start the experiment prevents the experiment from starting at all — FIS won't let you knowingly launch a fault on top of a system that's already unhealthy, which is the same incident-collision-avoidance discipline chaos programs eventually have to build for themselves at scale, given to you for free at the single-experiment level. Second, the alarm's own evaluation period sets a floor on how fast the abort can possibly react: a stop condition tied to a slow, multi-period alarm reacts only as fast as that alarm does, so pair a stop condition with an alarm built the way a fast-burn SLO alert is built — short period, few consecutive breaching datapoints — not a slow daily-average threshold meant for a different kind of alerting.

IAM: who's allowed to break what

☺ Like you're 10: Two separate permission slips are needed — one that says "FIS itself is allowed to touch these specific labeled things," and a completely different one that says "this person is allowed to press the button that starts an experiment at all."

FIS's governance model runs entirely on ordinary IAM, and it's worth separating the two roles that get confused most often. The first is the experiment role — the role FIS assumes to actually perform the actions in a template. Its trust policy must explicitly allow the FIS service principal to assume it, and its permissions policy should be scoped as tightly as the blast radius you intend, typically with a tag-based condition so the role can only ever reach resources that were deliberately opted in.

// Trust policy on the experiment role — only FIS, only this account, can assume it
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "fis.amazonaws.com" },
    "Action": "sts:AssumeRole",
    "Condition": { "StringEquals": { "aws:SourceAccount": "111122223333" } }
  }]
}
---
// Permissions policy — scoped to resources tagged FIS-Ready, not "every instance in the account"
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["ec2:StopInstances", "ec2:StartInstances", "ec2:DescribeInstances"],
      "Resource": "arn:aws:ec2:*:111122223333:instance/*",
      "Condition": { "StringEquals": { "aws:ResourceTag/FIS-Ready": "true" } }
    },
    {
      "Effect": "Allow",
      "Action": ["ssm:SendCommand", "ssm:ListCommands", "ssm:GetCommandInvocation", "ssm:CancelCommand"],
      "Resource": "*"
    },
    { "Effect": "Allow", "Action": ["cloudwatch:DescribeAlarms"], "Resource": "*" }
  ]
}

The second, easily overlooked surface is who's allowed to call FIS at all: an engineer or a CI pipeline needs its own IAM permissions for fis:CreateExperimentTemplate, fis:StartExperiment, and fis:StopExperiment before they can launch anything — and that's a completely separate policy from the experiment role above, attached to a human or a service identity rather than to FIS itself. Scoping that permission narrowly is the access-control layer that answers "who's allowed to break production on purpose" — the same question chaos engineering at scale covers organizationally through chaos champions and standing approval tiers, expressed here as a plain IAM policy instead of a process document. Because both layers are ordinary IAM, every CreateExperimentTemplate, StartExperiment, and StopExperiment call — who made it, when, against which template — lands in CloudTrail automatically, with no extra logging integration to build.

Day-to-day commands

☺ Like you're 10: A handful of commands cover almost everything: what faults exist, what's running right now, and how to make it stop.

# discover the current action and resource-type catalog — don't assume last year's list is complete
$ aws fis list-actions --query 'actions[].id'
$ aws fis get-action --id aws:ssm:send-command
$ aws fis list-target-resource-types

# templates
$ aws fis create-experiment-template --cli-input-json file://checkout-stop-template.json
$ aws fis list-experiment-templates
$ aws fis get-experiment-template --id EXT12345678abcdef0
$ aws fis update-experiment-template --id EXT12345678abcdef0 --cli-input-json file://updated.json
$ aws fis delete-experiment-template --id EXT12345678abcdef0

# running an experiment
$ aws fis start-experiment --experiment-template-id EXT12345678abcdef0
$ aws fis get-experiment --id EXP12345678abcdef0 --query 'experiment.state'
$ aws fis list-experiments --query 'experiments[].{id:id,state:state.status,template:experimentTemplateId}'
$ aws fis stop-experiment --id EXP12345678abcdef0   # manual abort, independent of any stop condition

# Terraform, if the template is managed as code
$ terraform plan -target=aws_fis_experiment_template.checkout_cpu_stress
$ terraform apply

Gotchas and failure modes

☺ Like you're 10: Most "why didn't this run" stories trace back to a helper program that wasn't actually awake, a permission slip that forgot one line, or a tag that matched more than you meant it to.

The SSM Agent has to be healthy before the experiment starts, not just installed

An OS-level action fails at the moment FIS tries to invoke it if the target instance's SSM Agent isn't running, isn't registered as managed in Systems Manager, or can't reach the SSM service endpoints — commonly because the instance sits in a private subnet with no NAT gateway and no VPC interface endpoints for Systems Manager. The error surfaces as an SSM command failure attached to that action, not as a clear "your agent is missing" message, so the fastest diagnostic is checking Systems Manager Fleet Manager for the instance's connection status before starting the experiment, not after it fails.

A missing or malformed trust policy fails the experiment, not the template creation

FIS will happily accept an experiment template referencing a role whose trust policy doesn't actually allow fis.amazonaws.com to assume it — the template itself creates without error. The failure only surfaces when you call start-experiment and FIS can't assume the role. Verify the trust policy once, up front, rather than discovering the gap the first time someone tries to run the thing for real.

Tag-based targeting with ALL is the classic blast-radius mistake

A tag value typo, or a tag that's broader than the author realized — Environment=prod matching resources across three unrelated services instead of the one intended — combined with selectionMode: ALL means every matching resource in the target's scope gets hit, not the handful the author had in mind. COUNT(n) and PERCENT(n) exist specifically to bound this failure mode; treat ALL as a mode you graduate into deliberately, after a template has proven itself at a smaller scope, the same blast-radius ladder chaos engineering describes in general.

Billing is per action-minute, across every target the action actually touches

FIS charges per action, per minute the action runs, and — this is the part that surprises people sizing a fleet-wide experiment — that's effectively multiplied by how many resources the action touches under a broad PERCENT or ALL selection. A five-minute CPU-stress action against COUNT(1) instance costs very differently from the same action against PERCENT(100) of a two-hundred-instance fleet. Treat any specific dollar figure as a moving target and verify current pricing on AWS's own FIS pricing page before sizing an experiment that spans a large fleet.

EKS support is the one place "no agent to run" doesn't hold

Every other gotcha on this page assumes FIS is agentless or leans on an agent (SSM) you likely already run for other reasons. The FIS extension for EKS breaks that pattern: it's a real, additional in-cluster workload that has to be installed, kept upgraded alongside the cluster itself, and mapped into the cluster's RBAC so the FIS experiment role can actually act on pods and nodes. Teams that assume EKS chaos experiments are as zero-friction as an EC2 stop-instances action are usually surprised by this the first time they try to set one up.

AWS FIS vs. Gremlin vs. LitmusChaos vs. Chaos Monkey

☺ Like you're 10: All four break things on purpose — they just disagree about who should be allowed to run the tool, what kinds of things it can reach, and who you have to trust to keep it safe.

The decision is rarely "which of these is best" in the abstract — it's "what does my infrastructure actually look like." FIS's whole case rests on staying inside AWS; the moment your footprint doesn't, a different tool's tradeoffs start to win.

The case for FIS as the default inside AWS comes down to what you don't have to do. There's no new vendor for security to review, because the experiment role's calls are ordinary AWS API calls subject to the same SCPs, permission boundaries, and CloudTrail logging as everything else in the account. There's no new fleet of agents to deploy and patch for the most common fault categories, because control-plane actions ride the AWS API you already have credentials for. And AWS's own resilience tooling is aware of it: production readiness reviews can point directly at FIS experiment templates as the evidence a resilience claim was actually tested, and AWS Resilience Hub can assess an application against a stated resiliency policy and recommend or generate FIS templates aligned to the gaps it finds — worth checking against current Resilience Hub documentation, since that integration has continued to evolve. None of that argues FIS is a better chaos engine than Gremlin's or Litmus's in the abstract — it argues that for a workload that never leaves AWS, the cheapest, most auditable starting point is the tool that was never a stranger to your account in the first place.

Where AWS FIS fits in the SREF blueprint

☺ Like you're 10: The exam won't ask you to write a CLI command — it wants you to recognize "this is a managed chaos-experiment service" from a description, and to know why an IAM-native tool behaves differently from a third-party SaaS one.

The DevOps Institute SRE Foundation (SREF) exam is closed-book and tests tool categories rather than vendor-specific syntax, as SRE Tools & Automation covers in full — AWS FIS sits in the chaos-engineering/resilience-testing row alongside Gremlin and LitmusChaos, and the concepts that actually get tested — hypothesis-driven experiments, blast-radius control, and the role of an automated stop condition — live in chaos engineering and chaos engineering at scale. If you're pursuing an AWS credential alongside SREF, FIS also shows up as exam-relevant material in its own right on AWS DevOps Engineer Professional and, at an awareness level, AWS Solutions Architect Associate — treat the experiment-template shape and the IAM role split on this page as the transferable knowledge either way, and verify exact CLI flags and current pricing against AWS's own FIS documentation before you rely on them for anything graded.