Gremlin

Chaos Monkey, covered elsewhere in this course, kills things at random and offers no apology. Gremlin is the commercial answer to a question a lot of teams ask right after reading about Chaos Monkey: "that's a great idea, but how do we run it without someone accidentally taking down payments at 2pm on a Tuesday?" Gremlin is a SaaS chaos-engineering platform — founded in 2016 by Kolton Andrus and Matthew Fornaciari, both alumni of resilience work at Netflix and Amazon — built around a deliberately narrow set of attacks, a scoping model that limits exactly what gets touched, a halt mechanism that can stop an experiment automatically, and scheduling that turns one-off chaos days into a recurring, governed practice. This page covers the platform itself: its attack library, its architecture, the blast-radius and halt controls that are its actual product, and the Scenarios feature that chains attacks into repeatable, schedulable experiments. Gremlin's own vendor certification is a separate credential, covered on Gremlin Enterprise Chaos Engineering Certification.

What Gremlin is and the problem it solves

☺ Like you're 10: It's a company you pay to give you a safe, labeled box of ways to break your own stuff on purpose, plus a button that undoes it instantly.

Everything this course's chaos engineering page teaches — hypothesis-driven experiments, a blast-radius ladder from staging to full production, an abort mechanism worth trusting — is a discipline. Gremlin is one commercial product built to enforce that discipline in software rather than leave it to a team's own tooling and self-restraint. Where Netflix's open-source Chaos Monkey is a single, narrow capability (terminate a random instance, on a schedule, with no scoping UI and no built-in halt), Gremlin sells a full platform: a curated library of attacks across compute, state, and network, a targeting system built around tags rather than hard-coded hostnames, a one-click and API-driven halt on every running attack, automatic halting tied to a live health signal, and a workflow layer — Scenarios — that chains attacks together and can run on a schedule without a human in the loop. The pitch, in short, is that a mature chaos program needs guardrails as much as it needs fault injection, and building those guardrails yourself on top of a bare open-source tool is exactly the kind of recurring, undifferentiated engineering work SRE calls toil.

Architecture: the agent, the control plane, and no inbound ports

☺ Like you're 10: A tiny program sits on each of your machines and only ever calls out to Gremlin's website to ask "anything to do?" — nobody can call in and reach it from outside.

Gremlin splits into two pieces, and the split is the whole security story. The agent is a small, privileged process that runs on every host you want to be able to attack — installed as a native package on a Linux (or Windows) host, as a sidecar/co-located Docker container, or as a Kubernetes DaemonSet via Gremlin's official Helm chart so one pod-per-node lands automatically as the cluster scales. The control plane is Gremlin's own multi-tenant SaaS backend — the UI, the API, attack scheduling, Scenario storage, and Status Check evaluation all live there, not on your infrastructure. The connection between the two runs in exactly one direction: the agent polls out to the control plane over outbound HTTPS on 443 and asks what to do; nothing ever connects in to an agent from the outside. That single design choice is why security teams tend to approve Gremlin faster than a tool that would otherwise need an open inbound port on every production host — there is no listening socket for an attacker to find.

Because the agent, not the control plane, is what actually runs an attack, most Resource, State, and Network attacks require the agent to hold elevated local privileges — root on a Linux host, or an equivalently privileged Kubernetes security context, to manipulate CPU scheduling, mount namespaces, or the network stack. ALFI works differently: instead of a host-level agent reaching into the OS, you add a Gremlin SDK (Go and Java are the longest-supported) directly into your application, and the fault is injected at a specific call site you've instrumented — a downstream HTTP call, a database query — rather than at the whole host. ALFI is the right tool exactly when the blast radius you want is narrower than "this entire pod," and the tradeoff is real engineering lead time: it needs a code change and a deploy, not just an agent already sitting there.

The attack library

☺ Like you're 10: Four toolboxes: one starves the machine of horsepower, one changes what the machine is doing, one messes with what the machine can talk to, and one breaks a single phone call from inside your own app.

Blackhole and Certificate Authority are worth calling out specifically because they test failure modes host-level Resource and State attacks structurally can't reach. A CPU or Shutdown attack proves a host can disappear; Blackhole proves a specific dependency can disappear while everything else keeps working, which is the far more common real-world failure — a downstream payment processor times out, not the whole data center. Certificate Authority exists because certificate-trust failures are notoriously hard to rehearse safely any other way: you can't casually expire a production CA to see what breaks, but you can tell one target to stop trusting it for ninety seconds and watch.

Scoping the blast radius and the halt controls

☺ Like you're 10: You pick exactly which machines get hit by their name tag, not by guessing — and there are three separate ways to make it stop, including one that stops itself.

Every Gremlin attack is targeted, not fired at "production" in general. Tags — pulled automatically from cloud-provider metadata (region, availability zone, instance type) or set by hand — are the unit of scoping, and a target selection is either Exact (name the precise hosts or containers) or Random (hit a percentage or a fixed count drawn from everything a tag filter matches). This is the mechanism behind the blast-radius ladder this course already teaches: "one instance" is an Exact target of one; "a small percentage of traffic" is a Random target at a low percent against a tag that matches an entire fleet.

Stopping an attack has three independent layers, and a mature Gremlin practice uses all three rather than trusting any one of them alone:

• Manual halt. A single click in the UI, or gremlin attack halt <attack-id> from the CLI, stops one running attack immediately. A separate "halt all" reaches every attack running across an entire account at once — the button an incident commander wants within one click during a live response, not three menus deep.
• Max attack duration. Every attack requires an explicit, bounded length when it's launched — Gremlin has no concept of an attack that just runs forever. The agent self-terminates the fault at that duration even if nobody ever presses halt, which caps the worst case of "we all forgot about the CPU attack from Thursday" at one thing: a duration you set, not an open-ended incident.
• Status Checks — the automatic one. A Status Check wires a health signal — a Datadog or New Relic monitor, a Prometheus query, or a plain HTTP endpoint expected to return healthy — into a running attack or Scenario. If the check goes unhealthy while the attack is live, Gremlin halts it automatically, without waiting for a human to notice the dashboard.

Scenarios, Recommended Scenarios, and scheduling

☺ Like you're 10: A single attack is one prank; a Scenario is a whole choreographed routine — break this, wait, check that it's still okay, break the next thing — and you can tell it to repeat every week on its own.

A single attack answers one question. A Scenario chains multiple attacks and pause/check steps into one repeatable experiment — inject latency on the payments dependency, wait sixty seconds, confirm checkout completion is still above threshold, then Blackhole a second dependency and confirm again — closer to the hypothesis-driven experiment this course already teaches than any single attack is on its own. Gremlin ships a library of Recommended Scenarios — pre-built templates aimed at common resilience questions like cross-AZ redundancy or resource exhaustion — as a starting point; they're a reasonable first Scenario to run, not a substitute for writing a specific hypothesis about your own system once you've outgrown the template.

Scenarios can be defined and stored three ways: hand-built in the UI, created via the REST API, or declared as code through Gremlin's official Terraform provider — the last of which is what makes a chaos experiment reviewable in a pull request instead of living only as a click-path someone has to remember. And a Scenario doesn't need a human to launch it each time: scheduled Scenarios run on a recurring cadence Gremlin's control plane triggers itself, turning chaos from an occasional event someone has to remember to organize into a routine, continuously-running check on the system — the same shift chaos engineering at scale covers as a program matures past ad hoc experiments. That scheduling machinery is also what operationalizes the larger, human-facing game days this course already teaches: a Gremlin Scenario can be the fault a game-day facilitator triggers on cue, with the halt and Status Check layers already wired in as the exercise's safety net.

# Illustrative shape of a Gremlin Scenario defined via Terraform — confirm exact
# resource and attribute names against the current gremlin/gremlin provider docs,
# since a commercial provider's schema shifts across releases more than an OSS one.
resource "gremlin_scenario" "cross_az_payments_check" {
  name        = "Cross-AZ payments dependency check"
  description = "Blackhole the us-east-1b payments replica; checkout should fail over cleanly"
  hypothesis  = "Checkout completion rate stays above 99% while the replica is unreachable"

  step {
    attack {
      type     = "blackhole"
      duration = 300
      target {
        strategy = "Exact"
        tags     = { az = "us-east-1b", role = "payments-replica" }
      }
    }
  }
  step {
    status_check {
      source    = "datadog"
      query     = "avg:checkout.completion_rate{env:prod}"
      threshold = 0.99
      # a breach here triggers the automatic halt described above
    }
  }

  schedule {
    recurrence = "weekly"
    day        = "wednesday"
  }
}

Day-to-day commands

☺ Like you're 10: A handful of commands launch an attack against a host, a container, or a whole tag group — and one command always stops whatever's running.

# Illustrative CLI shape — verify exact subcommands and flags against Gremlin's
# current CLI reference before running these against anything real.

# a CPU attack against one exact host, 60 seconds, one core
$ gremlin attack-host cpu --length 60 --cores 1 \
    --target-type exact --target-value web-prod-14

# latency injected into 25% of containers carrying a tag, chosen at random
$ gremlin attack-container latency --length 120 --delay 300ms \
    --target-type random --target-percent 25 --tag env:staging

# blackhole a specific egress port on a Kubernetes-scheduled agent
$ gremlin attack-container blackhole --length 180 --egress-port 5432 \
    --target-type exact --tag role:payments-replica --tag az:us-east-1b

# stop one attack immediately, by ID
$ gremlin attack halt 8f1c2a90-...

# list everything currently running, across the whole account
$ gremlin attack list --running

# run a stored Scenario on demand instead of waiting for its schedule
$ gremlin scenario run cross-az-payments-check

Gotchas and failure modes

☺ Like you're 10: Most Gremlin surprises come from something outside Gremlin itself — a free tier that's smaller than you assumed, a network layer between your tag and the actual blast radius, or a health check that's too slow to catch what it's watching for.

• The free tier isn't the full library. Gremlin Free has historically been limited to a single attack (Shutdown) against unlimited hosts — a reasonable way to kick the tires, and a common source of "why can't I run a Blackhole attack" confusion for a team that assumed a signup unlocked the whole catalog described on this page. Confirm current tier boundaries and pricing on Gremlin's own site before planning a rollout around a specific attack type.
• Privileged agent access is a prerequisite, not a detail. Resource, State, and Network attacks need the agent to hold real OS-level privileges. A hardened host or a restrictive Kubernetes PodSecurityContext can silently prevent an attack from running at all, or let it run partially — which is worse than a clean failure, because it can look like "the fault didn't do anything" instead of "the fault couldn't apply."
• Status Checks have a polling interval, not zero latency. If a health check evaluates every sixty seconds and a real breach and recovery both happen inside that window, the auto-halt may never fire at all — the attack simply ends on its own max-duration timer having done more damage than the Status Check ever detected. Tune the check's interval and threshold to the actual attack duration, don't just wire one up and assume it's watching continuously.
• Network attacks don't always respect the boundary you drew. A Blackhole or packet-loss attack applied inside a Kubernetes CNI overlay, or behind a NAT gateway, can affect more than the single pod a tag suggested — the actual blast radius is a property of the network path, not just the tag filter. Validate scope with a Random target of one before trusting Exact targeting at scale on a network topology you haven't tested chaos against before.
• ALFI needs lead time Resource/State/Network attacks don't. Because it's an SDK integrated into your own code, ALFI can't be pointed at a service that hasn't already shipped the instrumentation — plan it as a deliberate engineering task with a deploy attached, not something you can decide to try five minutes before a game day.

Alternatives and when to reach for something else

☺ Like you're 10: Other tools also break things on purpose — some are free and DIY, some only work inside one cloud, and each trades away a piece of what Gremlin bundles together.

The practical rule most teams land on: reach for Chaos Monkey, Litmus, or Chaos Mesh when the environment is homogeneous (all Kubernetes, all AWS) and a team is comfortable building and testing its own safety tooling; reach for Gremlin when the environment is heterogeneous, when the org wants a single auditable place attacks get requested and approved from, or when the built-in halt/Status Check/scheduling layer is worth more than the subscription it costs to not build that layer yourself. See the SRE toolchain for how the whole chaos-tooling category sits alongside monitoring, tracing, and paging, and Drill — Design a Chaos Experiment or Capstone Part 6 — Chaos Engineer It for hands-on practice designing one against a service you control.

Going further

☺ Like you're 10: This page is the platform; a completely separate page covers the exam you can take about it.

The canonical source is Gremlin's own documentation and pricing pages — confirm current attack-catalog details, tier boundaries, and the exact Terraform provider schema there, since all three are the kind of detail a commercial vendor changes faster than an open-source project's stable API. Pair this page with chaos engineering for the hypothesis-driven discipline Gremlin is built to enforce, chaos engineering at scale for how scheduled Scenarios fit into a maturing program, and SLIs, SLOs & error budgets for how much error budget an experiment like Rocky's is actually allowed to spend. If it's the credential rather than the platform you're after, that's Gremlin Enterprise Chaos Engineering Certification — a distinct page, worth reading only once this one feels comfortable.