SRE Tools & Automation
This domain is the one candidates most often study in the wrong direction — memorizing which vendor makes which product instead of the two ideas the exam actually asks about. First: where a described practice sits on the automation maturity ladder, from a human reading a runbook off a wiki page to a system that notices, decides, and fixes a problem entirely on its own. Second: which functional category a tool belongs to — metrics, tracing, paging, chaos engineering, or SLO tracking — regardless of which specific product implements it. Nail those two ideas and you can answer a Tools & Automation question about a product you've never heard of, because the exam is testing the category and the maturity rung, not the logo.
Picture three ways a kitchen handles a fire. Stage one: a sign on the wall says "if you smell smoke, walk to the panel and pull the lever" — a person has to notice, decide, and act. That's a runbook. Stage two: someone rigs the lever so pulling it once runs the whole shutdown sequence instead of five separate steps — a person still has to notice the fire and choose to pull it, but now one action does the work of many. That's runbook automation. Stage three: a heat sensor in the ceiling notices the fire itself and fires the sprinklers with nobody touching a lever at all — the decision was made in advance, not in the moment. That's autonomous remediation. All three protect the same kitchen. The exam wants you to know which stage a scenario describes, not which brand of sprinkler is bolted to the ceiling.
Domain 5 of 8: what this part of the exam actually tests
☺ Like you're 10: This part checks whether you can sort a described tool or habit into the right bucket — which rung of the ladder, which drawer of the toolbox — not whether you've memorized a pricing page.
SRE Tools & Automation is domain five of the eight that make up the DevOps Institute SRE Foundation (SREF) syllabus, sitting between Monitoring & Service Level Indicators and Anti-Fragility & Learning from Failure. The full map, in order, is SRE Principles & Practices, SLOs & Error Budgets, Reducing Toil, Monitoring & SLIs, this page, Anti-Fragility, Organizational Impact, and SRE, Other Frameworks & the Future. DevOps Institute doesn't publish a rigid fixed percentage per domain the way some vendor exams do, and across accredited courseware Tools & Automation tends to be one of the lighter-weighted domains on the roughly 40-question, 60-minute, closed-book paper — the mechanics of the exam itself (current timing, pass mark, question count) are covered fully on The SREF Exam, and it's worth confirming those specifics there before you book, since DevOps Institute has revised format details before.
What this domain covers, specifically: the judgment of how much human decision-making should stay in an operational loop, and the ability to recognize a tool's job from a functional description even when its name is unfamiliar. It sits downstream of two domains you should already have solid: Reducing Toil supplies the criteria for which manual work is even worth automating, and Monitoring & SLIs supplies the signal that any of this automation reacts to. This page is the machinery that sits on top of both.
The automation maturity ladder — from runbook to autonomous remediation
☺ Like you're 10: Four rungs, one question at each: who notices the problem, and who decides to act on it?
Every operational response — human or machine — answers the same two questions: who notices and who decides. The four rungs below differ only in how those two questions get answered, and every rung after the first still involves a script. That last point is the exam's favorite trap, so hold onto it.
- 1 · Runbook. A documented, step-by-step procedure a human reads and executes by hand during an incident — a human notices (usually via a page), a human diagnoses, a human types every command. Its value is consistency: even an on-call engineer three weeks into the job can follow it at 3am. See Capstone Part 3 — Write the On-Call Runbook for what a good one actually looks like.
- 2 · Runbook automation. The decision still belongs to a human, but the mechanical steps get compiled into one script or one-click job — a human still decides when to run it, but no longer manually types each command. This is what most teams mean when they say "we automated the runbook," and it's a genuinely useful rung: it turns a 12-step procedure prone to fat-fingering step 7 into a single, tested action. Real examples: PagerDuty Process Automation (the product built from Rundeck, which PagerDuty acquired in 2022 — verify current naming against PagerDuty's own docs, since it's been repackaged more than once), Ansible AWX/Tower job templates, AWS Systems Manager Automation documents, or a ChatOps slash command like
/restart checkout-prodwired to a webhook that runs a script. - 3 · Human-approved (semi-autonomous) remediation. The system itself detects the anomaly and proposes a fix, but execution waits behind an approval gate. The decision splits: the system decides what, a human still decides whether. A Slack bot posting "error-budget burn rate hit 14× on checkout, recommend rollback to
v1.4.2— react ✅ to proceed" is this rung exactly. - 4 · Autonomous remediation (self-healing). The system detects, decides, and executes with no human in the loop; a human is notified after, not asked before. Most engineers already run examples of this without labeling it "SRE automation": a kubelet restarting a container that fails its liveness probe, a Kubernetes HPA scaling pods on CPU, a cloud auto-scaling group replacing an unhealthy instance, or Argo Rollouts / Flagger auto-aborting a canary when its metric analysis breaches a threshold — see release engineering & progressive delivery and Kubernetes Reliability Patterns for both in depth.
Every rung from 2 through 4 involves a script. The exam trap is assuming "there's a script" means "it's automated remediation" — it doesn't. The rung is set entirely by who decides to trigger the action: a human clicking a button is rung 2 no matter how sophisticated the script behind that button is; a system acting on its own analysis is rung 4 even if the resulting change is trivial. Read the scenario for the decision-maker, not the tooling.
Guardrails: the price of removing the human from the loop
☺ Like you're 10: The more a system can do without asking, the more it needs its own leash — a limit on how much damage it's allowed to do per minute, and a way to yank it back fast.
Climbing the ladder trades response speed for blast radius, and a responsible rung-3 or rung-4 system earns that trade with explicit guardrails rather than assuming good intentions. Five that show up repeatedly in real incident postmortems and in exam scenarios:
- Blast-radius limits. Rate-limit how much an autonomous action can do per window — "restart at most one pod per five minutes," not the whole deployment at once — so a false-positive detection can't cascade into a self-inflicted outage.
- A kill switch. One flag or command that disables the automation immediately, independent of whatever triggered it. Timmy's question from earlier in this course — does it have a timeout? — is really this same guardrail asked a different way.
- An audit log. Every autonomous action needs the same paper trail a human action gets: who (or what) triggered it, what it changed, and when. An automated rollback that isn't logged is invisible in the incident timeline, which makes the postmortem that follows incomplete by design.
- Idempotency. An action retried twice — because a detector fired twice, or a network blip caused a duplicate trigger — must not double the effect. Restarting an already-restarting pod should be a no-op, not two restarts.
- Staged trust. Canary the automation logic itself the same way you'd canary a code change — run it in propose-only (rung 3) mode, watch it for a while, and only promote it to fully autonomous (rung 4) once its proposals have been right often enough to trust unattended.
The most widely cited industry cautionary tale here is Knight Capital Group's August 2012 trading incident: a deployment script failed to update one of eight production servers, leaving old dormant test code active on it. That code began executing live trades with no human reviewing them in real time, and by the time anyone intervened the firm had lost roughly $440–460 million in about 45 minutes — figures vary slightly by source, so verify the exact number if you cite it elsewhere, but the order of magnitude and the timescale are the point. The lesson the exam wants you to draw isn't "automation is dangerous" — it's that removing a human from the loop also removes the human who would have caught the mistake, so the guardrails above aren't optional polish on rung 3 and 4 automation; they're the thing that makes the trade defensible at all. See SRE anti-patterns & pitfalls for more failure modes in this shape.
Pick one genuinely recurring manual task from your own on-call rotation — something you or a teammate has done by hand more than twice. Place it on the ladder honestly: is it still rung 1 (a runbook nobody's scripted), or already rung 2 (a script that still needs a human to click it)? Then ask what it would take to move it up exactly one rung — not straight to rung 4 — and name the one guardrail from the list above that rung would need before you'd trust it. If you can't name a guardrail you'd actually be comfortable with, that's a legitimate reason to stay one rung lower for now.
The toolchain categories the exam expects you to recognize
☺ Like you're 10: Five drawers in the same toolbox — know what's in each drawer and you can find the right one even for a brand you've never held before.
Underneath the ladder sits the actual software that makes any of it possible, and the exam groups it into functional categories rather than testing product trivia — a discipline made easier to justify by the fact that the SREF exam is closed-book: you can't look a vendor name up mid-question, so the only thing that reliably works is knowing what a category of tool does. The five categories below are exactly the ones this course's SRE toolchain reference catalogs in full; this table is the exam-relevant summary, not a replacement for it.
| Category | What it actually does | Representative tools | Covered in depth |
|---|---|---|---|
| Metrics & monitoring | Collects numeric time series — request rate, error count, latency percentiles, resource usage — on a regular interval, queryable for dashboards and alerts. | Prometheus, Grafana, Datadog, InfluxDB | Monitoring & observability |
| Distributed tracing | Shows where in a multi-service call chain time actually went, as a tree of timed spans stitched together by a propagated request ID. | OpenTelemetry, Jaeger, Honeycomb, Zipkin | Monitoring & observability |
| On-call & paging | Routes an alert to a human, escalates if they don't acknowledge in time, and coordinates the incident once it's live. | PagerDuty, Opsgenie, Grafana OnCall, VictorOps | Incident management & on-call |
| Chaos engineering | Deliberately injects failure — killed processes, added latency, resource exhaustion — to verify resilience mechanisms hold under real conditions, not just on paper. | Chaos Monkey, Gremlin, Litmus, AWS FIS | Chaos engineering |
| SLO / error-budget tracking | Computes rolling-window SLI compliance against a target, tracks burn rate, and fires multi-window burn-rate alerts — error-prone to hand-roll correctly. | Nobl9, Sloth | SLIs, SLOs & error budgets |
Two categories sit just outside the exam's core five but are worth knowing for the same reason: log aggregation (Elastic Stack, Loki) and load/performance testing (k6, Locust, Apache JMeter, Gatling), the latter feeding directly into capacity planning & performance. If you want vendor-specific depth beyond what the SREF asks for, Prometheus Certified Associate and OpenTelemetry Certified Associate are the natural next credentials for the metrics and tracing rows respectively.
Reading an exam question: category over brand
☺ Like you're 10: If a question names a product you've never heard of, don't panic — read what it does, not what it's called, and the category falls right out.
Two worked examples in the exact shape the SREF likes to ask them. Cover the answer and reason it out first.
Q1. A team wants a system that pages the primary on-call engineer only after a service's error-budget burn rate has exceeded a fast-burn threshold across two separate lookback windows, and stays silent on any single-window spike. Which category of tool is being described? A) A generic dashboarding tool B) A distributed tracing backend C) A dedicated SLO / error-budget tracking tool D) A chaos-engineering platform Correct: C. A generic dashboard (A) can plot burn rate if you build the query yourself, but it has no built-in concept of a rolling compliance window or a multi-window comparison — that math is exactly what purpose-built SLO tooling exists to compute. See multi-window burn-rate alerting for the arithmetic behind this pattern, and Drill — Redesign a Noisy Alert to practice spotting the single-window false positive this design is built to avoid. Q2. An on-call engineer runs the same five commands from a wiki page every time a queue-depth alert fires. The team wires those five commands into a single Slack-triggered job that any on-call engineer can invoke with one click — but a human still has to click it. What rung of the automation maturity ladder does this describe? A) Manual runbook B) Runbook automation C) Human-approved auto-remediation D) Fully autonomous self-healing Correct: B. The decision to act is still a human's; only the mechanical execution moved into a script. Nothing in this scenario has the system detecting or deciding anything on its own — that's what would push it to C or D.
Q1's answer leans on the arithmetic covered in Multi-Window, Multi-Burn-Rate Alerting, and Drill — Redesign a Noisy Alert is the hands-on practice for spotting exactly the single-window false positive it's built to avoid. More of exactly this question pattern lives in Practice · Monitoring & SRE Tools and the full SREF Practice Questions bank; Answer Triage covers the general technique of eliminating options by what a category can't do, which is exactly the move both answers above rely on.
That's the whole domain: a four-rung ladder that's really just "who decided," a five-category toolbox that's really just "what does it compute," and the discipline to answer both from the description in front of you rather than from a memorized product list. Next in the blueprint is Anti-Fragility & Learning from Failure — the domain that asks what you do once automation like this has been tested against the chaos-engineering row above and something still breaks anyway.
Benny the Beaver: I wired our checkout restart into one Slack command. Runbook automation, done — pretty much autonomous now, right?
Nutty the Squirrel: Wrong drawer. You still click the button. That's rung two, not rung four.
Foxy: So what actually turns rung two into rung four?
Recon the Robot: BEEP. Nobody clicks. I watch the canary's own metrics, and if they breach, I roll it back before a human's even read the alert.
Timmy the Turtle: And when you roll back on your own, Recon — does it have a limit? One bad rollback per five minutes, not the whole fleet at once?
Recon the Robot: Affirmative. Rate-limited, logged, reversible. Otherwise Benny doesn't get to trust me with the pager.
Nutty the Squirrel: And that's the whole domain in one exchange — who decided, and what stops it from deciding too much.
1. Name the one factor that distinguishes "runbook automation" from "autonomous remediation" — it isn't whether a script is involved. 2. Name the five toolchain categories this domain expects you to recognize by function, and one representative tool for each. 3. Name two guardrails a responsible rung-4 (autonomous) system needs before it's trusted to act without a human approval gate, and why each matters. 4. In the burn-rate worked example, why is a generic dashboarding tool the wrong answer even though it could technically plot the same data?
Check your answers
- Who decides to trigger the action. Runbook automation still has a human deciding when to run the script; autonomous remediation has the system itself detecting and deciding, with a human only notified afterward. A script exists at both rungs.
- Metrics & monitoring (e.g. Prometheus, Grafana), distributed tracing (e.g. Jaeger, OpenTelemetry), on-call & paging (e.g. PagerDuty, Opsgenie), chaos engineering (e.g. Gremlin, Chaos Monkey), and SLO/error-budget tracking (e.g. Sloth, Nobl9).
- Any two of: blast-radius limits (caps how much damage one bad decision can cause), a kill switch (a way to disable the automation immediately, independent of its trigger), an audit log (so an autonomous action shows up in the incident timeline the same way a human action would), idempotency (a retried action can't double its own effect), or staged trust (running propose-only before promoting to fully autonomous).
- Because a generic dashboard has no built-in concept of a rolling compliance window or a multi-window comparison — it can chart whatever query you hand it, but it doesn't compute burn rate against a target or evaluate two lookback windows against each other on its own. That's specifically what purpose-built SLO tooling exists to do.