The SRE toolchain
Every practice covered so far in this course — SLIs and error budgets, observability, on-call, chaos engineering — is implemented with real software, and this page is a map of what's actually running in production SRE teams today. You'll leave with a category-by-category reference of representative tools and, more importantly, a clear sense of where tooling stops mattering and discipline starts.
A fully stocked toolbox doesn't make you a good carpenter. You still need to know which joint to cut, how deep to drill, and when to stop and re-measure. The tools in this page's list are drills, saws, and levels — real, useful, worth knowing by name — but a team that owns every tool on this list and never actually defines an SLO or writes a postmortem is a carpenter with a beautiful toolbox and a crooked house.
Metrics and monitoring
This is the foundation layer: numeric time series — request rate, error count, latency percentiles, CPU, queue depth — collected on a regular interval and queryable for dashboards and alerts. See monitoring & observability for how these feed SLIs directly.
- Prometheus — a pull-based, open-source time-series database with its own query language (PromQL); the de facto standard for Kubernetes-native metrics and the model most other tools in this category converge toward.
- Grafana — the dashboarding and visualization layer, usually paired with Prometheus or another time-series backend rather than replacing one; also ships its own alerting engine.
- Datadog — a commercial, fully hosted metrics/APM/logs platform that trades self-hosting effort for a managed pipeline and a large library of pre-built integrations.
- InfluxDB — a purpose-built time-series database used as a Prometheus alternative or for higher-cardinality / longer-retention workloads than Prometheus's local storage handles comfortably.
Distributed tracing
Metrics tell you that latency went up; a trace tells you where in a multi-service call chain the time actually went. A trace is a tree of timed spans — one per service hop — stitched together by a propagated request ID, which is exactly the piece pure metrics can't reconstruct after the fact.
- OpenTelemetry (OTel) — the CNCF-governed instrumentation standard (SDKs, a collector, and a wire protocol) for emitting traces, metrics, and logs; it's an instrumentation layer, not a backend, so traces still need to land somewhere.
- Jaeger — an open-source tracing backend originally built at Uber, commonly paired with OpenTelemetry as the storage and query layer.
- Honeycomb — a commercial observability platform built around high-cardinality event data and fast ad-hoc querying, positioned as a distinct alternative to the metrics-plus-traces split.
- Zipkin — an earlier open-source tracing system (Twitter-originated) that established much of the span/trace model OpenTelemetry and Jaeger build on; still in production use, though newer instrumentation increasingly targets OTel first.
SRE toolchain, by function
───────────────────────────────────────────────────────────
Metrics & monitoring Prometheus · Grafana · Datadog · InfluxDB
Distributed tracing OpenTelemetry · Jaeger · Honeycomb · Zipkin
On-call & paging PagerDuty · Opsgenie · Grafana OnCall · VictorOps
Chaos engineering Chaos Monkey · Gremlin · Litmus · AWS FIS
Load / perf testing k6 · Locust · Apache JMeter · Gatling
SLO / error-budget Nobl9 · Sloth · Google Cloud SLO monitoring
───────────────────────────────────────────────────────────
None of these rows fix an undefined SLO or an unsustainable
on-call rotation. The tool sits under the practice — it never
replaces it.
On-call, incident, and paging tools
These tools route an alert from a monitoring system to a human, manage the escalation policy if that human doesn't acknowledge in time, and coordinate the incident once it's live. See incident management & on-call for the human process these tools support.
- PagerDuty — the long-standing market leader for alert routing, on-call scheduling, and escalation policies; also offers incident-response workflow features (status pages, response coordination) beyond raw paging.
- Opsgenie — Atlassian's paging and on-call scheduling product, commonly chosen for its integration with Jira and Confluence in Atlassian-centric shops.
- Grafana OnCall — an open-source-first paging tool that integrates natively with Grafana alerting, useful for teams that want on-call scheduling without a separate commercial contract.
- VictorOps (Splunk On-Call) — a paging and incident-collaboration tool, now part of Splunk's observability suite following its acquisition.
Chaos engineering tools
These deliberately inject failure into a running system — killed processes, network latency, resource exhaustion — to verify that the resilience mechanisms covered in reliability patterns actually work under real conditions rather than only on paper. See chaos engineering for the practice and safety controls around running these.
- Chaos Monkey — Netflix's original tool, and the one that popularized the term; randomly terminates instances in production to force resilience to failure by default.
- Gremlin — a commercial chaos-engineering platform offering a broad library of attack types (CPU, network, state) with built-in blast-radius controls and scheduling.
- Litmus — a CNCF chaos-engineering project purpose-built for Kubernetes, expressing experiments as Kubernetes custom resources.
- AWS Fault Injection Service (FIS) — AWS's managed chaos-experiment service, scoped to AWS resources (EC2, ECS, RDS, and others) with native IAM-governed safety stops.
Load and performance testing
Load testing generates synthetic traffic against a system to validate capacity assumptions before real users do — the tooling side of capacity planning & performance. These matter most right before a launch, a known traffic spike, or after any architecture change that could shift a bottleneck.
- k6 — an open-source, developer-centric load-testing tool with tests written in JavaScript, built for CI integration and scriptable scenarios.
- Locust — a Python-based load-testing tool where user behavior is defined as code, popular for its distributed, cluster-scale test execution.
- Apache JMeter — the longest-established tool in this category, GUI- and Java-based, with broad protocol support beyond plain HTTP (JDBC, JMS, and others).
- Gatling — a Scala-based load-testing tool known for high throughput per test node and detailed HTML reporting out of the box.
SLO and error-budget tracking
Purpose-built tooling for the machinery in SLIs, SLOs & error budgets: computing SLI compliance against a target over a rolling window, tracking burn rate, and firing multi-window burn-rate alerts — work that's technically possible in a general dashboarding tool but error-prone to hand-roll correctly.
- Nobl9 — a commercial SLO-management platform that ingests metrics from multiple backends (Prometheus, Datadog, and others) and centralizes SLO definitions, burn-rate alerting, and error-budget reporting across teams.
- Sloth — an open-source tool that generates Prometheus recording and alerting rules from a declarative SLO spec, so multi-window burn-rate alerts don't have to be hand-written PromQL.
- Google Cloud SLO monitoring — SLO tracking built into Cloud Monitoring for services running on GCP, following the same math this course teaches directly from the source that popularized it.
Every category above has real, meaningful differences between its options — self-hosted versus managed, open-source versus commercial, breadth versus depth — and choosing well among them is a legitimate engineering decision. But the choice is bounded: a team running Prometheus, Grafana, PagerDuty, and Gremlin with no defined SLOs, no error budget, and a rotation nobody sustains is worse off than a team running spreadsheets and shell scripts with all three of those things actually in place. Tooling accelerates a practice that already exists; it does not create one.
That's the note to end this reference on. The tools above will change — new entrants replace old ones, acquisitions consolidate categories, self-hosted options mature or stagnate — on a timescale much faster than the underlying discipline does. An SLO, a blameless postmortem, and a sustainable on-call rotation are the same ideas whether they're implemented in Prometheus or in whatever replaces it a decade from now. If you're evaluating a team's SRE maturity, or your own, start with the questions this course has spent every other page on — are SLOs actually defined and reviewed, are postmortems actually written and actioned, is on-call actually sustainable — and treat the tool inventory as a distant second question.
1. Name one representative tool each for distributed tracing and for chaos engineering, and state in one sentence what each category is for. 2. What's the difference in scope between a general dashboarding tool like Grafana and a purpose-built SLO tool like Sloth or Nobl9? 3. Why would a team with a full, well-chosen toolchain but no defined SLOs still be considered immature from an SRE standpoint?
Check your answers
- Tracing example: Jaeger, OpenTelemetry, Honeycomb, or Zipkin — used to see where time went across a multi-service call chain via a tree of connected spans. Chaos example: Chaos Monkey, Gremlin, Litmus, or AWS FIS — used to deliberately inject failure and verify resilience mechanisms actually hold under real conditions.
- Grafana visualizes whatever time series you point it at but has no built-in concept of an SLO target, rolling compliance window, or burn rate; SLO-specific tools compute that math (multi-window burn-rate alerting, error-budget consumption) directly from a declarative SLO definition instead of requiring it to be hand-built.
- Because tools support the practice, they don't substitute for it — reliability comes from the discipline of defining and tracking SLOs, running blameless postmortems, and sustaining on-call, not from which software sits underneath that discipline. A well-tooled team without those practices is still just firefighting with better dashboards.