Sloth

Multi-window, multi-burn-rate alerting walked through, by hand, everything a single SLO needs: seven recording rules computing an error ratio at seven different windows, and four burn-rate conditions folded into a page tier and a ticket tier. That was one service, one SLO. Sloth is an open-source, single-purpose generator — you hand it roughly fifteen lines of declarative YAML naming a target percentage and the two PromQL queries that count good and bad events, and it compiles the entire rule set above from that formula, deterministically, the same way every time, for as many SLOs as you own. What used to be hand-typed PromQL that somebody had to re-derive correctly on every review becomes a build artifact: generated once from a source of truth, diffable in a pull request, and never hand-verified line by line again.

What Sloth is and the problem it solves

☺ Like you're 10: It reads a short "here's my target, here's how I count failures" file and writes out the whole multi-window alert rule set for you — the exact same math, every time, instead of somebody retyping it.

Sloth is a small, independent open-source project (Apache-2.0 licensed, written in Go, created by Xabier Larrakoetxea) with one job: implement the SRE Workbook's multi-window, multi-burn-rate method as a compiler rather than a set of instructions a human follows. You describe a service level objective declaratively — a target percentage and the two queries that separate good events from bad ones — and Sloth emits the complete, ready-to-load Prometheus rule group: every recording rule the burn-rate math needs, and the alerting rules that page or ticket off of it. It is not a monitoring backend, not a dashboard, and not a place your SLO data lives — it has no database and answers no queries at read time. It is closer in spirit to a compiler or to helm template: it runs, it produces text, and then it's out of the picture until the spec changes again.

The problem: burn-rate math, multiplied by every SLO you own

One SLO's worth of hand-written rules, as the worked example on the burn-rate page shows, is already a page of YAML: seven recording rules for the seven windows the four-tier table needs, plus four alert conditions correctly paired long-window-AND-short-window and correctly multiplied against 1 − SLO. A team with five services and one SLO each is already maintaining thirty-five recording rules and roughly twenty comparisons by hand. A team with fifty services, some carrying two or three SLOs each — availability and a latency SLO, say — is maintaining rule counts in the high hundreds, all of which must stay faithful to the same formula, and none of which Prometheus itself checks for correctness beyond valid PromQL syntax. A typo'd multiplier, a threshold computed against the wrong 1 − SLO, or two services whose windows have quietly drifted out of the standard 1h/6h/1d/3d shape are all silent failures — the rules still evaluate, they just don't mean what anyone thinks they mean. Sloth removes the copy-paste step from that picture entirely: every SLO's rules are generated from the same formula, so a mistake, if there is one, is in one place — the generator — rather than scattered across however many services hand-typed it independently.

How it works: two ways to run the same generator

☺ Like you're 10: You can either run it once as a one-off command and save the output, or install it inside the cluster so it keeps regenerating automatically whenever the spec changes.

Sloth ships as a single binary with two genuinely different consumption models. Both read the exact same declarative spec shape; they differ only in who triggers generation and where the output ends up.

CLI mode: a one-shot generator for a GitOps pipeline

sloth generate is a stateless, one-shot command: point it at a spec file (or a directory of them), and it writes a plain Prometheus rule-group YAML file to disk and exits. Nothing about Sloth keeps running afterward — the output is exactly the kind of rules/*.yml file Prometheus's own rule_files: directive already expects, the same mechanism shown loading rules/checkout-slo.yml in Prometheus's own config example. The natural home for this mode is CI: a pull request changes the SLO spec, a pipeline step runs sloth generate and promtool check rules against the output, and the generated file is what actually ships — reviewable in the diff, and reproducible from the spec at any later point in history.

Kubernetes operator mode: PrometheusServiceLevel as a live CRD

The second mode installs Sloth's controller into the cluster (via its manifests or Helm chart) and registers a custom resource definition, PrometheusServiceLevel, in the API group sloth.slok.dev. You apply PrometheusServiceLevel objects the same way you'd apply a Deployment, and the controller watches them continuously, reconciling each one into a PrometheusRule object — the exact CRD the Prometheus Operator (or kube-prometheus-stack) already watches and loads automatically, no rule_files: edit or reload required. This mode trades the CI step for a live, in-cluster reconciliation loop: edit the CR, the generated PrometheusRule updates itself, and Prometheus picks up the change on its own next rule-reload cycle. It's the more "GitOps-native" of the two in the sense that a reconciler like Argo CD or Flux can own the PrometheusServiceLevel object exactly like any other manifest, and the rest of the generation happens inside the cluster rather than in CI.

The SLO spec: anatomy of a PrometheusServiceLevel

☺ Like you're 10: Five things to fill in: a name, a target percentage, how to count the bad ones, how to count all of them, and where the alerts should go.

Both modes read the same document shape. Extending the checkout API's 99.9% availability SLO already worked through on the SLIs, SLOs & error budgets and burn-rate pages, here's the spec that compiles into that page's entire rule set:

apiVersion: sloth.slok.dev/v1
kind: PrometheusServiceLevel
metadata:
  name: checkout-availability
spec:
  service: "checkout"
  labels:
    team: checkout-squad          # stamped onto every generated rule and alert
  slos:
    - name: "requests-availability"
      objective: 99.9              # a PERCENTAGE, not a ratio — 99.9, never 0.999
      description: "checkout API requests that don't return a 5xx"
      sli:
        events:
          # {{.window}} is substituted with each of the 7 windows Sloth needs —
          # you write the query ONCE, Sloth renders it at every window itself
          error_query: |
            sum(rate(http_requests_total{route="/checkout",code=~"5.."}[{{.window}}]))
          total_query: |
            sum(rate(http_requests_total{route="/checkout"}[{{.window}}]))
      alerting:
        name: CheckoutHighErrorRate
        labels:
          category: availability
        annotations:
          runbook: "https://runbooks.acme.internal/checkout-availability"
        page_alert:
          labels: { severity: page }
        ticket_alert:
          labels: { severity: ticket }

The events-based SLI: error_query and total_query

This is the events SLI type — the same request-based convention monitoring & observability already establishes as the default: two PromQL queries, one counting bad events and one counting all events, with the ratio between them being the SLI. Sloth's one real templating trick is the {{.window}} placeholder: you write error_query and total_query exactly once, each with that placeholder standing in for a Go range-vector duration, and Sloth renders both queries seven times — once per window the burn-rate math needs — rather than making you write out [5m], [30m], [1h], [2h], [6h], [1d], and [3d] variants by hand. Sloth also supports a raw SLI type (a single query that already returns the error ratio directly, for services that precompute it elsewhere) and a plugin system — a named, parameterized SLI template defined once and referenced by name from many SLO specs, so fifty services with an identical "HTTP 5xx ratio on this route" shape don't each restate the same two PromQL strings.

The alerting block: naming the page and the ticket

alerting.name is the base identity every generated alert is built from; page_alert and ticket_alert each accept their own extra labels and annotations, which is where Alertmanager routing actually gets its hooks — a severity: page label is what a routing tree matches on to reach a pager, and severity: ticket is what routes to a queue instead. Either alert tier accepts a disable: true flag if a given SLO genuinely shouldn't page — a batch-processing SLO you only ever want a ticket from, say — without deleting the SLO's recording rules along with it.

What actually gets generated

☺ Like you're 10: Out comes exactly what last chapter typed by hand — seven number-tracking rules and two real alerts — just written by the machine instead of a person.

Run sloth generate (or let the controller reconcile the CR above) and the output is a standard Prometheus rule group, structurally identical in shape to the hand-written example on the burn-rate page — the recording-rule names Sloth itself chooses differ slightly from the illustrative ones used there, so treat the exact series names below as representative rather than gospel and confirm them against your installed version's actual output:

Recording rules: one per window, plus budget bookkeeping

groups:
  - name: sloth-slo-sli-recordings-checkout-requests-availability
    rules:
      - record: slo:sli_error:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{route="/checkout",code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{route="/checkout"}[5m]))
        labels: { sloth_service: checkout, sloth_slo: requests-availability, team: checkout-squad }
      # ...identical shape repeated for ratio_rate30m, 1h, 2h, 6h, 1d, 3d

  - name: sloth-slo-meta-recordings-checkout-requests-availability
    rules:
      - record: slo:current_burn_rate:ratio
        expr: slo:sli_error:ratio_rate5m / (1 - 0.999)
      - record: slo:period_error_budget_remaining:ratio
        expr: 1 - ((sum_over_time(slo:sli_error:ratio_rate5m[30d])) / (1 - 0.999))
      - record: sloth_slo_info
        expr: vector(1)
        labels: { sloth_service: checkout, sloth_slo: requests-availability, sloth_objective: "99.9" }

Two alerting rules that reproduce the four-tier table

  - name: sloth-slo-alerts-checkout-requests-availability
    rules:
      - alert: CheckoutHighErrorRate
        expr: |
          (
            slo:sli_error:ratio_rate1h  > (14.4 * (1 - 0.999))
            and
            slo:sli_error:ratio_rate5m  > (14.4 * (1 - 0.999))
          )
          or
          (
            slo:sli_error:ratio_rate6h  > (6 * (1 - 0.999))
            and
            slo:sli_error:ratio_rate30m > (6 * (1 - 0.999))
          )
        labels: { severity: page, category: availability }
        annotations:
          runbook: "https://runbooks.acme.internal/checkout-availability"

      - alert: CheckoutHighErrorRate
        expr: |
          (
            slo:sli_error:ratio_rate1d  > (3 * (1 - 0.999))
            and
            slo:sli_error:ratio_rate2h  > (3 * (1 - 0.999))
          )
          or
          (
            slo:sli_error:ratio_rate3d  > (1 * (1 - 0.999))
            and
            slo:sli_error:ratio_rate6h  > (1 * (1 - 0.999))
          )
        labels: { severity: ticket, category: availability }
        annotations:
          runbook: "https://runbooks.acme.internal/checkout-availability"

Read those two alert expressions against the four-row table on the burn-rate page and they're the same eight comparisons, just folded together with or into two alerts instead of stated as four: the page alert is "fast tier OR moderate tier," the ticket alert is "slow tier OR slowest tier," each internal pair still joined by the AND that makes the short window responsible for a fast reset. That folding is the one editorial decision Sloth makes on your behalf — Alertmanager routes on the severity label either way, so two alerts with four total conditions between them behaves identically, at the routing layer, to four separate alerts.

Day-to-day commands

☺ Like you're 10: Check the spec for mistakes, turn it into rules, then make sure the rules themselves are valid before Prometheus ever sees them.

# CLI mode
$ sloth version
$ sloth validate -i checkout-slo.yaml            # spec-level checks: objective range, required fields
$ sloth generate -i checkout-slo.yaml -o rules/checkout-slo.generated.yaml
$ sloth generate -i specs/ -o rules/              # a whole directory of specs at once, one output per file

# always validate the OUTPUT with Prometheus's own tool too — Sloth being happy
# doesn't guarantee the PromQL it emitted is syntactically sound on your Prometheus version
$ promtool check rules rules/checkout-slo.generated.yaml
$ promtool test rules rules_test.yaml             # unit-test the generated rules against synthetic series

# commit the generated file — it's a build artifact, and the diff is the whole point:
# a change in `objective:` should visibly change every threshold in the PR
$ git add rules/checkout-slo.generated.yaml && git commit -m "checkout: 99.9% -> 99.95%"

# Kubernetes operator mode — after installing the controller (manifests or Helm chart)
$ kubectl apply -f checkout-prometheusservicelevel.yaml
$ kubectl get prometheusservicelevels -A          # every SLO spec currently registered
$ kubectl get prometheusrule checkout-availability -o yaml   # the rules the controller reconciled
$ kubectl describe prometheusservicelevel checkout-availability   # reconcile status and any spec errors
$ kubectl logs -l app.kubernetes.io/name=sloth -n monitoring    # the controller's own reconcile log

Gotchas and failure modes

☺ Like you're 10: A perfect recipe machine still can't tell if you gave it the wrong ingredients — and it only writes a new recipe card when somebody actually asks it to.

Garbage in, mathematically perfect garbage out

Sloth has no way to know whether error_query and total_query are the right SLI for the thing users actually experience. If total_query silently double-counts retries, or error_query misses a whole error-code class because someone typed 5.. instead of 5\d\d, Sloth will compile a flawless, internally consistent four-tier alert set around a wrong number — the generator's correctness is entirely downstream of the two queries you hand it. Review the SLI queries themselves with the same rigor monitoring & observability asks for elsewhere; a generator cannot substitute for that judgment.

The "did anyone regenerate" trap (CLI mode)

Nothing forces you to re-run sloth generate and commit the diff when the spec changes — that discipline lives entirely in your CI pipeline, and a CI step that only lints the spec without also regenerating and diffing the output can let the two drift silently. The failure mode is exactly the one Helm's rendered-manifests pattern warns about: a reviewer approves a spec change believing it changed production behavior, while the actually-loaded rules file — last regenerated three commits ago — still reflects the old objective. A CI check that fails the build unless sloth generate's fresh output byte-matches the committed file closes this gap completely.

Operator mode's own watch-mismatch trap

The controller happily reconciles a PrometheusServiceLevel into a PrometheusRule object even when the Prometheus Operator isn't watching that namespace or doesn't match its label selector — the object exists, kubectl get prometheusrule shows it, and Prometheus never loads it anyway. This is the identical root cause behind any ruleSelector or serviceMonitorSelector mismatch, and the fix is the same: check what labels and namespaces the Prometheus Operator's own CR is configured to watch before assuming a missing alert means Sloth failed, rather than that nobody was ever watching the object it produced.

The rest worth knowing

• Renaming an SLO changes alert identity. alerting.name and the SLO's own name feed into every generated series and alert label. Rename either and any Alertmanager silence or routing rule keyed to the old identity goes stale — treat a rename with the same care as renaming any other versioned, routed artifact.
• objective is a percentage, not a ratio. 99.9 is correct; 0.999 is a common typo that produces an absurd, nowhere-near-real target and a generator that's happy to compile it anyway, because nothing about the spec format catches an off-by-100 semantically.
• Sloth is not a runtime SLO engine. It computes nothing at query time and holds no dashboard of its own by default — it ships example Grafana dashboards designed to read the recording-rule names it generates, but Grafana (or another tool) still has to do the actual visualizing. Contrast this with Nobl9, which is a hosted platform that both defines and continuously evaluates SLOs itself.
• A spec change in the compliance-window default (moving off the standard 30-day window) needs the multipliers re-derived, exactly as the burn-rate page warns — see SLO windows & composite SLOs for what a non-default window changes, and check current Sloth documentation for how that period is configured in your installed version before assuming a flag name here.

Where Sloth sits in this course's stack

☺ Like you're 10: It sits right between the SLO math you learned and the Prometheus that actually runs it — turning one into the other automatically.

Sloth's whole reason for existing is the seam between two pages this course already covers: the burn-rate formula in multi-window, multi-burn-rate alerting, and the Prometheus rule-loading mechanism — rule_files:, or a PrometheusRule object under the Prometheus Operator — described in Prometheus. It consumes the SLO target defined the way SLIs, SLOs & error budgets teaches, and its output feeds the exact alerting pipeline incident management & on-call assumes downstream. It's also complementary, not competing, with Grafana (which still draws the picture) and with a commercial platform like Nobl9, which typically ingests its data from a Prometheus deployment rather than generating rules for one. If you want hands-on reps building this exact pipeline rather than just reading about it, Capstone Part 2 — build the monitoring & alerting is where the burn-rate rules get built for real, and the SLO & error-budget drill is a faster, standalone rep of the underlying arithmetic Sloth automates.

Alternatives and when to choose it

☺ Like you're 10: Other tools solve the same "don't hand-type the alert math" problem — some by generating the same kind of file, some by running the whole thing themselves so you never touch PromQL at all.

The practical rule most Prometheus-native teams land on: reach for Sloth the moment you have more than two or three SLOs, specifically because that's roughly where hand-maintained burn-rate YAML starts drifting unnoticed rather than merely being tedious. Add a UI-forward option like Pyrra, Grafana dashboards on top of Sloth's own recording rules, or a hosted platform like Nobl9 only once reporting to people who don't read PromQL — not alerting itself — becomes the actual gap.

Going further

☺ Like you're 10: This page is enough to write a real spec and read real generated output — the project's own repository is where you go for the exact current CLI flags and CRD fields.

The canonical source is the project itself at github.com/slok/sloth — the README and the examples/ directory are worth reading end to end, since CLI flags, default windows, and the Kubernetes CRD's exact field names have shifted across releases and are worth confirming against your installed version rather than this page alone. Pair this page with multi-window, multi-burn-rate alerting for the formula being compiled, Prometheus for the rule-loading mechanism on the receiving end, and alert design & alert fatigue for why the page/ticket split this tool generates matters in the first place.