SLOs, Error Budgets & Toil

Somewhere between "it deployed successfully" and "the customer had a good experience" sits a gap that CI/CD green checkmarks don't measure. Site Reliability Engineering closes that gap with three ideas — the SLI, the SLO, and the error budget it produces — and this page teaches them for exactly one purpose: turning "should we ship this today?" from a gut call or a political argument into a number both Dev and Ops already agreed to respect. This is not the full SRE curriculum — measuring reliability, running on-call, and engineering for failure are each disciplines in their own right, and the platform's separate SRE course goes deep on all three. Here, the scope is narrower and sharper: how an error budget changes what a DevOps engineer ships today, and how toil quietly eats that budget before a release decision is even made.

SLIs, SLOs, and SLAs: the vocabulary a release decision runs on

☺ Like you're 10: One word for the thing you measure, one for the target you promised yourself, and one for the target you promised a customer with money attached.

Three terms get used loosely and mean precisely different things. A Service Level Indicator (SLI) is a quantitative measurement of some aspect of the service, expressed as a ratio: good events over valid events, over a window of time — "the fraction of checkout requests in the last five minutes that returned successfully and in under 300ms." A Service Level Objective (SLO) is the target you set for that SLI — "99.9% of checkout requests will succeed and be fast, measured over a rolling 30 days." A Service Level Agreement (SLA) is a contractual promise, usually to an external customer, with a penalty attached for missing it — a refund, a credit, an escape clause. The relationship only runs one direction: your SLO should always be stricter than your SLA, so you find out you're in trouble from your own dashboards before a customer finds out from a missed invoice credit.

Good SLIs share three properties. They're user-centric — measured as close to what the user actually experiences as you can get (a load balancer's request log, not a server's internal CPU graph). They're request-based or window-based — either a ratio of good-to-valid events (availability, correctness) or a ratio of good-to-total time (a batch job that either finished on time or didn't). And they're few — three or four SLIs per service is normal; thirty is a sign you're measuring the system instead of the user.

These four map directly onto the four golden signals already in this course's glossary — latency, traffic, errors, saturation — but an SLI narrows a golden signal into one specific, user-facing ratio you're willing to be held to, rather than a graph you merely watch.

Turning a target into a spendable number: the error budget

☺ Like you're 10: If your target is 99.9% good, the leftover 0.1% is a real, countable amount of "allowed to go wrong" — and every failure spends a little of it.

The move that makes SLOs actionable rather than aspirational is treating the gap between 100% and the SLO as a real, spendable quantity. Error budget = 100% − SLO, applied to the SLO's measurement window. A 99.9% availability SLO over a rolling 30-day window doesn't mean "try really hard" — it means the service is allowed exactly 0.1% of requests (or minutes, for a time-based SLI) to fail before the objective is breached, and that 0.1% converts to a concrete, countable number.

This is the number the whole rest of this page is about. It's the same number whether you spend it in one dramatic three-hour outage or in a thousand tiny half-second timeouts spread across the month — the budget doesn't care how it burns, only that it burns. And critically, it's a number both Dev and Ops can point to instead of arguing from incentive: this is precisely the wall of confusion problem — Dev rewarded for shipping, Ops rewarded for stability — resolved by giving both sides one shared, pre-agreed currency instead of two competing opinions.

Burn rate: how fast the budget is going, not just how much is gone

☺ Like you're 10: Spending the same allowance in one afternoon is a much bigger warning sign than spending it evenly across the whole month.

Burn rate is the speed at which a service is consuming its error budget, expressed as a multiple of the sustainable rate. A burn rate of 1× means the service is consuming budget exactly as fast as the SLO allows — spend it evenly and you land at exactly 0% remaining on the last day of the window, which is fine. A burn rate of 14.4× means the service is consuming a month's entire budget in about two days if it keeps up — which is the kind of number that should page someone at 3 a.m., not wait for a dashboard review. Alerting on the raw SLO ("we're below 99.9% right now") is too noisy for short blips and too slow for real incidents; alerting on burn rate across a short window paired with a longer confirming window is the widely-used pattern from Google's SRE literature for getting both fast detection and few false pages — the full design space of multi-window, multi-burn-rate alerting is exactly the kind of depth this page is deliberately not going into; that belongs to the SRE course's own treatment of measuring reliability.

These specific multipliers are the commonly cited starting point for a 99.9% target and are worth verifying against your own SLO tooling's defaults rather than copying blind — the point to take away isn't the exact numbers, it's the shape: fast, severe burn pages a human immediately; slow, steady burn opens a ticket for the next business day, because at 1× you still have days of runway to fix it calmly.

# Fast-burn alert: current 1h error rate exceeds 14.4x the rate
# a 99.9% SLO can sustain, confirmed on a shorter 5m window.
(
  1 - (
    sum(rate(http_requests_total{job="checkout",code!~"5.."}[1h]))
    /
    sum(rate(http_requests_total{job="checkout"}[1h]))
  )
) > (14.4 * 0.001)
and
(
  1 - (
    sum(rate(http_requests_total{job="checkout",code!~"5.."}[5m]))
    /
    sum(rate(http_requests_total{job="checkout"}[5m]))
  )
) > (14.4 * 0.001)

The core mechanic: the error budget as a release-cadence throttle

☺ Like you're 10: Plenty of allowance left, ship whatever you want. Almost out, stop building new stuff and go fix what's broken.

Here is the entire discipline compressed into one operating rule: while budget remains, ship — once it's gone, freeze features and fix reliability instead. That single sentence is what separates SRE-influenced DevOps from a team that merely has a nice-looking uptime dashboard. It converts an abstract cultural argument — "are we being too reckless" versus "are we being too slow" — into a number that was agreed on before anyone was emotionally invested in a specific release.

The mechanism runs both directions, which is the part teams new to this most often miss. A healthy budget is explicit permission to move faster: widen canary steps, skip a redundant manual approval gate, let a team ship more experimental changes, because the budget exists precisely so some risk is affordable. An exhausted budget is the opposite signal, and it has to bind on everyone, including the team with the launch date they really wanted to hit. Teams that do this well write a short, explicit policy in advance — before the first freeze conversation happens in the heat of an actual incident, where it will otherwise turn into a repeat of the very Dev-versus-Ops standoff SLOs exist to end.

Notice what a freeze is not: it's not a change freeze in the classic Ops sense of "nothing deploys." It's a feature freeze — reliability fixes, the exact changes that will stop the bleeding, are not just allowed during a freeze, they're the only thing that's supposed to be shipping. This is also where feature flags and progressive delivery earn their keep: a flagged feature can be merged and even deployed dark during a freeze, decoupling "the code is in production" from "the code is exposed to users," so a freeze slows down exposure without also stalling the pipeline that later ships it the moment budget recovers.

Toil: the tax that eats the budget before a release ever ships

☺ Like you're 10: If a helper has to do the exact same annoying chore by hand every single time something happens, that chore is toil — and every minute of it is a minute not spent making it not happen again.

Toil, in the SRE sense that this course's own glossary already defines, is manual, repetitive, automatable operational work that scales linearly with service size and produces no lasting engineering value — restarting a hung process by hand every time it hangs, manually copying a config to a fourth environment, running the same "is it back yet" check every five minutes during an incident because nothing pages you automatically. It is explicitly not all operational work; responding to a genuine incident is operations, but the manual, boilerplate steps inside that response — the ones a script could do — are toil. Google's original SRE guidance capped toil at roughly half of an SRE's time specifically so the other half stayed free for engineering that drives toil down; DevOps teams borrow the same cap as a useful diagnostic even without formally adopting the SRE title split.

Toil connects to the error budget in a way that's easy to miss because it happens upstream of the number, not in it. Every manual step in an incident response — a human who has to notice the alert, SSH in, and run a known fix by hand instead of a runbook automatically executing it — adds minutes directly onto time-to-restore, and every one of those minutes is error budget being spent slower than it had to be. A team that automates its top three toil sources doesn't just get a nicer week; it measurably shortens its own incidents, which is the same thing as making its error budget stretch further per incident. This is also the most concrete answer to "what do we do with a feature freeze" beyond just waiting: the engineering time a freeze frees up should go first at paying down the toil that made the last incident slower to resolve than it needed to be, not just idling on standby watching the burn-rate graph. The full treatment of measuring, automating, and budgeting toil is on-call culture & sustainable operations' territory; this page only needs the one connection: unmanaged toil is a hidden multiplier on how fast every incident burns the budget you're trying to protect.

Running it day to day: the release-decision playbook

☺ Like you're 10: Put the number somewhere everyone can see it before standup, agree on the rule ahead of time, and let a machine check it instead of arguing every morning.

None of the above matters if the budget lives in someone's head instead of somewhere a DevOps engineer checks before clicking deploy. A few practices make the difference between an error budget that's a real operating constraint and one that's a slide from a quarterly review nobody remembers by March:

• Dashboard it, don't bury it. SLO burn belongs on the same screen as build status — tools like Prometheus with Grafana, Datadog's SLO tracking, or a dedicated SLO platform like Nobl9 or the open-source Sloth generator all turn an SLI query into a burn-down chart a whole team can glance at.
• Write the policy down before you need it. The four-tier table above (or your team's version of it) should be a document with named sign-off from both engineering and whoever owns the roadmap, agreed on when the budget is healthy — not improvised during the incident that exhausts it.
• Gate the pipeline on it, don't rely on memory. A cheap, effective pattern is a pre-deploy check in CI/CD that queries current budget and fails the job — for a feature-labeled change, not a reliability fix — when it's below the freeze threshold, so the policy enforces itself instead of depending on someone remembering to check a dashboard under deadline pressure.
• Pair it with progressive delivery. A canary or staged rollout lets you see a new release's effect on the SLI before it's spent much budget at all, instead of finding out from the monthly number after a full rollout already burned through a week's worth.
• Close the loop with a blameless postmortem. Any incident that meaningfully dents the budget deserves the same blameless review this course already teaches — the goal is the same toil-reducing fix, not a name attached to the burn.

# Illustrative CI gate — block feature deploys once budget is thin,
# but let anything labeled a reliability fix through regardless.
- name: Check error budget before deploy
  if: ${{ !contains(github.event.head_commit.message, '[reliability-fix]') }}
  run: |
    remaining=$(curl -s https://slo.internal/api/v1/services/checkout/budget \
      | jq '.percentRemaining')
    if (( $(echo "$remaining < 10" | bc -l) )); then
      echo "::error::Error budget at ${remaining}% — feature deploys are frozen, see release policy"
      exit 1
    fi

Anti-patterns worth killing before they cost you a quarter

☺ Like you're 10: A perfect-score target sounds responsible but actually breaks the whole idea — and a budget nobody ever spends is just as wasted as one nobody ever tracks.

A 100% SLO. This is the single most common mistake teams make when they're new to the idea, and it inverts the entire point. A 100% target has a 0% error budget, which means any failure — even one caused by something totally outside your control, like a cloud provider's network blip — technically breaches it. That doesn't make the service more reliable; it makes the SLO meaningless, because it can never be used to make a real ship-or-hold decision. 100% is very rarely the right target even for critical systems; 99.95% or 99.99% with a real, spendable budget is almost always more useful than an aspirational number that offers zero room to operate in.

Never spending the budget. The opposite failure is quieter and shows up in mature teams: a service that sits at 99.99% actual reliability against a 99.9% SLO, month after month, while product complains that nothing ships fast. That unused budget is a signal the team is over-investing in reliability relative to what users actually need — and every hour spent gold-plating a system that's already well inside its target is an hour not spent shipping something users would have noticed more. An error budget is permission to take risk, not just a leash; a team that never spends it is paying a stability tax nobody asked for.

A policy that isn't written down, or isn't enforced. Both were covered above, but they're worth repeating together: an unwritten policy gets renegotiated every time it's inconvenient, and a written-but-unenforced one is functionally the same thing with extra paperwork. Either way, you're back to the exact incentive standoff between Dev and Ops that the wall of confusion page opened this course with — just with a fancier number attached to the argument.