Drill — SLO & Error-Budget Calculation
This is not a lesson — it's one scenario, worked end to end. Wavelength's search-api carries a stated SLO: 99.9% of requests return a non-5xx status code, measured over a rolling 30-day window. That window closes today, and you've been handed the raw traffic logs — real fields, real noise, two real incidents buried inside a month of otherwise unremarkable Tuesdays. Your job: turn those logs into an actual SLI, decide whether search-api met its SLO, and compute exactly how many minutes of error budget it has left — or how far into the red it's gone. Work it by hand or with a calculator before you open the answer key at the bottom; the arithmetic is the entire point of this drill, not just recognizing the formula.
Imagine your allowance for "oops money" is $43.20 a month — money you're only allowed to spend on real accidents, like a scraped knee or a broken glass. Today is the last day of the month, and your parents hand you every single receipt from the last 30 days: real "oops" receipts mixed in with a stack of receipts that don't count at all, like the automatic $0 "just checking the wallet is still there" slips the family robot prints every ten seconds. Your job is to throw out the robot's slips, add up only the real "oops" receipts, and find out whether you're still under $43.20 — or already borrowing against next month's allowance. That's this entire drill, with search requests standing in for dollars.
The scenario: search-api's window closes today
☺ Like you're 10: A search service promised "almost always works," the 30-day report card is due today, and you're the one filling it in.
search-api is the backend behind the search bar and autocomplete in the Wavelength app — a smaller, quieter service than the streaming edge itself, but one enough of the app depends on that it earned a formal SLO of its own: 99.9% of requests return a non-5xx status code, measured over a rolling 30-day window. That's the exact SLI shape this course established earlier — good events over valid events, availability expressed as "did the request come back without a server error." Today is 2026-08-16, the window closes, and the 30 days under measurement run from 2026-07-18 through today. Nobody has calculated this month's number yet.
Three things are asked of you, in order: (1) compute the actual SLI for this 30-day window as a precise percentage, (2) state plainly whether search-api met its 99.9% SLO, and (3) compute the exact error budget remaining, expressed in minutes — which may turn out to be a negative number. Nothing here requires a cluster, a CLI, or a dashboard login; it requires reading the data below correctly and not skipping a step.
Reading the raw logs
☺ Like you're 10: Not every line in the log is a real visitor — some are a robot checking the door is still unlocked, and those don't count.
Production logs for search-api look like the lines below — one per request, with a timestamp, method and path, HTTP status, and response time in seconds. Two lines below are worth noticing before you go any further: one is a health-check ping from the load balancer's uptime prober, and one is a 404 from a search query that matched nothing.
203.0.113.44 - - [29/Jul/2026:14:07:52 +0000] "GET /search?q=inception HTTP/1.1" 200 512 rt=0.084
203.0.113.201 - - [29/Jul/2026:14:07:53 +0000] "GET /search?q=blade+runner HTTP/1.1" 500 0 rt=2.941
10.4.0.12 - - [29/Jul/2026:14:07:53 +0000] "GET /healthz HTTP/1.1" 200 2 rt=0.001
203.0.113.88 - - [29/Jul/2026:14:07:54 +0000] "GET /search?q=xyzzynotashow HTTP/1.1" 404 128 rt=0.031
10.4.0.19 - - [29/Jul/2026:14:07:53 +0000] "GET /healthz HTTP/1.1" 200 2 rt=0.001
203.0.113.44 - - [29/Jul/2026:14:07:55 +0000] "GET /search?q=inception HTTP/1.1" 500 0 rt=3.102Two facts about this log stream matter more than any single line in it. First: search-api sits behind two load-balancer nodes, each of which pings /healthz every 10 seconds as an uptime check — that's 2 nodes × 6 pings/minute = 12 health-check hits every minute, every minute of every day, whether a single real user shows up or not. Those pings always return 200, and they are not real traffic — they don't belong in the SLI at all, on either side of the ratio. Second: the 404 above is a real user request that happened to match nothing — under the "non-5xx" SLI definition this course already established, a 404 still counts as good. Only 5xx counts as bad. Mixing those two facts up in either direction — counting health checks as real traffic, or quietly demoting 4xx to a failure — produces a wrong answer that still looks plausible, which is the most dangerous kind.
The trap in this drill isn't hidden in a single tricky line — it's hidden in what the daily totals below already include. Every day's "requests logged" count folds in that constant stream of health-check pings, and if you build your SLI's denominator straight from that column without stripping them out first, you'll compute a real number — it'll just be the wrong one.
Thirty days of daily counts
☺ Like you're 10: Nobody hand-counts 5.9 million lines — someone already added up each day's total for you, robot pings and all.
The full 30-day log stream runs past 5.9 million lines, so it's already been rolled up into one row per day: total requests logged that day (including the constant health-check pings described above) and how many of those returned a 5xx status. Two days stand out — flagged below — and are explained right after the table.
| Day | Date | Requests logged | 5xx errors |
|---|---|---|---|
| 1 | 2026-07-18 | 191,480 | 21 |
| 2 | 2026-07-19 | 184,180 | 24 |
| 3 | 2026-07-20 | 189,880 | 27 |
| 4 | 2026-07-21 | 185,480 | 30 |
| 5 | 2026-07-22 | 190,980 | 18 |
| 6 | 2026-07-23 | 219,880 | 21 |
| 7 | 2026-07-24 | 222,280 | 24 |
| 8 | 2026-07-25 | 191,480 | 27 |
| 9 | 2026-07-26 | 184,180 | 30 |
| 10 | 2026-07-27 | 189,880 | 18 |
| 11 | 2026-07-28 | 185,480 | 21 |
| 12 | 2026-07-29 | 190,980 | 3,850 |
| 13 | 2026-07-30 | 219,880 | 27 |
| 14 | 2026-07-31 | 222,280 | 30 |
| 15 | 2026-08-01 | 191,480 | 18 |
| 16 | 2026-08-02 | 184,180 | 21 |
| 17 | 2026-08-03 | 189,880 | 24 |
| 18 | 2026-08-04 | 185,480 | 27 |
| 19 | 2026-08-05 | 190,980 | 30 |
| 20 | 2026-08-06 | 219,880 | 18 |
| 21 | 2026-08-07 | 222,280 | 21 |
| 22 | 2026-08-08 | 191,480 | 24 |
| 23 | 2026-08-09 | 184,180 | 27 |
| 24 | 2026-08-10 | 189,880 | 2,640 |
| 25 | 2026-08-11 | 185,480 | 18 |
| 26 | 2026-08-12 | 190,980 | 21 |
| 27 | 2026-08-13 | 219,880 | 24 |
| 28 | 2026-08-14 | 222,280 | 27 |
| 29 | 2026-08-15 | 191,480 | 30 |
| 30 | 2026-08-16 | 184,180 | 18 |
Day 12 (2026-07-29): a deploy shipped at 14:03 UTC with a bug in a new query-ranking path. An on-call engineer caught the rising 5xx rate during a routine dashboard check 38 minutes later — there was no automated alert on this yet — and completed a rollback by 14:50 UTC, 47 minutes start to finish. Day 24 (2026-08-10): a batch reindex job against the catalog-index dependency started at 09:12 UTC and starved search-api's calls to it for about 22 minutes, producing a burst of 502s and 504s before an engineer manually killed the job.
Step 1 — filter out what isn't a valid request
☺ Like you're 10: Throw out the robot's "just checking" slips before you add anything up.
Sum the "Requests logged" column across all 30 rows, then subtract the health-check volume you already know: 12 pings/minute × 1,440 minutes/day × 30 days.
Total requests logged, 30-day window = 5,912,300
− health-check pings (12/min × 1,440 min/day × 30 days) = 518,400
──────────────────────────────────────────────────────────────────────
= Valid requests (the SLI denominator) = 5,393,9005,393,900 is the number every later step is built on. Skip this subtraction and every downstream number — the SLI, the compliance verdict, the minutes — is computed against the wrong denominator, and it won't announce itself as wrong.
Step 2 — compute the actual SLI
☺ Like you're 10: Good requests divided by real requests, turned into a percentage — that's the whole formula.
Sum the "5xx errors" column across all 30 rows — that includes both incident days, not just the baseline noise — then apply the good-over-valid formula this course established on the SLIs, SLOs & error budgets page.
Total 5xx errors, 30-day window = 7,156
Good requests = valid − 5xx = 5,393,900 − 7,156 = 5,386,744
SLI = good ÷ valid = 5,386,744 ÷ 5,393,900 = 0.998673 = 99.8673%
99.8673% < 99.9% SLO → the window MISSED its objectiveIf you'd skipped Step 1 and used the raw 5,912,300 as the denominator instead, you would have computed (5,912,300 − 7,156) ÷ 5,912,300 = 99.8790% — close enough to feel right, and different enough from 99.8673% to be a materially different number in a reliability report. Here it doesn't flip the verdict; on a service with a thinner margin, an uncorrected denominator like this is exactly the kind of error that turns a real breach into a report that reads "essentially compliant." Get the denominator right before you touch the percentage — this is the same warning the SLIs, SLOs & error budgets page gives about excluding health checks and bots from an SLI's valid-request count.
Step 3 — turn the shortfall into an error budget, in minutes
☺ Like you're 10: Turn "how many broken requests" into "how many minutes," using how fast requests normally arrive.
An error budget expressed as a count of allowed bad requests is precise but not very readable in an incident review or a release-freeze announcement — "minutes" is. To convert, first find the window's average request rate, then divide both the allowed and the actual bad-request counts by that same rate so they land in the same unit.
Average request rate = valid requests ÷ window minutes
= 5,393,900 ÷ 43,200
= 124.86 requests/minute
Allowed bad requests (0.1% of valid) = 0.001 × 5,393,900 = 5,393.9
Allowed, in minutes = 5,393.9 ÷ 124.86 = 43.20 min
Actual bad requests (5xx) = 7,156
Actual, in minutes = 7,156 ÷ 124.86 = 57.31 min
Remaining error budget = allowed − actual
= 43.20 min − 57.31 min
= −14.11 min → 14.11 minutes OVER budgetsearch-api didn't just miss its SLO — it overspent its entire 30-day error budget by roughly 14.11 minutes' worth of failed requests, concentrated almost entirely in two incidents 12 days apart rather than spread evenly across the month.
Notice that the "allowed, in minutes" figure above — 43.20 minutes — landed on exactly the same number the flat calendar formula from SLIs, SLOs & error budgets gives for a 99.9% SLO over 30 days: (100% − 99.9%) × 43,200 min = 43.2 min. That's not a coincidence — it's algebra. Dividing the allowed request count by the average request rate cancels the traffic volume out entirely, because both numerator and denominator were built from the same total. A request-based error budget converted through the window's own average rate always collapses to the same minutes figure as the flat time-based formula, whatever the service's actual traffic looks like. What traffic volume does change is how many real, individual failed requests that budget represents.
What happens next
☺ Like you're 10: When you've spent past your allowance, the answer isn't "try harder" — it's "stop taking chances until it refills."
Under the error-budget policy this course already established, an exhausted — here, overspent — budget has one correct response: freeze new risky releases for search-api, and shift the team's priority toward reliability work until the budget recovers as the window rolls forward. That recovery isn't automatic tomorrow — Day 12's 3,850-error spike stays inside the trailing 30-day window for another 12 days, and only ages out once the window's start date moves past 2026-07-29. Until then, every day's recalculation still carries that spike's weight. See SLO windows & composite SLOs for how the choice of window shape changes how quickly a single bad day like this one stops dominating the number.
The reliability work itself is concrete, not vague: root-cause both incidents through a proper blameless postmortem (the companion drill, Write the Postmortem, walks that process on a similar scenario), add a canary gate to the ranking-path deploy pipeline so a bug like Day 12's is caught on a slice of traffic instead of 100% of it (see release engineering & progressive delivery), and put a timeout and circuit breaker on the catalog-index call so a slow dependency degrades gracefully instead of piling up 502s.
Both incidents were only caught by a human noticing a dashboard, 38 minutes and an unspecified delay after the fact — this window's budget was already unrecoverable well before anyone calculated the monthly number. A properly tuned fast-burn alert would have paged on Day 12 within minutes of the spike starting, while there was still budget left to protect. See the companion drill Redesign a Noisy Alert and the deep dive on multi-window, multi-burn-rate alerting for how that alert should actually be built.
Try it yourself: tighten the SLO
☺ Like you're 10: Same broken month, stricter promise — see how much worse the answer gets.
Recompute Steps 2 and 3 assuming search-api's SLO were 99.95% instead of 99.9% — same 30-day window, same logs, same 7,156 total 5xx errors. How many minutes over or under budget does that stricter target come out to? Work it before checking the answer key below — the method is identical, only the allowed-percentage input changes.
Foxy: 99.8673% sounds close enough to 99.9% that I want to call it fine. It's a rounding error.
Sol the Sloth: ...It's not a rounding error, it's fourteen and a bit minutes of budget that doesn't exist. I converted it through the average request rate twice to be sure.
Timmy the Turtle: Then the migration we wanted to ship this week doesn't go out. We don't spend budget we've already overspent.
Foxy: Both incidents together are less than an hour of the whole month. Feels harsh to freeze over that.
Sol the Sloth: The policy doesn't grade on "it was only two bad days." It grades on the number. The number says over budget.
Professor Owl: And the number only means anything because Sol got the denominator right before touching the percentage. That's the actual discipline here — not the freeze, the arithmetic that justified it.
1. What two totals go into the SLI's denominator and numerator once the raw logs are correctly filtered, and what exactly gets subtracted out in Step 1 — and why doesn't a 404 get subtracted the same way? 2. What is search-api's actual SLI for this window, to four decimal places, and does it meet its 99.9% SLO? 3. What is the exact remaining error budget in minutes, and what does a negative value mean operationally? 4. At a 99.95% SLO instead of 99.9%, how many minutes over or under budget does the same month come out to?
Check your answers
- The denominator (valid requests) is total requests logged minus the 518,400 health-check pings, giving 5,393,900. The numerator (good requests) is that same 5,393,900 minus the 7,156 requests that returned a 5xx status, giving 5,386,744. Health-check pings are subtracted because they aren't real traffic at all — they don't belong in the ratio on either side. A 404 is not subtracted, because the SLI is defined as "non-5xx" — a 404 is a real, valid request that still counts as good under that definition.
- SLI = 5,386,744 ÷ 5,393,900 = 99.8673%. Since 99.8673% is below the 99.9% SLO,
search-apidid not meet its objective for this window. - Allowed budget in minutes = 43.20 (0.1% of the 43,200-minute window, via the average request rate of 124.86 req/min). Actual failed-request time = 57.31 minutes. Remaining = 43.20 − 57.31 = −14.11 minutes — the service is 14.11 minutes past zero, not merely close to it. Operationally, a negative remaining budget means the service is already in a release freeze under the standard error-budget policy: no new risky changes until the window rolls forward and the deficit clears.
- At 99.95%, allowed bad requests = 0.0005 × 5,393,900 = 2,696.95, equivalent to 21.60 minutes (again matching the flat formula: (100% − 99.95%) × 43,200 = 21.6 min). Actual is unchanged at 57.31 minutes. Remaining = 21.60 − 57.31 = −35.71 minutes — a stricter SLO doesn't change what happened in production, only how much further underwater the same incidents leave the budget.