Run a Reliable Service — start here
This is the front door to the SRE course's capstone: six connected parts that take one real service description — checkout-api, the last synchronous step in a purchase funnel — from a plain-English description with no SLO, no monitoring, no on-call rotation, and no incident history, to a service that's been measured, watched, paged on, broken on purpose, and hardened, in that order. Every part reads an artifact the part before it produced by name — Part 3's runbook quotes Part 1's error-budget policy verbatim, Part 6 spends error budget Part 1 calculated — so nothing you build early goes stale later. Read this page once, then start with Part 1 and work forward. Short on time? The fast path below tells you exactly where you can stop and still walk away with something real.
Imagine a shop that's been ringing up sales for years without anyone ever agreeing on how "reliable" the checkout counter is supposed to be — no bell that rings when the line backs up, no card taped under the register telling the cashier what to do when the reader jams, and no record of what happened the one time the whole line stalled. This capstone gives that shop all of that, one piece at a time, against one real checkout: first the definition of "reliable" as an actual number (Part 1), then a bell that rings early enough to matter (Part 2), then the card under the register telling the cashier exactly what to do when it rings (Part 3), then a real jam, handled for real and written up afterward without blaming the cashier (Part 4), a dry run before the biggest rush of the year to see how many customers the counter can actually handle (Part 5), and finally, once it's all in place, someone deliberately jams the reader on a quiet Tuesday to prove the bell, the card, and the fix all actually work (Part 6).
checkout-api through all six parts, someone lays out the whole arc and hands each part to the specialist who owns it.The one service every part builds on
☺ Like you're 10: One catch-up paragraph if you haven't read Part 1 yet — what checkout-api is, what it depends on, and what stays fixed for the rest of the build.
checkout-api is the last synchronous step in a purchase funnel: an authenticated client sends POST /v1/checkout, the service validates the cart against an inventory service, calls an external payment processor to authorize and capture the charge, writes an order row to its own database, and returns a response. It runs behind a load balancer across two availability zones and serves roughly 5,000,000 valid checkout requests over a rolling 30-day window — quiet overnight, spiking well past that average during flash sales. There's no retry queue behind it and no background job absorbing a failure: a failed checkout isn't delayed, it's abandoned. Part 1 — Define the SLOs gives the full service description, a fact table, and a dependency diagram; this page only needs the summary, because every part after Part 1 refers back to the exact same one.
| Fact | Value |
|---|---|
| Service | checkout-api, owned by the Checkout Reliability team |
| Entry point | POST /v1/checkout, synchronous HTTP |
| Dependencies | an inventory service, an external payment processor, and an order database |
| Traffic | ~5,000,000 valid requests / rolling 30-day window, with flash-sale spikes above baseline |
| Starting point (before Part 1) | No SLO, no monitoring, no alerts, no on-call rotation, no incident history |
Part 1 needs a text editor and a calculator — no cluster, no account, no CLI. Later parts introduce exactly the tool each one needs, not everything up front: Part 2 stands up a metrics and alerting stack, Part 3 wires up a paging tool, Part 5 needs a load generator, and Part 6 needs a fault-injection tool. Work the parts in order — each one assumes the exact artifacts (the SLI table, the SLO numbers, the error-budget policy, the runbook) the part before it produced, by name, not a rough paraphrase of them.
How the six parts fit together
☺ Like you're 10: One chain, not six separate exercises — each part hands the next one something it can't do without.
Parts 1 and 2 are the fast path — they define what "reliable" means for checkout-api and build the alerting that watches it. Part 3 turns that alerting into something a human actually acts on. Part 4 is where the first three parts get tested against something real: a deliberately injected failure, handled live, budget spent against the exact number Part 1 calculated. Parts 5 and 6 close the loop — sizing the service for more traffic than it's ever seen, then proving the hardening holds under a fault injected on purpose rather than hoping it would.
Every part opens by naming exactly which artifact it's inheriting from the part before it, and exactly what new artifact it hands to the part after. If a part ever feels like it's starting from nothing, that's the signal to go back and finish the part before it — not a gap in the part you're reading.
The six parts, in order
☺ Like you're 10: Check one off as you finish it — your browser remembers, nobody else sees it.
Work them in order; each one names the exact artifact it's picking up from the part before. Every card below links to that part's concept lesson (read it first if you want the theory before the build) and to the part itself.
checkout-api's SLIs (availability, latency), set the SLOs (99.9% availability / 99.5% latency under 300ms, both over a rolling 30-day window), do the arithmetic for the error budget, and write the error-budget policy that says what happens once it's spent.checkout-api could follow it.checkout-api's three dependencies, and respond for real: acknowledge the page, declare a severity, name an Incident Commander, work the runbook. It deliberately spends real minutes off Part 1's 43.2-minute budget. Afterward, write a blameless postmortem with a full five-whys chain.checkout-api's next flash sale beyond the ~5,000,000-request monthly baseline, then load-test it to find its actual breaking point against Part 1's 300ms latency SLO — the request rate where p99 latency or error rate crosses the line.checkout-api's three dependencies (for example: "losing the payment processor for 30 seconds causes zero customer-visible checkout failures, because of the fix Part 4 shipped"), run it at the smallest blast radius first, and widen it only once the hypothesis holds — deliberately spending a bounded slice of whatever error budget remains, exactly as Part 1's policy permits.The fast path — if you only have time for Parts 1 and 2
☺ Like you're 10: No time for all six? Do the first two and you still walk away with something real — you just skip the incident, the capacity plan, and the chaos test for now.
If you only have an afternoon, work Part 1 — Define the SLOs and Part 2 — Build the Monitoring & Alerting and stop there. That gets you the SLO-and-monitoring core: a real SLI table, a defined SLO with an exact error budget in both minutes and requests, a written error-budget policy, and the alerting stack that would actually catch a budget burn early instead of near the end of the 30-day window. What you consciously skip is everything downstream of an alert firing — a tested runbook, a real incident response, a capacity plan, and a chaos-tested hardening pass. That's a genuine, honest gap, not a rounding error: an SLO nobody's on call for is a number on a page. But Parts 1 and 2 are also not wasted effort if you never come back — every later part reads Part 1's artifacts by name, so whenever you do return, Part 3 picks up exactly where you left off.
Even two full parts too much right now? The standalone drills work the same arithmetic and the same alerting judgment in isolation, without the full build: Drill — SLO & Error-Budget Calculation and Drill — Redesign a Noisy Alert.
Professor Owl: Six parts, one checkout-api. Before anyone touches Part 2 — does everyone actually have Sol's numbers?
Sol the Sloth: Ninety-nine point nine percent availability, ninety-nine point five latency, both over thirty days. Forty-three point two minutes of budget. Checked it twice.
Ellie the Elephant: Then I finally have something to alert on. Before Part 1, checkout-api didn't even have a target to burn.
Pip the Hummingbird: And once Ellie's alert fires, it's coming straight to whoever's on call — which means Part 3 needs a real runbook before Part 4 gets to break anything.
Timmy the Turtle: It quotes Sol's policy verbatim, not a rewrite. Two versions of "what happens when the budget hits zero" is worse than one clear version.
Foxy: Which leaves Part 4 to me. Once something actually breaks, I want to know why — all the way down, not just the first excuse.
Rocky the Raccoon: And Part 6 is mine. I get to spend whatever budget's left proving Part 4's fix actually holds — on purpose, before checkout-api finds out the hard way.
1. What's the one service every part of this capstone builds around, and what four things does it start with none of? 2. Give checkout-api's two SLOs and the error budget each one produces. 3. Which two parts make up the fast path, and what do you deliberately not have if you stop there? 4. Why does Part 3's runbook quote Part 1's error-budget policy verbatim instead of writing its own escalation rules from scratch?
Check your answers
checkout-api— it starts with no SLO, no monitoring or alerts, no on-call rotation, and no incident history.- Availability: 99.9% over a rolling 30-day window, producing a 43.2-minute (or 5,000-failed-request) error budget. Latency: 99.5% under 300ms over the same window, producing a 25,000-slow-request budget.
- Part 1 (Define the SLOs) and Part 2 (Build the Monitoring & Alerting). Stopping there means no tested on-call runbook, no real incident response, no capacity plan, and no chaos-tested hardening — the SLO and the alert exist, but nothing downstream of the alert firing has been built or proven yet.
- Because the capstone is built so each part inherits the exact artifact the part before it produced, by name, instead of re-deriving its own version. A runbook with its own separate escalation logic could drift from the actual policy over time and hand an on-call responder two conflicting sets of instructions during a real page.
Ready? Start with Part 1 — Define the SLOs, where Sol turns a plain-English description of checkout-api into a real SLI table, a real SLO, and an exact error budget. Come back to this page any time you need the map.