Best Practices & the SRE Operating Model
Every mechanic on this page already has its own lesson elsewhere in this course: production readiness reviews score a service against six categories, an error budget is 100% minus your SLO, on-call pages the right person when it burns, and a postmortem investigates the system instead of the person. This page is about something narrower and, in practice, harder: whether an organization actually runs those four mechanics as a living operating model — enforced under real launch-date pressure, real fatigue, and a real quarter where the numbers are inconvenient — or lets each one quietly decay into paperwork nobody reads. One discipline turns out to hold all four gates open at once: every single one of them only works if it has a named owner, a real due date, and someone whose job is to check that the date was actually met.
A hospital doesn't get to call itself a Level I trauma center just because the sign says so — an outside board actually walks the building and checks specific things: is a trauma surgeon in the building 24/7, is the blood bank stocked, can the OR be ready in minutes. That's gate one. The hospital also has a written rule for what happens when it's overwhelmed — divert ambulances past a specific patient count, agreed in advance, not improvised in the moment. That's gate two. Doctors work shifts short enough that a tired surgeon isn't the one holding the scalpel at hour twenty, and they get paid for carrying the pager even on a quiet night. That's gate three. And after a bad case, the whole team reviews what went wrong in a morbidity and mortality conference — but the review only means something if its recommended fixes actually get built into the next protocol, with someone's name and a deadline on each one, not just discussed once and forgotten. That's gate four. A hospital that skips any of these isn't secretly fine because nothing's gone wrong yet — it's one bad week away from finding out the hard way. Running an SRE program is the same four-gate discipline, aimed at software instead of patients.
One discipline, four gates
☺ Like you're 10: A team with the SRE title but no real gates is just ops with a new name badge — the gates are what make the title true.
What is SRE? already named the trap: an organization can rename its ops team "SRE" without adopting a single mechanism, and end up running exactly the process it had before under a new badge. This page is where that trap gets concrete. The four gates below — a production readiness review with real veto power, a written and enforced error-budget policy, an on-call rotation designed to be survived rather than merely staffed, and a postmortem process whose action items actually close — are not four independent best practices to adopt piecemeal. They're one operating model, because they share a single point of failure: every one of them is, underneath, a promise written down in advance that gets tested precisely when honoring it is inconvenient. A PRR is a promise to say no to a launch date. An error-budget policy is a promise to freeze a roadmap. An on-call design is a promise to pay for interrupted sleep and cap how much of it one person absorbs. A postmortem action item is a promise that a fix found during a bad week actually gets built. All four decay the same way, too: the promise stays true right up until the first moment it's expensive to keep, and if nothing enforces it at that exact moment, it quietly becomes theater — a rubric nobody scores honestly, a policy nobody actually invokes, a rotation nobody protects, a ticket nobody closes. See SRE anti-patterns & pitfalls for the mirror image of this page — the specific ways each of these four gates is observed failing in the wild.
Gate one — a production readiness review with a real veto
☺ Like you're 10: The review only means something if it's allowed to say "not yet" and make it stick.
Production readiness reviews covers the six-category checklist and the pass / conditional-pass / fail mechanics in depth; the operating-model question here is different: does your PRR actually have the authority its name implies? A PRR that can only recommend — one that gets overruled the moment a VP wants a launch date held — isn't a gate, it's a suggestion with a form attached, and everyone on the review board learns that within one cycle. A PRR with real teeth has three properties that have nothing to do with the checklist itself. First, the reviewer isn't the on-call engineer scheduled to inherit the pager that week, reviewing under time pressure with an obvious conflict of interest — it's a standing board, usually senior SREs rotating through review duty, with enough seniority and organizational distance from the requesting team to say no and make it stick. Second, the outcome is binding: a fail means the owning team launches self-supported and holds its own pager until it re-applies, not "launches anyway, SRE will figure it out." Third, a PRR is not a one-time stamp. Services get re-reviewed on a fixed cadence for anything tier-1 — commonly annually — and immediately after a material change: a new critical dependency, a rewrite of the data layer, a tenfold jump in traffic, or a transfer of ownership to a different team. A service PRR'd once in 2022 against an architecture that no longer exists has a PRR in name only.
Real organizations also inherit services that were already on the pager before any PRR process existed, and pretending otherwise doesn't make the gap go away. The honest move is a grandfather clause with teeth, not a blanket exemption: run the review anyway, score it honestly, and where it fails, open a remediation plan with the exact same discipline that makes a postmortem action item real — a named owner, a due date, and a spot on someone's dashboard, not a permanent asterisk next to the service's name. A service that's been "grandfathered pending remediation" for three years didn't get grandfathered; it got exempted, and the PRR process just quietly stopped applying to it.
## PRR sign-off — payments-checkout-api
Score: 5/6 categories PASS, 1 CONDITIONAL (Capacity)
Decision: CONDITIONAL PASS — provisional on-call support
Blocking item:
- Load test at 3x current peak traffic not yet run
Owner: @maria.k (owning team lead)
Due: 2026-09-05
Tracked: JIRA CAP-4471
Reviewers: @j.chen (SRE board, lead) · @d.osei (SRE board)
Re-review trigger: item CAP-4471 closes, OR 2027-08-17 (12-month cadence)The single most common way a PRR program dies is a launch date colliding with a real finding. The service fails capacity review two weeks before a marketing-committed date, and the fastest path to "yes" is quietly reclassifying a blocking finding as a non-blocking one, or waving it through on a verbal "we'll fix it after launch." The date that gets kept is never the remediation date — it's the launch date. If your PRR pass rate is suspiciously close to 100%, that's not evidence the checklist is being met; it's evidence the checklist has stopped being able to fail anything.
Gate two — an error-budget policy that's written down before you need it
☺ Like you're 10: Agree on the rule while everyone's calm, not in the middle of the argument it's supposed to settle.
SLIs, SLOs & error budgets defines the arithmetic; Google & the error-budget policy covers what enforcement actually looked like at the company that invented the mechanism. The operating-model requirement underneath both is simpler to state than to do: the policy has to exist in writing, agreed by both SRE leadership and the product or engineering leadership who'll feel it, before the first time a budget actually hits zero — not drafted afterward to justify whatever decision already got made. A policy negotiated in the room where a launch is on the line isn't a policy; it's a post-hoc rationalization with a document template. A real written error-budget policy names, specifically: which SLO and measurement window it governs; exactly what a freeze restricts (typically no new feature launches, no risky migrations, no non-essential config changes) and what stays exempt (security patches, fixes for the very incident that burned the budget, previously-approved compliance deadlines); who has the standing authority to grant an exception and under what bar (commonly a joint sign-off from an SRE lead and the product lead, never a unilateral call by whoever's on call that day); and the exact condition that lifts the freeze — the budget crossing back above the threshold, measured the same way it was measured going in, not a vibe that things feel better now.
# Error-Budget Policy — checkout-api
# Agreed by: SRE leadership + Checkout eng leadership · reviewed quarterly
1. Governs: checkout-api availability SLO, 99.9% over a rolling 30-day
window (43.2 min allowed downtime / 30 days).
2. Trigger: budget consumption crosses 100% of the 30-day window at
any point — checked continuously, not just at month-end.
3. While frozen:
- No new feature launches to checkout-api.
- No non-essential config or infra changes.
- Exempt: fixes for the incident that burned the budget; security
patches; previously-approved compliance deadlines.
4. Exception process: a launch can proceed during a freeze only with
joint written sign-off from the SRE lead AND the Checkout eng lead,
logged in #checkout-error-budget with the specific risk accepted.
No unilateral exceptions from whoever is on call that day.
5. Unfreeze: automatic once 30-day burn drops back under 100%,
measured by the same query used to declare the freeze.
6. Audit: every freeze this policy actually triggered, and whether it
was honored without an exception, is reported quarterly alongside
SLO attainment.Test the policy while the budget is healthy. Run a tabletop exercise — see the incident response tabletop drill in this course's labs — where the budget is hypothetically at zero, and walk the exception process for real with the actual people who'd be in that room. A policy nobody has ever rehearsed enforcing gets improvised the first time it matters, and improvisation under launch pressure reliably favors the launch.
Gate three — an on-call rotation designed to be survived
☺ Like you're 10: A rotation that quietly burns people out isn't cheaper than one that pays for sleep — it's just billing the cost later, as attrition.
Incident management & on-call covers what happens once a page fires; this is about the design of the rotation itself, which decides how much of a person's life the pager is allowed to consume before the organization notices. The default most teams land on is a one-week primary rotation with a one-week secondary the week before or after — long enough to be predictable and plan around, short enough that no single person absorbs more than a handful of consecutive bad nights. Some teams push the primary shift shorter, to eight or twelve hours, specifically for services with high page volume or severe consequences, on the reasoning that bounding continuous sleep disruption to a single overnight window matters more than administrative convenience. Whatever the length, treat pages-per-shift as a first-class health metric, not an afterthought: a rotation that consistently produces more than one or two pages a shift, especially overnight, isn't an on-call design problem to route around — it's an alert-design problem, a toil problem, or a reliability problem the rotation is quietly absorbing instead of surfacing, and the fix belongs upstream of the schedule, not in another warm body added to it.
Compensation is not a nicety — an unpaid on-call rotation is one of the most reliable predictors of SRE attrition in the industry, and it shows up as resignations, not as a line item anyone tracked. Two components are worth separating: a stipend for carrying the pager at all, paid regardless of whether it rings, because availability itself has a cost even on a quiet week; and separate compensation — overtime pay or equivalent comp time — for actual incident work, especially anything after hours or overnight. Exactly what's required varies by jurisdiction — several countries and a handful of U.S. states have specific statutes governing on-call pay, so check your own jurisdiction's requirements rather than assuming a policy that's legal for one team's location covers a distributed one — but the baseline principle holds everywhere the law is silent: if the pager can wake someone up, the organization should pay for the privilege of owning that risk, not treat it as an unstated part of the job description.
Follow-the-sun — handing the pager between teams in different time zones so nobody's rotation crosses their own overnight — is the single best fix for on-call fatigue where a team can actually run it honestly, and the single most common thing smaller teams fake. It needs real headcount to work: commonly cited guidance is at least two engineers per region across at least three regions — six to eight people, minimum — so each region's own on-call still has a secondary and nobody's "day" rotation quietly becomes a solo overnight because their region-mate is on leave. A four-person team spread across two time zones calling itself follow-the-sun hasn't eliminated night pages; it's just relabeled the same fatigue and thrown away the one thing an honest single-region rotation has going for it — everyone knowing exactly when their bad week is coming. If your team is too small for real follow-the-sun, the better move is a plain single-region weekly rotation with a strict secondary and an automatic escalation timeout, plus a hard conversation about trimming how many services sit on that pager — see SRE team topologies for how team shape and service count interact — rather than a rotation that pretends to be something it isn't.
| Model | Minimum team size | Best for | Failure mode if forced too small |
|---|---|---|---|
| Weekly primary + secondary, single region | 4–6 engineers | Most services; predictable, easy to plan a life around | Same 1–2 people carry most weeks if the pool is smaller than it looks |
| Shortened 8–12h "tour" shifts, single region | 6–8 engineers | High page-volume or high-severity tier-1 services | More handoffs mean more chances for context to drop between shifts |
| Follow-the-sun, 3 regions | 6–8+ engineers, ≥2 per region | Global services where overnight pages are otherwise routine | Faked with too few people: night pages don't disappear, they just get relabeled |
A rotation with a primary and no automatic escalation timeout has a single point of failure baked into the design itself: if the primary sleeps through a page, misses a notification, or has their phone die, nothing happens next until a human notices. Every rotation needs a secondary and an escalation policy that pages the secondary automatically after a fixed window — commonly five to fifteen minutes — with no step in that chain that depends on a person remembering to check. Tools like PagerDuty, Opsgenie, and Grafana OnCall encode exactly this escalation chain as configuration — a primary layer, a secondary layer, and an automatic timeout between them — rather than leaving it to a human to remember at 3 a.m.
Gate four — postmortem action items that actually close
☺ Like you're 10: The review isn't the finish line — the fix landing in production is.
Postmortems & blameless culture already covers why an action item belongs in the issue tracker as a normal ticket with a named owner and a due date, not as a paragraph of prose promising to "look into it." The operating-model gap sits one level up: whose job is it to notice when that ticket goes stale? Writing a well-formed action item and then never looking at it again produces exactly the same outcome as never writing it — a postmortem archive full of diagnosed, unfixed root causes, each one waiting to cause the same incident again under someone else's watch. A functioning program answers that ownership question explicitly, usually by giving one role — an SRE lead, a program manager, whoever runs the standing reliability review — the specific job of chairing a recurring, dedicated meeting (commonly monthly) whose entire agenda is open postmortem action items: what's overdue, what's blocked, what needs an escalation.
That review needs a real escalation path for items that blow past their due date, not an infinite string of quiet re-dates. A common pattern: an item more than 30 days overdue gets escalated automatically to the owning team's engineering manager, and one more cycle overdue after that reaches the same leadership review that looks at incident counts and SLO attainment. And the completion rate itself — what fraction of last quarter's action items actually closed on time — belongs as a published number in the same report as your reliability metrics, covered in measuring & reporting reliability, not as an internal detail only the SRE team sees. A team's postmortem culture is exactly as healthy as its least comfortable metric, and "percentage of action items that actually got done" is usually the most uncomfortable number in the whole report — which is precisely why it belongs in front of leadership, not filed away.
| Postmortem | Action item | Owner | Due | Status |
|---|---|---|---|---|
| 2026-06-02 checkout outage | Add load test for connection-pool reuse under concurrency to CI | @d.osei | 2026-07-01 | Closed on time |
| 2026-06-02 checkout outage | Lower alert threshold from static 90% to burn-rate based | @j.chen | 2026-07-15 | 18 days overdue — escalated to EM |
| 2026-07-11 search latency incident | "Investigate caching layer" (no ticket, no owner) | — | — | Not a real action item — reopen as a ticket |
The four-gate operating-model scorecard
☺ Like you're 10: Score yourself honestly on all four, and the lowest score tells you exactly what to fix first.
These four gates are the whole page condensed into one table. Score your own organization 0 (not really), 1 (partly, inconsistently), or 2 (solidly, every time) on each row — out of 8 total.
| Gate | What decays without enforcement | How to run it for real — score 0–2 |
|---|---|---|
| Production readiness review | A checklist that can't say no is a suggestion with a form attached. | Standing review board, binding fail outcome, re-reviewed on a fixed cadence and after material change. |
| Error-budget policy | A policy drafted after the fact rationalizes whatever was already decided. | Written and agreed before the budget hits zero; named freeze scope, named exception authority, audited quarterly. |
| On-call rotation design | An unpaid, unbounded rotation bills its cost later, as attrition. | Sane shift length, real compensation, honest follow-the-sun only with the headcount to back it, secondary + auto-escalation. |
| Postmortem follow-through | An action item nobody revisits is a diagnosed root cause left to recur. | Owned recurring review of open items, real escalation for overdue ones, completion rate published alongside SLO attainment. |
Score your real organization against the four rows above, 0–2 each, out of 8. Then do two things. First, for whichever row scored a 2, name the evidence — an actual PRR that failed a service, an actual freeze that actually happened, an actual overdue action item that got escalated — not just the fact that a document describing the practice exists somewhere. A policy that's never been tested is a 1, not a 2, no matter how well it reads. Second, for whichever row scored lowest, that's not a coincidence — it's very likely the exact gate that's about to get tested by your next real incident, because the gates that decay first are the ones nobody's checked lately.
Benny the Beaver: Marketing already announced the launch date. Can we just conditional-pass the capacity finding and load-test after we ship?
Timmy the Turtle: Conditional pass needs a real due date on a real ticket, not "after we ship." Give me an owner and a date, or it's a fail.
Foxy: And while we're at it — is the error-budget freeze policy actually written down anywhere, or are we all just agreeing to remember it?
Sol the Sloth: ...Not written. Which means the first time the budget hits zero, we'll be negotiating the rule and the exception in the same meeting.
Pip the Hummingbird: Speaking of exhausted — this rotation paged four times last night. That's not bad luck, that's a rotation absorbing an alerting problem.
Professor Owl: Then it's the same fix in all three cases. Name an owner, put a date on it, and check back — that's the whole operating model, applied wherever it's inconvenient today.
That's the operating model in full: a PRR that can actually say no, an error-budget policy written down before anyone needs it, an on-call rotation designed to be survived rather than merely staffed, and postmortem action items that are tracked to a real close. None of the four is exotic — every mechanic here is documented, in depth, elsewhere in this course. What separates a program that runs this well from one that's SRE in name only is whether each gate still holds on the day it's inconvenient to enforce it, and whether someone's job is specifically to check.
1. Name the three properties that give a production readiness review real veto power, beyond the checklist itself. 2. What has to be true about an error-budget policy for it to actually function, and what happens if it's written after a budget's already been burned? 3. What headcount does honest follow-the-sun on-call actually require, and what happens when a smaller team fakes it anyway? 4. What's the single mechanism that keeps all four gates on this page from decaying into paperwork?
Check your answers
- A standing review board without a conflict of interest (not the on-call engineer inheriting the pager that week); a binding outcome (a fail means the owning team launches self-supported until it re-applies); and a fixed re-review cadence plus re-review after any material change, so a PRR isn't a permanent, one-time stamp.
- It has to be written down and agreed by both SRE and product/engineering leadership before the first time a budget actually hits zero — naming the governed SLO/window, exactly what's frozen and what's exempt, who can grant an exception, and the exact condition that lifts the freeze. Written after the fact, it isn't a policy — it's a rationalization for whatever decision already got made.
- Commonly cited guidance is at least two engineers per region across at least three regions — six to eight people, minimum. A smaller team that fakes follow-the-sun doesn't eliminate night pages; it relabels the same fatigue and loses the predictability an honest single-region rotation would have given it.
- A named owner, a real due date, and someone whose job is to check that the date was actually met — the same discipline that makes a postmortem action item real is what keeps a PRR remediation plan, an error-budget exception, and an on-call escalation from becoming theater.