Interview Prep

An SRE interview is not a trivia contest about Prometheus syntax. It is a sustained check on whether you reason about reliability as a number that gets chosen, measured, and spent — rather than as a feeling that things are going well. This page is a question bank and a technique guide built directly from this course's twelve domains: fifty-five technical questions with model answers, a set of open-ended scenario prompts with a method for attacking them, behavioral questions shaped so you can slot your own experience into them, the questions worth asking the interviewer back, and an honest plan for the last week before the call.

What the role actually gets asked

☺ Like you're 10: Most SRE hiring processes are four different tests wearing one name — and each one is checking something different, so answering all four the same way costs you.

SRE hiring processes vary a lot between organizations — a four-person startup and a company with a dedicated SRE org will not run the same loop, and some places fold SRE hiring into a general infrastructure track entirely. But across most processes you will meet some combination of four distinct stages, and knowing which one you are in is half of answering well.

The topic map: what gets probed, and how hard

☺ Like you're 10: Twelve subjects, but they don't all show up equally often — here's which ones to know cold and where each one lives in this course.

Below are this course's twelve domains with a rough sense of how heavily each tends to feature in an SRE interview. To repeat the caveat above: this emphasis is this course's judgment, derived from how load-bearing each concept is to the discipline and how often it appears as a prerequisite for other answers — not a measured distribution from any real hiring process. Treat it as a study-order hint, not a prediction.

🟢 Warm-up — definitional fluency, answerable in 30–60 seconds.🟡 Medium — a structured answer plus a concrete example.🔴 Hard — trade-off reasoning, arithmetic, or a scenario with no single right answer.

Domain 1 · What is SRE?

☺ Like you're 10: These four are the "do you actually know what this job is" questions, and they nearly always come first.

Four questions, and the trap in all of them is the same: reciting a definition instead of naming a mechanism. Anyone can say "SRE is about reliability." What separates a strong answer is being able to point at the specific, falsifiable machinery — an SLO, an error budget, a toil ceiling — that makes the claim checkable. Full treatment in What is SRE?.

🟢 1.1 · What SRE actually is

They ask: "How would you describe what Site Reliability Engineering is, to someone who's only ever worked in a traditional ops role?"

Start with the origin because it makes the definition concrete rather than abstract. SRE was created at Google around 2003, when Ben Treynor Sloss was asked to build a small team responsible for keeping google.com running and chose to staff it with software engineers rather than conventional operators. His own summary of the result is still the tightest definition available: SRE is what happens when you ask a software engineer to design an operations function.

Then say what that inversion actually changes. A traditional ops team is measured on keeping a system up by intervening; an SRE team is measured on keeping a system up by writing software that makes intervention unnecessary. Recurring manual work stops being "the job" and becomes a bug to fix in code — reviewed, tested, and version-controlled like any other production software. That single shift changes who gets hired, what gets rewarded, and what the team spends a Tuesday afternoon doing.

Close with the mechanisms, because that's what makes the claim auditable rather than aspirational: SLIs and SLOs, an error budget derived from the SLO, an error-budget policy that halts feature launches when the budget is spent, blameless postmortems with tracked action items, and an explicit ceiling on how much operational work an engineer absorbs before it counts as a staffing failure rather than a personal one.

🟡 1.2 · SRE versus DevOps

They ask: "What's the difference between SRE and DevOps? Aren't they the same thing with different job titles?"

They are related, not interchangeable, and the cleanest framing is the one that circulates among practitioners: class SRE implements interface DevOps. DevOps is a set of cultural principles — commonly summarized as CALMS: Culture, Automation, Lean, Measurement, Sharing — describing how development and operations should relate. It defines the contract. It deliberately does not mandate a mechanism for getting there.

SRE is one opinionated implementation of that contract with the defaults filled in, and the defaults are concrete and measurable: service level indicators and objectives, an error budget derived from them, a formal policy that pauses feature launches when the budget is exhausted, blameless postmortems whose action items are tracked to completion, and a cap on operational load. You can practice DevOps without any of those. You cannot practice SRE without them and still call it SRE honestly.

It's worth adding that this isn't a hierarchy — SRE isn't "advanced DevOps." They are a philosophy and one concrete implementation of it, and plenty of high-functioning teams implement DevOps principles through mechanisms that aren't Google's.

🟡 1.3 · The 50% rule

They ask: "What's the 50% rule, and what should a team actually do when it's consistently over it?"

Google's SRE model includes a guideline with no real DevOps equivalent: an SRE team should spend at most half its time on operational work — on-call, tickets, and the manual operational tasks that collectively count as toil — and at least half on engineering: writing automation, improving reliability, building tooling.

The critical framing is that the number is a ceiling, not a target. Sustained work meaningfully above 50% isn't a sign the team needs to try harder; it's a signal the team is understaffed, under-automated, or has accepted a service that shouldn't have passed its production readiness review. The standard response is structural: push the work back to the product-development team that owns the service, freeze new toil-generating launches, or add headcount. Working longer hours is explicitly not on that list, because it hides the signal the cap exists to produce.

🔴 1.4 · Spotting SRE that isn't SRE

They ask: "You join a company whose ops team was renamed 'SRE' last year. How would you tell whether they actually practice SRE, and what would you change first?"

Renaming an ops team without adopting the mechanisms is the most common failure mode in this discipline, and it's diagnosable with a small number of questions. Ask: are there SLOs, and were they derived from user impact or copied from a template? Is there an error-budget policy in writing, agreed before it was needed? Has that policy ever actually blocked a launch? Are postmortems written for every incident above a defined bar, and do their action items get tracked to completion in the same tracker as feature work? Is there a toil number, and has it ever triggered pushback rather than absorption?

The sharpest single test, and the one worth saying out loud: has any of this machinery ever made someone do something they didn't want to do? An SLO that has never prompted a conversation about slowing down, a policy that has never blocked a launch, and a postmortem process that has never survived an uncomfortable repeat name are all decoration. A mechanism that never bites isn't a mechanism.

On what to change first: resist the urge to start with tooling. The highest-leverage first move is usually a single well-chosen SLO on one user-facing critical path, derived from actual support-ticket evidence rather than from what the service already achieves, with an error-budget policy attached that names exactly what a freeze restricts and who can grant an exception. One SLO that genuinely constrains behavior is worth more than twelve that don't. This is covered in depth in SRE anti-patterns and the SRE operating model.

Domain 2 · SLIs, SLOs & error budgets

☺ Like you're 10: This is the domain interviewers dig into hardest, and it's the one where they'll actually make you do arithmetic out loud.

Seven questions, because this is the load-bearing domain — weak answers here quietly weaken your monitoring, release and capacity answers too, since all three reference it. Expect at least one question where you're asked to compute something live. Practice saying the arithmetic out loud slowly rather than fast and wrong; Sol's whole methodology on the SLO lesson is that the number has to come out exactly right.

🟢 2.1 · SLI, SLO, SLA

They ask: "Walk me through SLI, SLO and SLA. Why is the SLA usually looser than the SLO?"

An SLI is the measurement: a quantitative indicator expressed as a ratio of good events over valid events across a window — the percentage of requests served under 300ms, the percentage returning a non-5xx status. Note the shape: it's a proportion, not a raw average. "Average latency was 180ms" is not an SLI, because averages hide the tail and give you nothing to grade pass/fail against.

An SLO is the internal target set for that SLI over a defined window — 99.9% of requests under 300ms over a rolling 28 days. It's the number your team is accountable to, set collaboratively with the people who depend on the service, and revisited deliberately rather than treated as physics.

An SLA is a business contract with a customer that attaches consequences — usually service credits — to a stated commitment. Miss it and you owe someone something.

The SLA is set looser than the SLO on purpose, and the reason is an early-warning window. A common shape is a 99.9% internal SLO against a 99.5% external SLA; the slack between them absorbs normal operational noise without triggering a customer-facing penalty. More importantly, if you're consistently missing the SLO but still meeting the SLA, you have a real reliability problem that hasn't yet become a contractual one — which is exactly the moment to fix it. Set the two identically and you've deleted that warning window: every SLO miss becomes a contract breach simultaneously.

🟢 2.2 · What makes a good SLI

They ask: "Give me an example of a bad SLI and tell me what makes it bad."

The canonical bad SLI is any resource-based metric: CPU utilization, JVM garbage-collection pause time, queue depth. These are real signals worth watching operationally, but they are not SLIs, because a user does not experience your CPU. They experience whether their request succeeded and how long it took.

A good SLI is request-based and measured from the outside in, at the boundary the user actually crosses — an API gateway, a load balancer, or better, the client itself. It maps to a user-visible property: availability (did it succeed), latency (was it fast enough), correctness (was the answer right), or freshness (was the data current). And there should be few of them — roughly two to five per service. A service with twenty SLIs doesn't have twenty things users care about; it has an SLI set nobody will consult during an incident.

Worth adding: getting the denominator right matters as much as the threshold. An SLI that silently drops the requests that are timing out will report suspiciously good numbers precisely when things are worst.

🟡 2.3 · Error-budget arithmetic, out loud

They ask: "A service has a 99.95% availability SLO over a 30-day window. How much downtime does that allow? And what if we tightened it to 99.99%?"

Do this slowly and show the working — the interviewer wants to see the method, not just the number.

30-day window   = 30 × 24 × 60 = 43,200 minutes

99.9%  SLO → 0.1%  of 43,200 = 43.2 minutes allowed
99.95% SLO → 0.05% of 43,200 = 21.6 minutes allowed
99.99% SLO → 0.01% of 43,200 =  4.32 minutes allowed

Each added "nine" divides allowed downtime by roughly 10×.

So the answer is 21.6 minutes at 99.95%, and 4.32 minutes at 99.99%. Then add the part that makes it an SRE answer rather than an arithmetic one: each added nine divides the allowed downtime by about ten and typically multiplies the engineering cost of holding that line. The jump you were asked about isn't a small tightening — it removes roughly 17 of the 21.6 minutes the team currently has, which means a single ten-minute incident would go from spending about half the month's budget to blowing through more than twice it. Ask what the business case is before agreeing to it.

It's also worth naming the direction the budget should be reasoned from: the right SLO is the loosest target users won't notice you missing. Reliability beyond that point is waste, not virtue.

🟡 2.4 · What an error budget is for

They ask: "What is an error budget, and what should a team do differently when it still has budget versus when it's exhausted?"

An error budget is 100% minus the SLO, converted into a real quantity of allowed failure over the window. A 99.9% SLO doesn't mean "never fail" — it means you are explicitly permitted to fail 0.1% of the time.

The reframing that matters is that the budget is a resource to spend deliberately, not a warning to avoid. While budget remains, the team has explicit permission to take risk: ship faster, push a riskier migration, run a chaos experiment against production, skip an extra round of canarying. That's "burning" the budget — trading allowed unreliability for velocity, on purpose.

When the budget is exhausted, a pre-agreed policy flips the posture: no new risky releases, and the team's priority shifts to reliability work until the budget recovers as the window rolls forward or a fix demonstrably stops the bleed. That freeze is the mechanism that gives reliability work a standing claim on the roadmap, instead of losing every prioritization fight to the next quarter's features by default.

🔴 2.5 · Burn rate and why one threshold isn't enough

They ask: "Define burn rate. Then: a service has a 99.95% SLO and is currently running a 0.6% error ratio. What's the burn rate, and roughly how long until the budget is gone?"

Burn rate normalizes the observed badness against the budget itself: burn rate = observed bad-event ratio ÷ (1 − SLO). A burn rate of 1 means you're consuming budget at exactly the sustainable pace — sustain it for the whole window and you land at precisely zero remaining as the window closes. A burn rate of N means N times faster than sustainable.

1 − SLO = 1 − 0.9995 = 0.05%
burn rate = 0.6% / 0.05% = 12

time to exhaust a full budget = compliance_window / burn_rate
                              = 30 days / 12 = 2.5 days

Then explain why this matters more than a raw error percentage: 1% errors is catastrophic against a 99.99% SLO and comfortably survivable against a 99% one. Because burn rate is already normalized, the same alert rule travels unmodified across services with completely different targets, as long as each plugs in its own 1 − SLO.

If they push further, the follow-up is why a single window can't do the job. Alerting on the SLO's own 30-day compliance window is maximally precise and completely useless as a warning — by the time a 30-day average crosses the line the budget is already spent, so you've built an accurate obituary rather than an alarm. Alerting on a raw 5-minute error rate at the SLO's own threshold pages constantly, because 0.1% is a long-run target, not a value real traffic holds steady at minute to minute. Detection time and precision are in tension, and one window sits at exactly one point on that curve. Full derivation in multi-window, multi-burn-rate alerting.

🔴 2.6 · Multi-window alerting and the AND

They ask: "Why do burn-rate alerts pair a long window with a short one, and why are they joined with AND rather than OR?"

A burn-rate alert on a long window alone — say the trailing hour — has good precision, because an isolated two-minute blip barely moves an hour's average. But it has a less obvious problem called slow reset. Suppose a genuine 12-minute outage happens and gets fixed. The trailing 1-hour window still contains those 12 bad minutes for the rest of the hour, so the alert keeps firing — for up to 48 more minutes after the fix actually landed. That's not exactly a false positive, but it's a stale one, and it teaches whoever's on call to distrust the page.

The fix is to pair every alert with a short window, conventionally around one-twelfth the length of the long one, and require both to breach the same burn-rate threshold simultaneously. During a real ongoing incident, both windows run hot together, so the alert fires promptly. The moment the incident is genuinely fixed, the short window — which only looks back a few minutes — clears almost immediately, and because the condition is an AND, the alert clears with it, without waiting for the long window's trailing average to roll the bad data out.

Using OR instead reintroduces exactly the noise the short window was added to prevent: a five-minute blip alone can now trip the alert. The AND is load-bearing.

The tier structure that sits on top of this — commonly a fast page tier and a slower ticket tier, each with a long/short pair — is a widely used starting template adapted from Google's SRE Workbook, not a universal constant. Every multiplier in it is derived from a policy input (what percentage of budget you're willing to burn before alerting) divided by the long window as a fraction of the compliance window. Change the compliance window and every multiplier must be recomputed from the formula; the percentages carry over, the multipliers do not.

🔴 2.7 · Composite SLOs and chained services

They ask: "Three services on a checkout path each hit a 99.9% SLO, and every dashboard is green. Customers still complain. What's going on?"

If the user's journey requires every service on the critical path to succeed, that's a logical AND, and the journey's availability is the product of the individual availabilities — not the minimum, and not the average.

0.999 × 0.999 × 0.999 = 0.997002999 ≈ 99.70%

Converted to a 30-day error budget (43,200 minutes):
  single 99.9% hop:    (100% − 99.9%)  × 43,200 = 43.2   min
  composite (3 hops):  (100% − 99.70%) × 43,200 ≈ 129.47 min

129.47 / 43.2 ≈ 3× the single-hop budget.

So the journey can be broken for nearly three times as many minutes across the month as any single team's dashboard suggests, with nobody lying. "Every service dashboard is green and the composite journey SLO is red" isn't a contradiction — it's what an AND across components does to a probability.

Two things make a strong answer here. First, the independence caveat: the multiplication assumes each hop fails independently, and real systems routinely violate that — shared base images, a shared cluster, a shared AZ, one shared TLS certificate. When failures are positively correlated the real joint-failure rate is worse than the formula predicts, so treat 99.70% as an optimistic ceiling for a chain that shares infrastructure. Second, the fixes: allocate budget unevenly so a hub service that sits inside twenty journeys carries a tighter target than a leaf that sits in one; remove hops from the AND entirely where you can (a cache that serves a recent-good response, a circuit breaker with a defined fallback, an analytics write moved off the synchronous path onto a queue); and give the critical-user-journey SLO its own owner and its own alert, because no individual service team is naturally accountable for an AND that spans team boundaries. Worked through fully in SLO windows & composite SLOs.

Domain 3 · Toil & automation

☺ Like you're 10: Everyone can say "toil is boring work." The interview is checking whether you know the actual six-part definition and what you're supposed to do about it.

Four questions. The definitional one is close to a lock at screen stage, and its real value is the follow-up — "so what have you automated away?" — where you get to be specific about your own work. Detail in Toil & automation.

🟢 3.1 · The precise definition of toil

They ask: "What is toil? Is all operational work toil?"

No — toil is a specific technical term, not a synonym for work you don't enjoy. Work only counts as toil when it has all six of these properties:

• Manual — a human is executing it by hand, not supervising something that runs itself.
• Repetitive — you've done this exact task before and will do it again.
• Automatable — a machine could do it as well or better, with no judgment call that genuinely needs a person.
• Tactical — interrupt-driven and reactive (a disk filled up, a certificate expired), not something you chose as part of a plan.
• Devoid of enduring value — the service is in the same state afterwards as if the task had never been needed. You restored the status quo; you didn't improve anything.
• O(n) with service growth — it scales linearly with traffic, users, or machine count. Twice the fleet, twice the manual steps.

The property candidates most often miss is the last one, and it's the one that makes the distinction useful. Writing a design doc, debugging a novel outage, or building a dashboard can all be hard and unpleasant — none of it is toil, because each produces something that keeps paying off. Toil is work you'll be doing again next week in exactly the same shape, and more of as the system grows.

🟡 3.2 · The automation ladder

They ask: "You've identified a source of toil. How do you actually reduce it — and is full automation always the goal?"

Toil doesn't get eliminated in one leap; it climbs a ladder, and it's normal for different toil sources to sit on different rungs at once. Each rung removes a specific kind of human dependency:

1. Fully manual — an engineer does every step by hand, from memory. No repeatability guarantee at all.
2. Documented runbook — exact commands, exact order, so any on-call engineer executes it correctly under pressure. Still manual, but consistent and transferable.
3. Partially automated — a script or tool a human triggers and supervises. Execution is now consistent by construction; a human still decides when and confirms it worked.
4. Fully automated — the system detects the trigger and runs the fix itself, though a human may be notified and could intervene.
5. Self-service / autonomic — detection, decision and remediation all happen with no human in the path, not even as a notified bystander.

And no, rung 5 is not always the goal. The right stopping point depends on frequency, blast radius, and how much genuine judgment the task requires. What makes the ladder useful for prioritization is that a high-frequency toil source still sitting on rung 1 or 2 is usually the best return, because every step up removes a proportional slice of recurring load.

🟡 3.3 · Making the case for automation investment

They ask: "You want to spend two sprints automating something, and your product manager wants those sprints for features. How do you make the case?"

Automation competes for the same engineering time as feature work, so it needs the same kind of justification — with numbers, not conviction. The strongest case combines three things.

First, current cost: frequency multiplied by per-occurrence engineer-hours, pulled from actual ticket or on-call logs rather than estimated. Second, and more persuasive, the O(n) growth property — project what this toil bill looks like at 2× or 5× current scale. A toil source costing two hours a week today but scaling linearly with a fleet that's doubling every two quarters is a much stronger case than one costing ten hours a week but capped, because the capped one will never grow no matter what the service does. Third, the risk profile of manual execution: hand-run procedures under incident pressure are exactly where a mistake compounds into a bigger outage.

Be honest about the counter-argument too, because it makes the case more credible rather than less: automation that's brittle, undocumented, or unmaintained becomes a new toil source of its own — now someone babysits the automation. A script a human must still watch, restart when it fails silently, and patch every time the environment shifts hasn't actually left rung 3.

🔴 3.4 · A team reporting zero toil

They ask: "A team you're supporting reports their toil at under 10%. Do you believe them?"

Probably not, and the reason is worth stating carefully so it doesn't sound like an accusation. A team reporting very low toil is usually undercounting rather than toil-free, and the blind spots are consistent: toil buried inside on-call shifts that never gets logged as a ticket; "quick" manual steps embedded inside an otherwise-automated runbook; and toil individual engineers absorb quietly because doing it feels faster than arguing for automation time.

That last one is the important one culturally. Toil absorbed by a team's goodwill rather than pushed back is invisible precisely because the team is being helpful, and it's the first thing to balloon when the team is short-staffed. The fix isn't to challenge the number — it's to change how it's collected: sample actual on-call shift time rather than relying on ticket counts alone, count the manual steps inside runbooks explicitly, and make pushing work back a normal, expected action rather than an escalation.

The honest framing to offer: toil that isn't measured can't be capped, so a suspiciously low number is a measurement problem to fix, not a virtue to celebrate.

Domain 4 · Monitoring & observability

☺ Like you're 10: Four vitals, two checklists built on them, and one decision — what's worth waking a person up for — that matters more than all the rest.

Six questions. This domain and the SLO domain are the two universal prerequisites: scenario questions in every other domain will assume you can name what you'd look at and why. Detail in Monitoring & observability and alert design & alert fatigue.

🟢 4.1 · The four golden signals

They ask: "What are the four golden signals, and why is latency reported as a distribution rather than an average?"

Latency (time to service a request), traffic (demand on the system — requests per second, transactions per second, messages per second), errors (rate of failed requests), and saturation (how full the service is relative to its limit).

Latency goes as a distribution — p50, p95, p99 — because an average of 150ms can hide a p99 of three seconds hitting one request in a hundred, and that tail is where the complaints come from. There's a second, less-quoted reason worth adding: split successful latency from failed latency. A request that fails fast (a 400 in 8ms) and one that fails slow (a 500 after a 30-second timeout) are different problems, and averaging them into one number hides both.

If you want to show depth in one extra sentence: traffic is the denominator that makes the other three mean anything. Two hundred errors a second is background noise at 500,000 req/s and a full outage at 300 req/s. And saturation is the leading indicator of the four — a service at 90% memory isn't failing yet, but it predicts that latency and errors are about to move.

🟢 4.2 · USE and RED

They ask: "What are USE and RED, and when would you use each?"

USE — Utilization, Saturation, Errors — was popularized by Brendan Gregg for watching resources: a CPU, a disk, a connection pool, a queue. For each resource you ask what fraction of time it was busy, how much extra work it queued because it couldn't keep up, and how many error events it recorded. It's the method you reach for when root-causing "something upstream is slow" down to a specific saturated component.

RED — Rate, Errors, Duration — was popularized by Weaveworks for watching request-driven services end to end. It's essentially the golden signals with saturation dropped, because for a stateless request handler saturation usually surfaces one layer down, on the resources RED doesn't track — which is exactly where USE picks it up.

The answer that shows you understand rather than memorized: neither is a different theory from the golden signals. Both are the same four signals pre-filtered for what you're looking at, and a typical production stack runs both together — RED dashboards on every service in the request path, USE dashboards on the resources underneath them.

🟡 4.3 · Symptom-based versus cause-based alerting

They ask: "What's the difference between symptom-based and cause-based alerting, and why does one produce far fewer pages?"

Cause-based alerting pages on every internal condition that might eventually cause a problem: database CPU above 85%, one replica's disk above 90%, one dependency's error rate above 2%. It feels thorough and doesn't scale, for two reasons. A single real incident typically trips a dozen internal causes at once, each with its own rule, so one outage produces a dozen pages for one root cause. And many of those causes never become user-visible at all — a database can run at 85% CPU for hours with nobody's request getting slower, because it has headroom nobody documented. That alert fires anyway, gets acknowledged, does nothing, and trains whoever's on call to skim and dismiss.

Symptom-based alerting pages only when users are actually affected: p99 latency above an agreed threshold for five minutes, error rate above an agreed threshold, a golden-signal breach someone outside the company would notice. It produces dramatically fewer pages, and every one is actionable by construction — if it fired, someone is having a bad time right now, so there is always something worth doing, even if that something is triage.

Cause-level signals aren't worthless; they belong on the dashboard the person who was paged by a symptom uses to investigate why. The distinction worth stating precisely: dashboards are for humans who are already looking; alerts are for pulling a human's attention toward a problem in the first place, at 3am if necessary.

🟡 4.4 · Monitoring versus observability

They ask: "Is there a real difference between monitoring and observability, or is that just vendor language?"

There's a real distinction underneath the marketing, and the honest answer acknowledges both halves. Monitoring is watching a predefined set of signals against predefined thresholds — you decided in advance what could go wrong and instrumented for it, which is exactly what the golden signals are for. Observability is about whether the telemetry you emit lets you answer questions you didn't anticipate, without shipping new code to find out. Metrics tell you that latency went up; a trace tells you where in a multi-service call chain the time actually went, and a high-cardinality event lets you ask "which customer, on which build, in which region" after the fact.

Be honest that the boundary is fuzzy and contested, and that plenty of the distinction is vendor positioning. What isn't contested is the practical consequence: a system you can only monitor handles the failure modes you predicted, and a system you can genuinely observe also handles the ones you didn't. In an interview, the useful move is to say which problems in your own experience needed which — "our aggregate error rate told us something was wrong; it took traces to find that it was one downstream shard" is a far better answer than a definition.

🔴 4.5 · Auditing a noisy alerting setup

They ask: "Your on-call rotation is getting fifteen pages a shift. Where do you start?"

Start by naming the frame: a pager firing fifteen times a shift is not proof the system is fragile, it's proof the alerting is broken. It's a bug to file and fix, not a fact of on-call life to endure — and it has a cost ordinary toil doesn't, which is trust. Once responders learn to skim and dismiss, the real page gets missed too.

Then propose a structured audit rather than fixing whichever alert annoyed someone most recently, because that approach is slow and political. Export every page that fired over a window — ninety days is a reasonable default. Bucket every individual firing, not every rule, into one of five outcomes:

• Actioned — a human took a specific action that changed the outcome. This is the number you're protecting.
• Auto-resolved — cleared on its own before anyone did anything material. Either the threshold is too twitchy or this belongs behind auto-remediation, not a human.
• Duplicate / correlated — fired as a downstream symptom of an alert that had already paged for the same root cause. A grouping or inhibition gap, not a bad rule.
• Non-actionable / informational — real and confirmed, but nothing a human could usefully do right then. Wrong severity: it's a ticket or a dashboard panel.
• False positive — the condition wasn't real. A defect in the rule itself.

Roll each rule up to an actionability rate — actioned firings divided by total firings — and set a floor before you see the numbers so the data decides rather than whoever is most attached to a rule. Then dispose of everything under the floor: tighten thresholds, add grouping and inhibition for the correlated ones, demote the informational ones off the pager, rewrite or delete the false positives, and replace the auto-resolving ones with auto-remediation that pages only when the remediation itself fails.

🔴 4.6 · Instrumenting a service that has none

They ask: "You inherit a service with almost no instrumentation. What do you add first, and in what order?"

Work from the outside in, because that's the order that produces usable signal fastest.

First, a request-based SLI at the user-facing boundary — success rate and latency distribution measured at the gateway or load balancer, split by endpoint so a single critical path (checkout, login) can't be masked by a high-volume, low-stakes endpoint dragging the average up. Second, a blackbox or synthetic probe from outside the service's own metrics pipeline, so a total observability outage doesn't also leave you blind to the outage it was meant to reveal. Third, RED dashboards on the request path and USE dashboards on the resources beneath it, which gives whoever gets paged somewhere to look. Fourth, trace propagation, because the moment the service has more than one downstream you'll need to know which hop the time went to.

Alerts come last and stay small: typically one symptom-based alert for elevated error rate and one for elevated latency, both measured against what users experience. Everything else — replica lag, queue depth, individual resource utilization — lives on dashboards for triage, not on the pager. Once there's enough history to know what normal looks like, convert the two symptom alerts to burn-rate alerts against a documented SLO, which is the point where the alerting stops being "does this feel bad" and starts being "has this burned budget fast enough that someone needs to act now."

One caveat worth volunteering: don't set the SLO on day one from the first week of data. That's how a vanity SLO gets created — a target derived from what the service already does rather than from what users need.

Domain 5 · Incident management & on-call

☺ Like you're 10: Half of this domain is process knowledge and half is a quiet check on whether you'd be pleasant to share a pager with.

Five questions. This domain doubles as a behavioral probe more than any other — how you describe a rotation you've been on tells the interviewer a great deal about your last team's culture and your own tolerance for things being broken. Detail in Incident management & on-call and incident command for large incidents.

🟢 5.1 · Designing a rotation

They ask: "How would you structure an on-call rotation for a team of six?"

Two engineers deep: a primary who receives every page first, and a secondary paged automatically if the primary doesn't acknowledge within a set window. The secondary covers exactly the failure mode a primary-only rotation misses — the primary is on a flight, in the shower, or their phone died, and without a fallback the page silently drops until someone notices a red dashboard.

On length: a day-long rotation forces excessive handoffs and never lets anyone build situational awareness of what changed recently; a month-long rotation concentrates burnout in one person and lets fatigue go unnoticed until it's severe. Most engineering organizations converge on one week as the balance — long enough to build context on what's currently unstable, short enough that one bad week doesn't define someone's month. Some teams deliberately shorten the primary shift for services with high page volume, on the reasoning that bounding continuous sleep disruption matters more than administrative tidiness.

Then add the piece candidates usually skip: the handoff should be a structured event, not a passing Slack message. A short call or a written doc covering open incidents, anything running degraded, recent deploys that haven't fully baked, and any alert currently flapping or known-noisy. Without it, every incoming primary rediscovers context the outgoing one already had.

🟢 5.2 · Severity levels

They ask: "What separates a SEV1 from a SEV2, and why does it matter that the distinction is written down in advance?"

A typical three-tier matrix runs roughly like this. SEV1 is a full outage or critical-path failure — the core product is down or unusable for most users. It pages the primary immediately, 24/7, and typically auto-declares an incident channel with an incident commander assigned within minutes. SEV2 is partial degradation: a subset of users or a non-critical feature affected, or error rate and latency crossing an SLO threshold without full unavailability. It pages during business hours; overnight it may queue for the next shift or page at reduced urgency, depending on policy. SEV3 is a minor issue — a cosmetic bug, a non-user-facing job failing, a metric drifting but well inside its budget. It's a ticket, worked in business hours, and never pages anyone.

The exact tiers vary between organizations, and it's fine to say so. What doesn't vary is why the matrix is written before it's needed: it encodes the judgment ahead of time instead of relitigating it mid-incident, and it does two jobs at once. It protects sleep by refusing to page below the line, and it protects trust in paging by guaranteeing that when the phone does ring, it's real. A team that pages SEV3-grade issues at 3am trains its own engineers to silence pages — which is precisely the failure a well-drawn matrix exists to prevent.

🟡 5.3 · Incident command roles

They ask: "During a major outage, why shouldn't the incident commander also be the one debugging?"

Because running the room is a full cognitive load on its own. The incident commander tracks who's doing what, decides what to try next, makes the call on risky mitigations — roll back versus forward-fix — and keeps the incident moving. An IC who's also staring at logs loses the coordination thread at exactly the moment it matters most.

The structure that supports this comes almost intact from wildland firefighting's Incident Command System, formalized in the 1970s when commanders realized crews from different agencies couldn't coordinate a fast-moving fire without a shared chain of command. At the fire, the incident commander stands back and decides where crews go; nobody expects them to swing an axe. Software incident response uses the same shape:

• Incident Commander — owns the incident, coordinates, escalates severity, makes the final call on mitigations. Does not debug.
• Ops / Comms lead — owns everything outward-facing: the status page, the executive pinging the channel, and a predictable stakeholder update cadence (every 15–30 minutes is a common default) so responders aren't interrupted to answer "any update?"
• Subject-matter responders — the engineers actually debugging: the database owner, the team that shipped the suspect deploy. Pulled in by the IC as the incident narrows, not paged by default for every SEV1.

Worth adding honestly: for a small incident, the responder who picks up the page effectively is the IC and there's nobody else to be. The role separation earns its overhead once the incident is big enough that people start talking over each other — which is exactly the trigger for formally declaring incident command rather than letting it emerge.

🟡 5.4 · What healthy on-call looks like

They ask: "How would you tell whether an on-call rotation is healthy? What number would you look at?"

Make it measurable rather than a vibe. The most useful single number is actionable pages per shift, tracked per rotation over time. Industry on-call surveys and most mature SRE organizations converge on a rough ceiling somewhere around two or three actionable pages per shift as a health signal — not a hard SLA, and worth calibrating to your own service, but a defensible starting bar. Below it, on-call is a manageable responsibility. Sustained well above it, the correct conclusion is not that the team needs to toughen up; it's that the alerting is broken, and the fix is alert redesign or automating away the repeat cause.

Two other signals worth naming. First, whether pages cluster overnight — a rotation with the same total volume but concentrated at 3am is a materially worse rotation. Second, compensation: on-call is work even during the hours nothing pages, because it constrains what an engineer can do with their evening. Organizations that treat it as an unpaid tax on seniority burn out their best people fastest; a visible stipend or comp time per rotation is table stakes anywhere running a 24/7 service. Raising this in an interview is legitimate and is also, quietly, a question about them.

🔴 5.5 · A page fires at 3am — walk me through it

They ask: "It's 3am. You get paged: checkout error rate is elevated. Talk me through the first fifteen minutes."

Answer as a method, not a script, and say the method out loud so the interviewer can follow your reasoning even if your guesses are wrong.

Acknowledge first, immediately — before anything else, so the escalation to secondary doesn't fire and so everyone watching knows a human is on it.

Establish user impact and scope — what fraction of requests, which endpoints, which regions, which customers. This is where the SLI you'd want to already exist earns its keep: if checkout has a request-based success-rate SLI split by endpoint, this takes seconds. Convert impact into burn rate if you can, because that's what tells you whether this is a "fix it now" or a "fix it carefully" problem.

Check what changed — recent deploys, config or feature-flag flips, infrastructure changes, and any dependency's own status. The overwhelming majority of incidents have a change behind them, and the change log is a faster search space than the codebase.

Decide mitigate versus diagnose, and say the decision out loud — this is the judgment the question is actually about. If a recent deploy correlates, roll back to the last known-good artifact first and diagnose afterwards; restoring the user experience is not the same task as understanding the bug, and doing them in the wrong order costs error budget. If nothing correlates, the next moves are the cheap generic mitigations: shed load, fail over, degrade the non-critical dependency, scale out the saturated tier.

Escalate on a clock, not on a feeling — decide in advance how long you'll work alone before pulling in the service owner or declaring incident command. "If I don't have a working hypothesis in ten minutes, I page the owning team" is a much better answer than "I'd escalate if I got stuck," because heroics under sleep deprivation is the exact failure mode a rotation is designed to prevent.

Keep a timeline as you go — timestamps in the incident channel as things happen, because reconstructing them afterwards is unreliable and the postmortem depends on it.

Domain 6 · Postmortems & blameless culture

☺ Like you're 10: Cheap questions to ask, extremely revealing to answer — how you talk about a past outage says what your last team was like.

Four questions. Watch your language here more than anywhere else on this page: if you describe a past incident by naming what a colleague did wrong, you have answered a question the interviewer didn't ask. Detail in Postmortems & blameless culture.

🟢 6.1 · Why blameless

They ask: "Why do we run blameless postmortems? Isn't there a point where someone genuinely was careless?"

Blamelessness is not about being nice — it's a deliberate information-gathering strategy, and framing it that way is what makes the answer credible. Almost every outage traces back to a human action somewhere: someone ran a migration, merged a config change, typed a flag wrong. If a postmortem's conclusion is "engineer X made a mistake," it has stopped investigating exactly where the useful information starts. The real question is never who did this but what let this be possible — why the pipeline allowed an unreviewed change to reach production, why monitoring didn't catch the regression before a customer did, why a command that deletes data had no dry-run mode.

The practical case is about what happens to information. When engineers expect blame they hedge: they omit that they were tired, that they skipped a step because the runbook was stale, that they'd noticed the same near-miss two weeks earlier and said nothing. Punishing whoever happened to be holding the pager when a systemic flaw finally triggered doesn't fix the flaw — it teaches everyone else to hide the same flaw better, and guarantees it fires again under someone else's watch.

The aviation parallel is worth one sentence if you want it: a pilot reporting a near-miss gets immunity for that report precisely because investigators want the full honest detail so they can fix the procedure or the instrument that made the error possible.

🟢 6.2 · What goes in the document

They ask: "What sections does a postmortem contain, and which one do teams most often get wrong?"

The sections that show up across nearly every mature program: a summary (three or four sentences — what broke, how long, who was affected, written so someone skimming fifty past postmortems gets the gist); a timeline (timestamped, usually in UTC, from triggering event through detection, escalation, mitigation and resolution); root causes — plural, deliberately; impact, quantified; what went well; what went poorly; and action items, each with a named owner and a due date.

Two get botched most often. Root causes gets written as singular — but real incidents almost never have one root cause. A migration that dropped a column is a trigger; the absence of a staging check, the absence of a rollback plan, and an alert threshold too loose to catch the early signal are also root causes, and each deserves its own action item. And impact gets written vaguely — "some users had a bad experience" — which makes it impossible to prioritize the fix against other work three weeks later. Quantify it: requests failed, customers affected, SLO minutes or error-budget percentage consumed, support tickets opened.

🟡 6.3 · Five whys, and where it goes wrong

They ask: "Walk me through the five whys on an outage. How do you know when you've gone deep enough?"

The technique, adapted from Sakichi Toyoda's work at Toyota, forces the investigation past the first answer, which is almost always superficial. You ask why repeatedly, each time treating the previous answer as the new thing to explain, until you land on a cause that is systemic and fixable rather than an individual's action in the moment. Five is a guideline — some chains resolve in three, some need seven.

Symptom: The site was down for 22 minutes.
Why? The database connection pool was exhausted and new requests
     couldn't get a connection.
Why? A deploy introduced a code path that opened a new connection
     per request instead of reusing one from the pool.
Why? The connection-pooling regression wasn't caught by the test
     suite before merge.
Why? There is no load test in CI that exercises connection reuse
     under concurrent traffic — only functional correctness is
     tested, not resource behavior under load.
Why? The team has never had a load-related production incident
     before, so load testing was never prioritized into CI.

Root cause: CI validates correctness but not resource usage under
concurrent load, so this class of regression merges undetected
until it meets production traffic.

Two tells that you've stopped too early or gone wrong. First: if the chain terminates on a person — "the engineer should have known better" — back up and ask why that person was ever in a position for one individual action to cause an outage-scale failure. That next why is where the fixable cause lives. Second: five whys is a single linear chain and real incidents are usually a tree. If the timeline also shows the alert fired forty minutes late and the runbook step for cache invalidation was wrong, those are separate root causes needing their own chains and their own action items — not footnotes on the primary one.

🔴 6.4 · A postmortem process that's become theater

They ask: "A team writes a postmortem for every incident, and the same class of failure keeps recurring. What's broken?"

Almost certainly the loop, not the document. An action item that never gets done is worse than not writing the postmortem at all, because it converts real engineering hours into a document that creates the appearance of a fix without the substance of one — and next time the same failure fires, the archive will show it was already diagnosed and then dropped.

The diagnostic is a rising stack of postmortems whose action items sit open for months with no owner following up. The fix isn't writing better postmortems; it's fixing the tracking and accountability loop around the ones already written. Concretely: store each action item as a normal ticket linked back to the doc, not as prose inside it — a prose item ("we should add a load test") has no owner, no due date, and no way to appear on anyone's sprint board, so it will be rediscovered, unfixed, in the next postmortem for the same cause. Some teams put a hard SLA on postmortem action items, track completion rate on a dashboard, and escalate slippage, treating that completion rate as a reliability metric in its own right.

There's a second, subtler failure worth naming if you want to show real depth: a process that is formally blameless for a first incident but shifts tone the moment the same engineer's name appears a second or third time. Nobody has to announce that shift for it to work — every other engineer notices it happen to a colleague once, and the hedging comes straight back, specifically among the people whose names recur most, which in most rotations means whoever has been on call longest. Watch for phrasing drifting from systemic ("the deploy pipeline allowed this to reach prod without a canary step") to personal ("this is the third time this team has skipped canarying").

Domain 7 · Capacity planning & performance

☺ Like you're 10: The domain where they check whether you can actually reason with numbers — and where "we'll just autoscale" is the answer that ends the conversation.

Five questions. This is where quantitative reasoning shows up, and it's also the domain candidates most often collapse into a single word. Detail in Capacity planning & performance and queueing theory for SRE.

🟢 7.1 · Load, stress and soak tests

They ask: "What's the difference between a load test, a stress test and a soak test?"

A load test holds traffic at an expected level to confirm the system meets its latency and error-rate targets there. A stress test pushes past expected levels until something breaks, to find the actual ceiling. A soak test holds elevated traffic for an extended period — hours, not minutes — to surface slow failure modes a short burst never would: memory leaks, connection-pool exhaustion, disk fill, file-descriptor leaks.

The addition that makes this a strong answer rather than a definition: the most valuable output of a stress test isn't a number, it's the shape of the failure. Does p99 degrade gracefully as load climbs, or does the system fall off a cliff at a specific rate? Does it fail closed — rejecting excess cleanly with backpressure or load shedding — or fail open, queueing everything until memory is exhausted and the process dies? A system that degrades gracefully is far safer to run near its limit than one that behaves perfectly right up until it doesn't.

One more, if you want to pre-empt the follow-up: test against a realistic traffic mix, not raw requests per second. Hammering one cheap endpoint reports a far higher capacity than the system actually has once real users mix in expensive queries, cache misses and write-heavy paths. The breaking point is a property of the traffic shape, not just the volume.

🟡 7.2 · Headroom, and what it's for

They ask: "You've load-tested and found the ceiling. How much headroom do you leave, and why?"

There's no single correct percentage, and saying so is part of the right answer — it depends on how volatile demand is, how quickly you can add capacity, and how bad an overload actually is for that service. What matters is that headroom absorbs two different kinds of surprise, and the second is the one people forget.

The first is a demand spike the forecast didn't predict — content going viral, a competitor's outage sending you their traffic. The second is a capacity-side event where you have less of your own fleet than planned: a failed availability zone, a bad deploy being rolled back while nodes drain, maintenance taking part of the fleet offline. Headroom sized only for demand spikes is the common planning gap, and the arithmetic is unforgiving: a system running at 80% of measured ceiling across three AZs looks safe until you lose one, at which point the remaining two are instantly well over 100% of their fair share.

A common starting heuristic is enough headroom to absorb the loss of one redundant zone or region — frequently landing somewhere around 30–50% above steady-state baseline for a system built on N+1 or N+2 redundancy — plus additional margin for demand volatility on top. Then commit to revisiting it on a cadence with the forecast, rather than treating it as decided once.

🟡 7.3 · Forecasting demand

They ask: "How do you forecast capacity for the next two quarters?"

Two inputs, and a forecast built on only the first is the most common mistake.

The baseline is historical growth: pull request rate, storage, or active users over the past several months and fit a trend — linear for a service growing at a steady absolute rate, exponential for one compounding month over month. Plot the data before assuming which regime applies; most services blend both at different points in their lifecycle.

The second input is everything the business already knows is coming and that has therefore never appeared in the historical data: a marketing campaign with an announced date, a partner integration going live, a pricing change expected to shift usage, a seasonal peak. These get added as explicit adjustments, usually as a multiplier on top of the organic trend, and they have to be sourced from the teams running those events rather than guessed. "Marketing expects a 4× spike in signups during launch week" is a capacity input, and the only way to get it is to ask.

Then say the thing that makes it a practice rather than a spreadsheet: the forecast is a living document revisited on a cadence — monthly or quarterly — because both the trend and the calendar keep moving.

🔴 7.4 · Why latency exploded between 70% and 90%

They ask: "Traffic grew about 30% and average CPU went from 70% to 90%. Latency didn't grow 30% — p99 went up several times over. Why?"

Because queueing delay is not proportional to utilization. The relationship has a (1 − ρ) in the denominator, so the curve stays nearly flat across most of its range and then turns almost vertical exactly where a naive capacity plan assumes it's still safe.

The simplest model that produces the right shape is M/M/1 — Markovian arrivals, exponential service times, one server. Under it, mean time in system is the baseline service time divided by (1 − ρ):

Service takes 5 ms when nothing is queued → μ = 200 req/s

At ρ = 0.7 (λ = 140 req/s):  W = 5 / (1 − 0.7) = 16.7 ms
At ρ = 0.9 (λ = 180 req/s):  W = 5 / (1 − 0.9) = 50.0 ms

Traffic:  140 → 180 req/s = +28.6%
Latency:  16.7 → 50.0 ms  = ×3.0

A 28.6% increase in traffic produces a 3× increase in mean latency. That asymmetry is the whole answer, and it's arithmetic rather than a war story.

Two extensions if they push. First, the tail is worse than the mean by a knowable amount — under M/M/1's assumptions, time in system is exponentially distributed, so p99 sits near mean × ln(100) ≈ 4.6× the mean. Treat that multiplier as an optimistic floor rather than a number to copy, because it's exact only under assumptions real traffic doesn't meet. Second, real traffic is burstier than the model assumes, and burstiness makes it worse, not better. Kingman's formula generalizes the result by multiplying the same utilization term by a variability factor built from the coefficients of variation of arrivals and service times — retry storms, GC pauses, cache-miss stampedes and cron synchronization all push that factor above one.

And the operational conclusion, which is what they actually want: an average-utilization dashboard is structurally incapable of warning you before the cliff. A five-minute average of 70% can contain twenty seconds where instantaneous utilization was 95%, and because the delay curve is convex those seconds contribute far more than their share of queueing delay — which surfaces in p99 while the average line barely moves. Plan against tail latency and instantaneous saturation, not against mean CPU.

🔴 7.5 · Horizontal versus vertical scaling

They ask: "When would you scale up rather than out?"

Mature systems use both at different layers rather than picking one dogmatically, so answer as a trade-off rather than a preference.

Vertical — bigger instances — is operationally simple: no sharding, no cross-node coordination, no distributed-systems concerns. Its costs are a hard ceiling (the largest instance type available), usually a restart or brief downtime to resize, and — the one people forget — it does nothing for availability. A bigger single instance is still a single point of failure. It's the natural first move for stateful systems that are hard to shard, like a single primary database, until that ceiling is reached.

Horizontal — more instances — has effectively no ceiling and directly improves availability, since losing one of many degrades capacity rather than taking the service down. Its cost is complexity: the workload has to be stateless or shardable, you need load balancing and service discovery, and coordination problems appear that a single bigger box never had — cache consistency, distributed locking, connection-pool sizing per node.

In practice, stateless application tiers are the easy horizontal case and usually pair with autoscaling on a live signal. Stateful tiers, primary databases especially, are frequently scaled vertically first, with sharding and read replicas introduced only once vertical headroom is exhausted, because each of those adds real operational cost.

One nuance worth volunteering: horizontal scaling can make a downstream problem worse. Doubling your stateless app tier doubles the connections landing on the database behind it, and a service that was fine at 40 pods can exhaust a connection pool at 80 — which is a capacity decision the app-tier autoscaler knows nothing about.

Domain 8 · Reliability patterns

☺ Like you're 10: The "do you actually build things" domain — retries, timeouts, breakers and bulkheads are where hand-waving gets caught fastest.

Six questions. These come up in both the deep-dive and the design stages, and the follow-ups tend to be sharp because the failure modes are concrete and checkable. Detail in Reliability patterns and distributed systems reliability fundamentals.

🟢 8.1 · The circuit breaker state machine

They ask: "Explain the circuit breaker pattern. What are its states and what does each one do?"

A circuit breaker sits in front of a call to a downstream dependency and tracks its recent success and failure rate. Three states, named after the electrical breaker it borrows from.

Closed is normal operation: calls pass through and the breaker counts failures over a sliding window. If the failure rate crosses a threshold — 50% over the last 20 requests is a common default, though the numbers are configuration, not doctrine — it trips.

Open: for a fixed cooldown, often tens of seconds, every call fails immediately without attempting the network request at all, returning an error or a fallback straight away.

Half-open: after the cooldown, a small number of test requests are let through. Succeed and the breaker closes and normal traffic resumes; fail and it reopens and the cooldown restarts.

The half-open state is the part worth understanding rather than memorizing: it's how the breaker discovers recovery without slamming a still-fragile dependency with full traffic the instant the timer expires.

Say what the open state protects, because that's the actual point: it protects both sides. It stops your service piling up threads and connections waiting on something already struggling, and it stops you hammering that dependency while it's trying to recover — retries against an overloaded service are frequently what turns a brief blip into a prolonged outage.

🟢 8.2 · Backoff and jitter

They ask: "Why isn't exponential backoff enough on its own?"

Because every client following the same schedule retries in synchronized waves. Everyone's second attempt lands at roughly the same moment, then everyone's third, and so on — so the retry traffic still arrives as a series of spikes against a service that's trying to recover.

Jitter fixes it by adding randomness to the wait. The variant worth naming specifically is full jitter: pick a random wait uniformly between zero and the current backoff ceiling, rather than taking the ceiling and adding a little noise on top. AWS's 2015 analysis of the problem found full jitter spreads retries most effectively and does measurably less total work against a recovering service than backoff alone.

# exponential backoff with full jitter
ceiling  = min(max_delay, base_delay * 2^(attempt - 1))
wait     = random_uniform(0, ceiling)     # the WHOLE wait is random

attempt 1 fails → ceiling 200ms  → wait somewhere in 0–200ms
attempt 2 fails → ceiling 400ms  → wait somewhere in 0–400ms
attempt 3 fails → ceiling 800ms  → wait somewhere in 0–800ms

Worth adding that you shouldn't hand-write this loop — resilience4j's Retry module, Polly, and most gRPC and HTTP clients expose base delay, cap and jitter strategy as configuration.

🟡 8.3 · When retrying is unsafe

They ask: "A checkout POST now retries five times with exponential backoff. Ship it?"

Not yet — not without an idempotency key. Retries are only safe when repeating the call has the same effect as doing it once. Retrying a GET is almost always fine. Retrying a POST that charges a card or creates an order is not, because attempt two can succeed downstream while the response back to you is lost, and attempt three then charges the card again.

The fix is a client-supplied idempotency key that the downstream service uses to recognize a repeat of the same logical request and return the original result rather than performing the side effect again. Payment APIs such as Stripe's require an idempotency header for exactly this reason. A retry loop is incomplete for any state-changing call until the request carries one.

Two further points if you want to go deeper. Retries also need a budget, not just a count — an unbounded retry policy across a deep call chain multiplies load geometrically, so each hop retrying three times across four hops is up to 81 attempts for one user request. And retries should be paired with a circuit breaker, so a dependency that has no chance of succeeding stops being retried at all.

🟡 8.4 · Timeouts and deadline propagation

They ask: "Why does every network call need a timeout, and how should timeouts behave across a call chain?"

A call with no timeout is a call that can wait forever, and forever is exactly how one slow dependency becomes a full outage of the service calling it. Threads or connections pile up waiting on a response that never arrives, the pool exhausts, and every other request — including ones with nothing to do with the failing dependency — starts failing too.

So every network call needs an explicit timeout, set separately for connection establishment and for waiting on a response. And in a call chain the timeout budget should shrink as it propagates: if a user-facing request has a two-second deadline and the first two hops have already spent 1.5 seconds, the third hop must not still be given the full two seconds. gRPC propagates a deadline through the call context for precisely this reason, so each hop knows how much time is genuinely left rather than starting its own fresh clock.

The failure mode to name: static per-hop timeouts that don't compose. Four hops each with a "reasonable" two-second timeout produce an eight-second worst case for a user who gave up after three.

🔴 8.5 · Circuit breakers versus bulkheads

They ask: "You already have circuit breakers. Why would you also need bulkheads?"

Because they solve different halves of the same problem, and running one without the other leaves a real gap.

A circuit breaker acts over time against one specific dependency: it decides whether to keep calling it at all. A bulkhead acts at any single moment across dependencies: it caps how much of a shared resource pool — thread-pool slots, connection-pool capacity, semaphore permits — one dependency's calls can consume, so a slow downstream can only exhaust its own allotment. The name comes from the watertight compartments that let a ship's hull take damage in one section without sinking.

The gap each leaves alone is the crux. A breaker with no bulkhead can still let a slow but not yet failing dependency exhaust the shared thread pool before the failure rate ever crosses the trip threshold — the calls aren't erroring, they're just taking eight seconds each, and the breaker has nothing to count. A bulkhead with no breaker keeps politely retrying a dependency that has no chance of succeeding, inside its own compartment, forever.

Sizing the compartments is itself a capacity exercise rather than a one-time guess — which is why this connects back to capacity planning rather than being purely a coding pattern.

🔴 8.6 · Graceful degradation versus load shedding

They ask: "Graceful degradation and load shedding both trade completeness for availability. What's the actual difference, and how do you decide what to degrade?"

They trigger on completely different conditions, which is the distinction worth leading with.

Graceful degradation applies when one specific, non-critical dependency is down. Instead of failing the whole request, the service serves a reduced version that omits what that dependency provided — a product page whose recommendation service is unavailable still renders the product, price and buy button, it just drops the "customers also bought" rail.

Load shedding is a different lever applied at the front door under overall overload: the service deliberately rejects a fraction of requests outright, typically a 503 with a Retry-After, to protect its ability to serve the rest with acceptable latency. The alternative — accepting everything and letting everyone get proportionally slower — is usually worse, because as latency climbs upstream timeouts start firing and triggering retries, adding more load to an already-overloaded system.

On how to decide what to degrade: the classification has to be made upfront, at design time, not during the incident. Every dependency gets labeled critical (its failure should fail the request) or non-critical (its failure should degrade the response), and every non-critical one needs an explicitly written fallback — because the default behavior of most code is to let a failed call propagate as an exception all the way up. A dependency you intended to be optional but never wrote a fallback for is a hard dependency in practice.

Two production refinements worth naming: priority-aware shedding, which drops low-priority batch traffic before user-facing traffic, and adaptive concurrency limits, which estimate a safe in-flight request count from observed latency rather than using a fixed number. And validate both paths deliberately with fault injection — see chaos engineering — rather than discovering a broken fallback during a real incident.

Domain 9 · Release engineering & progressive delivery

☺ Like you're 10: Where reliability meets shipping speed — and where the interviewer finds out whether you'd be the person who blocks everything or the person who makes shipping safe.

Four questions. Common in organizations hiring an SRE specifically to stop bad launches without stopping good ones. Detail in Release engineering & progressive delivery.

🟢 9.1 · What progressive delivery combines

They ask: "What is progressive delivery, and what does it actually consist of?"

It isn't a single new technique — it's three existing ones wired together so that none of them needs a human in the loop for the common case. Alone, each is limited: a canary without automated analysis is just a smaller blast radius still watched by a person; a feature flag without a kill switch tied to health signals is just a manual toggle; automated rollback without a known-good target to revert to isn't reliably automatable at all.

• Canary release — the new version runs alongside the old, receiving a deliberately small slice of real traffic, usually selected by header, cohort or a hash of session ID rather than pure randomness, so the same users stay on the same version for the duration.
• Feature flags — a runtime switch that decouples deploying code from activating it. The new binary can already be on every host; the flag is what exposes the new path, and flipping it back is a config change, not a redeploy.
• Automated rollback — a controller watching the canary's metrics against the running baseline, reverting the moment the comparison crosses a threshold, without waiting for someone to notice.

The framing that makes this land: progressive delivery does not reduce the chance a bad release happens. It reduces blast radius and detection time when one does. A canary at 1% traffic that fails still fails — the win is that 1% of users saw it for one bake window instead of 100% of users seeing it until a human noticed. It's a faster, smaller failure detector, not a substitute for testing before the canary starts.

🟡 9.2 · What the health-check gate actually checks

They ask: "What does the gate between canary traffic steps evaluate, and why compare against a live baseline rather than a fixed threshold?"

A typical gate checks two distinct kinds of thing, and conflating them is a common design mistake.

First, golden-signal deltas: is the canary's error rate, p99 latency or saturation meaningfully worse than the currently-running stable version — not worse than some fixed number in isolation. Comparing against a live baseline is what lets the same gate stay correct through a normal daily traffic swing that would otherwise trip a fixed threshold. Both versions are experiencing the same Tuesday at the same hour; the only variable left is the change.

Second, SLO burn rate: is the canary cohort burning its share of the error budget faster than the sustainable rate. This catches a canary that technically passes a raw error-rate threshold while consuming budget at ten times the pace the window can sustain — the same multi-window burn-rate reasoning used for paging, applied to a promotion decision.

If the gate passes for the configured bake time — commonly 5–15 minutes, long enough to catch a slow leak or a cache-warming artifact, short enough to keep the pipeline moving — traffic promotes. If it fails, the controller reverts to the last known-good version and halts. It does not retry with less traffic, which would just repeat the same failing comparison. And a rollback should page or at minimum log loudly with the specific metric and threshold that tripped, so the failure is diagnosable rather than merely reverted.

🟡 9.3 · Why rollback needs an immutable artifact

They ask: "Your rollback procedure rebuilds the previous git tag. What's wrong with that?"

It reintroduces exactly the risk progressive delivery exists to remove. If rolling back means recompiling, a dependency resolver could pick up a different transitive version than it did last time, a base image could have been repatched, or a compiler flag could differ — any of which means the "known-good" version you rolled back to isn't actually the one that was known good.

The discipline is build once, promote everywhere: an artifact built one time, hashed, and stored immutably, so rollback means redeploying an exact previously-verified digest rather than rebuilding anything. Practically, a canary and its baseline should reference the artifact by immutable digest, not by a mutable tag like :latest or even :stable — a tag can be repointed underneath a rollout, a digest cannot. And the rollback target the controller stores should be captured before the canary starts receiving traffic, not looked up afterwards.

There's a timing argument too, which is usually the one that persuades people: a rollback that redeploys a known artifact takes seconds; one that reruns a build pipeline takes as long as the pipeline does — burning error budget the entire time, during an incident, on a build that could itself fail.

🔴 9.4 · Release trains versus continuous progressive delivery

They ask: "When is a release train the better model, and can the two be combined?"

A release train inverts progressive delivery's core assumption. Instead of each change shipping the moment it's ready, changes accumulate against a fixed cadence and whatever has merged and passed CI by the cutoff ships together, on schedule, whether or not any individual change is urgent. Anything that misses waits for the next train.

The trade is real in both directions. Progressive delivery minimizes blast radius per change and shortens time-to-detect for a bad release, at the cost of more moving release infrastructure and constant background rollout activity. Release trains minimize release-process overhead and give downstream teams — QA, docs, customers on a regulated update schedule — a predictable rhythm to plan around, at the cost of batching: when a train breaks something, isolating which of a dozen changes caused it is strictly harder than isolating one canary.

Trains fit where cadence is genuinely a constraint rather than a habit: mobile apps bound by app-store review latency, on-premises software customers install on their own schedule, or anything with a regulated change-approval window.

And yes, they combine — this is the answer that shows you've thought about it rather than picked a side. A mature setup often runs a train as the outer cadence (what ships this week) with progressive delivery as the inner mechanism (how each of this week's changes rolls out safely once it's in the train). Batching decides when; progressive delivery decides how safely.

Domain 10 · Chaos engineering

☺ Like you're 10: Asked more often as a "would you start this here" question than a "have you run this at scale" question — so answer with the safety controls, not the war stories.

Four questions. Many teams asking about chaos want to start a program rather than hand you an existing one, so the strongest answers emphasize how you'd make it safe and get buy-in. Detail in Chaos engineering and chaos engineering at scale.

🟢 10.1 · What Chaos Monkey did, and why

They ask: "Where did chaos engineering come from, and what did Chaos Monkey actually do?"

It traces to Netflix's migration to AWS around 2010. Running on someone else's infrastructure meant instances could and did disappear without warning, and Netflix's engineers recognized that a system which merely could tolerate that was not the same as one that had been shown to. Chaos Monkey randomly terminated production instances during business hours, on a schedule, with no warning to the team on call.

The randomness and lack of warning were the design, not an accident. If termination is rare and unpredictable, engineers write code that assumes instances live forever and that assumption goes untested until the day a real termination cascades. If termination happens constantly and on purpose, resilience to it stops being optional — code that can't survive an instance disappearing gets found and fixed within days rather than once a year. Business hours matters too: the point is that engineers are awake and watching.

Netflix extended it into the "Simian Army" — latency injection, conformity checks, and tools that simulated the loss of an entire availability zone or region — and the practice generalized into what the industry now calls chaos engineering, formalized in the 2020 O'Reilly book of the same name by Casey Rosenthal and Nora Jones.

🟡 10.2 · What makes a real experiment

They ask: "How do you design a chaos experiment?"

It starts with a specific, falsifiable hypothesis about steady-state behavior — not an intention to see what breaks. A well-formed hypothesis has three parts:

• The fault — what you're injecting: a timeout, a dependency outage, added latency, an instance kill.
• The scope — which service, which environment, which percentage of traffic.
• The expected steady state — the specific metric or user-visible behavior that should hold anyway, stated as a testable threshold. "Checkout completion rate stays above 99%" — not "things still work."

So "if the recommendations service times out, checkout should still complete, degraded to a static fallback list, with completion rate above 99%" is a hypothesis. "Let's see what happens if we kill the recommendations service" is not — it has no stated pass condition, so an engineer watching the aftermath has no way to distinguish a working fallback from a lucky escape.

Then run it like a scientific test: inject, measure the steady-state metric, record whether the hypothesis held. Say the thing most candidates miss — a failed hypothesis is not a failed experiment. It's the experiment doing its job, surfacing a gap between the resilience you designed and the resilience you have, before a real failure finds it. And if every experiment a team runs confirms its hypothesis, that's usually a sign they're only testing failure modes already known to be handled; the useful experiments are the ones with real uncertainty attached.

🟡 10.3 · Blast radius control

They ask: "How do you run chaos experiments in production without causing the outage you're trying to prevent?"

By bounding in advance how much damage a wrong hypothesis can do, and expanding that bound only after confidence has been earned at a smaller one. The progression is consistent across mature programs:

1. Non-production first — staging or a dedicated chaos environment that mirrors production topology closely enough that results transfer. A wrong hypothesis costs engineering time and nothing else.
2. One instance — a single host or pod in production, chosen so redundancy should absorb its loss with no visible effect. The smallest unit of real-world signal available.
3. A small percentage of traffic — 1%, then 5%, then larger, usually gated behind the same traffic-shifting infrastructure that makes canary releases safe.
4. Full production traffic — by which point the same hypothesis has already held at every smaller scope and the rollback has been exercised, not just planned.

Every step up should be gated on a real abort mechanism rather than intention: an automated rollback that trips the instant the steady-state metric breaches threshold, a manual kill switch, and a dashboard watched live for the duration. Say the sharp version out loud, because it's the sentence that shows you'd be trusted with this: an experiment with a real fault and no tested abort path is not a controlled experiment — it's an outage you scheduled yourself. If the team can't name the exact metric, threshold and window that trigger the abort, the experiment shouldn't touch production traffic yet, no matter how small the blast radius looks on paper.

🔴 10.4 · Chaos experiments versus game days

They ask: "What's the difference between a chaos experiment and a game day, and how would you get buy-in to start either one here?"

An automated chaos experiment mainly tests the system: does the failover code work, does the fallback path hold. A game day mainly tests the team: does the right person get paged, does the runbook match reality, does the on-call engineer reach for the correct dashboard, does the incident-commander role get filled without confusion. A game day is typically a single large fault injected once, with a facilitator who knows what's coming and a responding team that ideally doesn't — the closer to a surprise, the more honestly it tests instinct rather than a script someone just read. Explicit start and stop conditions, and a debrief immediately afterwards structured much like a blameless postmortem, because it surfaces the same kinds of gap: a missing dashboard, a stale runbook step, an alert that never fired.

On buy-in — which is usually the real question behind this one — production experiments need more than technical readiness. The on-call team has to know experiments are happening and how to distinguish one from a real incident; leadership has to agree to the acceptable risk, usually expressed as an error-budget allowance the experiment is permitted to spend; and there needs to be a channel where anyone watching can request a pause. The honest starting proposal is small: one hypothesis, in staging, about a failure mode that has already caused a real incident here — because "we've had this outage before and I want to prove the fix works" is a far easier sell than "I'd like to break production."

Domain 11 · Measuring & reporting reliability

☺ Like you're 10: The senior-flavoured domain — it's less about systems and more about whether the numbers you report actually change what anyone does.

Three questions. These skew toward roles where you'll talk to product and leadership rather than only to systems. Detail in Measuring & reporting reliability.

🟢 11.1 · What belongs on an SLO dashboard

They ask: "What should an SLO dashboard show?"

Three things, at a glance, per service: current SLI attainment against the target line; error budget remaining in the current window; and the burn-rate trend over recent days. Deliberately a small fixed set of panels, not a general-purpose metrics explorer — a dashboard that requires someone to build a query before they can answer "are we okay" has failed its actual job, which is to be glanceable during a standup, an incident, or a Friday-afternoon check.

Burn rate earns its place alongside current attainment because the same attainment snapshot can hide very different trajectories: an incident three weeks ago and a slow leak this morning can produce identical current numbers with completely different urgency. And the dashboard should be public within the organization — visible to the team, their manager, and product stakeholders — not buried in a personal workspace only SREs check.

🟡 11.2 · Vanity uptime

They ask: "A team reports 99.99% uptime and customers are still complaining. What's likely going on?"

Most likely they're reporting infrastructure uptime — the percentage of time VMs, containers or nodes were reachable and healthy — while the metric users actually feel is end-to-end request success. Those two numbers are not the same, and the infrastructure one is almost always the more flattering.

The gap hides real pain. Infrastructure can report four nines while a misconfigured load balancer, a saturated connection pool, a slow downstream dependency or a bad deploy causes a couple of percent of requests to fail or time out. The servers are up; the requests are not succeeding. That's what "vanity uptime" means — a number that's technically true, easy to hit, and disconnected from what anyone downstream cares about.

Name the specific blind spots to show it isn't a slogan: a server-side 2xx rate misses requests that never reached your servers at all (DNS failures, TLS handshake failures, client-side timeouts, CDN-layer errors); host uptime misses application-layer failures entirely (a healthy VM running a crash-looping process, or a service returning 500s while health checks pass); and an aggregate success rate across all endpoints can mask a critical path like checkout or login failing badly while high-volume, low-stakes endpoints drag the average back up.

The fix is the same discipline as choosing an SLI: measure at the boundary the user actually crosses — client-observed success rate, synthetic checks exercising the real user path, real-user-monitoring latency — not at the boundary that's most convenient to instrument. The test to offer: if a number in your report can be true while a user is actively having a bad time, it's the wrong number, no matter how good it looks.

🔴 11.3 · Making an error budget actually bite

They ask: "Your error budget is exhausted and a launch is committed for next week. Product leadership wants an exception. What do you do?"

Start by conceding the real point: exceptions are not automatically wrong. Sometimes a fix genuinely has to ship through a freeze, and a policy that can never bend is one that gets ignored entirely. What matters is that the exception is explicit, visible and logged, granted by whoever the written policy says can grant it — commonly a joint sign-off between an SRE lead and the product lead, never a unilateral call by whoever happens to be on call — with the specific risk being accepted written down.

Then name the failure mode you're guarding against, because that's the substance of the answer. A freeze policy waived every time it collides with an important launch isn't a policy, it's a suggestion, and everyone on the team learns that within one cycle. The first exception feels reasonable; so does the second and the third; by the tenth, the special case has quietly become the actual policy. So track the exception rate as its own metric alongside SLO attainment, and agree the acceptable ratio before you need it — a small number of overrides is healthy, overrides as the normal path is the policy failing.

Finally, offer the structural fix rather than relying on someone's willingness to say no under pressure: encode the check. A deploy pipeline can query remaining budget the same way it checks test results and refuse to proceed without an explicit, logged override. That doesn't remove exceptions — it changes the default from "ships unless someone objects" to "blocked unless someone visibly overrides," and every override becomes a reviewable data point.

Domain 12 · The SRE toolchain

☺ Like you're 10: Usually a filter, not a focus — they want to know you've touched the categories and don't mistake owning tools for having a practice.

Three questions. Answer by category rather than by brand loyalty; the tools change faster than the discipline does. Detail in The SRE toolchain.

🟢 12.1 · The categories

They ask: "Walk me through the tooling you'd expect an SRE team to have."

Answer by function, because that's how the toolbox is actually organized and it stays correct as vendors change.

One distinction worth getting right unprompted, because it's a common trip-up: OpenTelemetry is an instrumentation standard — SDKs, a collector, a wire protocol — not a backend. Traces still have to land somewhere, which is what Jaeger, Honeycomb or a commercial platform is for.

🟡 12.2 · Grafana versus a purpose-built SLO tool

They ask: "We already have Grafana. Why would we need a separate SLO tool?"

Grafana will plot whatever time series you point it at, but it has no built-in concept of an SLO target, a rolling compliance window, or a burn rate. Those have to be hand-built — and building them correctly is more error-prone than it looks, particularly multi-window burn-rate rules, which is exactly the kind of PromQL that's easy to get subtly wrong and hard to notice is wrong.

A purpose-built tool computes that math from a declarative SLO definition instead. Sloth generates Prometheus recording and alerting rules from a short SLO spec, so the burn-rate queries are derived rather than hand-written. Nobl9 and cloud-provider SLO products go further and centralize SLO definitions and error-budget reporting across teams, which matters once you have more SLOs than one person can hold in their head.

The honest caveat, which is worth volunteering: none of this is required. Plenty of teams run perfectly good SLO practice on hand-written recording rules. The question to ask is whether the number of SLOs has grown past the point where correctness by inspection is realistic — that's the trigger for a dedicated tool, not the existence of the tool itself.

🟡 12.3 · Tools versus practice

They ask: "Which observability stack do you prefer, and why?"

Answer the question — have a real preference and a real reason, because refusing to pick reads as evasion. Then reframe, because the reframe is what the question is often actually testing.

Every category has meaningful differences between its options — self-hosted versus managed, open source versus commercial, breadth versus depth — and choosing well is a legitimate engineering decision with real cost and operational consequences. But the choice is bounded. A team running a best-in-class stack with no defined SLOs, no error budget and a rotation nobody sustains is worse off than a team running spreadsheets and shell scripts with all three of those actually in place. Tooling accelerates a practice that already exists; it does not create one.

The forward-looking version, which is a good closing line: the tools in every one of these categories will change on a much faster timescale than the discipline does. An SLO, a blameless postmortem and a sustainable on-call rotation are the same ideas whether implemented in Prometheus or in whatever replaces it. So when evaluating a team's maturity — or your own — start with whether SLOs are defined and reviewed, whether postmortems are written and actioned, and whether on-call is actually sustainable, and treat the tool inventory as a distant second question.

Scenario & design questions

☺ Like you're 10: These have no single right answer on purpose — the interviewer is watching how you think, not waiting for a specific sentence.

Scenario questions are deliberately under-specified. There is no canned answer to memorize, and trying to produce one is the most common way to do badly at them: a candidate who launches straight into a solution has skipped the part being assessed. What is being assessed is method — whether you clarify before you design, state assumptions out loud, reason from user impact backwards, and name your own trade-offs before the interviewer has to.

🟡 S1 · "This service has no SLOs. Define them."

Clarify first: what does the service do, who consumes it, and is it in a synchronous user path or a background pipeline? Is there existing telemetry, and is there any record of what users have complained about? Does it have an SLA already, and if so, what does it promise?

Then work the method. Identify the critical user journey rather than the service boundary — what is the user actually trying to accomplish. Pick the SLI shape that matches how the service is consumed: a synchronous API wants availability and latency SLIs measured at the boundary the user crosses; a background pipeline's real promise is closer to "did the job finish by its deadline," which is a freshness or correctness SLI, because nobody is holding an HTTP connection open waiting for it. Keep the set small — two to five.

Derive the target from evidence outside the SLI. This is the step that separates a real SLO from a vanity one. Support-ticket volume, churn correlated with slow periods, and direct user research are all legitimate inputs; "what the service currently achieves" is not, because a target set that way is engineered never to be breached and therefore never to trigger the one behavior an SLO exists to trigger. Aim for the loosest target users won't notice you missing.

Choose the window deliberately and say why. A rolling window for the internal number that governs day-to-day behavior, because it has no known reset date to wait out; a calendar-aligned window is legitimate for external reporting that has to line up with a billing cycle. Twenty-eight days is a common internal choice specifically because it contains exactly four of every weekday, so a service with day-of-week seasonality isn't judged partly on which weekdays happened to fall in the window.

Attach a policy, or you've built a metric rather than a mechanism. Write down what a freeze restricts, what's exempt (security patches, fixes for the incident that burned the budget), who can grant an exception, and the exact condition that lifts the freeze — measured the same way it was measured going in.

Trade-off to name: a tight SLO on a service that doesn't need it burns engineering effort on reliability no user perceives; a loose one erodes trust before anyone gets a signal. Say which way you'd err for this service and why.

🔴 S2 · "p99 latency tripled at 3am. Nothing was deployed. Go."

The absence of a deploy is the interesting constraint — it removes the cheapest hypothesis and forces you to show a real search strategy.

Clarify: is it all endpoints or some? All regions or one? Did traffic change? Did the error rate move, or only latency? Is p50 up too, or only the tail?

That last question is the highest-value one you can ask, and asking it unprompted lands well. If p50 moved with p99, something got uniformly slower — a dependency, a resource ceiling, a config change. If only the tail moved, you're looking at something that affects a subset of requests: a hot shard, a cache-hit-rate collapse, a slow replica in the pool, garbage-collection pauses, or a queue that only some requests wait in.

Then work the layers, outside in: the golden signals for the service itself (did saturation move before latency — that's the leading indicator); its immediate dependencies' own latency and error rates; the resources beneath it via USE (connection pools, thread pools, disk, network); and anything that changed that wasn't a deploy — a feature flag flipped, a config push, a cron or batch job that starts at 3am, a scheduled backup, a certificate rotation, a dependency's own deploy, or a traffic pattern change from a different time zone waking up.

Say the 3am-specific hypotheses out loud, because the time is in the question for a reason: scheduled jobs, backup windows, log rotation, index rebuilds, and batch pipelines all cluster overnight, and any of them can create contention that only shows up in the tail. Also consider that at low overnight traffic, a fixed-cost background task consumes a much larger fraction of capacity than it does at midday, and caches that are warm at peak can go cold overnight.

Decide and act rather than only diagnose: convert the latency change into burn rate to decide urgency, and name the cheap mitigations available while you investigate — shed load, fail away from a suspect replica or zone, raise a timeout that's causing pile-up, degrade a non-critical dependency.

Close the loop: whatever the cause turns out to be, the postmortem question is why this class of change wasn't visible. A 3am latency regression with no deploy usually means there's a category of change — scheduled work, flags, dependency releases — that isn't in the change log the on-call engineer searches first.

🔴 S3 · "Design the reliability strategy for a new payments service."

Clarify: what does "payments" mean here — card authorization, internal ledger movement, payouts? What are the correctness requirements versus the availability requirements? What's the regulatory or audit context? What does the business lose per minute of downtime, and is anyone actually able to answer that?

Lead with the property that makes payments different: for most services availability is the headline concern, and for a payments path correctness usually outranks it. Double-charging a customer is worse than being briefly unable to charge them. That single sentence reframes the whole design and is the thing the interviewer is listening for.

From there the design writes itself in a defensible order. Idempotency everywhere on state-changing operations, with client-supplied keys, because retries are inevitable and a retry without a key is a duplicate charge. A consistency choice made explicitly rather than inherited: for the ledger, prefer consistency over availability during a partition — refusing to serve is recoverable, an inconsistent ledger may not be. Timeouts, bounded retries with full jitter, and circuit breakers on every synchronous hop to the card network or any third party, with the vendor's own reliability treated as a hard ceiling on yours — you cannot promise more availability than a hard dependency does, without an explicit fallback for the gap. Async where it's genuinely optional: receipts, notifications and analytics come off the synchronous critical path so they can't multiply into the user-facing SLO.

SLOs scoped to the journey, not the service, because the checkout path crosses several systems and each team's green dashboard can coexist with a red journey. Progressive delivery with an immutable-digest rollback, because the cost of a bad release here is measured in money rather than in patience. Reconciliation as a first-class reliability control — a scheduled comparison between your ledger and the provider's, alerting on divergence, because in payments the failure you most fear is the silent one.

Trade-off to name: every consistency and idempotency guarantee costs latency and complexity. Say where you'd spend it (the ledger write path) and where you wouldn't (a fraud-scoring enrichment that can degrade to a default).

🟡 S4 · "The team is drowning in pages. Fix it."

Clarify: how many pages per shift, what fraction overnight, how long has it been like this, and is anyone leaving because of it? Are the pages mostly one service, or spread?

Frame it as a bug, not a stamina problem — sustained high page volume is evidence the alerting is broken, and it costs something ordinary toil doesn't, which is trust. Then propose the structured audit rather than fixing the most-complained-about alert: export ninety days of firings, bucket each individual firing as actioned, auto-resolved, duplicate/correlated, non-actionable, or false positive, roll up to a per-rule actionability rate, and set the floor before you look at the numbers.

Map buckets to fixes: auto-resolved means either the threshold is too twitchy or the condition wants auto-remediation with a page only when the remediation fails; duplicates want grouping and inhibition, not threshold changes; non-actionable ones are the wrong severity and belong on a dashboard or a ticket queue; false positives are defects in the rule. And convert what survives to symptom-based, burn-rate-driven alerts so the remaining pages mean "budget is burning fast enough to matter" rather than "a number moved."

Do the immediate humane thing too, and say so: while the audit runs, temporarily route the worst-offending non-actionable rules off the pager rather than making people endure another three months of them for the sake of clean data.

Measure it worked: pages per shift, the overnight fraction, and per-rule actionability, tracked over the following quarter — not "the team says it feels better."

🔴 S5 · "We need to go from three nines to four. What changes?"

Clarify first, and slightly push back: why four? Is a customer contract driving it, is it a competitive claim, or is it an aspiration someone stated in a planning meeting? What does a minute of downtime actually cost this business? This pushback is part of the answer, not a dodge — the discipline's own position is that reliability beyond what users notice or the business case justifies is an unbudgeted subsidy from engineering time.

Make the arithmetic concrete: over a 30-day window, three nines allows about 43 minutes and four nines allows about 4.3. That's not a small tightening — it means a single 10-minute incident blows more than twice the entire month's budget.

Describe what actually changes, because it's qualitative, not just "more of the same." Moving from roughly three to roughly four nines typically requires genuine redundancy across failure domains rather than within one, automated failover that has actually been exercised rather than merely configured, load shedding and graceful degradation so overload degrades instead of collapsing, and enough discipline in rollout — canarying, staged deploys, immutable rollback targets — that a bad change is caught in minutes rather than becoming a long incident. Going further, toward five nines, generally means multi-region architecture, materially stricter change management, and a standing practice of proving the resilience works under real injected failure rather than assuming it.

Name the ceiling nobody controls: the network path to a typical mobile client is often itself well short of four nines, so past a point you're buying reliability the user structurally cannot perceive. Also check the composite: if the journey crosses four services, every hop needs to be around 99.975% for the journey to reach 99.99% — so "we'll make the backend four nines" may not deliver a four-nines experience at all.

Trade-off to name: each nine costs more per minute of downtime avoided than the last, because the cheap fixes get used up first. Offer the alternative that's often better: tighten the SLO only on the one critical user journey that justifies it, and leave the rest where they are.

🟡 S6 · "Take this on-call handover: one service is unowned. What do you do?"

Clarify: unowned in what sense — no team on the org chart, or a team that exists but doesn't respond? Is it in a user-facing path? What pages does it currently generate, and who's been silently absorbing them?

Stabilize before you organize. First, make sure the pages it generates actually reach someone with a documented escalation path, even if that's temporarily you — an alert routing into a void is worse than an alert nobody has tuned. Second, find out what depends on it and what it depends on, because an unowned service in a critical path is a much more urgent problem than an unowned batch job.

Then run it as a readiness problem rather than a staffing argument. Score it honestly against the review categories you'd apply to any new service — is there a request-based SLI, is there capacity evidence, are its dependencies enumerated and wrapped, has rollback been exercised, is ownership named, is there a documented SLO — and turn each failure into a remediation item with a named owner and a due date. That converts "nobody owns this" from a complaint into a tracked list, which is the only form in which it tends to get fixed.

Resist the two easy wrong answers, and say why you're resisting them: quietly absorbing it permanently means the organization never learns it has a gap, and the load grows silently until it's someone's whole week. Refusing to touch it at all means users pay for an org-chart problem. The defensible position is explicit and time-boxed — provisional support with a written remediation plan and a date, escalated if the date slips.

Behavioral questions, STAR-shaped

☺ Like you're 10: Same four-part shape every time — what was happening, what you were responsible for, what you actually did, and how it turned out. The story has to be yours.

STAR is Situation (brief context), Task (what you specifically owned), Action (what you did, in first person singular), Result (what changed, ideally with a number). Two rules make the difference. Keep Situation short — most candidates spend two thirds of the answer on setup. And say I, not we, in the Action: interviewers are trying to isolate your contribution, and a wall of "we" makes that impossible.

The skeletons below are shapes, not scripts. Each one shows what a strong answer contains; the content has to come from your own work, because a rehearsed story from someone else's incident collapses on the first follow-up question.

🟢 B1 · "Tell me about an incident you handled."

Situation — two sentences: what the service did, what broke, who was affected. Resist the urge to explain the architecture.

Task — your specific role. Were you the first responder, the incident commander, the person who found it? Say which, plainly.

Action — narrate your decisions in order, and include at least one judgment call with the reasoning attached: why you rolled back rather than forward-fixed, when you escalated and what triggered it, what you tried that didn't work. Including a dead end makes the story more credible, not less.

Result — quantify it: duration, users or requests affected, budget consumed, and what changed afterwards. Then add the sentence most candidates omit — what the postmortem action item was and whether it actually closed.

What they're scoring: whether you reason from user impact, whether you escalate on a clock rather than on ego, and whether your language is systemic. If you have no dramatic incident, use an ordinary one — a well-told 20-minute degradation beats a badly-told regional outage.

🟡 B2 · "Tell me about a time you caused an outage."

Answer it. Deflecting ("I haven't") reads as either inexperienced or dishonest, and this question is not a trap — it's a check on whether blamelessness is something you extend to yourself.

Situation and Task — brief, and own it in the first sentence without hedging.

Action — what you did once you realized: how fast you told someone, whether you mitigated before diagnosing, whether you asked for help. Speed of disclosure is a large part of what's being measured.

Result — impact, then the systemic finding. This is the pivot that makes the answer strong: what condition allowed a single action to have that blast radius, and what changed so it can't happen the same way again. A missing confirmation prompt, an absent dry-run mode, a command with no guardrail, a pipeline that didn't require review.

What they're scoring: whether you can hold "I made a mistake" and "the system let me" at the same time, without collapsing into either self-flagellation or blame-shifting. Both collapses are read as risk.

🟡 B3 · "Tell me about a time you pushed back on a launch."

Situation — what was about to ship and what specifically worried you. Be concrete: no rollback path, no load evidence, an unowned dependency, a burned budget.

Task — whether you had actual authority or only influence. Both are legitimate stories; be honest about which it was.

Action — how you made the case. The strongest version is evidence plus an alternative rather than a veto: not "this isn't safe," but "here's the specific risk, here's what it would cost if it fires, and here's a smaller version we could ship this week instead." Also say who you brought in and how you escalated.

Result — what happened, including if you lost. Losing a push-back and describing what you did next — logging the accepted risk, adding monitoring for the specific failure mode, agreeing a rollback trigger in advance — is often a better answer than winning, because it shows you can disagree and commit.

What they're scoring: whether you're a partner or a blocker. SREs who can only say no get routed around within two quarters.

🟡 B4 · "Tell me about something you automated."

Situation — the toil source, with its actual shape: how often, how long each time, who was doing it. Numbers here are cheap and land well.

Task — how you chose this one over the others competing for the same time. The good answer references frequency and growth, not annoyance.

Action — which rung of the ladder it started on and which it ended on, and why you stopped there. Explicitly deciding "rung 4 was enough because this needs a judgment call" is a stronger answer than reflexively reaching for full automation.

Result — hours reclaimed, pages eliminated, or errors avoided — and the honest maintenance cost, because automation that nobody maintains becomes a new toil source with a babysitter.

What they're scoring: prioritization, not cleverness. Anyone can write a script; choosing the right script is the job.

🟡 B5 · "Tell me about a disagreement with a product manager."

Situation — the concrete decision at issue: a launch date against a burned budget, feature work against reliability work, an SLO target neither side liked.

Task — what outcome you were responsible for, and what they were responsible for. Naming their legitimate incentive out loud is a strong move — it shows you understand the disagreement is structural rather than personal.

Action — how you converted an argument about priorities into a decision about numbers. This is where error budgets earn their reputation: "we're not arguing about whether reliability matters, we're looking at a budget that's 140% consumed" moves a conversation in a way opinion doesn't. Say what you conceded, too.

Result — the decision, and whether the working relationship survived it. The relationship half is genuinely part of the answer.

What they're scoring: whether you can disagree without making it a values fight. Framing every conflict as engineering-versus-product reads as someone who will generate escalations.

🟡 B6 · "Tell me about a bad on-call rotation."

Situation — what made it bad: volume, overnight concentration, unowned services, noisy alerts, or too few people in the rotation.

Task — whether you owned fixing it or were living in it. Both work.

Action — what you did with the data rather than the feeling. Tracked pages per shift, audited firings for actionability, pushed a toil source back to the owning team, changed the escalation policy, argued for compensation. If you mostly endured it, say what you'd do differently now — that's a legitimate and often more honest answer.

Result — the number that moved, and whether it stayed moved.

What they're scoring: whether you treat an unsustainable rotation as a system to fix or a badge to wear. Candidates who describe brutal on-call with pride are heard as a future retention problem.

🔴 B7 · "Tell me about a time you were wrong about a root cause."

Situation — an incident where the obvious cause turned out not to be the cause.

Task — what you believed and, importantly, what evidence you had for believing it. A wrong hypothesis with good reasoning behind it is a much better story than a guess.

Action — what changed your mind, and how quickly. Did you have a mechanism for testing the hypothesis, or did you commit to it until it failed? Did someone else's evidence move you?

Result — what the cause actually was, and — the part that makes this answer land — what you changed about how you investigate. Correlating deploys before logs, keeping a written hypothesis list during an incident, or asking "what would I expect to see if I'm wrong."

What they're scoring: intellectual honesty under time pressure, which is the single most useful trait in an incident and the hardest to fake.

🔴 B8 · "Tell me about improving reliability without more headcount."

Situation — the constraint, stated plainly. Fixed team, growing service, rising pages, or a reliability commitment made above your pay grade.

Task — what you were being asked to deliver.

Action — the leverage you found. The strong answers usually come from one of a few places: removing work rather than absorbing it (pushing toil back, retiring a service, deleting alerts), making one fix that closes a class of incidents rather than one incident, replacing human judgment with a gate the pipeline enforces, or changing what the team says yes to via a readiness bar. Name what you deliberately chose not to do — that's the real content of a constraint story.

Result — the metric that moved: pages per shift, budget consumed per month, time to restore, or toil percentage.

What they're scoring: whether you can prioritize under a real constraint, and whether your instinct under pressure is to work more hours or to remove work. Almost every SRE role has some version of this constraint, which is why this question is common at senior level.

Smart questions to ask them

☺ Like you're 10: The questions you ask are also answers — they show what you already know matters, and they're how you find out whether this job is the one you think it is.

Every interview ends with "any questions for us?" and it is not a formality. For an SRE role it's the single best opportunity to find out whether the team practices the discipline or has the title, and a well-chosen question demonstrates judgment more efficiently than another answer would. Pick three or four — not a list you read out — and follow up on whatever the answer reveals.

About the practice

• "Do your services have SLOs, and how were the targets chosen?" The follow-up is the real question: derived from user evidence, or set from what the service already achieved? The second is how a vanity SLO is born, and the answer tells you a lot.
• "Has an error budget ever actually stopped a launch here?" The single most diagnostic question on this page. A policy that has never once been invoked is decoration, and hearing "not yet, but…" is genuinely useful information rather than a dealbreaker — it tells you what the job's first year is.
• "What happens to a postmortem action item after the review meeting?" Listen for whether items become tracked tickets in the same system as feature work, and whether anyone checks the completion rate. "They go in the doc" means the loop isn't closed.
• "What's your production readiness bar for taking on a new service, and can it actually say no?" A review that can only recommend isn't a gate. Ask what happens when it fails something two weeks before a committed launch date.

About on-call and the day-to-day

• "How many actionable pages does a typical shift get, and what fraction are overnight?" Ask for the number, not the adjective. "It's pretty quiet" and "about two a week, rarely overnight" are very different answers.
• "What's the split between operational work and engineering work for this team right now — honestly?" The word "honestly" gives them permission to be real, and the answer to a 50%-rule question tells you whether you're joining an engineering team or a ticket queue.
• "Is on-call compensated, and how?" Entirely fair to ask, and how the question is received is itself informative.
• "What's the biggest source of toil the team hasn't got to yet?" This is a gift of a question — people enjoy answering it, and it tells you what your first quarter would actually look like.

About the role and the org

• "Is this team embedded with one product team, centralized across many services, or building a platform others self-serve?" These are genuinely different jobs with different daily work and different career shapes, and job adverts frequently don't distinguish them.
• "Who owns the reliability of a journey that crosses several teams' services?" If the answer is "each team owns theirs," they may not yet have hit the composite-SLO problem — which is either an opportunity or a warning depending on their scale.
• "What would you want the person in this role to have changed twelve months in?" Forces a specific answer and surfaces expectations that would otherwise appear in your first performance review.
• "What's the most recent incident you learned something from?" How readily someone answers, and whether they name a person or a system, tells you more about the culture than any values page.

The last week before the interview

☺ Like you're 10: Don't try to learn new things this week. Practice saying what you already know, out loud, until it comes out clean.

An honest plan, with an honest caveat: if the interview is in a week, this week is for consolidation and rehearsal, not for learning a domain from scratch. Cramming an unfamiliar topic produces answers that survive exactly one follow-up question, which is worse than saying "I haven't worked with that." Plan around what you already know and make it retrievable under pressure.

Where to go from here

☺ Like you're 10: This page tells you what gets asked; the rest of the course is where the actual answers live.

This page is deliberately a rehearsal surface, not a replacement for the material. Every answer above is a compressed version of a lesson that treats the same idea properly, and the compression is exactly what makes an answer collapse under a third follow-up question. If a question here felt thin when you said it out loud, the lesson linked beside it is where the depth is.

Three companions in this course are worth pairing with this page specifically. Flashcards is Remy's territory and the right tool for the warm-up tier — definitional recall drilled until it's reflex, which is precisely what a screen stage rewards. Self-Check is untimed active recall across every domain, which is a faster way to find your gaps than reading. And the worked incident case study is the closest thing here to a scenario question with the answer written out — read the timeline, stop before the postmortem section, and work out the root causes yourself before checking.

One last honest note. Nothing on this page is a substitute for having operated something real. The strongest answer to any question here is your own system, your own incident, your own number — and the purpose of the model answers is to show you what a good version of that answer contains, so you can find the equivalent in your own experience and say it clearly. Adapt, don't recite.