Practice & Reference · Flashcards
Flashcards
38 cards drawn from every module in this course, front to back. The front holds a question, the back holds the answer — click or tap a card to flip it, and search to jump straight to a topic.
☺ Explain it like I'm 10
Rereading a page about error budgets feels like studying because the words look familiar, but recognizing a fact and being able to produce it from memory on your own are two different skills — like humming along to a song versus singing it with no music playing. Flipping a card forces the harder skill: you have to attempt the answer before you're allowed to check it. A few cards most days beats cramming the whole deck once — the same facts take less effort to recall each time you come back to them.
Foundations
Who created Site Reliability Engineering at Google, roughly when, and how did he describe the resulting discipline in his own words?
Tap / Enter to flip →
Answer
Ben Treynor Sloss, around 2003, tasked with building a seven-person team to keep google.com running. He described SRE as "what happens when you ask a software engineer to design an operations function."
Foundations
What's the relationship between SRE and DevOps, and name one concrete mechanism SRE has that DevOps as a philosophy doesn't mandate.
Tap / Enter to flip →
Answer
"class SRE implements interface DevOps" — DevOps is a cultural philosophy (CALMS) with no mandated mechanism; SRE is one prescriptive implementation with concrete mechanisms such as SLOs and error budgets, a formal error-budget release freeze, or the 50% toil cap.
Foundations
What is the 50% rule, and is it a target or a ceiling?
Tap / Enter to flip →
Answer
An SRE team should spend at most 50% of its time on operational work (on-call, tickets, toil) and at least 50% on engineering. It's a ceiling — sustained overage is treated as a staffing or under-automation problem, not tolerated as normal.
Foundations
Define a Service Level Indicator precisely — what shape must it have?
Tap / Enter to flip →
Answer
A quantitative measurement expressed as a ratio of good events to valid events over a time window — e.g. "% of requests served under 300ms." Not a raw number like "average latency was 180ms": averages hide tail behavior and give nothing to grade pass/fail against.
Foundations
What makes a good SLI, and roughly how many should a typical service have?
Tap / Enter to flip →
Answer
Request-based — measured at the boundary the user crosses, not internal resource health — and directly tied to user-visible behavior: availability, latency, correctness, freshness. The SRE book's guidance is roughly two to five SLIs per service.
Foundations
What's an SLO, and what's "the right SLO" according to this course?
Tap / Enter to flip →
Answer
A target value for an SLI over a defined window — e.g. "99.9% of requests under 300ms, measured over a rolling 28-day window." The right SLO is the loosest one users won't notice you missing; reliability beyond that point is waste, not virtue.
Foundations
Why is an SLA typically set looser than the internal SLO? Give a common example gap.
Tap / Enter to flip →
Answer
So normal operational noise doesn't trigger a customer-facing penalty, and so missing the SLO while still meeting the SLA gives early warning before a contract is actually breached. A common pattern: 99.9% internal SLO vs. 99.5% external SLA.
Foundations
Give the error budget formula, and compute the allowed downtime for a 99.9% SLO over a 30-day window.
Tap / Enter to flip →
Answer
Error budget = (100% − SLO) × window. 30 days = 43,200 minutes; 0.1% of that = 43.2 minutes of allowed downtime.
Foundations
Name all six properties that must be true for work to count as "toil" under the SRE book's definition.
Tap / Enter to flip →
Answer
Manual, repetitive, automatable, tactical (interrupt-driven, not planned), devoid of enduring value, and O(n) — scales linearly with service growth.
Foundations
List the five rungs of the automation ladder, and what marks the top rung as different from the one below it.
Tap / Enter to flip →
Answer
Fully manual, documented runbook, partially automated (human-triggered script), fully automated (system fixes it, human notified), self-service/autonomic. The top rung removes the human from the loop entirely — no notified bystander at all — while rung 4 still has one.
Running Reliable Systems
Name Google's four golden signals.
Tap / Enter to flip →
Answer
Latency, traffic, errors, saturation.
Running Reliable Systems
Why should latency always be reported as a distribution with successful and failed requests split apart, rather than a single average?
Tap / Enter to flip →
Answer
An average hides tail behavior — a 150ms average can mask a p99 of 3 seconds hitting one request in a hundred. A fast failure (400 in 8ms) and a slow failure (500 after a 30-second timeout) are different problems that averaging together hides.
Running Reliable Systems
What's the core difference between symptom-based and cause-based alerting, and why does symptom-based scale better?
Tap / Enter to flip →
Answer
Symptom-based pages only on user-visible conditions (elevated latency, error rate) and is always actionable. Cause-based pages on internal conditions (e.g. database CPU at 85%) that may never become user-visible — a single incident can trip a dozen cause-based alerts at once, training on-call to skim and dismiss.
Running Reliable Systems
What's the functional difference between a dashboard and an alert?
Tap / Enter to flip →
Answer
A dashboard is for a human who's already investigating to dig through data. An alert/page is what pulls a human's attention toward a problem in the first place — including at 3am. Conflating the two causes either alert fatigue or a genuine symptom sitting unnoticed on a dashboard.
Running Reliable Systems
What rotation length do most engineering organizations converge on for on-call, and why?
Tap / Enter to flip →
Answer
One week — long enough to build real context on what's currently unstable, short enough that one bad week doesn't define someone's month. A day-long rotation forces excessive handoffs; a month-long one concentrates burnout risk.
Running Reliable Systems
During a live incident, why shouldn't the Incident Commander also be the one debugging the problem?
Tap / Enter to flip →
Answer
Coordinating the response — tracking who's doing what, deciding next steps, keeping the incident moving — is a full cognitive load on its own; an IC who's also staring at logs loses the coordination thread exactly when it matters most.
Running Reliable Systems
What separates a SEV1 from a SEV2 in a typical severity matrix?
Tap / Enter to flip →
Answer
SEV1 is a full outage or critical-path failure affecting most users — pages the primary immediately, 24/7. SEV2 is partial degradation or an SLO threshold crossed without full unavailability — typically pages only during business hours.
Running Reliable Systems
What's a rough industry ceiling for a healthy number of actionable pages per on-call shift, and what should a team do if it's sustained well above that?
Tap / Enter to flip →
Answer
Roughly two to three actionable pages per shift. Sustained volume well above that means the alerting is broken, not that the team needs to toughen up — the fix is tighter thresholds, deduplication, or automating away the repeat cause.
Running Reliable Systems
Why is blame counterproductive in a postmortem, as an information-gathering strategy rather than a matter of niceness?
Tap / Enter to flip →
Answer
When engineers expect blame, they hedge — omitting that they were tired, skipped a stale runbook step, or noticed the same near-miss weeks earlier. That's exactly the detail needed to find the systemic cause; punishing the person just teaches everyone else to hide the same flaw better.
Running Reliable Systems
In the five whys technique, what's the sign that a chain has gone wrong, and what should you do?
Tap / Enter to flip →
Answer
The chain has gone wrong if it terminates on an individual's action ("the engineer should have been more careful") instead of a systemic or process gap. Back up a step and ask why that person's action was able to cause an outage-scale failure at all.
Running Reliable Systems
What's the difference between a load test, a stress test, and a soak test?
Tap / Enter to flip →
Answer
A load test holds traffic at expected levels to confirm targets are met. A stress test pushes traffic past expected levels until something breaks, to find the real ceiling. A soak test holds elevated traffic for hours to surface slow failure modes like memory leaks or connection pool exhaustion.
Running Reliable Systems
What's the structural trade-off between vertical and horizontal scaling?
Tap / Enter to flip →
Answer
Vertical scaling (bigger instances) is operationally simple but has a hard ceiling, usually needs a restart, and doesn't improve availability. Horizontal scaling (more instances) has effectively no ceiling and improves availability, at the cost of needing a shardable or stateless workload and added coordination.
Engineering for Reliability
Name the three states of a circuit breaker and what triggers the transition between them.
Tap / Enter to flip →
Answer
Closed (calls pass through, failures counted over a sliding window), open (a threshold — commonly 50% failure rate — trips it; calls fail immediately for a cooldown, often 20-60 seconds), half-open (a small number of test requests decide whether it closes again or reopens).
Engineering for Reliability
What is "full jitter," and why does exponential backoff alone not solve the synchronized-retry problem?
Tap / Enter to flip →
Answer
Plain exponential backoff still has every client follow the identical schedule, so retries land in synchronized waves. Full jitter picks a random wait uniformly between zero and the current backoff ceiling — not "ceiling plus a little noise" — spreading retries out most effectively.
Engineering for Reliability
What does the bulkhead pattern do, and what's it named after?
Tap / Enter to flip →
Answer
It isolates resource pools — threads, connections — per dependency, so a slow downstream can only exhaust its own allotment, not resources every other dependency also needs. Named after a ship's watertight compartments, which let the hull take damage in one section without sinking the vessel.
Engineering for Reliability
What condition triggers graceful degradation, versus what condition triggers load shedding?
Tap / Enter to flip →
Answer
Graceful degradation triggers when one specific, non-critical dependency fails — the service serves a reduced response missing that dependency's contribution. Load shedding triggers on overall system overload, rejecting a fraction of requests outright (often a 503 with Retry-After) to protect latency for the rest.
Engineering for Reliability
What three components does progressive delivery combine into one mechanism, and what's each one limited to on its own?
Tap / Enter to flip →
Answer
Canary release (still human-watched alone), feature flags (a manual toggle alone), and automated rollback (no reliable known-good target alone). Combined, none of them needs a human in the loop for the common case.
Engineering for Reliability
A progressive delivery health-check gate typically evaluates two categories of signal — name them.
Tap / Enter to flip →
Answer
Golden-signal deltas (is the canary's error rate, latency, or saturation meaningfully worse than the live baseline) and SLO burn rate (is the canary cohort burning error budget faster than the window can sustain).
Engineering for Reliability
Why does automated rollback require a reproducible, immutably-addressed build rather than rebuilding from source?
Tap / Enter to flip →
Answer
Rebuilding on rollback can silently pull a different transitive dependency version, a repatched base image, or a different compiler flag — the "known-good" rebuild may not match what was actually good. Rollback should redeploy an exact digest (e.g. checkout-svc@sha256:...) captured before the rollout started.
Engineering for Reliability
What did Netflix's Chaos Monkey actually do, and why were randomness and lack of warning central to its design?
Tap / Enter to flip →
Answer
It randomly terminated production instances during business hours with no warning to on-call. The randomness was the point — a rare, announced termination lets engineers keep assuming instances live forever; a constant, unpredictable one forces that assumption to be fixed immediately because it's tested every few days.
Engineering for Reliability
What three parts does a well-formed chaos engineering hypothesis need?
Tap / Enter to flip →
Answer
The fault (what's being injected), the scope (which service, environment, percentage), and the expected steady-state (a specific, testable threshold that should hold — not just "things still work").
Engineering for Reliability
List the blast-radius progression for a chaos engineering program, smallest to largest scope.
Tap / Enter to flip →
Answer
Non-production/staging, then one production instance, then a small percentage of production traffic, then full production traffic — each step gated by a real, tested abort mechanism, widened only after the smaller scope has held.
Practice & Reference
What three things should an SLO dashboard show at a glance, and why does burn rate matter as much as current attainment?
Tap / Enter to flip →
Answer
Current SLI attainment against the target, error budget remaining in the window, and the burn-rate trend. Burn rate matters because it distinguishes a fast-moving incident about to breach the budget from a slow, manageable leak — the same attainment snapshot can hide very different trajectories.
Practice & Reference
What is "vanity uptime," and give an example of a metric that produces it.
Tap / Enter to flip →
Answer
Reporting a technically-true but flattering number disconnected from actual user experience — e.g. infrastructure/host uptime, which can read 99.99% while a saturated connection pool or bad deploy causes 2% of real requests to fail, because the servers are "up" even though requests aren't succeeding.
Practice & Reference
What concretely happens to the release process when a service's error budget hits zero, under an enforced error budget policy?
Tap / Enter to flip →
Answer
New feature releases pause and the team's priority shifts to reliability work — fixing the burn's root cause, hardening the weakest dependency — until the budget recovers as the window rolls forward or a fix demonstrably stops the bleed.
Practice & Reference
What's the purpose of a quarterly reliability review beyond just reporting the SLO numbers?
Tap / Enter to flip →
Answer
It forces a recurring, structured look at SLO performance with the people who can act on it, and is the forum for renegotiating targets that traffic or dependencies have outgrown — keeping the SLO honest instead of letting it drift silently or get quietly ignored.
Practice & Reference
What does a distributed trace tell you that a metrics dashboard alone can't?
Tap / Enter to flip →
Answer
Metrics tell you that latency went up; a trace tells you where in a multi-service call chain the time actually went — a tree of timed spans, one per service hop, stitched together by a propagated request ID.
Practice & Reference
Why would a team with a full, well-chosen toolchain but no defined SLOs still be considered immature from an SRE standpoint?
Tap / Enter to flip →
Answer
Tools support the practice, they don't substitute for it — reliability comes from the discipline of defining and tracking SLOs, running blameless postmortems, and sustaining on-call, not from which software sits underneath. A well-tooled team without those practices is still just firefighting with better dashboards.