Practice & Reference · Glossary
Glossary
Every term this course uses without re-explaining it inline, gathered into one alphabetical, searchable reference — SLIs, SLOs, and error budgets, on-call and incident response, capacity and resilience patterns, progressive delivery, and chaos engineering.
☺ Explain it like I'm 10
If a page uses a term you haven't seen defined yet — SLO, blast radius, canary release — it's here in a sentence or two, not buried in a paragraph you've already scrolled past. Search the box below or scan the full list; nothing here assumes you already speak the vocabulary.
AlertingThe subsystem that evaluates metrics or logs against a condition and notifies a human or automation when it's met. Effective alerting fires on symptoms tied to user impact — an SLO burn rate, a failed request rate — rather than on every raw signal that crosses a static threshold.
Availability (the "nines")The fraction of time or requests a service behaves correctly, usually written as a percentage. "Three nines" (99.9%) allows roughly 43 minutes of downtime a month; "four nines" (99.99%) allows roughly 4.3 minutes. Each additional nine cuts the allowed downtime, and the error budget behind it, by about 10x.
Blameless cultureAn incident-review norm that treats failures as gaps in the system and process rather than individual negligence, on the premise that engineers act reasonably given the information they had at the time. It matters because accurate root-cause information only surfaces when people aren't protecting themselves from blame.
Blast radiusThe scope of users, requests, or infrastructure that a given failure or change can affect. Shrinking blast radius — through canaries, cells, sharding, or bulkheads — is a primary lever for making incidents cheaper, independent of how often they happen.
Bulkhead patternPartitioning resources — thread pools, connection pools, compute — per dependency or tenant so that one overloaded or failing component can't exhaust resources the rest of the system needs. Named for the watertight compartments in a ship's hull that keep one breach from sinking the whole vessel.
Burn rateThe speed at which a service is consuming its error budget, expressed as a multiple of the sustainable rate. A burn rate of 10x means the budget for the full window would be exhausted in a tenth of the time; burn-rate alerts page on this ratio instead of on raw error counts.
Canary releaseDeploying a new version to a small slice of traffic or instances first, comparing its error rate and latency against the existing baseline, and only proceeding to a full rollout if the comparison stays clean.
Capacity planningForecasting future resource demand — compute, storage, network — from growth trends and provisioning ahead of need, balancing the cost of held headroom against the risk and cost of running out.
Chaos engineeringThe discipline of deliberately injecting failure into a system in a controlled, observed way to verify it degrades and recovers as designed, instead of assuming resilience because the architecture diagram says so.
Circuit breakerA client-side guard that stops sending requests to a dependency once its failure rate crosses a threshold, failing fast locally instead of piling up latency and retries against a service that's already struggling. It periodically lets a trial request through to detect recovery and close again.
Disaster recoveryThe plans, backups, and failover procedures for restoring service after a large-scale failure — region loss, data corruption, catastrophic outage — that exceeds what normal incident response is built to handle.
Distributed tracingInstrumentation that follows a single request as it crosses process and service boundaries, recording each hop (a span) so latency and errors can be attributed to the specific service in the call chain that caused them, rather than guessed at from aggregate metrics.
Error budgetThe amount of unreliability a service is allowed in a period while still meeting its SLO, computed as (1 − SLO) × total volume. Spending it authorizes risk-taking — releases, migrations, experiments; exhausting it triggers a release freeze or a shift to reliability-first work.
Escalation policyThe predefined chain of who gets paged next, and after how long, if the primary on-call doesn't acknowledge or resolve an alert.
FailoverAutomatically or manually redirecting traffic from a failed primary component to a healthy standby or replica.
Feature flagA runtime switch that turns a code path on or off without a deploy, used to decouple shipping code from exposing it to users and to kill a misbehaving feature instantly instead of waiting on a rollback.
Five whysA root-cause technique that repeatedly asks "why did that happen" of the answer to the previous question, until the chain reaches a systemic cause instead of stopping at the first symptom encountered.
Follow-the-sunAn on-call model that hands the primary rotation between teams in different time zones so no single team is paged overnight, at the cost of coordination overhead across regions and handoffs.
The four golden signalsGoogle's four metrics for monitoring a user-facing system: latency, traffic, errors, and saturation. Used as the minimum viable dashboard for judging whether a service is currently healthy.
Game dayA scheduled, announced exercise where a team runs a failure scenario — often via chaos-engineering tooling — against production or a production-like environment to rehearse detection and response before a real incident forces the rehearsal.
Graceful degradationDesigning a system to shed lower-priority functionality under stress while preserving its core function, rather than failing completely once any part of it is overloaded.
Horizontal scalingAdding more instances of a service behind a load balancer to absorb more load, as opposed to making a single instance bigger.
Hypothesis-driven testingThe chaos-engineering practice of stating an explicit, falsifiable prediction about how the system will behave under an injected fault, then running the experiment to confirm or refute it — the difference between an experiment and just breaking things.
Incident commanderThe person who owns coordination during an active incident — driving the response, assigning workstreams, communicating status — without necessarily being the one debugging the fix themselves.
JitterRandomized variation added to a retry delay so that many clients retrying after the same failure don't all retry at exactly the same moment and recreate the overload they were backing off from.
Load sheddingDeliberately rejecting a portion of incoming requests — usually the lowest-priority ones — when a system is over capacity, so the requests that do get served stay fast instead of letting everything queue and eventually time out.
Load testingDriving a system with synthetic traffic at or beyond expected peak to find its breaking point and validate capacity assumptions before real users find it for you.
MTBFMean time between failures: the average interval between successive failures of a system, used as a measure of how often it breaks between incidents.
MTTDMean time to detect: the average time from when a problem starts to when monitoring or a human notices it.
MTTRMean time to repair (also seen as resolve or recover): the average time from detecting an incident to restoring service. Often a more tractable target to improve than reducing how often failures happen in the first place.
N+1 redundancyProvisioning one more unit of capacity than the minimum needed to serve current load, so the system can lose any single unit and keep operating without degradation.
ObservabilityThe property of a system that lets engineers infer its internal state from its external outputs — metrics, logs, traces — specifically the ability to answer questions about failure modes nobody anticipated in advance, not just the ones a dashboard was built for.
On-callThe rotation of engineers responsible for responding to production alerts outside normal working hours, typically for a defined shift with a defined escalation path if they don't respond in time.
Percentile latency (p50/p99)Latency measured at a given percentile of requests rather than as an average. p50 is the median request; p99 is the request slower than 99% of the rest. The tail percentiles, not the average, expose the experience of a service's worst-served users.
PostmortemThe written record produced after an incident that reconstructs the timeline, root cause, impact, and follow-up actions — the mechanism that converts an outage into a durable process improvement instead of a story people just remember for a while.
Progressive deliveryRolling out a change gradually and observing it at each stage — canary, then a percentage-based ramp, then full — with automated or manual gates that can halt or roll back based on live signals rather than a single all-or-nothing deploy.
RED methodA monitoring approach for request-driven services: track Rate, Errors, and Duration for every service. Complements the more infrastructure-focused USE method.
RedundancyMaintaining more than one instance of a component so the failure of any single one doesn't take down the function it provides.
Reliability reviewA periodic, structured assessment of a service's SLOs, error-budget trend, toil load, and open risks — distinct from a postmortem, which is scoped to a single incident rather than the service's overall health.
ReplicationKeeping synchronized copies of data across multiple nodes or locations so a copy remains available if any one node is lost.
Retry with exponential backoffA retry strategy where each successive attempt waits longer than the last, typically doubling, so a client backs off a struggling dependency instead of hammering it at a constant rate right as it's trying to recover.
RollbackReverting a deployed change to the previous known-good version — usually the fastest way to end an incident that was caused by a recent release.
RPORecovery point objective: the maximum acceptable amount of data loss, measured as a span of time, between the last good backup or replica and a failure.
RTORecovery time objective: the maximum acceptable time to restore service after a disaster-level failure.
RunbookA documented, step-by-step procedure for diagnosing and resolving a specific known failure mode, written so an on-call engineer unfamiliar with the internals can follow it correctly under pressure at 3 a.m.
Single point of failureAny component whose failure alone is sufficient to take down the whole system, because no redundant path exists around it.
SLAService level agreement: an externally facing, often contractual, commitment about service performance, typically carrying financial or credit penalties for missing it. Usually set looser than the internal SLO so there's margin between "we're worried" and "we're in breach."
SLIService level indicator: an actual measured metric of a service's behavior, such as the proportion of requests served within 300ms over a given window.
SLOService level objective: an internal target for an SLI over a period — for example, 99.9% of requests succeed within 300ms over a rolling 28 days — that the error budget and release decisions are measured against.
SRE (site reliability engineering)The discipline, pioneered at Google, of applying software-engineering practices to operations problems — treating reliability as a measurable product feature with an explicit, spendable budget rather than an unbounded goal to maximize.
TimeoutA configured limit on how long a caller will wait for a response before giving up, preventing a slow dependency from holding resources indefinitely on the caller's side.
ToilOperational work that is manual, repetitive, automatable, tactical, and grows linearly with service size rather than producing lasting engineering value. SRE teams track it explicitly because unchecked toil crowds out the work that actually improves reliability.
Toil budgetA cap, often expressed as a share of an SRE team's time — commonly cited around 50% — above which toil is treated as a problem to automate away rather than accepted as normal workload.
USE methodA monitoring approach for resources — CPU, disk, network: check Utilization, Saturation, and Errors for each one. Complements the more request-focused RED method.
Vertical scalingIncreasing the resources — CPU, memory — of a single instance to handle more load, as opposed to adding more instances.