Reliability, SLOs & Incident Management

When you run a platform, you are not shipping a feature that a few people use for an hour a day — you are running the ground that every other team stands on. If the platform wobbles, every service on it wobbles at once. So reliability stops being a vague virtue (“let’s try not to break things”) and becomes an engineering discipline with numbers, budgets, and rituals. This page goes well past the exam blueprint into how senior platform teams actually do it: how to measure reliability with SLIs and SLOs, how to spend an error budget on purpose, how to design services that bend instead of snapping, how to run a humane on-call and a blameless postmortem, and how to break things deliberately (chaos engineering) so they don’t break by surprise. The goal at the end is a platform that mostly heals itself.

SRE for platform teams

☺ Like you’re 10: Because everyone stands on the platform, keeping it steady is a real job with real numbers — not just “try hard and hope.”

Site Reliability Engineering (SRE) is the practice, born at Google, of treating reliability as a software problem rather than a heroics problem. Instead of a rotating cast of tired humans manually holding a system together, you write code, define measurable targets, and let data decide what to work on next. For a platform team the fit is exact, because the platform has an unusual property: it is not a production system, it is production for everyone else.

The platform is production

A normal service team owns one blast radius — their own users. A platform team owns a shared blast radius. When your ingress controller, your CI runners, your GitOps reconciler, or your secrets injector has a bad five minutes, it is not one product that suffers — it is potentially every product at once, plus every developer who was mid-deploy. This changes how you reason about risk in two ways. First, the correlation of failures is high: a single shared dependency going down looks, to your customers, like the whole company going down. Second, your “customers” are internal developers (Dot and her teammates) who will happily route around a flaky platform — back to snowflake scripts and tickets — the moment it stops being trustworthy. Reliability is therefore not a nice-to-have; it is the thing that keeps your platform from being abandoned.

Reliability is a feature

The most useful mindset shift is to treat reliability as a feature you build and ship, with a backlog, an owner, and a definition of done — not as a background hum that “ops handles.” That means reliability work competes for prioritisation like any other feature, and it means you can be deliberate about how much of it to build. Here is the counter-intuitive part senior engineers internalise: 100% is the wrong target. Chasing the last fraction of a nine costs exponentially more and, worse, your users can’t even perceive it because their own network, browser, and phone are already less reliable than your service. The right target is “reliable enough that no one is choosing a competitor or filing a ticket because of downtime” — and then you stop, because every nine beyond that is money and velocity you could have spent shipping.

Error budgets align speed and stability

The genius of SRE is the error budget: the mathematical inverse of your reliability target. If you promise 99.9% reliability, you are explicitly saying you may be unreliable 0.1% of the time — and that 0.1% is a budget you are allowed to spend. It dissolves the oldest fight in software. Developers want to ship fast (which risks breakage); operators want stability (which resists change). The error budget makes them the same team: while budget remains, the platform team welcomes risky changes, fast rollouts, and experiments, because there is room to absorb a mistake. When the budget is exhausted, a pre-agreed policy kicks in — freeze risky changes, redirect effort to hardening — until reliability is earned back. Nobody argues about feelings; they read the number. To read that number you need real telemetry, which is why this page and Observability are two halves of one whole: observability is how you see reliability, SLOs are how you decide about it.

SLIs, SLOs & SLAs in depth

☺ Like you’re 10: First you pick a way to measure “is it working?” (an SLI). Then you pick a promise about that number (an SLO). An SLA is the same promise but with money attached if you miss.

These three initials get muddled constantly, so let’s pin them down precisely, then dig into the hard part — choosing the measurement and defending the target.

SLI vs SLO vs SLA

An SLI (Service Level Indicator) is a carefully-defined measurement of some aspect of service health, almost always expressed as a ratio of good events to valid events — e.g. “the proportion of HTTP requests served in under 300 ms with a non-5xx status.” An SLO (Service Level Objective) is a target for that SLI over a window — e.g. “99.9% of requests, measured over a rolling 30 days.” An SLA (Service Level Agreement) is a contract with an external customer that ties an objective to a consequence, usually a refund or service credit. The rule of thumb: your internal SLO should be stricter than any SLA you sign, so you get warned and act long before you owe anyone money.

Choosing good SLIs

The single most common mistake is measuring what is easy (CPU, memory, disk) instead of what the user feels. Your users do not experience CPU utilisation; they experience “did my request succeed, and was it fast?” Good SLIs are therefore user-journey-centric and drawn from a small, well-understood menu. Google’s framing pairs each SLI with the events it counts, so the ratio is unambiguous.

☺ Like you’re 10: Measure the thing a person actually notices — “did it work and was it quick?” — not how warm the computer is.

Two subtleties separate a novice SLI from an expert one. First, measure at the right vantage point: metrics scraped at the server miss failures that never reached the server (a broken load balancer, DNS, or TLS handshake), so the truest availability SLI is measured as close to the user as you can get — the load balancer, the mesh sidecar, or synthetic probes. Second, use percentiles, not averages, for latency: an average hides the long tail, and it is the p99 request — the slowest 1% — that makes users rage-quit. An SLO like “p99 latency under 500 ms” expresses this as a threshold-count SLI.

Setting realistic SLOs & the error-budget policy

An SLO target should be derived from evidence, not vibes. Start by measuring your current performance for a few weeks — if you are already delivering 99.92%, promising 99.99% is a fantasy that will keep you permanently in violation and destroy the signal. Pick a number your history says is achievable and that your users actually need, then translate it into a budget so the stakes are concrete. The arithmetic is worth memorising because it reframes the conversation from “nines” to “minutes you can afford to be down.”

The SLO is only half the mechanism. The other half is the error-budget policy — a document, agreed before any incident, that states exactly what happens when the budget runs low or out. A typical policy: while > 50% of the budget remains, ship freely; below 25%, require extra review on risky changes; at 0%, freeze all non-reliability changes and redirect the team to hardening until the trailing-window SLO recovers. The power is that it is decided calmly in advance, so in the heat of a bad month nobody has to win an argument about slowing down — the policy already did.

Multi-window, multi-burn-rate alerting

Now the deep part. Naïve SLO alerting (“page me if the error ratio over 5 minutes exceeds 0.1%”) is a disaster: it is far too noisy (a brief blip pages you at 3am for nothing) and far too slow to matter (a slow leak never trips the short window). The state of the art is multi-window, multi-burn-rate alerting, and the key concept is burn rate: how fast you are consuming the error budget relative to “exactly on target.” A burn rate of 1 means you will spend precisely your whole budget over the SLO window and finish at target. A burn rate of 14.4 means you are burning 14.4× too fast — at that pace you would exhaust a 30-day budget in about 2 days, or 2% of it in a single hour.

You run several burn-rate alerts at once, each pairing a fast window (to react quickly) with a slow window (to suppress false alarms). Requiring both windows to be hot means a 30-second blip won’t page you, but a genuine sustained fire will — fast. The canonical set for a 99.9% SLO:

In Prometheus this is a recording rule that pre-computes the error ratio over each window, plus Alertmanager rules that fire when a fast and slow window both exceed the burn-rate threshold. Pre-computing keeps the alert query cheap and instant:

groups:
- name: checkout-slo
  rules:
  # 1) Pre-compute the bad-event ratio over several windows.
  - record: job:slo_errors:ratio_rate5m
    expr: sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
        / sum(rate(http_requests_total{job="checkout"}[5m]))
  - record: job:slo_errors:ratio_rate1h
    expr: sum(rate(http_requests_total{job="checkout",code=~"5.."}[1h]))
        / sum(rate(http_requests_total{job="checkout"}[1h]))

  # 2) Fast burn: 14.4x the 0.1% budget, confirmed by BOTH windows => page.
  - alert: CheckoutErrorBudgetFastBurn
    expr: |
      job:slo_errors:ratio_rate5m > (14.4 * 0.001)
      and
      job:slo_errors:ratio_rate1h > (14.4 * 0.001)
    for: 2m
    labels: { severity: page }
    annotations:
      summary: "Checkout is burning its error budget 14.4x too fast"
      runbook: "https://runbooks.acme.dev/checkout/error-budget"

Reliability engineering patterns

☺ Like you’re 10: Build things that bend instead of snap — wait a moment and retry, give up if something takes too long, and stop asking a friend for help once they’ve clearly fainted.

SLOs tell you whether you’re reliable; these patterns are how you become reliable. They are mostly about surviving the failure of the things you depend on — because in a distributed platform, something downstream is always, at any given moment, a little bit broken.

Health probes: liveness, readiness, startup

Kubernetes gives every container three distinct health checks, and confusing them is a classic source of self-inflicted outages. A readiness probe answers “should this Pod receive traffic right now?” — when it fails, the Pod is pulled from Service endpoints but not restarted, which is exactly what you want when a dependency is briefly unavailable. A liveness probe answers “is this container wedged and beyond saving?” — when it fails, the kubelet kills and restarts the container. A startup probe answers “has the app finished booting yet?” — it holds off the other two probes during a slow start so a JVM that takes 90 seconds to warm up isn’t murdered at second 30.

Timeouts, retries, backoff & jitter

Every network call must have a timeout. A call with no timeout is a slow-motion outage waiting to happen: when the callee hangs, the caller’s threads or connections pile up waiting forever, exhaust the pool, and the caller falls over too — the failure propagates upstream. Retries handle transient blips, but naïve retries are dangerous. Retrying immediately, in a tight loop, from thousands of clients at the exact moment a service is struggling is how you turn a brief degradation into a full collapse — a retry storm. The fix is exponential backoff with jitter: wait longer after each failure (1s, 2s, 4s…) and add a random component so a thousand clients don’t all retry on the same synchronised tick (a thundering herd). Cap the total attempts, and — critically — only retry operations that are idempotent, or you may charge a customer’s card three times.

Circuit breakers, bulkheads & backpressure

When a dependency isn’t blipping but is genuinely down, retries are pointless cruelty — you’re hammering a corpse and stealing your own resources while you do it. A circuit breaker watches the recent failure rate to a dependency and, once it crosses a threshold, trips: it stops sending calls and fails fast (returning a cached value or a clear error) instead of waiting on timeouts. After a cool-off it lets a trickle of “probe” requests through (half-open) and, if they succeed, closes again. This gives the sick dependency room to recover instead of being kept pinned under load.

A bulkhead (named after a ship’s watertight compartments) isolates resources so one failure can’t sink the whole vessel: give each downstream dependency its own connection pool or thread pool, so a hang talking to the recommendations service can’t consume every thread and starve the unrelated checkout path. Backpressure is the discipline of signalling “I’m full, slow down” upstream — bounded queues that reject when full, rate limits, and concurrency caps — rather than silently accepting unlimited work until you OOM. In a service mesh you get circuit breaking, outlier detection, and pool limits declaratively; see how the mesh wires this in Networking.

Graceful degradation & load shedding

The most resilient systems have a plan for “I can’t do everything, so what’s the most valuable thing I can still do?” Graceful degradation means shedding non-essential features to keep the core alive: an e-commerce page that can’t reach the recommendations service should still render the product and the buy button — just without the “you might also like” strip. Load shedding is deliberately dropping some requests (ideally the least important, or the ones already past their deadline) when you’re over capacity, because serving 80% of users perfectly beats serving 100% of users a spinning error as the whole thing melts down. Both are conscious trade-offs you design in advance, and both depend on having decided which parts of your service are load-bearing and which are garnish.

Designing for failure

☺ Like you’re 10: Don’t keep all your eggs in one basket, and never let a routine chore — like swapping a broken shelf — knock all the eggs off at once.

Resilience patterns keep a single service standing; this section keeps the fleet standing when the infrastructure underneath it fails — and it always eventually does. The mantra is “assume everything fails, and make sure no single failure takes down more than a slice.” Much of this is about spreading your workload across failure domains and making routine operations (upgrades, drains) safe.

Spreading across failure domains

A failure domain is a boundary within which a single fault takes out everything: a node, a rack, an availability zone (AZ), a region. The first rule is to run every important workload with multiple replicas spread across at least three AZs, so losing a whole zone — a genuinely routine cloud event — costs you a third of your capacity, not all of it. Kubernetes gives you two tools. Pod anti-affinity asks the scheduler to keep replicas apart (“don’t co-locate two of these on one node”). The newer, cheaper topology spread constraints express “spread these Pods evenly across zones, and only tolerate a skew of 1,” which is exactly what you want for zonal resilience. Getting this right is deeply tied to how the scheduler places work — the mechanics live in Scaling & Scheduling.

PodDisruptionBudgets & safe drains

Here is a failure mode people discover the hard way. Kubernetes distinguishes involuntary disruptions (a node crashes) from voluntary ones (you drain a node to upgrade it). During a voluntary disruption, nothing by default stops the system from evicting every replica of your service at once — so a routine node upgrade can briefly take your whole app to zero. A PodDisruptionBudget (PDB) is the guardrail: it tells the eviction API “never voluntarily take me below N available (or above M unavailable) Pods.” With a PDB in place, kubectl drain and the cluster autoscaler and node auto-upgrades all respect your minimum, evicting Pods gradually and waiting for replacements to become Ready. This is what makes rolling node upgrades boring instead of an outage.

apiVersion: apps/v1
kind: Deployment
metadata: { name: checkout }
spec:
  replicas: 6
  template:
    spec:
      terminationGracePeriodSeconds: 45          # time to drain in-flight work
      topologySpreadConstraints:                 # even across zones
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector: { matchLabels: { app: checkout } }
      containers:
      - name: app
        image: registry.acme.dev/checkout:1.9.2
        readinessProbe:                          # pulled from LB when unhealthy
          httpGet: { path: /readyz, port: 8080 }
          periodSeconds: 5
        livenessProbe:                           # local check only — no deps!
          httpGet: { path: /livez, port: 8080 }
          periodSeconds: 10
          failureThreshold: 3
        lifecycle:
          preStop:                               # stop taking new traffic, then bleed out
            exec: { command: ["sh","-c","sleep 10"] }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: checkout-pdb }
spec:
  minAvailable: 4                                # never drop below 4 of 6 during drains
  selector: { matchLabels: { app: checkout } }

Graceful shutdown & rolling upgrades

Even with a PDB, a Pod that is being evicted must shut down cleanly, or users mid-request get abrupt resets. The correct dance is subtle and worth knowing: when a Pod is terminating, Kubernetes simultaneously sends SIGTERM and begins removing it from Service endpoints — but endpoint removal is eventually-consistent, so for a moment traffic still arrives at a Pod that’s already shutting down. The fix is a preStop hook that sleeps briefly (letting endpoint removal propagate) before the app stops accepting connections, and an application that, on SIGTERM, stops taking new work but finishes in-flight requests before exiting — all within terminationGracePeriodSeconds. Pair this with a progressive rollout (rolling update with maxUnavailable: 0, or a canary) and node upgrades and deploys become non-events. This connects directly to the safe-rollout machinery in CI/CD & Progressive Delivery.

Incident management

☺ Like you’re 10: When something breaks, you need one calm person in charge, clear job labels, and a way to tell everyone what’s happening — not five people all yanking the same rope.

No matter how well you engineer, incidents happen. What separates a mature platform team is not the absence of incidents but the calm competence of the response. Incident management is the set of roles, rituals, and tools that turn a chaotic scramble into a rehearsed play.

On-call rotations & humane paging

Someone has to be reachable when the pager fires, and that responsibility rotates through the team on a schedule managed by tools like PagerDuty or Opsgenie, which handle escalation (if the primary doesn’t acknowledge in N minutes, page the secondary, then the manager). The humane part is a design goal, not a nicety: burned-out responders make worse decisions and quit. That means a rotation large enough that on-call comes around infrequently, a hard cap on pages per shift (if it’s exceeded, that’s a reliability bug to fix, not a person to push harder), compensation or time-off-in-lieu for nights, and — the golden rule — every page must be both urgent and actionable. Non-urgent alerts go to tickets or dashboards; only “a human must act now to protect users” earns the right to wake someone.

Severity levels

Not every incident deserves a war room at 3am. A severity scale (SEV) lets everyone instantly agree on how big this is and how hard to pull the alarm, so the response is proportional. Publishing the scale in advance removes the in-the-moment debate about whether something “counts.”

The incident-commander model

The single most valuable import from emergency services is the Incident Commander (IC) role, adapted from firefighting’s Incident Command System. During a significant incident, one person becomes the IC — and crucially, the IC does not fix the problem. The IC coordinates: they hold the current state of the world, decide what happens next, delegate investigation and mitigation to specific named people, and keep the response from devolving into six engineers silently theorising in parallel. Supporting roles peel off as needed: a Communications Lead owns external and stakeholder updates so the IC isn’t interrupted, and an Operations/Ops Lead or scribe drives the technical work and keeps a timestamped log. The magic of the model is that it makes coordination someone’s explicit job, which is precisely the thing that collapses under pressure when it belongs to no one.

Comms & status pages

During an incident, communication is half the job. Internally, everyone converges on one channel (a dedicated Slack/Teams room per incident) and one document — no side conversations, so the timeline is coherent for the postmortem. Externally, a status page tells customers you know and you’re on it; a slightly-late honest update beats silence every time, because silence is what erodes trust and floods your support queue. The two reliability metrics that anchor all of this are MTTA (mean time to acknowledge — how fast a human engages) and MTTR (mean time to recovery — how fast users stop hurting). Notice the priority order that falls out: mitigate before you diagnose. Roll back, fail over, or drain the bad node to stop customer pain first; understand the root cause after. Curiosity is for the postmortem, not the outage.

Learning from incidents

☺ Like you’re 10: After something breaks, don’t look for someone to blame — look for the reason the system let it happen, and fix that so it can’t happen the same way again.

An incident you don’t learn from is an incident you’ll have again. The postmortem is where a platform team converts pain into durable improvement — but only if the culture around it is right.

Blameless postmortems

The foundational principle is blamelessness. When an engineer runs a command that takes down the cluster, the mature question is not “how could you be so careless?” but “how did a system permit a single command to do that, and why did it look reasonable at the time?” This isn’t about being soft — it’s cold pragmatism. In a blameful culture, people hide mistakes, and hidden mistakes can’t be fixed; you lose the very information you need to get more reliable. Blamelessness assumes everyone acted rationally given what they knew, and hunts for the systemic gaps — missing guardrails, confusing tooling, absent confirmations — that let a good person make a bad outcome. A blameless postmortem for a significant incident should be expected, not optional, and it should be published widely so the whole org learns.

Root cause vs contributing factors

Beware the phrase “the root cause.” Real incidents in complex systems almost never have a single cause — they have a chain of contributing factors that had to line up, like holes in slices of Swiss cheese lining up to let a hazard through. A deploy bug alone was survivable; it became an outage because the canary analysis was misconfigured and the alert was too slow and the rollback runbook was stale. Techniques like the “Five Whys” are useful for walking backward through that chain, but treat them as a way to surface multiple contributing factors, not to crown one villain. The richest lessons usually live in the factors that turned a small fault into a big one — the missing guardrails — not in the initial trigger.

Action items & error-budget-driven prioritisation

A postmortem with no follow-through is theatre. Every one should end with concrete action items, each with an owner and a due date, tracked in the same backlog as feature work so they can’t quietly evaporate. The hard part is prioritisation, and this is where the error budget closes the loop beautifully: incidents that burned a lot of budget justify spending engineering time on prevention, and a blown budget gives the reliability work the authority to jump the queue. Prefer action items that remove a whole class of failure (“add an admission check so no one can ever apply this misconfiguration” — see policy-as-code) over one-off patches. That is the difference between getting more reliable and merely getting more tired, and it feeds directly into the disciplines in Best Practices (and the traps in Anti-patterns).

Chaos engineering

☺ Like you’re 10: Instead of waiting to be surprised when something breaks, you break it on purpose during the day — carefully, in a small corner — to check your safety nets actually work.

You can’t know your system survives a zone failure until a zone fails. Chaos engineering is the practice of finding out on your own terms — injecting controlled failures while you’re watching and rested, rather than discovering the weakness at 3am when you’re not. It is a scientific method, not random vandalism.

Hypothesis-driven experiments & steady state

A chaos experiment starts by defining steady state: a measurable definition of “normal and healthy,” usually one of your SLIs (“99.9% of checkouts succeed under 300 ms”). Then you form a hypothesis: “if we kill one of the three checkout replicas, steady state will be unaffected because the other two absorb the load and Kubernetes reschedules the third.” Then you inject the failure and measure. If steady state holds, you’ve earned confidence; if it breaks, you’ve found a weakness cheaply, in daylight, with everyone watching — which is the entire point. The valuable outcome isn’t the chaos; it’s the hypothesis being confirmed or, better, refuted.

Blast-radius control

The one non-negotiable of chaos engineering is blast-radius control — you must be able to limit and instantly stop the harm. That means starting small (one Pod, not the whole service; a fraction of traffic, not all of it), running in staging before production, having an abort switch that halts and reverts the experiment the moment steady state degrades past a threshold, and only expanding the blast radius as confidence grows. Chaos without a small, bounded, abortable blast radius isn’t engineering — it’s an outage you scheduled. Start in a pre-prod environment, graduate to production off-peak, and always announce experiments so nobody mistakes a drill for a real fire.

Game days & the tooling

A game day is a scheduled, human-in-the-loop exercise where the team deliberately triggers a realistic failure and practises the whole response — detection, paging, the IC model, mitigation, comms — as a rehearsal. It tests the humans and the runbooks as much as the software, and it’s where you discover the on-call doc is out of date before a real incident does. For the technical injection, two CNCF projects lead in Kubernetes. Chaos Mesh offers rich, declarative fault types as CRDs — Pod kills, network latency and partitions, IO faults, clock skew, stress — scoped by label selector. LitmusChaos takes a workflow-and-hub approach with a large catalogue of reusable experiments and GitOps-friendly pipelines. A minimal Chaos Mesh experiment that kills one checkout Pod for two minutes, scoped tightly to a single namespace and one replica:

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: checkout-kill-one
  namespace: chaos-testing
spec:
  action: pod-kill
  mode: fixed          # kill exactly N, not a percentage
  value: "1"           # blast radius = ONE pod
  duration: "2m"       # auto-revert after two minutes
  selector:
    namespaces: [ "shop-prod" ]
    labelSelectors:
      app: "checkout"
# Steady-state hypothesis (checked out-of-band on the SLO dashboard):
#   checkout success ratio stays > 99.9% and p99 < 300ms.
#   If it dips, abort the experiment immediately.

Toward self-healing

☺ Like you’re 10: The best fix is one where the computer notices the problem and quietly fixes it itself, so nobody has to wake up at all.

The north star of platform reliability is a system that mostly heals itself — where the boring, repetitive fixes happen automatically and humans are reserved for genuinely novel problems. You never reach perfect autonomy, but every step toward it buys back sleep and reduces the chance of a tired human fat-fingering a 3am fix.

Runbooks as code

A runbook is the documented procedure for handling a known situation. The evolution goes in three stages. First, a good prose runbook: clear, current, linked directly from the alert (note the runbook: annotation in the Prometheus rule earlier — every page should carry a link to exactly what to do). Second, runbook as code: the steps become an executable script or a parameterised job, so the response is a reliable command instead of a human copy-pasting under stress. Third, the script becomes trusted enough to run without a human in the loop. Crucially, much of Kubernetes is already self-healing: a crashed Pod is restarted, an unhealthy one is pulled from load balancing, a failed node’s Pods are rescheduled, and the Horizontal Pod Autoscaler adds capacity under load — all reconciliation loops from the Kubernetes substrate doing their job without a page.

Automated remediation: alert to action

The frontier is closed-loop automated remediation: an alert doesn’t (only) page a human — it triggers a safe, well-scoped fix directly. A memory leak that reliably recovers on restart can auto-restart the offending Pod; a known bad node can be automatically cordoned and drained; a stuck queue can be auto-flushed. The discipline that keeps this from becoming its own hazard: automate only remediations that are safe, idempotent, well-understood, and rate-limited with a circuit breaker — if the auto-remediator fires more than a few times in an hour, it must stop and escalate to a human, because a fix that keeps re-firing means the automation is masking a deeper problem instead of solving it. Wire the alert to a webhook receiver that runs the vetted job:

# Alertmanager routes an actionable alert to a remediation webhook
route:
  routes:
  - matchers: [ 'alert = "PodMemoryLeak"', 'auto_remediate = "true"' ]
    receiver: remediator
    group_wait: 30s
receivers:
- name: remediator
  webhook_configs:
  - url: http://remediator.ops.svc/restart-pod
    send_resolved: false
# The remediator service enforces the guardrails BEFORE acting:
#   - verify the alert is one on the vetted allow-list
#   - refuse if it has already acted > 3 times in the last hour  (circuit breaker)
#   - perform the minimal idempotent action (rollout restart of ONE workload)
#   - always page a human as well, with a note of what it did

Reducing toil

SRE has a precise word for the enemy here: toil — work that is manual, repetitive, automatable, tactical, and scales linearly with the size of the system, producing no lasting value. On-call that is all toil burns people out and, worse, doesn’t make the platform any better. Mature teams cap the fraction of time spent on toil (Google’s famous target is under 50%) and spend the rest engineering the toil away — because a self-service platform is, in a sense, one enormous toil-elimination machine: every ticket it removes is toil deleted for both the platform team and its users (that’s the whole thesis of self-service and platform-as-a-product). Reliability, in the end, is a flywheel: measure with SLOs, spend the budget deliberately, respond to incidents humanely, learn blamelessly, test with chaos, and automate the fixes — and each turn makes the next incident smaller, rarer, and quieter. This is the operational backbone of the whole platform you assemble across this course, from the architecture up.