Kubernetes Reliability Patterns

Reliability patterns covered the mechanisms you build into your own application code — retries, circuit breakers, timeouts, graceful degradation. This page goes one layer down, into the orchestrator sitting underneath that code, because Kubernetes ships its own reliability primitives, and each one fails in its own specific, recognizable way when it's misused. Three mechanisms carry almost all of that weight: the health-check triad that decides whether a Pod keeps receiving traffic or gets killed and restarted, the PodDisruptionBudget that decides whether a planned maintenance operation is even allowed to touch a given Pod, and the resource requests and limits that quietly decide which Pod gets sacrificed first when a node runs short of memory. None of this is exam-blueprint trivia. Get any one of the three wrong and you don't get a warning — you get an outage with a specific, traceable shape, one experienced Kubernetes operators learn to recognize on sight. We'll take each mechanism apart: what Kubernetes actually does under the hood, the real incident pattern each mistake produces, and the fix.

The health-check triad: liveness, readiness & startup probes

☺ Like you're 10: Three different checks, three completely different consequences when they fail — one restarts the container, one just quietly removes it from the lineup, and one is a grace period that protects the other two from firing too early.

A kubelet doesn't know whether the process it started is actually working — from the outside, a hung process and a healthy one look identical. Probes are how you tell it. Kubernetes ships exactly three, and the single most important fact about them is that they don't do the same thing when they fail — conflating them is where nearly every probe-related outage starts.

A liveness probe answers one question only: is this process's own internal state stuck? Fail it failureThreshold times in a row and the kubelet kills the container and restarts it, full stop — that's the entire remedy it knows how to apply. A readiness probe answers a different question: is this Pod ready to receive traffic right now? Fail it and the Pod is removed from the Service's Endpoints/EndpointSlices — traffic stops arriving — but the container itself is left completely alone, still running, free to recover and rejoin on its own. A startup probe answers neither question; it exists purely to buy time. While it's running, the kubelet disables the liveness and readiness probes entirely, so a slow-starting process — a JVM warming up, a large in-memory cache loading — can't get killed by an impatient liveness timeout before it's even had a chance to finish booting. The instant the startup probe succeeds once, it steps aside permanently for that container's lifetime, and liveness and readiness take over.

containers:
  - name: checkout
    image: registry.acme.io/checkout:2.4.1
    startupProbe:
      httpGet: { path: /healthz/startup, port: 8080 }
      periodSeconds: 2
      failureThreshold: 30        # up to 60s to finish booting before liveness/readiness engage
    livenessProbe:
      httpGet: { path: /healthz/live, port: 8080 }
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3         # ~30s of a stuck event loop before a restart
    readinessProbe:
      httpGet: { path: /healthz/ready, port: 8080 }
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 2         # ~10s before traffic is pulled

Two easy-to-miss details in the API itself. First, successThreshold on a liveness or startup probe is forced to 1 by the apiserver's own validation — you cannot require two consecutive successes to un-fail a liveness check, only readiness allows a higher value, which matters if you want a Pod to prove itself stable for a couple of checks before it's trusted with traffic again. Second, an exec probe (running a command inside the container) counts every concurrently-running probe process against the container's own resource limits, which under enough load can itself become a source of the very slowness the probe is trying to detect.

The liveness-probe outage: conflating liveness with readiness

☺ Like you're 10: Point the "call an ambulance" check at a downstream dependency, and one hiccup in that dependency gets every single kid an ambulance ride at the exact same moment — which is exactly the wrong response and makes the original problem worse.

Here is the single most common Kubernetes reliability incident, and it's entirely self-inflicted: a service exposes one /healthz endpoint that checks its own process health and pings its database, its cache, and maybe a downstream API, and that same endpoint gets wired up as both the liveness probe and the readiness probe. It looks thorough. It is a loaded gun.

Walk through what happens when the database has a routine 90-second blip — a failover, a connection-pool exhaustion event, anything transient:

• Every replica fails the shared health check at roughly the same time, because they're all checking the same now-unreachable dependency. This isn't staggered; it's correlated by construction.
• Because the same endpoint backs the liveness probe, the kubelet doesn't just pull traffic — it restarts every container, near-simultaneously, across the whole fleet. Restarting the application does nothing to fix the database; the liveness probe has diagnosed a problem it has no ability to remedy.
• The restarted containers all reconnect at once, landing a connection-pool stampede on a database that was already struggling and is now trying to recover from a herd of simultaneous new connections instead of a steady trickle — frequently extending the outage well past the length of the original blip.
• If the database is still unreachable when the new containers run their own startup/liveness checks, they fail again immediately, and Kubernetes escalates into CrashLoopBackOff — an exponential restart backoff (10s, 20s, 40s, 80s, 160s, capped at 300s) that was designed to protect the cluster from a genuinely broken container, not to speed up recovery from a transient dependency blip. A team can now be looking at a 20-minute self-inflicted outage caused by a 90-second database hiccup, with the majority of that 20 minutes spent doing nothing but waiting out backoff timers on containers that were never actually broken.

The fix is almost insultingly simple once the failure mode is named: give liveness and readiness genuinely separate endpoints and genuinely separate logic. /healthz/live checks nothing but the process's own ability to respond — no network calls out. /healthz/ready checks the dependencies that actually determine whether this replica can usefully serve a request right now. The two endpoints will disagree constantly and correctly: a Pod can be perfectly alive (its own event loop is fine) while simultaneously not ready (its database is down) — that disagreement is the entire point of having two separate probes in the first place, not a bug in your health-check design.

Readiness, Endpoints & the graceful-shutdown race

☺ Like you're 10: Telling every other machine "stop sending mail to this address" doesn't happen instantly — it takes a moment to spread the news, so the address has to keep answering the door for a little while after it's told everyone to stop.

Readiness has a second, subtler failure mode that shows up specifically during rollouts and scale-downs, not during incidents: dropped requests during ordinary, planned Pod termination. When a Pod is deleted, two things are supposed to happen together — the kubelet sends SIGTERM to the container, and the EndpointSlice controller removes the Pod from the Service's endpoint list so traffic stops routing to it. In practice these are not perfectly synchronized: endpoint removal has to propagate from the apiserver out to every node's kube-proxy (or every sidecar in a service mesh), and that propagation takes real, non-zero time — typically low hundreds of milliseconds, sometimes more under load. If the container stops accepting connections the instant it receives SIGTERM, there's a real window where a Pod that's already technically dead is still receiving traffic that hasn't heard the news yet, and those requests fail.

The standard fix is a deliberate, small wait built into shutdown itself, using a preStop lifecycle hook:

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 10"]
terminationGracePeriodSeconds: 30   # must exceed preStop sleep + actual shutdown time

The preStop hook runs before SIGTERM is delivered to the main process, so during that sleep window the container keeps its server open and keeps serving any in-flight or newly-arriving requests exactly as before — it just no longer advertises itself as ready, and endpoint removal has the whole sleep window to finish propagating cluster-wide before the process actually begins shutting down. Get terminationGracePeriodSeconds wrong relative to the preStop sleep and the fix backfires: if the grace period is shorter than the sleep plus real shutdown work, the kubelet sends SIGKILL at the deadline regardless of where the process is in its shutdown sequence, which can be worse than not having the hook at all. A related, narrower extension point is worth knowing by name even without a deep dive here: Pod readiness gates let an external controller (the AWS Load Balancer Controller registering a target group is the canonical example) inject an additional condition that must also be true before a Pod counts as Ready — useful when "ready" depends on state outside the cluster entirely.

PodDisruptionBudgets: voluntary disruption, and only voluntary disruption

☺ Like you're 10: The promise "never leave a classroom with fewer than two kids" only applies to planned trips out of the room — it does nothing at all if the classroom's roof caves in.

A PodDisruptionBudget (PDB) is a promise you attach to a set of Pods: no planned maintenance operation may reduce the number of healthy, matching Pods below a floor you set. The floor is expressed as minAvailable (an absolute count or a percentage of the selected Pods that must stay Ready) or maxUnavailable (the mirror — how many may be down at once) — the two are mutually exclusive on a single PDB, pick one framing per workload.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-pdb
  namespace: shop
spec:
  minAvailable: 2
  selector:
    matchLabels: { app: checkout }
  unhealthyPodEvictionPolicy: AlwaysAllow   # 1.27+: already-broken pods can still be evicted

The word doing all the work in the definition is voluntary. Kubernetes draws a hard line between two categories of disruption, and a PDB only governs one of them:

• Voluntary disruption — anything initiated by an actor deliberately asking Kubernetes to remove a Pod in an orderly way: kubectl drain ahead of a node upgrade, Cluster Autoscaler or Karpenter consolidating nodes to save cost, a managed node-group rolling update, or a descheduler rebalancing the cluster. All of these go through the same API call, and a PDB is checked before that call is allowed to proceed.
• Involuntary disruption — a node crashing, a kernel panic, an out-of-memory event that takes the whole node down, a hardware fault, a spot/preemptible instance being reclaimed by the cloud provider with little or no notice. Nothing in Kubernetes intercepts these; there's no orderly API call for a PDB to gate, because nobody asked permission. A PDB does nothing whatsoever to protect against this category — that protection comes only from actual redundancy (enough replicas, spread across enough failure domains), which is a separate design decision a PDB cannot substitute for.

Since Kubernetes 1.27, an optional unhealthyPodEvictionPolicy field refines this further: the default, IfHealthyBudget, only allows evicting an already-unhealthy Pod when doing so wouldn't itself violate the budget, which can strand a broken Pod on a node you're trying to drain — a Pod that's already failing has nothing left to protect by counting it against the budget. Setting AlwaysAllow (as above) lets already-unhealthy Pods be evicted regardless of the current budget, so draining isn't held hostage by a Pod that was never contributing to availability in the first place. Verify the current default and GA status against Kubernetes's own release notes for the version you're running — this is exactly the kind of default that has shifted across recent releases.

PDBs meeting node draining: the eviction API and the deadlock trap

☺ Like you're 10: The office doesn't just take a kid out of class — it politely asks first, and if the answer is "our promise says no," it has to wait and ask again later, sometimes forever.

Mechanically, kubectl drain does not delete Pods directly. For every Pod on the node it POSTs to that Pod's eviction subresource, and the apiserver's built-in disruption controller checks that request against every PDB whose selector matches the Pod before honoring it:

curl -sk -X POST \
  https://$APISERVER/api/v1/namespaces/shop/pods/checkout-7d9f-x8/eviction \
  -H 'Content-Type: application/json' \
  -d '{"apiVersion":"policy/v1","kind":"Eviction",
       "metadata":{"name":"checkout-7d9f-x8","namespace":"shop"}}'

# 200 OK                 → evicted; disruptionsAllowed had room, and was decremented
# 429 Too Many Requests  → would violate the PDB; caller (drain / autoscaler /
#                          descheduler) backs off and retries

kubectl get pdb checkout-pdb -n shop
# NAME           MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
# checkout-pdb   2               N/A               1                    14d

That ALLOWED DISRUPTIONS column — the PDB's live status.disruptionsAllowed — is the actual headroom every drain, autoscaler consolidation, and rolling node upgrade is negotiating against in real time. It's a single shared number: the moment one eviction succeeds and drops the healthy count to the floor, it falls to zero and every subsequent eviction attempt against that PDB's Pods gets a 429 until a replacement Pod somewhere else becomes Ready and pushes the healthy count back up.

This produces a genuinely nasty, easy-to-create failure mode: a PDB deadlock. A 3-replica Deployment with minAvailable: 3 — or equivalently maxUnavailable: 0 — is a promise that literally cannot be honored while also evicting even one Pod: evicting any one of the three would drop the healthy count to 2, below the floor, so disruptionsAllowed sits permanently at zero. kubectl drain against a node holding one of those Pods will retry, back off, and eventually time out with an error. Cluster Autoscaler and Karpenter behave more quietly and, arguably, more dangerously: they simply treat that node as non-scalable and skip it, silently, with no error surfaced anywhere — a fleet-wide habit of setting maxUnavailable: 0 "to be safe" is a common, hidden reason cost-saving node consolidation quietly stops working across an entire cluster. A cluster-wide Kubernetes version upgrade can stall on exactly this one node, blocking every node behind it in the rollout.

Requests, limits & QoS classes: the real eviction mechanism

☺ Like you're 10: The classroom rule about who loses their table first when the cafeteria runs short isn't a vibe — it's arithmetic Kubernetes runs automatically the moment you write down what each kid reserved versus what they're actually eating.

What requests and limits actually do, down at the cgroup

A Pod's resources.requests and resources.limits aren't advisory notes for a human reading YAML — they're translated directly into Linux cgroup settings the kernel enforces. Requests feed the scheduler's bin-packing math (the sum of every Pod's requests on a node must fit within that node's Allocatable capacity) and set the container's cpu.shares — a relative weight the kernel's CFS scheduler only consults when the node is actually CPU-contended; an idle node ignores it entirely. Limits are enforced hard, and CPU and memory diverge completely in how:

• CPU is compressible. A limit sets a CFS quota (cpu.cfs_quota_us per cpu.cfs_period_us). Exceed it and the kernel simply withholds further CPU time from the container until the next scheduling period — no error, no crash, just added latency that shows up as p99 spikes and, if you're scraping it, a rising container_cpu_cfs_throttled_periods_total. The process is never killed for this.
• Memory is incompressible. A limit sets a hard cgroup ceiling (memory.max on cgroup v2). Exceed it and there is no "wait for the next period" option — the kernel's OOM killer fires inside that container's own cgroup and kills a process immediately. The Pod's container status reports OOMKilled, exit code 137 (128 + signal 9).

That asymmetry — throttle versus kill — is exactly why teams argue endlessly about setting CPU limits (the downside is bounded and just latency) while nobody seriously argues against setting memory requests and limits at all (the downside of skipping them is an unpredictable node-wide event instead of one contained, expected container death).

QoS classes: computed, not chosen

You never set a Pod's Quality-of-Service class directly — the apiserver derives it automatically, at admission time, purely from the requests/limits shape across every container in the Pod:

# Guaranteed — requests == limits, every resource, every container
resources: { requests: { cpu: "500m", memory: "512Mi" },
             limits:   { cpu: "500m", memory: "512Mi" } }

# Burstable — a request is set, but limits are looser (or missing)
resources: { requests: { cpu: "250m", memory: "256Mi" },
             limits:   { cpu: "1",    memory: "1Gi"   } }

# BestEffort — no requests or limits anywhere in the Pod
resources: {}

kubectl get pod checkout-7d9f-x8 -o jsonpath='{.status.qosClass}'
# Burstable

oom_score_adj is the mechanism that makes this real: it's a Linux kernel field the kubelet sets on every container process at creation time, and it's an additive offset the kernel's OOM killer folds into its own node-wide badness score when memory runs out and it has to pick a victim. A lower (more negative) value makes a process dramatically less likely to be the one killed; Guaranteed's -997 is about as close to "leave this alone" as the kernel allows without disabling the OOM killer for that process entirely.

Eviction order under node pressure: a worked example

Two entirely separate mechanisms operate at different layers, and mixing them up is a common source of confusion. The kubelet eviction manager is proactive: it watches signals like available memory against configured thresholds (for example --eviction-hard=memory.available<100Mi) and, once tripped, gracefully evicts whole Pods — sending SIGTERM, honoring a (capped) grace period — before the node actually runs out of memory. Its documented ranking order, applied across all Pods on the node regardless of QoS class, is: (1) Pods whose current usage exceeds their own request for the starved resource are considered first; (2) among those, rank by Pod Priority (a lower-priority Pod goes before a higher-priority one); (3) then by how far usage exceeds request, largest overage first. Only if the node is still critical after evicting every over-request Pod does it reach into Pods sitting within their requests at all. The kernel OOM killer is the separate, reactive last resort — it fires only if eviction didn't act in time, or memory hit zero headroom outright, and it picks a single process using oom_score_adj plus live usage, with no coordination with the kubelet at the moment of the kill; the kubelet only finds out and reports OOMKilled after the fact.

Put a concrete node under memory pressure and the order falls out directly:

• A BestEffort Pod, no request set, using 300Mi — it structurally "exceeds" its own zero-byte request the instant it uses anything at all. Evicted first.
• A Burstable Pod, requested 200Mi, currently using 600Mi — 400Mi over its request. Evicted next, ranked against any other over-request Pods by how far over they are and by Priority.
• A Guaranteed Pod, requested and limited to 500Mi, using 480Mi — under its request. Left alone, unless the node is still critical after everything else above is gone, which by that point is a genuinely dire node-level event.

In practice this also means Priority matters as a tiebreaker independent of QoS — cluster infrastructure add-ons (CoreDNS, the CNI plugin, kube-proxy) are typically deployed with system-node-critical or system-cluster-critical priority classes precisely so they're essentially never selected, regardless of their own QoS class, because losing the node's own networking mid-eviction would make everything else worse.

Where this fits

☺ Like you're 10: Every mechanism on this page is a guardrail that already exists, running whether you configured it thoughtfully or not — the only choice you actually get is whether it's working for you or quietly against you.

All three mechanisms on this page share one property worth naming explicitly: none of them are optional. A Pod with no probes still has an implicit always-succeeding liveness check; a workload with no PDB is simply undefended against every voluntary disruption a busy cluster runs constantly; every Pod gets a QoS class whether you thought about it or defaulted into BestEffort by omission. The only real choice is whether these run by design or by accident, and getting them wrong on purpose is a standard, recurring item on a production readiness review for exactly that reason. Capacity planning & performance is where the requests you set here become the input to a much bigger arithmetic problem — how much headroom the fleet needs in aggregate; monitoring & observability is where you'd actually alert on the symptoms these mechanisms produce (a rising restart count, an OOMKilled event, a PDB stuck at zero disruptionsAllowed), and multi-window, multi-burn-rate alerting is how you'd decide whether a spike in those symptoms is actually worth paging on. If you want to prove any of this holds under real pressure rather than trusting the YAML, chaos engineering — deliberately killing a node, or exhausting a Pod's memory limit on purpose — is exactly how you'd verify it before an unplanned node failure does the verification for you. These same mechanics — probes, disruption budgets, requests/limits, and QoS — sit squarely inside the CKA exam's workload and troubleshooting domains if you're studying toward it; verify the current exam blueprint on the CNCF's own page, since domain weights shift between versions.