Anti-Patterns & Pitfalls

Kubernetes' entire selling point is that it heals itself — kill a Pod and a controller replaces it, drain a node and workloads reschedule, and most days that reconciliation loop from The Object Model quietly papers over sloppy decisions before anyone notices. That's exactly what makes the seven mistakes on this page dangerous: none of them fail on day one. A :latest tag works fine until a registry push changes what it points to mid-incident. A missing resource limit works fine until a neighbor Pod gets noisy. A cluster-admin binding works fine until the credential holding it leaks. A cluster with no PodDisruptionBudget works fine until the exact week a node needs draining. Each trap below is a decision that looks free the day you make it and gets collected, with interest, the first time the cluster is actually under stress. Gizmo has a perfectly reasonable-sounding reason for every one of them; Timmy's question never changes — what happens the day this actually gets tested?

Why self-healing makes these traps quieter, not safer

☺ Like you're 10: A cluster that fixes small problems by itself can also be quietly hiding a big one — you just can't tell the difference from the outside until the big one finally shows up.

Every mistake below survives an audit that only checks whether the cluster is currently green. Pods are Running, the Deployment shows 3/3, the dashboard is all checkmarks — and none of that tells you whether the cluster would survive the specific stress each trap is vulnerable to. That's the throughline worth carrying through the whole page, the same test DevOps' own anti-patterns page runs against CALMS: would this configuration survive someone actually triggering the failure it's exposed to? A :latest tag survives a quiet week but not a mid-incident registry push. A missing PodDisruptionBudget survives months of nothing happening but not a single node drain. Reconciliation is real and valuable — it's also the reason these particular mistakes get to live undetected for so long.

1 · :latest and mutable tags in production

☺ Like you're 10: Ordering "whatever's newest" off a takeout menu is convenient — until "whatever's newest" turns out to be the dish that made everyone sick last week, and nobody can say which order caused it.

Gizmo's whisper: "Just tag the image latest and set imagePullPolicy: Always — every deploy always gets the newest build automatically, no version bumping, no digest wrangling." Why it tempts: it removes a whole category of CI busywork, and it works flawlessly in a dev cluster where re-pulling constantly is the entire point. Why it hurts: a tag is a mutable pointer, not a version. Two Pods from the exact same Deployment manifest, scheduled minutes apart, can silently end up running different code if someone pushes to the registry in between — there is no way to look at a running Pod and know which build it actually has. kubectl rollout undo can't roll back to a known-good build either, because the previous ReplicaSet's manifest also just says latest — the rollback target moved out from under you the moment the tag did. And imagePullPolicy: Always means every ordinary Pod restart — a crash, a reschedule, a routine node drain — can silently pull a newer, never-tested image straight into whatever incident is already in progress.

# Gizmo's version: convenient, and impossible to reason about later.
spec:
  containers:
    - name: api
      image: registry.example.com/api:latest
      imagePullPolicy: Always      # every restart can silently change the code

# Timmy's fix: an immutable reference, resolved once at build time.
spec:
  containers:
    - name: api
      # A digest is content-addressed — it cannot be moved. Restarting
      # this Pod, on any node, at any point, gets exactly this build.
      image: registry.example.com/api@sha256:3f29c2e1b6c1d4a9...
      imagePullPolicy: IfNotPresent
# CI still tags releases (api:1.14.2) for humans to read — the tag is
# just no longer what the cluster actually schedules against.

The fix: pin deployments to an image digest, not a tag — the tag stays as a human-readable label in your release notes, the digest is what the manifest actually references, and it cannot be silently repointed underneath you. kubectl's own rollout history becomes meaningful again once every entry maps to one immutable build.

2 · Skipping resource requests and limits

☺ Like you're 10: A lunch table with no assigned seats and no rule about portion size works great — right up until the biggest kid at the table starts eating everyone else's food, and the smallest kid gets moved off the table first when it runs out of room.

Gizmo's whisper: "Don't bother with requests and limits — let the scheduler figure it out. Setting real numbers means load-testing every service first, and we don't have time for that this sprint." Why it tempts: it's genuinely one less thing to tune, and on an idle cluster with plenty of headroom nothing appears to go wrong. Why it hurts: a Pod with no requests lands in the BestEffort QoS class, which is first in line to be evicted the moment any node runs short of memory — regardless of how important that Pod actually is. A Pod with no limits can consume an entire node's spare CPU or memory on its own, and when the node finally runs out, the kubelet doesn't necessarily kill the Pod that caused it — it kills by QoS class and usage-over-request, which is often some unrelated, well-behaved neighbor. SRE's Kubernetes reliability patterns page covers the exact eviction order this produces in full.

$ kubectl get events -n checkout --sort-by=.lastTimestamp | tail -3
5m   Warning  Evicted     pod/cart-service-7c9d-x2f9k    The node was low on resource: memory.
5m   Warning  OOMKilled   pod/cart-service-7c9d-x2f9k    Container cart exceeded memory
4m   Normal   Scheduled   pod/payments-5b8f-q7m1p         Successfully assigned checkout/payments...
# "cart-service" never leaked a byte — it just never asked for memory,
# so it was BestEffort, and "payments" (also unlimited) had grown quietly
# for weeks until the node finally ran out and picked a victim.

The fix: set requests to a real, measured baseline (not a guess) and limits to a ceiling you can justify, even a generous one — Guaranteed or Burstable QoS is always safer than leaving anything to fall back to BestEffort by omission.

3 · cluster-admin as the default RoleBinding

☺ Like you're 10: Giving the intern a master key that opens every door in the building — the server room, the vault, the roof — because it's faster than figuring out which one door they actually need.

Gizmo's whisper: "This CI pipeline needs to deploy stuff. Just bind its ServiceAccount to cluster-admin so we never hit a permission error mid-release again." Why it tempts: a permission error at 2am, mid-deploy, is exactly the kind of thing everyone wants to eliminate forever, and cluster-admin genuinely eliminates it — every single time, for every single resource. Why it hurts: the moment that binding exists, the ServiceAccount's token is no longer "a deploy credential" — it's every credential. A leaked CI secret, a compromised build agent, or one badly written third-party GitHub Action with access to that token now has the ability to read every Secret in every namespace, delete any workload, and grant itself even more access on the way out. RBAC & Admission Control and DevSecOps' Kubernetes security deep dive both cover why this single binding is the most common real-world path from "one leaked token" to "full cluster compromise."

# Fix: a Role scoped to one namespace and the exact verbs a deploy needs.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployer
  namespace: checkout
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-deployer
  namespace: checkout          # scoped — this token is worthless outside it
subjects:
  - kind: ServiceAccount
    name: ci
    namespace: checkout
roleRef:
  kind: Role
  name: deployer
  apiGroup: rbac.authorization.k8s.io

The fix: bind every ServiceAccount to the narrowest Role that lets it do its actual job, scoped to the namespaces it actually touches — never a ClusterRoleBinding by default. Drill — Harden an RBAC Configuration is built entirely around finding and fixing bindings exactly like this one. If you're studying toward the CKS or KCSA security domains, this is one of the highest-yield topics on the whole exam blueprint — see the Golden Kubestronaut ladder's security tier, covered in full on the sibling Golden Astronaut course.

4 · No PodDisruptionBudget, no anti-affinity — the whole app on one node

☺ Like you're 10: Saving three copies of your homework — all on the same USB stick. It really is three copies, right up until the one stick gets lost.

Gizmo's whisper: "Three replicas is plenty of redundancy. A PodDisruptionBudget and pod anti-affinity rules are extra YAML for a problem we don't actually have." Why it tempts: the Deployment genuinely does say replicas: 3, and on a quiet day, with the scheduler making its own reasonable placement choices, all three copies really do look redundant. Why it hurts: without a podAntiAffinity rule, nothing stops the scheduler from happily placing all three replicas on the same node — it's optimizing for bin-packing, not for your specific redundancy story. And without a PodDisruptionBudget, nothing stops a routine, planned operation — a node drain for a Kubernetes upgrade, a cordoned node for maintenance, cluster-autoscaler consolidating capacity — from evicting all three at once, with zero replicas left standing during the exact rollout window meant to be the safe, boring one.

# The two objects that were missing.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
  namespace: checkout
spec:
  minAvailable: 2               # kubectl drain now waits/refuses until this holds
  selector:
    matchLabels: { app: api }
---
# Spread the replicas so a single node failure can't take all of them.
spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels: { app: api }

The fix: set a PodDisruptionBudget on every Deployment that matters, and pair it with podAntiAffinity so replicas actually land on different nodes in the first place — a PDB alone can only block a drain from over-evicting; it can't undo a scheduler decision that already put all your eggs in one basket. SRE's Kubernetes reliability patterns covers exactly how a PDB interacts with kubectl drain and voluntary eviction under the hood.

5 · Snowflake clusters — kubectl edit as the change-management process

☺ Like you're 10: Fixing a museum exhibit by hand, in the actual museum, instead of updating the blueprint everyone else builds the next exhibit from — the fix works today, but nobody else knows it happened.

Gizmo's whisper: "It's a one-line fix — just kubectl edit it directly in prod. No need to touch Git, open a PR, and wait for a pipeline for something this small." Why it tempts: it's genuinely faster in the moment, especially mid-incident when every extra minute feels expensive. Why it hurts: the live cluster and the Git repository that's supposed to describe it now disagree, and nobody wrote that disagreement down anywhere. If the cluster is running GitOps, the reconciler notices the drift on its next sync and silently reverts the hotfix back to whatever Git says — the "fix" quietly vanishes, often hours later, with no obvious connection back to the edit that made it. If it isn't running GitOps, the drift just accumulates: six months of hand-applied patches later, nobody can rebuild the cluster from source, and the only accurate description of what's actually running lives in the muscle memory of whoever typed each kubectl edit.

# The trap, end to end:
$ kubectl edit deployment api -n checkout        # bumps memory limit, prod-only
deployment.apps/api edited

# ...three hours later, the GitOps reconciler runs its normal sync...
$ argocd app get checkout-api
Sync Status:  OutOfSync -> Synced
# The reconciler just reverted the hand-edit back to what Git says,
# because Git — not the live cluster — is the source of truth.
# The original OOM problem the edit "fixed" is back within the hour.

The fix: every change to a cluster's desired state goes through Git — GitOps on Kubernetes is the whole discipline built around exactly this — and every cluster is provisioned the same reproducible way, whether that's kubeadm, a managed offering, or infrastructure-as-code, never a sequence of manual steps someone remembers from the last time. If you can't tear a cluster down and rebuild it from source in under an hour, it's a snowflake, whatever the dashboard says. This is the same shape of trap Platform Engineering's own anti-patterns page calls "ticket-ops in disguise" — a manual fix that feels fast today and becomes tribal knowledge nobody can audit tomorrow.

6 · Treating etcd as an afterthought

☺ Like you're 10: etcd is the one notebook holding every single fact about your cluster — which Pods exist, which Secrets are real, what the desired state even is. Put that notebook on the cheapest, slowest shelf in the building and never make a photocopy, and you've built the whole school on top of a notebook nobody is actually protecting.

Gizmo's whisper: "etcd's just internal bookkeeping — run it on whatever's left over, we'll sort out backups later once things settle down." Why it tempts: etcd never shows up in an app dashboard, nobody on the team owns it explicitly, and a cluster with healthy Pods gives zero visible signal that its control plane's storage is one bad disk away from disaster. Why it hurts: etcd is a Raft-based consensus store — Control Plane Internals covers the mechanism in full — which means it needs an odd number of members (3 or 5, never 2 or 4) to keep quorum, and it's unusually sensitive to disk latency, because every write has to fsync to a majority of members before it's acknowledged. Undersize that disk and API server latency creeps up cluster-wide, for reasons nothing in the API server's own logs will point you toward. Skip backups entirely and losing quorum — a bad upgrade, two members failing together, a botched disk resize — doesn't just take down the control plane, it can permanently erase your cluster's entire memory of itself: every object, every Secret, every RBAC binding, gone with no way to reconstruct it.

# A snapshot is the entire recovery story for etcd — there is no
# "undo" once quorum is actually lost, only "restore from snapshot."
$ ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%F-%H%M).db \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key

$ etcdctl snapshot status /backup/etcd-2026-08-27-0300.db --write-out=table
# Automate this on a cron or systemd timer, ship it off-node immediately,
# and actually rehearse the restore — an untested backup is a hope,
# not a recovery plan.

The fix: run etcd on its own fast, dedicated disk (SSD, low-latency, never shared with noisy neighbors), keep an odd-numbered member count, automate etcdctl snapshot save off-box on a real schedule, and rehearse a restore before you ever need one for real. Velero backs up Kubernetes objects and persistent volumes at the API level and is worth running too, but it is not a substitute for an etcd snapshot — Velero can't restore a control plane that has lost quorum entirely.

7 · Privileged Pods and hostPath as the default Pod spec

☺ Like you're 10: Giving every guest the master key and the safe combination "just in case they need something," instead of the one specific door they actually need.

Gizmo's whisper: "The app needs to read one file off the node. Just mount a hostPath volume and set privileged: true — it's faster than figuring out the exact capability it actually needs." Why it tempts: figuring out the minimal set of Linux capabilities a container genuinely needs takes real investigation, and privileged: true reliably makes every permission-denied error disappear at once. Why it hurts: a privileged container is effectively root on the node itself, with access to every device and every other container's filesystem through the shared kernel — a container escape from a privileged Pod isn't a contained incident, it's node compromise, and node compromise reaches the kubelet's own credentials and every Pod scheduled there, which is a very short hop back to trap 3's cluster-wide blast radius. A hostPath mount is nearly as dangerous on its own: mount the node's filesystem and a compromised container can read every other Pod's mounted Secrets sitting right there on disk, no RBAC check required, because the RBAC layer was never in the request path at all.

# Gizmo's version: root-equivalent on the node, no capability review needed.
spec:
  containers:
    - name: log-reader
      securityContext:
        privileged: true
      volumeMounts:
        - { name: nodefs, mountPath: /host }
  volumes:
    - name: nodefs
      hostPath: { path: / }

# Timmy's fix: the restricted Pod Security Standard, and exactly the
# one capability the container actually needs — nothing implicit.
spec:
  containers:
    - name: log-reader
      securityContext:
        runAsNonRoot: true
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]
        seccompProfile: { type: RuntimeDefault }

The fix: enforce the restricted Pod Security Standard at the namespace level via Pod Security Admission (or an admission controller like OPA Gatekeeper or Kyverno for finer-grained policy), so privileged Pods and raw hostPath mounts can't land in a normal namespace at all — not even by accident. Security: Defense in Depth and DevSecOps' Kubernetes security deep dive both build on exactly this pattern, and it's one of the core competencies on the CKS blueprint.

The whole rogues' gallery, at a glance

☺ Like you're 10: One cheat-sheet — the tell and the fix — for all seven traps at once.

Notice the shape of every fix on this page: each one trades a decision that looked free for one that costs a little more up front and pays it back the one day it's actually needed — a digest instead of a moving tag, a measured limit instead of an open-ended guess, a scoped Role instead of the master key, a PDB instead of hoping the scheduler spreads things out on its own, a Git commit instead of a hand-edit nobody else can see, a rehearsed etcd restore instead of an unopened backup folder, a dropped capability instead of a shortcut around it. None of these traps look like mistakes from inside a healthy-looking cluster — that's the entire reason they last. Keep asking Timmy's question — what happens the day this actually gets tested? — and most of them don't survive contact with it. The companion page, Best Practices & Operating Model, covers what the healthy version of each of these looks like in steady-state operation.