Triage — Workloads & Storage

Most broken things on a performance-based platform exam are broken at the workload layer: a pod that will not schedule, an image that will not pull, a container that dies and dies and dies, a probe that shoots a perfectly healthy app, or a volume that nobody will hand over. These failures are worth learning as a vocabulary, because the status column narrows the cause to a handful of possibilities before you have run a single extra command. This page is the pod-state matrix and the storage-binding chain in full — symptom, likely cause, the first command to run, and the typical fix — sitting on top of the same fixed diagnostic order you should be running on everything. It is one branch of the triage hub; the other branches cover networking, RBAC and admission and GitOps, observability and platform APIs.

The universal triage order — a 90-second recap

☺ Like you’re 10: Before you guess what’s wrong, always look in the same six places, in the same order. Guessing first is how you waste ten minutes.

The single biggest time sink under exam pressure is forming a hypothesis before gathering evidence. You see CrashLoopBackOff, you decide it must be the config map, you spend six minutes on the config map, and the answer was in the Events the whole time. So: do not think. Run the sequence. The sequence takes ninety seconds and it very frequently hands you the literal answer in plain English. Everything later on this page assumes you have already run it.

The six steps, in order

Steps 1–4 are the general Kubernetes ladder; step 5 is the platform-engineering addition that matters enormously on this exam, because so much of the platform is expressed as custom resources whose status.conditions block is the real error message. Only at step 6 do you get to have an opinion.

The 90-second opener

Keep this block in muscle memory. It is the same six commands whether the thing in front of you is a pod, a StatefulSet, or a volume that never appeared.

# The 90-second opener. Run this on every broken thing, every time.
kubectl config current-context                     # am I even on the right cluster?
kubectl get pods -n "$NS" -o wide                  # 1. wide list
kubectl describe pod -n "$NS" "$POD" | tail -30    # 2. Events live at the BOTTOM
kubectl get events -n "$NS" --sort-by=.lastTimestamp | tail -25   # 3. everything else that failed
kubectl logs -n "$NS" "$POD" --previous --tail=50  # 4. why the process exited
kubectl logs -n "$NS" "$POD" -c "$INIT" --previous # 4b. init containers need -c

# Cluster-wide sweep when you do not yet know where the damage is:
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get events -A --sort-by=.lastTimestamp | grep -i -E 'warn|fail|error' | tail -30

Pod-level failures

☺ Like you’re 10: Pods break in about ten recognisable ways. Learn the ten names and each one tells you where to look.

Pod states are a diagnostic vocabulary. The status column is not just a label — it narrows the cause down to a handful of possibilities before you have run a single extra command. This is the highest-value table on the page; if you internalise nothing else, internalise this.

The pod-state matrix

Pending and Unschedulable

A Pending pod has been accepted by the API server but the scheduler cannot place it. The scheduler is unusually chatty about why — its message enumerates every node and the reason each one was rejected, and that message alone almost always closes the case.

kubectl describe pod -n app web-5d8f | tail -20
# Events:
#   Warning  FailedScheduling  ...  0/3 nodes are available:
#     1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: },
#     2 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found.

# Read the arithmetic yourself:
kubectl describe node worker-1 | grep -A 8 'Allocated resources'
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory
kubectl get node worker-1 -o jsonpath='{.spec.taints}{"\n"}'
kubectl get node --show-labels | head            # does the nodeSelector label exist at all?
kubectl get resourcequota -n app                 # quota blocks admission, not scheduling
kubectl get pvc -n app                           # an unbound PVC pins the pod in Pending

ImagePullBackOff and ErrImagePull

ErrImagePull is the first failure; ImagePullBackOff is kubelet backing off after repeated failures. The Events line distinguishes the three causes precisely: not found or manifest unknown means a bad name or tag, unauthorized or authentication required means credentials, and a timeout or DNS error means the registry is unreachable from the node.

kubectl describe pod -n app api-7c9 | grep -A 5 Events
#   Failed to pull image "ghcr.io/acme/api:v1.4.3": ... denied: denied
#   -> unauthorized: the node has no credentials for a private registry

# Create the pull secret and attach it (either to the pod spec or the ServiceAccount):
kubectl create secret docker-registry ghcr-creds -n app \
  --docker-server=ghcr.io --docker-username="$USER" --docker-password="$TOKEN"

kubectl patch serviceaccount default -n app \
  -p '{"imagePullSecrets":[{"name":"ghcr-creds"}]}'   # applies to all pods using this SA

# Confirm what the pod is actually asking for (typos hide well in long refs):
kubectl get pod -n app api-7c9 -o jsonpath='{.spec.containers[*].image}{"\n"}'

Note that adding imagePullSecrets to a ServiceAccount only affects newly created pods, so delete the existing ones (or let the Deployment roll) afterwards. Secret handling more broadly is covered in Secrets Management.

CrashLoopBackOff

The container starts, exits, and kubelet restarts it with growing backoff (10s, 20s, 40s, up to five minutes). The crucial move is --previous: by the time you look, the current container may not have produced any output yet, and without -p you will see an empty log and conclude, wrongly, that there is nothing to find.

kubectl logs -n app worker-abc --previous --tail=60
kubectl logs -n app worker-abc --previous --all-containers=true

# The exit code narrows it further:
kubectl get pod -n app worker-abc -o jsonpath='{.status.containerStatuses[0].lastState.terminated}{"\n"}'
#  exit 1        generic application error — read the logs
#  exit 137      SIGKILL — usually OOMKilled, check reason field
#  exit 143      SIGTERM — killed during shutdown/eviction
#  exit 126      command found but not executable
#  exit 127      command not found — bad `command:` or wrong image entrypoint

# Reproduce interactively without the broken entrypoint getting in the way:
kubectl debug -n app worker-abc -it --image=busybox:1.36 --target=worker -- sh
kubectl run tmp-shell --rm -it --image=busybox:1.36 --restart=Never -- sh

☺ Like you’re 10: --previous means “show me the logs of the one that just died,” not “the one that’s about to die.” In a crash loop, the dead one is the only one that has anything to say.

OOMKilled and exit code 137

Exit 137 is 128 + 9, i.e. the process received SIGKILL. When the kernel cgroup OOM killer did it, the pod status carries reason: OOMKilled explicitly. There are only two real fixes: give it more memory, or make it use less.

kubectl describe pod -n app cache-0 | grep -A 4 'Last State'
#     Last State:     Terminated
#       Reason:       OOMKilled
#       Exit Code:    137

kubectl top pod -n app --containers        # needs metrics-server
kubectl get pod -n app cache-0 -o jsonpath='{.spec.containers[0].resources}{"\n"}'

# Fix: raise the limit (and keep requests honest so the scheduler places it correctly)
kubectl set resources deployment/cache -n app --limits=memory=1Gi --requests=memory=512Mi

CreateContainerConfigError

This one is a gift: it means the pod spec references a ConfigMap or Secret that does not exist, or a key inside it that does not exist, and kubelet says so verbatim.

kubectl describe pod -n app api-1 | grep -A 3 Events
#   Error: couldn't find key DATABASE_URL in ConfigMap app/api-config

kubectl get configmap api-config -n app -o jsonpath='{.data}{"\n"}' | tr ',' '\n'
kubectl describe secret api-creds -n app          # lists the keys and byte sizes, never the values

# Fix the key (or make the reference tolerant if it genuinely is optional):
kubectl create configmap api-config -n app \
  --from-literal=DATABASE_URL=postgres://db:5432/api \
  --dry-run=client -o yaml | kubectl apply -f -

A related state, CreateContainerError, usually means the container runtime could not start the process at all — a bad command, a missing binary, or a read-only root filesystem where the app expects to write. See Configuration Management for how to keep these references from drifting in the first place.

Init container failures

Init containers run to completion, in order, before any app container starts. A stuck Init:0/2 is not an app problem — it is a dependency problem, and the pod will sit there indefinitely, which is often exactly the “app never comes up” mystery you were sent to solve.

kubectl get pod -n app api-1                       # STATUS: Init:0/2  or  Init:CrashLoopBackOff
kubectl get pod -n app api-1 -o jsonpath='{range .spec.initContainers[*]}{.name}{"\n"}{end}'
kubectl logs -n app api-1 -c wait-for-db --previous
kubectl describe pod -n app api-1 | grep -A 12 'Init Containers'

Readiness vs liveness — two very different symptoms

Confusing these costs people marks. The symptom tells you which probe is at fault, and therefore which knob to turn.

kubectl describe pod -n app web-1 | grep -E 'Liveness|Readiness|Startup'
#   Liveness:   http-get http://:8080/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
#   Warning  Unhealthy  ...  Liveness probe failed: Get "http://10.1.2.3:8080/healthz": dial tcp 10.1.2.3:8080: connect: connection refused

# Test the endpoint from inside the cluster, exactly as kubelet would:
kubectl exec -n app web-1 -- wget -qO- --timeout=2 http://127.0.0.1:8080/healthz
kubectl port-forward -n app pod/web-1 8080:8080 &    # backgrounded: port-forward blocks
curl -sv localhost:8080/healthz

# Classic fix: the app takes 40s to boot, liveness starts probing at 0s.
kubectl patch deployment web -n app --type=json -p \
  '[{"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/initialDelaySeconds","value":45}]'

☺ Like you’re 10: Readiness means “don’t send me customers yet.” Liveness means “if I stop answering, please shoot me and start me again.” If you mix them up, a slow-but-healthy app gets shot forever.

Evicted pods and stuck Terminating

An Evicted pod was removed by kubelet under node pressure — memory, disk, or PID exhaustion. The pod object stays around in a Failed phase as a tombstone. Stuck Terminating, by contrast, is nearly always a finalizer: some controller registered a cleanup hook and either the controller is gone or its cleanup keeps failing.

kubectl get pods -A --field-selector=status.phase=Failed
kubectl describe node worker-2 | grep -E 'MemoryPressure|DiskPressure|PIDPressure'
kubectl describe pod -n app evicted-1 | grep -i 'Message'
#   Message: The node was low on resource: ephemeral-storage.

# Stuck Terminating — find out who is holding it:
kubectl get pod -n app stuck-1 -o jsonpath='{.metadata.finalizers}{"\n"}'
kubectl get pod -n app stuck-1 -o jsonpath='{.metadata.deletionTimestamp}{"\n"}'

# Prefer fixing the controller. Clearing a finalizer skips real cleanup (leaked cloud resources!):
kubectl patch pod stuck-1 -n app -p '{"metadata":{"finalizers":null}}' --type=merge

Storage failures

☺ Like you’re 10: A pod that wants a disk waits politely forever until it gets one. Your job is to find out why nobody is handing one over.

Storage failures nearly all present the same way — a pod stuck in Pending or ContainerCreating — so you must go one level down to the PVC and its events to tell them apart. That is the crucial habit: the pod is only the messenger, and describing it forever will not tell you which of the seven storage faults you are looking at. Background on the model is in Storage & State.

The storage failure matrix

PVC Pending — walking the binding chain

Start at the claim and read its events, then look at the class, then at the pool of volumes. The three Pending rows above are told apart entirely by the message on the PVC: a WaitForFirstConsumer notice is benign and means the class is deliberately deferring binding until a pod is scheduled; no persistent volumes available means nothing in the pool matches your size and access mode; and storageclass ... not found means the name is wrong or there is no default class marked.

kubectl get pvc -n app
kubectl describe pvc data-db-0 -n app | tail -15
#   Warning  ProvisioningFailed  ... storageclass.storage.k8s.io "fast-ssd" not found
#   Normal   WaitForFirstConsumer  ... waiting for first consumer to be created before binding

kubectl get sc                       # which class is (default)?
kubectl get pv -o custom-columns=NAME:.metadata.name,CAP:.spec.capacity.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName,STATUS:.status.phase,CLAIM:.spec.claimRef.name

Multi-Attach, full volumes and expansion

A Multi-Attach error means an RWO (ReadWriteOnce) volume is still attached to another node because the old pod has not fully gone — common after a node failure or during a rollout that overlaps. The VolumeAttachment objects tell you who is still holding it. A volume node affinity conflict is the mirror image: the pod landed on a node or zone the PV simply cannot attach to, which is exactly what WaitForFirstConsumer exists to prevent. And when the app itself reports No space left on device, the volume (or the node’s ephemeral storage) is genuinely full — expansion only works if the StorageClass allows it.

# Multi-Attach: find who still holds it
kubectl describe pod -n app db-0 | grep -i multi-attach
kubectl get volumeattachment | grep <pv-name>

# Is the disk simply full?
kubectl exec -n app db-0 -- df -h /var/lib/data

# Expand (only if the StorageClass has allowVolumeExpansion: true)
kubectl patch pvc data-db-0 -n app -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

A StatefulSet blocked on its own PVC

This is the storage failure that looks the most alarming and is the least alarming, because three broken pods are almost always one broken volume.

Where to go next

Workloads and storage are the first branch of the tree, and the one that pays back fastest — the pod-state matrix alone resolves a large share of broken-on-purpose tasks. When the pods are healthy and traffic still fails, or when a forbidden or a webhook denial is in the way, cross to Triage — Networking, RBAC & Admission. When the workload never got deployed in the first place, or the dashboards are empty, or a self-service claim went nowhere, cross to Triage — Delivery, Observability & Platform APIs. The shared diagnostic order, the decision flow and the “when you are stuck” checks all live on the triage hub. For the concepts underneath these symptoms, read the Kubernetes substrate, Scaling & Scheduling and Storage & State; for the muscle memory, drill the Command Reference and work through Practice Tasks before ticking off the final checklist.