Observability on Kubernetes

Before you ever install Prometheus or a log backend, Kubernetes already ships four native signals about the health of what's running: a minimal, in-memory metrics-server feeding kubectl top and the autoscaler; probes that continuously judge a container's health, not just restart it; a cluster Events API recording why something happened, with a memory of about an hour; and a per-node logging architecture that captures every container's stdout/stderr to disk, with no aggregation and no retention beyond the node itself. None of the four is a substitute for a real observability stack — this page is explicit about where each one stops and Prometheus, Loki, or Elasticsearch has to pick up. But all four exist on every cluster the moment kubelet starts, they cost nothing to turn on, and knowing exactly what each one does and doesn't remember is what separates a five-minute diagnosis from an hour of guessing.

Why this page stops short of Prometheus and Loki

☺ Like you're 10: This page is about what the school building itself gives you for free — the thermometer, the sign-in sheet, the diaries. The records office down the hall is real, important, and covered elsewhere.

A full observability stack on Kubernetes is usually Prometheus for metrics with real history and alerting, and Loki or Elasticsearch for searchable, durable logs — genuinely deep tools this course covers in their own right: Prometheus and Loki in Platform Engineering's tool guides, with the theory of what to watch and how to alert on it in SRE's Monitoring & Observability (the four golden signals, and the case for symptom-based alerting over cause-based). This page deliberately doesn't re-derive any of that. What it covers instead is the layer underneath — what Kubernetes itself hands you before a single extra Pod is installed, how kubectl top, a probe failure, and kubectl logs actually work under the hood, and exactly where each one's memory runs out. If you already run Prometheus and Loki, this page is still worth reading: it's the substrate they're built on, and every exporter and log shipper you install is reading data structures this page names directly.

metrics-server vs. Prometheus: two different questions

☺ Like you're 10: One tool answers "how much right now?" and forgets the answer the instant something newer comes in. The other answers "how has this changed?" and keeps every answer it's ever been given.

metrics-server is deliberately the smallest thing that could work: one Deployment that scrapes every kubelet's /stats/summary endpoint (itself backed by cAdvisor) on a short interval, holds only the latest CPU and memory number per Pod and Node in memory, and serves it through the metrics.k8s.io/v1beta1 API — a real Kubernetes API resource, not a special code path. Autoscaling: HPA, VPA & Cluster Autoscaler already covers why the HPA depends on this exact API for its Resource-type metrics, and what happens when metrics-server itself falls over; this page's angle is narrower — metrics-server has no history at all. Ask it what a Pod's CPU usage was five minutes ago and there's no answer to give you, because nothing was ever stored past the last scrape.

Prometheus exists to answer everything metrics-server structurally can't: it pulls a /metrics endpoint from every target on a schedule, writes every sample into a local time-series database with real retention (roughly two weeks by default, far longer with a remote-write backend like Thanos or Mimir), and answers arbitrary questions over that history with PromQL. It also isn't limited to CPU and memory — cAdvisor exposes deep per-container resource metrics natively through the kubelet (no separate install), while kube-state-metrics, a separate component you deploy, answers a completely different question: not "how much CPU is this Pod using" but "what does the API server think this object's state should be" — a Deployment's spec.replicas against its status.availableReplicas, a Pod's phase, a Node's conditions. A service can look perfectly healthy on every resource graph metrics-server or cAdvisor can produce while its rollout is silently stuck — that's exactly the gap kube-state-metrics closes, and exactly the reason production clusters run both metrics-server and Prometheus rather than picking one.

kubectl top isn't magic — it's a thin client for the exact same API the HPA reads. You can watch that directly, with no top subcommand involved at all:

# the raw API kubectl top and the HPA both read — one JSON snapshot, no history behind it
$ kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq '.items[] | {name:.metadata.name, cpu:.usage.cpu, mem:.usage.memory}'
{
  "name": "kind-worker",
  "cpu": "187432152n",
  "mem": "612548Ki"
}

$ kubectl top pods -n kube-system --sort-by=cpu --no-headers
metrics-server-6cbc7b8b98-9f2xh   9m     28Mi
coredns-5d78c9869d-4pqvl          3m     14Mi
# every number here is a live re-scrape's worth old, at most — ask again in ten minutes
# and the old numbers aren't sitting anywhere for you to compare against

Probes as a continuous signal, not just a restart trigger

☺ Like you're 10: A probe isn't only the button that restarts a container — it's also constantly leaving a paper trail, and most of that trail has nothing to do with restarting anything.

The full mechanics of liveness, readiness, and startup probes — the exact YAML fields, and what each failure does to a Pod — belong to Workloads & Scheduling's CKA-level treatment, and this page assumes you already have that vocabulary. What that page doesn't dwell on is everything a probe failure leaves observable along the way, which is where this page picks up. A failing readiness probe flips the Pod's Ready condition to False — visible in status.conditions, and in the READY column of a plain kubectl get pods — without touching restartCount at all, because nothing was killed. A failing liveness probe does two things at once: the kubelet emits a Warning/Unhealthy Event ("Liveness probe failed: …"), immediately followed by a second Event, reason Killing, and only that kill increments restartCount. Repeat that cycle enough times in a row and the kubelet stops retrying immediately — it backs off exponentially between restarts (10s, 20s, 40s, 80s, capped at 5 minutes), and CrashLoopBackOff is the name for exactly that backoff state, not a distinct failure mode of its own.

$ kubectl get pod checkout-7d4f9c-8k2wn -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'
Initialized=True
Ready=False
ContainersReady=False
PodScheduled=True

$ kubectl get events --field-selector involvedObject.name=checkout-7d4f9c-8k2wn --sort-by=.lastTimestamp
LAST SEEN   TYPE      REASON      OBJECT                              MESSAGE
41s         Warning   Unhealthy   pod/checkout-7d4f9c-8k2wn   Readiness probe failed: HTTP probe failed with statuscode: 503
9s          Warning   Unhealthy   pod/checkout-7d4f9c-8k2wn   Liveness probe failed: HTTP probe failed with statuscode: 503
8s          Normal    Killing     pod/checkout-7d4f9c-8k2wn   Container checkout failed liveness probe, will be restarted

None of that has to stay trapped in kubectl output. The kubelet exposes its own Prometheus-format metric for exactly this — prober_probe_total, a counter labeled probe_type (Liveness/Readiness/Startup), result (successful/failed/unknown), pod, namespace, and container — scraped straight off the kubelet's own /metrics endpoint, not cAdvisor's. Paired with kube-state-metrics' kube_pod_container_status_restarts_total, that's enough to alert on a probe-failure rate — rate(prober_probe_total{probe_type="Liveness",result="failed"}[5m]) > 0 — well before enough failures accumulate for CrashLoopBackOff to show up in a dashboard someone's actually watching.

Cluster Events: the audit trail with a short memory

☺ Like you're 10: The sign-in sheet by the door writes down real things that really happened — it's just that nobody's keeping it past today.

Every Unhealthy, Killing, Scheduled, BackOff, and FailedScheduling you've ever seen at the bottom of a kubectl describe — the diagnostic ladder Troubleshooting builds its whole "describe before logs" habit around — is a real API object, an Event, living in the events.k8s.io/v1 API group (the legacy v1 core Event type still exists and is what most tooling, including kubectl, still surfaces by default). Each one carries a type (Normal or Warning), a machine-readable reason, an involvedObject reference back to whatever it's about, a human-readable message, and — because the exact same reason firing repeatedly on the same object gets deduplicated into one record with a rising count rather than a flood of near-identical rows — firstTimestamp and lastTimestamp instead of just one.

$ kubectl get events -A --sort-by=.lastTimestamp | tail -5
$ kubectl get events -A -o json \
  | jq '.items[] | select(.type=="Warning") | {reason, count, message, lastTimestamp}'
{
  "reason": "Unhealthy",
  "count": 4,
  "message": "Readiness probe failed: HTTP probe failed with statuscode: 503",
  "lastTimestamp": "2026-08-27T09:41:56Z"
}

The part that catches people who've never had to debug an incident from three hours ago: Events are stored in etcd like everything else, but kube-apiserver's --event-ttl flag defaults to one hour, and apiserver actively garbage-collects anything older than that on its own — it isn't a display filter you can widen after the fact, the object is genuinely gone. On a managed control plane (EKS, GKE, AKS) that flag usually isn't yours to change at all. If an Event explaining why a Pod crash-looped six hours ago is something you'll ever want to look back on, the object itself has already expired by the time most people go looking — the only fix is running a small controller (a purpose-built event exporter is the common choice) that watches the Events API continuously and forwards each one to a durable backend — a log store, a webhook, a Slack channel — before that hour runs out, which turns an Event from a one-hour scratchpad into a permanent, searchable record.

Logging architecture: from a stdout line to a durable backend

☺ Like you're 10: Kubernetes writes every kid's diary to a shelf in that kid's own room. It never builds you the hallway to the archive — someone has to build that, and build it before the room gets cleared out.

Nothing about kubectl logs is centralized. Whatever a container writes to stdout/stderr is captured by the container runtime through the CRI logging interface and written to a plain file on that node's own disk, at /var/log/pods/<namespace>_<pod>_<pod-uid>/<container>/0.log (with legacy /var/log/containers/*.log symlinks kept around purely for older tooling that expects that path), one line per log line, in a fixed CRI format: an RFC 3339 timestamp, the originating stream, a tag — F for a complete line, P for a partial one that got split — then the message itself.

# kind runs each "node" as a Docker container, so its node filesystem is one exec away
$ docker exec kind-worker tail -3 /var/log/pods/default_checkout-7d4f9c-8k2wn_3f8a1c02-*/checkout/0.log
2026-08-27T09:41:03.395515030Z stdout F {"level":"info","msg":"listening on :8080"}
2026-08-27T09:41:52.812004112Z stderr F panic: dial tcp 10.96.4.11:5432: connect: connection refused
2026-08-27T09:41:52.812511008Z stderr F [signal SIGABRT: signal SIGABRT]

$ kubectl logs checkout-7d4f9c-8k2wn --previous   # same file, read the friendly way

Left alone, a chatty container would happily fill a node's disk one line at a time, so the kubelet enforces per-container rotation itself — not logrotate, kubelet's own config — via containerLogMaxSize (default 10Mi per file) and containerLogMaxFiles (default 5 kept, oldest deleted once the count is exceeded). That caps disk usage; it does nothing for retention across a Pod's lifetime. The instant a Pod is deleted, evicted, or its node is replaced, those files go with it — there is no built-in history layer here the way Prometheus's TSDB gives you one for metrics.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
containerLogMaxSize: 10Mi
containerLogMaxFiles: 5

Two patterns get logs off the node before that happens, and production clusters mostly reach for the first: a node-level DaemonSet agent — Fluent Bit, Fluentd, Vector, or Grafana Alloy (Promtail's successor) — running exactly one per node, tailing /var/log/pods directly, cross-referencing the Kubernetes API to attach Pod, namespace, and label metadata to every line, then forwarding batches to a backend like Loki or an Elasticsearch-family store (the ELK stack covers that Filebeat-based version of this same pattern in DevOps). One agent per node means N Pods on that node cost one shipper process, not N — cheap, and the default reasonable choice. The alternative is a sidecar in the Pod itself: either a thin "streaming" sidecar that just tails a file the app writes to instead of stdout and re-emits it to its own stdout — a trick that lets the node agent still pick it up for free, useful only when the app can't be made to log to stdout directly — or a full shipping sidecar that forwards logs itself, bypassing the node agent entirely for one Pod's worth of extra isolation or custom parsing, at the cost of one more running container, and one more thing to keep healthy, per Pod that needs it.

Putting the four signals together

☺ Like you're 10: None of these four tools is the whole picture by itself — but stacked together, in order, they usually tell you the whole story before you even need to open a dashboard.

A crash-looping Pod is the cleanest example of all four signals earning their keep in sequence, and it's exactly the same order Troubleshooting's get → describe → logs → exec ladder already teaches, now visible as four distinct native signals rather than one flat habit. kubectl top pod rules out (or confirms) the node simply being starved of CPU or memory. kubectl describe pod surfaces the Ready condition and the Events trail — Unhealthy, then Killing, with a rising count if it's been happening for a while — telling you which probe failed and how often, no logs needed yet. kubectl logs --previous reads the exact node-local file this page just walked through, and shows you the actual panic or stack trace the process left behind before it died. Only once that picture still doesn't add up do you reach for the layer above — Prometheus to ask "was this gradual, and when precisely did it start," Loki to ask "is this happening on any other Pod, and did it start right after a deploy." Nothing about that escalation is specific to Kubernetes; it's the same symptom-first instinct the four golden signals teach for any service. What's specific to Kubernetes is that the first three steps — top, describe, logs — are already sitting on every cluster you'll ever touch, free, the moment kubelet starts, well before anyone's installed anything with the word "observability" in its Helm chart.