Observability & Operations

A platform you can’t see is a platform you can’t trust. This domain is Ellie’s watchtower: the metrics, logs, and traces that let you understand what a running system is doing, the dashboards and alerts that surface trouble, the SLOs and delivery metrics that prove the platform is actually good, and the hands-on triage that gets a broken pod healthy again — fast. It’s 20% of the exam and the most operationally practical part of the whole cert, so we’ll be concrete: real Prometheus rules, real PromQL, and a real kubectl flow you can run under time pressure.

Observability vs monitoring

☺ Like you’re 10: Monitoring answers questions you thought of ahead of time. Observability lets you ask brand-new questions after something surprises you — without shipping new code first.

People use the words interchangeably, but the exam wants you to feel the difference. Monitoring is checking a predefined set of conditions you already knew to watch: CPU above 90%, disk nearly full, the health check failing. It’s a dashboard of gauges you decided on in advance. Observability is a property of the system — how well its external outputs (metrics, logs, traces) let you infer what’s happening inside, including states you never anticipated. Monitoring tells you that something is wrong; observability lets you explore why.

Known-unknowns vs unknown-unknowns

☺ Like you’re 10: Some problems you can guess in advance and set an alarm for. Others you’ve never seen before — and those are the scary ones.

Monitoring is built for known-unknowns: failure modes you can predict, so you pre-build a check and a threshold (“alert if the queue is longer than 1,000”). Distributed cloud-native systems mostly break in unknown-unknowns — novel, emergent failures nobody predicted: a rare interaction between two services under a specific traffic pattern, a slow dependency three hops away, a single misbehaving tenant. You can’t pre-write a dashboard for a bug you’ve never imagined. Observability’s promise is that with rich enough telemetry — especially high-cardinality data you can slice by user, version, region, endpoint — you can debug a novel problem without shipping a new build to add a log line.

Why cloud-native forces the shift

A monolith on one server had one place to look. A cloud-native platform is dozens of services, hundreds of ephemeral pods, and autoscalers moving workloads across nodes every minute. A single user request now fans out across many services, so “which box is slow?” is the wrong question — the request itself is the unit that’s slow, and it crossed ten processes to get that way. Pods are cattle: the one that failed may be gone before you can SSH to it. That’s why the three pillars exist — metrics to see the shape of the whole, logs to read the detail, traces to follow one request across every hop — and why you export telemetry out to durable backends before the pod that produced it disappears.

The three pillars: metrics, logs, and traces

☺ Like you’re 10: Three different kinds of sensor. Numbers over time (metrics), written diary entries (logs), and a GPS trail that follows one request through the whole city (traces).

Observability rests on three complementary signal types. None replaces the others — you use them together, jumping between them during an incident. On the exam’s tool list, Prometheus, OpenTelemetry, Grafana, and Jaeger all live in this domain.

Metrics — Prometheus and the pull model

☺ Like you’re 10: Prometheus is a robot that walks around every minute and reads the numbers off each service’s little scoreboard, then remembers them forever so you can chart them.

Metrics are cheap, numeric time series — a request counter, a latency histogram, memory-in-use — sampled continuously and stored with labels. Prometheus is the de-facto standard, and its defining trait is the pull model: rather than services pushing numbers to a server, Prometheus scrapes an HTTP /metrics endpoint on each target on a schedule. Pull means Prometheus controls the load, can tell instantly when a target is down (the scrape fails), and needs no credentials handed out to every app. Things that can’t expose metrics natively get an exporter that translates them — node_exporter for machine stats, kube-state-metrics for Kubernetes object state, blackbox_exporter for probing endpoints.

On Kubernetes you rarely edit Prometheus config by hand. The Prometheus Operator (the heart of the kube-prometheus-stack) gives you CRDs so scrape config becomes declarative and GitOps-friendly: a ServiceMonitor (or PodMonitor) selects which Services to scrape by label, and a PrometheusRule holds your recording and alerting rules. Add a Service with the right label and it’s monitored automatically — no ticket, no restart.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout
  namespace: monitoring
  labels:
    release: kube-prometheus-stack   # must match your Helm release name, so the Prometheus adopts it
spec:
  selector:
    matchLabels:
      app: checkout                  # scrape every Service labelled app=checkout
  namespaceSelector:
    matchNames: [ checkout ]
  endpoints:
    - port: metrics                  # the *named* Service port to scrape
      path: /metrics
      interval: 30s

PromQL is how you ask questions of that data. rate() over a counter gives per-second throughput; histogram_quantile() turns a latency histogram into a p99; label matchers slice by code, job, or route. Recording rules precompute expensive expressions into new series (so dashboards and alerts stay fast), and alerting rules fire when an expression stays true for a duration. Here are the golden signals as real PromQL:

# Traffic — requests per second over the last 5 minutes
sum(rate(http_requests_total{job="checkout"}[5m]))

# Errors — fraction of responses that are 5xx
sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
  / sum(rate(http_requests_total{job="checkout"}[5m]))

# Latency — 99th-percentile request duration from a histogram
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket{job="checkout"}[5m])) by (le))

# Saturation — fraction of node memory in use
1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

Logs — Loki, Fluent Bit, and structured logging

☺ Like you’re 10: Logs are the diary each service scribbles. If everyone writes in the same neat format (not messy handwriting), you can search the whole city’s diaries at once.

Logs are timestamped event records — the detail you read once a metric tells you where to look. The modern cloud-native default is Loki (from Grafana Labs), deliberately built “like Prometheus, but for logs”: it indexes only a small set of labels (namespace, app, pod) rather than the full text of every line, which makes it dramatically cheaper to run. You query it with LogQL, whose label selectors mirror PromQL, so jumping from a metric spike to the matching logs feels seamless. A collection agent runs on every node as a DaemonSet and ships container logs to the store: Fluent Bit (tiny, fast, written in C — the common choice on Kubernetes), Fluentd (heavier, huge plugin ecosystem), or Grafana Alloy (Grafana’s current collector, which supersedes its older Promtail agent).

The single highest-leverage habit is structured logging: emit logs as JSON key-value pairs (level, msg, trace_id, tenant, duration_ms) instead of free-form prose. Structured logs are machine-parseable, so you can filter and aggregate them like data — and including the trace_id is what lets you pivot from a log line straight to the full distributed trace.

Traces — OpenTelemetry, Jaeger, and Tempo

☺ Like you’re 10: A trace is a GPS trail for one request as it hops from service to service, so you can see exactly which stop made it slow.

Distributed tracing answers the question metrics and logs can’t: for a single slow request that touched ten services, where did the time go? A trace is a tree of spans; each span is one unit of work (an HTTP call, a DB query) with a start, a duration, and attributes. What stitches spans across process boundaries is context propagation: the trace ID and parent span ID travel with the request, conventionally in the W3C traceparent header, so a downstream service knows it’s continuing the same trace.

OpenTelemetry (OTel) is the CNCF standard that unifies all of this: one vendor-neutral set of APIs, SDKs, and the OpenTelemetry Collector for generating and shipping all three signals — traces, metrics, and logs. You instrument once with OTel and stay free to switch backends. The trace backends — storage, query, and UI — are typically Jaeger (a CNCF-graduated project) or Tempo (Grafana’s object-storage-backed store that’s cheap because it indexes by trace ID and leans on your metrics and logs to find the ID). That “instrument with OTel, store in Jaeger/Tempo, view in Grafana” chain is exactly the diagram above, and a favourite exam pairing.

Grafana — the single pane of glass

☺ Like you’re 10: Grafana is the big wall of screens in the control room where all three kinds of sensor show up together, so you never juggle three different tools during a fire.

Grafana is the visualization layer that ties the pillars together. It connects to many data sources — Prometheus for metrics, Loki for logs, Tempo/Jaeger for traces — and renders dashboards, tables, and heatmaps over all of them. Its real power during an incident is correlation: you spot a latency spike on a Prometheus panel, click through to the Loki logs for that exact window, and follow an exemplar straight into the Tempo trace for one slow request — three pillars, one workflow, no context-switching. Dashboards are just JSON, so you keep them in Git and roll them out with GitOps like everything else. (Grafana visualizes; the alerting engine underneath is usually Prometheus + Alertmanager, up next.)

What to measure: golden signals, RED, and USE

☺ Like you’re 10: You can measure a million things, so smart people picked the tiny handful that actually predict trouble. Watch those first.

A blank Grafana is paralysing — you can graph anything. Three well-known frameworks tell you what to graph. Google’s SRE book names four golden signals for any user-facing service:

Two narrower cousins are worth knowing by name. RED (Tom Wilkie) — Rate, Errors, Duration — is the request-centric view, perfect for a microservice: how many requests, how many failed, how long they took. USE (Brendan Gregg) — Utilization, Saturation, Errors — is the resource-centric view, perfect for infrastructure: for each resource (CPU, disk, network) how busy is it, how much work is queued, and is it throwing errors. Rule of thumb: RED for your services, USE for your nodes and hardware, golden signals as the umbrella over both.

Alerting without the fatigue

☺ Like you’re 10: An alarm should only ring when a real person is actually being hurt — not every time a gauge twitches. Too many false alarms and everyone stops listening.

Prometheus evaluates alerting rules and forwards firing alerts to Alertmanager, which decides what actually reaches a human. Alertmanager’s jobs are worth memorising:

The rule that separates a calm on-call from a miserable one: alert on symptoms, not causes. Page on what the user feels — high error rate, high latency, the SLO burning — not on internal causes like “CPU is 85%.” High CPU that harms nobody is not an emergency; a symptom-based page is almost always actionable. Here’s a PrometheusRule with a recording rule feeding a symptom-based, error-budget-aware alert:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: checkout-slo
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: checkout.slo
      rules:
        # recording rules: precompute the 5xx ratio once, reuse everywhere
        - record: job:slo_errors:ratio_rate5m
          expr: |
            sum by (job) (rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
              / sum by (job) (rate(http_requests_total{job="checkout"}[5m]))
        - record: job:slo_errors:ratio_rate1h
          expr: |
            sum by (job) (rate(http_requests_total{job="checkout",code=~"5.."}[1h]))
              / sum by (job) (rate(http_requests_total{job="checkout"}[1h]))
        # fast-burn symptom alert: BOTH windows above 14.4x a 99.9% budget → page a human
        # (the long window says it is real, the short window says it is still happening)
        - alert: CheckoutErrorBudgetFastBurn
          expr: |
            job:slo_errors:ratio_rate1h > (14.4 * 0.001)
              and job:slo_errors:ratio_rate5m > (14.4 * 0.001)
          for: 2m
          labels:
            severity: page
          annotations:
            summary: "Checkout is burning its error budget fast"
            description: "5xx ratio {{ $value | humanizePercentage }} over 1h — 14.4x the 99.9% SLO burn rate."

Platform efficiency: SLIs, SLOs, error budgets, and DORA

☺ Like you’re 10: Instead of chasing “zero bugs forever” (impossible), you agree on a target like “works 99.9% of the time,” and the 0.1% you’re allowed to miss becomes a budget you get to spend on shipping new things.

This is the second competency — measuring and improving platform efficiency with deployment metrics and performance indicators. It has two halves: reliability targets (SLI/SLO/error budgets) and delivery metrics (DORA).

SLI, SLO, and the error budget

An SLI (Service Level Indicator) is a measured number: the fraction of requests that are fast and successful — e.g. good requests / total requests. An SLO (Service Level Objective) is your target for that SLI over a window: “99.9% of checkout requests succeed under 300 ms, measured over 30 days.” The genius follows: 100% − SLO = your error budget. A 99.9% SLO grants a 0.1% budget — about 43 minutes of badness per 30-day month. That budget reframes reliability from a moral fight (“never break anything”) into an economic one: as long as budget remains, you’re free to ship fast and take risks; when it’s spent, you freeze features and stabilise. It turns Dev-vs-Ops tension into one shared number.

You alert on the burn rate — how fast the budget is draining — not on a raw threshold. The Google SRE workbook’s multi-window, multi-burn-rate pattern is the exam-relevant recipe: a fast-burn alert (e.g. 14.4× the budget over a 1-hour and 5-minute window) pages immediately because you’d exhaust a month’s budget in ~2 days; a slow-burn alert (e.g. 6× over 6 hours) opens a ticket. Two windows keep it both urgent and un-flappy. This is symptom-based alerting done right — it fires on user-visible pain, scaled to how much reliability you’re actually losing.

DORA — measuring delivery, not just uptime

☺ Like you’re 10: Four scorecards for how well the team ships: how often, how fast, how often it breaks, and how quickly you recover. A good platform makes all four better.

SLOs prove the platform is reliable; DORA metrics (from the DevOps Research & Assessment program, the Accelerate research) prove it’s fast — and they’re the clearest evidence that your platform investment is paying off. There are four, split into throughput and stability:

Notice how tightly this couples to the rest of the course: deployment frequency and lead time are what GitOps and CI/CD buy you; change failure rate is what canaries and automated analysis hold down; and MTTR is exactly this domain — the faster your traces and dashboards find the cause, the faster you restore. A platform team that can move all four in the right direction has quantitative proof it’s helping Dot ship.

Diagnosing and remediating incidents

☺ Like you’re 10: When a pod is sick, you don’t guess — you ask it three questions in order: what’s your status, what happened (events), and what did you say before you died (logs).

This is the hands-on exam skill: a pod is unhealthy and you have minutes to fix it. Resist the urge to guess. Run a disciplined triage flow — the events at the bottom of describe and the logs of the crashed container solve the large majority of cases.

# 1. Wide view: which pods are unhealthy? how many restarts? which node?
kubectl get pods -n checkout -o wide

# 2. The single most useful command — events are at the BOTTOM of describe
kubectl describe pod checkout-7d9f8b6c5-abcde -n checkout

# 3. Logs from the current AND the previous (crashed) container
kubectl logs checkout-7d9f8b6c5-abcde -n checkout
kubectl logs checkout-7d9f8b6c5-abcde -n checkout --previous   # what it said before it died

# 4. Cluster-level events, newest last (scheduling, image, volume failures)
kubectl get events -n checkout --sort-by=.lastTimestamp

# 5. Resource pressure? (needs metrics-server)
kubectl top pod -n checkout

Those five commands map onto a small set of failure signatures you should recognise on sight. This table is the highest-value thing on the page for the performance-based exam:

For a genuinely novel failure — no familiar signature — this is where observability earns its keep: the dashboards narrow where, the logs tell you what, and a trace shows the exact hop that broke, so you diagnose an unknown-unknown without redeploying to add instrumentation. And remember the platform-native fix: because everything is reconciled from Git, your fastest remediation for a bad rollout is often to revert the PR and let the reconciler restore the last-good state.

Observability is the watchtower that makes every other domain safe to move fast in: GitOps can auto-heal because Ellie can see drift, canaries can promote themselves because she can read the metrics, and the platform can prove its worth because she measures the SLOs and DORA. Next, Timmy adds the guardrails — including the audit logs that make “who changed what” answerable — so speed never outruns safety.