Exam Prep · Practice · Observability & Operations

Practice — Observability & Operations

This page holds the six performance-based tasks for the CNPE’s Observability & Operations domain, which is worth 20% of the exam — roughly a fifth of your marks, split between instrumenting the platform so signals exist and operating it by acting on them. Drill them the way the exam will meet you: cold, from an empty terminal with only the official docs open, time-boxed to 5–7 minutes each, verified with the “Done when” command rather than with your eyes — and open the worked solution only after you have genuinely attempted the task and written something, even something wrong. These six are lifted unchanged from the full practice bank, where they sit alongside the other four domains and a 120-minute mock exam.

☺ Explain it like I’m 10

Imagine your house could tell you things: “the tap in the bathroom is dripping,” “the fridge is working twice as hard as normal,” “this room costs the most to heat.” Without those senses you only find out something is wrong when the floor is already wet. These six challenges are about giving a computer platform senses — little meters that count things, a tracker that follows a request as it visits five rooms, and a bill that says which room spent the most — and then using them to find and fix the dripping tap before anyone notices.

🐘Your host for this topic: Ellie the Elephant — she never forgets a metric, a trace or a bill. Ellie remembers exactly what the graph looked like before the incident, which is the whole reason observability beats guessing.

How to drill this domain

☺ Like you’re 10: Set a timer, try it yourself, and only lift the flap once you’ve had a proper go. Peeking early teaches you to read answers, not to write them.

Expect Prometheus Operator objects, PromQL you must write from memory, an OpenTelemetry pipeline, at least one broken pod to diagnose, and something about cost or SLOs. The two halves of the domain fail differently: instrumenting tasks fail silently — your object is valid, created, and completely ignored because a label selector didn’t match — while operating tasks fail loudly and reward whoever knows which single command reveals the cause. Practise both until the reflex is automatic.

Run each task like a sitting. Empty terminal, 5–7 minute timer, official docs only. When the timer rings, stop and record the outcome honestly: solved, solved-but-slow, or failed. Then read the worked solution — including the Why paragraph, which is where the exam-day gotcha usually hides — and put every failure on a list to re-do cold 48 hours later. A task you got wrong is worth ten you got right.

Background reading for this domain: observability, reliability & incidents, FinOps. When a task stalls, reach for the troubleshooting playbook and the command reference rather than the answer key, and triage the broken-workload task with triage: workloads.

Instrumenting — making the signals exist

☺ Like you’re 10: First you have to fit the meters and the trackers. If nothing is counting, nothing can tell you anything later.

The first three tasks build the sensing layer: getting Prometheus to actually scrape a service and alert on it, writing the four golden signals as PromQL you can produce under pressure, and standing up an OpenTelemetry Collector so traces have somewhere to go. Every one of them has a silent-failure mode, so verify with the “Done when” command every single time.

O1 · Scrape a service and make an alert fire

The payments API exposes Prometheus metrics on port 9090 at /metrics, but nothing scrapes it, so the on-call has no idea when it degrades. The cluster runs kube-prometheus-stack with the Prometheus Operator.

Your task:

  1. Create a ServiceMonitor that selects the payments-api Service and scrapes its metrics port every 30 seconds.
  2. Create a PrometheusRule with an alert PaymentsHighErrorRate that fires when the 5xx ratio exceeds 5% for 2 minutes, severity critical.
  3. Confirm the target is up and force the alert to fire.

Done when: the target appears UP in Prometheus (/targets), and curl -s localhost:9090/api/v1/alerts | jq '.data.alerts[].labels.alertname' shows PaymentsHighErrorRate in state firing.

Show the worked solution 
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payments-api
  namespace: payments
  labels:
    release: kube-prometheus-stack     # MUST match the Prometheus serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: payments-api
  namespaceSelector:
    matchNames: [payments]
  endpoints:
    - port: metrics                    # the *name* of the Service port, not the number
      path: /metrics
      interval: 30s
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: payments-rules
  namespace: payments
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: payments.slo
      rules:
        - alert: PaymentsHighErrorRate
          expr: |
            sum(rate(http_requests_total{job="payments-api",code=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{job="payments-api"}[5m])) > 0.05
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "Payments API 5xx ratio above 5%"
            runbook_url: "https://runbooks.acme.io/payments-high-error-rate"
kubectl apply -f servicemonitor.yaml -f prometheusrule.yaml
kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090 &
curl -s 'localhost:9090/api/v1/targets' | jq '.data.activeTargets[].labels.job'
curl -s 'localhost:9090/api/v1/rules'   | jq '.data.groups[].rules[].name'
curl -s 'localhost:9090/api/v1/alerts'  | jq '.data.alerts[]'

Why: the failure that costs people points here is the label selector chain. The Prometheus CR has a serviceMonitorSelector; if your ServiceMonitor’s labels don’t match it, your object is created, valid, and completely ignored. Check it with kubectl -n monitoring get prometheus -o yaml | grep -A5 serviceMonitorSelector before you write the ServiceMonitor. Second gotcha: endpoints[].port takes the Service port’s name.

O2 · Write the four golden signals in PromQL

A new service is onboarding and the team wants a dashboard. You have been asked to supply the four golden-signal queries for a service exposing standard http_requests_total and http_request_duration_seconds_bucket metrics, plus container resource metrics from cAdvisor.

Your task:

  1. Write a PromQL expression for each of latency (p99), traffic, errors (as a ratio) and saturation (CPU against limit).
  2. Verify each returns data against a live Prometheus.
  3. Add a recording rule for the error ratio so the dashboard query stays cheap.

Done when: all four expressions return a non-empty result in the Prometheus expression browser, and job:http_errors:ratio5m appears as a new series after the recording rule loads.

Show the worked solution 
# LATENCY - p99 over 5m, per service (histogram_quantile needs the le label kept)
histogram_quantile(0.99,
  sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))

# TRAFFIC - requests per second
sum by (service) (rate(http_requests_total[5m]))

# ERRORS - fraction of requests returning 5xx
sum by (service) (rate(http_requests_total{code=~"5.."}[5m]))
  /
sum by (service) (rate(http_requests_total[5m]))

# SATURATION - CPU used as a fraction of the container's limit
sum by (pod) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))
  /
sum by (pod) (kube_pod_container_resource_limits{resource="cpu"})
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: golden-signals
  namespace: monitoring
  labels: { release: kube-prometheus-stack }
spec:
  groups:
    - name: golden-signals
      interval: 30s
      rules:
        - record: job:http_errors:ratio5m
          expr: |
            sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
            / sum by (job) (rate(http_requests_total[5m]))

Why: two rules cover most PromQL mistakes. First, always rate() a counter before you sum() it — summing raw counters and then rating gives nonsense whenever a pod restarts. Second, histogram_quantile needs the le label, so it must appear in your sum by (…). Recording rules pre-compute expensive expressions on a schedule so dashboards and alerts read a single cheap series.

O3 · Wire an OpenTelemetry Collector to Jaeger

Three services already emit OTLP traces, but there is nowhere to send them; developers are debugging a cross-service latency problem by reading logs side by side. Jaeger is installed in the observability namespace.

Your task:

  1. Deploy an OpenTelemetry Collector with an otlp receiver on gRPC 4317 and HTTP 4318.
  2. Add a batch processor and a memory_limiter, and export via OTLP to the Jaeger collector.
  3. Send a test span and find it in the Jaeger UI.

Done when: the collector pod is Running with no config errors in its logs, and a trace sent to otel-collector:4317 is searchable by service name in Jaeger.

Show the worked solution 
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel
  namespace: observability
spec:
  mode: deployment
  config:
    receivers:
      otlp:
        protocols:
          grpc: { endpoint: 0.0.0.0:4317 }
          http: { endpoint: 0.0.0.0:4318 }
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 80
        spike_limit_percentage: 20
      batch:
        timeout: 5s
        send_batch_size: 1024
      resource:
        attributes:
          - key: cluster
            value: prod-eu
            action: upsert
    exporters:
      otlp/jaeger:
        endpoint: jaeger-collector.observability:4317
        tls: { insecure: true }
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, resource, batch]   # memory_limiter FIRST
          exporters: [otlp/jaeger]
kubectl apply -f otel-collector.yaml
kubectl -n observability logs deploy/otel-collector | head -30    # config errors surface here

# send a test span
kubectl -n observability run tracegen --rm -it --restart=Never \
  --image=ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest \
  -- traces --otlp-insecure --otlp-endpoint otel-collector:4317 --traces 5

Why: a Collector config is always the same four sections — receivers, processors, exporters, and the service.pipelines block that actually connects them. Defining a receiver but forgetting to list it in a pipeline is the single most common mistake, and it fails silently. Order in processors matters: memory_limiter goes first so it can shed load before anything expensive, and batch goes last, closest to the exporter. Jaeger accepts OTLP natively now, so no jaeger exporter is needed.

🐘 Ellie’s drill · 20 min

Do O1, O2 and O3 back to back on one throwaway cluster, 6 minutes each, timer running, nothing open but the Prometheus Operator and OpenTelemetry docs. Between tasks, close every tab and clear the terminal — the exam does not let you carry momentum. Then, before you look at a single answer key, run only the “Done when” commands and grade yourself on those alone. If a task passed but took ten minutes, that counts as a fail for exam purposes: put it on the re-do list.

Operating — acting on the signals

☺ Like you’re 10: Now that the house can talk, you have to listen and actually fix the dripping tap — and check what it all costs.

The last three tasks are the other half of the domain: diagnosing three differently-broken pods with nothing but kubectl, attributing spend back to the namespace that caused it, and turning dashboards and SLOs into Git artifacts so they survive a pod restart and stop paging people over blips. These are the highest-value minutes on this page, because they are the ones that mirror a real on-call morning.

O4 · Triage three broken pods and remediate

You are handed a namespace with three unhealthy workloads: api is in CrashLoopBackOff, worker keeps being OOMKilled, and report has been Pending for ten minutes. Fix each with the smallest correct change.

Your task:

  1. Determine the root cause of each pod’s state using only kubectl.
  2. Remediate all three so they reach Running and Ready.
  3. State, for each, which command revealed the cause.

Done when: kubectl -n broken get pods shows all three Running with READY 1/1, and kubectl -n broken get events --sort-by=.lastTimestamp shows no new warnings.

Show the worked solution 
# --- the universal triage sequence, in order ---
# each workload is a Deployment, so its pods carry a generated suffix:
# select them with -l app=<name> rather than guessing the pod name.
kubectl -n broken get pods -o wide --show-labels
kubectl -n broken describe pod -l app=api         # Events at the bottom = 80% of answers
kubectl -n broken get events --sort-by=.lastTimestamp | tail -20

# 1. CrashLoopBackOff -> read the previous container's logs
kubectl -n broken logs -l app=api --previous --tail=50
#   "FATAL: env DATABASE_URL not set"  -> missing key in the referenced ConfigMap/Secret
kubectl -n broken get deploy api -o jsonpath='{.spec.template.spec.containers[0].envFrom}'
kubectl -n broken create configmap api-config \
  --from-literal=DATABASE_URL=postgres://api:s3cret@db.broken.svc:5432/api \
  --dry-run=client -o yaml | kubectl -n broken apply -f -
kubectl -n broken rollout restart deploy/api

# 2. OOMKilled -> confirm the reason, then raise the memory limit
kubectl -n broken get pod -l app=worker \
  -o jsonpath='{.items[0].status.containerStatuses[0].lastState.terminated.reason}'
#   OOMKilled
kubectl -n broken set resources deploy/worker --limits=memory=512Mi --requests=memory=256Mi

# 3. Pending -> the scheduler explains itself in the events
kubectl -n broken describe pod -l app=report | grep -A5 Events
#   "0/3 nodes are available: 3 Insufficient cpu"     -> request too large, or quota
kubectl -n broken describe resourcequota
kubectl -n broken set resources deploy/report --requests=cpu=200m

Why: each state has one authoritative source. CrashLoopBackOff means the container started and exited, so the truth is in logs --previous — the current container is too young to have logged it. OOMKilled is a kernel verdict recorded in lastState.terminated.reason, and the fix is a limit change or a memory leak, never a restart. Pending means the pod was never scheduled, so the scheduler’s message in the events tells you exactly which predicate failed — insufficient resources, an unsatisfied nodeSelector or taint, an unbound PVC, or a quota rejection. Learn the trio and you will recognise them instantly under time pressure.

O5 · Attribute spend per namespace with OpenCost

Finance wants to know which of the twelve tenant namespaces drove last month’s 30% cluster cost increase. OpenCost is installed and scraping into Prometheus, but nobody has ever queried it.

Your task:

  1. Query the OpenCost allocation API for the last 7 days, aggregated by namespace.
  2. Produce the same answer in PromQL from OpenCost’s exported metrics.
  3. Identify the top three namespaces by total cost.

Done when: a single command prints a namespace-to-cost mapping, and you can name the top three spenders and say whether the cost is CPU, memory or storage.

Show the worked solution 
kubectl -n opencost port-forward svc/opencost 9003:9003 &

# allocation API - aggregate 7 days of spend by namespace
curl -sG 'http://localhost:9003/allocation' \
  --data-urlencode 'window=7d' \
  --data-urlencode 'aggregate=namespace' \
  --data-urlencode 'accumulate=true' \
| jq -r '.data[0] | to_entries
         | sort_by(-.value.totalCost)
         | .[] | [.key, (.value.totalCost|floor)] | @tsv'

# same shape, by controller, to find the specific workload inside the namespace
curl -sG 'http://localhost:9003/allocation' \
  --data-urlencode 'window=7d' --data-urlencode 'aggregate=controller' | jq .
# PromQL equivalent: monthly CPU + RAM request cost by namespace
sum by (namespace) (
    sum by (namespace, pod) (kube_pod_container_resource_requests{resource="cpu"})
  * on() group_left() avg(node_cpu_hourly_cost) * 730
)
+
sum by (namespace) (
    sum by (namespace, pod) (kube_pod_container_resource_requests{resource="memory"} / 1024^3)
  * on() group_left() avg(node_ram_hourly_cost) * 730
)

Why: OpenCost’s job is to join what ran (Kubernetes resource requests and usage over time) with what it costs (node hourly pricing), then let you slice by any Kubernetes dimension — namespace, controller, label, or a custom aggregate. The allocation API is far quicker than hand-rolled PromQL under exam pressure. Note the finding usually matters more than the number: cost driven by requests rather than usage means over-requesting, which is a rightsizing problem, not a scaling one. See FinOps.

O6 · Ship a dashboard and an SLO burn-rate alert as code

Your Grafana is full of hand-built dashboards that vanished when someone recreated the pod. Meanwhile the payments SLO — 99.9% availability over 30 days — is tracked in a spreadsheet, and the team gets paged for every brief blip.

Your task:

  1. Provision a Grafana dashboard from a ConfigMap using the sidecar label, so it survives a pod restart.
  2. Add a multi-window, multi-burn-rate alert: page on a fast burn (14.4× over 1h and 5m) and ticket on a slow burn (6× over 6h and 30m).
  3. Confirm the dashboard is loaded and the rules are evaluating.

Done when: the dashboard appears in Grafana after kubectl delete pod on Grafana, and both alert rules are listed at /api/v1/rules in Prometheus with a non-null evaluation time.

Show the worked solution 
apiVersion: v1
kind: ConfigMap
metadata:
  name: payments-dashboard
  namespace: monitoring
  labels:
    grafana_dashboard: "1"          # the sidecar watches for exactly this label
data:
  payments.json: |
    { "title": "Payments SLO", "uid": "payments-slo", "panels": [ ... ] }
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: payments-slo
  namespace: monitoring
  labels: { release: kube-prometheus-stack }
spec:
  groups:
    - name: payments.slo.burnrate
      # these read recording rules you must define first - one per window,
      # same shape as job:http_errors:ratio5m in O2 (5m, 30m, 1h, 6h)
      rules:
        - alert: PaymentsErrorBudgetFastBurn
          expr: |
            job:http_errors:ratio5m{job="payments-api"} > (14.4 * 0.001)
            and
            job:http_errors:ratio1h{job="payments-api"} > (14.4 * 0.001)
          for: 2m
          labels: { severity: critical, page: "true" }
          annotations:
            summary: "Burning 30d error budget 14.4x - budget gone in ~2 days"
        - alert: PaymentsErrorBudgetSlowBurn
          expr: |
            job:http_errors:ratio30m{job="payments-api"} > (6 * 0.001)
            and
            job:http_errors:ratio6h{job="payments-api"} > (6 * 0.001)
          for: 15m
          labels: { severity: warning, page: "false" }

Why: the grafana_dashboard: "1" label is the whole trick for dashboards-as-code with kube-prometheus-stack — the sidecar container watches ConfigMaps carrying it and drops the JSON into Grafana’s provisioning directory, so the dashboard is now a Git artifact your GitOps controller reconciles. The multi-window part of the burn-rate alert is what stops flapping: requiring both a long window (is this sustained?) and a short one (is it still happening now?) gives you fast detection without paging on a thirty-second blip. The 0.001 is the error budget for a 99.9% target.

◆ Key idea

Every task on this page is the same loop at a different altitude: make a signal exist, make it trustworthy, then make a decision with it. ServiceMonitor, PromQL, Collector pipeline, triage command, allocation query, burn-rate alert — six mechanisms, one idea. If a task stalls, ask “what signal would tell me the answer, and does it exist yet?” and build backwards from that.

When you have all six clean and inside the clock, go back to the full practice bank and interleave them with the other domains — GitOps & Continuous Delivery, Platform APIs & Self-Service, Platform Architecture & Infrastructure and Security & Policy Enforcement — because context-switching between domains under a single clock is itself a skill the exam tests.

🐢 Timmy’s checkpoint

1. Your ServiceMonitor is created and valid, but the target never appears in Prometheus — what is the first thing you check, and with which command? 2. Why must you rate() a counter before you sum() it, and which label must survive your sum by (…) for histogram_quantile to work? 3. In an OpenTelemetry Collector config, which processor goes first in a pipeline and which goes last, and why? 4. Name the authoritative source of truth for each of CrashLoopBackOff, OOMKilled and Pending. 5. What makes a burn-rate alert multi-window, and what problem does the second window solve?

Check your answers
  1. The label selector chain — the Prometheus CR’s serviceMonitorSelector must match your ServiceMonitor’s labels. Check it before you write the object with kubectl -n monitoring get prometheus -o yaml | grep -A5 serviceMonitorSelector. Second suspect: endpoints[].port takes the Service port’s name, not its number.
  2. Summing raw counters and then rating gives nonsense the moment a pod restarts and its counter resets — rate() handles the reset per-series, so it must come first. histogram_quantile needs the le label, so le must appear in your sum by (…).
  3. memory_limiter goes first so it can shed load before anything expensive runs; batch goes last, closest to the exporter. And whatever you define, it does nothing until it is listed in the service.pipelines block — the most common silent failure in a Collector config.
  4. CrashLoopBackOff → kubectl logs <pod> --previous (the container that actually died). OOMKilled → .status.containerStatuses[0].lastState.terminated.reason, a kernel verdict. Pending → the scheduler’s message in the pod’s Events, which names the exact failed predicate.
  5. It requires both a long window and a short window to be burning — the long window answers “is this sustained?” and the short window answers “is it still happening right now?”, which gives fast detection without paging on a thirty-second blip. The threshold multiplies the error budget (e.g. 14.4 * 0.001 for a 99.9% target).