The Capstone · Part 5 of 6 · pairs with Observability & Operations (D4)

Capstone Part 5: Observability Stack & Triage

Four parts in, your platform-dev cluster ships itself, provisions its own databases, and hands developers a self-service button — and you still can't answer "is ledger actually okay right now?" without guessing. This part gives Ellie her watchtower: kube-prometheus-stack installed the same GitOps way as everything else in this capstone, a real /metrics endpoint added to ledger itself, a ServiceMonitor that finds it, a golden-signals Grafana dashboard and a PrometheusRule shipped as code, and then — because a dashboard nobody has ever watched fail is a dashboard you don't trust — two deliberate incidents you cause on purpose and triage with nothing but kubectl describe and logs --previous. This is the hands-on twin of Observability & Operations, CNPE Domain 4 at 20% of the exam.

⚖ CNPA vs CNPE — That 20% weighting, and this whole build-it-then-break-it drill, is CNPE-specific — CNPA is closed-book multiple-choice with no cluster and no lab component at all. But golden signals, what a ServiceMonitor and PrometheusRule do, and the OOMKilled-vs-CrashLoopBackOff distinction are still fair game for CNPA's closed-book recall, just tested as concepts rather than something you diagnose live.

⚠ Where you are arriving from, and where you're headed

Arriving: the platform-dev cluster with Argo CD's root App-of-Apps reconciling apps/ in your platform-capstone repo (Part 1); ledger shipping through an Argo Rollouts canary from a Tekton pipeline that builds registry.local:5001/ledger (Part 2); a LedgerDatabase custom resource reconciled by the ledger-db-operator (Part 3); and Backstage running with ledger registered in the Software Catalog alongside a second, template-scaffolded service called invoices (Part 4). Nothing is instrumented. Nothing is scraped. If ledger silently starts failing half its requests right now, the only way you'd know is a developer complaining. Leaving this page: kube-prometheus-stack running in platform, reconciled by Argo CD like everything else; ledger itself carrying a real /metrics endpoint; a ServiceMonitor scraping it; a golden-signals Grafana dashboard and a symptom-based PrometheusRule both shipped as Git-committed manifests, not clicked together by hand; and two incidents you triggered and diagnosed yourself — an OOMKilled pod and a CrashLoopBackOff pod — using only kubectl describe and logs --previous. Part 6 picks up exactly here and adds the policy layer on top of a platform you can finally see.

☺ Explain it like I'm 10

You've built a toy factory (Part 2), taught it to invent new toy parts (Part 3), and put a friendly button on the front (Part 4) — but nobody's watching the factory floor. Today Ellie the elephant moves in. She wires a thousand tiny sensors onto ledger so she can see its pulse (metrics), builds a big screen showing the four things that matter most (a dashboard), and rigs a bell that only rings when something is actually wrong (an alert). Then — because a fire alarm nobody has ever heard go off might just be broken — you and Ellie set two small, controlled fires on purpose and time how fast you can find them with nothing but your own two eyes on kubectl.

🐘Your host for this part: Ellie the Elephant — she never forgets a metric, log, or incident. Everything in this part is Ellie building her watchtower over ledger, then testing it against a fire she lit herself.

⚠ Read this before Milestone 1

Same ground rules as every part of this capstone: everything below runs on the local, throwaway platform-dev cluster, nothing touches production or costs money, and chart versions, image tags and CRD API versions drift — treat every manifest below as the shape of the answer and check kubernetes.io/docs, kubernetes.io/blog, and the Prometheus/Grafana tool pages if something 404s. The one genuinely new wrinkle this part adds: you are about to edit ledger-src for the first time since Part 2 to add instrumentation, then ship it through the exact same pipeline you already built. If that pipeline is rusty, re-read Part 2 before Milestone 4.

What this part assumes and what it produces

☺ Like you're 10: Everything from before still runs exactly the same — you're only adding sensors and a screen.

You need everything from Parts 1–4 still running: platform-dev, Argo CD reconciling from platform-capstone, the ledger Rollout healthy, the ledger-db-operator and its LedgerDatabase Ready, and Backstage with ledger and invoices in the catalog. You also need Helm installed locally (Argo CD renders the chart for you, but Helm's own CLI is handy for inspecting values.yaml when something doesn't match this page), and push access to your ledger-src and platform-capstone repos from Part 2. Nothing from Parts 1–4 gets restructured — this part only adds a new folder and touches one existing file in each of two repos.

This page assumes you've read Observability & Operations — golden signals, the three pillars, PromQL, symptom-based alerting, and the kubectl triage flow all come from there. This page is the hands; that lesson is the theory. If you haven't worked the standalone Observability Labs, this part is a faster, narrower version of the same muscle, aimed at one real service instead of a throwaway sandbox.

Extending the shared world model

Four new names join the table every earlier part has kept current:

Thing	Name	Introduced
The metrics stack	`kube-prometheus-stack` Helm release, in the `platform` namespace	Part 5 — this page
The scrape target	`ledger`'s own `/metrics` endpoint, added to `ledger-src`	Part 5
The discovery object	`ServiceMonitor/ledger`, in the `ledger` namespace	Part 5
The add-on manifests	`platform-addons/observability/` in `platform-capstone`	Part 5 (folder named back in Part 1)

◆ Key idea

Notice the pattern holding since Part 1: every new capability arrives as one more file under apps/, reconciled by the same root you applied by hand exactly once. Today that means two new files — apps/kube-prometheus-stack.yaml and apps/ledger-observability.yaml — and the root fans them out automatically. You will not run a single helm install or kubectl apply against a monitoring object anywhere on this page.

Installing kube-prometheus-stack the GitOps way

☺ Like you're 10: Instead of clicking through an installer, you write down "I want the sensor kit" on the poster, and the robot builds it.

kube-prometheus-stack bundles the Prometheus Operator, Prometheus, Alertmanager, Grafana, kube-state-metrics and node-exporter into one Helm chart. Because Argo CD can render a Helm chart directly from its repository — no separate helm install, no drift between what you ran once and what's in Git — you install it exactly like every other add-on in this capstone: a new Application under apps/.

# apps/kube-prometheus-stack.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kube-prometheus-stack
  namespace: platform
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://prometheus-community.github.io/helm-charts
    chart: kube-prometheus-stack
    targetRevision: 65.*                     # a semver range; pin an exact version once you're happy
    helm:
      releaseName: kube-prometheus-stack     # every default Service/Secret name below assumes this
      valuesObject:
        prometheus:
          prometheusSpec:
            serviceMonitorSelector:
              matchLabels:
                release: kube-prometheus-stack
            ruleSelector:
              matchLabels:
                release: kube-prometheus-stack
        grafana:
          defaultDashboardsEnabled: true
  destination:
    server: https://kubernetes.default.svc
    namespace: platform
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Commit and push it. Because the root App-of-Apps from Part 1 already recurses over apps/, this is the entire installation:

git add apps/kube-prometheus-stack.yaml
git commit -m "install kube-prometheus-stack via GitOps"
git push

kubectl -n platform get applications kube-prometheus-stack -w
# NAME                     SYNC STATUS   HEALTH STATUS
# kube-prometheus-stack    Synced        Healthy

kubectl -n platform get pods
# prometheus-kube-prometheus-stack-prometheus-0        Running
# kube-prometheus-stack-grafana-...                    Running
# alertmanager-kube-prometheus-stack-alertmanager-0    Running
# kube-prometheus-stack-kube-state-metrics-...         Running
# kube-prometheus-stack-prometheus-node-exporter-...   Running

Open three port-forwards and keep them running in three tabs for the rest of this part — nearly every "done when" below is a curl against one of them, because a screenshot proves nothing and a query result proves everything:

kubectl -n platform port-forward svc/kube-prometheus-stack-prometheus 9090:9090
kubectl -n platform port-forward svc/kube-prometheus-stack-grafana 3000:80
kubectl -n platform port-forward svc/kube-prometheus-stack-alertmanager 9093:9093

# Grafana's generated admin password (user: admin)
kubectl -n platform get secret kube-prometheus-stack-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d ; echo

⚠ The two selector fields matter more than they look

serviceMonitorSelector and ruleSelector both require the label release: kube-prometheus-stack. Skip this and Prometheus's default install still finds some ServiceMonitors — the ones the chart itself ships — but silently ignores every ServiceMonitor and PrometheusRule you write yourself, with no error anywhere. This single missing label is the most common reason "I added a ServiceMonitor and nothing showed up." Remember it before Milestone 7.

Instrumenting ledger: a real `/metrics` endpoint

☺ Like you're 10: A sensor kit is useless pointed at a toy with no sensors built in — so first you build the sensors into the toy itself.

Part 2's ledger-src app has a health check and a balance endpoint, and nothing Prometheus can scrape. Before anything else on this page can work, you have to open that repo again — for the first time since Part 2 — and add real instrumentation with the standard prometheus_client library: a request counter labelled by method, path and status code, and a latency histogram. This is the exact shape the paired lesson's golden-signals PromQL assumes.

# ledger-src/app.py — Part 2's file, with instrumentation added
import os
import random
import time
from flask import Flask, jsonify, request, Response
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST

app = Flask(__name__)
BUGGY = os.environ.get("LEDGER_BUGGY", "false").lower() == "true"
VERSION = os.environ.get("LEDGER_VERSION", "dev")

REQUESTS = Counter(
    "http_requests_total", "Total HTTP requests", ["method", "path", "code"]
)
LATENCY = Histogram(
    "http_request_duration_seconds", "Request latency in seconds", ["method", "path"]
)

@app.before_request
def _start_timer():
    request._start_time = time.time()

@app.after_request
def _record_metrics(response):
    if request.path != "/metrics":              # don't let scrapes pollute their own metrics
        REQUESTS.labels(request.method, request.path, response.status_code).inc()
        LATENCY.labels(request.method, request.path).observe(time.time() - request._start_time)
    return response

@app.get("/healthz")
def healthz():
    return jsonify(status="ok"), 200

@app.get("/metrics")
def metrics():
    return Response(generate_latest(), mimetype=CONTENT_TYPE_LATEST)

@app.get("/")
def ledger():
    if BUGGY and random.random() < 0.7:
        # the "bad" build from Part 2: about 70% of requests fail on purpose
        return jsonify(error="ledger entry corrupted", version=VERSION), 500
    return jsonify(service="ledger", version=VERSION, balance=4200), 200

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

# ledger-src/requirements.txt — one new line
flask==3.0.3
prometheus-client==0.20.0

Commit and push ledger-src, then ship it exactly the way Part 2 taught — run ledger-ci again with a new tag. Nothing about the pipeline changes; you're only proving, again, that the platform you already built handles this like any other release:

git -C ledger-src add app.py requirements.txt
git -C ledger-src commit -m "add /metrics endpoint (prometheus_client)"
git -C ledger-src push

kubectl create -f - <<'EOF'
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: ledger-ci-run-
  namespace: platform
spec:
  pipelineRef: { name: ledger-ci }
  taskRunTemplate: { serviceAccountName: ledger-ci }
  params:
    - { name: repo-url, value: "https://github.com/YOU/ledger-src.git" }
    - { name: config-repo-url, value: "https://github.com/YOU/platform-capstone.git" }
    - { name: image, value: "registry.local:5001/ledger" }
    - { name: tag, value: "1.4.0" }
    - { name: build-arg-buggy, value: "false" }
  workspaces:
    - { name: source, volumeClaimTemplate: { spec: { accessModes: ["ReadWriteOnce"], resources: { requests: { storage: 1Gi } } } } }
    - { name: config, volumeClaimTemplate: { spec: { accessModes: ["ReadWriteOnce"], resources: { requests: { storage: 256Mi } } } } }
EOF

kubectl argo rollouts get rollout ledger -n ledger --watch

Watch 1.4.0 promote itself through the canary exactly like every previous release — 10% → 50% → 100% — and confirm the new endpoint is live before you go any further:

kubectl -n ledger port-forward svc/ledger 8080:80
curl -s http://localhost:8080/metrics | grep http_requests_total
# http_requests_total{code="200",method="GET",path="/"} 3.0
# http_requests_total{code="200",method="GET",path="/healthz"} 12.0

Giving Prometheus a target: the ServiceMonitor

☺ Like you're 10: A label on the front door that tells the sensor robot "scrape here, on this port, at this path."

Three Services already front ledger since Part 2 — ledger, ledger-stable, ledger-canary — all selecting the same pods. Scraping all three would triple-count every series, so patch only the plain ledger Service: name its port and give it a label the ServiceMonitor can key on that ledger-stable/ledger-canary don't carry.

# ledger/service.yaml — only the FIRST Service changes; ledger-stable and ledger-canary are untouched
apiVersion: v1
kind: Service
metadata:
  name: ledger
  namespace: ledger
  labels:
    app: ledger
    monitoring: scrape          # NEW — the one label that distinguishes this Service from its siblings
spec:
  selector: { app: ledger }
  ports:
    - name: http                # NEW — named so the ServiceMonitor can reference it by name
      port: 80
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: ledger-stable
  namespace: ledger
spec:
  selector: { app: ledger }
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: ledger-canary
  namespace: ledger
spec:
  selector: { app: ledger }
  ports:
    - port: 80
      targetPort: 8080

Add the ServiceMonitor and the PrometheusRule (next section) under the platform-addons/observability/ folder Part 1 named in advance, plus one more Argo CD Application to reconcile that folder:

platform-capstone/
├── apps/
│   ├── ledger.yaml
│   ├── kube-prometheus-stack.yaml           # from the previous section
│   └── ledger-observability.yaml            # NEW — points Argo CD at platform-addons/observability/
├── ledger/
│   ├── service.yaml                          # patched above
│   ├── rollout.yaml
│   └── analysistemplate.yaml
└── platform-addons/
    └── observability/                        # NEW — the folder Part 1 named in advance
        ├── servicemonitor.yaml
        ├── prometheusrule.yaml
        ├── grafana-dashboard-configmap.yaml
        └── kustomization.yaml

# platform-addons/observability/servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ledger
  namespace: ledger
  labels:
    release: kube-prometheus-stack       # required — see the warning above
spec:
  selector:
    matchLabels:
      monitoring: scrape                 # matches ONLY the plain ledger Service
  namespaceSelector:
    matchNames: [ ledger ]
  endpoints:
    - port: http                         # the named port from ledger/service.yaml
      path: /metrics
      interval: 30s

# apps/ledger-observability.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ledger-observability
  namespace: platform
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/YOU/platform-capstone.git
    targetRevision: main
    path: platform-addons/observability
  destination:
    server: https://kubernetes.default.svc
    namespace: ledger                    # fallback only — every manifest below sets its own namespace
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [ CreateNamespace=true ]

# platform-addons/observability/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - servicemonitor.yaml
  - prometheusrule.yaml
  - grafana-dashboard-configmap.yaml

Push all of it, then prove Prometheus actually found the target — not "the ServiceMonitor object exists," but "Prometheus is really scraping it":

git add ledger/service.yaml platform-addons/observability apps/ledger-observability.yaml
git commit -m "wire up ledger monitoring: ServiceMonitor + PrometheusRule + dashboard"
git push

kubectl -n platform get applications ledger-observability -w

curl -sG http://localhost:9090/api/v1/targets --data-urlencode 'state=active' \
  | jq '.data.activeTargets[] | select(.labels.job=="ledger") | {health, lastError}'
# {"health":"up","lastError":""}

curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=http_requests_total{namespace="ledger"}' \
  | jq '.data.result | length'
# a number greater than zero

Golden signals in PromQL, and the dashboard as code

☺ Like you're 10: Four questions worth asking any service, turned into four little math sentences Prometheus can answer.

With real data flowing, ask the four questions the paired lesson teaches — traffic, errors, latency, saturation — against ledger specifically:

# Traffic — requests per second, last 5 minutes
sum(rate(http_requests_total{namespace="ledger"}[5m]))

# Errors — fraction of responses that are 5xx
sum(rate(http_requests_total{namespace="ledger",code=~"5.."}[5m]))
  / sum(rate(http_requests_total{namespace="ledger"}[5m]))

# Latency — 99th percentile request duration
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket{namespace="ledger"}[5m])) by (le))

# Saturation — ledger's own container memory vs its limit
sum(container_memory_working_set_bytes{namespace="ledger",container="ledger"})
  / sum(kube_pod_container_resource_limits{namespace="ledger",container="ledger",resource="memory"})

Now ship those four as a dashboard checked into Git instead of four panels you built by clicking and will never reproduce. kube-prometheus-stack's Grafana runs a sidecar that auto-loads any ConfigMap labelled grafana_dashboard: "1" in its own namespace — so the dashboard becomes just another file next to the ServiceMonitor:

# platform-addons/observability/grafana-dashboard-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: ledger-golden-signals
  namespace: platform                   # same namespace as the Grafana pod
  labels:
    grafana_dashboard: "1"              # the label kube-prometheus-stack's sidecar watches for
data:
  ledger-golden-signals.json: |
    {
      "title": "Ledger — Golden Signals",
      "uid": "ledger-golden-signals",
      "schemaVersion": 39,
      "time": { "from": "now-1h", "to": "now" },
      "panels": [
        {
          "id": 1, "title": "Traffic (req/s)", "type": "timeseries",
          "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
          "datasource": { "type": "prometheus", "uid": "Prometheus" },
          "targets": [
            { "expr": "sum(rate(http_requests_total{namespace=\"ledger\"}[5m]))", "legendFormat": "req/s" }
          ]
        },
        {
          "id": 2, "title": "Errors (5xx ratio)", "type": "timeseries",
          "gridPos": { "x": 12, "y": 0, "w": 12, "h": 8 },
          "datasource": { "type": "prometheus", "uid": "Prometheus" },
          "targets": [
            { "expr": "sum(rate(http_requests_total{namespace=\"ledger\",code=~\"5..\"}[5m])) / sum(rate(http_requests_total{namespace=\"ledger\"}[5m]))", "legendFormat": "5xx ratio" }
          ]
        },
        {
          "id": 3, "title": "Latency p99 (s)", "type": "timeseries",
          "gridPos": { "x": 0, "y": 8, "w": 12, "h": 8 },
          "datasource": { "type": "prometheus", "uid": "Prometheus" },
          "targets": [
            { "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{namespace=\"ledger\"}[5m])) by (le))", "legendFormat": "p99" }
          ]
        },
        {
          "id": 4, "title": "Saturation (memory vs limit)", "type": "timeseries",
          "gridPos": { "x": 12, "y": 8, "w": 12, "h": 8 },
          "datasource": { "type": "prometheus", "uid": "Prometheus" },
          "targets": [
            { "expr": "sum(container_memory_working_set_bytes{namespace=\"ledger\",container=\"ledger\"}) / sum(kube_pod_container_resource_limits{namespace=\"ledger\",container=\"ledger\",resource=\"memory\"})", "legendFormat": "mem/limit" }
          ]
        }
      ]
    }

Confirm it loaded without ever opening the Grafana UI to build it by hand:

kubectl -n platform logs -l app.kubernetes.io/name=grafana -c grafana-sc-dashboard --tail=20
# should log that it found and wrote ledger-golden-signals.json

curl -s -u admin:<the password from earlier> http://localhost:3000/api/search?query=Ledger | jq '.[].title'
# "Ledger — Golden Signals"

Alerting on ledger: a PrometheusRule

☺ Like you're 10: A bell that only rings when users are actually feeling pain — plus a quieter one that watches for a container getting killed for using too much memory.

Following the paired lesson's rule — alert on symptoms, not causes — write one alert on the user-visible signal (error rate) and two on the specific failure signatures this part is about to cause on purpose, sourced straight from kube-state-metrics (bundled in the chart you already installed):

# platform-addons/observability/prometheusrule.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: ledger-alerts
  namespace: ledger
  labels:
    release: kube-prometheus-stack       # required, same as the ServiceMonitor
spec:
  groups:
    - name: ledger.rules
      rules:
        - record: ledger:http_errors:ratio_rate5m
          expr: |
            sum(rate(http_requests_total{namespace="ledger",code=~"5.."}[5m]))
              / sum(rate(http_requests_total{namespace="ledger"}[5m]))

        - alert: LedgerHighErrorRate
          expr: ledger:http_errors:ratio_rate5m > 0.05
          for: 5m
          labels: { severity: page }
          annotations:
            summary: "ledger error rate above 5%"
            description: "{{ $value | humanizePercentage }} of ledger requests failed (5xx) over 5m."

        - alert: LedgerContainerOOMKilled
          expr: kube_pod_container_status_last_terminated_reason{namespace="ledger",container="ledger",reason="OOMKilled"} == 1
          for: 0m
          labels: { severity: page }
          annotations:
            summary: "a ledger container was OOMKilled"
            description: "pod {{ $labels.pod }} in ledger was last terminated with reason OOMKilled."

        - alert: LedgerCrashLooping
          expr: increase(kube_pod_container_status_restarts_total{namespace="ledger",container="ledger"}[15m]) > 3
          for: 5m
          labels: { severity: page }
          annotations:
            summary: "ledger is crash-looping"
            description: "pod {{ $labels.pod }} restarted more than 3 times in the last 15 minutes."

Confirm the rule loaded, then force the error-rate alert to fire on purpose, safely, before you touch anything that resembles a real incident:

curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[] | select(.name=="ledger.rules") | .rules[].name'

# ship a buggy build the way Part 2's Milestone 10 taught, just to make the error-rate alert real
kubectl create -f - <<'EOF'
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: ledger-ci-run-
  namespace: platform
spec:
  pipelineRef: { name: ledger-ci }
  taskRunTemplate: { serviceAccountName: ledger-ci }
  params:
    - { name: repo-url, value: "https://github.com/YOU/ledger-src.git" }
    - { name: config-repo-url, value: "https://github.com/YOU/platform-capstone.git" }
    - { name: image, value: "registry.local:5001/ledger" }
    - { name: tag, value: "1.4.1-bad" }
    - { name: build-arg-buggy, value: "true" }
  workspaces:
    - { name: source, volumeClaimTemplate: { spec: { accessModes: ["ReadWriteOnce"], resources: { requests: { storage: 1Gi } } } } }
    - { name: config, volumeClaimTemplate: { spec: { accessModes: ["ReadWriteOnce"], resources: { requests: { storage: 256Mi } } } } }
EOF

curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.labels.alertname=="LedgerHighErrorRate")'
curl -s http://localhost:9093/api/v2/alerts | jq '.[].labels.alertname'

Notice what you just watched: Part 2's canary analysis catches the same bad build within its own 60-second window and aborts it long before your 5-minute for: window would ever page anyone — which is exactly the point. Fast automated analysis and slower symptom-based alerting are two different nets over the same failure, and a real one occasionally slips past the first.

Breaking it on purpose, part one: OOMKilled

☺ Like you're 10: Deliberately give the toy less room to breathe than it needs, and watch — and time — exactly how it dies.

Because selfHeal: true has been on since Part 1, you cannot fake an incident with a live kubectl patch — Argo CD reverts it in seconds. The only way to break ledger for this drill is the only way that's ever real: commit the bad state to Git and let the reconciler apply it faithfully, exactly as designed.

# ledger/rollout.yaml — add an unrealistically low memory LIMIT (Part 2 never set one)
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: ledger
  namespace: ledger
spec:
  replicas: 5
  revisionHistoryLimit: 3
  selector:
    matchLabels: { app: ledger }
  template:
    metadata:
      labels: { app: ledger }
    spec:
      containers:
        - name: ledger
          image: registry.local:5001/ledger:1.4.2-oom-drill
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 3
            periodSeconds: 5
          resources:
            requests: { cpu: 25m, memory: 32Mi }
            limits: { memory: 16Mi }              # deliberately too small — Flask alone needs more
  strategy:
    canary:
      canaryService: ledger-canary
      stableService: ledger-stable
      steps:
        - setWeight: 10
        - pause: { duration: 60s }
        - analysis:
            templates: [ { templateName: ledger-smoke-test } ]
            args: [ { name: service-name, value: ledger-canary } ]
        - setWeight: 50
        - pause: { duration: 60s }
        - analysis:
            templates: [ { templateName: ledger-smoke-test } ]
            args: [ { name: service-name, value: ledger-canary } ]
        - setWeight: 100

Before you push, open two terminals so you're ready to react the moment the bad revision lands — you have roughly a minute before the canary analysis notices the failing pod and Argo Rollouts starts scaling it back down, taking the evidence with it:

# terminal 1 — watch pods appear and die
kubectl -n ledger get pods -l app=ledger -w

# terminal 2 — fire the change the instant terminal 1 shows a new pod
git add ledger/rollout.yaml
git commit -m "drill: OOM the canary on purpose"
git push
kubectl argo rollouts get rollout ledger -n ledger --watch

The moment terminal 1 shows the new canary pod, run the triage flow — fast, because this window closes:

# 1. wide view — restart count climbing?
kubectl -n ledger get pods -l app=ledger -o wide

# 2. events are at the BOTTOM of describe
kubectl -n ledger describe pod <the new canary pod>
# Last State:  Terminated
#   Reason:    OOMKilled
#   Exit Code: 137

# 3. logs from the crashed container — the tell here is what's MISSING
kubectl -n ledger logs <pod> --previous
# usually just the ordinary Flask startup banner, then nothing — the kernel kills the
# process, the app never gets a chance to log a reason. That absence IS the signature.

⚠ OOMKilled has no application log to blame

This is the single most useful thing to internalise from this drill: OOMKilled is a kernel decision, not an application error, so logs --previous will look ordinary right up until it stops — there is no stack trace, no "out of memory" line written by ledger itself. The proof lives entirely in describe pod's Last State block and exit code 137. If you go looking for a log line explaining the OOM kill, you'll waste the whole triage window; that's the failure mode Part D of Part 2 and Milestone 12 here are both built to cure.

Revert it the only way this platform allows — in Git — and confirm the automated canary abort (or your own recovery) leaves ledger healthy again:

git revert HEAD --no-edit
git push

kubectl argo rollouts get rollout ledger -n ledger
# Healthy, 100% on the last known-good revision

kubectl -n ledger get pods -l app=ledger -o jsonpath='{..image}'

Breaking it on purpose, part two: CrashLoopBackOff

☺ Like you're 10: This time the toy refuses to even start — and unlike the last one, it leaves you a note explaining why, if you know where to look.

Commit a second, different kind of failure: override the container's own command so it never starts the Flask app at all — a stand-in for "someone committed a bad rollback command" or "a config generator produced garbage." Revert the OOM drill first if you haven't already, so you're testing one fault at a time.

# ledger/rollout.yaml — spec.template.spec.containers[0], nothing else changes
        - name: ledger
          image: registry.local:5001/ledger:1.4.2-oom-drill
          command: ["python", "-c"]
          args:
            - "import sys; print('ledger: fatal: unable to reach payments-core, refusing to start', file=sys.stderr); sys.exit(1)"
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 3
            periodSeconds: 5
          resources:
            requests: { cpu: 25m, memory: 32Mi }
            limits: { memory: 128Mi }

kubectl -n ledger get pods -l app=ledger -w &

git add ledger/rollout.yaml
git commit -m "drill: crash the canary on purpose"
git push

Because the crashing pod's readiness probe never has a chance to pass, this canary never reports Healthy on its own — don't wait out the ten-minute progressDeadlineSeconds timeout to see that happen. Instead, once you've captured what you need below, reach for the same panic button Part 2's Milestone 11 taught:

# 1. wide view — status column tells you the shape of the failure
kubectl -n ledger get pods -l app=ledger -o wide
# NAME                     READY   STATUS             RESTARTS
# ledger-7c9...-abcde       0/1    CrashLoopBackOff   4 (38s ago)

# 2. events at the bottom of describe — note the WAITING reason this time, not a Last State
kubectl -n ledger describe pod <the crashing pod>
# State:       Waiting
#   Reason:    CrashLoopBackOff
# Last State:  Terminated
#   Reason:    Error
#   Exit Code: 1

# 3. logs from the crashed container — THIS time there's a real message
kubectl -n ledger logs <pod> --previous
# ledger: fatal: unable to reach payments-core, refusing to start

# once you've captured the above, stop waiting on the progress deadline — abort by hand
kubectl argo rollouts abort ledger -n ledger

◆ Key idea — tell the two failures apart on sight

You just produced the two failure modes the exam most wants you to distinguish instantly. OOMKilled: Last State: Terminated, reason OOMKilled, exit code 137, and logs --previous is ordinary right up to a silent stop — the kernel acted, not the app. CrashLoopBackOff: State: Waiting, reason CrashLoopBackOff, and logs --previous usually hands you the application's own explanation, because the app chose to exit. One has no note; the other does. Learning to check the right field in ten seconds is worth more than memorising either definition.

Revert and confirm recovery the same way as the first drill:

git revert HEAD --no-edit
git push

kubectl argo rollouts get rollout ledger -n ledger
# Healthy, 100%, back on the last known-good, working image and command

argocd app get ledger --hard-refresh

What "done" looks like for Part 5

☺ Like you're 10: A cluster that can finally answer "is it okay?" without you guessing — and two fresh scars proving you know how to read the answer when it says no.

At the end of this part, kube-prometheus-stack runs in platform, reconciled by Argo CD; ledger exposes real Prometheus metrics through ledger-src's new /metrics endpoint; a ServiceMonitor, a golden-signals Grafana dashboard, and a PrometheusRule are all Git-committed manifests under platform-addons/observability/, not clicked together by hand; and you have personally triggered, timed, and correctly diagnosed both an OOMKilled pod and a CrashLoopBackOff pod using nothing but kubectl describe and logs --previous. Nothing from Parts 1–4 changed shape — you gave the existing platform eyes, and then tested the eyes. Part 6 starts from exactly this state and adds the policy layer: Kyverno, RBAC, NetworkPolicy, mTLS, a Trivy scan, tenancy, and cost visibility — then chains every part of this capstone together into one automatic golden path.

🦆 Dot's-eye view

"Before this part, when someone asked if ledger was healthy, I'd port-forward in and poke at it by hand. Now I open one Grafana dashboard and I know in five seconds — traffic's steady, errors are near zero, p99 is fine. And the two times Ellie broke it on purpose, I actually recognised the shape of the failure before describe even confirmed it. That's the whole difference between panicking and triaging."

🎬 At the Platform Guild
🦊Foxy: Why bother building two whole incidents on purpose? Can't we just trust the dashboard once it's up?
🐘Ellie: A dashboard nobody has watched fail is a dashboard you're hoping works, Foxy. I don't hope. I break it, I watch the alert fire, I read the pod myself — then I trust it.
🤖Recon: BEEP. And notice — you couldn't even sabotage the live pod directly. selfHeal reverted every attempt until you committed the bad state to Git properly. Even your fire drill had to go through the front door.
👺Gizmo: Boooring. Just page yourself "everything's fine!" every morning and skip the actual watching. Nobody checks the pages anyway. 🤑
🐢Timmy: That's exactly how a real page gets ignored, Gizmo. Every alert we ship pages on a symptom a human can act on, or it doesn't page at all.
🦆Dot: Honestly, the OOM one got me. No log line, nothing — just a silent stop and an exit code. I'd have stared at empty logs for ten minutes before this drill. Now I check describe first, every time.

Extending the pattern to invoices

☺ Like you're 10: The same trick works for the second toy from Part 4 — you just do it again, a little faster the second time.

Part 4 scaffolded a second real service, invoices, through the golden-path Software Template — and it deserves the same eyes. The recipe is identical, just repointed: add a named, labelled port to the invoices Service in its namespace, add a second ServiceMonitor (with namespaceSelector.matchNames: [invoices]), and either extend ledger-alerts with an or across both namespaces or add a small invoices-alerts PrometheusRule beside it. If invoices' own scaffolded code has no /metrics endpoint yet, that's exactly the gap this section of Part 4's golden-path Software Template should really close for every future service — bake the same prometheus_client instrumentation and a ServiceMonitor straight into the template's skeleton/, so the next scaffolded service arrives already observable, the same way it already arrives with a pipeline and a canary wired in. Treat that as a stretch goal rather than a new milestone below — the milestones focus on proving the pattern once, deeply, against ledger.

Milestones

☺ Like you're 10: Tick each box only once you've watched it happen on your own screen — especially the two break-it drills, which only count if you actually read the pod yourself.

Work these in order — each depends on the cluster and repo state from the one before. Progress saves in this browser.

0 / 14 milestones complete

🐢 Timmy's checkpoint

1. Why does the ServiceMonitor select only the plain ledger Service and not ledger-stable/ledger-canary too? 2. Name the one label every ServiceMonitor and PrometheusRule in this part needs, and what happens silently if you forget it. 3. Why couldn't you fake either incident with a live kubectl patch? 4. What field distinguishes OOMKilled from CrashLoopBackOff in kubectl describe pod, and which one typically leaves a useful message in logs --previous? 5. Why does the error-rate alert's 5-minute window rarely get the chance to fire on a bad canary, when Part 2's own analysis catches it first?

Check your answers

All three Services select the exact same pods, so scraping all three would create three duplicate copies of every series — one per Service — for no benefit. Labelling only the plain ledger Service and matching on that label keeps the target set to exactly one.
release: kube-prometheus-stack (matching the Helm release name). Forget it and the object is silently ignored — no error, no event, it just never shows up in Prometheus's target or rule list.
Because selfHeal: true has been enforced on the ledger Application since Part 1 — any live-only kubectl patch is reverted back to what's in Git within seconds. The only way to make a change stick is to commit it, which is also exactly how a real bad change reaches production.
OOMKilled shows up as Last State: Terminated, Reason: OOMKilled, exit code 137 — a kernel decision, so logs --previous is ordinary right up to a silent stop. CrashLoopBackOff shows State: Waiting, Reason: CrashLoopBackOff, and because the application chose to exit, logs --previous usually carries its own error message.
Part 2's canary analysis runs a smoke test within a 60-second pause and fails fast on a failureLimit of zero, so a bad canary is usually aborted well inside a minute or two — long before a symptom-based alert's for: 5m condition has stayed true long enough to fire. Fast automated analysis and slower alerting are two different safety nets over the same class of failure.

Part 5 gave the whole capstone eyes: a real metrics endpoint on ledger, a dashboard and alert shipped as code, and two incidents you caused and correctly diagnosed with your own hands. Continue to Capstone Part 6 — Security & the Golden Path, where Timmy adds policy, RBAC, NetworkPolicy, mTLS, a Trivy scan, tenancy and cost visibility — and chains every part of this capstone into one automatic golden path. Or step back to the full lab track to see how this part fits the rest of the capstone, and revisit Observability & Operations and the standalone Observability Labs for the concepts behind what you just built.