Capstone Part 4 — Observability

This is Part 4 of the five-part capstone running through this whole course: one continuously evolving service, kubestronaut-tracker, carried from a bare git repo through GitOps delivery and a progressive canary rollout, into a mesh with STRICT mutual TLS and Kyverno enforcing admission in Part 3. It runs, it's delivered safely, it's locked down — and nobody watching from outside the mesh can see a single number or a single request path through it. This part gives it eyes: real golden-signal metrics scraped through the exact mTLS wall Part 3 just built, zero-code OpenTelemetry tracing that follows one request across the mesh boundary itself, a dashboard with one panel per signal, and precisely two Prometheus alerts wired to fire — not the twenty a nervous team is tempted to write. Part 5 links straight to the dashboard this part builds.

What you're building on, and what this part adds

☺ Like you're 10: The ship is built, sealed, and flying. This part only adds instruments — nothing here reopens a hatch Part 3 closed.

This page assumes the whole world described on the lab track hub and everything Part 3 left behind: a kubestronaut-tracker Deployment and a single-replica Postgres in the tracker namespace, sidecar-injected, sitting behind a STRICT PeerAuthentication; a Kyverno ClusterPolicy in enforce mode rejecting anything with a :latest tag, a missing sidecar-injection label, or a root user; and everything add-on-shaped — Argo CD, the Argo Rollouts controller, istiod, Kyverno — living unmeshed in platform. This page adds four things to that picture, all in platform except the app changes, which land in tracker:

Everything below assumes the mesh and policy vocabulary from Istio and Kyverno, and goes deep on two tools already covered on their own pages: Prometheus for the scrape model, relabeling, and alerting rules underneath the Operator's CRDs, and the OpenTelemetry Collector for the five component types and the declared-vs-wired trap this page's own pipeline has to get right. Read those first if ServiceMonitor, PrometheusRule, or OpenTelemetryCollector feel like unfamiliar shapes — this page puts them to work against a real mesh rather than re-explaining them. Platform Engineering's Prometheus page and its OpenTelemetry page cover the same Operator layer from the CNPE's own angle, if that framing helps.

Installing kube-prometheus-stack into platform

☺ Like you're 10: One chart installs the whole instrument panel at once — the gauges, the recorder, and the alarm bell — instead of three separate installs that have to be wired together by hand.

On a Kubernetes platform, hand-writing prometheus.yml the way a plain-VM install would is the wrong layer to work at — the Prometheus page's raw config is what the Operator generates from your CRDs, not what you edit directly. kube-prometheus-stack installs the Prometheus Operator, a Prometheus instance it manages, a dedicated Alertmanager, and Grafana, all pre-wired to each other:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace platform --create-namespace \
  --set grafana.adminPassword='...' \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues=false

kubectl -n platform get pods -l "release=kube-prometheus-stack"
# prometheus-kube-prometheus-stack-prometheus-0   Running
# alertmanager-kube-prometheus-stack-alertmanager-0   Running
# kube-prometheus-stack-grafana-...                   Running

The three *SelectorNilUsesHelmValues=false flags matter more than they look: by default the chart's own Prometheus CR only picks up ServiceMonitor, PodMonitor, and PrometheusRule objects carrying the chart's own release label, which quietly ignores every CRD you write yourself unless you either match that label or flip these three flags to make Prometheus watch every CRD of each kind, cluster-wide. Skip this and the ServiceMonitor two sections down applies cleanly, reports no error anywhere, and simply never gets read — the exact kind of "declared but not wired" trap the Collector page warns about, one CRD layer up.

Scraping through STRICT mTLS: the merged-metrics port Istio already gives you

☺ Like you're 10: Instead of knocking on the ship's front door — which is bolted since Part 3 — you read the vitals off the one maintenance porthole that was never sealed, because even a locked ship still has to let a doctor's stethoscope through.

Istio's own gotcha list names this exact failure first among the four things STRICT mTLS commonly breaks: "a Prometheus instance scraping a meshed namespace from outside the mesh." That's precisely this page's situation — kube-prometheus-stack lives unenrolled in platform, and every pod it wants to scrape in tracker now refuses any caller that can't present a valid mesh certificate. A plain ServiceMonitor pointed at kubestronaut-tracker's own port 3000 gets a connection reset, every single scrape, forever — and Prometheus's /targets page reports it as DOWN with no hint that the fix is anywhere near mTLS.

The fix doesn't touch Part 3's PeerAuthentication at all. Every sidecar-injected pod already runs an istio-agent that answers on port 15020 — the same port kubelet's own httpGet liveness and readiness probes use, which is why Istio exempts it from mTLS enforcement in the first place: a kubelet probe can't present a mesh certificate either, and the platform can't function if STRICT silently breaks every pod's own health checks. That exemption has a second job: by default the agent also scrapes the application's own /metrics endpoint over localhost — a loopback hop inside the same pod, never crossing the mTLS boundary at all — and merges the result with Envoy's own proxy-level stats, serving the combined set at 15020/stats/prometheus. Point the ServiceMonitor there instead of at the app's own port, and the scrape succeeds under STRICT with zero exceptions carved into the mesh:

# tracker/observability/servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kubestronaut-tracker
  namespace: tracker
  labels: { release: kube-prometheus-stack }   # picked up even without the Selector flags above
spec:
  selector:
    matchLabels: { app: kubestronaut-tracker }
  namespaceSelector:
    matchNames: [tracker]
  endpoints:
    - port: http-envoy-prom      # the istio-proxy container's own declared port name, 15020
      path: /stats/prometheus
      interval: 15s
      relabelings:
        - action: replace
          targetLabel: app
          replacement: kubestronaut-tracker

Instrumenting kubestronaut-tracker: the four golden signals as real metrics

☺ Like you're 10: Two counters and a stopwatch, attached to every request, answer "how busy," "how often it fails," and "how fast" — the fourth signal, saturation, Kubernetes already tracks for you.

Traffic, errors, and latency live inside the application and need instrumenting directly; saturation is a pod-level metric kube-prometheus-stack's own kubelet and cAdvisor scrape jobs already collect for every container on the cluster, no application code required. Add prom-client, and label by the Express route pattern, never the resolved path — POST /certs/:id/complete has exactly one distinct route value no matter which certification ID hits it, while the raw path would mint a brand-new, permanently-indexed series per certification, the identical cardinality trap the Prometheus page warns about:

// metrics.js — mounted once, before any route handler
const client = require("prom-client");
client.collectDefaultMetrics({ prefix: "tracker_" });

const httpRequests = new client.Counter({
  name: "http_requests_total",
  help: "Total HTTP requests",
  labelNames: ["route", "method", "status"],
});
const httpDuration = new client.Histogram({
  name: "http_request_duration_seconds",
  help: "HTTP request duration in seconds",
  labelNames: ["route", "method", "status"],
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2, 5],
});

function metricsMiddleware(req, res, next) {
  const end = httpDuration.startTimer();
  res.on("finish", () => {
    // req.route.path keeps ":id" literal — never the resolved certification ID
    const route = req.route ? req.route.path : "unmatched";
    const labels = { route, method: req.method, status: res.statusCode };
    httpRequests.inc(labels);
    end(labels);
  });
  next();
}

module.exports = { metricsMiddleware, register: client.register };

// index.js — wire it in, next to the existing /health, /certs, /progress routes
const { metricsMiddleware, register } = require("./metrics");
app.use(metricsMiddleware);
app.get("/metrics", async (req, res) => {
  res.set("Content-Type", register.contentType);
  res.end(await register.metrics());
});

Redeploy through the same Argo CD sync Part 3 left running, and confirm the scrape end to end before touching anything else — /targets in Prometheus's own UI should show the kubestronaut-tracker job as UP, and a real query should return real numbers:

# traffic: requests per second, by route
sum(rate(http_requests_total{job="kubestronaut-tracker"}[5m])) by (route)

# errors: error rate as a fraction of total requests
sum(rate(http_requests_total{job="kubestronaut-tracker", status=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="kubestronaut-tracker"}[5m]))

# latency: p99 across the pod, from the histogram
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket{job="kubestronaut-tracker"}[5m])) by (le)
)

Zero-code tracing: the OpenTelemetry Operator's Instrumentation CRD

☺ Like you're 10: Instead of importing a tracing library into your own code, you stick one annotation on the pod, and something else quietly slips the library in for you before your code even starts.

The Collector page mentions the OTel Operator's second CRD in passing — Instrumentation, which injects auto-instrumentation into an annotated pod's init container without a single import statement in kubestronaut-tracker's own source. Install the Operator (it needs cert-manager for its webhooks, exactly as the Collector page's own install walkthrough covers), then declare one namespace-scoped profile:

# tracker/observability/instrumentation.yaml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: tracker-instrumentation
  namespace: tracker
spec:
  exporter:
    endpoint: http://otel-collector.platform.svc:4317   # crosses the mesh boundary OUTBOUND — no mTLS wrinkle
  propagators: [tracecontext, baggage]
  sampler:
    type: parentbased_traceidratio
    argument: "1"       # 100% for this lab; a real fleet tunes this down hard

# one annotation on the Deployment's pod template — the whole opt-in
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-nodejs: "true"

Redeploy, and the Operator's mutating webhook adds an init container that copies the Node.js auto-instrumentation agent into the pod and sets NODE_OPTIONS to load it before index.js ever runs — http, express, and the Postgres client library all get instrumented automatically, each database call showing up as its own child span with zero lines added to metrics.js or index.js. The propagators line matters as much as the injection itself: it's what makes the app read an incoming traceparent header from the mesh's ingress hop and continue that same trace, rather than quietly starting a brand-new one at the app boundary every single time.

A Collector pipeline that catches both the app's spans and the mesh's own

☺ Like you're 10: The app writes its own diary page about the request; separately, every relay station along the way writes its own note too — and gluing all those pages together by their shared ticket number is what "one trace" actually means.

Auto-instrumentation gets the application's own spans out via OTLP. It says nothing about the two Envoy sidecars the request also passes through on its way in and out of the mesh — and those hops are exactly where a real production trace earns its keep, because a request that looks fine inside the app but times out at a sidecar is invisible without them. Istio's Telemetry API can point Envoy's own tracing straight at an OTLP endpoint, no extra code anywhere:

# platform/observability/istio-telemetry.yaml
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default-tracing
  namespace: istio-system
spec:
  tracing:
    - providers:
        - name: otel-tracing
  # the otel-tracing provider name is defined once, in istiod's own MeshConfig,
  # pointing at otel-collector.platform.svc:4317 — the same Collector below

Both sources — the app's own OTLP export and Envoy's own OTLP export — land on one OpenTelemetryCollector in platform, which does the simplest thing a Collector can do: receive, batch, forward:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-collector
  namespace: platform
spec:
  mode: deployment
  config:
    receivers:
      otlp:
        protocols:
          grpc: { endpoint: 0.0.0.0:4317 }
          http: { endpoint: 0.0.0.0:4318 }
    processors:
      memory_limiter:            # ALWAYS first — see the Collector page's own gotcha list
        check_interval: 1s
        limit_percentage: 80
        spike_limit_percentage: 25
      batch: { timeout: 5s }      # ALWAYS last
    exporters:
      otlp/jaeger:
        endpoint: jaeger-collector.platform.svc:4317
        tls: { insecure: true }
      debug:
        verbosity: detailed
    service:
      pipelines:
        traces:
          receivers:  [otlp]
          processors: [memory_limiter, batch]
          exporters:  [otlp/jaeger]   # forgetting this line is the whole page's classic bug

# a minimal all-in-one Jaeger, throwaway-lab-grade — a real fleet runs Jaeger's own separated collector/query/storage
kubectl -n platform create deployment jaeger \
  --image=jaegertracing/all-in-one:1.60 --port=16686
kubectl -n platform expose deployment jaeger --port=16686 --target-port=16686 --name=jaeger-query
kubectl -n platform expose deployment jaeger --port=4317 --target-port=4317 --name=jaeger-collector

A dashboard with one panel per golden signal, not twenty

☺ Like you're 10: Four gauges on one panel, checked into the same repo as everything else, so the dashboard travels with the cluster instead of living only in one person's browser tab.

The kube-prometheus-stack chart's built-in sidecar watches for any ConfigMap carrying the label grafana_dashboard: "1" and mounts it straight into Grafana's provisioning directory — no click-and-forget dashboard, just a file Argo CD reconciles like any other manifest:

# platform/observability/tracker-dashboard.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kubestronaut-tracker-dashboard
  namespace: platform
  labels: { grafana_dashboard: "1" }
data:
  tracker.json: |
    {
      "title": "kubestronaut-tracker — golden signals",
      "panels": [
        { "title": "Request rate by status",
          "targets": [{ "expr": "sum(rate(http_requests_total{job=\"kubestronaut-tracker\"}[5m])) by (status)" }] },
        { "title": "Error rate",
          "targets": [{ "expr": "sum(rate(http_requests_total{job=\"kubestronaut-tracker\",status=~\"5..\"}[5m])) / sum(rate(http_requests_total{job=\"kubestronaut-tracker\"}[5m]))" }] },
        { "title": "p99 latency by route",
          "targets": [{ "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job=\"kubestronaut-tracker\"}[5m])) by (le, route))" }] },
        { "title": "Pod memory (saturation)",
          "targets": [{ "expr": "container_memory_working_set_bytes{namespace=\"tracker\",container=\"app\"}" }] }
      ]
    }

That fourth panel — saturation — costs zero application code, exactly as the Collector page's own agent-vs-gateway framing predicts: container_memory_working_set_bytes comes from cAdvisor, which kube-prometheus-stack already scrapes for every pod on the cluster the moment it's installed. Four panels, not a wall of them — the same restraint the alert count below insists on, applied to a dashboard instead of a page.

Two PrometheusRule alerts, not twenty

☺ Like you're 10: Only two warning lights are wired to interrupt anyone. Everything else lives on the dashboard above, for a human to check on their own schedule.

On the Operator layer, an alert is a Kubernetes object, not a line in a rule_files path — but the alerting principles underneath are exactly the ones the raw Prometheus page teaches: page on symptoms a user actually feels, and stop at two:

# platform/observability/tracker-alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kubestronaut-tracker-alerts
  namespace: platform
  labels: { release: kube-prometheus-stack }
spec:
  groups:
    - name: kubestronaut-tracker.alerts
      rules:
        - alert: TrackerHighErrorRate
          expr: |
            sum(rate(http_requests_total{job="kubestronaut-tracker", status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{job="kubestronaut-tracker"}[5m])) > 0.05
          for: 10m
          labels: { severity: page }
          annotations:
            summary: "kubestronaut-tracker error rate above 5% for 10m"

        - alert: TrackerHighLatency
          expr: |
            histogram_quantile(0.99,
              sum(rate(http_request_duration_seconds_bucket{job="kubestronaut-tracker"}[5m])) by (le)
            ) > 1.5
          for: 10m
          labels: { severity: page }
          annotations:
            summary: "kubestronaut-tracker p99 latency above 1.5s for 10m"

Nothing here watches pod memory, CPU, or restart count directly — those are causes, not symptoms, and they already have a panel above for a human to notice on their own schedule. If a genuinely new failure mode later slips past these two, the honest fix is a better symptom alert, never a growing pile of cause-based ones.

Watching an alert actually transition, and reading the trace back

☺ Like you're 10: Break it on purpose, on a day when breaking it doesn't matter, so the first time you watch an alert fire and a trace explain why isn't during a real incident.

Prove the whole pipeline at once by injecting a real fault — point kubestronaut-tracker's DB connection string at a host that doesn't exist and redeploy through Argo CD the same way Part 3 taught:

# edit the DB host in the app's ConfigMap, commit, let Argo CD sync it in
kubectl -n tracker get configmap kubestronaut-tracker-config -o yaml
# ...change DB_HOST to something unreachable, commit to the repo, wait for sync...

# generate traffic against POST /certs/:id/complete — every one now 5xxs

# watch the alert move: inactive -> pending -> firing
watch -n 5 'kubectl -n platform exec deploy/kube-prometheus-stack-alertmanager -- \
  wget -qO- localhost:9093/api/v2/alerts | jq ".[].labels.alertname"'

Confirm it sits in pending for close to the full 10-minute window before firing — flap suppression working as designed. Then pull one failing trace out of Jaeger for the same time window and read it end to end: spans ① and ② (the ingress and inbound sidecar hops) show healthy latency, span ③ (the app handler) shows the request arriving fine, and the Postgres child span is exactly where the trace goes red — the same failure the dashboard's error-rate panel and the TrackerHighErrorRate alert are reporting from a completely different angle, all three agreeing on the same five minutes. Revert the DB host and confirm the alert clears back to inactive before moving on.

What "done" looks like for Part 4

☺ Like you're 10: A ship that shows its own vital signs, keeps one traceable ticket per crew request even through a locked hatch, and rings exactly two bells — never more, never fewer.

At the end of this part, kube-prometheus-stack watches kubestronaut-tracker through the one port STRICT mTLS was always going to leave open; the app exposes real golden-signal metrics via prom-client; zero-code OpenTelemetry instrumentation and Istio's own Envoy tracing both export to one Collector, which forwards a single, correlated, five-span trace per request to Jaeger; a four-panel dashboard lives as a labeled ConfigMap next to everything else Argo CD reconciles; and exactly two PrometheusRule alerts are wired and proven to transition. Nothing here is thrown away — Part 5 reads today's work directly:

Milestones

☺ Like you're 10: Tick each box only once you've actually watched it happen on your own cluster, not because the step "sounds right."

Work these in order — each depends on the cluster state from the one before. Progress saves in this browser.

Part 4 gave kubestronaut-tracker eyes it can keep even behind a locked mesh: real golden-signal metrics scraped through the one door STRICT mTLS was always going to leave open, one correlated trace per request spanning both the app and the mesh's own Envoy hops, a four-panel dashboard checked in as code, and two proven alerts. Continue to Capstone Part 5 — The Portal, where this dashboard gets linked from a Backstage catalog entry anyone on a team could find without asking you directly. Or step back to the full lab track to see how this part fits the rest of the capstone, revisit Prometheus, Grafana, the OpenTelemetry Collector, and Istio for the concepts behind what you just built, and go diagnose a deliberately broken pipeline under time pressure in Drill — Trace a Broken Telemetry Pipeline.