Capstone Part 5 — Incident Response

This is the fifth of six parts building one continuous project: parcel-api, gated by the pipeline you wired in Part 1, running on the cluster Part 2 provisioned, shipped through the canary rollout Part 3 built, watched by the dashboards and alerts Part 4 wired. Every part before this one was preparation. This part is where that preparation gets tested against something it didn't see coming: a canary release that behaves fine at idle and falls over under real concurrent traffic, a real page from your own alerting stack, a severity call you have to get right under time pressure, an incident you run using nothing but the process from incident management, and a real code fix shipped back through Part 1's exact gated pipeline — not just an infrastructure rollback. By the end you'll have a fired-and-resolved incident, a full blameless postmortem with a five-whys root cause, and the real DORA numbers this one incident produced.

What this part assumes, and what it produces

☺ Like you're 10: Everything from Parts 1–4 still needs to be standing — this page doesn't rebuild any of it, it breaks it on purpose.

This part assumes Parts 1–4 are done and still running. You'll need everything those parts already put on your laptop — kubectl, the argo rollouts kubectl plugin, git, and the gh CLI authenticated against your parcel-api repo from Part 1 — plus a way to generate concurrent HTTP load (hey is used below; ab or a small concurrent curl loop work identically). Nothing here needs a second person, though a study partner makes the incident-commander section considerably more real.

One boundary worth naming up front, the same way Part 1 scoped what it wasn't building yet: if a shipment record itself ever came back lost, duplicated, or misrouted to the wrong destination, that's not this scenario — data-integrity bugs in the shipment record are a different, always-critical incident regardless of blast radius. Today's incident is a clean read-path availability problem on one route, never a corrupted or missing shipment.

The scenario: a canary regression under real concurrent traffic

☺ Like you're 10: A new feature works perfectly every time you test it once — the bug only shows up once a bunch of requests hit it at the exact same time.

Build 2.3.0 adds one new route to parcel-api: GET /shipments/export, a bulk export of shipments matching an optional destination filter, meant for logistics partners running scheduled reconciliation jobs. It shipped through Part 1's gated pipeline like everything else — lint clean, tests green, image built and pushed as registry.internal/parcel-api:2.3.0 — and Part 3's Rollout is currently sitting at its first canary step, setWeight: 20, paused. Two of your ten parcel-api pods are running 2.3.0; eight are still on stable's 2.2.0.

The bug is a one-line mistake with an outsized blast radius. The new handler builds a carrier-rate lookup table — a ~50,000-entry object meant to be computed once and reused — but the helper that builds it got left inside the request handler during development instead of being hoisted to module scope, where every other piece of shared state in this file already lives:

// src/index.js — the regression shipped in build 2.3.0
app.get("/shipments/export", (req, res) => {
  const { destination } = req.query;
  // BUG: rebuilds a ~50,000-entry lookup table on every single request
  // instead of once at startup — this was supposed to be a module-level
  // constant, computed alongside `shipments` at the top of the file
  const carrierRateTable = buildCarrierRateTable();
  const rows = [...shipments.values()]
    .filter((s) => !destination || s.destination === destination)
    .map((s) => ({ ...s, rateEstimate: carrierRateTable[s.destination] }));
  res.status(200).json(rows);
});

Under a single request this is invisible — one extra allocation, a few milliseconds, nobody notices. Under real concurrency it is not: twenty or thirty simultaneous calls to /shipments/export each build their own multi-megabyte copy of the same table at once, and the canary pods' resource limits — copied unchanged from the Rollout spec that has served /healthz and the two shipment routes comfortably for months — were never re-sized for this new route:

# rollout.yaml — Part 3's canary, template excerpt
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: parcel-api
  namespace: parcel
spec:
  replicas: 10
  strategy:
    canary:
      canaryService: parcel-api-canary
      stableService: parcel-api-stable
      steps:
        - setWeight: 20
        - pause: { duration: 5m }
        - analysis: { templates: [{ templateName: parcel-api-error-rate }] }
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100
  selector:
    matchLabels: { app: parcel-api }
  template:
    metadata:
      labels: { app: parcel-api }
    spec:
      containers:
        - name: parcel-api
          image: registry.internal/parcel-api:2.3.0   # canary — Part 3 fills this in per rollout
          ports: [{ containerPort: 8080 }]
          resources:
            requests: { cpu: 100m, memory: 96Mi }
            limits: { cpu: 250m, memory: 128Mi }   # unchanged since Part 3 — never re-sized for /shipments/export

The trigger is ordinary, not exotic: a logistics partner's scheduled reconciliation job starts polling /shipments/export?destination=... with real concurrency, exactly the kind of traffic pattern a single sequential Jest call from Testing in the Pipeline never exercises. A small provided script reproduces that trigger against your own cluster:

# chaos/simulate-partner-export.sh — provided for Part 5
#!/usr/bin/env bash
set -euo pipefail
echo "[scenario] parcel-api:2.3.0 is already at canary weight 20% (Part 3) - idle and healthy."
echo "[scenario] simulating a partner reconciliation job: 25 concurrent GETs to /shipments/export"
hey -z 6m -c 25 "http://localhost:8080/shipments/export?destination=Pune,%20IN"

Run that against a port-forward of the main parcel-api Service (kubectl -n parcel port-forward svc/parcel-api 8080:80) and the bug is live, harmless-looking traffic hitting an already-deployed canary — nothing here required you to write buggy code, only to generate the real, concurrent volume that makes an already-latent bug real.

Detection: the alert Part 4 built actually fires

☺ Like you're 10: The alarm you built weeks ago has no idea today is a test — it just sees the numbers cross the line and rings anyway.

Within about a minute of the load starting, canary pods begin failing their liveness checks and restarting. Part 4's two rules — recapped here, not redefined — both trip:

# Part 4's alert rules, recapped — parcel-api-alerts.yaml
groups:
  - name: parcel-api
    rules:
      - alert: ParcelApiHighErrorRate
        expr: |
          sum(rate(http_requests_total{service="parcel-api",code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{service="parcel-api"}[5m]))
          > 0.02
        for: 3m
        labels: { severity: page }
      - alert: ParcelApiPodCrashLooping
        expr: increase(kube_pod_container_status_restarts_total{namespace="parcel"}[10m]) > 3
        for: 2m
        labels: { severity: page }

PagerDuty pages your primary on-call schedule with both:

ALERT FIRING: ParcelApiPodCrashLooping
  namespace:  parcel
  fired:      09:17 UTC
  value:      4 restarts / 10m on a parcel-api canary pod

ALERT FIRING: ParcelApiHighErrorRate
  service:    parcel-api
  fired:      09:17 UTC
  value:      error ratio 3.8% (threshold: 2%, for 3m)
  summary:    blended across every route and both tracks - this
              number alone does not yet say who's affected

3.8% blended is real, but it's not the whole story, and the alert payload says so on purpose. Before doing anything else, narrow it — first by route:

sum(rate(http_requests_total{service="parcel-api",code=~"5.."}[5m])) by (route)
/
sum(rate(http_requests_total{service="parcel-api"}[5m])) by (route)

# route="/shipments/export"   ~0.19
# route="/healthz"            0
# route="/shipments"          0
# route="/shipments/:id"      0

Every route except the new one is clean — the 3.8% blended number was diluted by all the healthy traffic on the other three routes. Now split that one route by track, using exactly the parcel-api-canary / parcel-api-stable Service pair Part 3 built for this purpose:

sum(rate(http_requests_total{route="/shipments/export",code=~"5.."}[5m])) by (track)
/
sum(rate(http_requests_total{route="/shipments/export"}[5m])) by (track)

# track="canary"   ~0.95
# track="stable"   ~0.0

That's the actual shape of the incident: one route, on one track, currently carrying 20% of traffic. Per incident management's severity scale, this is SEV2 — "significant degradation: a major feature is down, or a meaningful subset of users can't complete a core workflow" — not SEV1, which requires nearly all users affected. Acknowledge the page in PagerDuty first, before opening a single dashboard, exactly as that page recommends: acknowledging stops the secondary on-call from being paged behind you while you're still just looking.

Declaring command and running the room

☺ Like you're 10: One person decides what happens next and posts updates on a timer — the other keeps digging, without stopping to answer "any update?" every two minutes.

Per incident management, declare yourself Incident Commander now, before touching a single mitigating command: the IC's job is deciding and communicating, not debugging. Open a running, timestamped log in one place — a shared doc, a #parcel-api-incidents channel, even a plain text file — and set a communication cadence. The page doesn't give SEV2 an exact number, but it's explicit that cadence is set by severity and scales with the response-time expectation attached to that tier; SEV2's own 15-minute response window is the natural cadence to match here, so post an update every 15 minutes, even when the honest content is "still investigating."

If you're working solo, keep both hats but make the split real: say the IC decision out loud and log it before you switch to typing the diagnostic or mitigating command yourself. It feels artificial for about a minute and then stops — which is the actual point of practicing it here, on an incident small enough that getting it wrong costs nothing.

09:17 UTC  ParcelApiPodCrashLooping and ParcelApiHighErrorRate
           both fire. Paged.
09:18 UTC  Acknowledged in PagerDuty.
09:18 UTC  By-route query: error rate isolated to /shipments/export
           (~19%); every other route clean.
09:18 UTC  Declared. SEV2 - a meaningful subset (one route, one
           track), not nearly all users. IC: you.
09:19 UTC  First Comms update: "Investigating elevated errors on
           /shipments/export. No other route affected. Next
           update 09:34."
09:20 UTC  By-track query on /shipments/export: canary ~95%,
           stable ~0%. kubectl confirms 2 canary pods
           CrashLoopBackOff, 8 stable pods Running.

Diagnosing and mitigating: aborting the canary

☺ Like you're 10: Confirm what's actually broken before you touch anything, then reach for the fastest safe switch — not the slowest correct fix.

kubectl confirms exactly what the by-track query implied — the canary pods aren't returning errors, they're dying and restarting mid-request:

kubectl -n parcel get pods -l app=parcel-api
# parcel-api-7f9c4d-h2k9p   1/1 Running            (stable x8)
# parcel-api-a3e1c9-x7m2q   0/1 CrashLoopBackOff    (canary x2)

kubectl -n parcel describe pod parcel-api-a3e1c9-x7m2q | grep -A3 "Last State"
# Last State:  Terminated
#   Reason:    OOMKilled
#   Exit Code: 137

kubectl -n parcel top pod -l rollouts-pod-template-hash=a3e1c9
# parcel-api-a3e1c9-x7m2q   210m   126Mi   <- pinned right against the 128Mi limit

The recent-change check closes the loop: 2.3.0's only diff from 2.2.0 is the new /shipments/export route, and its handler is exactly the one shown in the scenario section above. Per deployment strategies, a canary's entire value proposition is a rollback that's a single command, not a redeploy — so mitigate by aborting, not by trying to patch the handler live under pressure:

kubectl argo rollouts abort parcel-api -n parcel

kubectl argo rollouts get rollout parcel-api -n parcel --watch
# canary weight: 20 -> 0
# stable weight: 80 -> 100
# within about a minute, all ten pods are back on 2.2.0

Watch the same by-route/by-track queries settle before declaring anything resolved — mitigation stops user impact, it is deliberately not the same thing as the underlying issue being fixed:

09:21 UTC  kubectl describe confirms OOMKilled/137 on both canary
           pods, pinned at ~126Mi against a 128Mi limit.
09:22 UTC  Correlated to the recent diff: buildCarrierRateTable()
           called per-request in 2.3.0's new handler.
09:23 UTC  IC decision, logged before typing it: abort the canary.
09:23 UTC  kubectl argo rollouts abort parcel-api -n parcel
09:25 UTC  /shipments/export error rate: 19% -> 2% -> 0%.
09:29 UTC  Clean for a sustained 4 minutes across every route and
           both queries. Impact is over. IC marks this mitigated,
           not yet resolved.

Resolving it for real: the fix ships back through Part 1's pipeline

☺ Like you're 10: The actual fix has to go through the exact same locked door every other change goes through — no shortcuts just because it's urgent.

With impact stopped, write the real fix: hoist buildCarrierRateTable() out of the request handler and compute it once, alongside shipments, at module load:

// src/index.js — the fix
const shipments = new Map();
const carrierRateTable = buildCarrierRateTable();   // computed once, at startup

app.get("/shipments/export", (req, res) => {
  const { destination } = req.query;
  const rows = [...shipments.values()]
    .filter((s) => !destination || s.destination === destination)
    .map((s) => ({ ...s, rateEstimate: carrierRateTable[s.destination] }));
  res.status(200).json(rows);
});

Ship it exactly the way Part 1 requires — a short branch, a PR, the build-and-test check, a squash merge — because "ships the fix back through this same gated pipeline" is the whole point of resolving this incident here rather than by hand-patching a running pod:

git checkout -b fix-export-rate-table-allocation
git commit -am "fix: hoist carrier rate table out of the /shipments/export handler"
git push -u origin fix-export-rate-table-allocation

gh pr create --fill --base main
gh pr checks --watch          # build-and-test passes on the real fix
gh pr merge --squash --delete-branch

gh run list --branch main --limit 1   # confirm main rebuilt registry.internal/parcel-api:<new-sha>

Only now, with the root cause actually fixed and merged — not merely stopped — does the IC declare resolved. The next time Part 3's rollout cadence comes back around for this service, it's this corrected image that starts a fresh canary at weight 20, not 2.3.0 again.

09:31 UTC  Branch cut, fix written and tested locally.
09:40 UTC  PR opened; build-and-test passes; squash-merged to main.
09:41 UTC  Root cause fixed and merged, not just mitigated. IC
           declares resolved.
09:42 UTC  Final Comms update; incident closed.

From mitigated to resolved: the DORA numbers this incident produced

☺ Like you're 10: Two different stopwatches were running the whole time — one for "when did users stop hurting," one for "when was it actually fixed" — and they read different numbers on purpose.

Incident management tracks mean-time-to-mitigate as closely as mean-time-to-resolve, because from a user's perspective the incident effectively ends at mitigation. Both clocks started at the same alert, 09:17, and read very differently:

MTTM (detect -> mitigate):  09:17 -> 09:23  =  6 minutes
MTTR (detect -> resolve):   09:17 -> 09:41  = 24 minutes

Both numbers matter, for different reasons: 6 minutes is roughly how long real users saw errors on /shipments/export, which is what a customer would report if you asked them. 24 minutes is the number measuring success: the DORA metrics defines as time to restore service — the DORA metric, because it counts the full path back to a fixed, merged, no-longer-latent root cause, not just the moment impact stopped.

This deploy also counts against this month's change failure rate: by DORA's own definition, any deployment to production that causes a degraded service and requires remediation — a hotfix, a rollback, a patch — is a failed deployment, and 2.3.0 required exactly that. The canary didn't prevent this from being a change failure; what it did was bound the blast radius to 20% of traffic on one route for six minutes, instead of 100% of parcel-api for however long a plain rolling deploy would have taken to notice and reverse — which is precisely what deployment strategies promises a canary buys you, not a promise that canaries eliminate bad deploys.

Writing the blameless postmortem

☺ Like you're 10: Fill in the same sections every time, in the same order, using only what's actually in your timestamped log — and every action item lands on a system, never a name.

Culture & collaboration is explicit that blameless doesn't mean consequence-free — it means separating "what broke and why" from "whose fault is it," because the second question makes people hide the exact information the first one needs. Draft the postmortem the moment the incident closes, using only the action log above as source material:

POSTMORTEM - parcel-api /shipments/export availability incident
Date: [your run date]  ·  Author: [you]  ·  Status: Reviewed

SUMMARY
A canary release (2.3.0) of parcel-api introduced a per-request
allocation regression in the new GET /shipments/export handler.
Under real concurrent traffic from a simulated partner
reconciliation job, canary pods were repeatedly OOMKilled,
failing roughly 95% of export requests routed to the canary
track (20% of total traffic) for about 6 minutes. Mitigated by
aborting the canary; resolved by hoisting the offending
allocation out of the request path and merging the fix through
the standard gated pipeline.

TIMELINE (UTC)
09:17  ParcelApiPodCrashLooping and ParcelApiHighErrorRate both
       fire. Paged and acknowledged.
09:18  By-route query isolates the problem to /shipments/export;
       every other route is clean. Declared SEV2.
09:20  By-track query: canary ~95% error rate, stable ~0%.
09:21  kubectl describe confirms OOMKilled, pinned at ~126Mi
       against a 128Mi limit.
09:22  Recent-change check: 2.3.0's only diff is the new export
       handler.
09:23  Canary aborted. 100% of traffic back on stable within
       about a minute.
09:29  Clean for a sustained 4 minutes. Impact over; marked
       mitigated, not yet resolved.
09:41  Root-cause fix merged to main through the standard
       build-and-test gate. IC declares resolved.

ROOT CAUSE(S)
1. The new /shipments/export handler rebuilt a ~50,000-entry
   lookup table on every request instead of once at startup.
2. The canary's resources.limits were inherited unchanged from
   the existing Rollout spec and never re-evaluated for a route
   with a materially different memory profile.
3. No stage in the pipeline (lint, test, build) or in Part 4's
   synthetic checks exercises a route under concurrent or
   realistic-volume traffic - every check calls each route once,
   sequentially, against a handful of fixture shipments.

IMPACT
~28 of ~150 requests to /shipments/export failed (timeout or
502) between 09:17-09:23 UTC - all of them on the canary track.
The 80% of export traffic on stable, and every other parcel-api
route, stayed healthy throughout. This deployment counts as one
change failure for this month's DORA change failure rate.

WHAT WENT WELL
- The by-route and by-track queries Part 4 set up made the true
  blast radius (one route, one track, 20% of traffic) obvious
  within about three minutes of the page, instead of it being
  guessed at from the blended alert number alone.
- The canary's own rollback mechanism (abort) restored 100% of
  traffic in about a minute - no redeploy, no waiting.
- The real fix went through the exact same gate every other
  change does; nothing was hand-patched on a live pod.

WHAT WENT POORLY
- resources.limits were copied forward from an existing Rollout
  spec without being re-justified against the new route's memory
  profile.
- No stage anywhere in the pipeline tests concurrent or
  realistic-volume behavior for any route.

ACTION ITEMS
1. Add a concurrency/volume check to the pipeline: seed several
   thousand fixture shipments and hit any collection-returning
   route with real concurrency, asserting peak memory per pod
   stays under a fraction of its configured limit.
   Owner: platform   Due: before the next canary attempt
2. Require re-justifying resources.limits - not copying them
   forward - in review, whenever a PR changes a route already
   flagged "collection-returning."
   Owner: process / review guidelines   Due: this week
3. Add a policy-as-code check that blocks a Rollout from
   progressing past its first canary step unless resources.limits
   were explicitly touched in the same PR that changed the route.
   Owner: platform   Due: Part 6

Read the root-cause section again and notice what it never says: it never says "whoever wrote the handler should have known better." Every line lands on a gap in the pipeline, in review practice, or in resource governance — fixable things that protect whoever touches this route next, not a verdict on whoever touched it this time.

Five whys, and the action items that open Part 6

☺ Like you're 10: Keep asking "why" about your own last answer until you stop landing on a person and start landing on a missing check.

The root-cause section above is the output of running five whys against the symptom in the log, refusing to stop at the first plausible-sounding answer — the same disciplined, small-batch thinking what is DevOps traces back to Lean's Toyota Production System roots, applied here to a single incident instead of a whole release process:

Symptom: ~95% of requests to /shipments/export on the canary
         track failed for about 6 minutes, while the same route
         on stable, and every other route, stayed healthy.

Why?    Canary pods were being OOMKilled and restarted faster
        than they could finish serving in-flight requests.
Why?    The new /shipments/export handler in build 2.3.0 rebuilds
        a large lookup table on every request instead of once at
        process startup.
Why?    The helper was defined inside the handler for convenience
        during development and never hoisted out before merge -
        the diff read as "add one export endpoint," not as a
        change to per-request memory behavior.
Why?    Part 1's test suite and Part 4's synthetic checks each
        call every route once, sequentially, against a handful
        of fixture shipments - a table rebuilt once per call is
        invisible at that scale, and nothing in the pipeline ever
        calls a route with real concurrency before it reaches
        even a 20%-weighted canary.
Why?    No stage in the pipeline tests behavior under concurrent
        or realistic-volume traffic for any route - lint, test,
        and build all verify a single response is correct, never
        the resource cost of producing many of them at once.

Root cause: the pipeline validates functional correctness against
small, sequential fixture calls but has no gate for memory
behavior under concurrent, realistic-volume traffic, so a
per-request-allocation regression invisible at fixture scale
passed every stage and reached a live canary before real
concurrent traffic exposed it. The canary's 20%-weight cap is
what kept this a SEV2 instead of a SEV1 - not anything upstream
catching the bug first.

Store every action item as a real, owned ticket, not a sentence buried in a document nobody reopens — a finding with no owner and no due date gets rediscovered, unfixed, in the next postmortem for the same root cause. Two of today's three don't just close the loop in the abstract; they set up exactly where this capstone goes next:

See Compliance as Code & Policy Enforcement for the general Policy Decision Point / Policy Enforcement Point pattern behind that last action item — Part 6 is where you'll actually wire an admission-control policy to enforce it.

Milestones

☺ Like you're 10: Tick each box only once you've actually watched it happen on your own screen, not because the step "sounds right."

Work these in order — later ones depend on the incident state from earlier ones. Progress saves in this browser.

Part 5 gave you a real fired-and-resolved incident, a severity call made on evidence instead of a guess, a fix shipped back through the exact same gate every other change goes through, and a full blameless postmortem with three tracked action items — one of which is the literal starting point for what comes next. Continue to Capstone Part 6 — Security Hardening, where the policy-as-code action item above becomes a real admission-control gate, or step back to Ship It — Start Here to see how this capstone's six parts fit together. For more reps outside this capstone's own story, the diagnose-a-production-incident drill and the blameless-postmortem drill both isolate one half of what this page just made you do end to end, and culture & collaboration is worth revisiting for the generative-culture research behind why blameless postmortems actually work.