Drill — Roll Back a Bad Deploy
Reading deployment strategies and actually reverting one at 100% traffic, seventeen minutes before a scheduled flash-sale email blast, with a curl loop printing 500s in another terminal, are two different skills — the lesson page only teaches the first. This drill is the second. You'll stand up cart-api, part of Northwind Retail's checkout system from this course's own case study, on a real local canary — Argo Rollouts, on your own kind cluster — bring it to a healthy, fully-promoted baseline, then recreate the exact mistake that starts most real rollback stories yourself: a hotfix that skips the line under deadline pressure. From there the clock is yours. Every scene below hands you exactly what a real responder would have at that second and makes you run the actual command before you're allowed to see what a fast, correct rollback does differently from a slow, technically-also-correct one.
Your little brother swapped the good cookies for burnt ones on the tray you'd already set out, right as guests started arriving. Yelling "stop serving cookies!" (abort) does nothing if the tray's already empty and every guest already has one. Baking a whole new safe batch from scratch (the slow-but-safe fix) works, but takes forever while guests are still eating burnt ones the whole time it bakes. The fast, correct move is grabbing the good cookies you already baked earlier — you already know they're good, you already tested them — and swapping the whole tray back in one motion, not one cookie at a time. That's the difference this drill is really about.
How this drill works
☺ Like you're 10: Build the mess yourself first, then race to clean it up before the countdown ends, checking each answer only after you've already committed to a command.
This is a hands-on drill, not a tabletop — every scene below has a real command to run against a real (throwaway) cluster. You will deliberately break your own deployment to create the emergency, then work the clock to fix it. Each scene ends with a single bolded decision and the exact command it implies; commit to it — say it out loud, type it, or at minimum decide it in your head — before you open the Check the response box underneath, the same way a real page doesn't wait for you to feel ready. The scenario runs against a fictional deadline, a 15:00 UTC flash-sale email blast expecting a 6× traffic spike on checkout the instant it sends, so every minute you spend deciding is a minute that email gets closer to landing on a broken checkout page.
Everything here runs on a local, throwaway cluster — kind — and nothing touches production or costs money. Tear it down when you're done: kind delete cluster --name shipit-drill, docker rm -f registry. Second: Argo Rollouts' CLI flags and exact status text drift between minors. Treat every command below as the shape of the answer, not a magic incantation — if something doesn't match, run kubectl argo rollouts version and check that version's own docs. Reacting correctly when a tool's behavior isn't exactly what the last minor did is itself part of the skill this drill is drilling.
Before you start: stand up cart-api on a canary
☺ Like you're 10: Build a small, working checkout counter and get it running smoothly before you let anyone break it.
You need Docker, kind, kubectl, and the kubectl argo rollouts CLI plugin. Make the cluster, install the Argo Rollouts controller, and stand up a throwaway local registry:
kind create cluster --name shipit-drill kubectl create namespace argo-rollouts kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml kubectl -n argo-rollouts rollout status deploy/argo-rollouts-controller-manager --timeout=120s brew install argoproj/tap/kubectl-argo-rollouts # or the release binary for your OS/arch kubectl argo rollouts version kubectl create namespace cart docker run -d -p 5000:5000 --restart=always --name registry registry:2
cart-api is deliberately tiny — one file, one behavior toggle — so the whole drill fits in one page and you can read exactly what "bad" means instead of trusting a black box:
// server.js
const http = require("http");
const BAD = process.env.BAD_RELEASE === "true";
let n = 0;
http.createServer((req, res) => {
if (req.url === "/healthz") { res.writeHead(200); return res.end('{"status":"ok"}'); }
if (req.url === "/cart/checkout") {
n++;
if (BAD && n % 5 < 2) { // the hotfix's actual bug, ~2 in 5 requests
res.writeHead(500, { "Content-Type": "application/json" });
return res.end('{"error":"connection pool exhausted"}');
}
res.writeHead(200, { "Content-Type": "application/json" });
return res.end('{"status":"checkout ok"}');
}
res.writeHead(404); res.end();
}).listen(8080, () => console.log(`cart-api up (BAD_RELEASE=${BAD})`));# Dockerfile FROM node:20-alpine WORKDIR /app COPY server.js . ARG BAD_RELEASE=false ENV BAD_RELEASE=$BAD_RELEASE EXPOSE 8080 CMD ["node", "server.js"]
Build and push the known-good baseline, v1.8.0 — this is the version everything rolls back to later, so build it once now and don't touch it again:
docker build -t localhost:5000/cart-api:v1.8.0 . docker push localhost:5000/cart-api:v1.8.0
Apply the Rollout and Service below. This is a basic canary — no service mesh, no Ingress controller — Argo Rollouts approximates each step's traffic weight by the ratio of new-to-old replica counts behind one plain Service, which is exactly enough for this drill and needs nothing extra installed:
# rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: cart-api
namespace: cart
spec:
replicas: 5
revisionHistoryLimit: 5
selector:
matchLabels: { app: cart-api }
template:
metadata:
labels: { app: cart-api }
spec:
containers:
- name: cart-api
image: localhost:5000/cart-api:v1.8.0
ports: [{ containerPort: 8080 }]
readinessProbe:
httpGet: { path: /healthz, port: 8080 }
periodSeconds: 3
strategy:
canary:
steps:
- setWeight: 20
- pause: { duration: 30s }
- setWeight: 60
- pause: { duration: 30s }
- setWeight: 100
---
apiVersion: v1
kind: Service
metadata: { name: cart-api, namespace: cart }
spec:
selector: { app: cart-api }
ports: [{ port: 80, targetPort: 8080 }]kubectl apply -n cart -f rollout.yaml kubectl argo rollouts get rollout cart-api -n cart --watch # Ctrl-C once it reads Healthy, 5/5
A brand-new Rollout has no prior "stable" revision to canary against, so it skips the steps and goes straight to Healthy with all 5 replicas on v1.8.0. That's your baseline. Leave a port-forward and a load loop running in two more terminals — you'll want both live for the rest of this drill:
# terminal 2
kubectl -n cart port-forward svc/cart-api 8080:80
# terminal 3 — the "curl loop" every scene below refers to
while true; do curl -s -o /dev/null -w "%{http_code} " http://localhost:8080/cart/checkout; sleep 0.2; doneThe scenario: 15:00 UTC, and a hotfix skips the line
☺ Like you're 10: A small, real, tempting shortcut — ship the fix straight to everyone right now instead of testing it on a few people first — sets up everything that happens next.
| Thing | Value |
|---|---|
| Service | cart-api — Northwind Retail's checkout cart, same fictional company as the course case study |
| Deployment strategy in place | Canary via Argo Rollouts — 20% → 60% → 100%, 30s holds |
| Current stable | v1.8.0, healthy, 5/5 |
| The deadline | Marketing's flash-sale email sends at 15:00 UTC — expected 6× checkout traffic the instant it lands |
| The mistake, 14:42 UTC | 🦫 Benny ships a one-line copy fix to the cart page, sees the clock, and force-promotes straight to 100% instead of waiting out the canary steps — "it's one line, it's fine, we don't have time" |
Recreate Benny's mistake yourself — this is the only step in the whole drill where you're deliberately doing the wrong thing on purpose:
docker build --build-arg BAD_RELEASE=true -t localhost:5000/cart-api:v1.9.0-hotfix . docker push localhost:5000/cart-api:v1.9.0-hotfix kubectl argo rollouts set image cart-api -n cart cart-api=localhost:5000/cart-api:v1.9.0-hotfix kubectl argo rollouts promote cart-api -n cart --full # 14:42 UTC — skips every step, straight to 100%
Watch your terminal 3 loop. Within a few seconds it stops being a wall of 200s.
14:43 UTC — the page fires
☺ Like you're 10: Before you touch anything, look — is the fire still spreading, or did it already burn out and stop?
curl loop, terminal 3, last 20:
200 200 500 200 500 500 200 200 500 200 500 200 200 500 500 200 500 200 500 200
PAGERDUTY — cart-oncall
14:43 UTC ALERT: cart-api checkout error rate 41% (baseline <0.5%)
17 minutes to flash-sale send41% of checkouts failing, and you don't yet know whether this rollout is still mid-canary — where a partial rollback might already be underway on its own — or whether it already finished promoting. Decision: what's the very first command you run?
Check the response
kubectl argo rollouts get rollout cart-api -n cart — before touching anything else. Every command after this one depends entirely on what it says: if the Rollout reads Progressing, mid-step, an abort immediately snaps it back to stable. If it reads Healthy — fully promoted, no canary in flight — abort has nothing to abort. Reaching for a fix before reading status is the single most common wasted-seconds mistake under a real page: you run:
NAME KIND STATUS AGE INFO ⟳ cart-api Rollout ✔ Healthy 9m └──# revision:2 └──⧉ cart-api-7f9c8d-2 ReplicaSet ✔ Healthy 1s stable Images: localhost:5000/cart-api:v1.9.0-hotfix (stable)
Benny's --full promote already finished. This Rollout isn't mid-canary — it's already Healthy, fully on the bad image, marked stable. There is nothing in progress to interrupt. That single fact is the whole drill's turning point.
14:44 UTC — abort, undo, or something faster?
☺ Like you're 10: Three different ways to say "go back," and only one of them actually beats the clock.
You know three commands exist. abort is the one everyone reaches for first because it's the word that sounds fastest. undo is the one that sounds correct because it's what you'd type on a plain Deployment. And there's a third option: explicitly set the image back to v1.8.0 yourself and force it, the same --full flag Benny used to get you into this. Decision: which one do you run, sixteen minutes before the email fires?
Check the response
Set the image back to the known-good tag and force-promote it — not undo, and don't bother with abort at all.
# what abort would tell you, if you tried it — nothing to abort kubectl argo rollouts abort cart-api -n cart # error: rollout is not in a progressing state, nothing to abort # the fast, correct move — 14:44 UTC kubectl argo rollouts set image cart-api -n cart cart-api=localhost:5000/cart-api:v1.8.0 kubectl argo rollouts promote cart-api -n cart --full kubectl argo rollouts get rollout cart-api -n cart --watch
abort only has power over a rollout that's actively Progressing — it works by scaling the in-flight canary ReplicaSet back down and the previous stable one back up. Once a rollout has finished promoting, there's no "in flight" left for it to touch; you already proved that at 14:43. undo would technically work, but it sets the Rollout's template back to the prior revision's spec and lets the controller treat that as a new rollout — which means it walks the same canary ladder again: 20%, a 30-second hold, 60%, another hold, 100%. That's roughly 90 seconds during which some fraction of live traffic is still landing on the bad replicas as they scale down in step with the good ones scaling up — safe eventually, but not fast. Explicitly setting the image to the tag you already trust and forcing --full skips every step and cuts every replica over in the time it takes five pods to become ready, typically single-digit seconds. Skipping canary steps is reckless heading into an unverified image — that's exactly Benny's mistake. Skipping them heading back to an image you already ran cleanly for nine minutes is a different call entirely, and it's the right one.
14:47 UTC — the migration wrinkle
☺ Like you're 10: Rolling back the app is only half the swap — if the old app still needs a drawer you just emptied, it breaks in a whole new way.
Pods are cutting over. Then a second alert:
#cart-team — Slack, 14:47 UTC
@benny: wait — v1.9.0-hotfix's migration also ran. it dropped
legacy_sku from cart_items, the copy fix touched that
column's default text. v1.8.0 still SELECTs it.
DB LOG — 14:47:12 UTC
ERROR: column "legacy_sku" does not exist at character 34
STATEMENT: SELECT id, sku, legacy_sku, qty FROM cart_items WHERE ...Your fast app rollback is landing v1.8.0 pods that immediately start erroring on a column that no longer exists. A generated "down" migration would ADD COLUMN legacy_sku back — but naively re-running it blind, without checking what happened in between, risks silently losing any writes that happened to the new column shape while v1.9.0 was live. Decision: run the generated down-migration as-is, or something else?
Check the response
Something else — a compensating migration, not a blind down-migrate. First, check whether anything actually wrote data in the two minutes v1.9.0 was live in a shape v1.8.0 can't read back (in this case: no, the drop was the only schema change, nothing new was written to reconcile). Only then re-add the column, nullable, with the same default it had before:
ALTER TABLE cart_items ADD COLUMN legacy_sku text NULL DEFAULT '';
This is database change management's expand/contract pattern applied under fire: dropping a column belongs in a later, separate deploy, only after every version of the app that might read it is fully retired — never bundled into the same release as the code that stops using it. Benny's hotfix violated that rule by shipping the drop and the code change together, which is exactly what turned a one-line copy fix into a two-system incident. Blindly trusting a generated down-migration under pressure is its own failure mode: it reverses the schema, but says nothing about data written in between, and "probably fine" is not a sentence that belongs in a rollback plan.
14:53 UTC — green for ninety seconds
☺ Like you're 10: A clean minute and a half feels like victory, but the sale email is still seven minutes out — is ninety seconds actually proof, or just a good moment?
curl loop, terminal 3, last 30: 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 7 minutes to flash-sale send
Every request for the last ninety seconds has been a clean 200. Decision: declare it resolved now, with seven minutes of runway before the email, or hold longer?
Check the response
Hold — two full clean minutes, minimum, not ninety seconds — and use the runway you have, don't burn it declaring early. Ninety seconds clean is a good sign, not proof: a connection pool that was mid-exhaustion when the fix landed doesn't necessarily finish settling the instant the bad replicas disappear, and the compensating migration from the last scene only just committed. You have seven minutes before the real test — 6× traffic at 15:00 UTC — which is more than enough room to watch two full clean minutes and then run one deliberate burst of load yourself to approximate the coming spike, rather than finding out at 15:00 UTC alongside marketing. Declaring resolved on a good-looking moment instead of a sustained window is the same mistake as celebrating the tray swap before checking whether a burnt cookie was already handed out — the visible fix isn't the same claim as "durable."
14:56 UTC. Two clean minutes confirmed, one synthetic 6×-load burst run and absorbed cleanly. You declare resolved four minutes before the send. 15:00 UTC: the email goes out. Checkout holds.
15:04 UTC — lock the door before it happens again
☺ Like you're 10: The fire's out — now make it harder for the next person in a hurry to skip the same step that started this one.
The incident is over, but the exact command that started it — promote --full on an unfinished canary — is still one keystroke away from anyone with cluster access, the next time someone's seventeen minutes from a deadline. Decision: what do you actually change before you close this out, versus what do you just write down and hope people remember?
Check the response
Change something enforced, don't just write a reminder. A wiki note saying "don't force-promote under deadline pressure" is exactly the kind of guidance the pressure itself makes people skip — that's the whole reason this incident happened once already. The enforceable version is RBAC on who can patch the Rollout resource's promotion behavior at all:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: rollout-promoter, namespace: cart }
rules:
- apiGroups: ["argoproj.io"]
resources: ["rollouts"]
verbs: ["get", "list", "watch", "patch"] # patch covers promote/abort/retry
# bind this Role only to an sre-oncall group, via a RoleBinding —
# everyone else can watch a Rollout and read its status, nobody
# else can push one forward or skip its stepsThis doesn't remove the ability to force-promote in a genuine emergency — the on-call group still has it, which is exactly what this drill just used. It removes the ability to do it solo, on impulse, seventeen minutes before a deadline, with nobody else in the loop — the actual failure mode, not the tool. Pair it with a written postmortem, blameless, naming Benny's action in the same neutral tense as every other fact, per incident management — the fix belongs to the system that let one keystroke skip a safety step, not to Benny for being in a hurry under a deadline the whole company set.
Rolling back under every other strategy
☺ Like you're 10: Different swap tricks need different quick moves to undo them — know all four before you need one for real.
Deployment strategies covers four patterns; this drill drilled the canary lever specifically because it's the one with the sharpest, least-obvious failure mode. The other three each have their own single fastest move, worth having cold before you're the one holding the pager:
| Strategy | Fastest rollback lever | What it doesn't fix |
|---|---|---|
| Rolling | kubectl rollout undo deployment/cart-api — reverts to the previous ReplicaSet revision | Still ramps pod-by-pod under the Deployment's own maxSurge/maxUnavailable, not instant |
| Blue-green | Swap the router back to the idle environment — a target-group or DNS cutover, seconds | Nothing, if blue was kept warm and idle — that's the whole point of paying for two environments |
| Canary (this drill) | If mid-rollout: abort. If already fully promoted: explicit set image to the last-good tag, then promote --full | Any schema or data change bundled into the same release — see the migration scene above |
| Feature flag | Flip the flag off at the flag service — no deploy at all, milliseconds | Anything not gated by that flag, including a bundled migration or infra change riding along with it |
Notice the pattern repeats: every "what it doesn't fix" column is really the same warning from database change management and feature flags & progressive delivery in different clothes — a fast application-layer rollback only undoes what that layer actually controls. A destructive migration, a config change applied outside the deploy pipeline, or a third-party webhook fired mid-rollout all survive every one of the levers above.
Reaching for abort out of habit without checking rollout status first — it silently does nothing on a fully-promoted rollout, costing you the seconds you least have. Running undo under real time pressure without knowing it re-walks the canary ladder — safe, but slower than you assumed when you typed it. Trusting a generated down-migration without checking what it actually reverses, versus what data moved while the bad version was live. And declaring resolved on the first clean-looking moment instead of a sustained window — the version of "it's fixed" that gets walked back publicly ten minutes later.
Score the drill
☺ Like you're 10: Add up your points against what you actually decided before reading each answer — not what looks obvious now that you already know it.
| Decision | Points | Full credit requires |
|---|---|---|
| 1. First command at 14:43 | 4 | Checked rollouts get rollout status before running any fix |
| 2. Abort, undo, or fast revert at 14:44 | 4 | Went straight to explicit set image + promote --full, recognized abort was a no-op and undo was slower |
| 3. The migration at 14:47 | 4 | Chose a compensating migration over a blind down-migrate, after checking for interim writes |
| 4. Declare resolved at 14:53 | 4 | Held for a full sustained window instead of a 90-second good moment |
| 5. Lock the door at 15:04 | 4 | Chose an enforced RBAC change over a written reminder alone |
Every fast, correct move in this drill was fast specifically because it targeted a version already proven healthy — nine minutes of clean traffic on v1.8.0 is what made skipping the canary steps backward the right call. The same skip, forward, into v1.9.0-hotfix, is exactly what started the incident. Speed isn't the variable that makes a promote safe or reckless — what you already know about the target is.
17–20: this is what a fast, correct rollback actually looks like under a real deadline. 11–16: solid instincts — reread whichever scene cost you points before your next real on-call shift. 0–10: reread deployment strategies in full, then run this drill again from the top — the gap is usually a missing mental model of what each command actually does, not a lack of effort.
Benny the Beaver: In my defense, it was one line. Cart copy text. What could it possibly—
Timmy the Turtle: "It's one line" is exactly what everyone says right before I have to explain why abort just told them there's nothing to abort.
Foxy: Wait, so the fast fix is to skip the canary steps too — just backward instead of forward? Isn't that the same shortcut Benny took?
Timmy the Turtle: Same command, opposite risk. Forward, into v1.9.0, nobody had proven it safe yet. Backward, into v1.8.0, you'd already watched it run clean for nine minutes. Skipping steps isn't the sin — skipping them into the unknown is.
Pip the Hummingbird: And I paged cart-oncall the second the error rate crossed 40% — which is exactly why you had seventeen minutes instead of finding out from the sale email bouncing.
Timmy the Turtle: Ninety seconds of green doesn't get my sign-off, either. Hold for the real window, then declare it — every time.
1. Why did kubectl argo rollouts abort fail to do anything against Benny's fully-promoted hotfix, and what state does a Rollout need to be in for abort to have any effect? 2. What's the operational difference between kubectl argo rollouts undo and an explicit set image + promote --full when both are reverting to the same known-good version, and why does that difference matter with a clock running? 3. Why wasn't running the generated down-migration automatically the safe choice, and what's the correct fix instead? 4. Name the single fastest rollback lever for each of the other three deployment strategies — rolling, blue-green, and feature flag — and one thing each of them still doesn't fix.
Check your answers
abortonly has power over a rollout that's activelyProgressing— it scales the in-flight canary ReplicaSet down and the previous stable one back up. Oncepromote --fullhad already finished, the Rollout readHealthywith no canary in flight, so there was nothing forabortto act on.undosets the Rollout's template back to the prior revision and lets the controller treat it as a new rollout, which walks the same canary step ladder again — roughly 90 seconds during which some traffic still lands on the bad replicas as they scale down. An explicitset imageto the trusted tag plus a forced--fullpromote skips the steps entirely and cuts every replica over in single-digit seconds, because the target is already proven safe rather than unverified.- Because a generated down-migration reverses the schema but says nothing about data written in the interim shape while the bad version was live — running it blind risks silently losing that data. The correct fix is a compensating migration, informed by actually checking what changed in between, following the expand/contract pattern: never drop what a still-running version needs in the same release that stops using it.
- Rolling:
kubectl rollout undo deployment/<name>— doesn't fix the fact it still ramps pod-by-pod rather than instantly. Blue-green: swap the router back to the idle environment — doesn't fix anything if the idle environment wasn't actually kept warm. Feature flag: flip the flag off at the flag service — doesn't fix anything not gated by that flag, including a bundled migration.
Every command above transfers straight to whatever your real stack runs — Flagger instead of Argo Rollouts, a different flag service, a different migration tool — because the decisions, not the exact flags, are the muscle memory this drill built. Next up: Drill — Diagnose a Production Incident picks up after a rollback like this one doesn't fully explain what happened, and Capstone Part 3 — Deployment Strategy is where you build the canary and its analysis gate from nothing instead of inheriting one. If abort catching you off guard was the real lesson here, reread deployment strategies' rollback-design section before your next real on-call shift.