Hands-On Labs · The Capstone

Ship it, start to finish

The DOP-C02 exam is closed-book, multiple-choice — nobody hands you a broken cluster during the test. But every scenario question it asks ("which deployment strategy limits blast radius here," "why did this pipeline just fail," "which service closes this security gap") is really asking whether you've built the thing it's describing. This hub is where you build it. You'll take checkout-svc — a small stand-in for Northwind Retail's Postgres-backed checkout service, the same fictional system this course's case study already follows — from a bare git repo holding nothing but application code, through six connected stages, to a pipeline a real on-call engineer could trust: tested, reconciled from code, deployed safely, watched, paged on, and hardened. Each stage opens by stating exactly what "done" looks like. If you only have an afternoon, the fast path below still gets you real reps across all six.

☺ Explain it like I'm 10

Reading a recipe doesn't teach you to run a kitchen. This is the part where you cook. You'll take one small dish — checkout-svc — from a cold kitchen with no process at all into a working line, one station at a time: a prep line that checks its own work before it fires a ticket, a pantry stocked from a recipe card instead of someone's memory, an expo who plates a new dish for two tables before serving the whole room, gauges on the pass-through window so you know a dish is running hot before a table complains, a runner who pages the chef the instant a plate comes back, and a health inspector who won't let anything unsafe leave the kitchen. Same one dish. Same one line. Six stations, in order.

🦫🛠️Your host for this track: Benny the Beaver — the builder. He hands each stage to the right specialist (Timmy, Recon, Ellie, Pip, Sol) and won't call a stage "done" until its own done-when line is actually true. The whole Guild is on call.
⚠ Before you start

Every stage below is local and throwaway — nothing here touches production or costs real money. You'll want Docker (or Podman), a local Kubernetes cluster via kind (minikube works too — translate the commands), kubectl, helm, terraform, and git. Stage 5 benefits from a free-tier PagerDuty or Opsgenie account so a page reaches a real device — no card required for the free tier, and you can skip it and simulate the page if you'd rather not sign up for anything. Commands and version numbers drift — always check each project's current docs. Tear everything down when you're done (kind delete cluster --name checkout-dev, docker rm -f registry) and nothing lingers.

The world you're building

☺ Like you're 10: Before any building starts, here's the map: one cluster, one small service, two drawers to keep things tidy in, and one repo that grows a new room every stage.

The hub you're reading and all six stages build inside one continuous project, so nothing you set up in Stage 1 goes stale by Stage 6. Here is the whole world, once, so you never have to re-derive it:

checkout-dev (kind cluster)
├── namespace: platform          # Argo Rollouts, kube-prometheus-stack, Vault, Kyverno…
├── namespace: checkout           # the checkout-svc Deployment/Rollout, Service, Postgres, ServiceMonitor…
└── (git) checkout-svc/
    ├── app/, tests/, Dockerfile       # the bare repo you start with
    ├── .github/workflows/ci.yml       # Stage 1 — pipeline foundation
    ├── infra/                         # Stage 2 — infrastructure as code (Terraform)
    ├── deploy/                        # Stage 3 — deployment strategy (canary Rollout)
    ├── observability/                 # Stage 4 — dashboards & alert rules
    ├── runbooks/                      # Stage 5 — incident response
    └── policy/                        # Stage 6 — security hardening
◆ Key idea

Every stage opens by stating exactly what state your cluster and repo are in when you arrive (inherited from the stage before) and what state they're in when you leave (handed to the next one). If a stage ever feels like it's starting from nothing, re-read its opening paragraph — you're missing something the previous stage was supposed to leave behind.

The six stages

☺ Like you're 10: Six chapters of the same story. Each one hands the next chapter a repo and a cluster that are a little more finished than the ones before.

Work them in order. Each stage pairs 1:1 with a DOP-C02 exam domain, so you always know which slice of the real exam your hands are earning you reps in — and across all six, that's every domain, at its real weight. Each has its own checkbox milestones and its own "done when" bar to clear before you move on.

#StagePairs withDone when
1Pipeline Foundation — 🦫 Benny & 🐢 TimmyCI/CD Pipelines + Version Control & Branching · SDLC Automation (D1, 22%)a PR triggers the pipeline, a clean merge produces an immutable SHA-tagged image, and a red pipeline provably can't merge.
2Infrastructure as Code — 🤖 ReconInfrastructure as Code + Configuration Management · Config Management & IaC (D2, 17%)a rerun of terraform apply reports zero changes, and a manually deleted resource is caught by plan and restored by apply.
3Deployment Strategy — 🐢 Timmy & 🐘 EllieDeployment Strategies · Resilient Cloud Solutions (D3, 15%)a healthy build promotes through every canary step unattended, and a broken one aborts back to stable with no human touching a button.
4Observability — 🐘 EllieMonitoring & Observability · Monitoring & Logging (D4, 15%)your dashboard shows real traffic on all four golden signals, the alert fires on a fault you injected, and you can state root cause in one sentence.
5Incident Response — 🐦 Pip & 🤖 ReconIncident Management · Incident & Event Response (D5, 14%)a real page reaches you inside a stated SLA, the runbook gets you to mitigation, and a blameless postmortem lands in the repo.
6Security Hardening — 🐢 Timmy & 🤖 ReconSecurity & Compliance (D6, 17%)an unsigned or :latest-tagged manifest is refused at admission, and your pod's DB credential is a short-lived Vault lease, not a static secret.

Deep-dive the concept behind any stage via its linked lesson before or after you build — the lesson explains why, the stage makes you prove it works.

The fast path — for an afternoon

☺ Like you're 10: No time for six chapters? Here's the one-page summary of the whole story, still worth doing.

The six stages are the deep version of this track — real manifests, real breakage, real triage. If you only have an afternoon, or you want one condensed pass before the exam, work this shorter nine-item version instead. It touches every domain at a lighter depth, using the exact same checkout-svc/checkout-dev world as the stages above, so nothing you build here is wasted if you come back and go deep later.

0 / 9 fast-path items complete
0Stand up the ground — 🦉 Professor Owl
Install Docker, kind, kubectl, terraform, and git. Create the cluster with kind create cluster --name checkout-dev. Start a throwaway local registry (docker run -d -p 5000:5000 --name registry registry:2) and point registry.internal at 127.0.0.1 in your hosts file. Initialize the checkout-svc repo with a minimal app (GET /health, GET /cart/:id), a Dockerfile, and one passing test — nothing else yet.
Done when: kubectl get nodes shows a Ready node, your local registry answers curl localhost:5000/v2/_catalog, and docker build && docker run serves /health on your laptop.
1A pipeline that guards the door — 🦫 Benny & 🐢 Timmy
Add a CI workflow (GitHub Actions, or a Jenkinsfile if you'd rather run Jenkins) that on every PR installs dependencies, runs the test suite, and builds registry.internal/checkout-svc:$GIT_SHA; on merge to main, it pushes the image. Protect main so a red pipeline can't merge. Break a test on purpose in a PR and confirm the merge button won't go green.
Done when: a PR triggers the pipeline, a clean merge produces an immutable SHA-tagged image in your registry, and a failing test provably blocks the merge.
2The pantry, written down — 🤖 Recon
Write Terraform for the checkout and platform namespaces, a ResourceQuota, and your registry credentials as a kubernetes_secret resource — no more kubectl create by hand. Run terraform apply, then run it again against unchanged config and confirm zero diff. Now cause drift: kubectl delete the quota by hand, watch terraform plan flag it, then apply to restore it.
Done when: a second, unchanged terraform apply reports no changes, and a manually deleted resource is caught by plan and put back by apply.
3Plate it for two tables first — 🐢 Timmy & 🐘 Ellie
Deploy checkout-svc with an Argo Rollouts (or Flagger) canary: 10% → 50% → 100% with pauses, and an analysis step that queries error rate and aborts automatically if it trips. Ship a good version through and watch it promote unattended. Ship a version that 500s on /checkout and watch the rollout abort back to stable without you touching anything.
Done when: a healthy version promotes through every canary step on its own, and a broken version is automatically rolled back before it reaches 100%.
4Gauges on the pass-through window — 🐘 Ellie
Install kube-prometheus-stack into platform. Add a ServiceMonitor for checkout-svc (request count, latency histogram, error count on /metrics), a Grafana panel per golden signal, and a PrometheusRule alerting on error rate. Inject a fault — drop the memory limit until it OOMKilleds, or point it at a bad DB host — and triage with kubectl describe and kubectl logs --previous.
Done when: your dashboard shows real traffic on all four golden signals, the alert fires on the injected fault, and you can write the root cause in one sentence.
5A runner who actually pages you — 🐦 Pip & 🤖 Recon
Wire the Stage 4 alert to a real pager (PagerDuty or Opsgenie's free tier), routed to your own phone or Slack. Re-trigger the fault with the pager live and time your own acknowledgment and recovery. Mitigate using a one-page runbook you write beforehand. Close with a written blameless postmortem: timeline, contributing factors, and an action item assigned to the system, never to yourself.
Done when: a real page reaches you within a stated SLA, the runbook gets you to mitigation without improvising from zero, and a blameless postmortem exists in the repo.
6The health inspector — 🐢 Timmy & 🤖 Recon
Move the DB credential out of a plain Kubernetes Secret and into Vault (dev-mode server is fine), fetched by the pod at startup. Add a SAST scan and an image scan to the Stage 1 pipeline, gated so a Critical CVE blocks the push. Add a Kyverno (or OPA/Gatekeeper) policy — audit first, then enforce — blocking :latest tags and requiring non-root. Generate an SBOM and sign the image.
Done when: a :latest-tagged or unsigned manifest is refused at admission, your running pod's DB credential is a short-lived Vault lease, and your pipeline fails on an injected Critical CVE.
7Know what the meal costs — 🦥 Sol
Tag every resource your Stage 2 Terraform created with a cost-center label. Pull a rough runner-minute figure for your Stage 1 pipeline and a rough compute cost for the checkout namespace. Compare one pipeline run before and after you add caching or parallelize a slow step.
Done when: you can name, even roughly, in real currency, what one deploy of checkout-svc costs from commit to running pod.
CapstoneProve it worked — 🦥 Sol & the whole Guild
Pull your own DORA four out of what you actually built, not out of a textbook: deployment frequency from how often Stages 1–3 shipped, lead time for changes from your pipeline's own commit-to-production timestamps, change failure rate from how many of your deploys needed the Stage 3 abort or a Stage 5 incident, and MTTR straight off your Stage 5 postmortem. Compare your four numbers honestly against the Elite/High/Medium/Low tiers.
Done when: you have four real numbers of your own, and you can say — honestly, not hopefully — which tier they land in.
Concept: Measuring Success: the DORA Metrics · This is where every stage above gets weighed.
🦥 Sol's-eye view

"Whichever version you did — the fast path or the six deep stages — don't just tell me it worked. Count it. If your own deployment frequency and change failure rate don't beat what you started with, something above still needs fixing, not a better story about it. If the numbers move, you've built the thing this whole course is actually about."

What you'll have built

☺ Like you're 10: By the end, your laptop holds a small working version of the whole restaurant this course keeps describing — the prep line, the pantry, the expo, the gauges, the runner, and the inspector.

Finish the six stages (or the fast path) and you'll have hands-on reps across every DOP-C02 domain, at its real exam weight: SDLC Automation (D1, 22%), Configuration Management & IaC (D2, 17%), Resilient Cloud Solutions (D3, 15%), Monitoring & Logging (D4, 15%), Incident & Event Response (D5, 14%), and Security & Compliance (D6, 17%) — the full 100%. That's not exam trivia; it's the muscle memory that makes a scenario question answerable on sight instead of guessable by elimination. Pair this with the exam guide and study plan once they're up, and lean on Certifications for the wider landscape — including hands-on exams like the CKA, where you genuinely will be handed a broken cluster.

⚖ Closed-book, but still worth building — The AWS Certified DevOps Engineer – Professional exam (DOP-C02) is a closed-book exam of multiple-choice and multiple-response questions; there is no lab or task component, no terminal, no cluster to fix under a clock. (Exam format, length, and price drift over time — always confirm the current numbers on AWS's own exam page before you register.) So why build any of this? Because the exam's scenario questions are written by people who assume you've done exactly what these six stages walk you through — a question about which deployment strategy limits blast radius reads very differently once you've watched Stage 3's canary actually abort a bad build on its own. Closed-book doesn't mean untested; it means the test happens in your head, at the terminal, before you ever see the multiple-choice version of it.

🦫 Benny's challenge · going further

Already comfortable? Level up any stage: put the whole platform under GitOps so infra/, deploy/, and policy/ all reconcile from the same repo instead of being applied by hand; add a blue-green option alongside the canary and compare their rollback times; wire a second, dependent service and add a contract test between them; run a real chaos experiment (kill a pod, saturate the node, black-hole the database) during business hours instead of waiting for it to happen to you; or sign your image with cosign and make admission reject anything unsigned, not just anything on :latest.

🎬 At the Ship-It Guild
🦊

Foxy: Six stages feels like a lot for one checkout service. Can't I just skim the fast path and call it done?

🦫

Benny the Beaver: You can, Foxy, and you'll still walk away with real reps. But the fast path is the map — the six stages are the actual territory, with a pipeline that really breaks and a rollout that really aborts.

🐢

Timmy the Turtle: And I'm not signing off Stage 3 until a bad build actually gets caught and rolled back on its own. Watching that happen once teaches you more than reading about canaries ever will.

🐦

Pip the Hummingbird: Stage 5's the one people skip, and it's the one the exam loves to ask about sideways. Get paged for real just once, and every "what should you do first" question gets easier.

🦥

Sol the Sloth: And when you're done, don't just believe it worked — count it. Pull your own deployment frequency and change failure rate off what you actually built. Slow, honest arithmetic beats a hunch every time.

🦉

Professor Owl: One repo, six stages, one loop. Go build it.

🐢 Timmy's checkpoint

1. Which stage proves your infrastructure is actually idempotent, and what two-part test proves it? 2. Which stage stops a bad deploy from ever reaching 100% of production traffic without a human pressing anything? 3. The DOP-C02 exam never puts you in a terminal — so why does this hub still make you build a real pipeline instead of just reading about one?

Check your answers
  1. Stage 2 — Infrastructure as Code. The two-part test: running terraform apply a second time against unchanged config produces zero diff, and a resource you delete by hand out-of-band is both caught by terraform plan and restored by apply.
  2. Stage 3 — Deployment Strategy. The canary's analysis step checks error rate at each traffic step and auto-aborts back to the last stable version the moment it trips — no human has to notice or press a rollback button.
  3. Because the exam's multiple-choice scenarios describe real failures and real trade-offs, and they're far easier to answer correctly once you've watched the actual mechanism happen on your own cluster — a canary abort, a Vault lease expiring, a page arriving at 2 a.m. — than if you've only ever read a description of it.