CI/CD & Progressive Delivery Labs
This is the domain where the exam stops asking what you know and starts watching what you can build. Over twelve labs you’ll assemble a whole delivery path with your own hands: a Tekton Task that grows into a Pipeline, a workspace threading source between steps, results passed from one Task to the next, a container image built inside the cluster with no Docker daemon, credentials scoped to a ServiceAccount, a webhook that fires a run, an Argo Workflows DAG — and then the safe half: a canary that shifts traffic in steps, an AnalysisTemplate that queries Prometheus and aborts a bad version on its own, a blue/green preview you promote by hand, the same canary expressed in Flagger, and finally a feature flag that releases without deploying anything at all. Everything runs on a throwaway local cluster. Tick each lab off as you finish it — your progress saves in this browser.
Two machines live in this lesson. The first is a conveyor belt: you drop code on one end and a finished, packaged toy comes out the other, with no human carrying anything. The second is a careful shopkeeper: instead of handing the new toy to every child at once, she gives it to one child in ten, watches whether they cry, and only then hands out more — and if anyone cries, she takes it straight back. You’re going to build both belts and both shopkeepers today, on your own laptop, and then deliberately break things so you can watch the shopkeeper snatch the toy back.
These are local, throwaway labs. Nothing here should ever point at a real cluster, a real registry, or anything you’d miss — you will deliberately break workloads, turn on registry auth to watch a push fail, and ship images that crash on purpose. Tear it all down at the end with kind delete cluster --name cnpe-cicd. Also: versions and flags drift constantly in this corner of the ecosystem — Tekton’s API graduated v1beta1 → v1, the kubectl argo rollouts plugin ships new subcommands, Flagger’s chart values get renamed. Treat every URL and flag below as a shape to recognise, and follow each project’s current quickstart for exact install commands. Recognising the shape is what the exam tests; memorising a release URL is not. That quickstart-hopping is a prep-time habit, not an exam-day one: on the real CNPE, the allowlist is narrow — kubernetes.io/docs, kubernetes.io/blog, whatever task-specific link sits in that task’s own Quick Reference box, and local man/usr-share docs on the exam machine — not tekton.dev, argoproj.io, flagger.app, linkerd.io, or any other project’s own site.
⚖ CNPA vs CNPE — Twelve hands-on labs assembling and breaking a real pipeline on a live cluster is a CNPE-only format — CNPA has no lab or task component whatsoever, it’s a fully closed-book multiple-choice exam with zero external lookups of any kind. But the concepts these labs drill — Tasks vs. Pipelines, workspaces vs. results, in-cluster builds, and progressive delivery via canary, blue/green and feature flags — are exactly the kind of platform-engineering knowledge CNPA’s closed-book recall still tests, just as multiple-choice recognition rather than something you build and break yourself.
Before you start — set the stage
You need a cluster with a default StorageClass (kind and minikube both ship one), kubectl, helm, and Docker or Podman. Run this once; the rest of the labs build on it. The little in-cluster registry is deliberately insecure and anonymous for now — Lab 5 is where you lock it down and feel the consequences.
# a throwaway cluster + a working namespace for the pipeline kind create cluster --name cnpe-cicd # or: minikube start kubectl create namespace cicd kubectl config set-context --current --namespace=cicd # Tekton Pipelines + the tkn CLI (check tekton.dev for the current release URL) kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml kubectl -n tekton-pipelines rollout status deploy/tekton-pipelines-controller kubectl -n tekton-pipelines rollout status deploy/tekton-pipelines-webhook # ^ wait for the WEBHOOK too: apply a Task before it is ready and you get a # "failed calling webhook ... connection refused" that looks like bad YAML tkn version # brew install tektoncd-cli, or grab the binary # a throwaway OCI registry inside the cluster — anonymous for now kubectl create namespace registry kubectl -n registry create deployment registry --image=registry:2 kubectl -n registry expose deployment registry --port=5000 # reachable from any pod as: registry.registry.svc.cluster.local:5000
Work the labs in order — Labs 1–7 build the conveyor belt, Labs 8–12 build the safe release. Every lab names the command that proves it worked; if a command surprises you, that’s the lab doing its job. Deep-dive any concept through the linked lesson, and keep the command reference and the troubleshooting playbook open in another tab.
# lab1.yaml — a Task, then a Pipeline that runs two of them in order
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: say
spec:
params:
- name: message
type: string
default: "hello from a pod"
steps:
- name: echo # every step is a container in one pod
image: alpine:3.20
script: |
#!/bin/sh
echo "$(params.message)"
sleep 2
---
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: greet
spec:
params:
- name: who
type: string
tasks:
- name: first
taskRef: { name: say }
params:
- { name: message, value: "building for $(params.who)" }
- name: second
runAfter: [first] # drop runAfter and they run in parallel
taskRef: { name: say }
params:
- { name: message, value: "shipped for $(params.who)" }- Apply the manifest above:
kubectl apply -f lab1.yaml. - Run the Task on its own and watch the logs stream:
tkn task start say --param message="first run" --showlog. Note thattkncreated aTaskRunfor you —tkn taskrun list. - Now run the Pipeline:
tkn pipeline start greet --param who=checkout --showlog. - Look at the shape underneath:
kubectl get pipelinerun,taskrun,pod. One PipelineRun owns two TaskRuns; each TaskRun owns exactly one pod, and each step is a container in it. - Delete
runAfter: [first], re-apply, re-run, and compare timestamps withtkn pipelinerun describe --last— the two Tasks now overlap.
tkn pipelinerun list shows a run with status Succeeded, and kubectl get taskrun -l tekton.dev/pipeline=greet returns exactly two TaskRuns — the ones the PipelineRun owns, as opposed to the standalone one tkn task start made. You can say out loud which of Task / TaskRun / Pipeline / PipelineRun is a definition and which is an execution.# lab2.yaml — one PipelineRun, one PVC, two Tasks sharing it
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: clone-and-look
spec:
workspaces:
- name: shared # declared here, bound at run time
tasks:
- name: write
taskRef: { name: writer }
workspaces:
- { name: ws, workspace: shared }
- name: read
runAfter: [write]
taskRef: { name: reader }
workspaces:
- { name: ws, workspace: shared }
---
apiVersion: tekton.dev/v1
kind: Task
metadata: { name: writer }
spec:
workspaces: [{ name: ws }]
steps:
- name: write
image: alpine:3.20
script: |
#!/bin/sh
date > $(workspaces.ws.path)/built-at.txt
echo "wrote $(workspaces.ws.path)/built-at.txt"
---
apiVersion: tekton.dev/v1
kind: Task
metadata: { name: reader }
spec:
workspaces: [{ name: ws }]
steps:
- name: read
image: alpine:3.20
script: |
#!/bin/sh
ls -l $(workspaces.ws.path)
cat $(workspaces.ws.path)/built-at.txt # fails loudly if the volume didn't follow
---
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata: { generateName: clone-and-look- }
spec:
pipelineRef: { name: clone-and-look }
workspaces:
- name: shared
volumeClaimTemplate: # a fresh PVC per run, deleted with the run
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 1Gi- Confirm you have a default StorageClass:
kubectl get sc. Without one, the run will sit inPendingforever — that alone is worth seeing once. - Create everything in one pass:
kubectl create -f lab2.yaml. It must becreate, notapply—applyrejects thegenerateNameon the PipelineRun. Save that last PipelineRun block on its own asrun2.yamlso repeat runs are justkubectl create -f run2.yaml. - While it runs:
kubectl get pvc -w. Tekton materialised a PersistentVolumeClaim for this run alone. - Read the logs:
tkn pipelinerun logs --last -f. The reader Task must print a file it never created. - Now break it on purpose. In
run2.yaml, swap the wholevolumeClaimTemplateblock foremptyDir: {}andkubectl create -f run2.yamlagain. AnemptyDirworkspace is pod-local, and each TaskRun is its own pod — so the reader gets a fresh empty directory andcatdies withNo such file or directory. That is the error you meet in the exam when source “disappears” between build steps.
tkn pipelinerun logs --last shows the reader Task printing the timestamp the writer Task created, and kubectl get pvc showed a Bound claim during the run.# lab3.yaml — a Task that emits a result, a Task that consumes it
apiVersion: tekton.dev/v1
kind: Task
metadata: { name: make-tag }
spec:
results:
- name: tag
description: the image tag this run should build
steps:
- name: compute
image: alpine:3.20
script: |
#!/bin/sh
# results are tiny values written to a file Tekton hands back — no trailing newline!
# NOTE the backticks: shell $( ) inside a Tekton script collides with Tekton's
# own $(...) substitution syntax, so use `...` for command substitution here.
TAG="1.0.`date +%s`"
printf '%s' "$TAG" | tee $(results.tag.path)
---
apiVersion: tekton.dev/v1
kind: Task
metadata: { name: use-tag }
spec:
params: [{ name: tag, type: string }]
steps:
- name: show
image: alpine:3.20
script: |
#!/bin/sh
echo "would push registry.registry.svc.cluster.local:5000/demo:$(params.tag)"
---
apiVersion: tekton.dev/v1
kind: Pipeline
metadata: { name: tag-flow }
spec:
results:
- name: chosen-tag # bubble it out of the Pipeline too
value: $(tasks.compute.results.tag)
tasks:
- name: compute
taskRef: { name: make-tag }
- name: consume
runAfter: [compute]
taskRef: { name: use-tag }
params:
- { name: tag, value: $(tasks.compute.results.tag) }- Apply the file, then run it:
kubectl apply -f lab3.yaml, thentkn pipeline start tag-flow --showlog. - Read the result back off the run:
tkn pipelinerun describe --last, then the raw form —kubectl get pipelinerun --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1:].status.results}'. - Prove the wiring is real: change
make-tagto print a different string, re-run, and confirm the second Task echoes the new value without you touching it. - Now the classic trap. Replace the
printf '%s' "$TAG"line withecho "$TAG"— the trailing newline lands inside the result, and the downstream image reference silently becomes invalid. Notice how quietly it breaks. - Ask yourself which of the two mechanisms you’ve now used carries files and which carries values. Workspaces move the source tree; results move a string like a digest, a tag, or a commit SHA.
tkn pipelinerun describe --last shows a Pipeline result chosen-tag whose value matches exactly what the consume Task echoed in its logs.# lab4.yaml — build and push an image from inside the cluster, no Docker daemon
apiVersion: tekton.dev/v1
kind: Task
metadata: { name: kaniko-build }
spec:
params:
- name: image
type: string
workspaces: [{ name: source }]
steps:
- name: seed # stand in for a git-clone Task
image: alpine:3.20
script: |
#!/bin/sh
cd $(workspaces.source.path)
printf 'FROM alpine:3.20\nRUN echo built-in-cluster > /hello.txt\nCMD ["cat","/hello.txt"]\n' > Dockerfile
- name: build-and-push
image: gcr.io/kaniko-project/executor:latest # userspace builder: no daemon, no socket
env:
- name: DOCKER_CONFIG # where Tekton drops ServiceAccount creds;
value: /tekton/home/.docker # kaniko would otherwise only read /kaniko/.docker
args:
- --context=$(workspaces.source.path)
- --dockerfile=$(workspaces.source.path)/Dockerfile
- --destination=$(params.image)
- --insecure # our throwaway registry is plain HTTP
- --skip-tls-verify
---
apiVersion: tekton.dev/v1
kind: TaskRun
metadata: { generateName: kaniko-build- }
spec:
taskRef: { name: kaniko-build }
params:
- name: image
value: registry.registry.svc.cluster.local:5000/demo:v1
workspaces:
- name: source
volumeClaimTemplate:
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 1Gi- Apply and run:
kubectl create -f lab4.yaml, thentkn taskrun logs --last -f. Watch Kaniko unpack the base image and push layers. Split the trailing TaskRun block into its ownrun4.yamlas well — Lab 5 re-fires it, andkubectl createon the whole file a second time would trip over the already-existing Task. - Prove the artifact exists — ask the registry, don’t trust the log:
kubectl run reg-check --rm -it --image=curlimages/curl --restart=Never -- curl -s http://registry.registry.svc.cluster.local:5000/v2/_catalog. - Then list the tags: swap the path for
/v2/demo/tags/listand confirmv1is there. - Inspect the pod that did it:
kubectl get pod -l tekton.dev/task=kaniko-build -o jsonpath='{.items[*].spec.containers[*].securityContext}'. Noprivileged: true, no mounted/var/run/docker.sock— that’s the whole point. - Say why in one sentence: mounting the Docker socket into a build pod hands that pipeline the node, and every workload on it.
{"repositories":["demo"]} and the tags endpoint lists v1 — an image built by a pod, with no Docker daemon anywhere in the cluster.# lab5.sh — put a lock on the registry, then hand the run a key
# 1. generate an htpasswd file (no local apache tools needed)
docker run --rm httpd:2 htpasswd -Bbn ci s3cr3t > htpasswd
kubectl -n registry create secret generic registry-auth --from-file=htpasswd
# 2. mount the file FIRST, then switch auth on — do it the other way round and the
# registry crash-loops on a missing /auth/htpasswd while you watch
kubectl -n registry patch deployment registry --type=json -p='[
{"op":"add","path":"/spec/template/spec/volumes","value":[{"name":"auth","secret":{"secretName":"registry-auth"}}]},
{"op":"add","path":"/spec/template/spec/containers/0/volumeMounts","value":[{"name":"auth","mountPath":"/auth"}]}]'
kubectl -n registry rollout status deployment/registry
kubectl -n registry set env deployment/registry \
REGISTRY_AUTH=htpasswd \
REGISTRY_AUTH_HTPASSWD_REALM=Registry \
REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd
kubectl -n registry rollout status deployment/registry
# 3. re-fire the Lab 4 TaskRun unchanged → the push now fails with UNAUTHORIZED. Good.
kubectl create -f run4.yaml
tkn taskrun logs --last -f
# 4. the fix: a credential the run can reach, attached to a ServiceAccount
kubectl -n cicd create secret docker-registry regcred \
--docker-server=registry.registry.svc.cluster.local:5000 \
--docker-username=ci --docker-password=s3cr3t
kubectl -n cicd create serviceaccount pipeline-pusher
kubectl -n cicd patch serviceaccount pipeline-pusher \
-p '{"secrets":[{"name":"regcred"}]}' # SA .secrets, NOT .imagePullSecrets
# a dockerconfigjson Secret needs no annotation — Tekton merges it straight into the
# run's docker config. The tekton.dev/docker-0 annotation is for the OTHER shape,
# a kubernetes.io/basic-auth Secret, which has no registry host of its own:
# kubectl -n cicd create secret generic regcred-basic \
# --type=kubernetes.io/basic-auth --from-literal=username=ci --from-literal=password=s3cr3t
# kubectl -n cicd annotate secret regcred-basic \
# tekton.dev/docker-0=http://registry.registry.svc.cluster.local:5000
# 5. the workspace stanza on its own, so tkn can bind it
cat > vct.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 1Gi
EOF
# 6. run it AS that identity
tkn task start kaniko-build --serviceaccount pipeline-pusher \
--param image=registry.registry.svc.cluster.local:5000/demo:v2 \
--workspace name=source,volumeClaimTemplateFile=vct.yaml --showlog- Run steps 1–2 above to put basic auth in front of the registry. Mount the Secret before flipping
REGISTRY_AUTHon, or you get a crash-looping registry for no teaching value. - Run step 3 — the same Kaniko TaskRun that worked in Lab 4. Read the failure carefully:
UNAUTHORIZEDfrom the registry, not from Kubernetes. Knowing which system said “no” is half of delivery triage. - Run step 4 to create the
kubernetes.io/dockerconfigjsonSecret and list it under the ServiceAccount’ssecrets. Read the commented block underneath too: thetekton.dev/docker-0annotation you keep seeing belongs to basic-auth Secrets, which carry no registry host of their own — a dockerconfigjson Secret already names its server, so it needs no annotation. - Run steps 5–6. Same Task, same image, new identity — it pushes. (Prefer YAML? Re-apply the Lab 4 TaskRun with
serviceAccountName: pipeline-pusheradded underspecand skipvct.yamlentirely.) - Confirm what the run actually ran as:
kubectl get taskrun --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1:].spec.serviceAccountName}'. - If the authenticated run still says
UNAUTHORIZED, the Secret reached the pod but Kaniko never looked at it: Kaniko only reads$DOCKER_CONFIG/config.json, which is why the Task setsDOCKER_CONFIGexplicitly. Tekton has moved that directory between releases — try/tekton/creds/.dockerif/tekton/home/.dockercomes up empty on your version.
UNAUTHORIZED, the run using pipeline-pusher succeeds, and kubectl run reg-check --rm -it --image=curlimages/curl --restart=Never -- curl -s -u ci:s3cr3t http://registry.registry.svc.cluster.local:5000/v2/demo/tags/list returns {"name":"demo","tags":["v1","v2"]}.# lab6.yaml — webhook ➜ EventListener ➜ TriggerBinding ➜ TriggerTemplate ➜ PipelineRun
apiVersion: v1
kind: ServiceAccount
metadata: { name: triggers-sa, namespace: cicd }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: triggers-role, namespace: cicd }
rules:
- apiGroups: ["triggers.tekton.dev"]
resources: ["eventlisteners","triggerbindings","triggertemplates","triggers","interceptors"]
verbs: ["get","list","watch"]
- apiGroups: ["tekton.dev"]
resources: ["pipelineruns","taskruns"]
verbs: ["create","get","list","watch"]
- apiGroups: [""]
resources: ["configmaps","secrets","serviceaccounts"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create","patch"] # the listener EMITS events; watch alone isn't enough
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: triggers-rb, namespace: cicd }
subjects: [{ kind: ServiceAccount, name: triggers-sa, namespace: cicd }]
roleRef: { kind: Role, name: triggers-role, apiGroup: rbac.authorization.k8s.io }
---
# the EventListener also reads two CLUSTER-scoped kinds at startup. Leave this out
# and the sink pod never becomes ready — the #1 cause of "my curl does nothing".
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: triggers-clusterrole }
rules:
- apiGroups: ["triggers.tekton.dev"]
resources: ["clustertriggerbindings","clusterinterceptors"]
verbs: ["get","list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: triggers-crb }
subjects: [{ kind: ServiceAccount, name: triggers-sa, namespace: cicd }]
roleRef: { kind: ClusterRole, name: triggers-clusterrole, apiGroup: rbac.authorization.k8s.io }
---
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerBinding
metadata: { name: from-push, namespace: cicd }
spec:
params:
- name: who
value: $(body.repository.name) # pull fields out of the webhook payload
---
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerTemplate
metadata: { name: run-greet, namespace: cicd }
spec:
params: [{ name: who }]
resourcetemplates:
- apiVersion: tekton.dev/v1
kind: PipelineRun
metadata: { generateName: greet-from-webhook- }
spec:
pipelineRef: { name: greet }
params:
- { name: who, value: $(tt.params.who) }
---
apiVersion: triggers.tekton.dev/v1beta1
kind: EventListener
metadata: { name: push-listener, namespace: cicd }
spec:
serviceAccountName: triggers-sa
triggers:
- name: on-push
bindings: [{ ref: from-push }]
template: { ref: run-greet }- Install Triggers and its core interceptors — two files, both needed (check the project for current URLs):
kubectl apply -f https://storage.googleapis.com/tekton-releases/triggers/latest/release.yamlthenkubectl apply -f https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml. Wait forkubectl -n tekton-pipelines get podsto settle. - Apply
lab6.yaml, then confirm the listener came up:kubectl get eventlistener push-listenershould readREADY: True, andkubectl get svc,pod -l eventlistener=push-listenershould show the Service and pod namedel-push-listenerthat Tekton generated for you. (Theeventlistener=label is on the generated objects, not on the EventListener itself — that’s why they’re two commands.) - Expose it locally:
kubectl port-forward svc/el-push-listener 8080:8080. - In another terminal, be GitHub:
curl -X POST localhost:8080 -H 'Content-Type: application/json' -d '{"repository":{"name":"checkout"}}'. - Watch a run appear out of nothing:
tkn pipelinerun list, thentkn pipelinerun logs --last— it should greetcheckout, the value lifted out of your JSON body. - If nothing happens, read
kubectl logs deploy/el-push-listener. Nine times in ten it’s RBAC — the listener’s ServiceAccount isn’t allowed to create PipelineRuns.
curl produces a new PipelineRun in tkn pipelinerun list, and its logs contain the repository name you put in the JSON body.# lab7.yaml — an Argo Workflows DAG: fan out, then fan in
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata: { generateName: fan-out-in- }
spec:
entrypoint: main
templates:
- name: main
dag:
tasks:
- name: checkout
template: echo
arguments: { parameters: [{ name: msg, value: "clone" }] }
- name: unit-tests # unit-tests and lint depend only on checkout
dependencies: [checkout] # → they run at the same time
template: echo
arguments: { parameters: [{ name: msg, value: "unit tests" }] }
- name: lint
dependencies: [checkout]
template: echo
arguments: { parameters: [{ name: msg, value: "lint" }] }
- name: package # waits for BOTH
dependencies: [unit-tests, lint]
template: echo
arguments: { parameters: [{ name: msg, value: "package image" }] }
- name: echo
inputs: { parameters: [{ name: msg }] }
container:
image: alpine:3.20
command: [sh, -c]
args: ["echo {{inputs.parameters.msg}}; sleep 10"]- Install into its own namespace:
kubectl create ns argo, then apply the currentquick-start-minimal.yamlfrom the Argo Workflows release you choose — the shape iskubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/<VERSION>/quick-start-minimal.yaml, and the-n argomatters because those manifests are namespace-scoped. Install theargoCLI too. Wait forkubectl -n argo rollout status deploy/workflow-controller. - Submit and watch the graph draw itself:
argo submit -n argo --watch lab7.yaml. - While it runs, confirm the fan-out is real:
kubectl -n argo get pods -w— unit-tests and lint pods must beRunningat the same moment, and package must not start until both are gone. - Inspect afterwards:
argo get -n argo @latestshows the DAG with per-node durations. - Compare shapes: Tekton expresses ordering with
runAfteron Tasks; Argo Workflows usesdependenciesin adag. Same idea — a graph of containers as a Kubernetes object — different dialect. The exam wants you to recognise both.
argo get -n argo @latest reports Succeeded and lists all four task nodes — checkout, unit-tests, lint, package — under the DAG node, and you saw the two middle pods Running at the same moment in kubectl -n argo get pods -w.Labs 1–7 got a change from “someone pushed” to “an image exists,” entirely inside the cluster, with no build server to babysit. Everything from here is the other half of the domain: getting that image in front of users without hurting anyone. Remember the sentence the rest of this page is built on — deploy is not release. Deploy means the new version is running; release means traffic reaches it. Labs 8–12 pry those two apart four different ways.
# lab8.yaml — a Deployment, converted
apiVersion: argoproj.io/v1alpha1
kind: Rollout # replaces kind: Deployment, same pod template
metadata:
name: demo
namespace: demo
spec:
replicas: 5
selector:
matchLabels: { app: demo }
template:
metadata:
labels: { app: demo }
spec:
containers:
- name: demo
image: argoproj/rollouts-demo:blue # each tag serves a different colour
ports: [{ containerPort: 8080 }]
resources:
requests: { cpu: 5m, memory: 32Mi }
strategy:
canary:
steps:
- setWeight: 20 # ~20% of replicas are the new version
- pause: { duration: 30s } # soak
- setWeight: 50
- pause: {} # {} with no duration = pause FOREVER (manual gate)
- setWeight: 100- Install the controller and the plugin:
kubectl create ns argo-rollouts, thenkubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml, then thekubectl argo rolloutsplugin (kubectl krew install argo-rollouts, or the release binary). Wait forkubectl -n argo-rollouts rollout status deploy/argo-rollouts— the plugin talks to the API server, but nothing progresses without the controller. kubectl create ns demoand applylab8.yaml. The first rollout of a brand-new Rollout skips the steps entirely — there’s no stable version to protect yet.- Open the live view in its own terminal:
kubectl argo rollouts get rollout demo -n demo --watch. - Ship a new version:
kubectl argo rollouts set image demo demo=argoproj/rollouts-demo:yellow -n demo. - Watch it climb to 20%, soak 30s, climb to 50% — then stop. That indefinite
pause: {}is a manual gate; the status readsPausedand it will wait all day. - Be the human in the loop:
kubectl argo rollouts promote demo -n demo. It finishes to 100% and the old ReplicaSet scales to zero. - One honest caveat to say out loud: with no traffic router configured, “20%” is approximated by replica counts, not a true per-request split. Real weighting needs a mesh or Gateway API
HTTPRoute.
kubectl argo rollouts get rollout demo -n demo showed status Paused at the 50% step, and after promote it reads Healthy with every pod on the yellow image (kubectl -n demo get pods -o jsonpath='{.items[*].spec.containers[*].image}').# lab9.yaml — let the canary judge itself against Prometheus
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: no-restarts
namespace: demo
spec:
args:
- name: ns
metrics:
- name: canary-restarts
interval: 30s
count: 6 # six measurements, then the step passes
failureLimit: 1 # a SECOND bad measurement aborts the rollout
# (kube-state-metrics scrapes every 30s — give
# the restart counter time to actually move)
successCondition: result[0] < 1
provider:
prometheus:
address: http://kube-prom-stack-kube-prome-prometheus.monitoring:9090
query: |
sum(increase(kube_pod_container_status_restarts_total{namespace="{{args.ns}}"}[2m])) or vector(0)
---
# patch the Lab 8 strategy so step 2 is an analysis instead of a blind soak
# strategy:
# canary:
# steps:
# - setWeight: 20
# - analysis:
# templates:
# - templateName: no-restarts
# args:
# - { name: ns, value: demo }
# - setWeight: 50
# - pause: { duration: 30s }
# - setWeight: 100- Install Prometheus:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts,helm repo update, thenhelm install kube-prom-stack prometheus-community/kube-prometheus-stack -n monitoring --create-namespace. Wait forkubectl -n monitoring get podsto goRunning, then confirm the Service name yourself:kubectl -n monitoring get svc -l app.kubernetes.io/name=prometheus. The chart derives it from your release name and truncates it — that exact string must match theaddressabove. - Sanity-check the query before you trust it with a rollout:
kubectl -n monitoring port-forward svc/kube-prom-stack-kube-prome-prometheus 9090:9090, openlocalhost:9090, and run the PromQL in the UI. A query that returns nothing will fail your analysis for the wrong reason — the single most common wiring mistake here. - Apply the
AnalysisTemplateand patch the Rollout’s steps as shown in the comment. - Ship a version that works:
kubectl argo rollouts set image demo demo=argoproj/rollouts-demo:green -n demo. Watchkubectl argo rollouts get rollout demo -n demo --watch— anAnalysisRunappears, takes its measurements, passes, and the rollout continues. - Now ship a version that crashes:
kubectl argo rollouts set image demo demo=busybox:1.36 -n demo. Busybox’s default command exits immediately, so the canary pods land inCrashLoopBackOff, restarts climb, the query breachesfailureLimit, and the rollout aborts on its own — traffic snaps back to the previous version, canary pods scale to zero, nobody paged. (Resist reaching for a tag that doesn’t exist:ImagePullBackOffpods never start, so they never restart, and this particular query would sail right past them. Pick the failure mode your metric can actually see.) - Read the verdict:
kubectl -n demo get analysisrun, thenkubectl -n demo describe $(kubectl -n demo get analysisrun -o name --sort-by=.metadata.creationTimestamp | tail -1)— it records the measured value that failed. - In production you’d query request success rate scoped to the canary Service, not restarts. Note the difference and why a query that lumps canary and stable together would happily promote a broken release.
AnalysisRun in Successful, and the bad image leaves an AnalysisRun in Failed with the Rollout back on the previous version — verified by kubectl argo rollouts get rollout demo -n demo reporting Degraded/aborted and no canary pods left.# lab10.yaml — blue/green: two Services, one manual flip
apiVersion: v1
kind: Service
metadata: { name: demo-active, namespace: demo }
spec:
selector: { app: demo-bg }
ports: [{ port: 80, targetPort: 8080 }]
---
apiVersion: v1
kind: Service
metadata: { name: demo-preview, namespace: demo }
spec:
selector: { app: demo-bg }
ports: [{ port: 80, targetPort: 8080 }]
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: demo-bg, namespace: demo }
spec:
replicas: 3
selector:
matchLabels: { app: demo-bg }
template:
metadata:
labels: { app: demo-bg }
spec:
containers:
- name: demo
image: argoproj/rollouts-demo:blue
ports: [{ containerPort: 8080 }]
resources:
requests: { cpu: 5m, memory: 32Mi }
strategy:
blueGreen:
activeService: demo-active # what users hit
previewService: demo-preview # what only you hit
autoPromotionEnabled: false # the flip is a human decision
scaleDownDelaySeconds: 30 # keep the old set warm for a fast undo- Apply
lab10.yaml. Argo Rollouts rewrites each Service’s selector with a hidden pod-template-hash — that’s how one label can address two different versions. - Ship a new version:
kubectl argo rollouts set image demo-bg demo=argoproj/rollouts-demo:purple -n demo. - Now prove deploy ≠ release. Two port-forwards, two browser tabs:
kubectl -n demo port-forward svc/demo-active 8081:80andkubectl -n demo port-forward svc/demo-preview 8082:80. The preview is purple; the active is still blue. The new version is deployed and serving nobody. - Verify from the API rather than your eyes:
kubectl -n demo get svc demo-active demo-preview -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.selector}{"\n"}{end}'— different hashes. - Flip it:
kubectl argo rollouts promote demo-bg -n demo. The active Service repoints in one instant — no ramp, no partial state. - Compare the two shapes in your own words: blue/green costs double the replicas and flips everyone at once; canary costs almost nothing extra and exposes a sliver at a time. Blue/green is the one you pick when a half-and-half state would corrupt data.
promote, demo-active returns the new colour.# lab11.yaml — the same canary, expressed as Flagger's Canary CR
apiVersion: flagger.app/v1beta1
kind: Canary
metadata: { name: podinfo, namespace: test }
spec:
targetRef: # your ORDINARY Deployment, untouched
apiVersion: apps/v1
kind: Deployment
name: podinfo
progressDeadlineSeconds: 60
service:
port: 80
targetPort: 9898
analysis:
interval: 15s
threshold: 5 # 5 failed checks → roll back
maxWeight: 50
stepWeight: 10 # 10, 20, 30, 40, 50, then promote
metrics:
- name: request-success-rate # built in when a mesh reports it
thresholdRange: { min: 99 }
interval: 1m
webhooks:
- name: load-test # generate traffic, or there's nothing to measure
url: http://flagger-loadtester.test/
timeout: 5s
metadata:
cmd: "hey -z 1m -q 10 -c 2 http://podinfo-canary.test/"- Flagger shifts traffic through a mesh or ingress, so install one. Shortest local path: Linkerd —
linkerd install --crds | kubectl apply -f -,linkerd install | kubectl apply -f -,linkerd viz install | kubectl apply -f -, thenlinkerd check. - Install Flagger’s CRDs and controller, pointed at that mesh:
helm repo add flagger https://flagger.app,helm repo update, thenhelm upgrade -i flagger flagger/flagger -n flagger-system --create-namespace --set meshProvider=linkerd --set metricsServer=http://prometheus.linkerd-viz:9090. TheCanaryCRD has to exist beforelab11.yamlwill apply at all. - Make a meshed namespace, put an ordinary Deployment in it, and add the load generator:
kubectl create ns test,kubectl annotate ns test linkerd.io/inject=enabled,kubectl -n test create deployment podinfo --image=stefanprodan/podinfo:6.0.0, thenhelm upgrade -i flagger-loadtester flagger/loadtester -n test. Flagger’s ownpodinfoexample works too and comes with an HPA. - Apply
lab11.yaml. Note the container name you actually created —kubectl -n test get deploy podinfo -o jsonpath='{.spec.template.spec.containers[*].name}'printspodinfohere, while Flagger’s upstream example names itpodinfod. Step 5 needs whichever one your Deployment has. - Watch Flagger take over:
kubectl -n test get svc,deploy. It createdpodinfo-primary,podinfo-canaryand the apex Service — from your one Deployment. Nothing of yours changed shape. - Trigger a canary:
kubectl -n test set image deployment/podinfo podinfo=stefanprodan/podinfo:6.0.1. Follow it withkubectl -n test get canary podinfo -wandkubectl -n flagger-system logs deploy/flagger -f. - Break it: start another canary, and while it is progressing point traffic at podinfo’s deliberate-failure endpoint —
kubectl -n test exec deploy/flagger-loadtester -- hey -z 2m -c 5 -q 10 http://podinfo-canary.test/status/500. Success rate drops below 99, thethresholdof failed checks is hit, and Flagger rolls back and marks the CanaryFailed. - Write down the one structural difference: Argo Rollouts replaces your Deployment with a
Rollout; Flagger wraps the Deployment you already have. Same outcome, opposite intrusion.
kubectl -n test get canary shows one run reaching Succeeded and a deliberately broken run reaching Failed, with kubectl -n test describe canary podinfo listing the weight steps and the halt.# lab12.yaml — flagd: the release switch lives outside the image
apiVersion: v1
kind: ConfigMap
metadata: { name: flags, namespace: demo }
data:
flags.json: |
{
"flags": {
"new-checkout": {
"state": "ENABLED",
"variants": { "on": true, "off": false },
"defaultVariant": "off"
}
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: flagd, namespace: demo }
spec:
replicas: 1
selector: { matchLabels: { app: flagd } }
template:
metadata: { labels: { app: flagd } }
spec:
containers:
- name: flagd
image: ghcr.io/open-feature/flagd:latest
args: ["start", "--uri", "file:/etc/flagd/flags.json"] # watches the file
ports: [{ containerPort: 8013 }]
volumeMounts:
- { name: flags, mountPath: /etc/flagd }
volumes:
- name: flags
configMap: { name: flags }- Apply
lab12.yamland port-forward the evaluation API:kubectl -n demo port-forward deploy/flagd 8013:8013. - Ask it what a user should see:
curl -s -X POST localhost:8013/flagd.evaluation.v1.Service/ResolveBoolean -H 'Content-Type: application/json' -d '{"flagKey":"new-checkout","context":{}}'. It answers"variant":"off"— the new behaviour is deployed but dark. Read thevariant, not thevalue: this is a Connect/protobuf JSON API, and protobuf JSON omits fields that equal their zero value, so"value":falsesimply won’t appear in the body. - Note the pod’s identity before you change anything:
kubectl -n demo get pod -l app=flagd -o wide. Write down the name, age and restart count. - Release, without shipping:
kubectl -n demo patch configmap flags --type=merge -p '{"data":{"flags.json":"{\"flags\":{\"new-checkout\":{\"state\":\"ENABLED\",\"variants\":{\"on\":true,\"off\":false},\"defaultVariant\":\"on\"}}}"}}'. Mounted ConfigMaps refresh on the kubelet’s sync cycle — give it up to a minute. - Re-run the same
curl. It now answers"variant":"on"— and this time"value":trueshows up too, becausetrueis not the zero value. - Re-check the pod. Same name, same age, zero restarts. Nothing was deployed; a behaviour was released. Flip
defaultVariantback tooffand you’ve just performed an instant rollback with no rollout at all. - Say where each tool belongs: a canary chooses which users hit the new pods; a flag chooses which users see the new behaviour. Mature teams run both, which is why the flag survives long after the rollout finishes — and why stale flags become their own kind of debt.
curl returns "variant":"off" and then "variant":"on","value":true, while kubectl -n demo get pod -l app=flagd shows the same pod name, the same AGE and 0 restarts throughout. If the flip never lands, kubectl -n demo logs deploy/flagd will say whether flagd saw the file change — the kubelet can take up to a sync period to swap a mounted ConfigMap.“Every one of these labs is invisible to me when it works — and that’s the review criterion. I push code. Something builds it, something scans it, something ships a sliver of it, and something watches the numbers. The day I hear about any of it is the day one of you built it wrong.”
What you’ll have built
☺ Like you’re 10: A belt that turns your code into a packaged toy by itself, and a shopkeeper who hands the new toy out slowly and takes it back the second anyone cries.
Twelve labs in, your throwaway cluster holds a complete, if tiny, delivery path — and, more usefully, your hands know the shapes. On the pipeline side: Tasks and Pipelines as ordinary Kubernetes objects, a run that is just another resource you can describe, a workspace carrying files between steps and results carrying values, a daemonless in-cluster image build, a push credential scoped to a ServiceAccount rather than sprayed across the cluster, a webhook that starts work, and the same graph expressed a second way as an Argo Workflows DAG. On the release side: a canary that ramps in explicit steps and holds at a manual gate, an analysis step that turns “is this healthy?” into a Prometheus query the platform runs for you, an automatic abort you watched with your own eyes, a blue/green preview that made deploy is not release undeniable, the same canary in the other major tool, and a feature flag that released a behaviour with no deployment at all.
That covers the delivery half of the exam’s 25% GitOps & Continuous Delivery domain, hands-on; the syllabus-shaped view of the same ground is the CNPA continuous-delivery module. Fold it into the wider platform through the full lab track, pair it with the GitOps labs so the pipeline you just built hands off to a reconciler instead of to kubectl, drill the same material under time pressure in the practice tasks, and rehearse the failures in delivery triage. When something in a lab refuses to work — and it will — resist the urge to skip it. The troubleshooting playbook exists precisely because that stuck moment is the exam.
Comfortable? Close the loop. One: make the Lab 4 pipeline end by committing the new tag to a config repo and let Argo CD deploy it — the pipeline never touches the cluster, exactly as the concept lesson insists. Two: add a Trivy scan Task that fails the run on a critical CVE, and a cosign signing step, then make admission reject anything unsigned. Three: install the NGINX ingress controller or Gateway API and give Lab 8 real weighted routing, so 20% means 20% of requests instead of 20% of replicas. Four: rewrite the Lab 9 analysis to query request success rate scoped to the canary Service — the version that would actually protect users. Five: put all of it under GitOps so the pipelines themselves reconcile from Git.
Foxy: My canary went to 20% and then just… sat there. Broken controller?
Pip: Read your own steps, Foxy. pause: {} with nothing in it pauses forever — that’s a manual gate you asked for. pause: { duration: 30s } is the soak.
Benny: And when the build “can’t find the source,” it’s the workspace, not the code. Files ride the workspace; strings ride results. Mix those up and everything looks haunted.
Gizmo: Just mount the Docker socket into the build pod. One line, works instantly, everyone does it. 🤑
Timmy: And hands that pipeline the entire node, Gizmo. Kaniko, Buildpacks, rootless BuildKit — build like every other unprivileged pod.
Dot: I flipped a flag this morning and released to 5% of users. No deploy, no ticket, no you. Best day I’ve had.
1. In Tekton, which object is a definition and which is an execution — and what do Workspaces carry that Results don’t? 2. Why does building with Kaniko beat mounting /var/run/docker.sock? 3. What is the difference between pause: {} and pause: { duration: 5m }? 4. Your canary’s AnalysisTemplate queries overall error rate across the whole Service. Why will that promote a broken release? 5. Structurally, how do Argo Rollouts and Flagger differ in what they do to your Deployment? 6. Lab 12 changed user-visible behaviour with zero deploys — name the principle.
Check your answers
TaskandPipelineare definitions;TaskRunandPipelineRunare single executions of them. Workspaces carry files (a shared volume — the source tree); Results carry small values (a tag, a digest, a commit SHA) from one Task into another’s params.- Kaniko builds each Dockerfile instruction in userspace inside an ordinary, unprivileged pod. Mounting the Docker socket (or running privileged) gives the build pod control of the node and every workload on it — a node takeover waiting to happen, and something any admission policy will flag.
pause: { duration: 5m }soaks for five minutes and then continues automatically.pause: {}pauses indefinitely — a manual gate that waits until someone runskubectl argo rollouts promote.- Because a 10–20% canary’s errors are drowned by 80–90% healthy traffic from the stable version: the blended number stays above your threshold and the analysis passes. Scope the query to the canary Service (pass it in as an arg), or you’re measuring the version you already trust.
- Argo Rollouts replaces the
Deploymentwith aRolloutresource carrying the same pod template plus astrategy. Flagger leaves yourDeploymentalone and wraps it with aCanaryresource, generating primary/canary Services and a primary copy of the workload. Same outcome, opposite level of intrusion. - Deploy is not release. The code shipped earlier and sat dark; flipping the flag released the behaviour — with no new pods, no restarts, and an instant rollback available by flipping it back.