Hands-On Capstone · Part 2 of 6 · pairs with CI/CD & Progressive Delivery · D2 · 25%

Capstone Part 2: Pipeline & Progressive Delivery

In Part 1 you made Git the control panel: a kind cluster named platform-dev, Argo CD reconciling an App-of-Apps, and a placeholder ledger workload that proved reconciliation beats drift and Git deletion prunes. That was plumbing with no water running through it. This part builds the water: a Tekton pipeline that builds the real ledger image with Kaniko — no Docker daemon, no mounted socket — and ends the only way a GitOps pipeline is allowed to end: by committing a new image tag to Git. Then you convert ledger from a plain Deployment into an Argo Rollouts Rollout with a canary strategy, wire in an automated analysis step, and prove the two endings out loud: a good release promotes itself from 10% to 50% to 100%, and a bad one aborts itself back to zero. This is the hands-on twin of the CI/CD & Progressive Delivery lesson — read that first if any term here is unfamiliar.

⚖ CNPA vs CNPE — That D2 · 25% weighting is CNPE-specific — CNPA has no hands-on lab and no matching per-domain percentage. But the underlying ideas here — Kubernetes-native pipelines, canary releases, automated rollback — are still fair game for CNPA's closed-book recall, just tested as concepts rather than something you build.

☺ Explain it like I’m 10

Part 1 hired the tireless robot and gave it a poster. Today you build the toy factory that draws new posters: a conveyor belt (Tekton) bolts the parts together, tests them, and — instead of walking the toy to the shelf itself — pins a new poster to the wall saying “the toy is now version 1.1.0.” The robot from Part 1 notices the poster changed and fetches the toy itself. Then you hire a careful traffic cop (Argo Rollouts) who never gives a brand-new toy to every kid at once — she gives it to one in ten, watches for tears, and only then gives it to everyone. If kids start crying, she snatches it back before you even notice.

🦫🐦Your hosts for this part: Benny the Beaver & Pip the Hummingbird — Benny builds the conveyor belt and teaches it to stop at the Git commit instead of the cluster; Pip is the traffic cop who ramps the canary and yanks it back the instant her smoke test turns red.
⚠ Read this before Milestone 1

Same ground rules as every lab on this site: this all runs on the local, throwaway platform-dev cluster from Part 1, nothing here touches production or costs money, and tool versions and flags drift — treat every command below as the shape of the answer and check each project’s current docs (the same docs you may open in the real CNPE exam: kubernetes.io/docs, kubernetes.io/blog) if something fails. One extra wrinkle this part adds: a local, in-cluster image registry has no official one-line installer, so the registry-wiring commands below are adapted from kind’s own documented local-registry recipe — if a push fails with a DNS error, the troubleshooting note in Part A tells you the fallback.

Arrival state → departure state

☺ Like you’re 10: Know exactly what your cluster looks like when you sit down, and exactly what it must look like before you stand up — no surprises for the next part.

What you inherit from Part 1: a running kind cluster named platform-dev; Argo CD installed and healthy in the platform namespace; a root Application in platform pointed at apps/ in the platform-capstone Git repo (App-of-Apps, prune: true, selfHeal: true); and a child Application named ledger pointed at ledger/ in that same repo, currently rendering a plain placeholder Deployment into the ledger namespace. You already proved that a manual kubectl scale gets reverted and that deleting a manifest from Git prunes the live object.

What you leave for Part 3: a real ledger-src application repo with a Dockerfile; a Tekton Pipeline in the platform namespace that clones it, builds the image with Kaniko, pushes it to a local registry as registry.local:5001/ledger:<tag>, and commits the new tag into platform-capstone; the ledger/ path in platform-capstone now holding a Rollout (not a Deployment) with a three-Service canary and a job-based AnalysisTemplate; and a demonstrated, hands-off loop where a good tag promotes to 100% and a bad tag aborts to 0% automatically. Part 3 picks up from here to give the platform a brand-new API noun for ledger’s data layer — it only needs the ledger namespace and the Argo CD Application to still exist, so don’t delete either when you tear down today’s scratch resources.

Before you build: prerequisites & repo layout

☺ Like you’re 10: Two new drawers today — one for the toy’s actual blueprint (source code), one more folder in the poster drawer you already have (the Rollout instead of the Deployment).

You need everything from Part 1 still running, plus a Tekton CLI (tkn, optional but handy) and the kubectl argo rollouts plugin (installed in Part B below). You also need a second Git repo — the app repo, separate from the platform-capstone config repo, exactly as GitOps Workflows teaches. Create an empty repo named ledger-src; Milestone 3 fills it in. In your existing platform-capstone fork, ledger/ currently holds only a placeholder deployment.yaml — Milestone 7 replaces its contents with the files shown below.

ledger-src/                       # NEW — the APP repo: code + Dockerfile, no manifests
├── app.py
├── requirements.txt
└── Dockerfile

platform-capstone/                  # your Part-1 fork — the CONFIG repo Argo CD watches
├── apps/
│   └── ledger.yaml                # unchanged — still points Argo CD at ledger/
└── ledger/                        # Milestone 7 replaces this folder's contents
    ├── service.yaml                # ledger + ledger-stable + ledger-canary Services
    ├── rollout.yaml                # replaces deployment.yaml — the canary strategy
    └── analysistemplate.yaml       # the job-based smoke test
0 / 12 milestones complete

Part A · Give ledger a registry and a Kubernetes-native pipeline

☺ Like you’re 10: Build the conveyor belt itself — where finished toys get shelved (a registry), and the machine that assembles them (Tekton) without ever needing the master key to the whole building (no Docker daemon).

Milestone 1Wire a local image registry for kind — 🦫 Benny
  1. Run a plain registry:2 container attached to the same Docker network your platform-dev nodes are on (kind creates a network literally called kind), and map it to a fixed local port.
  2. Add a host-file entry so your laptop can also docker push/pull under the same name you’ll use inside the cluster.
  3. Patch every kind node’s containerd so that a pull of registry.local:5001/… is transparently redirected to the registry container’s real address on the kind network — this is the exact mechanism kind’s own local-registry recipe uses, renamed here to match the course’s registry.local placeholder.
  4. Label the well-known local-registry-hosting ConfigMap so any registry-aware tooling can discover it.
Done when: curl http://registry.local:5001/v2/_catalog from your laptop returns {"repositories":[]}, and docker exec platform-dev-control-plane crictl pull registry.local:5001/library/busybox:1.36 (or an equivalent pull from inside a node) succeeds.
Concept: CI/CD & Progressive Delivery · Tekton
# --- run the registry on the same docker network as the kind nodes ---
docker run -d --restart=always -p 127.0.0.1:5001:5000 --network kind --name registry.local registry:2
echo "127.0.0.1 registry.local" | sudo tee -a /etc/hosts   # your laptop can now push/pull too

# --- point every kind node's containerd at it (adapted from kind's local-registry recipe) ---
for node in $(kind get nodes --name platform-dev); do
  docker exec "${node}" mkdir -p "/etc/containerd/certs.d/registry.local:5001"
  cat <<EOF | docker exec -i "${node}" cp /dev/stdin "/etc/containerd/certs.d/registry.local:5001/hosts.toml"
[host."http://registry.local:5000"]
EOF
done

# --- tell tooling the registry exists (kind's documented convention) ---
kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: local-registry-hosting
  namespace: kube-public
data:
  localRegistryHosting.v1: |
    host: "registry.local:5001"
    help: "https://kind.sigs.k8s.io/docs/user/local-registry/"
EOF
⚠ If a pod can’t resolve registry.local

The patch above fixes containerd’s own image pulls (what kubelet does for you). Kaniko pushing from inside a pod is a different code path — it just needs the pod to resolve the name over the network. Most kind setups get this for free because CoreDNS forwards non-cluster names to the node’s own resolver, and the node — itself a container on the kind network — resolves registry.local the same way any other container on that network would. If it still fails, add a hostAliases entry to the Kaniko step’s pod (docker inspect registry.local --format '{{"{{"}}.NetworkSettings.Networks.kind.IPAddress{{"}}"}}' for the IP) as a fallback — don’t spend more than five minutes on this before reaching for that escape hatch.

Milestone 2Install Tekton Pipelines — 🦫 Benny
  1. Apply the upstream release manifest. It installs its own controllers into their own tekton-pipelines namespace — that part isn’t user-configurable without a custom overlay, which is fine: your own Task/Pipeline/PipelineRun objects still live in your platform namespace right alongside Argo CD, same as the rest of this capstone’s add-ons.
  2. Optionally install the tkn CLI for a friendlier view of runs and logs.
Done when: kubectl -n tekton-pipelines get pods shows the controller and webhook Running, and tkn version (if installed) prints a client version.
Concept: Kubernetes-native pipelines · Tekton
kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
kubectl -n tekton-pipelines rollout status deploy/tekton-pipelines-controller
kubectl -n tekton-pipelines rollout status deploy/tekton-pipelines-webhook

brew install tektoncd-cli     # optional but recommended
tkn version
Milestone 3Write the real ledger service — 🦫 Benny & 🦆 Dot
  1. In your new ledger-src repo, add the three files below: a tiny Flask app with a health check and a “buggy mode” switch, its one dependency, and a Dockerfile.
  2. Commit and push to main. This replaces Part 1’s placeholder image with the service you’ll actually build, canary, and (on purpose) break.
Done when: ledger-src on GitHub shows all three files on main, and docker build -t ledger:local . succeeds on your laptop as a sanity check before you ever involve Kaniko.
Concept: Building images without a Docker daemon
# app.py — a deliberately tiny ledger service
import os
import random
from flask import Flask, jsonify

app = Flask(__name__)
BUGGY = os.environ.get("LEDGER_BUGGY", "false").lower() == "true"
VERSION = os.environ.get("LEDGER_VERSION", "dev")

@app.get("/healthz")
def healthz():
    return jsonify(status="ok"), 200

@app.get("/")
def ledger():
    if BUGGY and random.random() < 0.7:
        # the "bad" build: about 70% of requests fail on purpose
        return jsonify(error="ledger entry corrupted", version=VERSION), 500
    return jsonify(service="ledger", version=VERSION, balance=4200), 200

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)
# requirements.txt
flask==3.0.3
# Dockerfile
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
ENV LEDGER_VERSION=dev
EXPOSE 8080
CMD ["python", "app.py"]
Milestone 4Author the pipeline’s three Tasks — 🦫 Benny
  1. fetch-source clones a Git repo into a shared workspace — plain git, no Tekton catalog resolver to configure.
  2. build-and-push runs Kaniko against that checkout — an ordinary, unprivileged pod, no Docker socket, exactly the “daemonless build” the CI/CD lesson insists on.
  3. bump-tag clones platform-capstone, edits the image line in ledger/rollout.yaml, and pushes the commit — this is the pipeline’s only write to a place Argo CD can see. It never touches the cluster directly.
  4. Create the git-credentials Secret the last Task needs to push, and the ServiceAccount that carries it.
Done when: kubectl -n platform get task fetch-source build-and-push bump-tag lists all three, and kubectl -n platform get secret git-credentials exists with a token scoped to repo on your fork.
Concept: Tekton — pipelines as Kubernetes resources · Secrets Management
apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: fetch-source
  namespace: platform
spec:
  params:
    - name: repo-url
    - name: revision
      default: main
  workspaces:
    - name: source
  steps:
    - name: clone
      image: alpine/git:latest
      script: |
        #!/bin/sh
        set -eu
        git clone --branch "$(params.revision)" --depth 1 "$(params.repo-url)" "$(workspaces.source.path)"
apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: build-and-push
  namespace: platform
spec:
  params:
    - name: image                     # e.g. registry.local:5001/ledger
    - name: tag
    - name: dockerfile
      default: Dockerfile
    - name: build-arg-buggy            # "true" bakes in the broken build on purpose
      default: "false"
  workspaces:
    - name: source
  steps:
    - name: kaniko-build
      image: gcr.io/kaniko-project/executor:latest
      args:
        - --context=$(workspaces.source.path)
        - --dockerfile=$(workspaces.source.path)/$(params.dockerfile)
        - --destination=$(params.image):$(params.tag)
        - --build-arg=LEDGER_BUGGY=$(params.build-arg-buggy)
        - --insecure                  # our local registry has no TLS
        - --insecure-pull
        - --skip-tls-verify
apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: bump-tag
  namespace: platform
spec:
  params:
    - name: config-repo-url
    - name: image
    - name: tag
    - name: rollout-path
      default: ledger/rollout.yaml
  workspaces:
    - name: config
  steps:
    - name: patch-and-push
      image: alpine/git:latest
      script: |
        #!/bin/sh
        set -eu
        git clone "$(params.config-repo-url)" "$(workspaces.config.path)"
        cd "$(workspaces.config.path)"
        sed -i "s#image: .*#image: $(params.image):$(params.tag)#" "$(params.rollout-path)"
        git config user.email "tekton@platform-dev.local"
        git config user.name "ledger-ci"
        git add "$(params.rollout-path)"
        git commit -m "chore(ledger): bump image to $(params.tag)" || echo "no changes to commit"
        git push origin main
apiVersion: v1
kind: Secret
metadata:
  name: git-credentials
  namespace: platform
  annotations:
    tekton.dev/git-0: https://github.com     # tells Tekton's git-init to use this cred for this host
type: kubernetes.io/basic-auth
stringData:
  username: YOU
  password: ghp_your_token_with_repo_scope   # a token scoped only to repo — never a broad PAT
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ledger-ci
  namespace: platform
secrets:
  - name: git-credentials
Milestone 5Chain them into a Pipeline and run it by hand — 🦫 Benny & 🤖 Recon
  1. Apply the ledger-ci Pipeline below — fetch runs first, then build, then bump, ordered with runAfter, matching the CI/CD lesson’s Task/Pipeline/Workspace shapes exactly.
  2. Fire a PipelineRun for version 1.0.0 of the good build (build-arg-buggy: "false", the default).
  3. Watch it with tkn pipelinerun logs --last -f (or kubectl -n platform get pipelinerun -w), then confirm the two write-side-effects independently: the image landed in the registry, and a commit landed in platform-capstone.
Done when: tkn pipelinerun describe --last (or kubectl -n platform get pipelinerun) shows Succeeded; curl http://registry.local:5001/v2/ledger/tags/list lists "1.0.0"; and git log --oneline -1 in your platform-capstone clone shows a commit authored by ledger-ci bumping the image to 1.0.0 — and, critically, the pipeline never ran a single kubectl command against the cluster.
Concept: The pipeline’s real job: end by committing to Git · GitOps Workflows
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: ledger-ci
  namespace: platform
spec:
  params:
    - name: repo-url
    - name: revision
      default: main
    - name: config-repo-url
    - name: image
    - name: tag
    - name: build-arg-buggy
      default: "false"
  workspaces:
    - name: source
    - name: config
  tasks:
    - name: fetch
      taskRef: { name: fetch-source }
      params:
        - { name: repo-url, value: $(params.repo-url) }
        - { name: revision, value: $(params.revision) }
      workspaces:
        - { name: source, workspace: source }
    - name: build
      runAfter: [fetch]
      taskRef: { name: build-and-push }
      params:
        - { name: image, value: $(params.image) }
        - { name: tag, value: $(params.tag) }
        - { name: build-arg-buggy, value: $(params.build-arg-buggy) }
      workspaces:
        - { name: source, workspace: source }
    - name: bump
      runAfter: [build]
      taskRef: { name: bump-tag }
      params:
        - { name: config-repo-url, value: $(params.config-repo-url) }
        - { name: image, value: $(params.image) }
        - { name: tag, value: $(params.tag) }
      workspaces:
        - { name: config, workspace: config }
# pipelinerun-good.yaml — the good build, version 1.0.0
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: ledger-ci-run-
  namespace: platform
spec:
  pipelineRef: { name: ledger-ci }
  taskRunTemplate:
    serviceAccountName: ledger-ci      # Tekton v1: lives under taskRunTemplate now
  params:
    - { name: repo-url, value: "https://github.com/YOU/ledger-src.git" }
    - { name: config-repo-url, value: "https://github.com/YOU/platform-capstone.git" }
    - { name: image, value: "registry.local:5001/ledger" }
    - { name: tag, value: "1.0.0" }
    - { name: build-arg-buggy, value: "false" }
  workspaces:
    - name: source
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources: { requests: { storage: 1Gi } }
    - name: config
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources: { requests: { storage: 256Mi } }
kubectl create -f pipelinerun-good.yaml
tkn pipelinerun logs --last -f -n platform
curl -s http://registry.local:5001/v2/ledger/tags/list
git -C platform-capstone pull && git -C platform-capstone log --oneline -1

Part B · From Deployment to Rollout: canary the ledger service

☺ Like you’re 10: Hire the traffic cop, teach her which door is “a few kids” and which door is “everyone,” and give her a stopwatch test she can run herself before waving more kids through.

Milestone 6Install Argo Rollouts & its CLI plugin — 🦫 Benny
  1. Install the controller. Like Tekton, its upstream manifest targets its own argo-rollouts namespace — the Rollout, Service, and AnalysisTemplate objects it manages for ledger still live in your ledger namespace, right where the app itself lives.
  2. Install the kubectl argo rollouts plugin — you’ll live in it for the rest of this part.
Done when: kubectl -n argo-rollouts rollout status deploy/argo-rollouts reports success, and kubectl argo rollouts version prints a version.
Concept: Argo Rollouts — a Deployment with a strategy · Argo Rollouts
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
kubectl -n argo-rollouts rollout status deploy/argo-rollouts

brew install argoproj/tap/kubectl-argo-rollouts
kubectl argo rollouts version
Milestone 7Replace deployment.yaml with a Rollout + three Services — 🤖 Recon
  1. In platform-capstone, delete ledger/deployment.yaml and add ledger/service.yaml and ledger/rollout.yaml below. There’s no trafficRouting block — with no mesh installed yet (that arrives with Linkerd in Part 6), this is a basic canary: the main ledger Service load-balances across whatever pods exist, so the split is only as precise as the ratio of canary to stable replicas — exactly the “coarse, connection-sticky approximation” the CI/CD lesson warns about. canaryService/stableService still work without a mesh, though — Argo Rollouts patches their selectors with an internal rollouts-pod-template-hash label so each one always points at just the canary or just the stable pods, which is what Milestone 8’s analysis step relies on.
  2. Commit and push. Because Argo CD’s ledger Application already watches this path, it will apply whatever kind of object is there — the first sync after this commit replaces the Deployment with a Rollout entirely.
  3. Watch the swap: kubectl -n platform get application ledger flips through OutOfSync to Synced, and kubectl -n ledger get rollout,deploy,svc shows the Deployment gone and a Rollout in its place.
Done when: kubectl -n ledger get rollout ledger exists, kubectl -n ledger get deploy returns nothing named ledger, and argocd app get ledger reports Healthy — Argo CD ships a built-in health check for the Rollout kind, so that status only turns green once the canary has actually promoted.
Concept: Reconciliation & drift · Traffic shifting — how Pip routes the canary
# ledger/service.yaml — replaces the plain Service from Part 1
apiVersion: v1
kind: Service
metadata:
  name: ledger
  namespace: ledger
spec:
  selector: { app: ledger }
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: ledger-stable
  namespace: ledger
spec:
  selector: { app: ledger }              # Argo Rollouts adds the revision-scoping label at runtime
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: ledger-canary
  namespace: ledger
spec:
  selector: { app: ledger }              # same base selector; Rollouts scopes it to the canary pods
  ports:
    - port: 80
      targetPort: 8080
# ledger/rollout.yaml — replaces deployment.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: ledger
  namespace: ledger
spec:
  replicas: 5
  revisionHistoryLimit: 3
  selector:
    matchLabels: { app: ledger }
  template:
    metadata:
      labels: { app: ledger }
    spec:
      containers:
        - name: ledger
          image: registry.local:5001/ledger:1.0.0   # bump-tag rewrites this line
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 3
            periodSeconds: 5
          resources:
            requests: { cpu: 25m, memory: 32Mi }
  strategy:
    canary:
      canaryService: ledger-canary
      stableService: ledger-stable
      steps:
        - setWeight: 10
        - pause: { duration: 60s }
        - analysis:
            templates:
              - templateName: ledger-smoke-test
            args:
              - { name: service-name, value: ledger-canary }
        - setWeight: 50
        - pause: { duration: 60s }
        - analysis:
            templates:
              - templateName: ledger-smoke-test
            args:
              - { name: service-name, value: ledger-canary }
        - setWeight: 100
Milestone 8Wire in a job-based analysis — 🐦 Pip
  1. No Prometheus stack exists yet — that’s Part 5’s job. So this step’s judgment doesn’t query a metrics provider; it runs a Kubernetes Job that hammers the canary’s own Service with real HTTP requests and counts the failures — Argo Rollouts’ built-in job analysis provider treats a zero exit code as pass and nonzero as fail.
  2. Add the AnalysisTemplate below to ledger/analysistemplate.yaml, commit, and let Argo CD sync it in before you trigger a Rollout that references it.
Done when: kubectl -n ledger get analysistemplate ledger-smoke-test exists, and running kubectl -n ledger create job manual-smoke-test --image=curlimages/curl -- sh -c "curl -s -o /dev/null -w '%{http_code}\n' http://ledger-stable.ledger.svc.cluster.local/" by hand prints 200 against the current stable release.
Concept: Automated analysis & rollback · Observability & Operations
# ledger/analysistemplate.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: ledger-smoke-test
  namespace: ledger
spec:
  args:
    - name: service-name
  metrics:
    - name: smoke-test
      count: 1
      failureLimit: 0
      provider:
        job:
          spec:
            backoffLimit: 0
            template:
              spec:
                restartPolicy: Never
                containers:
                  - name: smoke-test
                    image: curlimages/curl:latest
                    command: ["sh", "-c"]
                    args:
                      - |
                        fail=0
                        for i in $(seq 1 20); do
                          code=$(curl -s -o /dev/null -w '%{http_code}' "http://{{args.service-name}}.ledger.svc.cluster.local/")
                          echo "attempt $i -> $code"
                          [ "$code" = "200" ] || fail=$((fail+1))
                        done
                        echo "failures: $fail / 20"
                        [ "$fail" -le 2 ]     # allow up to 10% error rate before failing the step

Part C · Prove it: promote and abort

☺ Like you’re 10: Now push two toys down the belt — a good one and a broken one — and watch the traffic cop treat them completely differently, with nobody telling her which is which.

Milestone 9Ship the good version end-to-end — 🐦 Pip & 🦆 Dot
  1. Run pipelinerun-good.yaml from Milestone 5 again, but with tag: "1.1.0" (still build-arg-buggy: "false"). Do nothing else.
  2. Watch the whole chain fire without you touching the cluster: the Pipeline builds and pushes registry.local:5001/ledger:1.1.0, bump-tag commits it into ledger/rollout.yaml, Argo CD notices the drift and syncs, and Argo Rollouts starts a new canary automatically.
  3. Follow it live with kubectl argo rollouts get rollout ledger --watch. You should see SetWeight:10, a pause, an AnalysisRun go Successful, SetWeight:50, another pause and pass, then SetWeight:100 — fully promoted.
Done when: kubectl argo rollouts get rollout ledger shows Healthy at 100% with both prior AnalysisRuns Successful, and kubectl -n ledger get pods -l app=ledger -o jsonpath='{..image}' shows only 1.1.0 — the canary ReplicaSet fully replaced stable, and you ran zero kubectl apply commands to make it happen.
Concept: Blue/Green and canary — two rollout shapes
Milestone 10Ship a bad version and watch it abort itself — 🐦 Pip
  1. Run the Pipeline once more with tag: "1.2.0-bad" and build-arg-buggy: "true" — same source, same Dockerfile, one build argument flipped, which is exactly how a real regression usually ships: nobody meant to break anything.
  2. Watch the same automatic chain fire — build, push, commit, sync, canary starts at SetWeight:10 — except this time the smoke-test Job curls the canary Service 20 times, hits roughly 14 failures (the app fails ~70% of requests in buggy mode), blows well past failureLimit: 0, and the step fails.
  3. Confirm the Rollout does not proceed to SetWeight:50: it marks itself Degraded with the analysis reported as Failed, scales the canary ReplicaSet back toward zero, and every request through the main ledger Service is served by the still-fully-scaled 1.1.0 stable ReplicaSet.
  4. Curl the main Service in a loop during and after the abort to prove users never saw the bad version at any real scale: kubectl -n ledger run curltest --rm -it --image=curlimages/curl -- sh -c "for i in $(seq 1 30); do curl -s http://ledger.ledger.svc.cluster.local/ | head -c 80; echo; sleep 1; done".
Done when: kubectl argo rollouts get rollout ledger shows the rollout Degraded with the analysis step Failed, kubectl -n ledger get pods -l app=ledger -o jsonpath='{..image}' shows 1.1.0 as the overwhelming majority (stable never lost scale), and no human ran kubectl argo rollouts abort — the platform caught it itself.
Concept: Automated analysis & rollback · Anti-Patterns
# Milestone 9 — good release
kubectl create -f - <<'EOF'
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: ledger-ci-run-
  namespace: platform
spec:
  pipelineRef: { name: ledger-ci }
  taskRunTemplate: { serviceAccountName: ledger-ci }
  params:
    - { name: repo-url, value: "https://github.com/YOU/ledger-src.git" }
    - { name: config-repo-url, value: "https://github.com/YOU/platform-capstone.git" }
    - { name: image, value: "registry.local:5001/ledger" }
    - { name: tag, value: "1.1.0" }
    - { name: build-arg-buggy, value: "false" }
  workspaces:
    - { name: source, volumeClaimTemplate: { spec: { accessModes: ["ReadWriteOnce"], resources: { requests: { storage: 1Gi } } } } }
    - { name: config, volumeClaimTemplate: { spec: { accessModes: ["ReadWriteOnce"], resources: { requests: { storage: 256Mi } } } } }
EOF
kubectl argo rollouts get rollout ledger --watch -n ledger

# Milestone 10 — bad release: same shape, tag + build-arg-buggy flipped
kubectl create -f - <<'EOF'
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: ledger-ci-run-
  namespace: platform
spec:
  pipelineRef: { name: ledger-ci }
  taskRunTemplate: { serviceAccountName: ledger-ci }
  params:
    - { name: repo-url, value: "https://github.com/YOU/ledger-src.git" }
    - { name: config-repo-url, value: "https://github.com/YOU/platform-capstone.git" }
    - { name: image, value: "registry.local:5001/ledger" }
    - { name: tag, value: "1.2.0-bad" }
    - { name: build-arg-buggy, value: "true" }
  workspaces:
    - { name: source, volumeClaimTemplate: { spec: { accessModes: ["ReadWriteOnce"], resources: { requests: { storage: 1Gi } } } } }
    - { name: config, volumeClaimTemplate: { spec: { accessModes: ["ReadWriteOnce"], resources: { requests: { storage: 256Mi } } } } }
EOF
kubectl argo rollouts get rollout ledger --watch -n ledger

Part D · Operate and triage

☺ Like you’re 10: Learn the traffic cop’s hand signals for the days she needs a human, and learn what a jammed conveyor belt looks like so you can un-jam it fast.

Milestone 11Manual controls: pause, promote, abort, undo — 🦫 Benny
  1. Ship version 1.3.0 (good, same as Milestone 9), but this time kubectl argo rollouts pause ledger -n ledger during the first 60s pause window and leave it paused indefinitely — a manual gate on top of the automated one.
  2. Inspect it with kubectl argo rollouts get rollout ledger -n ledger — note the weight is frozen at 10% until you act.
  3. Promote it by hand: kubectl argo rollouts promote ledger -n ledger skips straight to the next step (or --full to jump directly to 100%).
  4. Now practice the panic button: mid-rollout, run kubectl argo rollouts abort ledger -n ledger and watch it snap back to stable immediately, then kubectl argo rollouts undo ledger -n ledger to roll the desired image back to the previous good revision.
  5. Note what does not happen: none of this fights Argo CD. Pause/promote/abort only touch the Rollout’s status subresource, a runtime detail — the spec (what’s in Git) is untouched, so selfHeal has nothing to revert. Only undo, which changes what image is actually desired, is something you’d still want to follow up with a Git commit so the two don’t disagree at the next promotion.
Done when: you’ve paused a rollout indefinitely and resumed it, aborted one mid-ramp, and rolled one back with undo — and can explain in one sentence why none of the three caused Argo CD to report drift.
Concept: Argo Rollouts · Self-heal and prune
kubectl argo rollouts pause ledger -n ledger
kubectl argo rollouts get rollout ledger -n ledger      # weight frozen
kubectl argo rollouts promote ledger -n ledger          # advance one step
kubectl argo rollouts promote ledger -n ledger --full   # jump straight to 100%

kubectl argo rollouts abort ledger -n ledger            # panic button: back to stable now
kubectl argo rollouts undo ledger -n ledger             # roll the desired image back a revision
kubectl argo rollouts dashboard                         # optional UI on :3100
Milestone 12Break the pipeline itself — 🐢 Timmy & 🦊 Foxy
  1. In ledger-src, typo a Dockerfile instruction (e.g. COPY app.py . → COPY ap.py .) on a branch, and run the Pipeline against that revision.
  2. The fetch Task succeeds — the typo is invisible to Git. The build Task’s Kaniko step fails with a “file not found” error, and the whole PipelineRun stops there — bump never runs, so platform-capstone is never touched and Argo CD never even sees an attempted change.
  3. Diagnose it the way you would in the exam: tkn pipelinerun logs --last -n platform (or kubectl -n platform logs -l tekton.dev/task=build-and-push --tail=100) to read Kaniko’s own error, then kubectl -n platform get taskrun -l tekton.dev/pipelineRun=<name> to see exactly which Task failed and which succeeded before it.
  4. Fix the Dockerfile, re-run, and confirm the same tag now succeeds end to end.
Done when: you can point at the exact failed step in tkn pipelinerun logs --last within under a minute, and you can state — correctly — why a broken Dockerfile never reaches Git or the cluster at all.
Concept: Triage Playbook · Triage · Delivery
# the Part 2 triage kit — worth memorising before the exam
tkn pipelinerun logs --last -f -n platform
kubectl -n platform get pipelinerun,taskrun
kubectl -n platform logs -l tekton.dev/task=build-and-push --tail=100

kubectl argo rollouts get rollout ledger -n ledger
kubectl -n ledger get analysisrun
kubectl -n ledger describe analysisrun <name>
kubectl -n ledger logs job/<analysis-job-name>   # the smoke test's own curl output

What Part 3 inherits

☺ Like you’re 10: By the end, your cluster doesn’t just fix itself — it ships itself, and it knows when to say no.

You now have a genuine little delivery machine: a Tekton pipeline in the platform namespace that builds ledger with Kaniko (no Docker daemon anywhere), pushes to a local registry, and ends — the only way a GitOps pipeline should — by committing a tag to platform-capstone; an Argo Rollouts Rollout in the ledger namespace that turns every one of those commits into a supervised canary, 10% → 50% → 100%; and a job-based AnalysisTemplate that judges each step on real HTTP traffic and aborts automatically when a build is bad. Nobody ran kubectl apply against the cluster at any point in Milestones 9 or 10 — CI pushed and committed, GitOps and Rollouts did the rest.

🦆 Dot’s-eye view

“I bumped a build argument, pushed to ledger-src, and went to make coffee. By the time I got back the good version had already promoted itself to 100%. Then I shipped the buggy one on purpose just to see — and it caught itself before I’d even opened a terminal to check. That’s the whole pitch of this domain in one afternoon.”

Part 3 — hosted by 🦋 Mira the Butterfly, pairing with Platform APIs, CRDs & Operators — picks up exactly here: it gives the platform a brand-new API noun for ledger’s data layer, either a Kubebuilder-scaffolded operator or a Crossplane XRD/Composition/Claim, and proves that controller self-heals a deleted child object the same way Recon self-heals drift. Keep the ledger namespace and its Argo CD Application alive — Part 3 builds directly on top of what’s running in it right now.

Checkpoint

☺ Like you’re 10: Before you move on, make sure you could explain today’s whole loop out loud, from a code push to a self-judging rollout.

🎬 At the Platform Guild
🦊

Foxy: Wait — the pipeline builds the image, but it never deploys anything? What’s even the point of it then?

🦫

Benny: That is the point. My pipeline pushes an image and commits a tag, full stop. It never holds a cluster credential, so it can’t shove a bad build past Recon even if it wanted to.

🤖

Recon: BEEP. I saw the commit. I applied the Rollout. Pip takes it from there.

🐦

Pip: And I don’t trust a new build just because it exists. Ten percent of traffic, twenty curls, and if more than two come back ugly I snap it back to stable before anyone notices. zip

👺

Gizmo: Boooring. Just bake kubectl set image into the last Tekton step and skip Git entirely — so much faster! 🤑

🐢

Timmy: And now your pipeline needs cluster-admin, your deploys have no audit trail, and Recon can’t reconcile what he never wrote down. No.

🦆

Dot: I shipped a bug on purpose today and it fixed itself before I finished my coffee. I’m never going back to filing a rollback ticket.

🐢 Timmy’s checkpoint

1. Why does the bump-tag Task push to platform-capstone instead of the pipeline running kubectl apply directly? 2. Without a service mesh, how does the ledger Rollout approximate a 10% traffic split, and why is that only approximate? 3. What does Argo Rollouts’ job analysis provider consider a “pass”? 4. When you run kubectl argo rollouts abort, why doesn’t Argo CD immediately try to “fix” it back? 5. In Milestone 12, why does a broken Dockerfile never reach Git or the cluster at all?

Check your answers
  1. Because a pipeline ending in kubectl apply needs cluster credentials and can push changes past the reconciler, creating instant, unaudited drift. Committing to Git keeps GitOps the single, observed door into the cluster — CI pushes, CD pulls.
  2. The main ledger Service load-balances across whatever pods currently exist; with no mesh doing request-level weighted routing, the split is really just the ratio of canary pods to stable pods (e.g. roughly 1-in-10 replicas), which is coarse and connection-sticky rather than an exact per-request 10%.
  3. A zero exit code from the Job's pod. The smoke-test container runs its own curl loop and script logic, and only exits 0 if the observed failure count is within the allowed threshold — Argo Rollouts itself just watches whether the Job succeeded or failed.
  4. pause/promote/abort only change the Rollout’s status (a runtime detail), not its spec (what’s declared in Git) — Argo CD’s selfHeal only reverts drift in the spec, so there’s nothing for it to revert.
  5. Because the Pipeline’s build Task fails before the bump Task ever runs — runAfter ordering means a failed Kaniko build stops the whole PipelineRun, so the config repo is never touched and Argo CD never sees a change to sync.