Hands-On · GitOps Labs

GitOps Labs

GitOps is the joint-largest slice of the CNPE, and it is the slice most likely to be graded by a script that runs kubectl get against your cluster rather than by anything you can say about the four principles. So this page is not a lesson — it is twelve reps. You will install Argo CD and watch a first Application go Synced; make drift die in front of you; order a database migration with a PreSync hook and sync waves; stop Argo fighting an HPA; onboard a team by creating a folder; hand every pull request its own preview environment; then do the same job again the Flux way, with a HelmRelease that repairs its own drift and an automation that opens a commit to bump an image tag. The last lab breaks everything on purpose and makes you find it. Tick each one off — your progress saves in this browser.

☺ Explain it like I’m 10

You already read about the tireless robot who keeps your bedroom matching the poster on the wall. Today you hire the robot. You will give it a poster, watch it tidy up, then mess the room on purpose to see how fast it notices. Then you teach it harder tricks — do the chores in a certain order, ignore the one drawer it isn’t allowed to touch, take care of a second bedroom, and give every visitor their own guest room. At the end you deliberately hand it a broken poster, because knowing what a confused robot looks like is the difference between fixing it in two minutes and staring at it for twenty.

🦫🤖Your hosts for these labs: Benny the Beaver & Recon the Robot — Benny lays the rails (repos, folders, manifests) and Recon is the reconciliation loop that never sleeps. Timmy, Pip and Sol drop in where a lab needs a guardrail, a pipeline or a lazy-but-correct shortcut.
⚠ Read this before lab 1

Every lab here runs on a local, throwaway cluster — kind or minikube — and nothing touches production or costs money. Tear it down when you’re finished (kind delete cluster --name gitops) and nothing lingers. Second, and more important: tool versions, CLI flags and API versions drift. Argo CD renames flags between minors, Flux graduates APIs from v2beta2 to v2, and install URLs move. Treat every command below as the shape of the answer, not a magic incantation — if one fails, open that project’s current quickstart (the same docs you’re allowed to open in the exam), check argocd version / flux --version, and adapt. Learning to do exactly that under time pressure is the exam skill.

⚖ CNPA vs CNPE — Twelve hands-on reps on a live cluster is a CNPE-only format — CNPA has no lab component whatsoever, it's a fully closed-book multiple-choice exam with zero external lookups of any kind. But the GitOps concepts these labs drill — prune vs. selfHeal, sync waves and hooks, ApplicationSets, drift detection — are exactly the kind of platform-engineering knowledge CNPA's closed-book recall still tests, just as multiple-choice recognition rather than something you build and break yourself.

Before you start

You need four things on your laptop: a container runtime (Docker or Podman), kind (or minikube), kubectl, and git. The argocd and flux CLIs get installed in labs 1 and 9. Nothing else.

You also need two repos you can push to, because several labs (prune, App-of-Apps, ApplicationSets, image automation) only mean anything if you can change what’s in Git. First, fork argoproj/argocd-example-apps on GitHub — lab 2 deletes a file out of it. Second, create an empty repo called platform-config and give it the shape below; labs 3 and 5–11 all commit into it. It is the same layout as the GitOps lesson.

platform-config/                 # YOUR fork — this is the repo the reconciler watches
├── apps/                        # one Argo CD Application manifest per app (lab 5)
├── tenants/                     # one folder per team, fanned out by an ApplicationSet (lab 7)
│   ├── payments/
│   └── search/
└── workloads/
    └── web/                     # a plain Deployment + Service you will break on purpose
        ├── deployment.yaml
        ├── service.yaml
        └── kustomization.yaml

Work the labs in order — later ones reuse the cluster, the app and the repo from earlier ones. Each should take five to fifteen minutes. When a lab clicks, deep-dive the concept via its linked lesson: GitOps Workflows, Argo CD, Flux. When a lab fights you, the Triage Playbook and the Speed Reference are the two pages to keep open.

0 / 12 labs complete

Part 1 · Argo CD — from a first sync to a fleet

Lab 1Install Argo CD and sync your first Application — 🦫 Benny
  1. Make the cluster: kind create cluster --name gitops. Confirm with kubectl get nodes.
  2. Install Argo CD into its own namespace: kubectl create namespace argocd then kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml. Wait for it: kubectl -n argocd rollout status deploy/argocd-server.
  3. Install the argocd CLI — you need it from here to lab 12 (brew install argocd, or the release binary; both in the block below). Check it answers: argocd version --client.
  4. Get in. Read the bootstrap password, port-forward, and log in with the CLI (see the block below). Open https://localhost:8080 in a browser too — the resource tree is worth seeing at least once.
  5. Apply the Application below. Note there is no automated: block yet — this is deliberately a manual sync, so you can see the two halves of GitOps separately.
  6. Watch it sit at OutOfSync, then sync it by hand: argocd app sync guestbook. Watch the resource tree fill in.
Done when: argocd app get guestbook prints Sync Status: Synced and Health Status: Healthy, and kubectl -n guestbook get deploy lists guestbook-ui with 1/1 ready.
# --- install the argocd CLI ---
brew install argocd                       # macOS / Linux with Homebrew
# ...or grab the release binary (Linux amd64):
curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
sudo install -m 555 argocd /usr/local/bin/argocd && rm argocd
argocd version --client

# --- get in ---
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d ; echo
# newer CLIs have a shortcut for the same value:
argocd admin initial-password -n argocd

kubectl -n argocd port-forward svc/argocd-server 8080:443     # leave running
argocd login localhost:8080 --username admin --insecure       # another terminal; paste that password
# guestbook.yaml — apply with: kubectl apply -f guestbook.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd                    # Applications live beside the controller
spec:
  project: default
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: HEAD
    path: guestbook
  destination:
    server: https://kubernetes.default.svc   # "in-cluster"
    namespace: guestbook
  syncPolicy:
    syncOptions:
      - CreateNamespace=true           # Argo creates the destination namespace
Lab 2Turn on prune + selfHeal and watch drift die — 🤖 Recon
  1. Switch the app from “I sync when told” to “I enforce”: argocd app set guestbook --sync-policy automated --auto-prune --self-heal (or patch spec.syncPolicy.automated to {prune: true, selfHeal: true} — both shown below).
  2. Prove self-heal. Drift the cluster by hand: kubectl -n guestbook scale deploy/guestbook-ui --replicas=5. Immediately run kubectl -n guestbook get deploy guestbook-ui -w and watch the replica count fall back to 1 within seconds. Do it again with a deletion: kubectl -n guestbook delete svc guestbook-ui — it comes back.
  3. Look at what Recon saw: argocd app diff guestbook during the drift window, and argocd app history guestbook afterwards.
  4. Prove prune. Repoint the Application at your fork (argocd app set guestbook --repo https://github.com/YOU/argocd-example-apps.git), then delete guestbook/guestbook-ui-svc.yaml in your fork and push. Force a refresh with argocd app get guestbook --hard-refresh instead of waiting out the 3-minute poll.
  5. Now restore the file in Git (git revert the deletion and push), turn pruning off — argocd app set guestbook --auto-prune=false — and delete it again. This time the Service survives as an orphan and the app reports OutOfSync forever. That contrast is the whole point of the flag.
Done when: after a manual scale, kubectl -n guestbook get deploy guestbook-ui -o jsonpath='{.spec.replicas}' prints 1 again without you touching it; and after deleting a manifest from Git, kubectl -n guestbook get svc no longer lists guestbook-ui while argocd app get guestbook stays Synced.
# the CLI way
argocd app set guestbook --sync-policy automated --auto-prune --self-heal

# the declarative way — this is what you would actually commit
kubectl -n argocd patch application guestbook --type merge -p \
  '{"spec":{"syncPolicy":{"automated":{"prune":true,"selfHeal":true}}}}'

# watch the loop work
kubectl -n guestbook scale deploy/guestbook-ui --replicas=5
kubectl -n guestbook get deploy guestbook-ui -w      # 5 ... 5 ... 1
argocd app get guestbook --hard-refresh              # skip the 3-minute poll
Lab 3Sync waves & a PreSync hook that runs the migration — 🤖 Recon & 🐦 Pip
  1. In your fork create workloads/ordered/ and commit the three manifests below: a PreSync hook Job, a ConfigMap annotated sync-wave: "-1", and a Deployment in the default wave 0.
  2. Create an Application for that path with automated sync (argocd app create ordered --repo <your fork> --path workloads/ordered --dest-server https://kubernetes.default.svc --dest-namespace ordered --sync-policy automated --sync-option CreateNamespace=true).
  3. Watch the ordering happen live in one terminal: kubectl -n ordered get pods -w. The db-migrate pod must appear, run and Complete before the web pods are created. Hooks run outside the waves; PreSync always goes first.
  4. Confirm the wave ordering too: the ConfigMap (wave -1) is applied before the Deployment (wave 0), and Argo waits for each wave to become Healthy before starting the next.
  5. Bump the Deployment’s image tag in Git and push. The migration Job runs again on every sync — which is why real migrations must be idempotent. Then read the hook’s output: argocd app get ordered shows the hook phase, and kubectl -n ordered logs job/db-migrate shows what it printed (delete the hook-delete-policy annotation first if you want the pod to stick around).
Done when: argocd app get ordered --show-operation lists db-migrate under a PreSync phase with status Succeeded; and — once you have removed the hook-delete-policy annotation so the pod survives its own success — kubectl -n ordered get pods --sort-by=.metadata.creationTimestamp prints the db-migrate pod above every web-* pod.
# workloads/ordered/00-migrate.yaml — a hook, not a wave: PreSync runs before everything
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded   # tidy up only if it worked
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: busybox:1.36
          command: ["sh","-c","echo 'applying schema 42...'; sleep 10; echo migrated"]
---
# workloads/ordered/10-config.yaml — wave -1: lands before the Deployment
apiVersion: v1
kind: ConfigMap
metadata:
  name: web-config
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
data:
  GREETING: "hello from wave -1"
---
# workloads/ordered/20-web.yaml — no annotation = wave 0, the default
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector: { matchLabels: { app: web } }
  template:
    metadata: { labels: { app: web } }
    spec:
      containers:
        - name: web
          image: ghcr.io/stefanprodan/podinfo:6.7.0
          envFrom:
            - configMapRef: { name: web-config }
          resources:
            requests:
              cpu: 50m          # the HPA in lab 4 needs a request to compute a % against
Lab 4ignoreDifferences — stop Argo fighting the HPA — 🦫 Benny & 🦥 Sol
  1. Give the cluster metrics so an HPA can act: install metrics-server and patch it for kind’s self-signed kubelet certs (block below). Confirm with kubectl top nodes.
  2. Autoscale the lab-3 Deployment: kubectl -n ordered autoscale deploy/web --min=3 --max=6 --cpu-percent=80. The HPA immediately raises spec.replicas to 3 (it enforces minReplicas before it ever reads a metric). Check it with kubectl -n ordered get hpa web — the TARGETS column reads <unknown> until metrics-server has a sample, and stays <unknown> forever if the container has no CPU request, which is why lab 3’s manifest sets one.
  3. Watch the fight. Git says replicas: 2; the HPA says 3. With selfHeal on, Argo reverts to 2, the HPA pushes back to 3, forever. Observe it: kubectl -n ordered get deploy web -w and argocd app get ordered flapping between Synced and OutOfSync.
  4. Fix it properly. Add the ignoreDifferences block below to the Application and the RespectIgnoreDifferences=true sync option — without that option the field is only hidden from the diff, and the next sync still overwrites it.
  5. Note the alternative Sol would pick: delete the replicas: line from the manifest in Git entirely, so the field has no desired value to fight over. Two valid answers; know both.
Done when: kubectl -n ordered get deploy web -o jsonpath='{.spec.replicas}' stays at the HPA’s number (3 or more) for a full minute while argocd app get ordered continues to report Synced, and argocd app diff ordered shows no replica difference.
# metrics-server on kind needs one patch (self-signed kubelet certs)
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl -n kube-system patch deploy metrics-server --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
kubectl -n kube-system rollout status deploy/metrics-server
kubectl top nodes
# add to the Application spec — BOTH halves are required
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: web
      jsonPointers:
        - /spec/replicas          # the HPA owns this field, not Git
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions:
      - RespectIgnoreDifferences=true   # without this, sync still overwrites it
Lab 5App-of-Apps — one root app plants the rest — 🦫 Benny
  1. In your fork, put Application manifests (not workloads) in apps/: one file per app, each pointing at a different path — reuse guestbook and ordered from labs 1–3, plus one new one.
  2. Apply the single root Application below. It points at apps/ with directory.recurse: true, so its “workload” is other Applications.
  3. Watch the fan-out: kubectl -n argocd get applications -w. One apply produced N apps. In the UI, the root app’s resource tree is a list of Applications.
  4. Prove the cascade. Delete one child file from apps/ in Git and push. Because the root has prune: true, the child Application object is deleted — and because it carries the resources-finalizer.argocd.argoproj.io finalizer (a finalizer, not an annotation), its workloads are deleted with it rather than orphaned.
  5. Re-add the file. The whole environment comes back from one commit. That round trip is the disaster-recovery story you should be able to tell out loud.
Done when: kubectl -n argocd get applications lists root plus every child, all Synced; removing one child file from Git makes that child disappear from the same command within one refresh, taking its namespace’s workloads with it.
# root.yaml — the only thing you apply by hand, ever
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/YOU/platform-config.git
    targetRevision: main
    path: apps                       # a folder full of Application manifests
    directory:
      recurse: true
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd                # children are created here
  syncPolicy:
    automated: { prune: true, selfHeal: true }
---
# apps/guestbook.yaml — a child, committed to Git like any other manifest
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io   # delete my workloads when I am deleted
spec:
  project: default
  source:
    repoURL: https://github.com/YOU/platform-config.git
    targetRevision: main
    path: workloads/web
  destination:
    server: https://kubernetes.default.svc
    namespace: guestbook
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [ CreateNamespace=true ]
Lab 6AppProject — fence a tenant in — 🐢 Timmy & 🤖 Recon
  1. Apply the payments AppProject below. Read each list out loud as a sentence: this tenant may deploy only from this repo, only into namespaces matching payments-*, and may not create cluster-scoped resources at all.
  2. Break rule one. Try to create an app in that project from a repo the project doesn’t allow: argocd app create rogue --project payments --repo https://github.com/argoproj/argocd-example-apps.git --path guestbook --dest-server https://kubernetes.default.svc --dest-namespace payments-a. It is refused with a message naming the project.
  3. Break rule two. Create a legal app but aim it at --dest-namespace default. Refused again — the destination isn’t permitted.
  4. Break rule three. Commit a ClusterRole into the tenant’s path and let a permitted app sync it. The sync fails with a “resource … is not permitted in project” condition — check argocd app get <app> and kubectl -n argocd get application <app> -o jsonpath='{.status.conditions}'.
  5. Fix each violation the way a platform engineer would: widen the project deliberately (and reviewably, in Git) or move the resource somewhere it belongs. Never by dropping the project.
Done when: all three attempts above are rejected with an error that names the payments project, and kubectl -n argocd get appproject payments -o yaml shows the source, destination and resource restrictions you wrote.
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: payments
  namespace: argocd
spec:
  description: What the payments team is allowed to deploy
  sourceRepos:
    - https://github.com/YOU/platform-config.git      # this repo only
  destinations:
    - server: https://kubernetes.default.svc
      namespace: 'payments-*'                         # these namespaces only
  clusterResourceWhitelist: []                        # no cluster-scoped resources at all
  namespaceResourceBlacklist:
    - group: ''
      kind: ResourceQuota                             # tenants can't raise their own quota
    - group: ''
      kind: LimitRange
  roles:
    - name: read-only
      description: Tenant devs may look, and sync, but not delete
      policies:
        - p, proj:payments:read-only, applications, get, payments/*, allow
        - p, proj:payments:read-only, applications, sync, payments/*, allow
Lab 7ApplicationSet — onboard a team with a folder, then a whole second cluster — 🦫 Benny
  1. Part A — git directory generator. In your fork, put a tiny Deployment in tenants/payments/ and another in tenants/search/. Apply the first ApplicationSet below. Two Applications appear, each with its own namespace, from one manifest.
  2. Now do the thing the whole lab exists to prove: mkdir tenants/checkout, drop a manifest in, commit, push. Within requeueAfterSeconds a third Application and namespace exist — onboarding a team is a folder. Watch with kubectl -n argocd get applications -w.
  3. Delete the tenants/search/ folder and confirm the Application is removed. (ApplicationSet deletes generated apps by default; preserveResourcesOnDeletion changes that.)
  4. Part B — cluster generator (+5 min). Build a second cluster: kind create cluster --name edge. Register it with Argo CD using the internal kubeconfig — kind’s normal kubeconfig points at 127.0.0.1, which is unreachable from inside the first cluster. That gotcha is in the block below.
  5. Apply the cluster-generator ApplicationSet. One manifest, one app per registered cluster (in-cluster plus edge). Confirm with argocd cluster list and kubectl -n argocd get applications, then check the workload really landed on the second cluster: kubectl --context kind-edge get deploy -A.
Done when: adding tenants/checkout/ to Git makes a matching Application and namespace appear with no other action; and kubectl -n argocd get applications lists one agent-* app per entry in argocd cluster list, with the workload visible under kubectl --context kind-edge get pods -A.
# Part A — one Application per folder under tenants/
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: tenants
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: https://github.com/YOU/platform-config.git
        revision: main
        requeueAfterSeconds: 60          # how fast a new folder is noticed
        directories:
          - path: tenants/*
  template:
    metadata:
      name: '{{.path.basename}}'         # "payments", "search", "checkout"
    spec:
      project: default
      source:
        repoURL: https://github.com/YOU/platform-config.git
        targetRevision: main
        path: '{{.path.path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.path.basename}}'
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [ CreateNamespace=true ]
---
# Part B — one Application per registered cluster
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: monitoring-everywhere
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - clusters: {}                       # empty selector = every cluster Argo knows
  template:
    metadata:
      name: 'agent-{{.name}}'            # agent-in-cluster, agent-edge
    spec:
      project: default
      source:
        repoURL: https://github.com/YOU/platform-config.git
        targetRevision: main
        path: workloads/web
      destination:
        server: '{{.server}}'
        namespace: platform
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [ CreateNamespace=true ]
# registering a second kind cluster — the 127.0.0.1 trap
kind create cluster --name edge
kind get kubeconfig --name edge --internal > /tmp/edge.kubeconfig   # docker-network address
argocd cluster add kind-edge --kubeconfig /tmp/edge.kubeconfig --name edge
argocd cluster list          # in-cluster + edge, both with a recent "successful" status
Lab 8A preview environment for every pull request — 🐦 Pip & 🦫 Benny
  1. Create a read-only GitHub personal access token scoped to your fork (public-repo read is enough) and put it in a Secret: kubectl -n argocd create secret generic gh-token --from-literal=token=YOUR_TOKEN. Never commit the token itself — see Secrets Management for how this is done properly.
  2. Apply the pull-request ApplicationSet below. Nothing happens yet: there are no open PRs.
  3. Open a PR in your fork — change one line in workloads/web/deployment.yaml (say the image tag or a replica count) on a branch and open it against main.
  4. Within requeueAfterSeconds an Application named preview-<number> appears, syncing that PR’s head_sha into its own namespace. Confirm with kubectl -n argocd get applications | grep preview and kubectl get ns | grep preview. Port-forward into it and see the change running.
  5. Close the PR. The generator stops returning it, the Application is deleted, and prune takes the namespace with it. Zero cleanup tickets — that is the whole feature.
  6. Optional: uncomment the labels: list inside the github: block so only PRs carrying the preview label get an environment — on a busy repo you want that. Note where it lives: filters: only takes regexes (branchMatch, targetBranchMatch, titleMatch), label selection is a provider option.
Done when: opening a PR creates a preview-<n> Application within a minute, closing it removes both the Application and the namespace (kubectl get ns | grep preview returns nothing), and you never ran a single command in between.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: previews
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - pullRequest:
        github:
          owner: YOU
          repo: platform-config
          # labels:                    # step 6: uncomment to pick up only PRs
          #   - preview                #   carrying the "preview" label
          tokenRef:
            secretName: gh-token
            key: token
        requeueAfterSeconds: 60
        filters:                       # filters take regexes on branch/title, not labels
          - targetBranchMatch: main    # only PRs opened against main
  template:
    metadata:
      name: 'preview-{{.number}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/YOU/platform-config.git
        targetRevision: '{{.head_sha}}'      # the PR's commit, not main
        path: workloads/web
      destination:
        server: https://kubernetes.default.svc
        namespace: 'preview-{{.number}}'
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [ CreateNamespace=true ]

Part 2 · The same job, the Flux way

Everything above is one product’s opinion. Flux solves the same problem with small composable controllers and no UI — and the exam tool list names both. These three labs deliberately reconcile a similar app so the differences stand out: no Application object, an explicit interval on everything, and image automation built in rather than bolted on.

Lab 9Install Flux and reconcile an app with GitRepository + Kustomization — 🤖 Recon
  1. Install the CLI (brew install fluxcd/tap/flux or curl -s https://fluxcd.io/install.sh | sudo bash), then check the cluster is suitable: flux check --pre.
  2. Install the controllers: flux install. (The production path is flux bootstrap github --owner=YOU --repository=platform-config --path=clusters/dev --personal, which commits Flux’s own manifests into your repo so Flux manages Flux. You need that in lab 11 — do it now if you like.) Confirm: kubectl -n flux-system get pods.
  3. Create the source and the reconciler with two CLI calls, or apply the equivalent YAML below — read both, because the exam may ask for either.
  4. Verify: flux get sources git then flux get kustomizations. Both must show Ready: True with an applied revision like master@sha1:…. The workload: kubectl -n default get deploy podinfo.
  5. Compare drift behaviour with lab 2. Scale it by hand: kubectl -n default scale deploy/podinfo --replicas=5. Flux corrects it at the next interval, not in seconds — that is the design difference. Prove you can force it: flux reconcile kustomization podinfo --with-source brings it back immediately.
Done when: flux get kustomizations shows podinfo as Ready=True with an applied revision, and a hand-scaled Deployment returns to its Git replica count after flux reconcile kustomization podinfo --with-source (or by itself within one interval).
flux check --pre
flux install

flux create source git podinfo \
  --url=https://github.com/stefanprodan/podinfo \
  --branch=master --interval=1m

flux create kustomization podinfo \
  --source=GitRepository/podinfo --path="./kustomize" \
  --target-namespace=default --prune=true --interval=1m --wait=true

flux get sources git
flux get kustomizations
flux reconcile kustomization podinfo --with-source     # don't wait for the interval
# the same thing declaratively — this is what you commit
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 1m                     # Flux puts the cadence on every object
  url: https://github.com/stefanprodan/podinfo
  ref:
    branch: master
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 1m
  sourceRef: { kind: GitRepository, name: podinfo }
  path: ./kustomize
  targetNamespace: default
  prune: true                      # Flux's equivalent of Argo's prune
  wait: true                       # block until resources report Ready
Lab 10A HelmRelease that repairs its own drift — 🦫 Benny
  1. Apply the HelmRepository + HelmRelease below. Flux’s helm-controller does the install and every upgrade — you never run helm upgrade yourself again.
  2. Confirm the release: flux get helmreleases -A shows Ready=True with the resolved chart version, and the workload exists — kubectl -n default get deploy podinfo-helm. (If you happen to have the helm CLI, helm -n default list shows the same release, owned by helm-controller. It is not a prerequisite for this lab.)
  3. Drift it. kubectl -n default set env deploy/podinfo-helm DRIFT=yes. With driftDetection.mode: enabled, the controller notices the live state no longer matches the rendered chart and runs a corrective upgrade within the interval.
  4. Prove why it changed, not just that it did: kubectl -n default describe helmrelease podinfo-helm and look for a drift-detected event and an upgrade; flux logs --kind=HelmRelease --name=podinfo-helm shows the same story.
  5. Now carve out an exception — the same idea as lab 4. The ignore block tells drift detection to leave /spec/replicas alone so an HPA can own it. Add it, scale by hand, and confirm that change now survives while your env-var change still does not.
Done when: kubectl -n default get deploy podinfo-helm -o jsonpath='{.spec.template.spec.containers[0].env}' no longer contains DRIFT after one interval, describe helmrelease shows the drift-triggered upgrade, and a hand-set replica count survives once it is in the ignore list.
Concept: Flux · Helm · Config Management
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 10m
  url: https://stefanprodan.github.io/podinfo
---
apiVersion: helm.toolkit.fluxcd.io/v2       # older Flux: v2beta2
kind: HelmRelease
metadata:
  name: podinfo-helm
  namespace: default
spec:
  interval: 1m
  chart:
    spec:
      chart: podinfo
      version: '6.*'
      sourceRef:
        kind: HelmRepository
        name: podinfo
        namespace: flux-system
  driftDetection:
    mode: enabled                # warn | enabled | disabled
    ignore:
      - paths: ["/spec/replicas"]
        target: { kind: Deployment }   # let the HPA own this field
  values:
    replicaCount: 2
    ui:
      message: "reconciled by Flux"
Lab 11Image automation writes the tag bump back to Git — 🐦 Pip
  1. This lab needs Flux bootstrapped against your repo with write access, because the whole point is that Flux opens a commit. Export a GitHub token with repo scope first — export GITHUB_TOKEN=<your-pat>, bootstrap reads it from the environment and fails without it. Then, if you skipped it in lab 9: flux bootstrap github --owner=YOU --repository=platform-config --path=clusters/dev --personal --components-extra=image-reflector-controller,image-automation-controller.
  2. Commit a Deployment into clusters/dev/ whose image line carries the setter marker comment — the automation only edits lines it has been told to edit:
    image: ghcr.io/stefanprodan/podinfo:6.6.0 # {"$imagepolicy": "flux-system:podinfo"}
    Name it podinfo-auto and set metadata.namespace: default explicitly. Two reasons: lab 9’s Kustomization already owns a Deployment called podinfo in default and two reconcilers fighting over one object is a mess, and an explicit namespace means you know where to look for it.
  3. Apply the three objects below: ImageRepository (scan the registry), ImagePolicy (choose the newest tag in the 6.x range), ImageUpdateAutomation (rewrite Git and push).
  4. Watch the scan find tags: flux get image repository podinfo then flux get image policy podinfo. The policy prints the latest tag it selected.
  5. Now git pull your repo. There is a commit you did not write, authored by fluxcdbot, bumping that image line. The Kustomization then reconciles it into the cluster — the loop closes with Git still the single source of truth. Confirm with kubectl -n default get deploy podinfo-auto -o jsonpath='{..image}'.
Done when: git log --oneline -3 in your repo shows a commit authored by the Flux bot that changed the image tag, and kubectl -n default get deploy podinfo-auto -o jsonpath='{..image}' prints the exact tag that flux get image policy podinfo reports as the latest.
apiVersion: image.toolkit.fluxcd.io/v1beta2      # check `flux --version`: APIs move
kind: ImageRepository
metadata: { name: podinfo, namespace: flux-system }
spec:
  image: ghcr.io/stefanprodan/podinfo
  interval: 5m
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata: { name: podinfo, namespace: flux-system }
spec:
  imageRepositoryRef: { name: podinfo }
  policy:
    semver:
      range: 6.x                                 # never jump a major by accident
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata: { name: podinfo, namespace: flux-system }
spec:
  interval: 1m
  sourceRef: { kind: GitRepository, name: flux-system }
  git:
    checkout:
      ref: { branch: main }
    commit:
      author:
        name: fluxcdbot
        email: fluxcdbot@users.noreply.github.com
      messageTemplate: "chore: bump {{range .Updated.Images}}{{println .}}{{end}}"
    push: { branch: main }
  update:
    path: ./clusters/dev
    strategy: Setters                            # edit only marked lines

Part 3 · When the reconciler is unhappy

Lab 12Break it on purpose: OutOfSync, Degraded, and a hook that failed — 🐢 Timmy & 🤖 Recon
  1. Fault 1 — nothing renders. Point an Application at a folder that doesn’t exist: argocd app set ordered --path workloads/nope. The app goes Unknown with a ComparisonError. Find the real message: argocd app get <app>, then kubectl -n argocd logs deploy/argocd-repo-server --tail=50 — the repo-server renders manifests, so path, Helm and Kustomize errors surface there, not in the controller.
  2. Fault 2 — synced but sick. Put the path back (argocd app set ordered --path workloads/ordered), then in workloads/ordered/20-web.yaml set the image to ghcr.io/stefanprodan/podinfo:9.9.9-nope and push. The app reports Synced (Git and cluster agree!) but Degraded. Diagnose down the stack: kubectl -n <ns> get podsImagePullBackOff, then kubectl -n <ns> describe pod <pod> for the pull error. Internalise that Synced ≠ Healthy — it is the single most common misread on this domain.
  3. Fault 3 — the hook fails. In lab 3’s Job, change the command to sh -c "echo migrating; exit 1". The sync stops at PreSync; the Deployment is never updated; the app stays on the old version. Read the failed hook: argocd app get ordered shows the hook phase Failed, and kubectl -n ordered logs job/db-migrate shows why (the pod survives because HookSucceeded only deletes on success — that is the flag doing you a favour).
  4. Roll back like an adult. argocd app history <app> then argocd app rollback <app> <id> to get healthy fast — and then fix Git, because with selfHeal on, a rollback that isn’t in Git is temporary by design.
  5. Repeat one fault on the Flux side. Point the Kustomization’s path at a missing folder and diagnose with flux get all -A, kubectl -n flux-system describe kustomization podinfo, and flux logs --level=error --all-namespaces. Different words, same three questions: did it fetch, did it render, did it become healthy?
Done when: you have reproduced all three faults and read each one’s real error out of the right place (repo-server logs / kubectl describe pod / the hook Job’s logs) — and then, with Git repaired and no manual kubectl apply, argocd app get ordered prints Sync Status: Synced and Health Status: Healthy, and flux get kustomizations shows Ready=True again.
# the triage kit — worth memorising before the exam
argocd app get <app>                       # sync status + health + per-resource state
argocd app diff <app>                      # exactly which fields differ
argocd app history <app> ; argocd app rollback <app> <id>
argocd app sync <app> --dry-run
kubectl -n argocd logs deploy/argocd-repo-server --tail=50          # render errors
kubectl -n argocd logs statefulset/argocd-application-controller --tail=50
kubectl -n argocd get application <app> -o jsonpath='{.status.conditions}'

flux get all -A                            # every source, kustomization, helmrelease
flux logs --level=error --all-namespaces
flux reconcile kustomization <name> --with-source
kubectl -n flux-system describe kustomization <name>
kubectl -n <ns> get events --sort-by=.lastTimestamp | tail -20
🦆 Dot’s-eye view

“Lab 8 is the one that changed my life, honestly. Before preview environments I’d ask the platform team for a staging slot, wait a day, and find someone else’s branch on it. Now I open a PR and my reviewer gets a URL. If you build only one thing from this page for your real team, build that one.”

What you’ll have built

☺ Like you’re 10: By the end you have a cluster that fixes itself, does chores in the right order, ignores the one drawer it shouldn’t touch, gives every visitor a guest room, and hands you a clear complaint when something is wrong.

Twelve labs in, your throwaway cluster is a genuine little GitOps control plane: an Argo CD install reconciling several apps from your own repo with prune and selfHeal enforcing desired state; ordered rollouts via sync waves and a PreSync migration hook; an ignoreDifferences carve-out so autoscaling and GitOps coexist; an App-of-Apps root that can rebuild the whole environment from one commit; an AppProject that fences a tenant to one repo, one namespace pattern and no cluster-scoped resources; ApplicationSets that onboard a team by folder, fan out across two clusters, and hand every pull request a disposable environment; a parallel Flux stack with GitRepository, Kustomization, a drift-detecting HelmRelease and image automation committing tag bumps back to Git; and — the part the exam actually rewards — the reflex to tell OutOfSync from Degraded from a failed hook in under a minute.

That covers essentially all of Domain 2 — GitOps & Continuous Delivery, 25% and touches Domains 1, 3 and 5 through projects, tenancy and fleet fan-out. Next, take the same cluster into the CI/CD & progressive delivery labs, put the whole thing in context with the full lab track, and pressure-test the knowledge with practice tasks and the GitOps practice set. If you can rewrite the Application, ApplicationSet and Kustomization manifests from memory — check yourself against Know It Cold — this domain is finished.

🦫 Benny’s challenge · going further

Comfortable? Stack these on top. (1) Combine a matrix generator — every tenant folder × every cluster — and watch four apps appear from one manifest. (2) Put Argo CD itself under Argo CD, so the reconciler manages its own manifests (then break it and learn why people keep a bootstrap script). (3) Add a PostSync hook that only runs on success, and a SyncFail hook that posts the failure somewhere. (4) Encrypt a Secret with SOPS and let Flux decrypt it on apply, or wire External Secrets so the repo holds only a reference. (5) Replace the Deployment in lab 3 with an Argo Rollouts canary so a bad tag bump aborts itself instead of reaching every pod.

🎬 At the Platform Guild
🦊

Foxy: Lab 4 took me forty minutes. Argo kept resetting my replicas and I kept re-scaling. I nearly turned self-heal off.

🤖

Recon: BEEP. Working as designed. Git said two. You said three. I am not a negotiator.

🦫

Benny: And turning self-heal off would have “fixed” it the way removing the smoke alarm fixes burnt toast. The real fix is telling Recon which field he doesn’t own.

🦥

Sol: Or just… delete replicas from the manifest. No field, no fight. Least effort, correct answer. My favourite combination.

👺

Gizmo: OR you skip the whole page and kubectl apply straight at prod like a free spirit. Nobody checks. 😈

🐢

Timmy: Recon checks, Gizmo. Every three minutes. Forever.

🦆

Dot: And if lab 12 taught me one thing: when it’s red, read the repo-server logs before you panic. Nine times out of ten it’s a typo in a path.

🐢 Timmy’s checkpoint

1. An app shows Synced but Degraded — what does that tell you, and which command do you run next? 2. Which two settings must you add together so Argo CD stops overwriting a field an HPA owns? 3. You added tenants/checkout/ to Git and no Application appeared. Name two plausible causes. 4. Where does a PreSync hook run relative to sync wave -1? 5. In Flux, what makes drift get corrected — and how is that different from Argo CD’s selfHeal? 6. What does the {"$imagepolicy": …} comment in a manifest do?

Check your answers
  1. Git and the cluster agree — the manifests were applied successfully — but the workload isn’t healthy, so the bug is in the workload, not the reconciler. Next: kubectl -n <ns> get pods then kubectl describe pod (or logs --previous). Chasing the Argo logs here wastes minutes.
  2. spec.ignoreDifferences for /spec/replicas and the RespectIgnoreDifferences=true sync option. Without the second one the field is only hidden from the diff and the next sync still stomps it. The alternative correct answer: remove replicas from the manifest entirely.
  3. Any two of: the generator hasn’t requeued yet (requeueAfterSeconds / the 3-minute refresh — force it with a hard refresh or a webhook); the folder contains no manifests, so the directory generator skipped it; you pushed to a branch other than the revision the generator watches; or the ApplicationSet controller is erroring — check kubectl -n argocd logs deploy/argocd-applicationset-controller.
  4. Before it. Hooks are a separate phase: all PreSync hooks complete before any wave is applied, including negative waves. Waves order resources within the sync phase.
  5. Flux re-applies the rendered manifests on every interval (and HelmRelease adds explicit driftDetection), so drift disappears at the next tick — cadence is declared per object. Argo CD’s selfHeal reacts to watch events on live resources, so it typically reverts within seconds. Same outcome, different trigger.
  6. It is a setter marker: it tells the image-automation controller that this exact line is owned by the named ImagePolicy, so the controller may rewrite the tag there and commit the change back to Git. No marker, no edit.