GitOps Labs
GitOps is the joint-largest slice of the CNPE, and it is the slice most likely to be graded by a script that runs kubectl get against your cluster rather than by anything you can say about the four principles. So this page is not a lesson — it is twelve reps. You will install Argo CD and watch a first Application go Synced; make drift die in front of you; order a database migration with a PreSync hook and sync waves; stop Argo fighting an HPA; onboard a team by creating a folder; hand every pull request its own preview environment; then do the same job again the Flux way, with a HelmRelease that repairs its own drift and an automation that opens a commit to bump an image tag. The last lab breaks everything on purpose and makes you find it. Tick each one off — your progress saves in this browser.
You already read about the tireless robot who keeps your bedroom matching the poster on the wall. Today you hire the robot. You will give it a poster, watch it tidy up, then mess the room on purpose to see how fast it notices. Then you teach it harder tricks — do the chores in a certain order, ignore the one drawer it isn’t allowed to touch, take care of a second bedroom, and give every visitor their own guest room. At the end you deliberately hand it a broken poster, because knowing what a confused robot looks like is the difference between fixing it in two minutes and staring at it for twenty.
Every lab here runs on a local, throwaway cluster — kind or minikube — and nothing touches production or costs money. Tear it down when you’re finished (kind delete cluster --name gitops) and nothing lingers. Second, and more important: tool versions, CLI flags and API versions drift. Argo CD renames flags between minors, Flux graduates APIs from v2beta2 to v2, and install URLs move. Treat every command below as the shape of the answer, not a magic incantation — if one fails, open that project’s current quickstart (the same docs you’re allowed to open in the exam), check argocd version / flux --version, and adapt. Learning to do exactly that under time pressure is the exam skill.
⚖ CNPA vs CNPE — Twelve hands-on reps on a live cluster is a CNPE-only format — CNPA has no lab component whatsoever, it's a fully closed-book multiple-choice exam with zero external lookups of any kind. But the GitOps concepts these labs drill — prune vs. selfHeal, sync waves and hooks, ApplicationSets, drift detection — are exactly the kind of platform-engineering knowledge CNPA's closed-book recall still tests, just as multiple-choice recognition rather than something you build and break yourself.
Before you start
You need four things on your laptop: a container runtime (Docker or Podman), kind (or minikube), kubectl, and git. The argocd and flux CLIs get installed in labs 1 and 9. Nothing else.
You also need two repos you can push to, because several labs (prune, App-of-Apps, ApplicationSets, image automation) only mean anything if you can change what’s in Git. First, fork argoproj/argocd-example-apps on GitHub — lab 2 deletes a file out of it. Second, create an empty repo called platform-config and give it the shape below; labs 3 and 5–11 all commit into it. It is the same layout as the GitOps lesson.
platform-config/ # YOUR fork — this is the repo the reconciler watches
├── apps/ # one Argo CD Application manifest per app (lab 5)
├── tenants/ # one folder per team, fanned out by an ApplicationSet (lab 7)
│ ├── payments/
│ └── search/
└── workloads/
└── web/ # a plain Deployment + Service you will break on purpose
├── deployment.yaml
├── service.yaml
└── kustomization.yamlWork the labs in order — later ones reuse the cluster, the app and the repo from earlier ones. Each should take five to fifteen minutes. When a lab clicks, deep-dive the concept via its linked lesson: GitOps Workflows, Argo CD, Flux. When a lab fights you, the Triage Playbook and the Speed Reference are the two pages to keep open.
Part 1 · Argo CD — from a first sync to a fleet
- Make the cluster:
kind create cluster --name gitops. Confirm withkubectl get nodes. - Install Argo CD into its own namespace:
kubectl create namespace argocdthenkubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml. Wait for it:kubectl -n argocd rollout status deploy/argocd-server. - Install the
argocdCLI — you need it from here to lab 12 (brew install argocd, or the release binary; both in the block below). Check it answers:argocd version --client. - Get in. Read the bootstrap password, port-forward, and log in with the CLI (see the block below). Open
https://localhost:8080in a browser too — the resource tree is worth seeing at least once. - Apply the
Applicationbelow. Note there is noautomated:block yet — this is deliberately a manual sync, so you can see the two halves of GitOps separately. - Watch it sit at OutOfSync, then sync it by hand:
argocd app sync guestbook. Watch the resource tree fill in.
argocd app get guestbook prints Sync Status: Synced and Health Status: Healthy, and kubectl -n guestbook get deploy lists guestbook-ui with 1/1 ready.# --- install the argocd CLI ---
brew install argocd # macOS / Linux with Homebrew
# ...or grab the release binary (Linux amd64):
curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
sudo install -m 555 argocd /usr/local/bin/argocd && rm argocd
argocd version --client
# --- get in ---
kubectl -n argocd get secret argocd-initial-admin-secret \
-o jsonpath='{.data.password}' | base64 -d ; echo
# newer CLIs have a shortcut for the same value:
argocd admin initial-password -n argocd
kubectl -n argocd port-forward svc/argocd-server 8080:443 # leave running
argocd login localhost:8080 --username admin --insecure # another terminal; paste that password# guestbook.yaml — apply with: kubectl apply -f guestbook.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: guestbook
namespace: argocd # Applications live beside the controller
spec:
project: default
source:
repoURL: https://github.com/argoproj/argocd-example-apps.git
targetRevision: HEAD
path: guestbook
destination:
server: https://kubernetes.default.svc # "in-cluster"
namespace: guestbook
syncPolicy:
syncOptions:
- CreateNamespace=true # Argo creates the destination namespace- Switch the app from “I sync when told” to “I enforce”:
argocd app set guestbook --sync-policy automated --auto-prune --self-heal(or patchspec.syncPolicy.automatedto{prune: true, selfHeal: true}— both shown below). - Prove self-heal. Drift the cluster by hand:
kubectl -n guestbook scale deploy/guestbook-ui --replicas=5. Immediately runkubectl -n guestbook get deploy guestbook-ui -wand watch the replica count fall back to 1 within seconds. Do it again with a deletion:kubectl -n guestbook delete svc guestbook-ui— it comes back. - Look at what Recon saw:
argocd app diff guestbookduring the drift window, andargocd app history guestbookafterwards. - Prove prune. Repoint the Application at your fork (
argocd app set guestbook --repo https://github.com/YOU/argocd-example-apps.git), then deleteguestbook/guestbook-ui-svc.yamlin your fork and push. Force a refresh withargocd app get guestbook --hard-refreshinstead of waiting out the 3-minute poll. - Now restore the file in Git (
git revertthe deletion and push), turn pruning off —argocd app set guestbook --auto-prune=false— and delete it again. This time the Service survives as an orphan and the app reportsOutOfSyncforever. That contrast is the whole point of the flag.
kubectl -n guestbook get deploy guestbook-ui -o jsonpath='{.spec.replicas}' prints 1 again without you touching it; and after deleting a manifest from Git, kubectl -n guestbook get svc no longer lists guestbook-ui while argocd app get guestbook stays Synced.# the CLI way
argocd app set guestbook --sync-policy automated --auto-prune --self-heal
# the declarative way — this is what you would actually commit
kubectl -n argocd patch application guestbook --type merge -p \
'{"spec":{"syncPolicy":{"automated":{"prune":true,"selfHeal":true}}}}'
# watch the loop work
kubectl -n guestbook scale deploy/guestbook-ui --replicas=5
kubectl -n guestbook get deploy guestbook-ui -w # 5 ... 5 ... 1
argocd app get guestbook --hard-refresh # skip the 3-minute poll- In your fork create
workloads/ordered/and commit the three manifests below: aPreSynchook Job, a ConfigMap annotatedsync-wave: "-1", and a Deployment in the default wave0. - Create an Application for that path with automated sync (
argocd app create ordered --repo <your fork> --path workloads/ordered --dest-server https://kubernetes.default.svc --dest-namespace ordered --sync-policy automated --sync-option CreateNamespace=true). - Watch the ordering happen live in one terminal:
kubectl -n ordered get pods -w. Thedb-migratepod must appear, run andCompletebefore the web pods are created. Hooks run outside the waves;PreSyncalways goes first. - Confirm the wave ordering too: the ConfigMap (wave
-1) is applied before the Deployment (wave0), and Argo waits for each wave to become Healthy before starting the next. - Bump the Deployment’s image tag in Git and push. The migration Job runs again on every sync — which is why real migrations must be idempotent. Then read the hook’s output:
argocd app get orderedshows the hook phase, andkubectl -n ordered logs job/db-migrateshows what it printed (delete thehook-delete-policyannotation first if you want the pod to stick around).
argocd app get ordered --show-operation lists db-migrate under a PreSync phase with status Succeeded; and — once you have removed the hook-delete-policy annotation so the pod survives its own success — kubectl -n ordered get pods --sort-by=.metadata.creationTimestamp prints the db-migrate pod above every web-* pod.# workloads/ordered/00-migrate.yaml — a hook, not a wave: PreSync runs before everything
apiVersion: batch/v1
kind: Job
metadata:
name: db-migrate
annotations:
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/hook-delete-policy: HookSucceeded # tidy up only if it worked
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: busybox:1.36
command: ["sh","-c","echo 'applying schema 42...'; sleep 10; echo migrated"]
---
# workloads/ordered/10-config.yaml — wave -1: lands before the Deployment
apiVersion: v1
kind: ConfigMap
metadata:
name: web-config
annotations:
argocd.argoproj.io/sync-wave: "-1"
data:
GREETING: "hello from wave -1"
---
# workloads/ordered/20-web.yaml — no annotation = wave 0, the default
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 2
selector: { matchLabels: { app: web } }
template:
metadata: { labels: { app: web } }
spec:
containers:
- name: web
image: ghcr.io/stefanprodan/podinfo:6.7.0
envFrom:
- configMapRef: { name: web-config }
resources:
requests:
cpu: 50m # the HPA in lab 4 needs a request to compute a % against- Give the cluster metrics so an HPA can act: install metrics-server and patch it for kind’s self-signed kubelet certs (block below). Confirm with
kubectl top nodes. - Autoscale the lab-3 Deployment:
kubectl -n ordered autoscale deploy/web --min=3 --max=6 --cpu-percent=80. The HPA immediately raisesspec.replicasto 3 (it enforcesminReplicasbefore it ever reads a metric). Check it withkubectl -n ordered get hpa web— theTARGETScolumn reads<unknown>until metrics-server has a sample, and stays<unknown>forever if the container has no CPU request, which is why lab 3’s manifest sets one. - Watch the fight. Git says
replicas: 2; the HPA says 3. WithselfHealon, Argo reverts to 2, the HPA pushes back to 3, forever. Observe it:kubectl -n ordered get deploy web -wandargocd app get orderedflapping betweenSyncedandOutOfSync. - Fix it properly. Add the
ignoreDifferencesblock below to the Application and theRespectIgnoreDifferences=truesync option — without that option the field is only hidden from the diff, and the next sync still overwrites it. - Note the alternative Sol would pick: delete the
replicas:line from the manifest in Git entirely, so the field has no desired value to fight over. Two valid answers; know both.
kubectl -n ordered get deploy web -o jsonpath='{.spec.replicas}' stays at the HPA’s number (3 or more) for a full minute while argocd app get ordered continues to report Synced, and argocd app diff ordered shows no replica difference.# metrics-server on kind needs one patch (self-signed kubelet certs)
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl -n kube-system patch deploy metrics-server --type=json \
-p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
kubectl -n kube-system rollout status deploy/metrics-server
kubectl top nodes# add to the Application spec — BOTH halves are required
spec:
ignoreDifferences:
- group: apps
kind: Deployment
name: web
jsonPointers:
- /spec/replicas # the HPA owns this field, not Git
syncPolicy:
automated: { prune: true, selfHeal: true }
syncOptions:
- RespectIgnoreDifferences=true # without this, sync still overwrites it- In your fork, put Application manifests (not workloads) in
apps/: one file per app, each pointing at a different path — reuseguestbookandorderedfrom labs 1–3, plus one new one. - Apply the single
rootApplication below. It points atapps/withdirectory.recurse: true, so its “workload” is other Applications. - Watch the fan-out:
kubectl -n argocd get applications -w. One apply produced N apps. In the UI, the root app’s resource tree is a list of Applications. - Prove the cascade. Delete one child file from
apps/in Git and push. Because the root hasprune: true, the childApplicationobject is deleted — and because it carries theresources-finalizer.argocd.argoproj.iofinalizer (a finalizer, not an annotation), its workloads are deleted with it rather than orphaned. - Re-add the file. The whole environment comes back from one commit. That round trip is the disaster-recovery story you should be able to tell out loud.
kubectl -n argocd get applications lists root plus every child, all Synced; removing one child file from Git makes that child disappear from the same command within one refresh, taking its namespace’s workloads with it.# root.yaml — the only thing you apply by hand, ever
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/YOU/platform-config.git
targetRevision: main
path: apps # a folder full of Application manifests
directory:
recurse: true
destination:
server: https://kubernetes.default.svc
namespace: argocd # children are created here
syncPolicy:
automated: { prune: true, selfHeal: true }
---
# apps/guestbook.yaml — a child, committed to Git like any other manifest
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: guestbook
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io # delete my workloads when I am deleted
spec:
project: default
source:
repoURL: https://github.com/YOU/platform-config.git
targetRevision: main
path: workloads/web
destination:
server: https://kubernetes.default.svc
namespace: guestbook
syncPolicy:
automated: { prune: true, selfHeal: true }
syncOptions: [ CreateNamespace=true ]- Apply the
paymentsAppProjectbelow. Read each list out loud as a sentence: this tenant may deploy only from this repo, only into namespaces matchingpayments-*, and may not create cluster-scoped resources at all. - Break rule one. Try to create an app in that project from a repo the project doesn’t allow:
argocd app create rogue --project payments --repo https://github.com/argoproj/argocd-example-apps.git --path guestbook --dest-server https://kubernetes.default.svc --dest-namespace payments-a. It is refused with a message naming the project. - Break rule two. Create a legal app but aim it at
--dest-namespace default. Refused again — the destination isn’t permitted. - Break rule three. Commit a
ClusterRoleinto the tenant’s path and let a permitted app sync it. The sync fails with a “resource … is not permitted in project” condition — checkargocd app get <app>andkubectl -n argocd get application <app> -o jsonpath='{.status.conditions}'. - Fix each violation the way a platform engineer would: widen the project deliberately (and reviewably, in Git) or move the resource somewhere it belongs. Never by dropping the project.
payments project, and kubectl -n argocd get appproject payments -o yaml shows the source, destination and resource restrictions you wrote.apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: payments
namespace: argocd
spec:
description: What the payments team is allowed to deploy
sourceRepos:
- https://github.com/YOU/platform-config.git # this repo only
destinations:
- server: https://kubernetes.default.svc
namespace: 'payments-*' # these namespaces only
clusterResourceWhitelist: [] # no cluster-scoped resources at all
namespaceResourceBlacklist:
- group: ''
kind: ResourceQuota # tenants can't raise their own quota
- group: ''
kind: LimitRange
roles:
- name: read-only
description: Tenant devs may look, and sync, but not delete
policies:
- p, proj:payments:read-only, applications, get, payments/*, allow
- p, proj:payments:read-only, applications, sync, payments/*, allow- Part A — git directory generator. In your fork, put a tiny Deployment in
tenants/payments/and another intenants/search/. Apply the firstApplicationSetbelow. Two Applications appear, each with its own namespace, from one manifest. - Now do the thing the whole lab exists to prove:
mkdir tenants/checkout, drop a manifest in, commit, push. WithinrequeueAfterSecondsa third Application and namespace exist — onboarding a team is a folder. Watch withkubectl -n argocd get applications -w. - Delete the
tenants/search/folder and confirm the Application is removed. (ApplicationSet deletes generated apps by default;preserveResourcesOnDeletionchanges that.) - Part B — cluster generator (+5 min). Build a second cluster:
kind create cluster --name edge. Register it with Argo CD using the internal kubeconfig — kind’s normal kubeconfig points at127.0.0.1, which is unreachable from inside the first cluster. That gotcha is in the block below. - Apply the cluster-generator
ApplicationSet. One manifest, one app per registered cluster (in-clusterplusedge). Confirm withargocd cluster listandkubectl -n argocd get applications, then check the workload really landed on the second cluster:kubectl --context kind-edge get deploy -A.
tenants/checkout/ to Git makes a matching Application and namespace appear with no other action; and kubectl -n argocd get applications lists one agent-* app per entry in argocd cluster list, with the workload visible under kubectl --context kind-edge get pods -A.# Part A — one Application per folder under tenants/
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: tenants
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- git:
repoURL: https://github.com/YOU/platform-config.git
revision: main
requeueAfterSeconds: 60 # how fast a new folder is noticed
directories:
- path: tenants/*
template:
metadata:
name: '{{.path.basename}}' # "payments", "search", "checkout"
spec:
project: default
source:
repoURL: https://github.com/YOU/platform-config.git
targetRevision: main
path: '{{.path.path}}'
destination:
server: https://kubernetes.default.svc
namespace: '{{.path.basename}}'
syncPolicy:
automated: { prune: true, selfHeal: true }
syncOptions: [ CreateNamespace=true ]
---
# Part B — one Application per registered cluster
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: monitoring-everywhere
namespace: argocd
spec:
goTemplate: true
generators:
- clusters: {} # empty selector = every cluster Argo knows
template:
metadata:
name: 'agent-{{.name}}' # agent-in-cluster, agent-edge
spec:
project: default
source:
repoURL: https://github.com/YOU/platform-config.git
targetRevision: main
path: workloads/web
destination:
server: '{{.server}}'
namespace: platform
syncPolicy:
automated: { prune: true, selfHeal: true }
syncOptions: [ CreateNamespace=true ]# registering a second kind cluster — the 127.0.0.1 trap kind create cluster --name edge kind get kubeconfig --name edge --internal > /tmp/edge.kubeconfig # docker-network address argocd cluster add kind-edge --kubeconfig /tmp/edge.kubeconfig --name edge argocd cluster list # in-cluster + edge, both with a recent "successful" status
- Create a read-only GitHub personal access token scoped to your fork (public-repo read is enough) and put it in a Secret:
kubectl -n argocd create secret generic gh-token --from-literal=token=YOUR_TOKEN. Never commit the token itself — see Secrets Management for how this is done properly. - Apply the pull-request
ApplicationSetbelow. Nothing happens yet: there are no open PRs. - Open a PR in your fork — change one line in
workloads/web/deployment.yaml(say the image tag or a replica count) on a branch and open it againstmain. - Within
requeueAfterSecondsan Application namedpreview-<number>appears, syncing that PR’shead_shainto its own namespace. Confirm withkubectl -n argocd get applications | grep previewandkubectl get ns | grep preview. Port-forward into it and see the change running. - Close the PR. The generator stops returning it, the Application is deleted, and
prunetakes the namespace with it. Zero cleanup tickets — that is the whole feature. - Optional: uncomment the
labels:list inside thegithub:block so only PRs carrying thepreviewlabel get an environment — on a busy repo you want that. Note where it lives:filters:only takes regexes (branchMatch,targetBranchMatch,titleMatch), label selection is a provider option.
preview-<n> Application within a minute, closing it removes both the Application and the namespace (kubectl get ns | grep preview returns nothing), and you never ran a single command in between.apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: previews
namespace: argocd
spec:
goTemplate: true
generators:
- pullRequest:
github:
owner: YOU
repo: platform-config
# labels: # step 6: uncomment to pick up only PRs
# - preview # carrying the "preview" label
tokenRef:
secretName: gh-token
key: token
requeueAfterSeconds: 60
filters: # filters take regexes on branch/title, not labels
- targetBranchMatch: main # only PRs opened against main
template:
metadata:
name: 'preview-{{.number}}'
spec:
project: default
source:
repoURL: https://github.com/YOU/platform-config.git
targetRevision: '{{.head_sha}}' # the PR's commit, not main
path: workloads/web
destination:
server: https://kubernetes.default.svc
namespace: 'preview-{{.number}}'
syncPolicy:
automated: { prune: true, selfHeal: true }
syncOptions: [ CreateNamespace=true ]Part 2 · The same job, the Flux way
Everything above is one product’s opinion. Flux solves the same problem with small composable controllers and no UI — and the exam tool list names both. These three labs deliberately reconcile a similar app so the differences stand out: no Application object, an explicit interval on everything, and image automation built in rather than bolted on.
- Install the CLI (
brew install fluxcd/tap/fluxorcurl -s https://fluxcd.io/install.sh | sudo bash), then check the cluster is suitable:flux check --pre. - Install the controllers:
flux install. (The production path isflux bootstrap github --owner=YOU --repository=platform-config --path=clusters/dev --personal, which commits Flux’s own manifests into your repo so Flux manages Flux. You need that in lab 11 — do it now if you like.) Confirm:kubectl -n flux-system get pods. - Create the source and the reconciler with two CLI calls, or apply the equivalent YAML below — read both, because the exam may ask for either.
- Verify:
flux get sources gitthenflux get kustomizations. Both must showReady: Truewith an applied revision likemaster@sha1:…. The workload:kubectl -n default get deploy podinfo. - Compare drift behaviour with lab 2. Scale it by hand:
kubectl -n default scale deploy/podinfo --replicas=5. Flux corrects it at the next interval, not in seconds — that is the design difference. Prove you can force it:flux reconcile kustomization podinfo --with-sourcebrings it back immediately.
flux get kustomizations shows podinfo as Ready=True with an applied revision, and a hand-scaled Deployment returns to its Git replica count after flux reconcile kustomization podinfo --with-source (or by itself within one interval).flux check --pre flux install flux create source git podinfo \ --url=https://github.com/stefanprodan/podinfo \ --branch=master --interval=1m flux create kustomization podinfo \ --source=GitRepository/podinfo --path="./kustomize" \ --target-namespace=default --prune=true --interval=1m --wait=true flux get sources git flux get kustomizations flux reconcile kustomization podinfo --with-source # don't wait for the interval
# the same thing declaratively — this is what you commit
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: podinfo
namespace: flux-system
spec:
interval: 1m # Flux puts the cadence on every object
url: https://github.com/stefanprodan/podinfo
ref:
branch: master
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: podinfo
namespace: flux-system
spec:
interval: 1m
sourceRef: { kind: GitRepository, name: podinfo }
path: ./kustomize
targetNamespace: default
prune: true # Flux's equivalent of Argo's prune
wait: true # block until resources report Ready- Apply the
HelmRepository+HelmReleasebelow. Flux’s helm-controller does the install and every upgrade — you never runhelm upgradeyourself again. - Confirm the release:
flux get helmreleases -AshowsReady=Truewith the resolved chart version, and the workload exists —kubectl -n default get deploy podinfo-helm. (If you happen to have thehelmCLI,helm -n default listshows the same release, owned by helm-controller. It is not a prerequisite for this lab.) - Drift it.
kubectl -n default set env deploy/podinfo-helm DRIFT=yes. WithdriftDetection.mode: enabled, the controller notices the live state no longer matches the rendered chart and runs a corrective upgrade within the interval. - Prove why it changed, not just that it did:
kubectl -n default describe helmrelease podinfo-helmand look for a drift-detected event and an upgrade;flux logs --kind=HelmRelease --name=podinfo-helmshows the same story. - Now carve out an exception — the same idea as lab 4. The
ignoreblock tells drift detection to leave/spec/replicasalone so an HPA can own it. Add it, scale by hand, and confirm that change now survives while your env-var change still does not.
kubectl -n default get deploy podinfo-helm -o jsonpath='{.spec.template.spec.containers[0].env}' no longer contains DRIFT after one interval, describe helmrelease shows the drift-triggered upgrade, and a hand-set replica count survives once it is in the ignore list.apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: podinfo
namespace: flux-system
spec:
interval: 10m
url: https://stefanprodan.github.io/podinfo
---
apiVersion: helm.toolkit.fluxcd.io/v2 # older Flux: v2beta2
kind: HelmRelease
metadata:
name: podinfo-helm
namespace: default
spec:
interval: 1m
chart:
spec:
chart: podinfo
version: '6.*'
sourceRef:
kind: HelmRepository
name: podinfo
namespace: flux-system
driftDetection:
mode: enabled # warn | enabled | disabled
ignore:
- paths: ["/spec/replicas"]
target: { kind: Deployment } # let the HPA own this field
values:
replicaCount: 2
ui:
message: "reconciled by Flux"- This lab needs Flux bootstrapped against your repo with write access, because the whole point is that Flux opens a commit. Export a GitHub token with
reposcope first —export GITHUB_TOKEN=<your-pat>, bootstrap reads it from the environment and fails without it. Then, if you skipped it in lab 9:flux bootstrap github --owner=YOU --repository=platform-config --path=clusters/dev --personal --components-extra=image-reflector-controller,image-automation-controller. - Commit a Deployment into
clusters/dev/whose image line carries the setter marker comment — the automation only edits lines it has been told to edit:image: ghcr.io/stefanprodan/podinfo:6.6.0 # {"$imagepolicy": "flux-system:podinfo"}
Name itpodinfo-autoand setmetadata.namespace: defaultexplicitly. Two reasons: lab 9’s Kustomization already owns a Deployment calledpodinfoindefaultand two reconcilers fighting over one object is a mess, and an explicit namespace means you know where to look for it. - Apply the three objects below:
ImageRepository(scan the registry),ImagePolicy(choose the newest tag in the 6.x range),ImageUpdateAutomation(rewrite Git and push). - Watch the scan find tags:
flux get image repository podinfothenflux get image policy podinfo. The policy prints the latest tag it selected. - Now
git pullyour repo. There is a commit you did not write, authored byfluxcdbot, bumping that image line. The Kustomization then reconciles it into the cluster — the loop closes with Git still the single source of truth. Confirm withkubectl -n default get deploy podinfo-auto -o jsonpath='{..image}'.
git log --oneline -3 in your repo shows a commit authored by the Flux bot that changed the image tag, and kubectl -n default get deploy podinfo-auto -o jsonpath='{..image}' prints the exact tag that flux get image policy podinfo reports as the latest.apiVersion: image.toolkit.fluxcd.io/v1beta2 # check `flux --version`: APIs move
kind: ImageRepository
metadata: { name: podinfo, namespace: flux-system }
spec:
image: ghcr.io/stefanprodan/podinfo
interval: 5m
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata: { name: podinfo, namespace: flux-system }
spec:
imageRepositoryRef: { name: podinfo }
policy:
semver:
range: 6.x # never jump a major by accident
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata: { name: podinfo, namespace: flux-system }
spec:
interval: 1m
sourceRef: { kind: GitRepository, name: flux-system }
git:
checkout:
ref: { branch: main }
commit:
author:
name: fluxcdbot
email: fluxcdbot@users.noreply.github.com
messageTemplate: "chore: bump {{range .Updated.Images}}{{println .}}{{end}}"
push: { branch: main }
update:
path: ./clusters/dev
strategy: Setters # edit only marked linesPart 3 · When the reconciler is unhappy
- Fault 1 — nothing renders. Point an Application at a folder that doesn’t exist:
argocd app set ordered --path workloads/nope. The app goesUnknownwith aComparisonError. Find the real message:argocd app get <app>, thenkubectl -n argocd logs deploy/argocd-repo-server --tail=50— the repo-server renders manifests, so path, Helm and Kustomize errors surface there, not in the controller. - Fault 2 — synced but sick. Put the path back (
argocd app set ordered --path workloads/ordered), then inworkloads/ordered/20-web.yamlset the image toghcr.io/stefanprodan/podinfo:9.9.9-nopeand push. The app reportsSynced(Git and cluster agree!) butDegraded. Diagnose down the stack:kubectl -n <ns> get pods→ImagePullBackOff, thenkubectl -n <ns> describe pod <pod>for the pull error. Internalise that Synced ≠ Healthy — it is the single most common misread on this domain. - Fault 3 — the hook fails. In lab 3’s Job, change the command to
sh -c "echo migrating; exit 1". The sync stops atPreSync; the Deployment is never updated; the app stays on the old version. Read the failed hook:argocd app get orderedshows the hook phaseFailed, andkubectl -n ordered logs job/db-migrateshows why (the pod survives becauseHookSucceededonly deletes on success — that is the flag doing you a favour). - Roll back like an adult.
argocd app history <app>thenargocd app rollback <app> <id>to get healthy fast — and then fix Git, because withselfHealon, a rollback that isn’t in Git is temporary by design. - Repeat one fault on the Flux side. Point the Kustomization’s
pathat a missing folder and diagnose withflux get all -A,kubectl -n flux-system describe kustomization podinfo, andflux logs --level=error --all-namespaces. Different words, same three questions: did it fetch, did it render, did it become healthy?
kubectl describe pod / the hook Job’s logs) — and then, with Git repaired and no manual kubectl apply, argocd app get ordered prints Sync Status: Synced and Health Status: Healthy, and flux get kustomizations shows Ready=True again.# the triage kit — worth memorising before the exam
argocd app get <app> # sync status + health + per-resource state
argocd app diff <app> # exactly which fields differ
argocd app history <app> ; argocd app rollback <app> <id>
argocd app sync <app> --dry-run
kubectl -n argocd logs deploy/argocd-repo-server --tail=50 # render errors
kubectl -n argocd logs statefulset/argocd-application-controller --tail=50
kubectl -n argocd get application <app> -o jsonpath='{.status.conditions}'
flux get all -A # every source, kustomization, helmrelease
flux logs --level=error --all-namespaces
flux reconcile kustomization <name> --with-source
kubectl -n flux-system describe kustomization <name>
kubectl -n <ns> get events --sort-by=.lastTimestamp | tail -20“Lab 8 is the one that changed my life, honestly. Before preview environments I’d ask the platform team for a staging slot, wait a day, and find someone else’s branch on it. Now I open a PR and my reviewer gets a URL. If you build only one thing from this page for your real team, build that one.”
What you’ll have built
☺ Like you’re 10: By the end you have a cluster that fixes itself, does chores in the right order, ignores the one drawer it shouldn’t touch, gives every visitor a guest room, and hands you a clear complaint when something is wrong.
Twelve labs in, your throwaway cluster is a genuine little GitOps control plane: an Argo CD install reconciling several apps from your own repo with prune and selfHeal enforcing desired state; ordered rollouts via sync waves and a PreSync migration hook; an ignoreDifferences carve-out so autoscaling and GitOps coexist; an App-of-Apps root that can rebuild the whole environment from one commit; an AppProject that fences a tenant to one repo, one namespace pattern and no cluster-scoped resources; ApplicationSets that onboard a team by folder, fan out across two clusters, and hand every pull request a disposable environment; a parallel Flux stack with GitRepository, Kustomization, a drift-detecting HelmRelease and image automation committing tag bumps back to Git; and — the part the exam actually rewards — the reflex to tell OutOfSync from Degraded from a failed hook in under a minute.
That covers essentially all of Domain 2 — GitOps & Continuous Delivery, 25% and touches Domains 1, 3 and 5 through projects, tenancy and fleet fan-out. Next, take the same cluster into the CI/CD & progressive delivery labs, put the whole thing in context with the full lab track, and pressure-test the knowledge with practice tasks and the GitOps practice set. If you can rewrite the Application, ApplicationSet and Kustomization manifests from memory — check yourself against Know It Cold — this domain is finished.
Comfortable? Stack these on top. (1) Combine a matrix generator — every tenant folder × every cluster — and watch four apps appear from one manifest. (2) Put Argo CD itself under Argo CD, so the reconciler manages its own manifests (then break it and learn why people keep a bootstrap script). (3) Add a PostSync hook that only runs on success, and a SyncFail hook that posts the failure somewhere. (4) Encrypt a Secret with SOPS and let Flux decrypt it on apply, or wire External Secrets so the repo holds only a reference. (5) Replace the Deployment in lab 3 with an Argo Rollouts canary so a bad tag bump aborts itself instead of reaching every pod.
Foxy: Lab 4 took me forty minutes. Argo kept resetting my replicas and I kept re-scaling. I nearly turned self-heal off.
Recon: BEEP. Working as designed. Git said two. You said three. I am not a negotiator.
Benny: And turning self-heal off would have “fixed” it the way removing the smoke alarm fixes burnt toast. The real fix is telling Recon which field he doesn’t own.
Sol: Or just… delete replicas from the manifest. No field, no fight. Least effort, correct answer. My favourite combination.
Gizmo: OR you skip the whole page and kubectl apply straight at prod like a free spirit. Nobody checks. 😈
Timmy: Recon checks, Gizmo. Every three minutes. Forever.
Dot: And if lab 12 taught me one thing: when it’s red, read the repo-server logs before you panic. Nine times out of ten it’s a typo in a path.
1. An app shows Synced but Degraded — what does that tell you, and which command do you run next? 2. Which two settings must you add together so Argo CD stops overwriting a field an HPA owns? 3. You added tenants/checkout/ to Git and no Application appeared. Name two plausible causes. 4. Where does a PreSync hook run relative to sync wave -1? 5. In Flux, what makes drift get corrected — and how is that different from Argo CD’s selfHeal? 6. What does the {"$imagepolicy": …} comment in a manifest do?
Check your answers
- Git and the cluster agree — the manifests were applied successfully — but the workload isn’t healthy, so the bug is in the workload, not the reconciler. Next:
kubectl -n <ns> get podsthenkubectl describe pod(orlogs --previous). Chasing the Argo logs here wastes minutes. spec.ignoreDifferencesfor/spec/replicasand theRespectIgnoreDifferences=truesync option. Without the second one the field is only hidden from the diff and the next sync still stomps it. The alternative correct answer: removereplicasfrom the manifest entirely.- Any two of: the generator hasn’t requeued yet (
requeueAfterSeconds/ the 3-minute refresh — force it with a hard refresh or a webhook); the folder contains no manifests, so the directory generator skipped it; you pushed to a branch other than therevisionthe generator watches; or the ApplicationSet controller is erroring — checkkubectl -n argocd logs deploy/argocd-applicationset-controller. - Before it. Hooks are a separate phase: all
PreSynchooks complete before any wave is applied, including negative waves. Waves order resources within the sync phase. - Flux re-applies the rendered manifests on every
interval(andHelmReleaseadds explicitdriftDetection), so drift disappears at the next tick — cadence is declared per object. Argo CD’sselfHealreacts to watch events on live resources, so it typically reverts within seconds. Same outcome, different trigger. - It is a setter marker: it tells the image-automation controller that this exact line is owned by the named
ImagePolicy, so the controller may rewrite the tag there and commit the change back to Git. No marker, no edit.