Practice — GitOps & Continuous Delivery

Eight performance-based tasks for the single biggest slice of the CNPE blueprint — GitOps & Continuous Delivery is 25% of the exam, roughly one task in four on your paper. Drill them the way the exam will feel: cold, with only the official docs open and no worked solution in sight; time-boxed to five to seven minutes each, with a hard stop when the clock runs out; and open the answer key only after you have genuinely attempted the task on a real cluster. Peeking first turns a drill into reading, and reading is exactly what doesn't get you through a hands-on exam. This is one domain lifted out of the full practice bank — the background reading lives in GitOps workflows, CI/CD & progressive delivery and release engineering.

☺ Like you’re 10: These tasks are all about writing what you want in a Git folder and getting a robot in the cluster to make it real — and then rolling out a new version slowly enough to catch a mistake.

The biggest domain, and the most mechanical — which is good news, because mechanical tasks are the ones you can drill to reflex. Expect to create Argo CD Application resources, restructure manifests with Kustomize, build a pipeline, and configure a progressive rollout. Keep the command reference beside you for the CLI shapes, and when a drill goes sideways, the troubleshooting playbook is faster than guessing.

Reconciling desired state — Argo CD, Kustomize and Flux

The four foundational drills. Each one is “make a controller inside the cluster hold reality to what Git says,” from a single app, through a properly structured repo, to many tenants at once, and finally the same job done with the other major reconciler.

G1 · Make an app self-heal against drift

Your platform team has just adopted Argo CD, and the payments team’s manifests already live in the platform config repo at apps/payments/overlays/prod. Right now someone still deploys them by hand, and last week an engineer scaled the deployment during an incident and never scaled it back — nobody noticed for four days.

Your task:

1. Create an Argo CD Application named payments in the argocd namespace pointing at branch main, path apps/payments/overlays/prod of the config repo, targeting namespace payments on the in-cluster destination.
2. Enable automated sync with both prune and selfHeal, and have Argo CD create the destination namespace.
3. Prove drift is corrected: scale the deployment by hand to 7 replicas and show Argo CD reverts it.

Done when: kubectl -n argocd get app payments -o jsonpath='{.status.sync.status}{" "}{.status.health.status}' prints Synced Healthy, and after a manual kubectl scale, kubectl -n payments get deploy api -o jsonpath='{.spec.replicas}' returns to the value in Git within a minute.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/acme/platform-config.git
    targetRevision: main
    path: apps/payments/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: payments
  syncPolicy:
    automated:
      prune: true       # delete live resources removed from Git
      selfHeal: true    # revert out-of-band drift back to Git
    syncOptions:
      - CreateNamespace=true

kubectl apply -f payments-app.yaml
argocd app wait payments --health --timeout 120     # or watch the UI

# prove self-heal
kubectl -n payments scale deploy/api --replicas=7
sleep 60
kubectl -n payments get deploy api -o jsonpath='{.spec.replicas}'   # back to Git's value

G2 · Convert a flat Deployment into a Kustomize base and overlays

The search team keeps three near-identical copies of their manifests in manifests/dev, manifests/staging and manifests/prod. The three have already drifted — prod is two image tags behind and nobody spotted it, because diffing three whole directories by eye is hopeless.

Your task:

1. Create base/ containing the shared deployment.yaml, service.yaml and a kustomization.yaml.
2. Create overlays/prod/ that sets the namespace to search-prod, raises replicas to 5, and pins the image to ghcr.io/acme/search:2.3.1 — using Kustomize fields, not a copied Deployment.
3. Render and apply the prod overlay.

Done when: kubectl kustomize overlays/prod emits a Deployment with 5 replicas and the pinned tag, and kubectl -n search-prod get deploy search -o jsonpath='{.spec.template.spec.containers[0].image}' shows ghcr.io/acme/search:2.3.1.

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
labels:
  - pairs:
      app.kubernetes.io/name: search
    includeSelectors: true
---
# overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: search-prod
resources:
  - ../../base
replicas:
  - name: search
    count: 5
images:
  - name: ghcr.io/acme/search
    newTag: "2.3.1"
patches:
  - target: { kind: Deployment, name: search }
    patch: |-
      - op: add
        path: /spec/template/spec/containers/0/resources
        value: { requests: { cpu: 200m, memory: 256Mi }, limits: { memory: 512Mi } }

kubectl kustomize overlays/prod            # render only — always check before applying
kubectl apply -k overlays/prod
kubectl -n search-prod get deploy search -o jsonpath='{.spec.template.spec.containers[0].image}'

G3 · Fan one app across many namespaces with an ApplicationSet

Four tenant teams each need the same shared logging-agent deployed into their own namespace, with a per-tenant log level. Hand-writing four Application objects works today and breaks the moment a fifth team onboards.

Your task:

1. Create an ApplicationSet named logging-agent using a list generator with entries for tenants payments, search, checkout and identity, each carrying a logLevel.
2. Template one Argo CD Application per tenant, deploying into {{tenant}}, with automated sync and namespace creation.
3. Pass the per-tenant log level through as a Helm parameter (or Kustomize var).

Done when: kubectl -n argocd get applications lists four generated apps named logging-agent-<tenant>, all Synced, and each tenant namespace has the agent running.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: logging-agent
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - list:
        elements:
          - tenant: payments
            logLevel: info
          - tenant: search
            logLevel: debug
          - tenant: checkout
            logLevel: info
          - tenant: identity
            logLevel: warn
  template:
    metadata:
      name: 'logging-agent-{{.tenant}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/acme/platform-config.git
        targetRevision: main
        path: charts/logging-agent
        helm:
          parameters:
            - name: logLevel
              value: '{{.logLevel}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.tenant}}'
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [ CreateNamespace=true ]

G4 · Reconcile the same app with Flux

A second cluster in your estate was set up by a different team and runs Flux rather than Argo CD. You need the checkout app reconciled there from the same config repo, with a ten-minute interval and pruning enabled — and you need to be able to force an immediate reconcile during an incident.

Your task:

1. Create a GitRepository source named platform-config in flux-system tracking branch main, polled every minute.
2. Create a Kustomization named checkout reconciling ./apps/checkout/overlays/prod with prune: true and wait: true.
3. Force an out-of-band reconcile and confirm it succeeded.

Done when: flux get kustomizations checkout shows Ready=True with the applied revision, and flux reconcile kustomization checkout --with-source completes without error.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: platform-config
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/acme/platform-config.git
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: checkout
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: platform-config
  path: ./apps/checkout/overlays/prod
  targetNamespace: checkout
  prune: true
  wait: true
  timeout: 3m

# the CLI equivalents — much faster under exam time pressure
flux create source git platform-config \
  --url=https://github.com/acme/platform-config.git --branch=main --interval=1m
flux create kustomization checkout \
  --source=GitRepository/platform-config --path=./apps/checkout/overlays/prod \
  --prune=true --wait=true --interval=10m --target-namespace=checkout

flux reconcile kustomization checkout --with-source
flux get kustomizations checkout

Building the pipeline — Tekton and Argo Workflows

GitOps says what should be running; something still has to build the artefact and run the housekeeping. These two drills cover the Kubernetes-native pipeline engines the exam expects you to author by hand, not just invoke.

G5 · Build a Tekton Task and Pipeline, then run it

The platform’s golden-path pipeline has to build a container image from a Git repo and push it to the internal registry. There is no pipeline yet; you have Tekton Pipelines installed and a service account with registry credentials attached.

Your task:

1. Write a Tekton Task named build-push that takes an image parameter and a source workspace, and builds/pushes with Buildah (or Kaniko).
2. Write a Pipeline named app-ci that runs git-clone then build-push, passing the workspace between them.
3. Start it with tkn and confirm the PipelineRun succeeds.

Done when: tkn pipelinerun list shows the run Succeeded, and tkn pipelinerun logs -f shows the push step completing.

apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: build-push
spec:
  params:
    - name: image
      type: string
  workspaces:
    - name: source
  steps:
    - name: build-and-push
      image: quay.io/buildah/stable:latest
      workingDir: $(workspaces.source.path)
      securityContext:
        capabilities:
          add: ["SETFCAP"]
      script: |
        #!/usr/bin/env bash
        set -euo pipefail
        buildah bud --storage-driver=vfs -t "$(params.image)" .
        buildah push --storage-driver=vfs "$(params.image)"
---
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: app-ci
spec:
  params:
    - name: repo-url
    - name: image
  workspaces:
    - name: shared
  tasks:
    - name: fetch
      taskRef: { name: git-clone }          # from Tekton Hub / cluster catalog
      params:
        - name: url
          value: $(params.repo-url)
      workspaces:
        - name: output
          workspace: shared
    - name: build
      runAfter: ["fetch"]
      taskRef: { name: build-push }
      params:
        - name: image
          value: $(params.image)
      workspaces:
        - name: source
          workspace: shared

kubectl apply -f task.yaml -f pipeline.yaml
tkn pipeline start app-ci \
  --param repo-url=https://github.com/acme/checkout.git \
  --param image=registry.internal/acme/checkout:1.0.0 \
  --workspace name=shared,volumeClaimTemplateFile=pvc.yaml \
  --serviceaccount build-bot --use-param-defaults --showlog

G6 · Model a fan-out/fan-in job as an Argo Workflow DAG

Nightly platform housekeeping currently runs as one long shell script in a CronJob: it collects inventory from three sources, then produces a single report once all three finish. When one source is slow, everything is slow, and when one fails you can’t tell which.

Your task:

1. Write an Argo Workflow using a dag template with three parallel collection steps and one report step that depends on all three.
2. Pass a parameter into each collection step so they share one template.
3. Submit it and watch it to completion.

Done when: argo get @latest shows the three collect nodes running in parallel and report only after all three, with phase Succeeded.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: inventory-
spec:
  entrypoint: nightly
  templates:
    - name: nightly
      dag:
        tasks:
          - name: collect-clusters
            template: collect
            arguments: { parameters: [{ name: source, value: clusters }] }
          - name: collect-images
            template: collect
            arguments: { parameters: [{ name: source, value: images }] }
          - name: collect-costs
            template: collect
            arguments: { parameters: [{ name: source, value: costs }] }
          - name: report
            template: report
            dependencies: [collect-clusters, collect-images, collect-costs]
    - name: collect
      inputs:
        parameters: [{ name: source }]
      container:
        image: alpine:3.20
        command: [sh, -c]
        args: ["echo collecting {{inputs.parameters.source}}; sleep 5"]
    - name: report
      container:
        image: alpine:3.20
        command: [sh, -c]
        args: ["echo all sources collected - writing report"]

argo submit -n argo workflow.yaml --watch
argo get -n argo @latest
argo logs -n argo @latest

Progressive delivery — canaries that abort themselves

The last two drills are the highest-value ones in the domain, because they are where candidates most often lose marks: knowing the manifest shape is not enough, you also have to know which command promotes, which aborts, and what each leaves behind. Read CI/CD & progressive delivery first if the vocabulary is shaky.

G7 · Turn a Deployment into an analysed Argo Rollouts canary

Checkout is your highest-risk service and currently deploys with a plain rolling update — a bad image reaches 100% of users in about forty seconds. The team wants a canary that shifts traffic in stages and automatically aborts if the error rate rises.

Your task:

1. Convert the checkout Deployment into an Argo Rollout with a canary strategy: 20% → pause 2m → 50% → pause → 100%.
2. Add an AnalysisTemplate that queries Prometheus for success rate and fails below 95%, and wire it into the canary steps.
3. Trigger an update, promote through a pause, then trigger a second update and abort it.

Done when: kubectl argo rollouts get rollout checkout --watch shows the canary paused at 20%; after promote it advances; and after abort the rollout reports Degraded with all traffic back on stable.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
    - name: service
  metrics:
    - name: success-rate
      interval: 30s
      count: 4
      successCondition: result[0] >= 0.95
      failureLimit: 1
      provider:
        prometheus:
          address: http://prometheus-operated.monitoring:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service}}",code!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{service="{{args.service}}"}[2m]))
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
spec:
  replicas: 6
  selector:
    matchLabels: { app: checkout }
  template:                      # identical to your Deployment's pod template
    metadata:
      labels: { app: checkout }
    spec:
      containers:
        - name: app
          image: ghcr.io/acme/checkout:2.0.0
  strategy:
    canary:
      canaryService: checkout-canary
      stableService: checkout-stable
      analysis:
        templates: [{ templateName: success-rate }]
        args: [{ name: service, value: checkout }]
      steps:
        - setWeight: 20
        - pause: { duration: 2m }
        - setWeight: 50
        - pause: {}              # empty pause = wait for a human promote
        - setWeight: 100

kubectl argo rollouts get rollout checkout --watch
kubectl argo rollouts set image checkout app=ghcr.io/acme/checkout:2.1.0
kubectl argo rollouts promote checkout           # advance past an indefinite pause
kubectl argo rollouts promote checkout --full    # skip all remaining steps + analysis
kubectl argo rollouts abort checkout             # send 100% back to stable
kubectl argo rollouts undo checkout --to-revision=2

G8 · Configure a Flagger canary with automatic rollback

A different team already runs Linkerd (or Istio) and would rather not adopt a second rollout controller. Flagger is installed. They want the same protection as G7 but driven from their existing Deployment, with automatic promotion when metrics stay healthy.

Your task:

1. Create a Flagger Canary targeting the frontend Deployment and its Service on port 8080.
2. Step traffic 10% at a time up to 50%, analysing every 30 seconds with a request-success-rate threshold of 99% and a p99 latency threshold of 500 ms.
3. Trigger a rollout and watch Flagger promote it — then push a deliberately broken image and watch it roll back.

Done when: kubectl -n prod get canary frontend shows Succeeded in the STATUS column after a good release, and Failed with traffic returned to primary after the bad one; kubectl -n prod describe canary frontend shows the halt reason in events.

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: frontend
  namespace: prod
spec:
  provider: linkerd
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: frontend
  service:
    port: 8080
    targetPort: 8080
  analysis:
    interval: 30s
    threshold: 5          # consecutive failed checks before rollback
    maxWeight: 50
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange: { min: 99 }
        interval: 1m
      - name: request-duration
        thresholdRange: { max: 500 }
        interval: 1m
    webhooks:
      - name: load-test
        url: http://flagger-loadtester.test/
        timeout: 5s
        metadata:
          cmd: "hey -z 1m -q 10 -c 2 http://frontend-canary.prod:8080/"

kubectl -n prod set image deploy/frontend app=ghcr.io/acme/frontend:3.1.0
kubectl -n prod get canary frontend -w
kubectl -n prod describe canary frontend | tail -20   # events show each weight step

That’s the GitOps domain drilled. Anything that scored badly, re-do cold on a clean cluster before you move on — then head back to the full practice bank for the other four domains and the 120-minute mock. For the theory underneath these drills, see GitOps workflows and release engineering; for the CLI flags you fumbled, the command reference; and when a drill breaks in a way you don’t recognise, work it through the troubleshooting playbook or delivery triage rather than reaching straight for the answer key.