Exam Prep · Mock Exam · Set 2

Mock Exam · Set 2 — a fresh 120-minute sitting

This is a whole exam, not a drill. Eighteen performance-based tasks, weighted exactly like the CNPE blueprint, designed to be sat in one unbroken 120-minute block on a throwaway cluster with only the permitted documentation open. Every task here is new — none of them repeats a task from the five practice banks, so if you have already worked those, this is a genuine second look at your readiness rather than a memory test. Each task gives you a realistic platform situation, a short numbered brief, an objective done when check you can run yourself, and a point value; the worked solutions are folded away behind a summary so you can sit the paper honestly and mark it afterwards. Total the sheet at the end, compare it against the 64% bar, and let the misses — not the number — tell you what to study next.

☺ Explain it like I’m 10

Imagine you have been practising football drills for weeks: passing in one corner, shooting in another, defending in a third. Today is different — today is a real match. Two halves, a proper referee, a scoreboard, and no stopping to look things up in your notebook. You will be worse at it than you were at the drills, and that is exactly the point: a match tells you which parts of your game fall apart when the clock is running and nobody is helping you. Play it properly, then look at the scoresheet afterwards and see which corner of the pitch you need to go back to.

🐢🦫Your hosts for this topic: Timmy the Turtle & Benny the Beaver — Timmy enforces exam conditions (one timer, no pausing, only the docs you are actually allowed), and Benny checks your work actually runs: nothing scores here until the done-when command says so on a real cluster.

Before you start — exam conditions

☺ Like you’re 10: A practice match only helps if you play it like a real one — same rules, same clock, no help from the sidelines.

The most common way to waste a mock is to sit it comfortably: a pause here, a quick search there, a peek at a solution when a manifest will not validate. The result is a score that flatters you and teaches nothing. Set the conditions first, then start the clock.

One — build the cluster the night before. A throwaway kind or minikube cluster is enough, with Argo CD, Argo Rollouts, Flux, Tekton Pipelines, the kube-prometheus-stack, an OpenTelemetry Collector, KEDA, Crossplane with the patch-and-transform function, and Kyverno installed. If a component is missing when you reach its task, skip the task and move on — never stop the clock to install things.

Two — one timer, started once. 120 minutes, no pausing for coffee, a broken cluster or a stubborn CRD. If you lose ten minutes to your own laptop, that is data: the real exam will also eat time in ways you did not plan for. Sit it at roughly the hour your real exam is booked.

Three — only the permitted documentation, open before you start. The allowlist is narrower than most candidates expect: kubernetes.io/docs, kubernetes.io/blog, any task-specific documentation the exam links from its Quick Reference box, and the local docs on the exam desktop — man pages and whatever ships under /usr/share. That is the whole list. Argo CD, Flux, Tekton, Crossplane, Backstage, Prometheus, OpenTelemetry, Kyverno, Gatekeeper, Istio and Helm documentation are not permitted, and neither is this site, your notes, a search engine or an AI assistant.

Four — no copy-paste from the solutions, and no AI assistance of any kind. If you get stuck, spend your budget, fail the task, and read the solution after the sitting. A task you pasted your way through is a task you will fail again on exam day.

⚠ The allowlist is narrow — and everything here can change

That list changes how you must prepare: for the tools whose own documentation is not allowed, you need the resource shapes in your head or reachable from what to know cold — you cannot look up an Application, a HelmRelease, a ScaledObject or a ClusterPolicy mid-exam. What you can always do is interrogate the cluster: kubectl explain <kind> --recursive, kubectl api-resources and kubectl get <kind> <existing> -o yaml are not documentation, they are the live API, and they are your real reference. Separately: exam specifics move. Task counts, the 120-minute duration, the 64% pass mark, domain weights, the permitted-resource list and the Kubernetes version on the desktop have all changed before and will change again. Everything here is a study aid written at a point in time — the official CNCF certification pages and Linux Foundation training site are the only authority. Confirm before you book.

Five — read the whole paper first, then choose your order. Five minutes spent reading all eighteen tasks is the highest-return five minutes of the sitting. You will spot the three you can finish in four minutes each, and you will spot the one that is going to eat twelve. Bank the cheap points first. Sequential order is a trap in a weighted, partial-credit exam.

Your time budget

☺ Like you’re 10: Give every question a small share of the two hours, and when a question runs over its share, put a sticky note on it and go to the next one. You can always come back.

Eighteen tasks in 120 minutes is roughly six and a half minutes each — but the tasks are not equal. Budget deliberately: five minutes to read everything, about 108 minutes of task work, and seven minutes at the end to re-run your done-when checks. That last block matters more than it sounds; half-finished work you never verified is where points quietly die.

Block	Tasks	Budget	Running total	Note
Read the whole paper	—	5 min	5	Mark each task cheap / medium / expensive before you touch a terminal.
GitOps & Continuous Delivery	M1–M5	6 + 5 + 6 + 7 + 7 = 31 min	36	Biggest domain, most mechanical. M4 and M5 are the expensive pair.
Platform APIs & Self-Service	M6–M10	6 + 5 + 6 + 7 + 5 = 29 min	65	M10 is the cheapest task on the paper — do it early.
Observability & Operations	M11–M13	6 + 7 + 8 = 21 min	86	M13 is triage; it either resolves fast or swallows your slack.
Platform Architecture & Infrastructure	M14–M16	5 + 5 + 6 = 16 min	102	Three self-contained tasks. Good points-per-minute.
Security & Policy Enforcement	M17–M18	5 + 6 = 11 min	113	Two tasks worth 15 points between them — never skip these.
Verify & sweep flagged tasks	—	7 min	120	Re-run every done-when. Finish the nearly-done before starting anything new.

The flag-and-move rule: when a task passes roughly six minutes without a working done-when check, stop. Write one line about where you stalled, leave whatever you applied in place (partial credit is real), and go to the next task. The instinct to finish what you started is the most expensive habit in a timed performance exam — it trades a guaranteed five points elsewhere for a possible five points here. Timmy’s version: you are not paid to finish tasks, you are paid to bank points.

◆ Key idea

Points, not tasks, are the unit of this exam. With a 64% bar you can fail four or five tasks outright and still pass comfortably — but only if you actually attempted the cheap ones. The candidates who fail are almost never the ones who did not know enough; they are the ones who spent twenty-two minutes on a Crossplane composition and never opened the two security tasks worth fifteen points at the bottom of the paper.

GitOps & Continuous Delivery — M1 to M5 (25 points)

☺ Like you’re 10: These five are all about the robot that reads your instructions from a folder and keeps the cluster matching them — putting the steps in the right order, telling it what to ignore, and swapping in new versions carefully.

The biggest slice of the blueprint, and the one where speed pays most. Background reading lives in GitOps workflows, CI/CD & progressive delivery and release engineering — but read those after the sitting, not during it.

M1 · Order a release with sync waves and a PreSync hook

The ledger team’s deployment keeps failing on rollout: the new application version expects a database column that a migration job creates, and Argo CD applies the Deployment and the migration Job at the same time, so the pods crash-loop until the migration happens to finish. They also want the schema migration to run before anything else touches the database, and the migration Job object cleaned up when it succeeds.

Your task:

Turn the existing db-migrate Job into an Argo CD PreSync hook that is deleted once it succeeds.
Use sync waves so the ConfigMap and Secret land first (wave -1), then the Deployment (wave 0), then the Ingress (wave 1).
Sync the application and show the ordering actually happened.

Done when: kubectl -n ledger get jobs shows no leftover db-migrate Job after a successful sync, and kubectl -n argocd get app ledger -o jsonpath='{.status.operationState.syncResult.resources[*].hookPhase}' plus argocd app history ledger show the sync completed with the hook succeeded before the wave-0 resources. (5 points)

Show the worked solution

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  namespace: ledger
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: ghcr.io/acme/ledger-migrate:2.4.0
          command: ["/bin/migrate", "up"]
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ledger-config
  namespace: ledger
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
data:
  LOG_LEVEL: info
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ledger
  namespace: ledger
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec: {}   # unchanged
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ledger
  namespace: ledger
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec: {}   # unchanged

argocd app sync ledger
argocd app history ledger
kubectl -n ledger get jobs                 # db-migrate gone after success
kubectl -n argocd get app ledger -o jsonpath='{.status.operationState.phase}'

Why: hooks and waves solve different halves of the same problem. A hook runs at a phase of the sync (PreSync, Sync, PostSync, SyncFail) and is not part of the desired state at all, which is why it needs a hook-delete-policy or it accumulates. Waves order resources within a phase — Argo CD applies all resources in the lowest wave, waits for them to become healthy, then moves to the next. Unannotated resources default to wave 0, which is why partial annotation still works.

M2 · Stop a perpetual OutOfSync caused by an HPA

The search application has been showing OutOfSync in the Argo CD UI for a week, and self-heal keeps fighting the HorizontalPodAutoscaler: Git says replicas: 2, the HPA scales the Deployment to 6, Argo CD reverts it to 2, the HPA scales it back up. The on-call engineer has started ignoring Argo CD’s sync status entirely — which is worse than the original problem.

Your task:

Configure the search Application so Argo CD ignores differences in spec.replicas on the search Deployment only.
Make sure automated self-heal also respects that exclusion, not just the diff view.
Show the application returns to Synced while the HPA still holds the Deployment above the Git value.

Done when: kubectl -n argocd get app search -o jsonpath='{.status.sync.status}' prints Synced while kubectl -n search get deploy search -o jsonpath='{.spec.replicas}' is greater than the value committed in Git, and stays there for two minutes. (5 points)

Show the worked solution

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: search
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/acme/platform-config.git
    targetRevision: main
    path: apps/search/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: search
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: search
      namespace: search
      jsonPointers:
        - /spec/replicas
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - RespectIgnoreDifferences=true   # without this, self-heal still reverts

kubectl apply -f search-app.yaml
kubectl -n search get hpa search
kubectl -n argocd get app search -o jsonpath='{.status.sync.status}'
sleep 120 && kubectl -n search get deploy search -o jsonpath='{.spec.replicas}'

Why: the cleanest long-term fix is to remove replicas from the manifest entirely so nothing in Git claims ownership of it — but when the field must stay (a shared base, a chart default), ignoreDifferences is the escape hatch. The trap the exam loves is that ignoreDifferences alone only silences the diff; automated self-heal computes its own target and will still revert the field until you add RespectIgnoreDifferences=true. Scope the rule to one kind and name — a group-wide ignore hides real drift. More context in scaling & scheduling.

M3 · Deliver a Helm chart through Flux with drift detection

A platform component currently gets installed with helm upgrade --install from a laptop whenever someone remembers. Values have drifted from what is in the repo, nobody can say which version is deployed, and the last engineer who ran it has left. You are moving it onto Flux.

Your task:

Create a HelmRepository source for the chart repo with a 30-minute interval.
Create a HelmRelease that installs the chart into namespace platform, pins a chart version range, sets at least one value, and retries on failure.
Enable drift detection so a hand-edited Deployment is corrected, and prove the reconciliation works on demand.

Done when: flux get helmreleases -n platform shows Ready=True with the resolved chart version, and after kubectl -n platform scale deploy/<name> --replicas=9, flux reconcile helmrelease <name> -n platform restores the replica count from the chart values. (5 points)

Show the worked solution

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: podinfo
  namespace: platform
spec:
  interval: 30m
  url: https://stefanprodan.github.io/podinfo
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
  namespace: platform
spec:
  interval: 10m
  chart:
    spec:
      chart: podinfo
      version: "6.5.x"          # a range, so patch bumps flow automatically
      sourceRef:
        kind: HelmRepository
        name: podinfo
        namespace: platform
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
      remediateLastFailure: true
  driftDetection:
    mode: enabled               # correct out-of-band edits, don't just warn
  values:
    replicaCount: 2
    resources:
      requests:
        cpu: 50m
        memory: 64Mi

kubectl apply -f podinfo-helmrelease.yaml
flux get sources helm -n platform
flux get helmreleases -n platform

# prove drift detection
kubectl -n platform scale deploy/podinfo --replicas=9
flux reconcile helmrelease podinfo -n platform
kubectl -n platform get deploy podinfo -o jsonpath='{.spec.replicas}'   # back to 2

Why: the HelmRelease makes the Helm release itself a declared, reconciled object rather than the side effect of a command someone ran — so “which version is deployed?” is answered by kubectl get helmrelease, not by archaeology. driftDetection.mode: enabled is the part people forget: by default Flux only re-runs Helm when the desired state changes, so a hand-edited Deployment survives until the next upgrade. Remediation converts a bad upgrade from an outage into an automatic rollback. See configuration management.

M4 · Pass data between Tekton tasks with results and a shared workspace

The build pipeline currently does everything in one giant Task, so a failing test forces a full re-clone and re-build, and the image digest it produces is only visible by scraping the logs. The team wants the clone, build and deploy steps split, with the digest passed to the deploy step as real data.

Your task:

Write a Task build that consumes a workspace named source and emits a result named digest.
Write a Task deploy that takes a digest param and prints it.
Wire both into a Pipeline where deploy consumes build’s result, sharing one workspace backed by a volumeClaimTemplate in the PipelineRun.

Done when: tkn pipelinerun logs -f <run> shows the deploy step printing the exact digest the build step wrote, and kubectl get taskrun -o jsonpath='{.items[*].status.results[*].value}' (or tkn taskrun describe) shows the result recorded on the build TaskRun. (5 points)

Show the worked solution

apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: build
spec:
  workspaces:
    - name: source
  results:
    - name: digest
      description: The digest of the image we just built
  steps:
    - name: build
      image: alpine:3.20
      script: |
        #!/bin/sh
        set -eu
        echo "building from $(workspaces.source.path)"
        # backticks, not $( ) — Tekton owns the $( ) syntax inside a script
        DIGEST=`date +%s | sha256sum | cut -c1-64`
        echo -n "sha256:$DIGEST" > $(results.digest.path)
---
apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: deploy
spec:
  params:
    - name: digest
      type: string
  steps:
    - name: deploy
      image: alpine:3.20
      script: |
        #!/bin/sh
        echo "deploying image with digest $(params.digest)"
---
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: build-and-deploy
spec:
  workspaces:
    - name: shared
  tasks:
    - name: build
      taskRef:
        name: build
      workspaces:
        - name: source
          workspace: shared
    - name: deploy
      runAfter: [build]
      taskRef:
        name: deploy
      params:
        - name: digest
          value: $(tasks.build.results.digest)

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: build-and-deploy-
spec:
  pipelineRef:
    name: build-and-deploy
  workspaces:
    - name: shared
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 1Gi

Why: workspaces and results are the two ways Tekton moves things between Tasks, and they are not interchangeable. A workspace is a shared volume for bulk data — source trees, caches, build output. A result is a small string (kept in the TaskRun status, so keep it tiny) that becomes a first-class value another Task can take as a param, which also creates the dependency edge automatically. Writing a result is just writing to $(results.<name>.path) with no trailing newline — echo -n matters, because a stray newline breaks downstream comparisons. One trap in the script block: Tekton substitutes $( ) itself, so use backticks for real shell command substitution or your command text is left uninterpreted.

M5 · Blue-green a service with a preview check and a manual promotion

The checkout service cannot be canaried: a schema quirk means two versions must not serve production traffic simultaneously. The team wants the new version fully deployed and testable on a separate hostname first, then promoted all at once by a human, with the old ReplicaSet kept warm long enough to roll back instantly.

Your task:

Convert the checkout Deployment into an Argo Rollouts Rollout using the blueGreen strategy with an active Service and a preview Service.
Disable auto-promotion and keep the old ReplicaSet for at least 60 seconds after promotion.
Trigger a new revision, verify the preview Service points at the new pods while the active Service still points at the old ones, then promote.

Done when: after triggering a new image, kubectl -n checkout get svc checkout-preview checkout-active -o jsonpath='{.items[*].spec.selector.rollouts-pod-template-hash}' shows two different hashes and kubectl argo rollouts status checkout reports Paused; after kubectl argo rollouts promote checkout both selectors converge and status becomes Healthy. (5 points)

Show the worked solution

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
  namespace: checkout
spec:
  replicas: 4
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: ghcr.io/acme/checkout:3.1.0
          ports:
            - containerPort: 8080
  strategy:
    blueGreen:
      activeService: checkout-active
      previewService: checkout-preview
      autoPromotionEnabled: false      # a human promotes
      scaleDownDelaySeconds: 60        # old RS stays warm for instant rollback
      prePromotionAnalysis:
        templates:
          - templateName: smoke-check

# use the plugin — `kubectl set image` only understands built-in workload kinds
kubectl argo rollouts set image checkout checkout=ghcr.io/acme/checkout:3.2.0 -n checkout
kubectl argo rollouts get rollout checkout -n checkout --watch   # shows Paused at preview

kubectl -n checkout get svc checkout-preview -o jsonpath='{.spec.selector.rollouts-pod-template-hash}'
kubectl -n checkout get svc checkout-active  -o jsonpath='{.spec.selector.rollouts-pod-template-hash}'

kubectl argo rollouts promote checkout -n checkout
kubectl argo rollouts status checkout -n checkout      # Healthy
# if the preview looked wrong instead:
kubectl argo rollouts abort checkout -n checkout

Why: blue-green is the right shape whenever two versions must not coexist on live traffic — the cost is double the pods for the duration, the benefit is an instant, atomic cutover and an equally instant rollback. The controller manages both Services by injecting a rollouts-pod-template-hash selector, which is exactly why comparing the two selectors is a sound verification. scaleDownDelaySeconds is what makes rollback instant rather than a fresh scale-up; set it to zero and you have thrown away the main advantage. Compare against canary weighting in CI/CD & progressive delivery.

🦆 Dot’s-eye view

“Every one of those five tasks is invisible to me and I feel all of them. Waves mean my migration is not a race any more. The ignore rule means the deploy dashboard stops lying, so I trust it during an incident. The blue-green preview URL means I can actually click the new version before anyone else gets it. Nobody wrote a runbook for me — the platform just stopped having those failure modes.”

Platform APIs & Self-Service — M6 to M10 (25 points)

☺ Like you’re 10: These five are about inventing your own kind of Kubernetes object — teaching the cluster a new word — and then making it safe and easy for other people to use that word.

The other 25% domain. Half of it is CRD mechanics you can drill to reflex; the other half is the judgement of what to expose to developers and what to keep behind the curtain. Background: Platform APIs & operators, self-service & golden paths, and IaC & control planes.

M6 · Promote a CRD to a new version and migrate the stored objects

Your Environment custom resource has been in v1alpha1 for a year and teams depend on it. You are shipping v1beta1 with the same schema shape, and you want new writes stored in the new version, old clients still able to read, and a clear deprecation warning on the way out — without a conversion webhook.

Your task:

Add v1beta1 to the CRD as served: true, storage: true, and flip v1alpha1 to served: true, storage: false with a deprecation warning.
Set the conversion strategy explicitly to None and explain (to yourself) why that is safe here.
Migrate the existing stored objects and clean up status.storedVersions.

Done when: kubectl get crd environments.platform.acme.io -o jsonpath='{.status.storedVersions}' prints ["v1beta1"] only, and kubectl get environments.v1alpha1.platform.acme.io -A still returns the objects while printing the deprecation warning. (5 points)

Show the worked solution

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: environments.platform.acme.io
spec:
  group: platform.acme.io
  names:
    kind: Environment
    plural: environments
    singular: environment
    shortNames: [env]
  scope: Namespaced
  conversion:
    strategy: None          # identical schemas: no conversion needed
  versions:
    - name: v1alpha1
      served: true
      storage: false
      deprecated: true
      deprecationWarning: "platform.acme.io/v1alpha1 Environment is deprecated; use v1beta1"
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [tier]
              properties:
                tier: { type: string, enum: [dev, staging, prod] }
    - name: v1beta1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [tier]
              properties:
                tier: { type: string, enum: [dev, staging, prod] }

kubectl apply -f environments-crd.yaml

# 1. rewrite every stored object through the API so it is persisted as v1beta1
kubectl get environments.platform.acme.io -A -o json | kubectl replace -f -

# 2. then drop the old version from the stored list
kubectl patch crd environments.platform.acme.io --subresource=status --type=merge \
  -p '{"status":{"storedVersions":["v1beta1"]}}'

kubectl get crd environments.platform.acme.io -o jsonpath='{.status.storedVersions}'
kubectl get environments.v1alpha1.platform.acme.io -A   # still served, warns

Why: exactly one version may be storage: true, and status.storedVersions records every version etcd has ever persisted. You cannot remove a version from the CRD while it is still listed there — the API server refuses, because it would orphan unreadable objects. The order is therefore fixed: add the new storage version, rewrite every object so nothing is left encoded in the old one, then patch storedVersions. strategy: None simply serves the stored object under whichever version you ask for, only changing apiVersion — safe when schemas are compatible, dangerous when they are not, which is when you need a conversion webhook instead.

M7 · Make a custom resource scalable with the scale subresource

Application teams keep asking why kubectl scale does not work on the Worker resources your platform provides, and the autoscaling team cannot point an HPA at them either. Both are asking for the same missing feature.

Your task:

Add the scale subresource to the Worker CRD, mapping spec replicas, status replicas and the label selector.
Make sure the fields it references exist in the schema, and add a printer column for replicas.
Prove that kubectl scale works and that an HPA can target the resource.

Done when: kubectl scale worker/etl --replicas=4 succeeds and kubectl get worker etl -o jsonpath='{.spec.replicas}' prints 4, and an HorizontalPodAutoscaler whose scaleTargetRef is the Worker reaches a non-<unknown> target — kubectl get hpa etl -o jsonpath='{.status.conditions[?(@.type=="AbleToScale")].status}' prints True. (5 points)

Show the worked solution

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: workers.platform.acme.io
spec:
  group: platform.acme.io
  names: { kind: Worker, plural: workers, singular: worker }
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      subresources:
        status: {}
        scale:
          specReplicasPath: .spec.replicas
          statusReplicasPath: .status.replicas
          labelSelectorPath: .status.selector
      additionalPrinterColumns:
        - name: Desired
          type: integer
          jsonPath: .spec.replicas
        - name: Ready
          type: integer
          jsonPath: .status.replicas
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                replicas: { type: integer, minimum: 0, default: 1 }
                queue:    { type: string }
            status:
              type: object
              properties:
                replicas: { type: integer }
                selector: { type: string }

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: etl
  namespace: platform
spec:
  scaleTargetRef:
    apiVersion: platform.acme.io/v1
    kind: Worker
    name: etl
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

kubectl apply -f workers-crd.yaml
kubectl scale worker/etl --replicas=4                          # goes through /scale
kubectl get worker etl                                         # printer columns show Desired/Ready
kubectl get --raw /apis/platform.acme.io/v1/namespaces/platform/workers/etl/scale

kubectl apply -f worker-hpa.yaml
kubectl get hpa etl -o jsonpath='{.spec.scaleTargetRef.kind}'  # Worker
kubectl describe hpa etl                                       # <unknown> target means no selector

Why: the scale subresource makes a custom kind a first-class citizen of the scaling ecosystem — kubectl scale, the HPA and KEDA all speak /scale, not your bespoke field name. labelSelectorPath is the part candidates miss: kubectl scale works without it, but the HPA cannot compute per-pod metrics without a selector string and silently reports unknown. Every path must point at a field that exists in the schema and that your controller populates in status.

M8 · Wire owner references so children are garbage-collected

Your Environment controller creates a Namespace-scoped ConfigMap and a Secret for each environment. When a developer deletes their Environment, those children are left behind — the cluster has accumulated 200 orphaned ConfigMaps, and a re-created environment inherits stale config from its dead predecessor.

Your task:

Add an ownerReferences entry to the child ConfigMap pointing at its owning Environment, marked as the controller and blocking owner deletion.
Delete the owner and show the child is collected automatically.
Repeat the deletion with --cascade=orphan and observe the difference.

Done when: kubectl delete environment demo is followed within seconds by kubectl get cm demo-config returning NotFound, while kubectl delete environment demo2 --cascade=orphan leaves demo2-config in place with its ownerReferences intact. (5 points)

Show the worked solution

apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-config
  namespace: demo
  ownerReferences:
    - apiVersion: platform.acme.io/v1beta1
      kind: Environment
      name: demo
      uid: 4f8f0c1e-1e4b-4c2a-9a5d-0f8a1c7b2d33   # MUST be the live object's uid
      controller: true
      blockOwnerDeletion: true
data:
  TIER: dev

# fetch the real uid rather than typing one
UID=$(kubectl -n demo get environment demo -o jsonpath='{.metadata.uid}')
kubectl -n demo patch cm demo-config --type=merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"platform.acme.io/v1beta1\",\"kind\":\"Environment\",\"name\":\"demo\",\"uid\":\"$UID\",\"controller\":true,\"blockOwnerDeletion\":true}]}}"

kubectl -n demo delete environment demo
kubectl -n demo get cm demo-config          # NotFound — collected

kubectl -n demo delete environment demo2 --cascade=orphan
kubectl -n demo get cm demo2-config         # still there

Why: owner references are how Kubernetes does cleanup — the garbage collector, not your controller, deletes the children, so cleanup keeps working even if your operator is down. Three rules the exam checks: the uid must match the live owner (a stale uid makes the GC delete the child immediately), owner and dependent must share a namespace unless the owner is cluster-scoped, and only one reference may set controller: true. blockOwnerDeletion only matters for foreground deletion, where it holds the owner in Terminating until the child is gone. Use owner refs for structural cleanup and finalizers only for external resources the GC cannot see.

M9 · Move a Crossplane Composition to pipeline mode

Your XBucket Composition still uses the legacy inline resources array with patches. The platform team wants it on the function pipeline so they can later insert validation and a second function without rewriting everything, and so the composition stops depending on deprecated fields.

Your task:

Install the function-patch-and-transform Function package and wait for it to become healthy.
Rewrite the Composition to mode: Pipeline with a single step that feeds the existing resources and patches to that function.
Create a claim and show the composed resource still appears with the patched name.

Done when: kubectl get functions shows the function INSTALLED=True HEALTHY=True, kubectl get composition xbucket -o jsonpath='{.spec.mode}' prints Pipeline, and kubectl get managed lists the composed resource created from your claim. (5 points)

Show the worked solution

apiVersion: pkg.crossplane.io/v1     # pkg.crossplane.io/v1beta1 on Crossplane < v1.16
kind: Function
metadata:
  name: function-patch-and-transform
spec:
  package: xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:v0.7.0
---
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xbucket
spec:
  compositeTypeRef:
    apiVersion: platform.acme.io/v1alpha1
    kind: XBucket
  mode: Pipeline
  pipeline:
    - step: patch-and-transform
      functionRef:
        name: function-patch-and-transform
      input:
        apiVersion: pt.fn.crossplane.io/v1beta1
        kind: Resources
        resources:
          - name: bucket
            base:
              apiVersion: s3.aws.upbound.io/v1beta1
              kind: Bucket
              spec:
                forProvider:
                  region: eu-west-1
            patches:
              - type: FromCompositeFieldPath
                fromFieldPath: spec.parameters.region
                toFieldPath: spec.forProvider.region
              - type: FromCompositeFieldPath
                fromFieldPath: metadata.name
                toFieldPath: metadata.annotations[crossplane.io/external-name]

kubectl apply -f function.yaml
kubectl get functions -w                 # wait for INSTALLED/HEALTHY True
kubectl apply -f composition.yaml
kubectl apply -f claim.yaml
kubectl get claim,composite,managed
kubectl describe composite <name>        # events explain a failed pipeline step

Why: pipeline mode turns composition from a fixed patching engine into a sequence of composable steps, each a container that receives the observed state and returns desired state. The migration is mechanical — the old spec.resources array becomes the input of one function-patch-and-transform step — but it unlocks inserting validation, templating or environment lookups later without a rewrite. Note the function must be healthy before the Composition references it, or every claim fails with a pipeline step error. Crossplane API versions move quickly; confirm with kubectl explain composition.spec on the cluster in front of you. Background in IaC & control planes.

M10 · Expose your custom API to developers with an aggregated ClusterRole

You shipped the Worker API and immediately got a stream of tickets: developers with the standard edit role in their namespace get Forbidden when they try to create a Worker. Right now the platform team hand-writes a Role per namespace, which does not scale and drifts.

Your task:

Create a ClusterRole granting full CRUD on workers (and read on workers/status) that aggregates into the built-in edit and admin roles.
Do not create any new RoleBinding for the developers — rely on their existing edit binding.
Prove the permission arrived, and prove a viewer still cannot write.

Done when: kubectl get clusterrole edit -o yaml | grep -A3 workers shows the aggregated rule, and kubectl auth can-i create workers --as=system:serviceaccount:team-a:dev -n team-a prints yes while the same check against a view-bound identity prints no. (5 points)

Show the worked solution

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: platform:worker-edit
  labels:
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
  - apiGroups: ["platform.acme.io"]
    resources: ["workers"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["platform.acme.io"]
    resources: ["workers/status"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: platform:worker-view
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
rules:
  - apiGroups: ["platform.acme.io"]
    resources: ["workers"]
    verbs: ["get", "list", "watch"]

kubectl apply -f worker-rbac.yaml
kubectl get clusterrole edit -o yaml | grep -A4 'platform.acme.io'

kubectl auth can-i create workers --as=system:serviceaccount:team-a:dev -n team-a   # yes
kubectl auth can-i create workers --as=system:serviceaccount:team-a:reader -n team-a # no
kubectl auth can-i --list --as=system:serviceaccount:team-a:dev -n team-a

Why: the built-in view, edit and admin ClusterRoles are aggregated roles — the controller continuously unions in the rules of any ClusterRole carrying the matching label. That is the intended extension point for a custom API: label once, and every namespace where someone already holds edit gains the permission, forever, with no per-namespace bindings to drift. Writing bespoke Roles instead is the anti-pattern that produces the ticket queue you started with. Verifying with kubectl auth can-i --as is non-negotiable — a policy you did not test from the caller’s identity is a guess. See self-service & golden paths.

🐢 Timmy’s drill · 15 min

Before you sit the full paper, do a warm-up that is pure speed: from a cold terminal, produce a valid CRD skeleton with a status subresource and one printer column in under three minutes, using only kubectl explain customresourcedefinition.spec.versions --recursive as a reference. Then do the same for an Application, a HelmRelease and a ScaledObject from kubectl explain alone — no vendor docs, because on the day you will not have them. If any of the four takes over three minutes, that is your highest-value study target, ahead of any concept.

Observability & Operations — M11 to M13 (20 points)

☺ Like you’re 10: These three are about making sure the right person gets woken up for the right problem, that you are not shipping people’s private data into your tracing system, and that you can find a broken thing quickly.

Twenty percent of the paper, and the domain that most rewards a fixed method. When a triage task starts, resist the urge to guess — go through the same evidence order every time. Background: observability, reliability & incidents and the troubleshooting playbook.

M11 · Route alerts to the right team and suppress the noise

At 02:00 a node filled its disk and the platform team’s pager fired eleven times: one critical alert plus ten dependent warnings from the same namespace. Meanwhile a genuine payments critical alert went to a Slack channel nobody reads. Alertmanager currently has one route and one receiver.

Your task:

Build a routing tree: severity=critical with team=payments goes to the payments-pager receiver; any other severity=warning goes to platform-chat with a 12-hour repeat interval; everything else falls through to a default receiver.
Add an inhibition rule so a firing critical suppresses warning alerts sharing the same alertname and namespace.
Silence one specific alert for two hours while you fix it.

Done when: amtool config routes test --config.file=alertmanager.yaml severity=critical team=payments prints payments-pager and the same command with severity=warning prints platform-chat, and amtool silence query lists your active silence. (7 points)

Show the worked solution

route:
  receiver: platform-default
  group_by: ['alertname', 'namespace']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="critical"
        - team="payments"
      receiver: payments-pager
    - matchers:
        - severity="warning"
      receiver: platform-chat
      repeat_interval: 12h

inhibit_rules:
  - source_matchers: [ severity="critical" ]
    target_matchers: [ severity="warning" ]
    equal: ['alertname', 'namespace']

receivers:
  - name: platform-default
  - name: payments-pager
  - name: platform-chat

amtool check-config alertmanager.yaml
amtool config routes test --config.file=alertmanager.yaml severity=critical team=payments
amtool config routes test --config.file=alertmanager.yaml severity=warning
amtool config routes show --config.file=alertmanager.yaml

amtool silence add alertname=NodeDiskWillFill namespace=kube-system \
  --duration=2h --comment="disk expansion in progress, ticket PLAT-412"
amtool silence query

Why: routing is depth-first and stops at the first matching child unless it sets continue: true — so specific routes go above general ones, or the warning route swallows everything. The three intervals get mixed up constantly: group_wait holds the first notification of a new group so related alerts batch, group_interval is the wait before notifying about new alerts joining an existing group, and repeat_interval is how often to re-notify about alerts still firing. Inhibition is the structural fix for the 02:00 storm: express the causal relationship once instead of asking humans to ignore ten pages. Silences are for a known, time-boxed fix — always with a comment, because an uncommented silence outlives the incident and hides the next one.

M12 · Build an OpenTelemetry Collector pipeline that redacts and samples

Legal has found customer email addresses and Authorization headers in trace attributes in the tracing backend. Separately, the collector fell over twice last month under a burst of spans, and the backend bill is dominated by successful, fast requests nobody ever looks at.

Your task:

Configure a traces pipeline with a memory guard, an attribute processor that deletes the authorization header attribute and hashes user.email, tail sampling, and batching.
Sample so that all errors and all requests slower than 500 ms are kept, plus 5% of everything else.
Order the processors correctly and prove the pipeline is live.

Done when: the collector starts with no config error (kubectl -n observability logs deploy/otel-collector | grep -i "everything is ready"), kubectl -n observability get cm otel-collector-conf -o yaml shows the four processors in the pipeline in the required order, and a test span carrying user.email arrives at the backend hashed. (7 points)

Show the worked solution

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15
  attributes/redact:
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: user.email
        action: hash
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 500
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  otlp/jaeger:
    endpoint: jaeger-collector.observability.svc:4317
    tls:
      insecure: true
  debug:
    verbosity: normal

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, attributes/redact, tail_sampling, batch]
      exporters: [otlp/jaeger, debug]

kubectl -n observability create cm otel-collector-conf --from-file=config.yaml --dry-run=client -o yaml | kubectl apply -f -
kubectl -n observability rollout restart deploy/otel-collector
kubectl -n observability logs deploy/otel-collector | tail -20

Why: processor order is the whole task. memory_limiter must be first so it can shed load before anything else allocates — behind another processor it protects nothing. Redaction must happen before sampling and export so no unredacted attribute can escape on any path. tail_sampling must see a whole trace before deciding, which is why it buffers for decision_wait and why it must come before batch. batch is always last, immediately before the exporters, because batching is an export optimisation. Note also that tail sampling needs every span of a trace to reach the same collector instance, so a scaled-out deployment needs a load-balancing exporter in front of it — a detail worth knowing even when the task does not ask for it.

M13 · Triage a service that returns nothing at all

The orders team reports that their service “is up but returns connection refused from other namespaces.” The Deployment shows 3/3 pods Running, no restarts, no recent deploys, and the developer has already restarted the pods twice with no change.

Your task:

Determine why traffic to the orders Service never reaches a pod, working from evidence rather than guessing.
Fix the root cause with the smallest correct change.
Prove the Service now serves traffic from another namespace.

Done when: kubectl -n orders get endpointslices -l kubernetes.io/service-name=orders -o jsonpath='{.items[*].endpoints[*].conditions.ready}' shows true for every pod, and kubectl run probe --rm -it --restart=Never --image=curlimages/curl -n default --command -- curl -s -o /dev/null -w "%{http_code}" http://orders.orders.svc:80/healthz prints 200. (6 points)

Show the worked solution

# fixed evidence order — never start by editing YAML
kubectl -n orders get svc orders -o yaml | grep -A5 -E 'selector|ports'
kubectl -n orders get pods --show-labels
kubectl -n orders get endpointslices -l kubernetes.io/service-name=orders

# empty endpoints means either the selector misses, or pods are not Ready
kubectl -n orders describe pod -l app=orders | grep -A5 -E 'Readiness|Events'
kubectl -n orders get pods -o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}'

# the two classic root causes, both fixed in the Service/pod spec:

# (a) selector drift — Service selects app=orders, pods are labelled app=orders-api
apiVersion: v1
kind: Service
metadata:
  name: orders
  namespace: orders
spec:
  selector:
    app: orders-api          # now matches the pod labels
  ports:
    - name: http
      port: 80
      targetPort: http       # named port, resolved against the container

# (b) readiness probe pointed at the wrong port, so pods never became Ready
readinessProbe:
  httpGet:
    path: /healthz
    port: 8080               # was 8000; container listens on 8080
  initialDelaySeconds: 3
  periodSeconds: 5

kubectl -n orders get endpointslices -l kubernetes.io/service-name=orders
# --command is required: the curlimages/curl entrypoint is already `curl`
kubectl run probe --rm -it --restart=Never --image=curlimages/curl -n default \
  --command -- curl -s -o /dev/null -w "%{http_code}\n" http://orders.orders.svc:80/healthz

Why: “Running but unreachable” has a short suspect list, and EndpointSlices tell you which branch you are on in one command. Empty endpoints means the Service’s selector does not match any pod labels, or the matching pods are not Ready — a readiness probe pointed at the wrong port produces the second, and it is invisible in get pods output if you only read the STATUS column instead of READY. Non-empty endpoints but still refused means the problem is further out: targetPort versus the real container port, or a NetworkPolicy. Note that Running is about the container process, not about serving traffic — only the Ready condition gates endpoint membership. The full decision tree is in triaging workloads and networking.

Platform Architecture & Infrastructure — M14 to M16 (15 points)

☺ Like you’re 10: These three are about who gets a seat when the room is full, keeping the important box even after you throw away the label, and letting a helper go to sleep when there is no work.

Three self-contained tasks with excellent points-per-minute — do not leave them unattempted because they sit late in the paper. Background: platform architecture, storage & state and scaling & scheduling.

M14 · Guarantee platform components a seat with priority and preemption

During a batch-heavy afternoon the cluster filled up and the ingress controller’s replacement pod sat Pending for eleven minutes behind a queue of analytics jobs. Platform components must never lose a scheduling race to batch work again — but batch jobs must never evict each other, because a preempted job restarts from zero.

Your task:

Create a platform-critical PriorityClass with a high value that may preempt lower-priority pods.
Create a batch-background PriorityClass with a low value and a preemption policy of Never.
Assign them, fill a node with batch pods, and demonstrate preemption occurring.

Done when: kubectl get priorityclass shows both classes with the expected values, and after scheduling a platform-critical pod onto a full node, kubectl get events --field-selector reason=Preempted -A shows a batch pod evicted while kubectl -n batch get pods shows no batch pod preempting another. (5 points)

Show the worked solution

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: platform-critical
value: 1000000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "Ingress, CNI, CoreDNS and other components the platform cannot lose."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch-background
value: 100
globalDefault: false
preemptionPolicy: Never
description: "Best-effort batch. Waits for room; never evicts anything."

# in the pod template of the component that must win:
spec:
  priorityClassName: platform-critical
  containers:
    - name: controller
      image: ghcr.io/acme/ingress:1.9.2
      resources:
        requests: { cpu: 200m, memory: 256Mi }

kubectl apply -f priorityclasses.yaml
kubectl get priorityclass

kubectl -n batch scale deploy/cruncher --replicas=40    # fill the node
kubectl -n platform rollout restart deploy/ingress
kubectl get events -A --field-selector reason=Preempted
kubectl -n platform get pod -l app=ingress -o jsonpath='{.items[0].spec.priority}'

Why: priority does two separate things and the exam separates them. Priority orders the scheduling queue — higher-priority pending pods are considered first. Preemption is what happens when the highest-priority pod still does not fit: the scheduler evicts lower-priority pods to make room. preemptionPolicy: Never keeps the queue advantage while removing the eviction power, which is exactly right for long-running batch that cannot afford to be restarted. Values matter relatively, not absolutely, but stay below the reserved system classes (system-cluster-critical is two billion) or you will out-prioritise the control plane. Pair this with a PodDisruptionBudget: priority protects you from the scheduler, a PDB protects you from drains.

M15 · Ship a StorageClass that survives a deleted claim

A developer deleted a PersistentVolumeClaim while cleaning up a namespace and the underlying volume — with three days of unbacked-up data — was deleted with it, because the default StorageClass reclaims aggressively. You are adding a StorageClass for stateful platform data with different guarantees.

Your task:

Create a StorageClass platform-retain with reclaimPolicy: Retain, allowVolumeExpansion: true and volumeBindingMode: WaitForFirstConsumer.
Create a PVC that uses it and a pod that mounts the PVC, so binding actually happens.
Delete the PVC and show the PersistentVolume survives.

Done when: kubectl get sc platform-retain -o jsonpath='{.reclaimPolicy}{" "}{.allowVolumeExpansion}{" "}{.volumeBindingMode}' prints Retain true WaitForFirstConsumer, and after deleting the PVC, kubectl get pv shows the volume in phase Released rather than gone. (5 points)

Show the worked solution

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: platform-retain
provisioner: rancher.io/local-path      # use whatever your cluster actually has
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ledger-data
  namespace: platform
spec:
  storageClassName: platform-retain
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 2Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: ledger-writer
  namespace: platform
spec:
  containers:
    - name: app
      image: alpine:3.20
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: ledger-data

kubectl apply -f storage.yaml
kubectl -n platform get pvc ledger-data     # Pending until the pod schedules, by design
kubectl get pv

# expansion, while the claim still exists and where the provisioner supports it:
kubectl -n platform patch pvc ledger-data \
  -p '{"spec":{"resources":{"requests":{"storage":"5Gi"}}}}'

kubectl -n platform delete pod ledger-writer
kubectl -n platform delete pvc ledger-data
kubectl get pv                              # STATUS Released, not deleted

# a Released PV is not reusable until its binding is cleared:
kubectl patch pv <pv-name> --type=json -p '[{"op":"remove","path":"/spec/claimRef"}]'

Why: three fields, three distinct guarantees. reclaimPolicy: Retain means deleting the claim releases the volume but never the data — recovery becomes possible, at the cost of manual cleanup. allowVolumeExpansion lets a PVC grow in place by editing its request instead of a migration (the CSI driver must support it — some local provisioners, including the one kind ships, do not, and you get a “resizing not supported” event rather than an error at apply time). WaitForFirstConsumer delays binding until a pod is scheduled, so the volume lands in the same zone or node as the workload — without it, topology-constrained volumes strand pods as Pending forever. Note a Released PV is not automatically reusable: clear its spec.claimRef first. More in storage & state.

M16 · Scale a queue worker to zero with KEDA

The reconcile-worker Deployment runs 3 replicas 24/7 and processes a queue that is empty for about eighteen hours a day. Finance has noticed. A plain HPA has not helped, because it will not go below one replica and it scales on CPU, which stays flat regardless of queue depth.

Your task:

Create a KEDA ScaledObject targeting the Deployment with minReplicaCount: 0 and a sensible maximum.
Drive it from a Prometheus query on queue depth, with an activation threshold that distinguishes 0 to 1 from 1 to N.
Show the Deployment resting at zero replicas with an empty queue, and scaling up when work arrives.

Done when: kubectl -n jobs get scaledobject reconcile-worker -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' prints True, kubectl -n jobs get deploy reconcile-worker -o jsonpath='{.spec.replicas}' prints 0 with the queue empty, and the replica count rises after messages are published. (5 points)

Show the worked solution

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: reconcile-worker
  namespace: jobs
spec:
  scaleTargetRef:
    name: reconcile-worker
  minReplicaCount: 0
  maxReplicaCount: 20
  pollingInterval: 15        # how often KEDA asks Prometheus
  cooldownPeriod: 120        # quiet time before scaling back to zero
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-operated.monitoring.svc:9090
        query: sum(rabbitmq_queue_messages_ready{queue="reconcile"})
        threshold: "50"            # ~50 messages per replica
        activationThreshold: "1"   # below this, scale all the way to zero

kubectl apply -f scaledobject.yaml
kubectl -n jobs get scaledobject reconcile-worker
kubectl -n jobs get hpa                       # keda-hpa-reconcile-worker, created by KEDA
kubectl -n jobs get deploy reconcile-worker -w
kubectl -n jobs describe scaledobject reconcile-worker   # events explain a failing trigger

Why: the HorizontalPodAutoscaler cannot scale to zero, and it only understands metrics the metrics APIs expose. KEDA solves both: it is an operator that reads an event source directly and creates and manages an HPA for you for the 1-to-N range, while handling the 0-to-1 activation itself. That split is why there are two numbers — activationThreshold decides whether there is any work at all, threshold decides how much work each replica should carry. Do not hand-edit the generated HPA; KEDA owns it and will revert you. The cost of scaling to zero is cold-start latency on the first message, so cooldownPeriod is a real tuning decision rather than a formality. Compare the three autoscaling axes in scaling & scheduling and the spend argument in FinOps.

Security & Policy Enforcement — M17 to M18 (15 points)

☺ Like you’re 10: These two are about setting house rules for the cluster — one uses the rules Kubernetes already has built in, the other writes new rules that also tidy up after new teams automatically.

Only two tasks, but 15 points — the highest points-per-task on the paper. Never leave these unattempted. Background: security & policy enforcement and governance & compliance.

M17 · Enforce restricted Pod Security and fix what it breaks

An audit found containers in the payments namespace running as root with no seccomp profile. You are enforcing the restricted Pod Security Standard there — but the namespace is live, and a rollout that starts rejecting pods at 09:00 is not an acceptable way to find out what does not comply.

Your task:

Find out, without breaking anything, which existing workloads would be rejected by restricted.
Fix the offending Deployment’s pod template so it complies.
Apply enforce=restricted (pinned to a version) plus warn and audit, and prove a non-compliant pod is now rejected.

Done when: kubectl -n payments rollout status deploy/api succeeds after the label is applied, and kubectl -n payments run bad --image=nginx --restart=Never is rejected with a PodSecurity violation message. (8 points)

Show the worked solution

# 1. dry-run the label against the live API — prints warnings for every current violation
kubectl label --dry-run=server --overwrite ns payments \
  pod-security.kubernetes.io/enforce=restricted

# 2. make the pod template compliant with "restricted"
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          image: ghcr.io/acme/payments-api:4.2.0
          securityContext:
            allowPrivilegeEscalation: false
            privileged: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}

# 3. enforce, pinned to a version so a cluster upgrade cannot tighten the rules under you
kubectl label --overwrite ns payments \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=v1.31 \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted

kubectl -n payments rollout status deploy/api
kubectl -n payments run bad --image=nginx --restart=Never   # rejected: PodSecurity violation

Why: Pod Security Admission is built into the API server — no controller to install, no policy language, three labels per namespace. The exam-worthy trick is the first command: kubectl label --dry-run=server asks the API server to evaluate the change and return warnings without persisting it, giving you the full list of what would break before anything does. The three modes are complementary: warn tells the person applying, audit writes to the audit log, enforce rejects. Pinning enforce-version matters because the restricted profile gains requirements over releases, so an unpinned setting lets a cluster upgrade reject workloads that were compliant yesterday. PSA gates pod creation only — it never rewrites existing pods, which is why the Deployment must be fixed first.

M18 · Generate tenant guardrails automatically with Kyverno

Every new tenant namespace is supposed to get a default-deny NetworkPolicy and a standard set of labels. In practice the platform team applies them by hand, forgets roughly one namespace in five, and two of the missing ones turned up in a penetration test as wide-open lateral movement paths.

Your task:

Write a Kyverno ClusterPolicy whose first rule mutates every new Namespace carrying a platform.acme.io/tenant label to also carry platform.acme.io/managed: "true".
Add a second rule that generates a default-deny NetworkPolicy inside any such namespace, kept in sync if someone deletes it.
Create a tenant namespace and prove both rules fired.

Done when: after kubectl create ns tenant-x and labelling it, kubectl get ns tenant-x -o jsonpath='{.metadata.labels}' contains the managed label, kubectl -n tenant-x get netpol default-deny exists, and deleting that NetworkPolicy causes Kyverno to recreate it within a minute. (7 points)

Show the worked solution

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: tenant-baseline
spec:
  background: true
  rules:
    - name: label-managed
      match:
        any:
          - resources:
              kinds: [Namespace]
              selector:
                matchLabels:
                  platform.acme.io/tenant: "?*"
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              platform.acme.io/managed: "true"

    - name: generate-default-deny
      match:
        any:
          - resources:
              kinds: [Namespace]
              selector:
                matchLabels:
                  platform.acme.io/tenant: "?*"
      generate:
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        name: default-deny
        namespace: "{{request.object.metadata.name}}"
        synchronize: true
        data:
          metadata:
            labels:
              platform.acme.io/managed: "true"
          spec:
            podSelector: {}
            policyTypes:
              - Ingress
              - Egress

kubectl apply -f tenant-baseline.yaml
kubectl get clusterpolicy tenant-baseline          # READY True before you test

kubectl create ns tenant-x
kubectl label ns tenant-x platform.acme.io/tenant=acme-x
kubectl get ns tenant-x -o jsonpath='{.metadata.labels}'
kubectl -n tenant-x get netpol default-deny

kubectl -n tenant-x delete netpol default-deny
sleep 30 && kubectl -n tenant-x get netpol default-deny   # recreated
kubectl get policyreport -A                        # what fired, and where

Why: Kyverno’s three rule types answer three different questions — validate rejects bad input, mutate corrects it on the way in, and generate creates resources the user never asked for. generate is the one that turns policy from a gate into a golden path: the tenant gets a correct baseline for free instead of being told off for missing it. Two details decide whether this works: synchronize: true makes the generated resource continuously reconciled, so deleting or editing it is undone (without it, generation is one-shot), and background: true lets the policy also apply to namespaces that already existed when you installed it. Kyverno’s background controller needs RBAC to create NetworkPolicies — if the generate rule silently does nothing, check its ClusterRole before you doubt the policy. Compare with the reject-only approach in security & policy enforcement.

⚠ The two most expensive habits in this domain

First, applying an enforcing policy without measuring first. Both tasks above have a measure step — --dry-run=server for PSA, Audit mode for Kyverno — and skipping it is how a policy rollout becomes an incident. Second, testing a policy as yourself. You are almost certainly cluster-admin on your practice cluster, and admin bypasses more than you think: PSA exempts nothing by default but Kyverno policies commonly exempt system namespaces and service accounts. Always verify from the identity that will actually be affected — --as for RBAC, a real namespace for admission.

Score yourself

☺ Like you’re 10: Add up your points, turn it into a percentage, and then — more usefully — look at which questions you lost points on and go back and read about those.

Mark after a break — marking your own work while still adrenalised produces generous nonsense. Award full points only when the done-when check actually passed on the cluster, half when the resource exists and is broadly right but the check did not pass, and zero when you did not attempt it. There is no credit for “I knew how to do that one.”

Task	Domain	Points
M1 · Sync waves and a PreSync hook	GitOps	5
M2 · ignoreDifferences vs a fighting HPA	GitOps	5
M3 · Flux HelmRelease with drift detection	GitOps	5
M4 · Tekton results and a shared workspace	GitOps	5
M5 · Blue-green with manual promotion	GitOps	5
M6 · CRD version promotion and storage migration	Platform APIs	5
M7 · Scale subresource on a custom resource	Platform APIs	5
M8 · Owner references and garbage collection	Platform APIs	5
M9 · Crossplane Composition in pipeline mode	Platform APIs	5
M10 · Aggregated ClusterRole for a custom API	Platform APIs	5
M11 · Alertmanager routing, inhibition, silence	Observability	7
M12 · OTel Collector redaction and tail sampling	Observability	7
M13 · Triage a Service with no endpoints	Observability	6
M14 · PriorityClass and preemption policy	Architecture	5
M15 · StorageClass with Retain and expansion	Architecture	5
M16 · KEDA scale-to-zero on queue depth	Architecture	5
M17 · Restricted Pod Security, measured first	Security	8
M18 · Kyverno mutate + generate guardrails	Security	7
Total	All five domains	100

Computing your result. The points total 100, so your raw score is your percentage, and the pass mark is 64%. That leaves 36 points spendable — you can drop six of the five-point tasks outright and still clear the bar with six to spare. Then do the more useful arithmetic and total each domain separately: a 70 made of five even domains and a 70 made of four strong domains plus a zero are completely different results, and the second one fails a real exam the day the task draw is unkind.

Domain	Available	If you scored under two-thirds, go here
GitOps & Continuous Delivery	25	GitOps workflows, CI/CD & progressive delivery, then re-drill the GitOps bank.
Platform APIs & Self-Service	25	Platform APIs & operators, self-service, then the Platform APIs bank.
Observability & Operations	20	Observability, reliability & incidents, then the Observability bank.
Platform Architecture & Infrastructure	15	Platform architecture, storage & state, scaling, then the Architecture bank.
Security & Policy Enforcement	15	Security & policy, governance, then the Security bank.

What to do with each individual miss. Sort your zeros and halves into three piles, because they need three different treatments. A task you did not recognise at all is a knowledge gap: read the lesson page linked in its solution, then re-do the task cold two days later. A task where you knew the shape but burned the clock hunting field names is a speed problem, and the fix is the command reference plus kubectl explain drills, not more reading. A task that broke in a way you could not diagnose is a triage gap: work the troubleshooting playbook and triaging delivery until the evidence order is automatic.

What your score means

☺ Like you’re 10: One score does not tell you if you will pass — but the shape of it tells you what to practise next, which is more useful anyway.

Treat the number as a diagnostic, never as a prediction. A mock is harsher in some ways (no Quick Reference box tailored to the task, tasks written by someone who cannot see your cluster) and gentler in others (your own cluster, your own aliases, no exam-day nerves). Read the band, not the decimal.

Score	Read it as	Next move
80–100	Comfortably ready, with margin for a bad task draw.	Stop grinding tasks. Work the final checklist, skim the field notes for exam-day logistics, and book it.
64–79	A pass on this paper — but with no margin if two tasks go sideways.	Re-drill only your weakest domain, then sit a second timed paper. Speed work, not new topics.
50–63	Close. Usually a speed problem plus one weak domain, not a knowledge problem.	Two weeks of timed drills from the bank, plus what to know cold for the shapes you kept looking up.
Under 50	Genuine gaps across domains. Booking now wastes the sitting.	Back to the lessons for the two lowest-scoring domains, then the lab track to rebuild hands-on fluency before another mock.

One more pattern worth checking, independent of the total: how many tasks did you leave completely unattempted? If the answer is more than two, your problem is pacing rather than knowledge, and pacing is the cheapest thing on this list to fix. Sit the paper again in a fortnight, with a hard rule that you touch every task once before you return to any of them.

🎬 At the Platform Guild
🦆Dot: 71 out of 100. That is a pass! I am booking the exam tonight.
🐢Timmy: Show me the domain breakdown, not the total.
🦆Dot: …GitOps 25, Platform APIs 24, Observability 15, Architecture 7, Security 0. I ran out of clock before the last two.
🦫Benny: So you scored forty-nine out of fifty on the two domains you actually finished, and zero on the highest-value pair on the paper. That is not a knowledge problem, Dot. That is you refusing to abandon M9.
👺Gizmo: Easy — just work faster! Skip the verification sweep, that is seven whole minutes back. 🤑
🐢Timmy: The sweep is where she finds the two half-finished tasks, Gizmo. What she skips is the eleventh minute of a six-minute task.
🦉Professor Owl: Sit it again in a fortnight, Dot, in reverse order — Security first. If your score jumps twenty points without you learning a single new thing, you will never argue about pacing again.

🐢 Timmy’s checkpoint

1. Which documentation may you have open during the real exam, and name three popular projects whose own docs are not permitted. 2. In Argo CD, what does ignoreDifferences fail to do on its own, and which sync option completes it? 3. Order these OpenTelemetry Collector processors and say why: batch, tail_sampling, memory_limiter, attributes. 4. You must remove v1alpha1 from a CRD — what has to happen first, and which field proves it is safe? 5. What does preemptionPolicy: Never keep, and what does it give up? 6. Which single command tells you what enforcing restricted Pod Security would break, without breaking it? 7. Your score is 71 but Security is 0 — what is the diagnosis and the fix?

Check your answers

kubernetes.io/docs, kubernetes.io/blog, task-specific docs linked from the exam’s Quick Reference box, and local man//usr/share documentation. Not permitted: Argo CD, Flux, Tekton, Crossplane, Backstage, Prometheus, OpenTelemetry, Kyverno, Gatekeeper, Istio and Helm docs (any three). Your substitute is the live API: kubectl explain --recursive and reading an existing object.
It silences the diff but not automated self-heal, which recomputes its own target and reverts the field anyway. Add the sync option RespectIgnoreDifferences=true.
memory_limiter → attributes → tail_sampling → batch. The limiter must shed load before anything allocates; redaction must precede anything that exports; tail sampling must buffer whole traces before deciding; batching is an export optimisation and always sits last.
Every stored object must be rewritten under the new storage version (kubectl get … -o json | kubectl replace -f -), then status.storedVersions must be patched to list only the surviving version. That field is the proof — while the old version is listed, removal is refused.
It keeps the pod’s scheduling-queue priority (it is considered before lower-priority pending pods) and gives up its ability to preempt — it waits for room instead of evicting anything. Right for long batch jobs that would restart from zero.
kubectl label --dry-run=server --overwrite ns <ns> pod-security.kubernetes.io/enforce=restricted — the API server evaluates and returns warnings for every current violation without persisting the label.
A pacing problem, not a knowledge problem: you never reached the highest-value tasks. The fix is the flag-and-move rule plus a re-sit in reverse domain order — and it is the cheapest twenty points available to you.

That is the whole paper. Score it, write the domain breakdown somewhere you will see it, and let the two weakest numbers choose your next fortnight. When the numbers stop moving, the remaining work is logistical rather than technical: the exam guide covers booking, the environment and the rules; the field notes collect what people actually reported from the day; the final checklist is the last thing to read before you sit down; and the command reference, what to know cold and the glossary are the three pages worth a last skim on the morning. Back to the drill bank when you are ready to go again — or the course home to pick a lesson.

⏱ The six papers

Set 1 · Set 2 (you are here) · Set 3 · Set 4 · Set 5 · Set 6. Next up: Set 3. All six are weighted to the official blueprint and worth 100 points each; see the practice hub for how to sit them, and the exam guide for the real thing.