Drill — Write an Enforcing Kyverno Policy

Kyverno's real test isn't installing a policy — it's the moment you flip one from Audit to Enforce and watch actual traffic get refused for the first time. This drill puts you through exactly that moment, start to finish, on a throwaway cluster: install Kyverno, apply the same require-crew-label rule this course's Kyverno lesson walks through, and prove Audit mode's very real gap — a violation that gets recorded and never stopped. Then flip failureAction to Enforce and watch admission actually say no, with the policy's own message sitting in the error. The harder half comes after: PolicyExceptions are off by default for a reason, so you turn that on deliberately and scoped, write one exception for one legitimate workload, and prove — not assume — that it never widens past that single Deployment.

How this drill works

☺ Like you're 10: One policy, evolved in place through seven steps — work them in order, not as two separate puzzles.

Budget 25 to 40 minutes. Like the real lifecycle of a guardrail, this is one policy rewritten under you in place — install it soft, prove the soft version's gap, harden it, prove the hard version actually holds, then carve one narrow, reviewable exception into it rather than softening it back. Each step names exactly what to run and exactly what you should see; if your terminal disagrees in a way that looks structural rather than cosmetic, stop and re-read the previous step before continuing.

Stand up Kyverno on a throwaway cluster

☺ Like you're 10: Before the hall monitor can do anything, the hall monitor has to actually show up for work.

Create the cluster, two namespaces you'll use throughout the drill, and install Kyverno with Helm:

kind create cluster --name kyverno-drill
kubectl create namespace analytics
kubectl create namespace platform-exceptions

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace

kubectl -n kyverno rollout status deploy/kyverno-admission-controller --timeout=120s
kubectl get pods -n kyverno

deployment "kyverno-admission-controller" successfully rolled out

NAME                                             READY   STATUS    RESTARTS   AGE
kyverno-admission-controller-7c9d4f6b5d-x2q7f    1/1     Running   0          58s
kyverno-background-controller-6f8b9c7d4-m9k2p    1/1     Running   0          58s
kyverno-cleanup-controller-5d7c6b8f9-tz4jn       1/1     Running   0          58s
kyverno-reports-controller-8b6c5d7f4-vh3wr       1/1     Running   0          58s

Four controllers, four rows. If you only see the admission controller, give it another minute — the other three come up slightly behind it. The Kyverno lesson covers what each one actually does if any of these names are unfamiliar.

Done when: the rollout reports "successfully rolled out" and all four Kyverno pods show 1/1 Running.

Install the policy in Audit mode — and watch it wave a violation through

☺ Like you're 10: The monitor writes your name down. You keep walking. Nothing stops you.

Apply mission-control-baseline, requiring a crew label on every workload — the exact policy this course's Kyverno lesson works through, started here in its softest setting:

# mission-control-baseline.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: mission-control-baseline
spec:
  background: true
  rules:
    - name: require-crew-label
      match:
        any:
          - resources:
              kinds: [Deployment, StatefulSet, DaemonSet]
      exclude:
        any:
          - resources: { namespaces: [kube-system, kyverno] }
      validate:
        failureAction: Audit          # recorded, never blocked -- that's the whole point of this step
        message: "Workloads need metadata.labels.crew -- which flight owns this?"
        pattern:
          metadata:
            labels:
              crew: "?*"              # ?* = at least one character

kubectl apply -f mission-control-baseline.yaml
kubectl get cpol mission-control-baseline

NAME                       BACKGROUND   VALIDATE ACTION   READY   AGE
mission-control-baseline   true         Audit             true    4s

Now create a Deployment that carries no crew label at all — the honest, fast thing a rushed engineer actually does:

kubectl create deployment ground-control --image=nginx:1.27 -n analytics
kubectl get deploy -n analytics ground-control

deployment.apps/ground-control created

NAME             READY   UP-TO-DATE   AVAILABLE   AGE
ground-control   1/1     1            1           22s

It's running. Nothing rejected it. Now go find the part of Kyverno that actually noticed:

kubectl get polr -A -o wide

NAMESPACE   NAME                                     PASS   FAIL   WARN   ERROR   SKIP   AGE
analytics   cpol-mission-control-baseline-4f8c2b1a    0      1      0      0       0      31s

Done when: ground-control shows 1/1 Running in analytics despite carrying no crew label, and kubectl get polr -A -o wide shows a FAIL: 1 row for it that never stopped anything.

Flip failureAction to Enforce — and watch admission actually say no

☺ Like you're 10: Same monitor, same notebook — except now they're standing in the doorway.

Change exactly one line — failureAction: Audit to failureAction: Enforce — and reapply:

kubectl apply -f mission-control-baseline.yaml   # after editing failureAction to Enforce
kubectl get cpol mission-control-baseline

NAME                       BACKGROUND   VALIDATE ACTION   READY   AGE
mission-control-baseline   true         Enforce           true    1m12s

Try to create a second unlabeled Deployment:

kubectl create deployment ground-control-2 --image=nginx:1.27 -n analytics

Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:

resource Deployment/analytics/ground-control-2 was blocked due to the following policies

mission-control-baseline:
  require-crew-label: 'validation error: Workloads need metadata.labels.crew --
    which flight owns this? rule require-crew-label failed at path /metadata/labels/crew/'

No object was ever created — check kubectl get deploy -n analytics and ground-control-2 simply isn't there. Notice the message: it's the exact string you wrote in the policy's message field, not a generic Kubernetes error. That's not an accident — a rejection nobody can act on without opening a runbook is barely better than no rejection at all.

Done when: the second create attempt is refused outright, the error names require-crew-label and repeats your own message text, and kubectl get deploy -n analytics confirms ground-control-2 was never created.

Prove the rule isn't just blocking everything

☺ Like you're 10: The doorway monitor lets through anyone who actually has a pass — check that this is really true, don't just assume it.

A policy that blocks every create looks identical, from a five-second glance at a dashboard, to a policy that correctly blocks only the bad ones. Prove the difference by giving it what it actually asked for:

# ground-control-3.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ground-control-3
  namespace: analytics
  labels:
    crew: apollo
spec:
  replicas: 1
  selector: { matchLabels: { app: ground-control-3 } }
  template:
    metadata:
      labels: { app: ground-control-3, crew: apollo }
    spec:
      containers:
        - name: web
          image: nginx:1.27

kubectl apply -f ground-control-3.yaml
kubectl get polr -n analytics -o wide

deployment.apps/ground-control-3 created

NAMESPACE   NAME                                     PASS   FAIL   WARN   ERROR   SKIP   AGE
analytics   cpol-mission-control-baseline-4f8c2b1a    1      1      0      0       0      3m40s

PASS went from 0 to 1 while FAIL stayed at the single violation from before Enforce ever took effect — proof the rule is discriminating, not just refusing on principle.

Done when: ground-control-3 is created cleanly with no error, and the PolicyReport's PASS count increases without its FAIL count moving.

Turn on PolicyExceptions, deliberately and scoped

☺ Like you're 10: The hall pass system doesn't exist yet — the principal has to actually turn it on first, and decide which office is allowed to print one.

PolicyExceptions are disabled by default, on purpose: a platform that lets every tenant quietly loosen the rule they don't like is a platform with no rule at all. Turning the feature on is a deliberate, auditable admin action — two flags on the admission controller, not a default anyone stumbles into.

# chart values move between Kyverno releases -- confirm the exact key on your version
# before trusting any --set path, this one included
helm show values kyverno/kyverno | grep -A6 "extraArgs"

# the flag names themselves are stable across chart layouts: enablePolicyException
# and exceptionNamespaces -- restrict the second one tightly
helm upgrade kyverno kyverno/kyverno -n kyverno --reuse-values \
  --set-string admissionController.extraArgs.enablePolicyException=true \
  --set-string admissionController.extraArgs.exceptionNamespaces=platform-exceptions

kubectl -n kyverno rollout status deploy/kyverno-admission-controller --timeout=120s

Confirm it actually landed by reading the running container's own arguments, which is true regardless of which values path your chart version happened to use:

kubectl -n kyverno get deploy kyverno-admission-controller \
  -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep -i exception

--enablePolicyException=true
--exceptionNamespaces=platform-exceptions

--exceptionNamespaces=platform-exceptions is doing real work here: it means an exception can only ever be created inside that one namespace, so a tenant cannot self-grant an exemption from inside their own namespace no matter what the exception's own match block says.

Done when: both flags show up in the admission controller's live container args, exactly as set.

Try the legitimate case anyway — and watch it get rejected first

☺ Like you're 10: Before you print anyone a hall pass, make them actually get stopped at the door once — proves you're writing the pass because it's needed, not because you assumed it.

Say a batch workload genuinely can't carry a crew label yet — it's a shared vendor image nobody on your team templates. Resist the urge to write its exception first. Create it under Enforce exactly as it stands today:

kubectl create deployment batch-runner --image=apache/airflow:2.9.3 -n analytics

Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:

resource Deployment/analytics/batch-runner was blocked due to the following policies

mission-control-baseline:
  require-crew-label: 'validation error: Workloads need metadata.labels.crew --
    which flight owns this? rule require-crew-label failed at path /metadata/labels/crew/'

Good — same rejection, same message, no favoritism. This is the step most people skip on a real platform, and skipping it is how "exceptions" quietly turn into "things I assumed would be a problem and pre-exempted without ever checking." An exception you write against a rejection you actually watched happen is a different, more honest artifact than one you write speculatively.

Done when: batch-runner is rejected with the identical require-crew-label message, proving the rule treats it no differently than ground-control-2 earlier.

Write the scoped exception, then prove the scope

☺ Like you're 10: One pass, one kid's name on it, one door it's good for — check that it doesn't quietly work for the kid standing next to them too.

Write a PolicyException that names exactly one policy, one rule, and one resource — not a namespace, not a label selector, one literal name:

# batch-runner-exempt.yaml
apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: batch-runner-exempt
  namespace: platform-exceptions      # the one namespace --exceptionNamespaces allows
spec:
  exceptions:
    - policyName: mission-control-baseline
      ruleNames: [require-crew-label]
  match:
    any:
      - resources:
          kinds: [Deployment]
          namespaces: [analytics]
          names: [batch-runner]        # exactly one workload, never a whole namespace

kubectl apply -f batch-runner-exempt.yaml
kubectl get polex -n platform-exceptions

kubectl create deployment batch-runner --image=apache/airflow:2.9.3 -n analytics

policyexception.kyverno.io/batch-runner-exempt created

NAME                    AGE
batch-runner-exempt     3s

deployment.apps/batch-runner created

It went through this time — same manifest, same missing label, only the exception changed. Now prove the exception is exactly as narrow as its match block claims, by testing a workload it does not name:

kubectl create deployment ground-control-4 --image=nginx:1.27 -n analytics

Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:

resource Deployment/analytics/ground-control-4 was blocked due to the following policies

mission-control-baseline:
  require-crew-label: 'validation error: Workloads need metadata.labels.crew --
    which flight owns this? rule require-crew-label failed at path /metadata/labels/crew/'

Done when: batch-runner applies cleanly with the exception in place, and ground-control-4 — a different, unrelated unlabeled Deployment in the same namespace — is still rejected by the same rule and the same message.

Audit told you the truth without stopping anything, Enforce actually stopped it, and one narrow PolicyException let a real exception through without loosening the rule for anyone else — that's the whole lifecycle a guardrail goes through on a real platform. For the full architecture underneath what you just did — the four controllers, all four rule types, autogen, and every gotcha this drill deliberately avoided — see the Kyverno lesson; for the vendor-neutral argument for policy-as-code at all, see Policy-as-Code Philosophy. This exact rule reappears, wired into a real mesh and a real cluster, in Capstone Part 3 — Mesh & Policy, and the full domain breakdown behind it is on the KCA blueprint. If Kubernetes admission control and RBAC themselves are still shaky underneath this, the sibling Kubernetes course's CKS material covers that ground in depth. Ready for a different single skill? Try Drill — Trace a Broken Telemetry Pipeline, or step back to the capstone hub for the connected, six-part version.