Kyverno

The KCA blueprint puts 32% of its weight — more than the next two domains combined — on the single competency of writing policies, and shows the four rule types worked together as one file. This page pulls them apart again: each rule type gets its own worked example, its own gotcha, and enough surrounding architecture to explain why it behaves the way it does — the four controllers behind the engine, how a validate rule's writ never quite reaches a resource that already existed until a background scan says so, how generate keeps a promise long after the namespace that triggered it was created, and what a PolicyException actually buys you that editing the policy doesn't. The admission-control model underneath all of it — two windows, mutating then validating, before anything reaches etcd — belongs to every engine that can occupy those slots, not just this one, and has its own page; this one assumes that model and gets straight to Kyverno's specific fields.

Architecture: four controllers, and a webhook that grows itself

☺ Like you're 10: One crew member stands at the door checking everyone as they arrive; a second one quietly re-checks everyone already inside; a third writes up what both of them found; a fourth just tidies up on a schedule.

A modern Kyverno install splits the work across four separate controllers, and knowing which owns which job turns a confusing outage into a fast diagnosis. The admission controller serves the mutating and validating webhooks and is the only one actually sitting in the live request path — scale it first, and give it a PodDisruptionBudget, because its availability is now part of your cluster's availability. The background controller does two unrelated-sounding jobs that share one property — neither happens at admission time: it periodically re-evaluates validate rules against resources that already existed before a policy arrived, and it drives every generate and mutate-existing rule, queuing the work as an internal UpdateRequest object before applying it. The reports controller aggregates results from both of the above into PolicyReport and ClusterPolicyReport objects. The cleanup controller is the odd one out — it doesn't touch admission or reports at all, it just runs CleanupPolicy and ClusterCleanupPolicy resources on their own cron schedule, deleting whatever matches.

One detail explains why Kyverno stays cheap at scale: it dynamically rewrites its own webhook configuration to cover only the resource kinds your currently installed policies actually reference. Install one policy about Pod, and the API server only ever calls Kyverno for Pod admission — every other kind never pays the latency cost of a call that would do nothing.

The resource you write: ClusterPolicy, Policy, and how a rule gets selected

☺ Like you're 10: One rulebook works everywhere in the station; a smaller one only covers a single room — and every rule inside either one says exactly who it's checking before it says anything else.

Two kinds, identical schema, different reach. ClusterPolicy (cpol) applies across every namespace and is the only one that can match a cluster-scoped resource like Namespace at all — write a Policy targeting Namespace and it silently matches nothing, because a namespaced object can't reach outside its own namespace to begin with. Policy (pol) is confined to the namespace it lives in, the way a tenant is handed ownership of rules for their own space without needing cluster-wide access to write them.

Both hold spec.rules[], and every rule needs a match — any or all lists of filters on kinds, namespaces, names, label selectors, subjects and operations — with an optional exclude block that removes resources the way a spotlight removes shadow. preconditions sit beside match as a cheaper, JMESPath-driven gate for logic match can't express — skip this rule entirely unless X — and one precondition idiom is worth memorizing before you ever hit it in the wild: {{ request.operation || 'BACKGROUND' }}. A background scan carries no live admission request, so request.operation simply doesn't exist there; the || default catches that case and resolves to the literal string "BACKGROUND" instead of erroring the rule out. spec.background — true by default — is what lets the background controller pick a validate rule up at all for resources that predate the policy; set it to false deliberately for any rule that reads admission-only context like request.userInfo, because that context genuinely does not exist outside a live request and the rule would misbehave silently otherwise.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: mission-control-baseline
spec:
  background: true                       # scan existing resources too — the default
  rules:
    - name: require-crew-label
      match:
        any:
          - resources:
              kinds: [Deployment, StatefulSet, DaemonSet]
      exclude:
        any:
          - resources: { namespaces: [kube-system, kyverno] }
      preconditions:
        all:
          - key: "{{ request.operation || 'BACKGROUND' }}"
            operator: AnyIn
            value: [CREATE, UPDATE, BACKGROUND]
      validate:
        failureAction: Enforce           # current per-rule field; older spec.validationFailureAction still appears in the wild
        message: "Workloads need metadata.labels.crew — which flight owns this?"
        pattern:
          metadata:
            labels:
              crew: "?*"                 # ?* = at least one character
              =(replaced-by): "?*"       # =() : conditional anchor — IF this key exists, it must match too

That last line is a conditional anchor: =(field) means "only enforce this sub-pattern if the field is actually present" — useful for optional metadata you want validated when it exists without making it mandatory. Two siblings show up constantly in the policy library: +(field), "add this if absent," which only means something inside a mutate patch, and X(field), the negation anchor — the field must not be present at all. Getting the match/exclude selector subtly wrong, not a broken anchor, is the single most common reason a policy "does nothing" in practice — always prove a new rule against a resource you know should fail before trusting one that reports clean.

The four rule types, each doing a genuinely different job

☺ Like you're 10: Reject it, quietly fix it, print something new to go with it, or check its seal — one rule only ever does one of these.

Every rule carries exactly one of the four. Below, each gets its own worked example rather than one shared file, because the fields that matter differ completely between them.

Validate — say no, with a message someone can act on

This is the rule type the exam and real incidents both reach for most. Three fields carry the actual behavior: failureAction: Enforce — without it, nothing is ever blocked, only recorded — a match that genuinely selects the right kinds, and a message a human can act on without opening a runbook. pattern does structural matching; deny.conditions expresses arbitrary boolean logic pattern can't, and newer Kyverno also accepts a cel block using the same Common Expression Language Kubernetes itself uses natively in ValidatingAdmissionPolicy — worth learning once, because it transfers directly.

# deny with explicit conditions — reject the ':latest' tag outright
- name: block-latest-tag
  match:
    any:
      - resources: { kinds: [Pod] }
  validate:
    failureAction: Enforce
    message: "The ':latest' tag is not allowed — pin a version or a digest."
    deny:
      conditions:
        any:
          - key: "{{ images.containers.*.tag }}"
            operator: AnyIn
            value: ["latest"]

Mutate — fix it instead of rejecting it

Mutation turns Kyverno from a bouncer into a butler: instead of bouncing a request back for a missing field, it patches the field in and lets the request through. patchStrategicMerge reads like the manifest it's altering; patchesJson6902 is an RFC 6902 JSON patch for anything a strategic merge can't express, like inserting into a specific array index. A separate targets block lets a mutate rule act on something other than the object that triggered it — "mutate existing" — and the spec-level mutateExistingOnPolicyUpdate: true makes that fire again whenever the policy itself changes, not only when the trigger does.

- name: default-pull-policy
  match:
    any:
      - resources: { kinds: [Pod] }
  mutate:
    patchesJson6902: |-
      - op: add
        path: "/spec/containers/0/imagePullPolicy"
        value: IfNotPresent

- name: add-managed-by-label
  match:
    any:
      - resources: { kinds: [Deployment, StatefulSet] }
  mutate:
    patchStrategicMerge:
      metadata:
        labels:
          +(app.kubernetes.io/managed-by): mission-control   # +() = add only if absent

Generate — bootstrap something automatically

generate creates a companion resource the moment a trigger appears — the canonical, genuinely valuable example being a default-deny NetworkPolicy for every new namespace, so tenant isolation is the default instead of a task someone forgets in week one. data defines the resource inline; clone copies an existing object instead — the right shape for a registry pull secret you maintain in exactly one place. synchronize: true keeps the generated copy in lockstep with its source going forward, on both sides of the relationship.

- name: default-deny-per-namespace
  match:
    any:
      - resources: { kinds: [Namespace] }          # cluster-scoped target → must be a ClusterPolicy
  exclude:
    any:
      - resources: { names: [kube-system, kube-public, kyverno] }
  generate:
    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    name: default-deny
    namespace: "{{ request.object.metadata.name }}"  # a variable, resolved at runtime
    synchronize: true
    data:
      spec:
        podSelector: {}
        policyTypes: [Ingress, Egress]

- name: copy-registry-credentials
  match:
    any:
      - resources: { kinds: [Namespace] }
  generate:
    apiVersion: v1
    kind: Secret
    name: registry-credentials
    namespace: "{{ request.object.metadata.name }}"
    synchronize: true
    clone:                       # clone an existing object — never inline a secret in policy YAML
      namespace: platform-system
      name: registry-credentials

verifyImages — trust the seal, not the tag

This is where Kyverno meets the software supply chain: no valid signature, no pod. Keyless verification is the modern default — instead of managing a public key, you assert who signed the image and which OIDC issuer vouched for that identity, checked against the Fulcio certificate authority and the Rekor transparency log the same way Cosign & Sigstore covers in full. mutateDigest: true is on for a reason: a successful verification rewrites the tag to the exact digest that was just verified, which is why this rule type lives in the mutating phase rather than the validating one — closing the tag-mutation race where a tag gets re-pointed at a different image after the check ran.

- name: require-keyless-signature
  match:
    any:
      - resources: { kinds: [Pod] }
  verifyImages:
    - imageReferences: ["ghcr.io/mission-control/*"]  # scope it — never verify public base images this way
      mutateDigest: true              # rewrite tag -> digest once verified
      required: true
      attestors:
        - count: 1
          entries:
            - keyless:
                subject: "https://github.com/mission-control/*/.github/workflows/*"
                issuer: "https://token.actions.githubusercontent.com"
                rekor: { url: https://rekor.sigstore.dev }

Autogen: one Pod rule quietly becomes five

Write a rule that matches Pod, and Kyverno doesn't stop there — it automatically generates equivalent rules, rewritten against .spec.template.spec, for the controllers that actually create pods: Deployment, StatefulSet, DaemonSet, Job, and CronJob. Without it, a developer who deploys a broken Deployment would see the Deployment accepted cleanly and only the pods underneath silently fail to schedule — a confusing failure with no useful message anywhere near where the person actually looked. Autogen makes the rejection land at the exact object the developer applied. It only fires for a rule that matches Pod specifically and doesn't already carry an explicit rule for that controller kind; steer or disable it entirely with the pod-policies.kyverno.io/autogen-controllers annotation on the policy. The generated rules carry an autogen- prefix in reports, which is worth knowing before it shows up unexplained in your first kubectl get polr.

Background scans, PolicyReports, and PolicyExceptions

☺ Like you're 10: A second pass quietly re-checks everything already inside, writes up what it found on a report card, and there's one very specific way to excuse a single student from a single rule without changing the rule for the whole class.

Background scanning is what closes the gap admission control structurally can't: a policy installed today has zero effect on the thousand resources already running, because admission only fires on CREATE and UPDATE. The background controller periodically re-evaluates every validate rule with background: true — the default — against the resources already sitting in the cluster, and writes what it finds into a BackgroundScanReport, which the reports controller folds into the same PolicyReport/ClusterPolicyReport objects admission-time results land in. This is the honest way to measure a fleet before you ever flip a rule to Enforce: switching enforcement on doesn't retroactively fix anything already running, it only blocks the next change to it — so read the report first, or you'll find out the hard way during an incident that two hundred resources were already in violation.

kubectl get polr -A -o wide       # PolicyReports — one per namespace, pass/fail/warn/skip/error
kubectl get cpolr                 # ClusterPolicyReports — the cluster-scoped equivalent
kubectl describe polr -n payments # per-resource, per-rule detail — read the FAIL column first

PolicyExceptions are the deliberate, reviewable escape hatch: one named workload skips one named rule, through Git and code review, while the policy stays strict for absolutely everyone else. They're disabled by default — a platform admin turns them on with --enablePolicyException=true on the admission controller and restricts where they can be created at all with --exceptionNamespaces, so an exception isn't something any tenant can self-grant from inside their own namespace.

apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: batch-runner-exempt
  namespace: platform-exceptions      # wherever --exceptionNamespaces allows
spec:
  exceptions:
    - policyName: mission-control-baseline
      ruleNames: [require-crew-label]
  match:
    any:
      - resources:
          kinds: [Deployment]
          namespaces: [analytics]
          names: [batch-runner]        # exactly one workload, never a whole namespace

The kyverno CLI: apply, test, jp

☺ Like you're 10: Rehearse the rule against a stack of pretend toys before you ever let it near the real conveyor belt.

The standalone kyverno binary evaluates policies against manifests without a cluster, which is what makes changing a policy something you can safely do in a pull request rather than by hand against production. The curriculum names exactly three verbs worth having in muscle memory.

# apply — "what would these policies do to these resources?"
kyverno apply ./policies/ --resource ./manifests/
kyverno apply ./policies/ --resource ./manifests/ --policy-report
kyverno apply ./policies/ --cluster --policy-report     # dry-run against a LIVE cluster, nothing enforced

# test — declarative unit tests, discovered recursively as kyverno-test.yaml
kyverno test ./policies/

# jp — debug the expression language before it ever reaches a rule
kyverno jp query -i pod.yaml 'spec.containers[*].image'
kyverno jp function                                       # list Kyverno's custom JMESPath functions

# kyverno-test.yaml — pairs a policy with resources and asserts the expected result per rule
apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
  name: mission-control-baseline-test
policies:
  - ../policies/mission-control-baseline.yaml
resources:
  - resources/good-deployment.yaml
  - resources/unlabeled-deployment.yaml
results:
  - policy: mission-control-baseline
    rule: require-crew-label
    resources: [good-deployment]
    kind: Deployment
    result: pass
  - policy: mission-control-baseline
    rule: require-crew-label
    resources: [unlabeled-deployment]
    kind: Deployment
    result: fail

Wire kyverno test into whatever pipeline guards the policy repository itself, and a policy that no longer catches what it claims to catch fails CI before it ever reaches a cluster — the same discipline applied to application code, applied here to governance.

Gotchas and failure modes

☺ Like you're 10: Most surprises come from the guard being unreachable, a permission nobody granted, two rules disagreeing about who goes first, or forgetting that the guard only checks things as they walk in, never things already standing there.

A Kyverno outage can become a cluster outage

Kyverno sits in the API server's request path, so its own availability quietly becomes part of your cluster's. failurePolicy on its webhook configuration decides what happens when Kyverno can't be reached at all: Fail rejects the request — secure, but a Kyverno outage now blocks every matching write, including, in the worst case, the very fix that would bring Kyverno back. Ignore lets the request through unchecked — available, but your guardrail disappears at precisely the moment things are already going wrong. Run enough admission-controller replicas to survive a node loss, and always exclude kube-system and Kyverno's own namespace from anything that could plausibly block Kyverno fixing itself.

Generate needs RBAC you have to grant explicitly

Kyverno cannot create a resource it holds no permission for. Generating a plain NetworkPolicy works out of the box; generating a custom resource owned by another operator usually doesn't, and the symptom is deceptively quiet — the policy reports healthy, the rule shows no error, and nothing ever appears. The fix is a ClusterRole, aggregated into the background controller's role, naming exactly the group/kind it needs to create. This is ordinary Kubernetes RBAC, not a Kyverno-specific mechanism — the sibling Kubernetes course's CKS material covers the RBAC model itself in depth if it's rusty.

Enforcement never applies retroactively

Flipping a rule from Audit to Enforce does not touch the resources already running in violation of it — admission control only ever fires on the next create or update. The background scan will keep reporting those violations forever unless someone actually fixes or exempts them; it does not, and cannot, reach in and correct a resource that's already live. Read the PolicyReport before every enforcement flip, not after.

Ordering across policies is not something to depend on

Rules within one policy run in the order they're written, but ordering across separate ClusterPolicy objects carries no such guarantee — never design a rule whose correctness depends on another policy having already run first. Kyverno also shares the mutating-webhook chain with anything else registered on the same kinds, a mesh sidecar injector among them; depending on call order, a validate rule may or may not ever see a container another webhook added afterward.

Kyverno vs the alternatives

☺ Like you're 10: A few different kinds of guard exist for the same door. They trade off how much you have to teach them, how far past this one cluster they can see, and whether they even need to stand at the door at all.

The practical rule most platforms land on: reach for Kyverno first when the guardrails only ever need to look at Kubernetes objects, because generate and verifyImages turn governance into a paved road instead of a wall, and nothing else on this table offers both natively. Reach for OPA/Gatekeeper specifically when the same policy brain has to reason about something that isn't a Kubernetes object at all — Policy-as-Code Philosophy makes that fork the center of its argument, in depth. And treat the native CEL policies as a scalpel for the hot path, not a replacement for either engine: no generate, no image verification, on either of them, as of the versions this page was written against.