Kyverno

Kyverno is a policy engine built for Kubernetes and nothing else, and its whole pitch is that a policy is just another Kubernetes resource written in the YAML you already know — no new language to learn, no separate runtime to operate. It solves the platform problem of “we agreed nobody would run privileged containers, and we agreed every namespace gets a NetworkPolicy, and neither thing is actually true in the cluster,” by turning those agreements into resources that the API server enforces on every single request, forever, whether or not anyone is watching.

What Kyverno is and the problem it solves

☺ Like you’re 10: It’s a doorkeeper for your cluster whose rulebook is written in ordinary Kubernetes YAML, so anyone can read it.

Kyverno is a CNCF project — it moved up from the sandbox to incubating maturity in 2022 — that runs inside your cluster as a set of controllers and registers itself with the Kubernetes API server as an admission webhook. Every create, update and delete that reaches the API server can be routed through it, and Kyverno consults the policies you have installed to decide what happens next: allow it, reject it with a message, quietly modify it first, or generate some other resource alongside it.

The problem before policy-as-code

Every platform accumulates rules. Images must come from the company registry. Every workload must set resource requests, or the scheduler makes bad decisions and cost goes sideways. Every namespace needs an owner label so the on-call knows who to page. Without an engine those rules live in three unreliable places — a wiki page, a review checklist, and someone’s memory — and all three fail the same way: they scale with human attention, and human attention does not scale.

Policy-as-code moves those rules into version control and enforces them at the one chokepoint every change must pass: the API server’s admission path. Nothing enters the cluster without being checked, no matter who or what sent it — a developer, a CI job, an Argo CD sync, or a Crossplane composition.

Policies are ordinary Kubernetes resources

The design decision that defines Kyverno is that it invented no policy language. A Kyverno policy is a Kubernetes custom resource containing YAML that looks like a fragment of the manifest it is checking. If you want to require that securityContext.runAsNonRoot is true, you write a pattern containing securityContext: {runAsNonRoot: true} and Kyverno matches it structurally. That is the entire mental model, and it is why a policy written by the platform team is legible to the application team it constrains — a property that matters far more in practice than it sounds on a slide.

What it is not

Kyverno is not a general-purpose decision engine. Unlike plain OPA, you cannot point it at your microservice and ask it to authorize an HTTP request — it speaks Kubernetes objects only. It is not a runtime detector either: it decides what may enter the cluster, not what a container does once it is running, which is Falco’s job. And it is not a scanner — it will happily verify that an image is signed, but discovering that the image is full of CVEs belongs to Trivy in your pipeline.

Where it fits in a platform

☺ Like you’re 10: It sits right at the cluster’s front door, so everything else in the platform has to walk past it.

Kyverno belongs to the platform’s governance and control plane, not its data plane. It runs alongside the API server as an extension of it, which puts it in an unusually powerful and unusually dangerous position: it sees every request, and if it misbehaves it can block every request. Architecturally it is the enforcement half of the pairing that makes self-service safe — you can hand developers a wide-open namespace precisely because the guardrails are automatic rather than procedural.

Its neighbours

Upstream sits GitOps: policies are manifests, so Argo CD or Flux delivers them like anything else, and self-heal means nobody can quietly delete a guardrail. Beside it sits its direct alternative, OPA Gatekeeper, and the built-in floor beneath both, Pod Security Admission. Downstream, Sigstore/cosign produces the signatures Kyverno’s verifyImages rules verify. Kyverno also polices other tools’ resources — a Crossplane claim without a cost-centre label or a NetworkPolicy-free namespace are just objects — and its PolicyReport output feeds Governance & Compliance evidence and Prometheus alerting.

Why platform teams reach for it

Three reasons, in the order teams discover them. The learning curve: a team that has never written Rego ships a useful policy in an afternoon. Then mutate and generate — fixing a problem rather than merely rejecting it turns policy from an obstacle into a paved road, and a namespace that arrives with a default-deny NetworkPolicy, a pull secret and a quota already in it is a self-service feature, not a restriction. And finally the curated library at kyverno.io/policies, which means most rules you want are already written and tested.

CNPE domain relevance

Kyverno is on the official CNPE tool list and sits squarely in Security & Policy Enforcement — 15% of the exam — alongside OPA/Gatekeeper, Istio and Linkerd. It also brushes Platform APIs & Self-Service (25%), because policies constrain the custom resources a platform exposes, and the governance and multi-tenancy concerns inside Platform Architecture & Infrastructure (15%). The lesson page to study first is Security & Policy Enforcement; drill the manifests on Know Cold.

How it works — architecture and CRDs

☺ Like you’re 10: Kyverno plugs into the API server twice — once to change things on the way in, and once to say yes or no.

The Kubernetes admission path has two phases, and Kyverno registers in both. Mutating webhooks run first and may change the incoming object. Validating webhooks run second and may only accept or reject it. That ordering is fixed by Kubernetes, and it is why a Kyverno mutate rule can add a default that a Kyverno validate rule then successfully checks — the object being validated is the mutated one.

The controllers

A modern Kyverno install splits into four deployments, and knowing which is which turns a confusing outage into a five-minute diagnosis. The admission controller serves the webhooks and is the only one in the request path — if it is unavailable, admission is affected. The background controller handles background scanning of already-running resources plus generate and mutate-existing rules. The reports controller writes and maintains the PolicyReport objects. The cleanup controller serves CleanupPolicy resources, which delete matching objects on a cron schedule. Splitting them means a heavy background scan cannot starve the admission path.

One important detail: Kyverno dynamically configures its own webhook rules to match only the resource kinds your installed policies actually reference. Install one policy about Pods and the API server only calls Kyverno for Pods. That is a large part of why it stays cheap.

The custom resources it introduces

The four rule types

Every rule inside spec.rules[] has a name, a match (and optionally exclude) block that selects resources, optional preconditions, and exactly one of four actions. This table is worth memorising — it is the shape of the whole tool.

verifyImages living in the mutating phase surprises people. It is there because a successful verification can rewrite the image tag to its digest (mutateDigest, on by default), which pins you to the exact bytes you verified and closes the tag-mutation race.

Escape hatches: exceptions and CEL

Two newer features matter for real platforms. A PolicyException lets a specific workload skip a specific rule without weakening the policy for everyone — it is a resource, so it goes through review and GitOps rather than a Slack message. And recent Kyverno versions let a validate rule express its logic as CEL expressions, the same language Kubernetes uses natively in ValidatingAdmissionPolicy; Kyverno can even generate those native policies for you, moving enforcement into the API server itself with no webhook in the path.

# A CEL-based validate rule, plus a reviewable exception for one workload
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-replicas
spec:
  rules:
    - name: at-least-two-replicas
      match:
        any:
          - resources: {kinds: [Deployment]}
      validate:
        failureAction: Enforce          # newer per-rule form of validationFailureAction
        message: "Deployments must run at least 2 replicas."
        cel:
          expressions:
            - expression: "object.spec.replicas >= 2"
---
apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: batch-runner-exempt
  namespace: analytics
spec:
  exceptions:
    - policyName: require-replicas
      ruleNames: [at-least-two-replicas]
  match:
    any:
      - resources:
          kinds: [Deployment]
          namespaces: [analytics]
          names: [batch-runner]         # exactly one workload, not a whole namespace

The resources you will actually write

☺ Like you’re 10: Four shapes: say no, quietly fix it, make a new thing, and check the seal on the box.

Validate — say no, with a useful message

This is the rule type the exam is most likely to want. Note the three things that carry the marks: failureAction: Enforce (nothing is blocked without it), a match block that actually selects the right kinds, and a message a human can act on. The * wildcard means “every element of this list,” and the ? in a string pattern is a single-character wildcard.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: baseline-workload-hygiene
spec:
  background: true                       # also evaluate resources already running
  rules:
    # ---- 1. structural pattern match ----
    - name: require-non-root-and-limits
      match:
        any:
          - resources:
              kinds: [Pod]
      exclude:
        any:
          - resources:
              namespaces: [kube-system, kyverno]   # never police the police
      validate:
        failureAction: Enforce
        message: "Containers must run as non-root and declare CPU/memory limits."
        pattern:
          spec:
            containers:
              - securityContext:
                  runAsNonRoot: true
                resources:
                  limits:
                    memory: "?*"         # ?* = a non-empty string, i.e. "must be set"
                    cpu: "?*"

    # ---- 2. deny with explicit conditions ----
    - name: block-latest-tag
      match:
        any:
          - resources:
              kinds: [Pod]
      preconditions:
        all:
          - key: "{{request.operation || 'BACKGROUND'}}"   # the || default: no request in a background scan
            operator: AnyIn
            value: [CREATE, UPDATE]
      validate:
        failureAction: Enforce
        message: "Using the ':latest' tag is not allowed — pin a version or digest."
        deny:
          conditions:
            any:
              - key: "{{ images.containers.*.tag }}"
                operator: AnyIn
                value: ["latest"]

Two more shapes are worth recognising. anyPattern takes a list and passes if any one pattern matches — the way you express “either a ClusterIP service or an annotated LoadBalancer.” And anchors change matching semantics: =(field) means “if this key exists it must match,” +(field) is add-if-absent in mutate rules, and X(field) is the negation anchor — the field must not be present at all.

Mutate — fix it instead of rejecting it

Mutation is what turns Kyverno from a bouncer into a butler. The first rule below adds a default label to every new Deployment; the second demonstrates mutate existing, where a targets block makes the rule modify something other than the object that triggered it, and the spec-level mutateExistingOnPolicyUpdate: true makes it fire on policy create/update as well as on the trigger.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: platform-defaults
spec:
  mutateExistingOnPolicyUpdate: true       # spec-level: also act when the POLICY changes, not only the trigger
  rules:
    - name: add-owner-label
      match:
        any:
          - resources: {kinds: [Deployment, StatefulSet]}
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              +(app.kubernetes.io/managed-by): platform    # +() = add only if absent
              +(cost-centre): "unassigned"

    - name: drop-service-account-token
      match:
        any:
          - resources: {kinds: [Pod]}
      mutate:
        patchesJson6902: |-
          - op: add
            path: "/spec/automountServiceAccountToken"
            value: false

    - name: label-existing-namespaces
      match:
        any:
          - resources: {kinds: [ConfigMap], names: [platform-config]}
      mutate:
        targets:                                # targets = mutate something OTHER than the trigger
          - apiVersion: v1
            kind: Namespace
            name: "{{request.object.metadata.namespace}}"
        patchStrategicMerge:
          metadata:
            labels:
              policy.acme.io/reviewed: "true"

Generate — the paved road, automatically

A generate rule creates a resource in response to another resource appearing. The classic and genuinely valuable example is giving every new namespace a default-deny NetworkPolicy so that tenant isolation is the default rather than a task someone forgets. data defines the resource inline; clone copies an existing one (ideal for a registry pull secret you maintain in one place); and synchronize: true means Kyverno keeps the generated copy in step with the source and puts it back if someone deletes it.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: namespace-bootstrap
spec:
  rules:
    - name: default-deny-networkpolicy
      match:
        any:
          - resources: {kinds: [Namespace]}     # cluster-scoped → must be a ClusterPolicy
      exclude:
        any:
          - resources: {names: [kube-system, kube-public, kyverno]}   # names, not namespaces — a Namespace has no namespace
      generate:
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        name: default-deny
        namespace: "{{request.object.metadata.name}}"
        synchronize: true                        # re-create it if someone deletes it
        data:
          spec:
            podSelector: {}                      # every pod in the namespace
            policyTypes: [Ingress, Egress]       # deny both directions by default

    - name: copy-registry-pull-secret
      match:
        any:
          - resources: {kinds: [Namespace]}
      exclude:
        any:
          - resources: {names: [kube-system, kube-public, kyverno, platform-system]}
      generate:
        apiVersion: v1
        kind: Secret
        name: registry-credentials
        namespace: "{{request.object.metadata.name}}"
        synchronize: true
        clone:                                   # clone, don't inline — no secrets in policy YAML
          namespace: platform-system
          name: registry-credentials

verifyImages — trust the seal, not the label

This is where Kyverno meets the supply chain. A verifyImages rule verifies a cosign signature before the pod is admitted. Keyless verification is the modern default: instead of a public key you assert who signed it and which OIDC issuer vouched for that identity, checked against the Fulcio certificate authority and the Rekor transparency log.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-signed-images
spec:
  webhookTimeoutSeconds: 30            # registry round-trips are slow; the default 10s is tight
  rules:
    - name: require-keyless-signature
      match:
        any:
          - resources: {kinds: [Pod]}
      verifyImages:
        - imageReferences:
            - "ghcr.io/acme/*"          # only our own images; public bases are excluded
          failureAction: Enforce
          mutateDigest: true            # rewrite tag -> digest once verified
          verifyDigest: true
          required: true
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/acme/*/.github/workflows/*"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev

Day-to-day commands

☺ Like you’re 10: Mostly plain kubectl — plus one extra command for testing policies before they go anywhere near a real cluster.

Inspecting policies and results in a cluster

kubectl get clusterpolicies                 # short name: cpol
kubectl get policies -A                     # short name: pol (namespaced)
kubectl describe cpol baseline-workload-hygiene
kubectl get cpol baseline-workload-hygiene -o yaml | head -40

# results — the report card
kubectl get polr -A                         # PolicyReports, one per namespace
kubectl get cpolr                           # ClusterPolicyReports
kubectl get polr -A -o wide                 # the FAIL column is who is currently violating something
kubectl describe polr -n payments           # per-resource, per-rule detail

# is the engine itself healthy?
kubectl -n kyverno get pods
kubectl -n kyverno logs deploy/kyverno-admission-controller --tail=100
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations | grep -i kyverno

# prove enforcement actually works — the step people skip
kubectl run bad --image=nginx:latest --dry-run=server   # expect a 403 with your message

# and the universal recovery move when memory fails
kubectl explain clusterpolicy.spec.rules --recursive | head -60

That last one deserves its own sentence. Because Kyverno’s CRDs register their full OpenAPI schema in the cluster, kubectl explain is a complete, version-correct offline reference for every field on this page. See the command reference for more of these.

The kyverno CLI — test before you deploy

The standalone kyverno binary evaluates policies against manifests without a cluster, which is what makes policy safe to change. kyverno apply answers “what would this policy do to these resources?” and kyverno test runs declarative unit tests.

# what would this policy do to this resource?
kyverno apply policy.yaml --resource pod.yaml
kyverno apply ./policies/ --resource ./manifests/ --policy-report   # emit a PolicyReport

# dry-run a whole policy set against a LIVE cluster before enforcing — the safest audit there is
kyverno apply ./policies/ --cluster --policy-report

# run declarative unit tests (looks for kyverno-test.yaml files recursively)
kyverno test ./policies/

# helpers
kyverno create --help               # list the scaffolds the CLI can generate
kyverno jp query -i pod.yaml 'spec.containers[*].image'   # debug a JMESPath expression
kyverno version

Policy unit tests in CI

A kyverno-test.yaml file pairs policies with resources and asserts the expected result for each rule. Wire kyverno test into the pipeline that guards your policy repo and a broken policy never reaches a cluster — the same discipline you apply to application code, applied to governance.

apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
  name: baseline-workload-hygiene-test
policies:
  - ../policies/baseline-workload-hygiene.yaml
resources:
  - resources/good-pod.yaml
  - resources/root-pod.yaml
results:
  - policy: baseline-workload-hygiene
    rule: require-non-root-and-limits
    resources: [good-pod]
    kind: Pod
    result: pass
  - policy: baseline-workload-hygiene
    rule: require-non-root-and-limits
    resources: [root-pod]
    kind: Pod
    result: fail

Gotchas and failure modes

☺ Like you’re 10: The doorkeeper can be too polite, too strict, or standing in the wrong order — all three cause real problems.

Audit forever — the most common failure of all

A policy in Audit applies cleanly, looks correct in kubectl get cpol, dutifully logs every violation into a PolicyReport — and blocks precisely nothing. Teams switch on a dozen policies in audit “for a two-week soak,” the backlog is never triaged, and eighteen months later the dashboard is a wall of red that everyone has learned to ignore. Audit is a runway, not a destination: put a date on every audit-mode policy, and track “policies still in audit” as debt on the platform’s own scorecard. If an exam task or a ticket says prevent, block or reject, audit mode is a zero.

Webhook availability — the failure mode that takes the cluster with it

Kyverno sits in the API server’s request path, so its availability becomes your cluster’s availability. The failurePolicy on its webhook configuration decides what happens when Kyverno cannot be reached: Fail rejects the request (secure, but a Kyverno outage stops all matching writes — including, potentially, the writes that would fix Kyverno), and Ignore lets it through unchecked (available, but your guardrail evaporates exactly when things are going wrong). Neither is free. Run at least three admission-controller replicas with a PodDisruptionBudget, always exclude kube-system and Kyverno’s own namespace, and keep webhookTimeoutSeconds honest — a verifyImages rule reaching a slow registry inside a 10-second budget starts failing admission under load. Unpicking this class of symptom is covered in Triage: Delivery.

Mutate ordering and autogen surprises

Mutation rules inside a single policy execute in the order they are declared, but ordering across policies is not something you should rely on — so never write one policy that depends on another policy’s output. Kyverno also competes with other mutating webhooks: a service-mesh sidecar injector may run after Kyverno, meaning a container Kyverno never saw appears in the final pod, and your validate rule may or may not catch it depending on which webhook the API server calls last.

Then there is autogen. When a policy matches Pod, Kyverno silently generates equivalent rules for the pod controllers — Deployment, DaemonSet, StatefulSet, Job, CronJob — so that the rejection happens at the Deployment rather than mystifying you as a Deployment with zero pods and no error. This is almost always what you want, but it means the rule you read in the YAML is not the only rule running, and the generated rule names differ (autogen-…) which is confusing the first time you see it in a report. Narrow it with the pod-policies.kyverno.io/autogen-controllers annotation when you need to.

The quieter traps

Background scanning has no admission request behind it, so nothing derived from one is available. A rule referencing {{request.userInfo}}, {{request.roles}} or {{serviceAccountName}} is rejected outright unless the policy sets background: false, and {{request.operation}} simply resolves to nothing during a scan — which is why library policies write {{request.operation || 'BACKGROUND'}} and gate on the result. Policies never apply retroactively at admission, so switching on an Enforce rule does not fix the two hundred workloads already running; only the next update to them is blocked, which is a nasty surprise during an incident. A ClusterPolicy is required to match cluster-scoped resources such as Namespace — a namespaced Policy silently will not. And validate.pattern with a typo in a field name does not error; it simply matches nothing and passes everything, which is why kyverno test with a deliberately bad resource matters more than a policy that “applied successfully.”

Alternatives and when to choose it

☺ Like you’re 10: There are four ways to have rules. One is free but blunt, one is powerful but hard, one is built into Kubernetes, and Kyverno sits in the friendly middle.

The comparison that decides it

A practical rule

Start with Pod Security Admission — it is free, always on, and a sane floor for every tenant namespace; do not run a policy engine to reproduce what three labels already give you. Add Kyverno when you need rules PSA cannot express, and especially when you want mutate and generate, because those two turn governance into a paved road rather than a wall. Choose OPA/Gatekeeper instead when you already have Rego expertise or, more compellingly, when you need the same policy language across Kubernetes, your CI pipelines, your Terraform plans and your application authorization — that portability is Rego’s real advantage and Kyverno has no answer to it. Reach for native ValidatingAdmissionPolicy for simple, hot-path checks where removing a webhook from the critical path is worth losing expressiveness. See The Tool Landscape for how these sit among the other projects on the CNPE tool list, and Security & Policy Enforcement for the domain around them.

Exam relevance and going further

☺ Like you’re 10: On exam day you cannot open Kyverno’s website — so the YAML has to already be in your head.

Kyverno is on the official CNPE tool list. Expect a task that asks you to prevent something — non-root violations, missing labels, the :latest tag, an unsigned image — or to add a default with mutate, or to bootstrap a namespace with generate. Expect also to be asked to diagnose: a policy that exists but blocks nothing is a classic, and the answer is almost always the failure action.

The documentation allowlist — read this twice

⚖ CNPA vs CNPE — That allowlist is a CNPE-only mechanic: CNPA has no allowlist at all, because it is a fully closed-book multiple-choice exam — zero external resources, zero lookups of any kind, on anything. That’s stricter than CNPE, not looser. Even so, the rule types and failure-action reasoning on this page are worth knowing cold, since this concept-level knowledge still matters for CNPA’s closed-book recall.

What to be able to do without notes

Write a valid ClusterPolicy from a blank file: apiVersion: kyverno.io/v1, kind: ClusterPolicy, spec.rules[] with a name, a match.any[].resources.kinds block, and a validate with a message plus either a pattern or deny.conditions. Set the failure action to Enforce and know both spellings of that field. Name the four rule types and what each does. Know that background: true scans what is already running and that generate takes data or clone with synchronize. Know that results land in polr and cpolr. And always prove it: create a deliberately violating pod and show the 403. A policy that applied is not a policy that enforces. Practise the whole shape on Practice: Security.

Official resources for after the exam

Outside the exam, the canonical sources are kyverno.io/docs (the Writing Policies section is the one to read end to end), the curated, searchable library at kyverno.io/policies, the source and issue tracker at github.com/kyverno/kyverno, and the project’s CNCF page at cncf.io/projects/kyverno. Pair this page with Security & Policy Enforcement for the concepts, OPA Gatekeeper for the comparison, Sigstore & cosign for the signatures verifyImages checks, and the glossary whenever a term stops making sense.