Security & Policy Enforcement

A golden path is only golden if it’s safe to drive fast on. These guardrails let Dot ship in minutes without a 2am breach page: proving who every service is (mTLS and identity), limiting what each person and workload can do (RBAC and least privilege), checking every change before it lands (admission control and policy-as-code), and trusting what you run (a signed, scanned supply chain). Done right, security isn’t a gate people route around but the paved shoulder keeping them on the road.

Zero trust: never trust, always verify

☺ Like you’re 10: Being inside the building doesn’t mean you can open every door — everyone shows a badge at every door, every time.

The old model was a castle: a hard wall around the cluster, everything inside trusted. That fails the moment one pod is compromised — the attacker is now “inside the wall.” Zero trust flips it: nothing is trusted for where it comes from; every call must prove its identity and be explicitly authorised, every time. On a shared multi-tenant platform that’s not optional — one tenant’s bug must never become another’s breach.

mTLS with a service mesh (Pip’s wires)

☺ Like you’re 10: Two services don’t just talk — they check each other’s ID cards and whisper through a tube nobody else can hear.

Plain HTTP between services is a postcard: anyone on the path can read it or forge the sender. Mutual TLS (mTLS) fixes both — traffic is encrypted, and both ends present a certificate, so each proves its identity to the other. Per-app that’s miserable, so a service mesh does it: a sidecar (or per-node proxy in “ambient” mode) beside each pod transparently upgrades every connection to mTLS. The exam names two:

• Istio — feature-rich, built on Envoy. A PeerAuthentication set to STRICT requires mTLS (the default PERMISSIVE also accepts plaintext, for migration); access rules use AuthorizationPolicy.
• Linkerd — deliberately lightweight, on a Rust “micro-proxy”. It turns mTLS on automatically for all meshed pod-to-pod TCP with zero config; authorization uses Server and AuthorizationPolicy.

Both derive a workload’s identity from its Kubernetes ServiceAccount (a SPIFFE identity), so “who is this service?” has a cryptographic answer, not an IP you must trust. Here, Istio requires mTLS in a namespace, then lets only the frontend account call checkout:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments
spec:
  mtls:
    mode: STRICT              # reject any non-mTLS traffic in this namespace
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: checkout-allow-frontend
  namespace: payments
spec:
  selector:
    matchLabels: { app: checkout }
  action: ALLOW
  rules:
    - from:
        - source:
            # a principal is the SPIFFE ID minus its scheme, i.e.
            # spiffe://cluster.local/ns/<namespace>/sa/<serviceaccount>
            principals: ["cluster.local/ns/storefront/sa/frontend"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/api/charge"]

Read aloud: “in payments, refuse un-encrypted traffic, and only let frontend POST to /api/charge on checkout.” Every other caller is denied by default.

cert-manager: who mints the certificates

mTLS depends on a steady supply of short-lived certificates, and hand-rotated ones always expire at the worst moment. cert-manager is the Kubernetes-native certificate robot: you declare an Issuer / ClusterIssuer (backed by an internal CA, Vault, or an ACME provider like Let’s Encrypt) and a Certificate, and it obtains, stores it in a Secret, and renews before it lapses. It secures ingress TLS, and via istio-csr can even back the mesh’s workload identities — no more expired-cert outages.

NetworkPolicy: the network-layer complement

☺ Like you’re 10: Even with ID checks at every door, you brick up the hallways nobody should walk down — so a burglar can’t wander room to room.

mTLS secures traffic that is allowed; a NetworkPolicy decides which traffic may exist at all, at L3/4. It limits blast radius: if one pod is compromised, policy stops it reaching every other. The golden-path pattern is default-deny — deny everything in a namespace, then open only the flows each app needs:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments
spec:
  podSelector: {}                 # an empty selector = every pod in the namespace
  policyTypes: [Ingress, Egress]  # with no rules below, all in/out traffic is denied
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-checkout
  namespace: payments
spec:
  podSelector:
    matchLabels: { app: checkout }
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { team: storefront }
      ports:
        - protocol: TCP
          port: 8080

RBAC and least privilege

☺ Like you’re 10: Everyone gets a key that opens exactly the doors their job needs — and not one more.

Once a request is authenticated (“who are you?”), Kubernetes asks “are you allowed to do this?” That’s Role-Based Access Control (RBAC), governing humans and workloads through four small, composable objects:

Subjects come in three kinds: users and groups (people, authenticated by your identity provider — Kubernetes has no built-in user database) and ServiceAccounts (the identity a pod runs as). RBAC is purely additive — no “deny” rules, so a subject can do only what a binding grants. Below, checkout gets read-only config access in its namespace, nothing else:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: checkout
  namespace: payments
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: config-reader
  namespace: payments
rules:
  - apiGroups: [""]                       # "" = the core API group
    resources: ["configmaps"]
    verbs: ["get", "list", "watch"]       # read-only; no create/update/delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: checkout-config-reader
  namespace: payments
subjects:
  - kind: ServiceAccount
    name: checkout
    namespace: payments
roleRef:
  kind: Role
  name: config-reader
  apiGroup: rbac.authorization.k8s.io

The governing principle is least privilege: narrowest verbs, fewest resources, smallest scope, namespaced Roles over ClusterRoles. Two habits matter. Don’t let workloads use the default ServiceAccount — give each its own, and set automountServiceAccountToken: false where a pod needs no API access. And be ruthless about the top of the ladder.

Scoping platform vs tenant permissions

On an internal platform, RBAC draws the line between the platform team that builds the playground and the tenants who play in it. The platform team needs cluster-scoped power for controllers, CRDs, and mesh config, but even they should work through GitOps, not personal cluster-admin. Tenants get a namespace (or virtual cluster) and a curated Role: they manage their own Deployments, Services, and ConfigMaps, but cannot edit NetworkPolicies, escalate RBAC, or touch other namespaces — neither can bulldoze the other’s lane, the RBAC half of the multi-tenancy story.

The admission-control path — where policy plugs in

☺ Like you’re 10: Before anything is written into the cluster’s notebook, it passes through a hallway of checkpoints — any one can change it or turn it away.

To place policy engines, know where they sit. Every write is an API request that travels a fixed gauntlet inside the kube-apiserver before it’s saved — picture this path and half of Domain 5 clicks into place:

Mutating vs validating webhooks

☺ Like you’re 10: One checkpoint can fix your outfit (tuck in your shirt); the next only says yes or no — it can’t touch you.

Two kinds of admission webhook hang off that path, order exam-critical. Mutating webhooks run first and can change the object — inject a mesh sidecar, add default labels, set a securityContext. Validating webhooks run last and only accept or reject; they never modify. So a policy validates the final object — after defaults were injected — not the user’s original. Engines register as one or both.

Pod Security Admission — the built-in baseline

☺ Like you’re 10: Kubernetes has a built-in bouncer with three dress codes — pick one per room with a label on the door.

Before any full policy engine, know the built-in floor. Pod Security Admission (PSA) is a validating controller enforcing the Pod Security Standards at three levels: privileged (no restrictions), baseline (blocks obvious escapes: host namespaces, privileged containers), and restricted (hardened: non-root, no privilege escalation, seccomp). You opt a namespace in with labels, and each label pairs one of three modes — enforce (reject the pod), audit (let it through but record it in the audit log), warn (let it through and warn the applier) — with the level to apply:

apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    pod-security.kubernetes.io/enforce: restricted   # reject non-conforming pods
    pod-security.kubernetes.io/warn: restricted       # and warn the applier
    pod-security.kubernetes.io/audit: restricted      # and log it for review

PSA is coarse — three fixed levels, no custom rules — so teams graduate to Gatekeeper or Kyverno for bespoke policy. But it’s free, always-on, and a sane floor for every tenant namespace.

Policy engines: policy as code

☺ Like you’re 10: Instead of a human remembering the rules, you write them as code so the computer checks every request the same way, forever.

Policy-as-code means the rules governing your cluster — “no privileged pods,” “images only from our registry” — live in version control and are enforced automatically at admission, not in a wiki nobody reads. Two CNCF projects dominate, and the exam names both.

Kyverno — policies that look like manifests

Kyverno runs as an admission webhook and expresses policy as ordinary Kubernetes resources in the YAML you already write — anyone who can read a Deployment can read a policy. Beyond validate, it can mutate (add defaults), generate (a default NetworkPolicy per new namespace), and verifyImages (check signatures). Here it requires every pod to run non-root:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-non-root
spec:
  validationFailureAction: Enforce   # "Audit" to only report; "Enforce" to block
  background: true                    # also scan already-running pods
  rules:
    - name: run-as-non-root
      match:
        any:
          - resources:
              kinds: ["Pod"]
      validate:
        message: "Running as root is not allowed — set runAsNonRoot: true."
        pattern:
          spec:
            securityContext:          # pod level — a hardened version also checks
              runAsNonRoot: true      # each container, which can override the pod

OPA / Gatekeeper — Rego and constraints

☺ Like you’re 10: You write the logic once as a reusable stencil, then stamp out specific rules by filling in the blanks.

Open Policy Agent (OPA) is a general policy engine; Gatekeeper is its Kubernetes admission integration. Policy splits in two: a ConstraintTemplate carries the reusable Rego logic and defines a new CRD; a Constraint instantiates it with parameters and match rules. This example requires a label on namespaces:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names: { kind: K8sRequiredLabels }
      validation:
        openAPIV3Schema:
          type: object
          properties:
            labels: { type: array, items: { type: string } }
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels
        violation[{"msg": msg}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("missing required labels: %v", [missing])
        }
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: ns-must-have-owner
spec:
  enforcementAction: deny          # "dryrun" = audit only; "deny" = block; "warn"
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
  parameters:
    labels: ["owner"]

Audit vs enforce — roll out safely

☺ Like you’re 10: First the guard just writes down who’s breaking the rule; only once the backlog’s fixed do they start turning people away.

Both engines share a two-speed rollout, and using it well marks a senior engineer. In audit mode (validationFailureAction: Audit in Kyverno, enforcementAction: dryrun in Gatekeeper), violations are recorded but nothing is blocked — so you switch a policy on across a live cluster, see who breaks, and fix them without an outage. Then flip to enforce and it starts rejecting — guardrails added to a moving platform, no big-bang breakage.

Supply-chain security — trust what you ship

☺ Like you’re 10: Before you eat the sandwich, you want the list of ingredients, an allergy check, and a tamper-proof seal proving the kitchen actually made it.

Plenty of compromises never touch your front door — they arrive baked into a container image: a vulnerable dependency, or a tampered artifact from a poisoned build. Supply-chain security is trusting what you run: four questions, each with a tool, answered early in the pipeline (“shift left”) and re-checked at admission.

SBOM, scanning, signing, provenance

Sigstore/cosign is exam-worthy: cosign can sign with a key, but its clever mode is keyless — it gets a short-lived certificate from Fulcio tied to your CI’s OIDC identity (e.g. GitHub Actions), signs, and records it in Rekor, a public append-only transparency log where any later tampering is detectable — no long-lived key to leak. SLSA (Supply-chain Levels for Software Artifacts) then grades build integrity around a verifiable provenance attestation of which source and builder produced the artifact.

Verify at admission — closing the loop

☺ Like you’re 10: Signing the sandwich is pointless if nobody checks the seal — so the cluster checks before letting anything in.

Signing only helps if something verifies it before the image runs — otherwise you’ve locked the door and left the key in it. Here policy engines and the supply chain meet: a Kyverno verifyImages rule (or the Sigstore policy-controller) blocks any pod whose image isn’t signed by your identity. This one demands a keyless cosign signature from your GitHub org:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-signed-by-ci
      match:
        any:
          - resources:
              kinds: ["Pod"]
      verifyImages:
        - imageReferences: ["registry.acme.io/*"]
          attestors:
            - entries:
                - keyless:                                  # Sigstore keyless
                    subject: "https://github.com/acme/*"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev       # transparency log

Audit trails, compliance, and runtime

Prevention isn’t the whole story: you also have to prove what happened — for auditors and forensics — and catch the attack that slips past every gate. Three capabilities finish the domain.

Kubernetes audit logs

Audit logs are the API server’s chronological record of every request: who did what, to which resource, when, and whether it was allowed. Tune verbosity per rule with an audit policy — level None, Metadata, Request or RequestResponse — and ship the log off the cluster to your observability stack, which is what stops an attacker simply deleting their own tracks. When Foxy asks “who deleted the payments namespace at 03:14?”, the log answers — so enable it before the incident, not after.

Policy reports & compliance evidence

Both Kyverno and Gatekeeper continuously emit machine-readable results — Kyverno via the standard PolicyReport / ClusterPolicyReport CRDs, Gatekeeper via the audit violations recorded in each Constraint’s status, a live report card — so “are we compliant right now?” is a query, not a quarterly fire drill. Add CIS-benchmark checks (kube-bench) and compliance reports fall out as a by-product. Your SBOMs double as evidence: when a Log4Shell-style CVE drops, stored SBOMs answer “which of our 400 services ship this library?” in seconds, not weeks.

Runtime security with Falco

☺ Like you’re 10: Even with locked doors and ID checks, you still want a lifeguard watching the pool for someone who’s already drowning.

Everything so far runs at admission — before a pod starts. But a container that passed every check can still be exploited at runtime. Falco is the CNCF runtime-security watchdog: it watches the kernel’s syscalls (through an eBPF probe, or a kernel module on older setups) and alerts on behaviour a static check can’t see — a shell spawning in a container, an unexpected outbound connection, a process reading /etc/shadow. Admission asks “should this start?”; Falco asks “is something bad happening now?” — you need both. Here’s the whole domain as layers:

Wiring it into the platform

The senior insight: security must be the default, built into the road so nobody has to remember to be safe. Two placements do the work. First, shift left — SBOM, scanning, and signing live in Benny’s CI/CD pipeline, so problems surface in the PR, cheap to fix. Second, admission is the final gate — policy engines re-check at the cluster boundary, so a skipped pipeline can’t sneak in an unsigned or root-running image. Belt and braces.

When Mira’s self-service templates scaffold a service, it ships already secure-by-default: its own ServiceAccount, a default-deny NetworkPolicy, a restricted Pod Security level, mesh mTLS, and a signing step. Dot gets it all by clicking “new service” — the golden path is the secure path. Being declarative, it flows through GitOps: policies, RBAC, and mesh config reconcile from Git — one audit trail, one-commit rollback. See the reference architecture for how the layers fit and best practices for the guardrails-as-defaults pattern.

Guardrails turn “move fast and break things” into “move fast and break nothing”: the platform can hand Dot the keys because the road itself keeps her safe. Next, Timmy catalogues how teams get this wrong in Anti-Patterns & Pitfalls.