Security: Defense in Depth
Kubernetes doesn't have one security boundary — it has a stack of independent controls, and each one is structurally blind to almost everything the others check. A scanner has no idea what RBAC will later allow. RBAC has no idea whether the object it just authorized is actually safe to run. A NetworkPolicy has no idea a Pod's own code has already been compromised — it only cares where that Pod tries to go next. And nothing before this page's last layer knows a container is doing something it has never done before; catching that is its job alone. This page is the map: what runs before a workload ever reaches the cluster, what the cluster's own write-time gate checks — RBAC and admission control, already covered in full in RBAC & Admission Control — what keeps watching after that gate is finished, and how one real credential, a Secret, has to survive every layer without ever being the thing that breaks it. None of this is CKA-scoped; most of it is exactly what CKS exists to test.
Picture four slices of Swiss cheese lined up back to back, each one riddled with its own holes. Slice one is the scan before a container ships — it has a hole, because a scanner can only catch a problem someone has already published. Slice two is the guard at the door, checking the shape of everything coming in — but with no idea what's hidden inside a legitimate-looking box. Slice three is a fence around each room, stopping anything that slips past the guard from wandering into the next room. Slice four is a night watchman inside every room, who doesn't care how something got in — only that it's now doing something nothing honest in that room has ever done. No single slice blocks everything. But line up all four, and a threat has to find a hole in exactly the same spot on every slice at once — and that almost never happens by accident.
Split this stack into two questions and the rest of the page organizes itself: which layers try to prevent something from ever existing, and which layers assume prevention already failed and try to contain or detect instead? Supply chain, RBAC, and admission control are all prevention — every one of them runs before or at the moment an object is created, and a hole here means something that shouldn't exist now does. NetworkPolicy and runtime detection are both containment — they only ever get a turn once a Pod is already scheduled and running, which means by definition something upstream already let it through. Neither half is optional and neither can substitute for the other: a cluster with airtight admission and no runtime detection has no idea when its airtight admission was based on a spec that turned out to be lying.
Before the cluster: what shouldn't even reach the registry
☺ Like you're 10: Nobody wants to open the box marked "bad" — much better if it never gets shipped to the door in the first place.
The cheapest security fix is the one that never has to run in production, and that's the entire premise of the supply-chain layer. It starts with what's in the image: a minimal or distroless base with no shell and no package manager gives an attacker who does achieve code execution nowhere to pivot from — there's no /bin/sh to reach for, a pattern DevOps' immutable infrastructure & golden images covers from the build-pipeline side. From there, a scanner like Trivy checks every layer of the built image against known-CVE databases before it's ever pushed, an SBOM records exactly what's inside so a future disclosure is a minutes-long query instead of a days-long audit, and Sigstore & cosign signs the resulting artifact — keylessly, via OIDC identity, with the signature recorded in a public transparency log — so the cluster can later verify not just what it's about to run but that it came from the pipeline it claims to have come from. DevSecOps' Container & supply-chain security is the full treatment of this layer on its own; this page's job is only to place it correctly in the stack.
# A build pipeline's supply-chain gate, in order
trivy image --severity HIGH,CRITICAL --exit-code 1 registry.example.com/checkout:sha-9f1c2a
# Sign only after the scan passes — an unsigned image never reaches the cluster's allow-list
cosign sign --yes registry.example.com/checkout@sha256:9f1c2a...
# Later, at admission time (see the next section), a policy verifies this exact signature
cosign verify --certificate-identity-regexp "https://github.com/example-org/checkout/.*" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
registry.example.com/checkout@sha256:9f1c2a...Trivy can only flag a CVE that's already been published to a database it checks against. A zero-day, or a vulnerability disclosed the day after your last scan, sails through a perfectly clean report — the scan wasn't wrong, it was answering the only question it's structurally capable of answering: "what's already known, as of right now." That's not an argument against scanning; it's the exact reason a scan-and-ship mentality is a false sense of done, and why the next three layers exist at all. Treat "the scan passed" as "layer one did its job," never as "this image is safe."
Write-time: RBAC decides who, admission decides what
☺ Like you're 10: Two separate checks at the same door — first your wristband color, then, no matter the color, someone still looks at exactly what you're carrying.
Once an image clears layer one, the cluster's own write-time gate takes over, and this course already gives it a full page: RBAC & Admission Control covers the request pipeline in precise order, aggregated ClusterRoles, the escalate/bind/impersonate verbs, bound ServiceAccount tokens, the built-in admission plugins, the mutating-then-validating webhook chain, and Pod Security Admission in full — read it for the mechanics. What that page doesn't spend time on, because it's a supply-chain concern rather than an admission-mechanics one, is the one control that ties layer one and layer two together directly: image-signature verification at admission. A policy engine like Kyverno or OPA Gatekeeper can run verifyImages-style rules as a validating admission policy — rejecting any Deployment whose image isn't signed by a trusted identity, the same signature cosign sign produced back in layer one. Without this check, signing is only ever advisory; an unsigned or wrong-signer image is still perfectly free to get scheduled, because nothing at write-time was ever asked to look.
# A Kyverno ClusterPolicy enforcing layer one's signature at layer two's gate
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-signed-images
spec:
validationFailureAction: Enforce
rules:
- name: verify-checkout-signature
match:
any:
- resources: { kinds: ["Pod"] }
verifyImages:
- imageReferences: ["registry.example.com/checkout:*"]
attestors:
- entries:
- keyless:
subject: "https://github.com/example-org/checkout/.*"
issuer: "https://token.actions.githubusercontent.com"Pod Security Admission — the floor every namespace enforces whether you configured anything or not — is the other half of this layer, and it's covered in full on the RBAC page. The one thing worth restating here: PSA checks the shape of a Pod spec (root user, host mounts, capabilities), never the behavior of the code running inside it. A Pod can be perfectly restricted-compliant and still be running an application with a live remote-code-execution bug — which is exactly why write-time is only half the stack.
Continuous: NetworkPolicy decides where, runtime detection decides what happens next
☺ Like you're 10: The write-time gate only ever looks once, at the door. These two keep looking, for as long as the thing is actually inside.
Admission is the last check that ever inspects the object as submitted — the moment it's persisted to etcd and scheduled, admission is finished and never looks again. Everything from here on is watching a Pod that's already running. NetworkPolicy, the layer that decides where a Pod is allowed to talk, gets a full mechanical treatment — including the trap that a policy silently does nothing on a CNI that doesn't enforce it — in Networking & the CNI; the short version here is that a default-deny policy scoped tightly to a workload's actual dependencies is what turns "attacker got a shell in one Pod" into "attacker got a shell in one Pod that can reach exactly the two things it was already allowed to reach."
The layer nothing else on this page covers is runtime detection, and it exists because every control above it shares one blind spot: none of them can see what a container's process tree actually does after it starts. A tool like Falco — a CNCF-graduated engine built on eBPF — attaches at the kernel syscall level and evaluates live behavior against a ruleset: a shell spawning inside a container image that has never spawned one, a process reading /etc/shadow, an outbound connection from a Pod that has no business making one. Where a NetworkPolicy only ever answers "is this connection allowed," Falco can answer "did something inside this Pod just start behaving like nothing that Pod's own code has ever done" — a structurally different question, answerable only from inside the running container, which is why it's this page's fifth and final layer rather than a restatement of the fourth. DevSecOps' Container runtime security is the deep mechanical treatment — eBPF probes, the default ruleset, and routing alerts to a SIEM instead of a log file nobody reads.
# A Falco rule: alert the instant a shell appears inside a container
# that has no legitimate reason to ever spawn one.
- rule: Unexpected Shell in Payments Container
desc: A shell was spawned inside a container in the payments namespace
condition: >
spawned_process
and container
and k8s.ns.name = "payments"
and proc.name in (bash, sh, zsh)
and not proc.pname in (entrypoint.sh, healthcheck.sh)
output: >
Shell spawned in payments container
(user=%user.name container=%container.name image=%container.image.repository
parent=%proc.pname cmdline=%proc.cmdline)
priority: CRITICAL
tags: [container, shell, mitre_execution]Secrets: the one thing every layer has to protect, and none of them fully can
☺ Like you're 10: A secret has to survive walking through all five checkpoints without a single one of them accidentally leaving it lying on the counter.
A Kubernetes Secret is base64-encoded, not encrypted — anyone who can kubectl get secret -o yaml it can decode it in one command, which is exactly why layer two's RBAC gating get/list on secrets is doing real work here, not a formality. What Kubernetes stores at rest is whatever etcd holds, and by default that's the same unencrypted base64 — real encryption at rest requires an explicit EncryptionConfiguration on kube-apiserver, ideally backed by a KMS provider rather than a static key file sitting on a control-plane node. How that mounts into a Pod matters too: a Secret projected as a volume file lives only in an in-memory tmpfs and never shows up in a process listing or a crash dump the way an environment variable does, which is why envFrom: secretRef is the pattern worth defaulting away from, not toward. And a growing number of teams skip persisting the real credential in etcd — or in a Git repo it was templated from — entirely, using something like External Secrets & the secrets patterns to sync from Vault or a cloud secrets manager at the last possible moment, with the manager doing rotation instead of a human remembering to. DevSecOps' Secrets management covers the leak-scanning and rotation-policy side of this, and Cryptography & key management covers what "encrypted" is actually claiming underneath a KMS call. The thread that connects all five layers, restated: a secret should never be baked into layer one's image, should never be readable without layer two's explicit grant, should never leave etcd unencrypted under layer three's watch, should never be worth exfiltrating past layer four's policy, and its access pattern is exactly the kind of thing layer five should be watching for anomalies in.
Reproduce the last two layers of the walkthrough above on a real cluster. First, apply a default-deny NetworkPolicy to a test namespace that allows egress only to one specific in-namespace Service, then kubectl exec into a Pod there and try to reach anything else — confirm the connection actually hangs rather than just trusting the YAML. Then register the Falco rule from this page (or Falco's own default ruleset) and deliberately kubectl exec -it <pod> -- sh into a container that never normally gets a shell — watch the alert land. For the guided version that layers RBAC, admission, and hardening together, work through Drill: Harden an RBAC Configuration, or build the whole stack as part of a real cluster in Capstone Part 5 — Security & RBAC.
"The mistake I watch teams make isn't skipping a layer — it's treating whichever layer they finished most recently as 'done.' Ship a scanner, and suddenly every conversation about risk starts and ends with the scan result. Get admission policies airtight, and suddenly a Pod that passed admission is treated as trusted forever, not just trusted at the moment it was created. Each layer only ever answers the one question it was built to answer. The scan answers 'is anything already known wrong with this image.' Admission answers 'is this specific object shaped correctly.' Neither one has any opinion on what the running process does thirty minutes from now — that's not a gap in the tool, that's the tool working exactly as designed. Check every layer, on its own terms, and never let one stand in for another."
This page is the map; three of its five layers already have a full page of their own elsewhere. RBAC & Admission Control is the write-time gate in full mechanical detail, Networking & the CNI covers exactly how (and how not) NetworkPolicy gets enforced, and cross-course, DevSecOps' own Kubernetes security deep dive covers the same write-time-versus-continuous split from the security-team side of the table, alongside Container runtime security for eBPF mechanics past what fits on this page. None of it is CKA-scoped. If this is the direction you're headed, the CKS blueprint is the specialist certification built entirely around this stack, the associate-level KCSA blueprint is the on-ramp to it, Platform Engineering's Security & Policy Enforcement covers the same territory from a CNPE angle, and the wider CNCF security-track ladder beyond CKS lives on the sibling Golden Astronaut course.
Ellie the Elephant: Falco alert, three minutes ago — a shell spawned inside a payments Pod that's never spawned one in its entire recorded history. Logging it now, but somebody needs eyes on this immediately.
Foxy: That Pod passed admission this morning — restricted PSS, non-root, signature verified. Nothing about it looked wrong going in.
Timmy the Turtle: Because admission never checked what the app's own code does — it checked the shape of the spec, once, before anything ran. That's not a failure at the gate. It's the reason we have a gate after the gate.
Gizmo the Gremlin: Falco's just noisy anyway — half these alerts are nothing. Bump the rule threshold way up, or turn off enforcement on that namespace's NetworkPolicy so the on-call stops getting paged every night. 🤑
Pip the Hummingbird: Absolutely not touching the NetworkPolicy — it's the only reason that shell couldn't reach the database. Loosen it now and the next alert is a data breach instead of a Falco log line.
Benny the Beaver: Found the actual hole — a deserialization bug in a library that got a CVE published two days after our last scan. Patching the base image and re-signing now.
Timmy the Turtle: Good. Fix the layer that actually failed. The other four did exactly their job — that's the whole point of stacking them.
1. Put these five in the order a workload actually meets them: NetworkPolicy enforcement, admission control, runtime detection, image scan and sign, RBAC authorization. 2. Structurally — not as a tooling limitation, but by design — what is the one category of problem a vulnerability scanner can never catch, no matter how good it is? 3. A Pod's spec is completely valid, passes restricted Pod Security Admission, and RBAC correctly authorized the deploy. The application inside is exploited five minutes later. Did admission control fail here? Why or why not? 4. Name one thing Falco can catch that a NetworkPolicy structurally cannot, and explain the structural reason why NetworkPolicy can't. 5. Why does mounting a Secret as a projected volume file generally beat injecting it as an environment variable? 6. Of the five layers on this page, which three are prevention and which two are containment or detection — and what's the one-sentence rule for telling them apart?
Check your answers
- Image scan and sign, RBAC authorization, admission control, NetworkPolicy enforcement, runtime detection — supply chain and the write-time gate happen before or as the object is created; NetworkPolicy and runtime detection only ever act on a Pod that's already scheduled and running.
- A vulnerability not yet published to whatever database the scanner checks against — a genuine zero-day, or a CVE disclosed after the scan already ran. The scan can only ever compare against what's already known as of the moment it runs; it has no way to flag a problem nobody has documented yet.
- No — admission worked exactly as designed. Admission (and Pod Security Admission specifically) validates the shape of a Pod spec — user, capabilities, host access — not the runtime behavior of the application code running inside it. A perfectly shaped Pod can still contain a perfectly real, exploitable bug; catching that behavior is what the runtime-detection layer exists for, not admission.
- Falco can catch a process-level anomaly inside an already-permitted connection or already-running container — a shell spawning where one never has, a sensitive file being read, a privilege-escalation attempt — none of which involves a network connection a
NetworkPolicywould ever inspect. NetworkPolicy only evaluates packets against allowed source/destination/port rules; it has no visibility into what a process inside an already-permitted Pod is actually doing. - A Secret mounted as a projected volume lives only in an in-memory
tmpfsfile and doesn't appear in a process listing, a child process's inherited environment, or a crash dump — all places an environment variable routinely does show up, making it a much easier accidental leak. - Supply chain, RBAC, and admission control are prevention — all three run before or at the moment the object is created, trying to stop something that shouldn't exist from ever existing. NetworkPolicy and runtime detection are containment and detection — both only ever act on a Pod that's already running, so by definition something upstream already let it through. The rule: if it can still say no to the object existing, it's prevention; if it can only limit or flag what an already-running Pod does, it's containment.