DevSecOps in Depth · Kubernetes Security Deep Dive

Kubernetes Security Deep Dive

PodSecurityPolicy shipped in Kubernetes 1.3, was deprecated in 1.21, and removed outright in 1.25 — a reminder that a built-in Kubernetes security control can disappear out from under a cluster that leaned on it. Cloud security posture drew the line between what your cloud provider secures and what you do; this page picks up exactly where that line ends and goes one layer deeper, inside the cluster itself. Four controls do the actual work: Pod Security Standards constrain what a Pod is allowed to declare about its own privilege, NetworkPolicy constrains what an already-running Pod is allowed to talk to, admission control constrains what any object is allowed to become on its way into etcd, and RBAC constrains who's allowed to ask for any of it in the first place. By the end you should be able to design a default-deny namespace, reason precisely about the admission chain a write passes through, and run a structured RBAC review against a cluster carrying dozens of service accounts instead of eyeballing YAML and hoping.

☺ Explain it like I'm 10

Remember the apartment building from cloud security posture — the landlord secures the building, you secure your own unit? This page is about what happens once you're inside your unit. You don't get just one lock on the front door: you decide which rooms are even allowed to exist without their own extra lock (Pod Security Standards deciding what a room may claim about itself), which rooms are allowed to send mail to which other rooms (NetworkPolicy), a doorperson who inspects every delivery before it's allowed upstairs at all (admission control), and a master list of exactly which roommate holds a key to which room (RBAC). None of that is the landlord's job. That's all you, inside your own four walls.

🤖Your host for this topic: Recon the Robot — the reconciler who never negotiates with drift teaches IaC security & policy as code; this page is that same reconciling instinct pointed at the cluster's own security posture instead of a Terraform plan.

From PodSecurityPolicy to Pod Security Standards

☺ Like you're 10: PodSecurityPolicy was one complicated gatekeeper that got taken away; Pod Security Standards is three simple name tags — Privileged, Baseline, Restricted — that you stick on a namespace's door instead.

PodSecurityPolicy (PSP) was Kubernetes' original built-in answer to "what is a Pod allowed to declare about its own privilege" — no privileged containers, no host namespaces, no arbitrary host-path mounts, unless a policy explicitly permitted it. It was also, by near-universal agreement among the people who operated it, one of the worst-designed objects in the API: a PSP wasn't bound to a namespace, it was authorized through RBAC to a user or service account, and which policy actually applied to a given Pod depended on who created it — the creating identity's bound policies, resolved through an opaque and hard-to-audit ordering. Testing a policy change meant simulating the creation identity, not just the Pod spec. SIG Auth deprecated PSP in Kubernetes 1.21 and removed it in 1.25, replacing it with something deliberately simpler.

The replacement is the built-in Pod Security admission (PSA) controller, which enforces a specification called the Pod Security Standards — three policy levels, not a policy language you write yourself: Privileged (unrestricted — for the platform's own infrastructure workloads, CNI and CSI daemons, and little else), Baseline (blocks the known, obvious privilege-escalation paths while staying broadly compatible with common workloads), and Restricted (the current hardened best practice — heavily locked down, and the level you want on anything handling real traffic or data). Critically, the level attaches to the namespace, not to whoever happens to be creating the Pod — exactly the ambiguity that made PSP unauditable.

apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    pod-security.kubernetes.io/enforce: restricted   # reject non-compliant Pods
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted      # allow, but record in audit log
    pod-security.kubernetes.io/warn: restricted       # allow, but warn the caller

The three modes — enforce, audit, warn — are independently settable and that independence is the whole migration story: run a namespace at enforce: baseline with audit: restricted and warn: restricted for a few weeks, watch the audit log and the warnings client tools surface, fix what would actually break, and only then flip enforce up to restricted. No PSP-style simulation required — the signal is right there before you ever turn on rejection.

ControlBaselineRestricted
Privileged containers, host namespaces, hostPathBlockedBlocked
Added Linux capabilitiesNo additions beyond a small, safe default setMust drop: [ALL]; only NET_BIND_SERVICE may be added back
Volume typesUnrestricted beyond the hostPath rule aboveRestricted to a core set — configMap, secret, emptyDir, PVC, projected, downwardAPI
allowPrivilegeEscalationAllowedMust be false
runAsNonRootNot requiredRequired — true
seccompProfileNot requiredRequired — RuntimeDefault or Localhost

Keep one thing in mind about scope: PSA is a floor every cluster gets for free, checked in the same admission chain covered below — it isn't a replacement for a full policy engine. It has no opinion on labels, resource limits, image registries, or anything outside the specific fields the Pod Security Standards define; that's exactly the gap admission webhooks exist to fill.

NetworkPolicy design for default-deny namespaces

☺ Like you're 10: By default every Pod can talk to every other Pod — NetworkPolicy is how you turn that open floor plan into locked doors, one explicit allow at a time.

Kubernetes ships the NetworkPolicy API, but not an implementation of it. The object is a request; a CNI plugin has to actually enforce it in the dataplane — Calico, Cilium, Antrea, and Weave Net all do, but the simplest CNI setups, and some cloud-managed defaults, historically didn't without an add-on. Apply a picture-perfect NetworkPolicy to a cluster whose CNI ignores the object, and nothing happens: no error, no warning, just silent non-enforcement. Confirming your CNI actually implements NetworkPolicy is the first step, not an afterthought.

⚠ Watch out

A default-deny egress policy is the single most common way teams accidentally break their own cluster. Kubernetes DNS resolution is itself a network call — to CoreDNS in kube-system, port 53, UDP and TCP. Default-deny egress without an explicit allow rule for DNS doesn't produce an obvious error; it produces every Pod in the namespace silently failing to resolve any hostname, application and infrastructure alike, which is a much harder incident to diagnose than a policy that simply didn't apply.

The design pattern is two steps, applied to every namespace, not just the ones that feel sensitive — the cost of a mistaken allow rule is far lower than the cost of relying on someone to remember whether a policy exists at all:

# Step 1 — deny everything: an empty podSelector matches every Pod
# in the namespace, and no ingress/egress rules means no traffic is allowed.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
# Step 2 — allow only what's needed, rule by rule.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-from-frontend-and-dns
  namespace: payments
spec:
  podSelector:
    matchLabels: { app: api }
  policyTypes: [Ingress, Egress]
  ingress:
    - from: [{ podSelector: { matchLabels: { app: frontend } } }]
      ports: [{ protocol: TCP, port: 8080 }]
  egress:
    - to: [{ podSelector: { matchLabels: { app: db } } }]
      ports: [{ protocol: TCP, port: 5432 }]
    - to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } } }]
      ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }]

NetworkPolicy rules are additive across every policy that selects a given Pod — there's no "deny wins" precedence to reason about, which is exactly what made PSP's resolution order so painful by comparison. A Pod's effective ingress is the union of every ingress rule from every policy that selects it; if no policy selects a Pod at all for a given direction, that direction defaults to fully open, which is the entire reason the default-deny-all policy above has to exist explicitly, in every namespace, rather than being assumed.

payments namespace — default-deny-all (Ingress + Egress) frontend api db allow :8080 allow :5432 no rule — denied kube-dns kube-system · :53 allow DNS :53 (every pod) internet no egress rule — denied Rules are additive across every policy selecting a Pod — nothing named above is allowed by default.
🤖 Recon's workshop · 15 min

On a cluster with a NetworkPolicy-enforcing CNI (kind with Calico, or any managed cluster with the policy add-on enabled), apply the default-deny-all policy above to a scratch namespace, then kubectl exec into one Pod and try to curl another Pod's ClusterIP — it should hang and time out. Apply the allow-api-from-frontend policy next and repeat the same curl from a Pod labeled app: frontend; it should succeed instantly. Now try it from a Pod with no matching label at all. That gap between "denied" and "allowed" for one label change is the entire mechanism NetworkPolicy runs on.

Admission control: validating and mutating webhooks

☺ Like you're 10: Before any change is allowed to become permanent, it has to walk past two checkpoints — one that's allowed to quietly fix small things, and one that can only say yes or no.

Every write to the apiserver runs the same fixed pipeline: authentication, authorization (RBAC), mutating admission webhooks, schema and Pod Security Standards validation, then validating admission webhooks, and only then persistence to etcd. Mutating webhooks can rewrite the incoming object — stamp default resource limits onto a Pod that didn't set any, inject a sidecar, add a required label. Validating webhooks can only accept or reject — "no image may come from an unapproved registry," "every Deployment must carry a cost-center label." From a security-program standpoint this chain is the enforcement point for policy as code applied to the cluster itself, the same idea IaC security & policy as code covers for a Terraform plan, just one layer further downstream.

Two policy engines dominate real deployments, plus a newer option built directly into the apiserver:

EnginePolicy languageNeeds a webhook server?Reach for it when
OPA GatekeeperRego, wrapped in ConstraintTemplate CRDsYesComplex, reusable, org-wide policy shared across Kubernetes and non-Kubernetes systems alike — see OPA & Conftest.
KyvernoNative Kubernetes YAML, no new languageYesFast-to-write rules, built-in image signature verification and mutation — see Kyverno.
ValidatingAdmissionPolicyCEL, expressed directly in the policy objectNo — built into the apiserverSimple, high-availability-critical rules where you can't afford a webhook outage.

That last row matters more than it looks. A webhook server is a Pod you deployed, which means it can crash, get OOMKilled, or lose its last replica during a bad rollout — and every matching write is a hard dependency on it being up. The built-in ValidatingAdmissionPolicy (stable as of the Kubernetes 1.30 line — verify the version against current docs if it matters for you) sidesteps that failure mode entirely for the rules simple enough to express in CEL, because there's no separate process to keep alive:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata: { name: block-privileged-containers }
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: >-
        object.spec.containers.all(c,
          !has(c.securityContext) || !has(c.securityContext.privileged) ||
          c.securityContext.privileged == false)
      message: "Privileged containers are not allowed in this cluster."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata: { name: block-privileged-containers-binding }
spec:
  policyName: block-privileged-containers
  validationActions: [Deny]
⚠ Watch out

Whichever engine you pick, failurePolicy is a security decision, not a deployment detail. Fail ("closed") rejects the write if the policy check can't run, which is correct for security-critical rules but turns your policy service into a single point of cluster-wide failure — scope its rules narrowly and exclude kube-system. Ignore ("open") lets the write through if the check can't run, which keeps the cluster available but means your security posture degrades silently the moment the policy engine has a bad day. Pick deliberately per rule; don't let the default do the choosing for you.

RBAC review for a cluster with dozens of service accounts

☺ Like you're 10: Every Pod, controller, and CI pipeline usually acts as a ServiceAccount, not a person — and a cluster that's been alive for a year usually has more of them than anyone can name from memory.

The building blocks are small: a Role or ClusterRole is a set of allowed verb + resource + apiGroup rules, and a RoleBinding or ClusterRoleBinding grants that rule set to a subject — a user, a group, or a ServiceAccount. A ClusterRole can be bound with a namespaced RoleBinding, which is the common pattern for defining one reusable role and scoping it per namespace rather than duplicating the same rules under a dozen different names. What actually makes RBAC hard to reason about at scale isn't the primitives — it's the sprawl. Every namespace gets a default ServiceAccount automatically; every Pod that doesn't specify a serviceAccountName uses it; and unless automountServiceAccountToken: false is set, that default SA's token gets automounted into every Pod in the namespace whether anyone intended to grant it a live API credential or not. A cluster with dozens of namespaces easily carries hundreds of service accounts, most created implicitly, few reviewed deliberately.

A structured review works through six questions, in order, rather than reading Role YAML top to bottom and hoping something jumps out:

  1. Inventory. List every ServiceAccount that exists (kubectl get sa -A) and cross-reference against what's actually supposed to have an identity — deployments, CronJobs, CI runners — versus default SAs nobody assigned on purpose.
  2. Map bindings. For each ServiceAccount, find every RoleBinding and ClusterRoleBinding that references it. Nothing in the Role object itself tells you who holds it — the binding is the actual grant.
  3. Flag wildcards. Any rule with resources: ["*"], verbs: ["*"], or apiGroups: ["*"] gets manual sign-off, never a rubber stamp — it's the RBAC equivalent of the "Action": "*" IAM policies that cloud security posture already flagged as a top misconfiguration category on the cloud-provider side.
  4. Flag high-risk verbs, wildcard or not. get/list/watch on secrets, create on pods/exec or pods/attach (an interactive shell in a running Pod is close to node-level access), and the RBAC-specific verbs impersonate, bind, and escalate — Kubernetes prevents a subject from granting permissions it doesn't already hold, but a subject that holds bind or escalate can route around that protection if handed out carelessly.
  5. Count cluster-admin. Every ClusterRoleBinding to cluster-admin gets a named owner and a reason it still exists. "Nobody remembers why" is the answer far more often than it should be.
  6. Resolve aggregation. An aggregated ClusterRole pulls in rules from other ClusterRoles via an aggregationRule label selector, without listing them directly — reading the YAML of the aggregating role tells you nothing about its actual effective permissions until you resolve the aggregation.
# 1 — inventory
kubectl get sa -A --no-headers | wc -l

# 2/3/4 — map bindings and surface risky grants for one identity
kubectl who-can delete secrets -n payments             # forward: who can do this?
rbac-lookup payments:ci-deployer                        # reverse: what can this SA do?
rakkess --sa payments:ci-deployer                        # full verb x resource matrix

# 6 — trim a role down to what a service account actually used, from the audit log
audit2rbac --serviceaccount=payments:ci-deployer < audit.log > ci-deployer-role.yaml

# a dedicated scanner for exactly the risky patterns in steps 3-5
kubiscan -rr   # risky roles: wildcards, secrets access, exec/attach, bind/escalate
ToolAnswers
kubectl-who-can"Who can verb resource in this namespace?" — a forward query, krew plugin.
rbac-lookup"What can this user, group, or ServiceAccount do?" — a reverse query across every binding.
rakkessRenders a full verb-by-resource access matrix for one identity on a single screen.
audit2rbacGiven real audit-log activity, generates the minimal Role that would have sufficed — the RBAC equivalent of IAM Access Analyzer.
kubiscanScans existing Roles and bindings directly for the risky patterns above — wildcards, secrets access, exec/attach, bind/escalate.

Run the full six-question pass quarterly, and gate step 5 and 6 continuously: an admission policy from the previous section can reject a new ClusterRoleBinding to cluster-admin outside a documented break-glass process just as easily as it rejects a privileged Pod, which turns "we'll catch it at the next quarterly review" into "it never merges in the first place."

Defense in depth: layering the four controls

☺ Like you're 10: None of the four does another one's job — stack them, and a single compromised container has to fail past all four before it reaches anything interesting.

It's worth being precise about what each layer stops that the others structurally cannot, because that precision is what tells you whether you actually have defense in depth or four overlapping copies of the same check. RBAC governs what an identity — human or ServiceAccount — is allowed to ask the API to do; it has no opinion about what's running inside a container. Admission control, including Pod Security Standards as its built-in floor, governs what object is allowed to be created or updated at write time; once that object exists and is running, admission control is finished — it never looks at it again. NetworkPolicy is the one control still doing anything after the Pod is scheduled: it governs what an already-running Pod can reach over the network, for as long as the Pod lives.

Walk the scenario through: an attacker gets remote code execution inside a container via an application vulnerability — a dependency bug, an injection flaw, whatever it was, upstream of everything on this page. RBAC now decides whether that compromised Pod's mounted ServiceAccount token can do anything interesting against the API — list Secrets cluster-wide, create new Pods, escalate — and if the token was scoped tightly by the review above, that's a dead end. Admission and Pod Security Standards already decided, at deploy time, whether that container was even allowed to run as root or mount the host filesystem — under restricted, there's no privilege-escalation path off the container to begin with. NetworkPolicy decides whether that Pod can reach the database three services over, or exfiltrate data to the internet at all — under default-deny, that's a third dead end. Four independent failures are required for the compromise to go anywhere, not one.

Write time — gates what can start kubectl apply Authn AuthzRBAC Mutatingadmission Pod SecurityStandards Validatingadmission etcd Podrunning Runtime — keeps constraining it for as long as it runs NetworkPolicy enforced by the CNI dataplane continuous seccomp / AppArmor enforced by the kernel continuous Falco behavioral syscall detection continuous A gate checks once, at the top of this diagram. Everything in the bottom row is still watching an hour later.
◆ Key idea

RBAC, Pod Security Standards, and admission control are all write-time checks — they run once, when an object is created or changed, and then stop. NetworkPolicy, seccomp/AppArmor, and runtime detection are the only members of this list still doing anything an hour into a Pod's life. A cluster that only invests in the write-time gates has a strong front door and no interior walls.

Continuous verification: benchmarks, drift, and the runtime backstop

☺ Like you're 10: A policy that was correct on the day you wrote it and never checked again is a policy that's slowly becoming wrong — clusters drift the same way cloud accounts do.

Everything above assumes the policies are actually being enforced as designed, on nodes that are themselves configured correctly — an assumption worth checking on a schedule rather than trusting forever. kube-bench (Aqua Security) is an open-source implementation of the CIS Kubernetes Benchmark: it runs on control-plane and worker nodes and checks their raw configuration — kubelet flags, file permissions on manifests and certificates, etcd's listen address — against a published checklist, which is a different layer entirely from anything PSA or a NetworkPolicy governs. Polaris and kubescape audit workload manifests already running in the cluster against best-practice rules — missing resource limits, missing probes, containers still running as root — which is the useful complement to enforce-time admission control: it tells you what's already there from before the policy existed, not just what a new write would trigger.

Drift happens the same way it happens to a cloud account: someone runs kubectl edit directly against a running Deployment during an incident, or a break-glass ClusterRoleBinding granted at 2 a.m. never gets revoked once the incident closes. A GitOps-managed cluster — Argo CD or Flux reconciling from a Git repository — catches configuration drift the same way any Kubernetes controller catches Pod drift, because it's the identical reconciliation loop with the security policy itself as the desired state instead of a Deployment spec. RBAC bindings and NetworkPolicy objects checked into Git and reconciled continuously don't survive a stray kubectl edit the way a policy that only lived in someone's memory of "how we set it up" does.

No static policy anticipates every attack, which is why runtime detectionFalco, watching syscalls via eBPF for behavior no admission check could have predicted, like a shell spawned inside a container that's never supposed to spawn one, or a write to /etc in an image that's supposed to be immutable — sits as the backstop behind everything else on this page. Container Runtime Security covers that layer in full; this page is deliberately everything before a workload starts misbehaving, that one is everything after. And if any of this is the syllabus you're actually studying toward, the CKS — Kubernetes Security Specialist exam is close to a certification built around exactly these four controls plus supply-chain and runtime security.

🎬 At the Shift-Left Squad
🤖

Recon the Robot: New namespace, payments. Before anything else — restricted Pod Security Standard, default-deny NetworkPolicy, and I need the RoleBinding list before I sign off.

🦫

Benny the Beaver: It's just a demo service. Does it really need all four checks on day one?

🐢

Timmy the Turtle: Every namespace gets the gate, Benny. I don't have a "trust me, it's just a demo" setting.

🦝

Rocky the Raccoon: Went looking anyway. Found a ClusterRoleBinding to cluster-admin from a hackathon, eight months old. Nobody remembered it existed.

🤖

Recon the Robot: Which is exactly the drift I reconcile against. Four controls, reviewed on a cadence — not four controls set once at launch and never looked at again.

🦉

Professor Owl: Say it plainly for the room: Pod Security Standards decide what a Pod may claim about itself, admission control decides what's allowed to exist at all, NetworkPolicy decides who it can talk to once it's running, RBAC decides who's allowed to ask for any of it. Know which one you're missing, and you know exactly what a compromise costs you.

✓ Checkpoint

1. Why was PodSecurityPolicy removed, and what specifically made it hard to audit compared to its replacement? Name the three Pod Security Standards levels and the three independent enforcement modes. 2. Why can a picture-perfect NetworkPolicy silently do nothing at all, and what's the classic mistake teams make with a default-deny egress policy? 3. What's the practical trade-off between OPA Gatekeeper, Kyverno, and the built-in ValidatingAdmissionPolicy for enforcing policy at admission time? 4. Name three things a structured RBAC review should specifically flag, beyond just confirming a Role exists. 5. Using the compromised-container scenario, explain why an attacker has to defeat all four controls — RBAC, admission/Pod Security Standards, and NetworkPolicy — rather than just one.

Check your answers
  1. PSP was authorized through RBAC to the creating user or service account rather than attached to a namespace, so which policy applied to a given Pod depended on who created it — an ordering that was opaque and hard to test. It was deprecated in Kubernetes 1.21 and removed in 1.25, replaced by Pod Security admission enforcing the Pod Security Standards. The three levels are Privileged, Baseline, and Restricted; the three modes are enforce (reject), audit (allow, log), and warn (allow, warn the caller) — settable independently, which is what makes a safe migration path possible.
  2. NetworkPolicy is an API object, not an enforcement mechanism — it does nothing unless the cluster's CNI plugin actually implements it, and applying it to a CNI that doesn't produces no error at all. The classic mistake is a default-deny egress policy with no explicit allow rule for DNS (CoreDNS in kube-system, port 53), which silently breaks every hostname lookup in the namespace instead of failing loudly.
  3. OPA Gatekeeper and Kyverno both need a running webhook server and can express complex, reusable, or cross-system policy — Gatekeeper in Rego, Kyverno in native Kubernetes YAML. ValidatingAdmissionPolicy is built directly into the apiserver in CEL, so it has no separate process to keep available, which matters most for simple rules that are too security-critical to risk a webhook outage taking them offline.
  4. Any three of: wildcard resources/verbs/apiGroups; high-risk verbs like get/list/watch on secrets, create on pods/exec or pods/attach, or the RBAC-escalation verbs impersonate/bind/escalate; every ClusterRoleBinding to cluster-admin and whether it still has a reason to exist; and aggregated ClusterRoles, whose effective permissions aren't visible just from reading the aggregating role's own YAML.
  5. RBAC governs whether the compromised Pod's ServiceAccount token can do anything interesting against the API — a tightly scoped token is a dead end there. Admission control and Pod Security Standards already decided at deploy time whether the container was even allowed to run as root or mount the host filesystem, closing off privilege escalation off the container under a Restricted policy. NetworkPolicy governs whether the running Pod can reach anything else over the network at all — default-deny closes off lateral movement and exfiltration. All three have to fail independently before the compromise reaches anything beyond the one container it started in.