Drill — Container Escape Investigation

Step two — the identity check comes up empty, which is itself the finding. Nothing in the Kubernetes audit log around 16:42 shows a human or a CI identity issuing kubectl exec against checkout-7d9f-x8. What it does show, a few seconds after the token-read alert, is the stolen token trying to use itself:

$ kubectl -n kube-system logs deploy/audit-sink --since=15m \
    | jq 'select(.user.username=="system:serviceaccount:payments:checkout")'

{"kind":"Event","verb":"create","objectRef":{"resource":"pods","namespace":"kube-system"},
 "user":{"username":"system:serviceaccount:payments:checkout"},
 "responseStatus":{"code":403,"reason":"Forbidden",
   "message":"pods is forbidden: User \"system:serviceaccount:payments:checkout\" cannot create resource
   \"pods\" in API group \"\" in the namespace \"kube-system\""}}

The attacker used the stolen token to try creating a privileged pod in kube-system — the classic move to escape the pod boundary entirely by scheduling something with hostPID or a host mount. RBAC scoped to the checkout ServiceAccount's actual job returned 403. That pivot failed — but not because anything about this container stopped it.

Step three — nothing in the node's kernel logs shows a denial, because nothing here needed kernel-level permission it didn't already have. The container was running as root with a full default capability set, so chmod u+s /bin/dash is an entirely ordinary, permitted operation — no seccomp filter or AppArmor profile ever got a chance to object, because none was loaded.

$ kubectl -n payments get pod checkout-7d9f-x8 -o jsonpath='{.spec.containers[0].securityContext}{"\n"}'
{}
$ kubectl -n payments get pod checkout-7d9f-x8 -o jsonpath='{.spec.automountServiceAccountToken}{"\n"}'
true

Step four confirms it: an empty securityContext. No runAsNonRoot, no capability drop, no seccomp profile, and the ServiceAccount token auto-mounted by Kubernetes' own default — for a service that never once calls the Kubernetes API to do its job.

Verdict: confirmed compromise. Real RCE, a real reverse-shell callback, a real credential theft, and a real persistence mechanism already planted. The pivot into kube-system failed on RBAC scoping — a control that happened to exist for an unrelated reason, not because this workload was hardened. Treat that distinction as the whole lesson of this ticket: the team got lucky once, on a control nobody put there specifically to stop this. Rotate the token immediately regardless of its remaining TTL (a projected, auto-rotating token still narrows the exfiltration window, but "narrower" isn't "safe" — see Secrets management), and treat the image as burned: redeploy from a rebuilt, unmodified image rather than trusting the running one, since the setuid backdoor persists in the container's writable layer.

What would have changed the outcome: runAsNonRoot: true (with a fixed, non-zero runAsUser) is the control that actually stops this exact call. chmod's permission check passes automatically the moment the calling UID matches the file's own owning UID, and root — UID 0 — trivially matches /bin/dash's owner, also UID 0, with no capability check involved at all. Once the process runs as a UID that doesn't own the file, that same call needs CAP_FOWNER to override the ownership mismatch — which is exactly what capabilities.drop: ["ALL"] removes, closing the one bypass a misconfigured UID might otherwise leave open. allowPrivilegeEscalation: false is the backstop underneath both: it sets the kernel's no_new_privs flag, so even a setuid bit that somehow did get set is ignored at exec time — the persistent-root-shell trick stops working regardless of how the bit got there. And automountServiceAccountToken: false handles the earlier step: with it set, the token read in step three has nothing to read, because checkout never needed API access to begin with.

Step two — the Kubernetes audit log has the answer. The k8saudit plugin (or a direct pull from wherever your cluster ships its audit log — a webhook backend, a control-plane export, a sidecar) shows exactly who opened this session:

$ kubectl -n kube-system logs deploy/audit-sink --since=20m \
    | jq 'select(.objectRef.name=="inventory-sync-6c88f" and .objectRef.subresource=="exec")'

{"kind":"Event","verb":"create",
 "objectRef":{"resource":"pods","subresource":"exec","name":"inventory-sync-6c88f","namespace":"warehouse"},
 "user":{"username":"alex@acme.com","groups":["system:authenticated","break-glass-oncall"]},
 "requestURI":"/api/v1/namespaces/warehouse/pods/inventory-sync-6c88f/exec?command=bash&container=inventory-sync&stdin=true&stdout=true&tty=true"}

alex@acme.com, authenticated, through a break-glass-oncall group scoped specifically for exactly this. Cross-referencing the on-call ticketing system: INC-4471, "inventory-sync stuck, investigating," opened by the same engineer six minutes earlier — timestamps line up.

Steps three and four — nothing to check, and that's fine. No syscall was denied because nothing malicious was attempted; the pod's securityContext is relevant here only for what it means for what Alex's session could accidentally do, not for what stopped an attack, because there wasn't one.

Verdict: false positive for compromise — Falco did its job correctly by alerting on genuinely anomalous behavior; the anomaly just turned out to be a human, not an attacker. Closing the ticket as "no action" is not the same as closing it as "no finding," though. A regular bash exec straight into a running production container means every command Alex ran carried the container's real permissions against live state — an accidental rm or a typo'd config edit would have hit production exactly as hard as an attacker's would have. The fix isn't a Falco tuning change; it's a process one: prefer kubectl debug inventory-sync-6c88f -it --image=busybox --target=inventory-sync --copy-to=inventory-sync-debug, an ephemeral debug container attached alongside the real one, which gives Alex the tools to look around without executing anything inside the production container's own process namespace and writable layer.

What would have changed the outcome: nothing here needed to be prevented — the session was legitimate. But notice what a distroless build (see Container & supply-chain security) would have done to this exact ticket: with no bash binary in the image at all, this specific exec simply isn't possible, and the ephemeral-debug-container pattern stops being optional advice and becomes the only path in — the same tradeoff that page already named as the cost of going shell-less. Runtime hardening and image hygiene are different levers pulling toward the same outcome here.

Step three is where this ticket turns. The node's own kernel audit log, pulled directly from the worker the pod was scheduled on, shows the syscall this technique depends on getting refused before it could do anything:

$ ssh node-worker-14
$ sudo ausearch -m SECCOMP -ts recent

type=SECCOMP msg=audit(1755316682.771:48291): auid=4294967295 uid=10004 gid=10004 ses=4294967295
 pid=48211 comm="sh" exe="/bin/dash" sig=31 arch=c000003e syscall=165 compat=0
 ip=0x7f2a1c2a3f5b code=0x0

$ sudo dmesg | grep -i apparmor | tail -3
[884213.221004] audit: type=1400 audit(1755316682.900:9931): apparmor="DENIED" operation="mkdir"
 profile="cri-containerd.apparmor.d" name="/sys/fs/cgroup/rdma/w/" pid=48211 comm="sh"
 requested_mask="c" denied_mask="c" fsuid=10004 ouid=0

Read the seccomp line precisely: syscall=165 is mount on x86_64 — the release_agent technique needs a fresh, attacker-controlled cgroup hierarchy mounted to plant its own release_agent path, and code=0x0 means the seccomp filter killed the call outright before it returned to the process. The AppArmor line, a fraction of a second later, shows a second, independent denial — a write into the cgroupfs hierarchy the container's profile never grants, which would have stopped this specific write even if the mount call had somehow gotten through some other way.

Step four explains why both fired. This pod's securityContext, unlike ticket 1's, was actually filled in:

securityContext:
  runAsNonRoot: true
  runAsUser: 10004
  allowPrivilegeEscalation: false
  seccompProfile: { type: RuntimeDefault }
  capabilities: { drop: ["ALL"] }
# AppArmor — annotation form (pre-1.30) shown; see the native field below
# container.apparmor.security.beta.kubernetes.io/batch-reporting: runtime/default

Three layers, and the attacker needed all three to fail to succeed: capabilities were already dropped, so the process never held CAP_SYS_ADMIN — the capability the release_agent technique needs even to attempt a fresh cgroup mount, meaning this attack was already dead on arrival before seccomp ever got involved. Seccomp's RuntimeDefault profile blocks the mount syscall outright regardless of capability, an independent line of defense. And AppArmor's default profile denies the write path into cgroupfs on top of both, a third layer working on a different axis entirely — file paths and operations, not syscall numbers or capability bits.

Verdict: real attempt, blocked. This is a genuine, targeted CVE-2022-0492 exploitation attempt — not a misconfigured test, not a scanner false positive — that produced exactly one Warning-priority alert and two kernel-level denials, and never got further. Still worth an incident ticket and a rotation of whatever credentials that RCE could reach — a blocked escalation doesn't mean the initial code execution wasn't real — but the container escape itself never had a chance.

What made the difference: this is the one ticket in the drill where the hardening was already live, and it's the reason "attempt" and "attempt that worked" are different rows in an incident log instead of the same row.

1. Read the alert's own process and container fields first; cross-reference an identity that could explain it (Kubernetes audit log or a ticketing system); check whether the attempted action actually completed by pulling node-level kernel logs; and pull the pod's actual securityContext to know what was even in place. The order matters because each step either narrows or rules out a verdict category before you spend effort on the next one — jumping straight to "was this blocked" without first checking for a legitimate identity would waste time chasing a false positive as if it were a real incident.
2. Kubernetes RBAC scoped to the checkout ServiceAccount's actual permissions returned a 403 when the stolen token tried to create a Pod in kube-system. The drill insists this isn't "the workload being hardened" because that RBAC scoping exists for an unrelated reason — least-privilege IAM design, not a control anyone put there specifically to stop this attack — while the container itself had an empty securityContext: no capability drop, no non-root enforcement, no seccomp profile. The team was protected by a control that happened to exist, not by anything defending this specific workload.
3. The alert's parent field: ticket 1's shell was spawned by php-fpm (a service process reacting to input), while ticket 2's was spawned by containerd-shim (what a kubectl exec attach looks like from inside the container) — a lead worth chasing, not proof on its own. The Kubernetes audit log confirmed it, showing an authenticated exec subresource request from alex@acme.com through a scoped break-glass group, with timestamps matching an open incident ticket.
4. Dropped capabilities meant the process never held CAP_SYS_ADMIN, so the cgroup release_agent technique had no capability to build on in the first place. Seccomp's RuntimeDefault profile independently denied the mount syscall (syscall 165 on x86_64) the technique needs to set up a fresh cgroup hierarchy, regardless of capabilities. AppArmor's profile independently denied the write into the cgroupfs release_agent path itself, on the file-and-path axis rather than the syscall-number axis — a third, structurally different layer that would have stopped the write even if the mount had somehow gotten through.
5. allowPrivilegeEscalation: false sets the kernel's no_new_privs flag on the process. With that flag set, the kernel ignores setuid and setgid bits, and file capabilities, at exec time — so even if an attacker somehow still managed to set the setuid bit on a binary (which running non-root should already prevent, since chmod's ownership check fails once the process's UID no longer matches the file's, and dropped capabilities removes the CAP_FOWNER bypass for that mismatch), executing that binary would no longer grant the elevated privilege the bit is supposed to confer. It closes the specific loophole the other controls leave open if any one of them is misconfigured.