Container Runtime Security
Trivy scanned the image clean. Kyverno's admission policy let the Pod through because it ran as non-root with every capability dropped. Pod Security admission stamped it restricted. Six hours into production, a deserialization bug in the checkout service lets an attacker's input reach php-fpm, and php-fpm calls execve("/bin/sh") — and not one of those three checks fires again, because all three already finished their job the moment the container started. This page is about the layer that's still watching: anomalous process execution, unexpected network connections, and privilege escalation attempts inside a container that's already running, caught by eBPF-based runtime detection instead of anything baked into the image. By the end you should be able to explain why eBPF replaced kernel modules and ptrace for this job, name the syscall-level signal each of the three attack classes above actually produces, and read a Falco rule well enough to know exactly what it's watching for before the next page takes the tool itself apart.
Checking a lunchbox before school tells you exactly what's inside it — sandwich, no allergens, nothing spilled. It says nothing about who the kid trades sandwiches with once they're sitting in the cafeteria. Scanning a container image is the lunchbox check: thorough, useful, and finished the moment the container starts running. Runtime security is the part that keeps watching after that — not because the lunchbox check failed, but because it was never built to see behavior, only contents.
The image was clean. The behavior isn't.
☺ Like you're 10: A locked front door stops a stranger from walking in. It says nothing about what happens once someone who was already allowed inside starts opening drawers they've never touched before.
Every gate this course has covered so far runs once, at a specific moment, and then it's finished. Container & supply-chain security scanned the image before it was pushed. Kubernetes Security Deep Dive walked the write-time chain — RBAC, admission control, Pod Security Standards — that decided whether the object was even allowed to be created. All of that happens before or at the instant a container starts. None of it is still running an hour later, which is precisely the gap this page exists to close.
A build-time scan or an admission-time policy is structurally blind to an entire category of real incidents, not because either tool is poorly built, but because neither was designed to observe behavior over time:
- A CVE disclosed after the scan ran, against a package that was clean on Tuesday and is still sitting, unpatched and running, in production on Thursday.
- A legitimate, already-present binary used for something it was never meant for.
bash,curl, andpythonare supposed to be in the image — there's nothing "vulnerable" for a scanner to flag when an attacker simply uses tools that were always there. - A process the image's own entrypoint never listed at all, spawned because application logic reacted to attacker-controlled input in a way no manifest could have anticipated.
- Drift introduced after deploy — someone
execs into a running container during an incident and installs a package, or a file gets rewritten in a directory that was scanned clean at build time and never looked at again. - An escape attempt that targets the host kernel underneath a perfectly clean container userspace — no image scanner inspects the kernel a container happens to be running on.
Every item on that list happens after the last gate that could have said no has already said yes and moved on to the next request. Recall the defense-in-depth diagram from the previous page: RBAC, admission control, and Pod Security Standards all sat in the top row, labeled write-time, gating what's allowed to start. NetworkPolicy and runtime tooling sat in the bottom row, labeled continuous, still watching an hour later. This page is that bottom row, worked all the way down to the syscall.
eBPF: watching every syscall without a kernel module
☺ Like you're 10: Imagine a tiny, closely supervised helper who's allowed to stand right next to the kernel's front door and take notes on everyone who walks through — but who gets fired instantly, before it ever starts, if it tries to do anything except take notes.
eBPF — extended Berkeley Packet Filter, though almost nobody expands the acronym anymore — is a way to run small, sandboxed programs inside the Linux kernel, triggered by kernel events, without writing a kernel module and without recompiling the kernel itself. It grew out of classic BPF, a narrow 1992 mechanism for filtering network packets, and was generalized into a full in-kernel virtual machine starting in the mid-3.x kernel series, maturing rapidly through the 4.x and 5.x releases into the general-purpose observability and enforcement layer it is today. The programs attach to hooks: kprobes and kretprobes (dynamic tracing of almost any kernel function, entry and return), tracepoints (stable, kernel-maintained markers — the ones Falco relies on most heavily for syscall entry and exit), raw tracepoints, LSM hooks (Linux Security Module attachment points, available since kernel 5.7, which let an eBPF program participate in security decisions rather than just observe them), perf events, and cgroup hooks. For runtime container security specifically, the hooks that matter are the ones that fire on execve, connect, capset, and their neighbors — because those four syscalls are where process execution, network connections, and privilege changes all become visible to the kernel in the first place.
What makes this safe to run on every syscall, on every node, all the time, is the verifier. Before an eBPF program is allowed to load, the kernel statically analyzes it: every loop must be provably bounded, every memory access must be provably within bounds, every pointer must be provably valid for what it's being used for. A program that fails any of those checks is rejected outright — it never executes a single instruction. A program that passes gets JIT-compiled to native machine code, so the per-hook overhead of "run this check" is close to a native function call rather than a trap into a separate tracing process. That combination — provably safe before it runs, and near-native speed once it does — is the whole reason eBPF displaced the tooling that came before it for this exact job.
| Approach | Runs as | Container-aware? | Typical overhead | If it goes wrong |
|---|---|---|---|---|
| Custom kernel module | Out-of-tree code, compiled against one exact kernel build | No — needs manual PID-namespace correlation | Low once built, but brittle across kernel upgrades | A bug can crash the entire host, not just the module |
ptrace-based tracing | A userspace tracer process, one per tracee | Yes, but expensive to scope across a whole node | High — every traced syscall forces a context switch | The tracee stalls on every hit; doesn't scale to a busy node |
auditd | The kernel audit subsystem, delivered over netlink | No — one global backend, blind to namespace boundaries | Grows quickly as rules get more specific | Rarely crashes, but can throttle the whole system under load |
| eBPF (modern driver) | Verifier-checked, JIT-compiled in-kernel program | Yes — cgroup and PID-namespace metadata come for free | Low, typically single-digit percent CPU | Cannot corrupt the kernel — the verifier already rejected anything unsafe before load |
Two more pieces close the loop between "safe to run" and "actually useful at scale." CO-RE (Compile Once – Run Everywhere), backed by BTF (BPF Type Format, a compact description of a kernel's own struct layouts embedded in the kernel itself), lets one compiled probe read the right field offsets on whatever kernel it happens to land on, instead of being hand-compiled per kernel version the way the old kernel-module driver had to be — this is what let Falco drop its per-node build step entirely. And events don't reach userspace by being polled for; they're pushed through a ring buffer BPF map (the BPF_MAP_TYPE_RINGBUF type, added in kernel 5.8), a lock-light, shared-memory queue built specifically so a busy node generating thousands of syscalls a second doesn't turn "watch everything" into "fall behind everything."
eBPF is the mechanism, not the product. It gives you a fast, safe way to see every syscall in the kernel — it has no opinion whatsoever about which ones matter. Deciding that is a rules engine's job, which is exactly what the next three sections build toward, and what Falco does in full on the next page.
Signal one: anomalous process execution
☺ Like you're 10: If the family dog has only ever barked at the mail carrier, and one day it starts quietly opening kitchen cupboards, you don't need a list of "bad dog behaviors" written down anywhere — you already know that's not what this dog does.
Every containerized workload has an expected process ancestry. An nginx container spawns nginx worker processes, not bash. A Java service spawns JVM threads, not python -c. The syscall that creates every one of these processes is execve, and hooking it — via a kprobe or tracepoint on sys_execve, or the newer LSM bprm_check_security hook — gives a runtime tool visibility into every single process spawned inside every container on the node, each one already tagged with exactly which container and Pod it happened in.
Attackers know image scanners look for known-vulnerable or known-malicious files, so a common technique is to avoid dropping any new file at all and instead reuse binaries that are already sitting in the image, legitimately, for legitimate reasons — a pattern security teams call living-off-the-land. curl piped into a shell, base64 -d piped into an interpreter, a python -c one-liner, wget quietly fetching a second-stage payload — none of these binaries are vulnerable, so an SCA or image scan has nothing to flag. What's abnormal isn't the binary's presence; it's the pattern of how and when it got used, and that pattern only exists at runtime.
# Modeled closely on Falco's own default rule set — check falco_rules.yaml
# in your installed version for the exact current wording.
- rule: Terminal shell in container
desc: >
A shell was spawned by a process inside a container, with a terminal
attached — almost never legitimate for a service container.
condition: >
spawned_process and container
and shell_procs and proc.tty != 0
and container.id != host
output: >
A shell was spawned in a container with an attached terminal
(user=%user.name container_id=%container.id container_name=%container.name
shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline terminal=%proc.tty
image=%container.image.repository)
priority: NOTICE
tags: [container, shell, mitre_execution]Read the fields, not just the shape: condition is a boolean expression over live event data — proc.tty != 0 specifically excludes the shells a container legitimately starts with (PID 1, non-interactive) and keeps only ones with an attached terminal, which is what an attacker's interactive session actually looks like. output is a template, interpolated at alert time from the same event, so the alert that lands in Slack already carries the container name, the exact command line, and the parent process — not a bare "something happened" that someone still has to go dig context for. tags commonly carry a MITRE ATT&CK technique ID — T1059, Command and Scripting Interpreter, is the one this particular rule maps to — so a pile of Falco alerts can be rolled up into the same framework a threat-modeling or purple-team exercise already uses.
Run Falco locally against a scratch container — the project ships a container image with default rules preloaded. Start a plain nginx container next to it, then docker exec -it (or kubectl exec -it on a cluster) into that nginx container and just sit at the resulting shell for a second. Watch Falco's output: the "Terminal shell in container" rule should fire within a second of the shell attaching, carrying the exact container name and command line in its output. That's the entire mechanism this section describes, happening in front of you instead of read about.
Signal two: unexpected network connections
☺ Like you're 10: A locked door stops a stranger from walking in. It has nothing to say about a family member who's always allowed through that door suddenly calling a phone number nobody in the house has ever dialed.
The same hooking approach applied to connect, accept, and sendto gives per-process, per-container visibility into every network connection a container attempts — source and destination IP, port, and protocol, all tagged with exactly which process in which container made the call. That's a materially different signal from what NetworkPolicy gives you, and it's worth being precise about the difference rather than treating the two as redundant.
NetworkPolicy makes a binary allow/deny decision, once, ahead of time, based on selectors and CIDRs — a Pod either matches an allow rule or it doesn't. It has no concept of "this specific connection, from this specific process, right now, is unusual." A Pod with a broad egress rule permitting traffic to an internal CIDR range is fully compliant with its own NetworkPolicy the instant it starts beaconing out to a host inside that range it has never once talked to before in its entire history — NetworkPolicy has nothing further to say about that, because by its own rule, the connection is allowed. Runtime detection is the layer that still has an opinion after the policy has already said yes.
Concrete indicators worth building rules around: connections using the stratum+tcp:// mining-pool protocol or to known cryptomining infrastructure; outbound connections on ports classically associated with reverse shells (4444, 1337, and similar); DNS-based tells — DGA-style beaconing to freshly registered domains, or abnormally large and frequent TXT or NULL record queries, both signatures of DNS tunneling used to exfiltrate data or maintain command-and-control through a channel that's rarely blocked outright; and raw socket usage that bypasses a workload's normal application libraries entirely. Correlating destination IPs and domains against a live threat-intel feed — the same kind of feed secure by design & threat intel covers — turns a bare "unusual connection" alert into "unusual connection, to an IP a KEV bulletin flagged three days ago," which is a very different priority.
# Falco condition fragment — an unexpected outbound connection from a container # that should only ever be spoken to, never speak out condition: > outbound and container and not fd.sip in (allowed_egress_ips) and fd.sport != 0 # key fields available on a network event: # fd.sip / fd.sport — source IP / port (inside the container) # fd.rip / fd.rport — remote IP / port (the connection's destination) # fd.name — full socket descriptor, e.g. "10.2.4.9:51322->203.0.113.9:4444" # evt.type — connect, accept, sendto, ...
Signal three: privilege escalation inside the container
☺ Like you're 10: A container is supposed to believe it's the only thing running on the computer. Privilege escalation is what it looks like when the container figures out that isn't true, and starts quietly testing how much of the real machine underneath it can actually reach.
The syscall-level tells here cluster around identity and capability changes: setuid/setgid calls, capset altering a process's Linux capability set, a process calling prctl to try to work around PR_SET_NO_NEW_PRIVS, writes landing in /etc/passwd, /etc/shadow, /etc/sudoers, or a cron directory, and a newly created file getting its setuid bit set via chmod. Each of these is a legitimate syscall doing something a container almost never has a legitimate reason to do.
The sharpest version of this signal is a full container escape, and three real, disclosed vulnerabilities make the syscall-level tell concrete rather than theoretical. CVE-2019-5736 let a malicious container overwrite the host's own runc binary by writing through /proc/self/exe just as a subsequent exec operation was about to re-run it — the next command run against that container executed on the host instead. CVE-2022-0492 abused a legacy cgroup v1 feature: a process holding CAP_SYS_ADMIN inside a container could write to that cgroup's release_agent file, arranging for an arbitrary host-level command to run the moment the cgroup's last process exited. CVE-2022-0847, nicknamed Dirty Pipe and disclosed by Max Kellermann, was a kernel bug in how pipe buffers tracked a flag that let an unprivileged process overwrite bytes in a file it only had read access to — including a file bind-mounted read-only into a container — which is enough to corrupt /etc/passwd or a SUID binary and escalate straight through a boundary that was supposed to be immutable.
| Technique | What it does | Syscall-level tell |
|---|---|---|
| runc overwrite — CVE-2019-5736 | Overwrites the host's runc binary via /proc/self/exe, so the next exec against the container runs on the host. | An open() with write access resolving to /proc/*/exe or the runc binary path, from inside a container. |
| cgroup release_agent — CVE-2022-0492 | Abuses cgroup v1's release_agent hook to run an arbitrary host command when the cgroup empties. | A write to a path ending in release_agent or notify_on_release from inside a container. |
| Dirty Pipe — CVE-2022-0847 | Overwrites bytes in a file the process only has read access to, via a pipe-buffer flag bug — including read-only bind mounts. | Unusual splice()/pipe syscall sequences targeting a file the calling process has no write permission on. |
| docker.sock mount — misconfiguration, not a CVE | Grants direct, unauthenticated, root-equivalent control of the host's Docker daemon from inside the container. | A connect() to the Unix socket at /var/run/docker.sock, or the docker CLI itself running inside a container. |
If you find a Pod with /var/run/docker.sock bind-mounted into it, that is not a finding to triage on a severity scale — it's root on the node, full stop, and it should be treated as an active incident until proven otherwise rather than a backlog ticket. It's also exactly the kind of misconfiguration Rocky the Raccoon goes looking for in offensive security for DevSecOps, and the same "assume the process is hostile" reasoning zero trust for pipelines applies to what a compromised CI runner can reach on the underlying host.
What all four rows share is the reason behavior-based detection outperforms a CVE-specific signature here. A signature rule needs to know about CVE-2022-0847 by its exact identifier before it can catch it. A rule watching for "a process writes to a file it doesn't have permission for, via an unusual pipe-syscall sequence" catches Dirty Pipe and the next kernel bug that shares the same general shape, without a single update, because the signal is the syscall pattern itself, not the CVE number attached to whichever exploit happened to demonstrate it first. That's the single biggest practical argument for eBPF-based behavioral rules over pure vulnerability signatures: novel variants of a known technique class still trip the same rule.
From syscall to alert: Falco's architecture
☺ Like you're 10: Everything so far has been about what to watch for. This section is about the one program that's actually doing the watching, continuously, on every node in the cluster.
Falco is the tool that turns everything in this page from a description into a running system, and it's worth understanding its shape before the next page opens it up fully. Its driver — the component that actually captures raw syscall events — has evolved through three generations: a custom out-of-tree kernel module (Falco's roots trace back to Sysdig's own driver), then a legacy eBPF probe offered as an alternative, and now, in current releases, a modern eBPF driver built with CO-RE as the default — check the exact default against whatever version you deploy, since this has shifted release over release. The practical payoff of the modern driver is exactly the CO-RE point from earlier in this page: no more compiling a kernel-module build per node image, and lower overhead than either of its predecessors.
Above the driver, libscap captures the raw event stream off whichever driver is active, and libsinsp parses and enriches it — resolving the container ID against the container runtime's own socket, and, on Kubernetes, resolving Pod name, namespace, labels, and image from the API or a metadata fetcher, so a raw execve event becomes "this happened in checkout-7d9f-x8, namespace payments, image registry.acme.io/checkout:1.8.2" before a rule ever sees it. The rule engine then evaluates every enriched event against whatever's loaded — Falco's own defaults plus anything custom layered on top, in the shape shown throughout this page: rule, desc, condition, output, priority, tags.
Where the alert actually goes is its own layer. Falco writes natively to stdout, a file, or syslog — a reasonable baseline, and often where a first deployment starts. Falcosidekick, a companion project, fans a single alert stream out to more than fifty destinations — Slack, PagerDuty, Elasticsearch, S3, a generic webhook, a SIEM ingestion endpoint — without Falco itself needing to know anything about any of them. Falco Talon, a newer response engine, goes one step further: it can act on a match automatically, labeling or cordoning the node, deleting or network-isolating the offending Pod, or applying an emergency NetworkPolicy, turning a detection into a remediation with no human in the loop for the cases a team trusts enough to automate. Falco itself is a CNCF project, donated by Sysdig and graduated within the foundation in 2022 — worth confirming against the current CNCF landscape if its governance status matters for a compliance or vendor-risk conversation.
That's the shape of it. Installing Falco, writing rules from scratch instead of just reading them, and the operational details of running it as a DaemonSet across a real fleet are the whole subject of the next page: Falco.
Running it for real: overhead, tuning, and response
☺ Like you're 10: A smoke alarm that goes off every time someone makes toast gets its battery pulled within a week. A detection system nobody trusts is worse than no detection system at all, because everyone's already learned to ignore it.
The modern eBPF driver's overhead is typically low single-digit percent CPU on a busy node — a real improvement over both the legacy kernel-module driver and anything ptrace-based, but not zero, and worth benchmarking against your own workload rather than trusting a number measured on someone else's cluster with a very different syscall rate.
Falco needs to load an eBPF program and read raw syscalls across the entire node, which in practice means a privileged (or near-privileged, depending on kernel version and the specific capabilities granted — CAP_SYS_ADMIN, CAP_BPF, CAP_PERFMON) DaemonSet with hostPID and access to /proc and the kernel debug filesystem. That's an unusually large footprint for the very tool that's supposed to be watching for privilege escalation. Pin its image by digest, keep it current, and give its own RBAC and mounts the same scrutiny you'd give anything else carrying that grant — the watcher having a lot of privilege is the tradeoff, not an oversight, but it still needs to be treated as one.
The operational risk that actually kills a runtime-detection rollout is rarely overhead — it's false-positive fatigue. Start new or custom rules at a lower priority, or run Falco's -A/audit-style posture where practical, and watch a rule in shadow mode against real traffic before wiring it into a paging path. Build exceptions and macros scoped to a specific workload rather than disabling an entire rule cluster-wide the first time it's noisy on one deployment. Route only CRITICAL and EMERGENCY priority alerts somewhere that wakes a human up; let everything below that land in a dashboard or a ticket queue instead of a pager. Falco can also ingest the Kubernetes audit log as a second event source alongside syscalls — which is what lets it flag kubectl exec, kubectl cp, and anonymous API access as first-class detections, catching the human and API side of runtime abuse with the same rule language covered throughout this page.
None of this replaces the layers that came before it — it completes them. Kubernetes Security Deep Dive covers everything that decides what's allowed to start; this page covers everything that keeps watching once it has. When a rule like the ones on this page actually fires on something real, that alert is the start of Foxy's job, not the end of Recon's — and if you want to practice the whole chain hands-on, the container escape investigation drill walks through exactly this kind of finding from first alert to root cause.
Recon the Robot: New alert. Pod checkout-7d9f-x8 — a shell just spawned inside it. It's never done that once in forty days of history.
Benny the Beaver: That image passed every scan, Recon. Trivy came back clean, Kyverno let it through non-root, capabilities dropped. What's even left to catch?
Recon the Robot: Everything that happens after your image starts running, Benny. Nothing in a scan report sees an exec into a live container six hours later.
Timmy the Turtle: I gated the build. Recon gates what the build does once nobody's watching the pipeline anymore.
Foxy: Fair — but is it actually malicious, or is that just curl checking a health endpoint in a slightly odd way?
Recon the Robot: That's the tuning problem, and it's a fair one to ask every time. But this workload's process tree has never included /bin/sh before, not once. That's the signal — not the binary, the pattern.
Rocky the Raccoon: For what it's worth, this is exactly the box I'd tick first if I were breaking in. A shell inside a Pod is a foothold, not the disaster. What it reaches next is.
1. Name three specific things a build-time image scan or a write-time admission policy structurally cannot see, and explain why runtime detection is the only layer left that can catch them. 2. What does the eBPF verifier actually guarantee before a program is allowed to load, and why does that make eBPF safer for this job than a custom kernel module? 3. Give one behavioral signal each for anomalous process execution, an unexpected network connection, and privilege escalation inside a container — and name a real CVE that produces the third one. 4. What's the practical difference between what NetworkPolicy blocks and what runtime network-connection detection flags, given a connection that's technically inside an already-allowed CIDR? 5. Why does a behavior-based rule — a shell spawning inside a container that's never spawned one before — often catch novel exploits that a CVE-specific signature rule would miss entirely?
Check your answers
- Any three of: a CVE disclosed after the scan already ran, against a package already deployed; a legitimate, already-present binary (bash, curl, python) used abnormally, with nothing "vulnerable" for a scanner to flag; a process the image's entrypoint never declared, spawned by application logic reacting to attacker input; drift introduced after deploy, such as a live
execinstalling a package; or a host-kernel escape attempt underneath a userspace that scanned perfectly clean. Runtime detection is the only layer still running after the container has started, which is exactly when all of these become possible. - The verifier statically proves, before a single instruction runs, that every loop is bounded and every memory access stays within safe limits, rejecting the program outright if it can't prove that. A custom kernel module carries no such guarantee — a bug in it can crash the entire host — while a rejected eBPF program simply never loads.
- Process execution: a shell (
/bin/sh) spawned inside a container that has never spawned one, via a hook onexecve. Network: an outbound connection to a host or port the workload has never talked to before (e.g., a reverse-shell port), via a hook onconnect. Privilege escalation: a write to a cgroup'srelease_agentfile, corresponding to CVE-2022-0492 (also acceptable: CVE-2019-5736 for a write through/proc/self/exe, or CVE-2022-0847 "Dirty Pipe" for an unusual pipe-splice write to a read-only file). - NetworkPolicy makes a one-time, binary allow/deny decision ahead of time based on selectors and CIDRs; a connection inside an already-allowed CIDR is, by definition, compliant with the policy and NetworkPolicy has nothing further to say about it. Runtime detection still evaluates that specific connection, from that specific process, against the workload's own behavioral history, and can flag it as unusual even though NetworkPolicy already said yes.
- A signature rule has to know the exact CVE identifier or exploit pattern in advance to catch it. A behavior-based rule watches for the underlying syscall pattern shared by a whole class of techniques — an interactive shell where none should exist, a write to a file the process can't normally write — so it also catches the next exploit that produces the same pattern, without ever being updated for a new CVE number.
The next page takes the tool this page kept pointing at and opens it up completely: Falco — installation, the full rule-writing workflow, and running it across a real fleet. If runtime and container security is the syllabus you're actually studying toward, the CKS — Kubernetes Security Specialist exam leans heavily on exactly this material, and Part 4 of the capstone is where you harden and sign a container before any of this ever gets a chance to run.