Falco

Falco is a CNCF-graduated runtime security engine that sits on every node, watches the stream of system calls your containers make to the Linux kernel, and shouts when something happens that should never happen — a shell opening inside a running container, a process writing into /usr/bin, a workload reading /etc/shadow, a pod suddenly dialling an address it has never dialled before. It solves the platform problem that admission control structurally cannot: your manifests were perfect, your images were signed, everything passed the gate — and then, at 4am on a Tuesday, the process inside the container started behaving like an attacker.

What Falco is and the problem it solves

☺ Like you’re 10: It listens to everything your programs ask the computer to do, and rings a bell when something is never-normal.

Falco was created at Sysdig, donated to the CNCF in 2018, and graduated in 2024 — the first runtime security project to do so. It runs as a DaemonSet: one agent per node, watching that node’s kernel activity for every container scheduled onto it. It is a detection engine. It observes behaviour, matches it against a ruleset, and emits an alert with enough context — which pod, which namespace, which image, which user, which command line — for a human or an automation to act.

The gap admission control leaves

Everything else on the security page of a platform happens before a workload runs. Kyverno and OPA Gatekeeper inspect the object at admission. Trivy scans the image in the pipeline. Sigstore/cosign proves the bytes came from your build. Each of those is a check on a static artefact, and each is worth having — but all of them are finished by the time the container’s first process is exec’d.

That leaves a real gap, and it is not theoretical. Your image passed a scan on Monday and a critical CVE was published on Wednesday. Your application has a perfectly legitimate feature that turns out to be a path traversal. A dependency ships a post-install script nobody read. A valid credential leaks and someone uses it correctly, from the right place, to do something wrong. In every one of those cases the manifest is compliant, the image is signed, admission said yes — and the only remaining signal is what the process actually does at runtime.

Syscalls as the source of truth

Falco taps that signal at the narrowest, hardest-to-lie-about point in the system: the system call boundary. Every meaningful thing a program does — opening a file, spawning a child process, connecting a socket, mounting a filesystem, changing a namespace — must cross from user space into the kernel through a syscall. A process can obfuscate its source, rename its binary, or pack itself, but it cannot read a file without asking the kernel to open it.

So Falco does not try to understand your application. It watches execve, open, openat, connect, setns, chmod and their siblings, enriches each event with process lineage and container identity, and evaluates a set of boolean conditions written against those fields. That is genuinely the whole idea, and its simplicity is why it survives contact with software Falco has never heard of.

What it is not

Falco does not, by itself, stop anything. Out of the box it is a very loud smoke alarm, not a sprinkler — it writes an alert and moves on. (Response is a separate component, Falco Talon, covered below.) It is not a vulnerability scanner; knowing your image contains a critical CVE is Trivy’s job. It is not a network policy engine — it can tell you a pod made an unexpected outbound connection, but blocking that connection belongs to Cilium or a plain NetworkPolicy. And it is not a log aggregator, though its alerts should absolutely end up in one; see Observability.

Where it fits in a platform

☺ Like you’re 10: It lives on every machine, underneath everyone’s apps, watching the floor rather than the front door.

Falco belongs to the node plane — the layer of the platform that lives on the Kubernetes substrate itself rather than in the control plane. It is privileged infrastructure: it needs access to the kernel, so it runs with elevated permissions on every node, which makes it something the platform team owns absolutely and tenants never touch. Functionally, though, its output belongs to the observability and incident plane: alerts, routed somewhere a human sees them.

Its neighbours

Upstream of Falco sit the preventive controls — Kyverno at admission, cosign on the supply chain, secrets management keeping credentials out of images. Falco assumes all of them worked and asks what happens next. Downstream, its alerts flow into the same pipes as everything else: Prometheus and Alertmanager for on-call, Loki or a SIEM for retention and search, Grafana for the dashboard. Its deployment is delivered like every other add-on, by Argo CD or Flux from the config repo described in GitOps. And the runbook that says what to do when it fires belongs on Reliability & Incidents.

Prevention and detection are a pair

The clearest way to hold this in your head is a two-line rule. If you can express it as a property of the object, prevent it at admission. “No privileged containers,” “no hostPath mounts,” “images only from our registry” — those are Kyverno rules, and enforcing them at the gate is strictly better than detecting the consequences later. If it can only be expressed as a property of behaviour, detect it at runtime. “A shell started inside a container,” “something wrote a new binary and immediately executed it,” “a pod contacted the API server for the first time in its life” — no admission policy can see those, because none of them exist until the workload is already running.

CNPE domain relevance

Be clear about this: Falco is not on the official CNPE tool list. The exam’s security tooling is Kyverno, OPA/Gatekeeper and the service meshes. You will not be asked to write a Falco rule. What Falco gives you is the concept the exam does test — that Domain 5, Security & Policy Enforcement, is not only about admission, and that a reference platform has a runtime detection layer with somewhere for its alerts to go. If a scenario question describes a compromise that a policy engine could not have caught, runtime security is the shape of the answer. Study Security & Policy Enforcement for the domain, and know the difference cold.

How it works — architecture and components

☺ Like you’re 10: A little listener down in the kernel passes events up to a rule-checker, which passes alerts out to wherever you want them.

Falco is three layers stacked on each other: a driver that captures kernel events, a userspace engine that enriches and evaluates them, and an output stage that emits alerts. Nearly every operational problem you will ever have with Falco lives in one specific layer, so it pays to be able to name them.

The driver — three ways to see the kernel

Falco needs a privileged foothold in the kernel to observe syscalls, and it offers three of them — plus two special-case modes that do not capture host syscalls at all. All of them are selected in falco.yaml under engine.kind, and choosing correctly is the single most common installation decision.

The companion tool falcoctl handles the artefacts: it can download or build the right legacy probe or module for a node’s kernel, and it also manages rules and plugins as OCI artefacts, which is how a cluster keeps its ruleset current without a Helm upgrade.

The userspace engine and enrichment

Raw syscalls are almost useless on their own — openat(AT_FDCWD, "/etc/shadow", O_RDONLY) from PID 4471 tells you nothing actionable. The userspace engine (the libsinsp library) maintains live state: a process table with full parent lineage, open file descriptors, and container identity resolved from the container runtime via the CRI socket. That is what turns the raw event into “the process cat, spawned by bash, spawned by the entrypoint, inside container a1b2c3 from image ghcr.io/acme/api:1.4.3, in pod api-7f9 in namespace payments.”

Kubernetes metadata beyond what the CRI provides — labels, owner references, the full pod spec — comes from the k8smeta plugin in current Falco versions, fed by a small k8s-metacollector deployment that watches the API server once for the whole cluster and streams to each agent. This replaced the older design in which every Falco pod on every node held its own watch on the API server, which scaled badly on large clusters.

Sources and plugins

Syscalls are Falco’s native event source, but not its only one. Plugins add sources and fields: k8saudit makes Falco a detector over the Kubernetes audit log (“someone created a ClusterRoleBinding granting cluster-admin”), and there are equivalents for cloud trails and SaaS audit streams. Each rule declares which source it belongs to with a source: field, defaulting to syscall. This matters for the mental model: Falco is a general condition-matching engine over event streams, and syscalls happen to be the most valuable stream available.

The rules you will actually write

☺ Like you’re 10: A rule is one sentence: “when THIS happens, print THAT, and here’s how worried to be.”

Falco ships a substantial default ruleset (falco_rules.yaml) which you should treat as read-only — it is updated by the project and will be overwritten. Your own rules and your tuning go in falco_rules.local.yaml or files under /etc/falco/rules.d/.

Anatomy of a rule, a macro and a list

A rules file is a YAML list of three object types. A rule is the alert itself; a macro is a named, reusable condition fragment; a list is a named collection of values. Macros and lists exist so that the same idea (“this is a container,” “this is a shell,” “these are our trusted images”) is written once and referenced everywhere.

# /etc/falco/rules.d/acme-platform.yaml

# the minimum rules-engine version this file needs; check yours with `falco --version`
- required_engine_version: 0.26.0

# ---- lists: named collections of plain values ----
- list: acme_trusted_debug_images
  items: [ghcr.io/acme/toolbox, ghcr.io/acme/netshoot]

- list: acme_sensitive_paths
  items: [/etc/shadow, /etc/sudoers, /root/.ssh, /var/run/secrets]

# ---- macros: named, reusable condition fragments ----
- macro: acme_container
  condition: container.id != host          # a shipped default macro is simply "container"

- macro: acme_shell
  condition: proc.name in (bash, sh, zsh, ash, dash, ksh, csh, fish)

# ---- the rule ----
- rule: Shell spawned in a production container
  desc: >
    An interactive shell was executed inside a container in a production namespace.
    Legitimate for a named break-glass image; suspicious anywhere else.
  condition: >
    spawned_process
    and acme_container
    and acme_shell
    and k8s.ns.name in (payments, checkout)
    and not container.image.repository in (acme_trusted_debug_images)
  output: >
    Shell in prod container
    (user=%user.name uid=%user.uid shell=%proc.name parent=%proc.pname
    cmd=%proc.cmdline pod=%k8s.pod.name ns=%k8s.ns.name
    image=%container.image.repository:%container.image.tag)
  priority: WARNING
  tags: [container, shell, mitre_execution, T1059]
  source: syscall

Two details in that condition carry all the weight. spawned_process is a default macro shipped with Falco meaning “an execve-family syscall on the exit side of the call” — the exit side matters because that is when the new process’s name and arguments are actually known. And every %field in output is a filter field rendered at alert time, which is why writing a good output line is not cosmetic: it is the difference between an alert someone can triage in ten seconds and one that starts a twenty-minute investigation.

Priorities and the default ruleset

Every rule declares a priority, drawn from a fixed syslog-style ladder. Falco’s priority setting in falco.yaml sets the minimum level it will load and emit, and downstream routing almost always keys off it.

The default ruleset is worth reading once in full, because it is a compact education in what container attacks look like. The classics: Terminal shell in container, Write below binary dir, Read sensitive file untrusted, Launch Privileged Container, Contact K8S API Server From Container, Change thread namespace (a container-escape signal), and drift detection rules that fire when a brand-new executable appears in a container and is then run — an immutable-infrastructure violation that is almost always either a bad build or an intruder.

Tuning without forking — exceptions and override

You will need to silence legitimate behaviour that trips a default rule, and the wrong way to do it is to copy the rule into your own file and edit it, because you have now forked something the project updates. The right way is one of two mechanisms. An exceptions block attaches named, field-based carve-outs to a rule. An override block appends to (or replaces) a specific part of an existing rule by name.

# Tuning the SHIPPED rules without editing falco_rules.yaml

# 1. Append items to a shipped list — the cheapest, safest tuning of all.
- list: falco_privileged_images
  items: [ghcr.io/acme/node-agent]
  override:
    items: append

# 2. Append a clause to a shipped rule's condition.
#    Any list you reference must already be defined, so define it first.
- list: acme_sidecar_entrypoints
  items: [acme-sidecar, supervisord]

- rule: Terminal shell in container
  condition: and not proc.pname in (acme_sidecar_entrypoints)
  override:
    condition: append

# 3. A structured exception on your own rule: tuple-matched, not a blanket "not".
- rule: Shell spawned in a production container
  exceptions:
    - name: ci_debug_job
      fields: [k8s.ns.name, container.image.repository, proc.name]
      comps: [=, =, in]
      values:
        - [payments, ghcr.io/acme/ci-runner, [bash, sh]]
  override:
    exceptions: append

Prefer the third form where you can. A blanket and not proc.name = bash disables the detection everywhere; a tuple exception says “this exact image, in this exact namespace, running this exact shell” and leaves the rule doing its job everywhere else. That distinction is the difference between tuning and switching the alarm off — and it is exactly the same discipline as writing a narrow PolicyException rather than a wildcard exclude in Kyverno.

Deploying it — the Helm values that matter

Falco installs from the falcosecurity/falco chart, and four blocks of values account for most of what you will change: the driver, custom rules, output format, and the Falcosidekick subchart.

# values.yaml — installed by Argo CD / Flux from the config repo
driver:
  enabled: true
  kind: modern_ebpf              # engine.kind — no per-kernel artefact needed

collectors:
  kubernetes:
    enabled: true                # deploy k8s-metacollector for pod/ns enrichment

falco:
  json_output: true              # ALWAYS true in a cluster — machines parse this
  json_include_output_property: true
  priority: notice               # minimum priority to load and emit
  buffered_outputs: false
  syscall_event_drops:
    actions: [log, alert]        # tell me when the kernel buffer overflows
    rate: 0.03333
    max_burst: 10
  metrics:
    enabled: true                # Falco's own internals: drops, rule hits, resource use
    interval: 1h
    output_rule: true            # emit them periodically as an internal alert

# your rules, mounted into /etc/falco/rules.d/
customRules:
  acme-platform.yaml: |-
    - macro: acme_shell
      condition: proc.name in (bash, sh, zsh)
    - rule: Shell spawned in a production container
      desc: An interactive shell started inside a production container.
      condition: spawned_process and container and acme_shell and k8s.ns.name in (payments)
      output: Shell in prod (user=%user.name pod=%k8s.pod.name cmd=%proc.cmdline)
      priority: WARNING
      tags: [container, shell]

falcosidekick:
  enabled: true
  webui:
    enabled: true
  config:
    slack:
      webhookurl: ""             # from External Secrets — never in Git
      minimumpriority: warning
    alertmanager:
      hostport: http://alertmanager.monitoring:9093
      minimumpriority: notice
    loki:
      hostport: http://loki.monitoring:3100

Day-to-day commands

☺ Like you’re 10: Mostly you check the rule file is valid, check the agent is healthy, and read the alerts.

Validating and exploring rules

The falco binary can do a great deal without ever attaching to a kernel, which is what makes rule changes safe to review in a pipeline.

# validate rule files — wire this into CI for your rules repo
falco --validate /etc/falco/rules.d/acme-platform.yaml
# short form; -V takes one file and may be repeated
falco -V ./rules/acme-platform.yaml -V ./rules/acme-network.yaml

# what rules and fields do I actually have?
falco --list                            # every filter field, with descriptions
falco --list syscall                    # fields for the syscall source only
falco -L                                # list loaded rules
falco --list-events                     # every event type Falco can see

# dry-run the config without starting capture
falco --dry-run -c /etc/falco/falco.yaml

# replay a capture file instead of live syscalls — the best way to test a rule
falco -r ./rules/acme-platform.yaml -e ./captures/incident.scap

# run for 60 seconds, unbuffered, printing to stdout
falco -M 60 -U -o json_output=true

# support bundle: config, rules, versions, driver info — attach this to bug reports
falco --support | jq .

Running and inspecting it in a cluster

# is the agent healthy on every node?
kubectl -n falco get pods -o wide
kubectl -n falco get ds falco            # DESIRED should equal READY, always

# the alerts themselves — Falco writes them to stdout
# fromjson? skips the plain-text startup lines that would otherwise break jq
kubectl -n falco logs ds/falco --tail=50 | jq -R 'fromjson? | select(.priority=="Warning")'
kubectl -n falco logs ds/falco -c falco | grep -i "Terminal shell"

# did the driver actually load? this is where install failures show up
kubectl -n falco logs ds/falco -c falco | head -30
kubectl -n falco logs ds/falco -c falcoctl-artifact-install

# is the kernel dropping events? (see the gotchas below)
kubectl -n falco logs ds/falco | grep -i "drop"

# fan-out and response
kubectl -n falco logs deploy/falco-falcosidekick --tail=30
kubectl -n falco port-forward svc/falco-falcosidekick-ui 2802:2802

# prove it works — the step everyone skips
kubectl exec -it deploy/api -n payments -- bash -c 'cat /etc/shadow'
# then watch the alert appear in the Falco logs within a second or two

That last pair is the whole verification loop, and it belongs in your platform’s smoke tests. An unverified detector is indistinguishable from no detector; see the command reference for more of these one-liners.

Gotchas and failure modes

☺ Like you’re 10: A bell that rings all day is a bell nobody hears. Most of the work is making it ring only when it matters.

Tuning is the real work — and alert fatigue is the real failure

This is the thing to internalise above everything else on this page. Installing Falco takes twenty minutes. Making it useful takes weeks. The default rules are written to be broadly applicable, which means in your cluster a meaningful fraction of them will fire on entirely legitimate behaviour: your CI runner spawns shells, your log shipper reads paths that look sensitive, your service mesh sidecar changes namespaces at startup, your init container writes files that look like drift.

Teams that skip the tuning phase arrive at the same place every time: a Slack channel with four hundred alerts a day that everyone has muted, which is worse than having no runtime security, because it carries the appearance of coverage. Treat tuning as the project. Start with a soak: route everything to Loki or a SIEM only, send nothing to a human channel, and for two weeks aggregate by rule name. Silence or narrow the top offenders with tuple exceptions. Only then promote a small set of high-confidence rules to a channel someone reads, and a smaller set still — container escape, drift detection — to a pager. See Anti-patterns for the general shape of this mistake.

Driver and kernel compatibility

The second-most-common Falco outage is a node whose kernel the driver cannot handle. With kmod or legacy ebpf, the driver artefact is built against a specific kernel; a node pool upgrade, an autoscaled node from a newer image, or a distro that ships an unusual kernel all produce the same symptom — a Falco pod in CrashLoopBackOff or stuck in an init container, with the failure in the falcoctl-artifact-install logs rather than in Falco’s own. The whole point of modern_ebpf is that it removes this class of failure entirely by embedding a CO-RE probe in the binary, so use it wherever the kernel is ≥ 5.8.

Two related traps. Falco needs a real Linux kernel with the required capabilities, so managed environments where you do not control the node — most notably fully serverless container platforms — may not support it at all. And because Falco runs privileged on every node, any Kyverno or Pod Security policy you write must exclude its namespace, or your own guardrail will block your own security agent. Diagnosing a node-level DaemonSet that will not start is covered in Triage: Workloads.

Performance on busy nodes

Falco is not free. On a node running thousands of processes, the syscall stream is enormous, and both the kernel-side capture and the userspace evaluation cost CPU. In practice a well-tuned install is low single-digit percent of a node’s CPU, but a badly tuned one — every rule enabled, verbose sources on, a chatty workload — can be much worse. Give the DaemonSet explicit requests and limits, watch its own CPU as carefully as you watch its alerts, and remember the honest trade-off: broader capture means better detection and higher cost, and you are choosing a point on that line whether you think about it or not. The scheduling and cost pages both apply.

Detection is not prevention

The last trap is conceptual. A Falco alert means the thing already happened. Detection shortens dwell time; it does not stop the event. This is why Falco Talon exists: a response engine that consumes Falco’s output and matches its own rules to perform actions — terminate the offending pod, add a quarantine label, attach a deny-all NetworkPolicy, capture forensics, invoke a webhook. Automated response is powerful and genuinely dangerous, so start it in a dry-run posture, scope it to namespaces where killing a pod is safe, and never wire “terminate” to a rule you have not watched behave for a month.

Alternatives and when to choose it

☺ Like you’re 10: Some tools check the bag at the door, some watch the rooms — you want one of each, not two of the same.

The comparison that decides it

A practical rule

Read that table by column, not by row: only the first two columns are alternatives to each other. Choose Falco when you want a mature, portable, graduated detection layer with the largest community ruleset in the ecosystem and a huge fan-out story through Falcosidekick — it is the default answer, and it works on any Kubernetes distribution and outside Kubernetes too. Choose Tetragon when you are already running Cilium and specifically want in-kernel enforcement rather than detection-plus-response, accepting a tighter coupling to that stack. Do not choose between Falco and Kyverno; run both, because they answer different questions at different moments. Where all of these sit relative to one another is laid out on The Tool Landscape.

Exam relevance and going further

☺ Like you’re 10: You won’t be asked to write a Falco rule — but you should be able to say, in one sentence, why the front-door guard isn’t enough.

Falco is not on the official CNPE tool list, so calibrate your effort accordingly: no rule syntax to memorise, no CLI flags to drill. What is examinable is the concept it represents. Domain 5, Security & Policy Enforcement, expects you to reason about a defence-in-depth stack, and “runtime detection” is one of its layers. A scenario describing a compromise that passed every static check — a zero-day in a signed image, a leaked credential used correctly, an application vulnerability exploited at runtime — is asking you to notice that admission control could not have caught it.

The documentation allowlist — read this twice

⚖ CNPA vs CNPE — that whole allowlist mechanic is CNPE-specific — it exists only because CNPE is hands-on with a real cluster in front of you. CNPA is stricter, not looser: it is fully closed-book, multiple-choice, with zero lookups of any kind, so there is no man falco or Quick Reference box to lean on. That said, the runtime-detection concept above is still worth knowing cold for CNPA's closed-book recall.

What to be able to do without notes

Say in one sentence what Falco is: a node-level agent that watches kernel syscalls and alerts on behaviour that should never happen. Draw the three layers — driver, userspace engine, outputs — and name the driver options, with modern_ebpf as the modern default. State the prevention/detection split crisply enough to answer a scenario question: admission control stops bad configuration from being created; Falco catches bad behaviour once something is running; you want both. Name three default detections (terminal shell in a container, write below a binary directory, unexpected outbound connection). Know that Falcosidekick fans alerts out and Falco Talon performs response actions. And know the two biggest operational risks by name: false-positive tuning, and dropped events. Rehearse the surrounding domain on Practice: Security and Security & Policy Enforcement.

Official resources for after the exam

Outside the exam, the canonical sources are falco.org/docs (the Rules and Reference sections are the ones to read end to end), the searchable field reference at supported fields, the source at github.com/falcosecurity/falco, the shipped and community rules at falcosecurity/rules, the fan-out router at falcosidekick, the response engine at falco-talon, and the project’s CNCF page at cncf.io/projects/falco. Pair this page with Security & Policy Enforcement for the domain, Kyverno for the preventive half of the pairing, Observability for where the alerts land, Reliability & Incidents for what happens after one fires, and the glossary whenever a term stops making sense.