Linux Fundamentals for Platform Engineers

The LFCS blueprint teaches you the commands: systemctl status, journalctl -k, reading memory.max off a cgroup after a Pod dies. That's enough to pass a performance-based exam. It is not enough to explain why systemctl edit writes a drop-in instead of touching the vendor file, why an app with no signal handler can shrug off docker stop for ten full seconds, or why the word "container" describes zero new things the kernel actually knows about. This page assumes you've read the blueprint and can run the commands; what follows is the machinery underneath them — systemd as the process the kernel hands a very specific, non-optional job to; namespaces and cgroups as the only two primitives a "container" is built from, assembled here by hand with no runtime in sight; and journald as a log store whose most useful property isn't that it's structured, but that parts of it can't be forged by the process writing to it. The theme underneath all three: every abstraction this whole course teaches — a Pod, a sidecar, a reconciler, an eBPF datapath — is these three things, wearing a nicer name.

Why this floor doesn't disappear in a Kubernetes-first world

☺ Like you're 10: Every fancy button on the control panel is wired to a plain old switch somewhere below deck. Hiding the switch doesn't remove it — it just means fewer people know where it is when it sticks.

It's tempting to treat LFCS as the one credential on this ladder that's aimed at a job nobody does anymore — who SSHes into a bare node when the whole point of Kubernetes, and further up, of a managed control plane or GKE Autopilot or Fargate, is that nobody has to? The honest answer is that the layer doesn't go away under any of that; it goes further from view, and the number of people who still understand it goes down while the blast radius of not understanding it does not. A managed node pool still runs a kubelet that is still a systemd unit that can still fail to start. A serverless container still gets OOMKilled by the exact same kernel memory controller reading the exact same memory.max file, whether or not anyone on the team has ever seen a cgroup directory. The abstraction changes who is paged first, not what's actually true underneath the page.

systemd as PID 1: the one job every "first process" inherits

☺ Like you're 10: Whoever is born first in a family sometimes ends up responsible for cleaning up after everyone else, whether they signed up for it or not. PID 1 is that sibling — by kernel rule, not by choice.

The kernel starts exactly one process at boot — traditionally /sbin/init, which on essentially every modern distribution is systemd — and that process becomes PID 1. The kernel then treats PID 1 specially in two ways that have nothing to do with what systemd chooses to do and everything to do with what being PID 1 is. First: when a process's parent dies before it does, the kernel reparents the orphan to the nearest surviving ancestor marked to catch it, which by default is PID 1 — and PID 1 is expected to call wait() on it, or it lingers forever as a zombie, a dead process still holding a slot in the process table. Second: the kernel suppresses the default terminate-on-signal behavior for PID 1 specifically — a bare SIGTERM that would kill any ordinary process does nothing to PID 1 unless PID 1 has installed its own handler for it. Neither of these is a systemd feature. They are a rule about the number 1, and systemd is simply the process the kernel handed the number to.

systemd's actual job, on top of that inherited obligation, is to turn a pile of independent services into one coherent boot: a dependency graph where Wants=/Requires= answer "does this unit need to exist at all" and After=/Before= separately answer "in what order, if both are starting" — two different questions a unit file is required to answer independently, and conflating them is the single most common cause of a boot-time race.

# Not just "what failed" — the DEPENDENCY GRAPH that produced the boot
systemd-analyze blame                              # which units took longest to start
systemd-analyze critical-chain kubelet.service      # the slowest CHAIN kubelet actually waited on
systemd-analyze dot kubelet.service containerd.service | dot -Tsvg > deps.svg   # drawn, not guessed

# Wants=/Requires=  -> does this unit need to exist at all      (dependency)
# After=/Before=    -> in what order, IF both are starting      (sequence)
# A unit can Want another with no ordering constraint at all — and that's
# usually a bug, not a feature, unless you deliberately mean "parallel."

The same PID-1 rules apply one level down, inside a container's own PID namespace — and that's where the obligation quietly becomes your problem instead of systemd's. A container image typically ships no init system at all: your application binary is ENTRYPOINT, and it becomes PID 1 of its own, brand-new PID namespace the instant it starts.

# Inside a container with no init process, YOUR app is PID 1 — and
# inherits every obligation systemd normally absorbs on the host.
$ docker run -d --name demo my-app sleep 3600
$ docker exec demo ps aux
PID   USER     COMMAND
1     root     sleep 3600        # this literally is PID 1 in here

# my-app spawns a helper and never wait()s on it -> that helper becomes
# a zombie THIS container never reaps, same failure mode as a broken
# systemd unit, one namespace down. And `docker stop` sends SIGTERM —
# an app with no handler for it, running as PID 1, gets no default
# "just die": it waits out the full grace period until SIGKILL arrives.
$ docker run --init -d my-app     # --init quietly puts tini at PID 1
#   ^ tini's entire job is doing the two things above so your app
#     never has to: reap zombies, forward signals it doesn't handle.

One more parallel worth naming: systemd's Type=notify lets a service call sd_notify(0, "READY=1") over a private socket to tell systemd "I have started, but I am not yet ready" — the same problem, at the same layer, that Kubernetes solves one level up with a readinessProbe. Both exist because "the process started" and "the process can do useful work" are provably different facts, and a supervisor that conflates them routes traffic — or restarts a boot sequence — before the thing it's waiting on can actually answer.

Namespaces and cgroups: the primitives "container" is a nickname for

☺ Like you're 10: There is no "make a container" button anywhere in the kernel. There's a "make this process see a smaller world" button, and a separate "put this process on an allowance" button — and someone else's tool presses both for you, then calls the result a container.

Search the kernel's system-call table and you will not find create_container(). What you'll find are two unrelated mechanisms, each older than the word "container" in its current sense, composed by userspace tooling — runc, underneath containerd, underneath the kubelet — into the thing you experience as one. Namespaces give a process its own, private view of one kind of global kernel resource; a process inside a namespace can't see, and usually can't affect, whatever exists outside it. Cgroups (control groups) do the opposite job: they don't hide anything, they meter and cap it — how much memory, CPU time, I/O bandwidth and even how many PIDs a group of processes is allowed to consume, enforced by the kernel, not requested of it.

These aren't abstract — clone(), unshare() and setns() are ordinary system calls any process can make, with no container runtime involved at all. Building one namespace-and-cgroup box by hand, from a plain shell, is the single most useful thirty seconds you can spend on this material:

# No runc, no containerd, no Docker daemon — just the raw syscalls.
sudo unshare --pid --net --mount --uts --ipc --fork --mount-proc bash

# Inside this new shell:
hostname sandbox            # UTS namespace: changes in here, host untouched
ps aux                      # PID namespace: 'bash' really is PID 1 in here
ip addr                     # NET namespace: loopback only — no route out, by design
mount -t tmpfs tmpfs /mnt   # MOUNT namespace: this mount is invisible outside

# Now put it on a budget — cgroup v2, by hand, no systemd unit involved:
sudo mkdir /sys/fs/cgroup/sandbox
echo $$ > /sys/fs/cgroup/sandbox/cgroup.procs   # move THIS shell into it
echo 20M > /sys/fs/cgroup/sandbox/memory.max     # a hard ceiling, kernel-enforced
cat /sys/fs/cgroup/sandbox/memory.max            # confirm it actually stuck

# runc does exactly this sequence — plus a pivot_root and a capability
# drop — then execve()s your ENTRYPOINT. There is no further magic.

One structural detail worth knowing even though the LFCS domains don't name it directly: cgroup v2 unifies every controller (memory, cpu, io, pids) into one single hierarchy per process, replacing the older cgroup v1 model where a process could sit in a different, independently-managed tree per controller — a design that made consistent accounting genuinely difficult. Delegation down that one tree is explicit: a parent cgroup lists which controllers it hands down to its children in cgroup.subtree_control, which is exactly the file the kubelet's own cgroup driver writes to when it delegates kubepods.slice down to a per-Pod, then per-container, cgroup.

journald: what a binary, trusted-metadata log actually buys you

☺ Like you're 10: If a note is stapled to a package by the delivery driver instead of written by the sender, you can trust it more — the driver saw the package with their own eyes. journald staples some of its own notes on, and refuses to let the sender write those particular ones.

journald doesn't store plain text lines; it stores structured, indexed records, and the interesting part isn't the format, it's where each field came from. Some fields — MESSAGE, PRIORITY — are supplied by the sending process, and journald trusts them exactly as much as you'd trust anything a process claims about itself: not much. Other fields, prefixed with an underscore — _PID, _UID, _SYSTEMD_UNIT, _SYSTEMD_CGROUP, _BOOT_ID — are added by journald itself, read straight from the kernel's own record of the connection (via SO_PEERCRED on the logging socket) rather than accepted from anything the process said. A compromised or simply buggy process can put whatever it wants in MESSAGE. It cannot put whatever it wants in _SYSTEMD_UNIT — that field is the kernel's word, laundered through journald, not the process's.

# Two different kinds of field in the same binary record
journalctl -u containerd -o verbose -n 1
# MESSAGE=...                        <- supplied BY the process (an unverified claim)
# _PID=48213                         <- added BY journald from the KERNEL's own
# _SYSTEMD_UNIT=containerd.service      record of who actually sent this — the
# _SYSTEMD_CGROUP=/system.slice/...     process cannot forge these fields, even
#                                        if MESSAGE itself is a total lie.

# Persistence is not a given — check before you assume
ls -ld /var/log/journal 2>/dev/null || echo "no persistent journal on this host"
# Storage=auto (the default): journald writes to /var/log/journal if that
# directory EXISTS, and silently falls back to /run/log/journal — tmpfs,
# gone at reboot — if it doesn't. Two hosts on the same distro image can
# disagree here for no deeper reason than "one had the directory created
# for it, once, and one never did."
sudo mkdir -p /var/log/journal && sudo systemd-tmpfiles --create --prefix /var/log/journal

The correlation this trust buys is what makes journald a genuinely good fit for a node running containers, not just a legacy syslog replacement: because _SYSTEMD_CGROUP is stamped by journald from a source the process can't touch, every line a unit ever writes can be reliably grouped by the exact unit and cgroup that emitted it — including the previous boot's dying words, via journalctl -b -1, which is frequently the only trace left of a node that crashed rather than shut down cleanly.

Where the floor keeps receding — and why someone still has to stand on it

☺ Like you're 10: Moving further from the basement doesn't unplug the pipes. It just means fewer people remember they're there, until one leaks.

Everything this course teaches above this page is these same three mechanisms, restated at a different altitude. eBPF and the Cilium datapath is a program attached to a kernel hook, running faster than netfilter precisely because it skips layers of the same packet-filtering machinery this page's namespaces sit inside. The OpenTelemetry data model and every Prometheus node-exporter metric ultimately come from an agent reading /proc and cgroup controller files — the same files this page's unshare demo wrote to by hand. A GitOps reconciler is, on the node that runs it, a systemd unit with the exact PID-1-adjacent obligations described above, whether or not the team operating it has ever thought about it that way. None of that is a coincidence — it's the same small set of kernel primitives, composed a different number of times.

Which is the honest answer to the opening question. A fully managed control plane, a serverless platform, an autoscaling node pool — none of them delete namespaces, cgroups, systemd or journald. They move the pager for all four onto someone else's team, for as long as that team's abstraction holds. The day it doesn't — a node that won't drain, a sidecar that can't reach localhost, a crash with no journal to read — the floor is exactly where this page left it, and the platform engineer who's spent thirty minutes with unshare and a cgroup file finds it a great deal faster than the one who hasn't. For the exam-shaped version of everything above, the LFCS blueprint is the place to drill it under time pressure; for the two tools this page leaned on hardest, systemd & journald and LVM & Linux Storage Tools go further at reference depth; and the broken cgroup-limit drill is the hands-on version of the OOMKilled trace the blueprint walks through.