systemd & journald
The LFCS blueprint teaches enough systemctl and journalctl to pass a performance-based exam; Linux Fundamentals for Platform Engineers explains the kernel machinery — PID 1's inherited obligations, cgroup v2, journald's trusted metadata — underneath both tools. This page sits between them, at reference depth: every unit type you'll actually meet, the exact anatomy of a unit file, the two genuinely different questions Wants=/Requires= and After=/Before= answer, how restart policy turns a crash into either a quiet recovery or a loop that burns your StartLimitBurst, why systemctl edit is the only override that survives a package upgrade, the cgroup-backed resource directives you'll write directly into a unit file, and the journalctl filters that turn a vague "something's wrong on that node" into an exact line, in under a minute. Nothing here is exam trivia for its own sake — every directive below is one you'll actually type against a real node, usually at 2am.
Think of systemd as the head counselor at a summer camp. Every activity — swimming, the campfire, the radio hut — is a "unit," and the counselor's clipboard says, for each one: who has to already be set up before it starts (dependency), whether it should relight itself if the campfire goes out on its own or only if a camper puts it out on purpose (restart policy), and exactly how much firewood or radio battery it's allowed to burn through before someone steps in (a resource limit). If a counselor-in-training wants one activity to run differently, they don't scribble on the official clipboard — they pin a sticky note on top of it, so next summer's official rulebook update doesn't erase what they changed. And every single thing that happens at camp — who started what, when, and why it stopped — gets written into one shared logbook that campers themselves aren't allowed to edit, only the counselors can, which is exactly why you trust it when something goes wrong.
Units: the six kinds you'll actually meet
☺ Like you're 10: Not every activity at camp is "run this program" — some are "listen for a knock," some are "only on Tuesdays," and some are just a labeled folder that groups other activities together.
Everything systemd manages is a unit, and the file's extension tells you which of roughly a dozen unit types it is. In practice, five or six cover almost everything you'll touch:
| Unit | What it represents | Where you'll meet it |
|---|---|---|
.service | A managed process — start it, supervise it, restart it, budget it | containerd.service, sshd.service, kubelet.service — nearly everything below this table |
.socket | A listening socket, opened by systemd itself, before any service exists | Socket activation: the service that owns the socket only starts on the first real connection |
.timer | A scheduled trigger, paired with a same-named .service | The systemd-native replacement for cron — logs to the journal like everything else, for free |
.target | A named synchronization point — a label with no process of its own | multi-user.target, graphical.target — the things units point WantedBy= at |
.mount / .automount | A filesystem mount, generated automatically from /etc/fstab or written by hand | Making a mount a dependency another unit's After= can wait on |
.path | Watches a path with inotify and starts a paired unit when it changes | "Run this the moment a file appears" — cheaper than a polling timer |
A .service file has three sections, always in the same order, and every directive below belongs to exactly one of them:
# /etc/systemd/system/mission-agent.service [Unit] Description=Mission Control telemetry agent Documentation=https://internal.example/runbooks/mission-agent After=network-online.target containerd.service Wants=network-online.target BindsTo=containerd.service # dies WITH containerd, not just after it [Service] Type=notify # see the Type= table below ExecStartPre=/usr/local/bin/mission-agent-preflight ExecStart=/usr/local/bin/mission-agent --config /etc/mission-agent/config.yaml ExecReload=/bin/kill -HUP $MAINPID Restart=on-failure RestartSec=5s User=mission-agent Group=mission-agent WorkingDirectory=/var/lib/mission-agent EnvironmentFile=-/etc/default/mission-agent # leading '-' = fine if the file is missing [Install] WantedBy=multi-user.target
Type= is the directive people get wrong most, because it changes what "started" even means to systemd. simple (the default) considers the unit started the instant ExecStart's process exists — fine for a process that's ready almost immediately, wrong for one with real startup work, because anything ordered After= it may race a service that isn't actually ready yet. notify waits for the process to call sd_notify(3) with READY=1 over a private socket — the correct choice for anything with meaningful startup work, and the same "started ≠ ready" problem Kubernetes solves one layer up with a readinessProbe. forking is for old-school daemons that fork into the background and expect systemd to track the child, not the process it launched directly. oneshot is for a unit that's meant to run to completion and exit — the shape every .timer's paired service takes, and it accepts RemainAfterExit=yes to still report "active" after the process exits.
Dependency ordering: two different questions, asked separately
☺ Like you're 10: "Does this activity need swimming to have happened at all" and "does it need to happen before or after swimming" sound like the same question. They aren't, and mixing them up is how you get a race.
Wants= / Requires= / BindsTo= answer existence — does this unit need that unit to exist and be running, at all, ever? After= / Before= answer a completely separate question — ordering — if both units are starting during this transaction, which one goes first? Neither implies the other. Write Wants=containerd.service alone, with no After=, and systemd starts both units in parallel — containerd gets pulled in, but nothing guarantees your unit doesn't reach ExecStart first and fail against a socket that isn't listening yet. This is the single most common cause of a flaky boot-time race, and the fix is always the same: name the dependency in both directions if you actually need it in both.
| Directive | Axis | Behavior |
|---|---|---|
Wants= | Existence | Pull the target in if not already active; a failure of the target does not stop this unit |
Requires= | Existence | Same pull, but if the target fails or stops, this unit is stopped too — a hard dependency |
BindsTo= | Existence | Requires=, tightened further: this unit stops the instant the target stops, even a clean stop, not just a failure |
After= | Ordering only | If both units are starting in this transaction, start the target first — pulls in nothing by itself |
Before= | Ordering only | The mirror of After=, written from the other unit's side |
PartOf= | Existence, one-directional | Stopping or restarting the target propagates to this unit; the reverse does not |
If A.service requires B.service which requires A.service, systemd detects the cycle at transaction time and breaks it by deleting one of the ordering edges — the boot continues, but one of your two units starts in an order you didn't intend, and nothing loud tells you that happened. systemd-analyze verify unit-name.service catches malformed and circular dependencies before you ever run daemon-reload, and it costs nothing to run on every unit file before it ships.
Restart policy: telling systemd what a crash actually means
☺ Like you're 10: "Relight the campfire if it blows out" sounds simple until you ask: relight it if a camper stamps it out on purpose too? How many times in a row before you stop and ask an adult?
Restart= decides which exits trigger a restart at all, and the option people reach for by habit — always — is usually the wrong one, because it restarts even a deliberate, clean systemctl stop. on-failure is the sane default for almost everything: restart on a non-zero exit code, an uncaught signal, a timeout, or a watchdog miss — but a clean exit or an explicit stop is left alone.
| Value | Restarts on… |
|---|---|
no | Never (the default if Restart= is absent entirely) |
on-success | Only a clean exit — unusual, but real for a unit meant to keep re-running a job |
on-failure | Non-zero exit, uncaught signal, timeout, or watchdog miss — not a clean stop |
on-abnormal | An uncaught signal, timeout, or watchdog miss only — a clean exit code is not "abnormal" |
on-watchdog | Only a missed WatchdogSec= heartbeat |
on-abort | Only an uncaught signal |
always | Every exit, including a clean, deliberate systemctl stop — rarely what you actually want |
Restart policy alone can burn a node in a tight loop, so two more directives exist specifically to cap that: RestartSec= is the pause before each restart attempt, and StartLimitIntervalSec= / StartLimitBurst= together define a crash-loop breaker — allow at most Burst starts within IntervalSec, and past that, systemd stops trying and marks the unit failed rather than hammering a dependency that's already down. Hitting the limit is not itself a bug; it's the safety valve doing its job, and the fix is systemctl reset-failed unit-name after the real problem underneath is actually resolved — not raising the burst count until the loop stops complaining.
[Service] Restart=on-failure RestartSec=5s StartLimitIntervalSec=60s StartLimitBurst=4 # 4 restarts inside any 60s window, then give up and go 'failed'
Drop-ins: the only override that survives an upgrade
☺ Like you're 10: Don't scribble on the official rulebook — pin a sticky note on top of it, so the next official reprint doesn't erase what you changed.
Editing a vendor-shipped unit under /usr/lib/systemd/system/ directly works — until the package updates and silently overwrites it, taking your change with it. The correct mechanism is a drop-in: a small override file under /etc/systemd/system/<unit>.d/, layered on top of the vendor unit at parse time rather than replacing it. systemctl edit unit-name.service creates and opens exactly that file — an empty override.conf under a freshly-made .d directory — and runs daemon-reload for you the moment you save and exit; systemctl edit --full unit-name.service instead opens a complete, editable copy of the whole unit for the rare case a drop-in genuinely can't express what you need.
# systemctl edit mission-agent.service creates: # /etc/systemd/system/mission-agent.service.d/override.conf [Service] # Most directives just get ADDED to the vendor unit's list — # but list-type directives like ExecStart= and Environment= don't merge, # they APPEND. To truly replace one, reset it first with a bare '=': ExecStart= ExecStart=/usr/local/bin/mission-agent --config /etc/mission-agent/config.yaml --debug Environment=MISSION_AGENT_LOG_LEVEL=debug
That empty-assignment reset trips up almost everyone once: appending a second ExecStart= without first clearing the original doesn't override it, it queues a second command to run — which for most services is a hard failure, since only one ExecStart= is normally allowed to run at a time. systemctl cat unit-name.service shows the vendor unit and every applied drop-in, merged in the order systemd actually applies them — always run it after an edit to confirm the file did what you expected, before trusting systemctl status alone.
cgroup-backed resource limits, written in systemd's own dialect
☺ Like you're 10: Every cabin gets a firewood budget and a battery budget, set on the counselor's clipboard — and the counselor writes it into a form the actual supply shed understands, not a form of their own invention.
The [Service] resource directives aren't systemd's own invention — they're a friendlier spelling of cgroup v2 controller files, the same kernel primitives the deep-dive on namespaces and cgroups builds by hand from a bare shell, and the exact mechanism the kubelet writes on your behalf every time a Pod declares resources.limits. Setting them on a unit is the systemd-native equivalent of a Kubernetes resource limit, one process tree at a time.
[Service]
# Memory — cgroup v2's memory controller
MemoryHigh=384M # soft ceiling: throttled hard above this, but NOT killed
MemoryMax=512M # hard ceiling: the kernel OOM-kills this cgroup above it
MemorySwapMax=0 # forbid swap entirely for this unit — fail fast, don't thrash
# CPU — cgroup v2's cpu controller
CPUWeight=200 # relative share under contention, 1-10000, default 100 ('nice', not a hard cap)
CPUQuota=150% # hard cap: 1.5 cores' worth of wall-clock CPU time, enforced even when idle elsewhere
# I/O and process count
IOWeight=300 # relative disk I/O share under contention
IOReadBandwidthMax=/dev/sda 50M # a hard ceiling, per device
TasksMax=512 # cap on threads+processes this unit can fork — a fork-bomb backstopTwo behaviors are worth internalizing precisely because they read as similar and aren't. MemoryHigh= is a soft throttle — cross it and the kernel leans hard on reclaim and slows the cgroup down, but nothing is killed. MemoryMax= is a hard ceiling — cross it and the kernel's OOM killer terminates a process inside that cgroup, full stop, the exact mechanism the LFCS blueprint traces end-to-end from a Pod's resources.limits down to the oom_kill counter. Setting only MemoryMax= with no headroom below it in MemoryHigh= means the first sign of memory pressure is a kill, not a warning — give a service room to be throttled before it's room to be killed.
# confirm what actually got applied — read it back, don't just trust the unit file systemctl show mission-agent.service -p MemoryMax -p MemoryHigh -p CPUQuotaPerSecUSec systemd-cgls # the cgroup tree, as systemd sees it, unit by unit systemd-cgtop # top(1), but per cgroup — live CPU/memory/IO/tasks cat /sys/fs/cgroup/system.slice/mission-agent.service/memory.max # the raw file underneath the directive
Every managed service on a systemd host is already living inside a cgroup, whether or not anyone ever writes a resource directive for it — services are placed under system.slice by default, users under user.slice, and a container engine's own workloads under kubepods.slice, side by side. Writing MemoryMax= doesn't create the cgroup; it just puts a number in a file the kernel was already reading. That's why the fix-a-broken-cgroup-limit drill feels like a systemd exercise and a Kubernetes exercise at the same time — underneath, it's the same file either way.
systemctl: the day-to-day command set
☺ Like you're 10: One tool asks the head counselor about an activity, changes whether it runs, or asks it to reread the whole clipboard.
# inspecting a unit systemctl status mission-agent.service # human summary + last log lines systemctl cat mission-agent.service # the FULL merged unit — vendor file + every drop-in systemctl show mission-agent.service # every resolved property, machine-readable systemctl list-dependencies mission-agent.service # the dependency tree, as systemd resolved it # runtime state — doesn't touch boot-time behavior systemctl start|stop|restart|reload mission-agent.service systemctl is-active mission-agent.service systemctl is-failed mission-agent.service systemctl reset-failed mission-agent.service # clear a StartLimitBurst trip, after fixing the real cause # boot-time behavior — doesn't touch whether it's running right now systemctl enable mission-agent.service # symlinks it into its [Install] WantedBy= target systemctl disable mission-agent.service # removes that symlink only systemctl mask mission-agent.service # symlinks the unit to /dev/null — CANNOT be started, even by hand systemctl unmask mission-agent.service # after ANY unit-file edit made outside 'systemctl edit' systemctl daemon-reload # re-reads unit files from disk; does NOT restart anything running systemd-analyze verify mission-agent.service # catch syntax errors and cycles before you rely on it
Two pairs are worth memorizing precisely because they're independent: start/stop only affects the unit right now, this boot; enable/disable only affects whether it starts on the next boot, by wiring — or unwiring — the symlink its [Install] section names. Running one without the other is a normal, deliberate combination, not a mistake: enable --now does both at once when you actually want that. disable and mask are not the same guardrail — disable just removes the boot-time symlink, but the unit can still be started by hand or pulled in as someone else's dependency; mask makes that structurally impossible until it's explicitly unmasked, which is the right tool when you need a unit to genuinely never run, not just to stay off by default.
"I disabled a unit that kept misbehaving, restarted the box, and it came back anyway. Took me longer than I want to admit to remember that disable only removes the enablement symlink — it doesn't stop something else from Requires=-ing it back in as a dependency at boot. mask was the tool I actually needed, and the two minutes I spent reading man systemctl that morning saved me from ever making that mistake twice."
Template units are the other habit worth building early: a unit file named with an @, like getty@.service, is a template — %i inside it is substituted with whatever follows the @ at instantiation, so systemctl start getty@ttyS0.service starts one instance parameterized with ttyS0, and a fleet-management pattern like mission-agent@.service lets one unit file back an arbitrary number of independently start/stop-able instances — one per %i — without duplicating a single line of unit definition.
journalctl: filtering for incident response
☺ Like you're 10: The shared logbook has everything in it — the trick is asking it narrower and narrower questions until only the one line that matters is left on the page.
journald is the sink every unit's stdout/stderr and every structured log call lands in — no separate log file per service to hunt down. The skill worth building isn't knowing every flag, it's narrowing a search the same way every time: unit, then time window, then priority, then a text match, in that order, because each step throws away everything that can't possibly be the answer before the next step even runs.
# narrowing, in order — unit, window, priority, pattern journalctl -u mission-agent.service # everything this unit ever logged (this journal) journalctl -u mission-agent.service --since "10 min ago" # add a time window journalctl -u mission-agent.service --since today -p err # add a priority floor: emerg..err (0-3) journalctl -u mission-agent.service -g "connection refused" --since "1 hour ago" # add a regex match # boot-scoped views — essential after any crash or unexpected reboot journalctl -b # THIS boot only (the default view's actual scope) journalctl -b -1 # the PREVIOUS boot — did it crash, or shut down cleanly? journalctl --list-boots # every boot journald still has a record of, oldest first # live and structured journalctl -u mission-agent.service -f # follow live, like tail -f journalctl -u mission-agent.service -o json-pretty -n 1 # one entry, every field, human-readable journalctl -u mission-agent.service -o verbose -n 1 # same fields, systemd's own compact format journalctl -k # kernel ring buffer only — dmesg, OOM kills, driver errors journalctl _SYSTEMD_UNIT=mission-agent.service _PID=48213 # exact-match a KERNEL-verified field, not text # operational hygiene — journald is a classic /var space eater journalctl --disk-usage journalctl --vacuum-time=14d # drop entries older than 14 days journalctl --vacuum-size=500M # cap total journal size, oldest entries dropped first
The _-prefixed fields in that -o verbose output — _SYSTEMD_UNIT, _SYSTEMD_CGROUP, _PID, _BOOT_ID — are stamped on by journald itself, read from the kernel's own record of who's actually connected to the logging socket, not accepted from anything the process claimed. That's precisely why journalctl _SYSTEMD_UNIT=x is trustworthy in a way grepping for a string that looks like a unit name inside MESSAGE never fully is — the deep-dive covers exactly why that trust boundary exists. And persistence isn't automatic: journald's default Storage=auto only writes to disk under /var/log/journal if that directory already exists; otherwise every entry lives in volatile /run/log/journal and is gone the moment the box reboots — worth confirming with ls -ld /var/log/journal on any node before you're depending on journalctl -b -1 during an actual incident.
Gotchas and failure modes
☺ Like you're 10: Most surprises come from a command that quietly ran through the wrong shell, an edit that never got told to take effect, or forgetting that a saved log line is a choice someone made, not a guarantee.
ExecStart= is not a shell — pipes and globs silently don't work
Unless you wrap the command explicitly, ExecStart= runs the binary directly with execve(), with no shell in between — no |, no * glob expansion, no $VAR substitution from the environment the way a shell script would do it, no && chaining. A command that works perfectly when pasted into a terminal can fail silently or behave completely differently as a raw ExecStart= line, precisely because a terminal always runs it through a shell first. If you genuinely need shell features, be explicit about it: ExecStart=/bin/bash -c '/usr/local/bin/agent | logger -t agent'.
ExecStartPre failing means the unit never starts at all
By default a non-zero exit from any ExecStartPre= command aborts the start entirely — the main ExecStart= never even runs, and the failure reads, at first glance, exactly like ExecStart itself failed. systemctl status does show which Exec step actually failed, but only if you read past the top line; a prefix of - on the command (ExecStartPre=-/usr/local/bin/optional-step) makes a step's failure non-fatal when that's genuinely the intent.
Editing the unit file directly means it needs daemon-reload — a drop-in via systemctl edit does not
Any change to a unit file made with a plain text editor is invisible to a running systemd until something tells it to re-read unit files from disk — systemctl daemon-reload. Skip it, and systemctl status keeps showing the old configuration with no warning at all that a newer file exists on disk. systemctl edit is worth using specifically because it runs that reload for you automatically the moment you save and exit — one less step to forget under pressure.
Socket activation means "enabled" and "already listening" can both be true before the service exists
A .socket unit that's Requires=d by connection, not by boot, means the port can be open and accepting connections before its paired .service has ever started — the first real connection is what triggers the start. systemctl status foo.socket and systemctl status foo.service can legitimately disagree about "running" at the same moment, and that's the design working correctly, not a fault.
Foxy: mission-agent keeps flapping — restart, crash, restart, crash — and after four tries it just gives up and sits there marked failed. Did I break the restart policy?
Sol the Sloth: Slow down before you touch StartLimitBurst. That four-and-stop is StartLimitIntervalSec=/StartLimitBurst= doing exactly what it's for — it's not the bug, it's the alarm going off. Something underneath it is actually crashing.
Gizmo: Easy fix — set StartLimitBurst=999999 and Restart=always. Now it NEVER stops trying! Problem gone, forever! 🎉
Timmy the Turtle: Gizmo, that turns a contained failure into a process that spins the CPU in a permanent crash loop on that node, forever, instead of failing loud and visible in under a minute. That's not a fix, that's hiding the alarm bell.
Sol the Sloth: journalctl -u mission-agent.service -p err --since "10 min ago" first. Read the actual reason it's exiting before anyone touches the policy that's just reporting it.
Foxy: ...it's a config file it can't find. Someone's drop-in overrode ExecStart= and pointed --config at a path that doesn't exist on this box.
Sol the Sloth: There it is. Fix the path, systemctl reset-failed, systemctl restart. The four-strike limit did its job — it stopped the box from burning CPU on a crash loop while we actually looked.
1. What's the difference between what Wants=/Requires= answer and what After=/Before= answer, and why does a unit with only Wants= risk a startup race? 2. Restart=always restarts even a deliberate systemctl stop — which value doesn't? 3. Why does systemctl edit write to a .d/override.conf drop-in instead of the vendor unit file directly, and why does resetting a list-type directive like ExecStart= require an empty assignment first? 4. What's the practical difference between MemoryHigh= and MemoryMax=? 5. Put these journalctl filters in the order you'd actually apply them during an incident: priority, unit, pattern match, time window. 6. What's the difference between systemctl disable and systemctl mask? 7. Why can journalctl -b -1 come back completely empty on a node that just hard-crashed, even though nothing about that crash was unusual?
Check your answers
Wants=/Requires=answer whether the target must exist and be running at all (existence);After=/Before=answer, only when both units are starting in the same transaction, which one goes first (ordering). They're independent — naming onlyWants=pulls the dependency in but gives no guarantee it starts before this unit does, which is exactly the race.on-failure(alsoon-abnormal,on-watchdog,on-abort,on-success, andno) all leave a clean, deliberate stop alone; onlyalwaysrestarts through a deliberate stop too.- A drop-in layers on top of the vendor unit without ever touching the file itself, so a package upgrade that overwrites the vendor file doesn't erase the change. List-type directives like
ExecStart=andEnvironment=append to the vendor unit's value rather than replacing it, so a bareExecStart=with nothing after the=clears the inherited value first — without that, you end up with twoExecStart=lines queued instead of one replaced. MemoryHigh=is a soft ceiling — cross it and the kernel throttles the cgroup hard via reclaim, but nothing is killed.MemoryMax=is a hard ceiling — cross it and the kernel's OOM killer terminates a process in that cgroup outright.- Unit, then time window, then priority, then pattern match — each step narrows what the next step has to search, which is the whole point of doing it in that order rather than grepping the entire journal for a text pattern first.
disableonly removes the boot-time enablement symlink — the unit can still be started by hand or pulled in as someone else's dependency.masksymlinks the unit to/dev/null, making it impossible to start at all, by hand or as a dependency, until it's explicitlyunmasked.- journald's default
Storage=autoonly persists to disk if/var/log/journalalready exists; if it doesn't, every entry lives in volatile/run/log/journaland is destroyed by the very reboot the crash caused — nothing about the crash itself was unusual, the logging configuration just never survived it.