Hands-On Labs · Guided Drills

Drill — Fix a Broken cgroup Limit

A small service goes into a restart loop the moment real traffic hits it. systemctl status shows it flapping; nothing in the application logs looks wrong; the code is doing exactly what it was written to do. Somewhere between "the process wanted more memory" and "the process is dead" sits a single number, in a single file, that nobody has looked at since the day the unit was written. This drill hands you that exact situation — self-contained, no cluster, no capstone dependency, just one Linux host with systemd and a stopwatch. You'll reproduce a real OOM kill, trace it to its cgroup's own accounting files instead of guessing, measure the service's actual memory need instead of picking a number that "feels safe," and write a fix that survives repeated bursts, not just one lucky green run. Budget 25-40 minutes before you open the walkthrough below.

☺ Explain it like I'm 10

Picture a school backpack with a zipper that physically will not close past a certain bulge. Whoever bought that backpack sized it years ago, back when you only carried a pencil case and a sandwich — and most days that's still all it holds, so the zipper never even notices. But once a week is gym-kit day: shoes, shorts, a water bottle, on top of the usual stuff, way more than that zipper was ever sized for. It doesn't gently squeeze things down for you. It refuses outright — and here, "refuses" means the whole backpack gets confiscated on the spot and a brand-new one is handed to you, every single gym day, forever, until somebody actually measures what gym day needs and buys a backpack built for it.

🦥Your host for this drill: Sol the Sloth — the one member of Mission Control who refuses to move to the fix until the evidence has actually been read, which is exactly the discipline a cgroup limit rewards and a rushed guess punishes.

⚠ Before you start

You need a real Linux host or Linux VM with systemd (v245+) and cgroup v2 mounted as the unified hierarchy — true of a default Ubuntu 22.04+, Debian 12+, or Fedora box; confirm with stat -fc %T /sys/fs/cgroup/, which should print cgroup2fs. On macOS or Windows, a throwaway cloud VM or a local Linux VM (Multipass, UTM, Lima) is the fastest path — a container is not enough here, because you need root access to a real, writable /sys/fs/cgroup tree. You also need python3 and sudo. Everything you create lives under /opt/telemetry-relay and /etc/systemd/system; tear it down when you're done (commands at the end). systemctl's output format has changed slightly across versions — if a line below doesn't match yours exactly, trust your own terminal over this page.

What memory.max and MemoryMax= actually are

☺ Like you're 10: Two different names, written by two different tools, pointing at exactly the same one file.

Every service systemd starts gets its own cgroup automatically, whether or not the unit file ever mentions one — placed under system.slice by default, right next to kubepods.slice on a node that also runs a kubelet. Writing MemoryMax=64M into a [Service] block doesn't create that cgroup; it writes the number 67108864 into a kernel file the cgroup already had: memory.max, cgroup v2's hard ceiling for that process tree's memory. Cross it, and the kernel doesn't warn — it tries to reclaim, finds nothing reclaimable in freshly-allocated memory, and invokes its OOM killer scoped to just that cgroup. No Kubernetes component and no systemd component makes that call. The kernel does, synchronously, the instant the allocation would breach the limit — and systemd's only job afterward is to notice the process is gone and report why.

◆ Key idea

MemoryMax= is systemd's friendlier spelling of memory.max — the exact mechanism the LFCS blueprint traces for a Kubernetes Pod's resources.limits.memory, and the exact mechanism this drill traces for a plain systemd service. Same kernel file, same kill behavior, whether the number arrived via a unit file or a kubelet's cgroup driver.

Set up the scratch service

☺ Like you're 10: A tiny program that mostly sips memory, then once in a while gulps a lot of it at once — on purpose.

telemetry-relay batches incoming samples in memory and flushes them periodically. Most cycles it's trivial. Every third cycle, a real burst lands — the entire reason the service exists — and that burst is where today's bug lives:

sudo mkdir -p /opt/telemetry-relay

# /opt/telemetry-relay/relay.py
import time

SAMPLE_KB = 200

def flush(batch, label):
    kb = len(batch) * SAMPLE_KB
    print(f"flush ({label}): {len(batch)} samples, {kb} KB", flush=True)

def quiet_batch():
    return [bytearray(1024 * SAMPLE_KB) for _ in range(20)]    # ~4 MB - routine

def burst_batch():
    return [bytearray(1024 * SAMPLE_KB) for _ in range(750)]   # ~150 MB - a real burst

def main():
    cycle = 0
    while True:
        cycle += 1
        flush(quiet_batch(), "quiet")
        time.sleep(3)
        if cycle % 3 == 0:
            flush(burst_batch(), "burst")
        time.sleep(3)

if __name__ == "__main__":
    main()

Copy that in as root (sudo tee /opt/telemetry-relay/relay.py or your editor of choice), then wire the unit — the one that shipped with the bug already in it:

# /etc/systemd/system/telemetry-relay.service — BROKEN, as shipped
[Unit]
Description=Telemetry batch relay
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 /opt/telemetry-relay/relay.py
Restart=on-failure
RestartSec=2s
MemoryMax=64M

[Install]
WantedBy=multi-user.target

MemoryMax=64M was copied from heartbeat-pinger.service — a tiny neighbor that never carries more than a few kilobytes — back when telemetry-relay was first scaffolded and nobody had written the burst path yet. Nobody came back to revisit the number once the real job grew past it. Enable and start it:

sudo systemctl daemon-reload
sudo systemctl enable --now telemetry-relay.service
systemctl status telemetry-relay.service --no-pager

Watch it die

☺ Like you're 10: Two quiet flushes go fine. The third one doesn't even get to print.

Follow it live and let at least one full burst cycle run — about 18 seconds:

journalctl -u telemetry-relay.service -f

telemetry-relay[51204]: flush (quiet): 20 samples, 4000 KB
telemetry-relay[51204]: flush (quiet): 20 samples, 4000 KB
systemd[1]: telemetry-relay.service: A process of this unit has been killed by the OOM killer.
systemd[1]: telemetry-relay.service: Main process exited, code=killed, status=9/KILL
systemd[1]: telemetry-relay.service: Failed with result 'oom-kill'.
systemd[1]: telemetry-relay.service: Scheduled restart job, restart counter is at 3.
systemd[1]: Started telemetry-relay.service - Telemetry batch relay.

Two clean "quiet" flushes, then nothing — no "flush (burst)" line ever prints, because the process is dead before it reaches that print() call. Restart=on-failure brings it straight back, it replays the same two quiet cycles, and the third cycle kills it again. Left alone, this runs forever: a real service, doing real work, permanently unable to finish the one job it exists to do.

Diagnose it from the cgroup's own evidence, not a guess

☺ Like you're 10: Don't reread the Python file hunting for a bug that isn't there. Ask the kernel what it actually did, and ask the cgroup's own files to back that up.

systemctl status already tells you the shape of it — status 'oom-kill', not a Python traceback, not a nonzero exit code from the script itself:

systemctl status telemetry-relay.service --no-pager -l

● telemetry-relay.service - Telemetry batch relay
     Loaded: loaded (/etc/systemd/system/telemetry-relay.service; enabled)
     Active: active (running) since ...; 2s ago
   Main PID: 51301 (python3)
     Status: "Running"
     Memory: 3.9M (max: 64.0M available: 60.1M)
        CPU: 41ms
     CGroup: /system.slice/telemetry-relay.service
             └─51301 /usr/bin/python3 /opt/telemetry-relay/relay.py

Now confirm it's the kernel, not systemd, that pulled the trigger — journalctl -k is the kernel's own ring buffer, and systemd only ever reports what already happened to a process it didn't kill itself:

journalctl -k --since "-2 min" | grep -i -A2 "oom-killer\|out of memory"

kernel: python3 invoked oom-killer: gfp_mask=0x1100dca (GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
kernel: memory: usage 65536kB, limit 65536kB, failcnt 0
kernel: memory: cgroup: /system.slice/telemetry-relay.service
kernel: Memory cgroup out of memory: Killed process 51301 (python3) total-vm:302144kB, anon-rss:64012kB, file-rss:512kB, shmem-rss:0kB, UID:0 pgtables:340kB oom_score_adj:0

That third line is the whole diagnosis in one place: cgroup: /system.slice/telemetry-relay.service, usage 65536kB, limit 65536kB — the process's own cgroup, hitting its own declared ceiling exactly. Confirm it straight from the source file systemd is only reporting on, not systemd's summary of it:

cat /sys/fs/cgroup/system.slice/telemetry-relay.service/memory.max
# 67108864                      <- exactly 64M in bytes, matches MemoryMax= in the unit

cat /sys/fs/cgroup/system.slice/telemetry-relay.service/memory.events
# low 0
# high 0
# max 214
# oom 3
# oom_kill 3
# oom_group_kill 0

oom_kill is a running counter, not a flag — it climbs by exactly one every time this cgroup gets killed, and it will keep climbing for as long as the limit stays wrong. That counter is your proof, later, that a fix actually held rather than just not having failed yet.

Size the fix from evidence, not a guess

☺ Like you're 10: Before you pick the new backpack size, actually measure gym day — don't just grab the biggest one on the shelf.

The tempting fix is to delete MemoryMax= or crank it to something absurd like 8G and move on. Resist that — an unlimited service is a service with no guardrail at all, and the next real leak on this host won't be caught by anything until it takes the whole machine down with it. Measure the actual peak first. Temporarily loosen the ceiling so the process survives long enough to be measured, then poll its live usage through a full burst:

sudo systemctl set-property telemetry-relay.service MemoryMax=1G --runtime
sudo systemctl reset-failed telemetry-relay.service
sudo systemctl restart telemetry-relay.service

for i in $(seq 1 20); do
  systemctl show telemetry-relay.service -p MemoryCurrent
  sleep 1
done

MemoryCurrent=4116480
MemoryCurrent=4116480
MemoryCurrent=163987456   <- the burst cycle, right here
MemoryCurrent=4116480
...

Real peak: about 156 MB, once per burst, dropping straight back down once flush() clears the batch. Not a guess — a number read off the process's own cgroup while it was actually doing its worst-case work. Now write a limit that respects that number instead of ignoring it: a MemoryHigh= soft ceiling with real headroom above the measured peak, so future growth throttles instead of dying outright, and a MemoryMax= hard ceiling well clear of both:

sudo systemctl edit telemetry-relay.service

Paste this into the editor systemd opens (it writes a drop-in, not the vendor file — the only override that survives the next time someone reinstalls or edits the original unit):

# /etc/systemd/system/telemetry-relay.service.d/override.conf
[Service]
MemoryHigh=192M
MemoryMax=256M

Then clear the runtime-only measurement override, reload, and restart clean:

sudo systemctl daemon-reload
sudo systemctl reset-failed telemetry-relay.service
sudo systemctl restart telemetry-relay.service
systemctl cat telemetry-relay.service          # confirm the drop-in is layered on top of the vendor file

Prove it holds

☺ Like you're 10: One quiet cycle proves nothing. Three full bursts in a row, with the counter frozen, is the actual bar.

Watch it live across at least three burst cycles — about a minute — and check the counter, not just your own impression that "it seems fine now":

journalctl -u telemetry-relay.service -f
# ...let it run past three "flush (burst)" lines, then Ctrl-C

cat /sys/fs/cgroup/system.slice/telemetry-relay.service/memory.events | grep oom_kill
# oom_kill 3        <- unchanged from before the fix — no NEW kills, not "reset to zero"

The counter staying at exactly the value it had before the fix — not climbing, and not suspiciously reset — is what actually proves this: memory.events only ever counts up, for the life of that cgroup instance. Three real bursts landing clean, each one printing its own flush (burst) line where a kill used to happen instead, is the whole drill.

◆ Key idea

MemoryHigh= and MemoryMax= answer different questions and the fix needs both. MemoryHigh= is a soft ceiling — cross it and the kernel leans hard on reclaim and slows the cgroup down, but nothing dies. MemoryMax= is the hard ceiling — cross that and the OOM killer fires, full stop. Setting only MemoryMax= with nothing below it means the first sign of memory pressure a service ever gets is a kill, not a warning; the gap between the two numbers is deliberately the room to notice a problem before it becomes an outage.

🦥 Sol's-eye view

"Everyone wants to jump straight from 'it's crashing' to 'raise the number.' I won't. Read memory.max before you touch it — is it even the limit that's wrong, or is something upstream actually leaking? Read memory.events — has this happened three times today or three hundred? Measure the real peak before you write a new number, because a guess that happens to work today is a guess that will eventually stop working, at the worst possible hour, and nobody will remember why the ceiling is whatever it currently is."

🦥 Sol's challenge · going further

The minimum fix above is enough to pass the drill. Two ways to push further: read /sys/fs/cgroup/system.slice/telemetry-relay.service/memory.pressure during a burst — cgroup v2's PSI (pressure stall information) file, showing what fraction of recent time this cgroup actually spent stalled waiting on memory, which is a genuinely better early-warning signal than watching MemoryCurrent creep toward a ceiling. And add StartLimitIntervalSec=60 / StartLimitBurst=4 to the drop-in, then deliberately re-break MemoryMax= back to 64M — watch systemd give up restarting after four kills in the window instead of crash-looping forever, and notice that a restart limit isn't a bug, it's the alarm that stops a bad limit from burning CPU on a doomed retry loop all night.

0 / 8 steps complete

1Create the script and the broken unit, then enable and start it

Done when: systemctl status telemetry-relay.service shows Active: active (running) with a real Main PID.

2Watch it die: reproduce at least one OOM kill and restart

Done when: journalctl -u telemetry-relay.service shows Failed with result 'oom-kill' at least once.

3Confirm the kernel made the call, from journalctl -k

Done when: you can point to the exact Memory cgroup out of memory: Killed process line and read the cgroup path off it.

4Read memory.max and memory.events directly off the cgroup

Done when: you can state the exact byte value of memory.max and the current oom_kill count from the files themselves, not from systemd's summary.

5Measure the real peak by loosening the limit and polling MemoryCurrent

Done when: you have an actual measured peak in MB, not a guessed-at number.

6Write a right-sized MemoryHigh + MemoryMax drop-in via systemctl edit

Done when: systemctl cat telemetry-relay.service shows both directives coming from override.conf, layered over the original unit.

7Clear the runtime override, reload, reset-failed, restart clean

Done when: systemctl status shows Active: active (running) with no failed-state history left over.

8Prove it holds across at least three burst cycles

Done when: three flush (burst) lines print cleanly and memory.events' oom_kill counter hasn't moved since before the fix.

🎬 At Mission Control
🦊Foxy: Why not just delete MemoryMax= entirely? No limit, no kill, problem gone, right?
🦥Sol the Sloth: Slow down, Foxy. No limit doesn't fix anything — it just removes the one thing that would've told you when a real leak shows up. You've traded one bug for a worse, invisible one.
👺Gizmo the Gremlin: Boooring. I'd just set it to like 8G and never think about this service again. 🤑
🐢Timmy the Turtle: That's not sizing a limit, Gizmo, that's abandoning one. A guardrail set from a guess is barely a guardrail — the whole point is the number means something.
🐘Ellie the Elephant: And memory.events already remembers every kill this cgroup ever had. That counter is the only honest record — trust it over how "fine" things feel right now.
🦥Sol the Sloth: Measure the peak. Give it real headroom to be throttled before it's room to be killed. Then watch the counter, not the calendar, before you call it fixed.

🐢 Timmy's checkpoint

1. Why did the exact same script run cleanly for two out of every three cycles and only die on the third — what does that tell you about diagnosing a limit that "sometimes" breaks? 2. What's the practical difference between crossing MemoryHigh= and crossing MemoryMax=, and which one actually killed telemetry-relay? 3. What in journalctl -k proves the kernel, not systemd, made the kill decision — and what in journalctl -u proves systemd only reported and restarted after the fact? 4. Where does MemoryMax= actually get written under the hood, and what command reads that value straight from the source instead of trusting systemctl's summary? 5. Why measure the real peak before writing new numbers, instead of just picking something generously large? 6. What exactly proves the fix held, and why is "it hasn't failed in the last five minutes" not good enough on its own?

Check your answers

The script's own logic only allocates its large ~150 MB burst every third cycle by design — the bug was never intermittent in the way a race condition is, it was a fixed, deterministic pattern that simply doesn't show up on every single observation. A bug that fails the identical way at the identical point every time it's actually exercised is a sizing bug, not a flaky one — the fix is to make sure you actually watch a full cycle through the point where it happens, not to assume "the last two runs were clean" means it's fixed.
MemoryHigh= is a soft ceiling — cross it and the kernel throttles the cgroup hard via reclaim pressure, but nothing is killed. MemoryMax= is the hard ceiling — cross it and the kernel's OOM killer terminates a process in that cgroup outright. The original unit only had MemoryMax=64M with no MemoryHigh= below it, so the burst went straight from "fine" to "killed" with no warning stage in between.
journalctl -k is the kernel's own ring buffer; the line Memory cgroup out of memory: Killed process ... (python3) comes directly from the kernel's memory controller, naming the exact cgroup it acted on. journalctl -u telemetry-relay.service shows systemd's own lines — "A process of this unit has been killed by the OOM killer" and Failed with result 'oom-kill' — which are systemd noticing and reporting a death it didn't cause, followed by the restart it does own.
MemoryMax= is written into the cgroup's memory.max file, under /sys/fs/cgroup/system.slice/<unit>.service/ for a system service. cat that file directly (or systemctl show telemetry-relay.service -p MemoryCurrent -p MemoryMax for systemd's own reading of it) rather than trusting a human-readable summary alone.
A limit set from a guess either fails again later, when real usage grows past a number nobody actually validated, or wastes resources by being far larger than anything the service will ever need, hiding a genuine future leak behind a ceiling too generous to ever trip. Measuring the actual peak first — with the ceiling temporarily loosened so the process survives long enough to be observed — turns the new limit into a number you can defend instead of one you're hoping is big enough.
memory.events' oom_kill counter staying at the exact same value across at least three full burst cycles after the fix, not climbing and not suspiciously reset to zero — because that counter only ever increases for the life of the cgroup, it's the one piece of evidence that can't be faked by a run that simply hasn't hit the bad case yet. "Hasn't failed recently" is what the drill's original bug looked like for two out of every three cycles, right before it failed again.

Holding across three straight bursts? Good — that's the whole drill. Tear it down when you're done: sudo systemctl disable --now telemetry-relay.service && sudo rm -rf /etc/systemd/system/telemetry-relay.service* /opt/telemetry-relay && sudo systemctl daemon-reload. For the concepts underneath today's bug, see systemd & journald for every directive at reference depth, and Linux Fundamentals for Platform Engineers for cgroup v2's mechanics built by hand from a bare shell. The full walkthrough of a Pod hitting this exact same kernel file is in the LFCS blueprint. Ready for a different single skill? Try Drill — Diagnose a Stuck Argo CD Sync, or step back to Build Your Cert Tracker — Start Here for the full continuity version.