LFCS Mock Exam · Set 3

This is the third and final LFCS paper in this course, and it's deliberately lopsided in the way real exam prep should be once you know your weak spots. Nineteen performance-based tasks, one unbroken 120-minute sitting, still worth exactly 100 points split 25 / 25 / 20 / 20 / 10 across the same five official domains as Set 1 and Set 2 — but not split evenly across tasks this time. Networking gets six tasks here and Storage gets five, while Operations Deployment and Essential Commands shrink to three apiece to make room, because the two domains most real incident reviews actually live in deserve more independent repetitions under a clock, not just a bigger point total. Every task keeps the same shape as the other two papers — a plain-language brief, a done when check you run yourself against a real terminal, and a full worked solution folded under a summary that stays closed until you've either passed the check or given up on it — and every scenario below is new, not a reskin of anything from Set 1 or Set 2. Build the machine the night before, skim the pre-flight section, then start one timer and don't stop it until task nineteen is behind you.

Before you start — exam conditions

☺ Like you're 10: This drill only teaches you something if you run it exactly like the real thing — same clock, same rules, no notes you wouldn't have on the day.

Everything that made Set 1 and Set 2 honest applies here too, plus a slightly longer shopping list because this paper's lean into Networking and Storage touches tools the earlier two barely needed.

One — build the machine the night before, with more moving parts than usual. A throwaway VM works, but this paper wants at least three spare block devices attached (one for an LVM snapshot, two for a software RAID1 mirror) plus a second interface or two for VLAN/bridge/bond-adjacent work, and — for the network-block-device task — either a second small VM to act as the NBD server or nbd-server running locally as a stand-in against a loopback export. Install everything the paper touches before the clock starts:

# RHEL/Fedora family
sudo dnf install -y lvm2 mdadm quota nbd haproxy sssd sssd-ldap nftables git openssl \
  libvirt libvirt-daemon virt-install
sudo modprobe 8021q nbd

# Debian/Ubuntu family
# sudo apt install -y lvm2 mdadm quota nbd-client nbd-server haproxy sssd nftables git \
#   openssl libvirt-daemon-system virtinst

lsblk                          # confirm the spare disks for the snapshot + RAID tasks are visible and empty
ip -br link                    # confirm at least two extra interfaces exist for VLAN/bridge work

Two — one timer, started once. 120 minutes, no pausing for a stubborn typo or a VM that needs a reboot. Time lost fighting your own lab environment is real data about exam-day friction, not a reason to stop the clock.

Three — only the permitted resources, open before you start. Man pages, documentation installed by the distribution, and packages that ship with it — accessed from inside the exam terminal itself. No browser, no search engine, no this course, no AI assistant.

Four — no copying commands from the solution, and no AI assistance. If you get stuck, spend the budget, fail the task, and read the solution after the sitting ends.

Five — read all nineteen tasks before you touch a terminal. With six networking tasks bunched together, five minutes spent skimming the whole paper is what stops you from accidentally spending twenty minutes on the fourth one when the sixth was the two-command freebie.

Your time budget

☺ Like you're 10: Give every domain a slice of the two hours sized to how many jobs live in it, not to how important the topic sounds — Networking gets the biggest slice here because it has the most jobs, not because it matters more than Storage.

Nineteen tasks in 120 minutes still averages under six and a half minutes each once you carve out five minutes to read and six to verify at the end, but this paper's shape is lumpier than Set 1 or Set 2: Operations Deployment and Essential Commands get three bigger, denser tasks apiece, while Networking's six tasks and Storage's five are individually smaller and faster — the trade this paper is deliberately making is fewer points per task in exchange for more independent chances to get a wire or a disk right under pressure.

The flag-and-move rule still applies exactly as it did on the first two papers: when a task passes roughly its budgeted minutes without a passing done-when check, stop, write one line about where you stalled, leave whatever you've applied in place, and move to the next task. On a paper this heavy on Networking, the rule matters more than usual — six tasks means six chances to fall down the same rabbit hole, and only one of them is worth more than five points.

Operations Deployment — OD1 to OD3 (25 points)

☺ Like you're 10: Three bigger jobs this time — bringing a whole virtual machine to life, teaching this box to trust and install new software from a private source, and rescuing a machine that won't finish booting.

Fewer, denser tasks than Set 1 or Set 2 to make room for Networking and Storage below. Background: LFCS — the exam's Operations Deployment section.

OD1 · Bring a pre-built VM to life with libvirt and make it survive a reboot

A qcow2 disk image for a new edge node, /var/lib/libvirt/images/edge-01.qcow2, has been staged on this host by an earlier build step. It needs to become a running, named libvirt domain — 2 vCPUs, 2048 MB of RAM, attached to the default virtual network — that comes back up automatically the next time this host reboots, without anyone running a manual virsh start.

Your task:

1. Define and start a domain named edge-01 around the existing disk image, without reinstalling an OS onto it.
2. Confirm it's actually running, not just defined.
3. Enable autostart so the hypervisor brings it up on the next host boot.

Done when: virsh list --all shows edge-01 in state running; virsh dominfo edge-01 reports Autostart: enable; and virsh domifaddr edge-01 returns at least one lease once the guest has finished booting. (9 points)

virt-install \
  --name edge-01 \
  --memory 2048 \
  --vcpus 2 \
  --disk path=/var/lib/libvirt/images/edge-01.qcow2,format=qcow2 \
  --import \
  --os-variant generic \
  --network network=default \
  --graphics none \
  --noautoconsole

virsh list --all                 # edge-01 ... running

virsh autostart edge-01
virsh dominfo edge-01 | grep -i autostart      # Autostart: enable

# give the guest a minute to boot and request a DHCP lease, then:
virsh domifaddr edge-01

OD2 · Add a signed private package repository and pin an exact version

A vendor tool, telemetry-agent, ships from a private RPM repository at https://repo.mission.internal/rpm/el9/, GPG-signed. It needs to be installed at exactly version 2.4.1 — not whatever is newest — and it needs to stay at that version even after a routine dnf update runs later.

Your task:

1. Add the repository with signature checking enabled and import its GPG key.
2. Install exactly version 2.4.1 of telemetry-agent.
3. Lock that package so a later dnf update cannot silently move it to a newer version.

Done when: dnf repolist shows mission-tools enabled; rpm -q telemetry-agent reports exactly telemetry-agent-2.4.1; and after dnf versionlock list shows the package locked, running dnf update telemetry-agent reports nothing to do. (8 points)

# /etc/yum.repos.d/mission-tools.repo
[mission-tools]
name=Mission Tools
baseurl=https://repo.mission.internal/rpm/el9/
enabled=1
gpgcheck=1
gpgkey=https://repo.mission.internal/RPM-GPG-KEY-mission

rpm --import https://repo.mission.internal/RPM-GPG-KEY-mission
dnf makecache
dnf repolist                                        # mission-tools ... enabled

dnf list --showduplicates telemetry-agent           # confirm 2.4.1 is an available build
dnf install -y telemetry-agent-2.4.1
rpm -q telemetry-agent

dnf install -y python3-dnf-plugin-versionlock       # if not already present
dnf versionlock add telemetry-agent
dnf versionlock list

dnf update telemetry-agent                          # "Nothing to do."

OD3 · Recover a host stuck in emergency mode after a bad fstab edit

This host was rebooted after someone hand-edited /etc/fstab and it now drops into emergency mode instead of reaching a normal login — a systemd dependency failure tied to a mount unit. You have an emergency shell.

Your task:

1. From the emergency shell, identify which fstab entry is actually causing the failure, without guessing.
2. Fix the entry so the host can boot normally again.
3. Prove the fix is correct before committing to another reboot.

Done when: mount -a completes with no errors from the emergency shell; after a reboot, systemctl is-system-running reports running (or degraded with no failed mount units specifically); and systemctl list-units --failed shows no .mount units. (8 points)

# from the emergency shell
journalctl -xb | grep -iE "fstab|mount|dependency failed"
# ... reveals a mismatched UUID or a bad mount point on one specific fstab line

mount -o remount,rw /            # the emergency shell often mounts root read-only
vi /etc/fstab                    # fix the offending line — corrected UUID, or add nofail
                                  # for a device that isn't always guaranteed to be present

mount -a                         # rehearse the fix exactly as boot would apply it,
                                  # WITHOUT committing to a second reboot yet
systemctl default                # attempt to continue the current boot without rebooting

reboot                           # once mount -a is clean, confirm with a real reboot
systemctl is-system-running
systemctl list-units --failed

Networking — N1 to N6 (25 points)

☺ Like you're 10: Six jobs about making sure the right message reaches the right place — a private lane on a shared wire, a reply that goes back out the door it came in, a shared fence between two rooms, a newer kind of address, traffic split between two helpers, and a name that points to the right desk.

The biggest single block on this paper, both in points and in task count. Per LFCS — the exam and Platform Engineering's own Networking chapter, this is also the domain where real incidents concentrate. Use ip, not ifconfig; ss, not netstat; nft, not raw iptables; resolvectl, not a hand-edited resolv.conf, wherever the host runs systemd-resolved.

N1 · Tag a VLAN subinterface for isolated management traffic

Management traffic for the telemetry rack needs to move onto VLAN 100, isolated from the untagged production LAN sharing the same physical NIC, eth0. The subinterface needs address 10.100.0.5/24 and must survive a reboot.

Your task:

1. Create the tagged VLAN subinterface eth0.100 on eth0 and assign it the required address.
2. Persist it using your distribution's real network configuration mechanism.
3. Confirm the tag and address survive a service restart, not just a live check.

Done when: ip -d link show eth0.100 reports vlan protocol 802.1Q id 100; ip addr show eth0.100 lists 10.100.0.5/24; and restarting systemd-networkd (or the NetworkManager equivalent) leaves both in place. (5 points)

ip link add link eth0 name eth0.100 type vlan id 100
ip addr add 10.100.0.5/24 dev eth0.100
ip link set eth0.100 up
ip -d link show eth0.100                 # confirm: vlan protocol 802.1Q id 100

# /etc/systemd/network/10-eth0.network  (the parent interface — must reference the VLAN)
[Match]
Name=eth0
[Network]
VLAN=eth0.100

# /etc/systemd/network/20-eth0.100.netdev
[NetDev]
Name=eth0.100
Kind=vlan
[VLAN]
Id=100

# /etc/systemd/network/20-eth0.100.network
[Match]
Name=eth0.100
[Network]
Address=10.100.0.5/24

systemctl restart systemd-networkd
ip -d link show eth0.100
ip addr show eth0.100

N2 · Route return traffic out the interface it actually belongs to

This host is multi-homed: eth0 carries the default route via ISP-A, and eth1 is a secondary uplink through ISP-B at 198.51.100.20/24, gateway 198.51.100.1. Traffic sourced from 198.51.100.20 keeps leaving via the default route on eth0 instead of back out eth1, and ISP-B's edge firewall — correctly — drops it as spoofed.

Your task:

1. Add a policy rule so any packet sourced from 198.51.100.20 uses a separate routing table.
2. Populate that table with a default route via ISP-B's gateway, out eth1.
3. Persist both the rule and the table so they survive a network service restart.

Done when: ip route get 8.8.8.8 from 198.51.100.20 reports it would go via 198.51.100.1 dev eth1; ip rule show lists the custom rule; and restarting systemd-networkd leaves both the rule and the table populated. (4 points)

ip rule add from 198.51.100.20 table 100
ip route add default via 198.51.100.1 dev eth1 table 100
ip route add 198.51.100.0/24 dev eth1 src 198.51.100.20 table 100

ip rule show
ip route show table 100
ip route get 8.8.8.8 from 198.51.100.20        # confirm it now selects table 100 / eth1

# /etc/systemd/network/10-eth1.network  (additions)
[RoutingPolicyRule]
From=198.51.100.20
Table=100

[Route]
Gateway=198.51.100.1
Table=100

systemctl restart systemd-networkd
ip rule show
ip route show table 100

N3 · Bridge a physical interface for two isolated lab guests

A pair of local KVM guests need a private lab network on their own bridge — not the default NAT'd libvirt network — with a dedicated physical uplink, eth3, enslaved to it. The bridge itself needs a usable management address.

Your task:

1. Create bridge br0 and enslave eth3 to it.
2. Assign 10.50.0.1/24 to the bridge itself, not to eth3.
3. Confirm the slave interface is actually forwarding, not just attached.

Done when: bridge link show lists eth3 with master br0 state forwarding; ip addr show br0 shows 10.50.0.1/24; and ip addr show eth3 shows no IPv4 address of its own. (4 points)

ip link add name br0 type bridge
ip link set eth3 down
ip link set eth3 master br0
ip link set eth3 up
ip link set br0 up
ip addr add 10.50.0.1/24 dev br0

bridge link show                    # eth3 ... master br0 state forwarding
ip addr show br0                    # 10.50.0.1/24
ip addr show eth3                   # no IPv4 address — the bridge owns it now

# /etc/systemd/network/25-br0.netdev
[NetDev]
Name=br0
Kind=bridge

# /etc/systemd/network/25-br0.network
[Match]
Name=br0
[Network]
Address=10.50.0.1/24

# /etc/systemd/network/10-eth3.network
[Match]
Name=eth3
[Network]
Bridge=br0

N4 · Add a routable IPv6 address and prove connectivity, not just presence

The telemetry-archive host needs a routable IPv6 address on eth0 — 2001:db8:100::5/64 — with a default IPv6 route via 2001:db8:100::1, persisted, and actually reachable, not merely configured.

Your task:

1. Assign the address and add the default IPv6 route.
2. Persist both.
3. Prove the address is a global one and that the route actually carries traffic.

Done when: ip -6 addr show eth0 lists 2001:db8:100::5/64 with scope global (not only the automatic fe80:: link-local address); ip -6 route get 2001:db8:100::1 resolves via eth0; and ping -6 -c2 2001:db8:100::1 receives replies. (4 points)

ip -6 addr add 2001:db8:100::5/64 dev eth0
ip -6 route add default via 2001:db8:100::1 dev eth0

ip -6 addr show eth0                # confirm scope global, not just link-local
ip -6 route get 2001:db8:100::1
ping -6 -c2 2001:db8:100::1

# /etc/systemd/network/10-eth0.network  (additions)
[Network]
Address=2001:db8:100::5/64
Gateway=2001:db8:100::1

N5 · Load-balance two backends with a health check that actually removes a dead one

Two identical backend services run on 10.0.2.11:8080 and 10.0.2.12:8080. The public-facing host needs to round-robin across both on port 80, and a backend that stops responding needs to fall out of rotation automatically — not stay in it, quietly failing every request routed to it.

Your task:

1. Configure HAProxy to front both backends on port 80 with round-robin balancing.
2. Add an active health check against each backend.
3. Prove the failover actually works by stopping one backend and confirming the frontend keeps serving.

Done when: haproxy -c -f /etc/haproxy/haproxy.cfg reports no errors; repeated requests against the frontend return 200 and are served by both backends; and after one backend is stopped, requests against the frontend still return 200, served entirely by the surviving backend. (4 points)

# /etc/haproxy/haproxy.cfg (additions)
frontend telemetry_front
    bind *:80
    default_backend telemetry_back

backend telemetry_back
    balance roundrobin
    option httpchk GET /healthz
    server web1 10.0.2.11:8080 check
    server web2 10.0.2.12:8080 check

haproxy -c -f /etc/haproxy/haproxy.cfg       # validate BEFORE reload — never reload blind
systemctl enable --now haproxy
systemctl reload haproxy

for i in 1 2 3 4; do curl -s localhost/ -o /dev/null -w "%{http_code}\n"; done

# prove the health check actually removes a dead backend
systemctl stop web1-backend                  # simulate a failure on web1

for i in 1 2 3 4; do curl -s localhost/ -o /dev/null -w "%{http_code}\n"; done
# still all 200s — every request now served by web2 alone

N6 · Route one internal domain's DNS lookups to a different resolver

Queries for anything under *.mission.internal need to go to an internal resolver, 10.0.0.53, while every other lookup keeps using the normal upstream DNS. A previous engineer's fix was a single /etc/hosts entry, which doesn't scale past one hostname and breaks the moment a second internal host needs resolving.

Your task:

1. Point eth0's DNS at 10.0.0.53 for the mission.internal domain specifically, not globally.
2. Persist the configuration.
3. Confirm internal names resolve via the internal resolver while everything else still resolves normally.

Done when: resolvectl status eth0 shows DNS server 10.0.0.53 with domain ~mission.internal; resolvectl query relay.mission.internal reports it was answered via eth0's configured server; and resolvectl query example.com still resolves through the normal upstream. (4 points)

resolvectl dns eth0 10.0.0.53
resolvectl domain eth0 "~mission.internal"

resolvectl status eth0
resolvectl query relay.mission.internal      # answered via eth0's DNS server
resolvectl query example.com                 # still answered via the default upstream

# /etc/systemd/network/10-eth0.network  (additions)
[Network]
DNS=10.0.0.53
Domains=~mission.internal

systemctl restart systemd-networkd
resolvectl status eth0

Storage — S1 to S5 (20 points)

☺ Like you're 10: Five jobs about protecting and organizing what's already on a disk — a safety copy before a risky change, a mirror that survives one disk dying, a fair share of space per person, a disk borrowed from another machine, and figuring out why "full" doesn't always mean what you think.

Background: LFCS — the exam, and for the tool depth, LVM & Linux Storage Tools.

S1 · Snapshot a logical volume before a risky change, then roll back

A risky in-place migration is about to run against /dev/vg_data/lv_app, mounted at /srv/app. If it corrupts data, the fix needs to be a rollback, not a restore from a separate backup system that might not even exist yet.

Your task:

1. Create a 2G snapshot of lv_app before the migration.
2. Assume the migration went wrong. Roll lv_app back to the snapshot.
3. Confirm the rollback actually took effect.

Done when: before rollback, lvs vg_data lists lv_app_snap with origin lv_app; after lvconvert --merge and reactivating the volume, lvs vg_data no longer lists the snapshot at all; and the filesystem mounted at /srv/app reflects the pre-migration state. (4 points)

lvcreate -L 2G -s -n lv_app_snap /dev/vg_data/lv_app
lvs vg_data                                  # lv_app_snap ... Origin lv_app

# --- the migration runs here, and goes wrong ---

umount /srv/app
lvconvert --merge vg_data/lv_app_snap        # schedules the merge

lvchange -an vg_data/lv_app                  # deactivate...
lvchange -ay vg_data/lv_app                  # ...and reactivate to complete a merge on an in-use LV
mount /dev/vg_data/lv_app /srv/app

lvs vg_data                                  # lv_app_snap is gone — merged and consumed

S2 · Mirror two disks with software RAID1, then survive a simulated failure

A dataset on two spare disks, /dev/sdc and /dev/sdd, must survive a single disk failure without taking the mount offline. Build the mirror, mount it persistently, then prove it actually tolerates a failure.

Your task:

1. Create a RAID1 array across the two disks, put a filesystem on it, and mount it persistently at /srv/mirrored.
2. Simulate a failed member disk.
3. Recover the array by removing the failed member and adding a replacement.

Done when: cat /proc/mdstat initially shows md0 fully in sync across both members; after mdadm --fail, it shows the array degraded but still active, with /srv/mirrored still mounted and readable; and after removing the failed disk and adding a fresh one, /proc/mdstat shows the array resyncing back to full redundancy. (4 points)

mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdc /dev/sdd
cat /proc/mdstat                              # resync in progress, then active/in-sync

mkfs.xfs /dev/md0
mkdir -p /srv/mirrored
mount /dev/md0 /srv/mirrored

mdadm --detail --scan >> /etc/mdadm.conf      # Debian family: /etc/mdadm/mdadm.conf — check which yours reads
echo "/dev/md0  /srv/mirrored  xfs  defaults,nofail  0  2" >> /etc/fstab
mount -a

# simulate a failed disk
mdadm /dev/md0 --fail /dev/sdc
cat /proc/mdstat                              # sdc marked (F); array degraded but still active
ls /srv/mirrored                              # still readable — the mount never had to move

# recover: drop the failed member, add a replacement
mdadm /dev/md0 --remove /dev/sdc
mdadm /dev/md0 --add /dev/sde
cat /proc/mdstat                              # resyncing back to full redundancy

S3 · Cap one user's disk usage without touching anyone else's

/home lives on its own ext4 filesystem, and one account, intern, keeps filling it with a runaway build cache. Set a block quota for intern specifically — 5G soft, 6G hard, with the standard grace period — without capping any other user.

Your task:

1. Enable quota accounting on /home.
2. Set intern's soft and hard block limits.
3. Confirm the limits apply only to intern.

Done when: repquota /home lists intern with a 5G soft / 6G hard block limit; quota -u intern reflects the same; and repquota /home shows no limit set for any other user on the filesystem. (4 points)

# /etc/fstab — /home's line needs the usrquota option
/dev/vg_data/lv_home  /home  ext4  defaults,usrquota  0  2

mount -o remount /home
quotacheck -cum /home                         # creates aquota.user, initial usage scan
quotaon /home

# limits are in 1K blocks: 5G = 5242880, 6G = 6291456 — setquota takes no unit suffixes
setquota -u intern 5242880 6291456 0 0 /home

repquota /home                                # intern ... 5242880 6291456 ...
quota -u intern

S4 · Attach a remote block device over NBD and make it reattach on boot

A remote storage host exports a raw block device over the network at storage-01:10809, export name mission-blk. It needs to appear locally as a normal block device, hold a filesystem, mount at /mnt/remote-block, and come back automatically after a reboot — without anyone manually re-running nbd-client.

Your task:

1. Attach the export locally as /dev/nbd0, create a filesystem on it, and mount it.
2. Write a systemd unit that performs the attach automatically, ordered after the network is up.
3. Wire the mount to depend on that unit so it doesn't try to mount a device that was never attached.

Done when: lsblk /dev/nbd0 shows it attached and mounted at /mnt/remote-block; systemctl is-enabled nbd-mission-blk.service reports enabled; and after systemctl restart nbd-mission-blk.service && mount -a, the mount comes back with no manual nbd-client command typed by hand. (4 points)

modprobe nbd max_part=8
nbd-client storage-01 10809 /dev/nbd0 -N mission-blk
lsblk /dev/nbd0

mkfs.ext4 /dev/nbd0
mkdir -p /mnt/remote-block
mount /dev/nbd0 /mnt/remote-block

# /etc/systemd/system/nbd-mission-blk.service
[Unit]
Description=Attach mission-blk NBD export
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/modprobe nbd max_part=8
ExecStart=/usr/sbin/nbd-client storage-01 10809 /dev/nbd0 -N mission-blk
ExecStop=/usr/sbin/nbd-client -d /dev/nbd0

[Install]
WantedBy=multi-user.target

# /etc/fstab
/dev/nbd0  /mnt/remote-block  ext4  _netdev,x-systemd.requires=nbd-mission-blk.service  0  0

systemctl daemon-reload
systemctl enable --now nbd-mission-blk.service
mount -a

S5 · Diagnose a "disk full" error on a filesystem that isn't full

A batch job writing into /var/spool/jobs — its own ext4 filesystem — keeps failing with No space left on device, but df -h reports the filesystem is only 40% full by capacity. Find the real cause and fix it without deleting anything the job still needs.

Your task:

1. Explain the contradiction between the error and what df -h reports, without guessing.
2. Identify which directory is actually responsible.
3. Reclaim the resource that's actually exhausted, without touching unrelated live job files.

Done when: df -i /var/spool/jobs shows IUse% back under a safe threshold; the batch job's next write succeeds; and a file count of the directory before and after shows only stale files were removed. (4 points)

df -h /var/spool/jobs                         # 40% full by space — doesn't explain the error
df -i /var/spool/jobs                         # IUse% 100%, IFree 0 — every inode is used up

find /var/spool/jobs -xdev -type d -exec sh -c \
  'echo "$(find "$1" -maxdepth 1 | wc -l) $1"' _ {} \; | sort -n | tail
# identifies /var/spool/jobs/tmp/ as the directory holding millions of tiny files

# clean up genuinely stale files only — never a blind rm -rf
find /var/spool/jobs/tmp -type f -empty -mtime +7 -delete

df -i /var/spool/jobs                         # inodes freed, IUse% well under 100%

Essential Commands — E1 to E3 (20 points)

☺ Like you're 10: Three jobs — finding exactly which change broke something without checking every change one by one, freeing up space a log file is quietly hoarding, and proving a website's certificate is trusted all the way up, not just today.

Background: LFCS — the exam, and for the TLS half, cert-manager automates in a cluster exactly the chain problem E3 makes you diagnose by hand.

E1 · Find the exact commit that broke a script with a binary search, not a guess

deploy.sh in the ops-scripts repository started silently exiting 1 sometime in the last dozen commits, and nobody knows which one. HEAD is broken; the tag v3 is known good.

Your task:

1. Use a binary search over the commit history to find the exact first bad commit — not a manual walk through each one.
2. Confirm which commit it is.
3. Leave the repository in a clean, restored state when you're done.

Done when: the bisect process reports a single specific commit hash as the first bad commit; git status afterward shows a clean working tree with no detached-HEAD leftover; and that commit's diff plausibly touches deploy.sh. (7 points)

cd ops-scripts
git bisect start
git bisect bad HEAD
git bisect good v3
# git checks out a midpoint commit automatically

./deploy.sh --dry-run; echo $?          # test at each step git lands you on
git bisect good                          # or: git bisect bad — depending on the result
# ... repeat; git narrows the range by half each time

# once git reports "<hash> is the first bad commit":
git show <hash>                          # confirm the diff touches deploy.sh
git bisect log > bisect-report.txt      # optional record of the search
git bisect reset                         # return to the original branch and HEAD

E2 · Rotate a runaway log without leaking the disk space back to nothing

beacon-api.log under /var/log/beacon/ has grown to 40G because no rotation was ever configured, and it's now the actual cause of repeated disk-full alerts on /var. Configure rotation, and make sure the running service actually releases the old file's space rather than continuing to write into it under a different name.

Your task:

1. Configure daily rotation, keeping 14 days compressed.
2. Make sure the writing service reopens its log file after each rotation.
3. Prove a forced rotation right now actually frees space, not just renames the file.

Done when: logrotate -d /etc/logrotate.d/beacon-api reports no config errors; a forced rotation produces a rotated .1 file plus a small, fresh beacon-api.log; and checking the service's open file descriptors shows it holding the new file, not the rotated one. (7 points)

# /etc/logrotate.d/beacon-api
/var/log/beacon/beacon-api.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 beacon beacon
    postrotate
        systemctl kill -s HUP beacon-api.service
    endscript
}

logrotate -d /etc/logrotate.d/beacon-api      # dry run — shows what WOULD happen, changes nothing
logrotate -f /etc/logrotate.d/beacon-api      # force one rotation right now to prove it works

ls -la /var/log/beacon/                        # beacon-api.log.1 present, new beacon-api.log small again
ls -la /proc/$(pgrep beacon-api)/fd | grep beacon-api.log   # holding the NEW file, not the rotated one
df -h /var

E3 · Diagnose and fix a broken TLS chain, not a broken certificate

An internal API at api.mission.internal serves a certificate signed by a private intermediate CA, but some clients report a trust failure even though the certificate itself hasn't expired. Determine whether the service is actually sending the full chain, and fix it if it isn't.

Your task:

1. Determine how many certificates the live service actually sends during a handshake.
2. Prove whether the chain verifies all the way to a trusted root.
3. Fix the served chain if the intermediate is missing.

Done when: openssl s_client -showcerts against the live service returns two certificates — leaf then intermediate — not one; openssl verify -CAfile <root> -untrusted <intermediate> <leaf> reports OK; and a client that trusts only the root (not the intermediate directly) completes the handshake without a chain error. (6 points)

openssl s_client -connect api.mission.internal:443 -servername api.mission.internal \
  -showcerts </dev/null 2>/dev/null | grep -c "BEGIN CERTIFICATE"
# 1 — only the leaf is being sent. The intermediate is missing from the wire.

openssl verify -CAfile /etc/pki/ca-trust/mission-root-ca.pem \
  -untrusted intermediate-ca.pem server.pem
# fails without the intermediate supplied separately

# fix: serve leaf + intermediate concatenated, in that order
cat server.pem intermediate-ca.pem > /etc/nginx/tls/api.mission.internal-fullchain.pem
# update nginx's ssl_certificate directive to point at the fullchain file, not the bare leaf

nginx -t && systemctl reload nginx

openssl s_client -connect api.mission.internal:443 -showcerts </dev/null 2>/dev/null \
  | grep -c "BEGIN CERTIFICATE"
# now 2 — leaf and intermediate both sent

Users & Groups — U1 to U2 (10 points)

☺ Like you're 10: The smallest slice of the exam, but two of its cheapest points — giving every account a shared set of tools automatically, and letting one host trust a shared directory of accounts without breaking local logins if that directory ever goes quiet.

Only 10% of the paper, but per the flag-and-move discipline above, never the two tasks worth skipping.

U1 · Set a shared environment for everyone, plus an extra for one account

Every account on this host needs MISSION_HOME=/opt/mission and /opt/mission/bin on its PATH by default. One service account, svc-nav, additionally needs /opt/mission/nav-tools on its own PATH — a directory no other account should see added.

Your task:

1. Add the shared environment variable and PATH entry for every account, system-wide.
2. Add svc-nav's extra PATH entry without affecting any other account.
3. Confirm both layers apply correctly for the right accounts.

Done when: su - alice -c 'echo $PATH' (any other real account) includes /opt/mission/bin but not /opt/mission/nav-tools; su - svc-nav -c 'echo $PATH' includes both; and MISSION_HOME is set correctly under any login session. (4 points)

# /etc/profile.d/mission.sh — sourced automatically for every login shell
export MISSION_HOME=/opt/mission
export PATH="$PATH:/opt/mission/bin"
chmod 644 /etc/profile.d/mission.sh

# ~svc-nav/.bash_profile
[ -f /etc/profile ] && . /etc/profile
export PATH="$PATH:/opt/mission/nav-tools"

chown svc-nav:svc-nav ~svc-nav/.bash_profile

su - alice -c 'echo $MISSION_HOME; echo $PATH'    # bin present, nav-tools NOT present
su - svc-nav -c 'echo $PATH'                       # both present

U2 · Point one host at LDAP-backed accounts without breaking local logins

This host needs to authenticate one group of contractor accounts against an existing corporate LDAP directory at ldap://ldap.mission.internal, while every local account — including root — must keep working exactly as before if LDAP is briefly unreachable.

Your task:

1. Configure SSSD to look up accounts against the LDAP directory.
2. Set the NSS lookup order so local accounts are always checked first.
3. Prove local accounts survive an LDAP outage, while LDAP-only accounts don't.

Done when: getent passwd <ldap-user> returns a full LDAP-backed entry; /etc/nsswitch.conf lists files before sss on the passwd and group lines; and after systemctl stop sssd, getent passwd root and a local-account login both still work, while the LDAP-only account stops resolving until sssd is back. (6 points)

# /etc/sssd/sssd.conf
[sssd]
services = nss, pam
domains = mission.internal

[domain/mission.internal]
id_provider = ldap
auth_provider = ldap
ldap_uri = ldap://ldap.mission.internal
ldap_search_base = dc=mission,dc=internal
cache_credentials = true

chmod 600 /etc/sssd/sssd.conf
authselect select sssd --force
systemctl enable --now sssd

grep -E "^(passwd|group):" /etc/nsswitch.conf
# passwd: files sss
# group:  files sss

getent passwd contractor.jane          # resolves via LDAP through sssd
id contractor.jane
getent passwd root                     # resolves purely from /etc/passwd, no LDAP round trip

# prove the outage behavior
systemctl stop sssd
getent passwd root                     # still works — local, unaffected
getent passwd contractor.jane          # fails to resolve until sssd is back

Score yourself

☺ Like you're 10: Add up the points, turn it into a percentage, and then do the more useful thing — look at which topic cost you the most and go fix that specifically.

Mark after a short break — grading your own work while still adrenalized produces generous nonsense. Award full points only when the done-when check actually passed for real, half when the resource exists and is broadly right but the check didn't pass cleanly, and zero for anything unattempted.

Computing your result. The points total 100, so your raw score is your percentage, against the published 67% pass mark. Then do the more useful arithmetic: total each domain separately, and this time also compare against your Set 1 and Set 2 domain scores if you sat them. A domain that improved across all three sittings is a real signal; a domain that's flat or worse despite this paper giving it more repetitions is the one that needs a different kind of practice, not just more of the same kind.

What your score means

☺ Like you're 10: This is the last of the three papers, so the question now isn't "which set do I sit next" — it's "am I actually ready to book."

Treat the number as a diagnostic, never a prediction, same as the earlier two papers. But because this is the third and final sitting in this course, read the band below as a booking signal, not just a study signal.

One check worth running across all three sittings, not just this one: did your Networking and Storage scores actually improve from Set 1 through Set 2 to this paper? If they did, the extra repetitions this paper deliberately front-loaded did their job. If they didn't, the gap probably isn't repetition at all — it's a specific concept (policy routing, snapshot merge semantics, a health-check flag) worth reading about directly rather than drilling blind again.

That's all three papers. If you've sat all of them, you now have three domain-by-domain scorecards instead of one — look at the trend, not any single number, and let a domain that's stayed weak across all three sittings decide what you study before you book anything. For the deeper cross-reference against a platform engineer's actual daily use of this material, see Platform Engineering's own LFCS page, and for booking mechanics, retake terms and the retired LFCE, see LFCS — the exam and the official LFCS certification page.