Hands-On Labs · Guided Drills

Drill — Leaked Credential Triage

Eleven minutes ago, a gitleaks check failed on a pull request that had already been merged into a public repo you maintain — it flagged what looks like a live AWS access key, hardcoded in config/settings.py. The key has been sitting in the repository's history since its very first commit. You don't yet know for how long it's been publicly reachable, or whether anyone's actually used it. This page is one scenario, worked start to finish: the exact sequence a real leaked-credential incident calls for — revoke, scan, rotate, verify — run against a deliberately messy fake commit history that includes a false "fix," a second copy of the same key hiding on a branch that already got merged, and a rename thrown in to make a lazy search miss it. Time box: 25 minutes, start to a credential you've actually confirmed is dead — not one you're merely hoping is dead.

☺ Explain it like I'm 10

If you drop your house key on a busy sidewalk, the fix isn't sweeping the sidewalk so it looks like it never happened — someone may have already picked the key up before you even noticed it was gone. The fix is changing the lock. Sweeping the sidewalk is worth doing afterward, but only changing the lock actually stops a stranger from walking in, and that's the one step that has to happen first — before you've even finished figuring out who might have seen it fall.

🐘🦊Your hosts for this drill: Ellie the Elephant & Foxy — Ellie kills the old key's access the second it's confirmed leaked, no matter how incomplete the picture still is, and Foxy refuses to accept "we removed it from the file" as the end of the story until the full git history backs it up.

The scenario: a live key just went public

☺ Like you're 10: A real key is sitting somewhere public right now. The clock started the moment it was pushed, not the moment anyone noticed.

The repo is reporting-service, public on GitHub, and the failed check is a gitleaks job that only runs after merge — a gap this drill deliberately seeds, because it's exactly the kind of hole Security in CI/CD argues for closing by making secrets scanning a required, pre-merge check instead of an after-the-fact alert. That gap already happened; this drill starts from the alert you actually got, not the gate that should have stopped it. What you know right now: a key matching AWS's access-key-ID shape is present in config/settings.py on main. What you don't know yet, and have to find out: how long it's been there, whether it's the only copy, and whether it's been used by anyone who isn't you.

⚠ Before you start the clock

You need git installed to follow the history commands for real; gitleaks is worth having but not required — every finding below is also reachable with plain git log. The AWS CLI commands are the running example because IAM's revoke/rotate/verify calls are clean and well-documented, but the sequence is provider-agnostic: a GCP service-account key, an Azure client secret, a Stripe or GitHub token each have their own revoke and rotate calls, and each one still owes you the same four steps in the same order. Twenty-five minutes, starting now.

The repo you're triaging

☺ Like you're 10: Before you can fix a mess, you have to actually look at the whole mess — not just the one file someone already tried to clean up.

config/settings.py looked like this when the repo was created, seven commits ago:

# config/settings.py — the very first commit, 1a77003
AWS_ACCESS_KEY_ID = "AKIAIOSFODNN7EXAMPLE"
AWS_SECRET_ACCESS_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
S3_EXPORT_BUCKET = "reporting-export-prod"

Two commits later, someone noticed and "fixed" it — moved the key out to environment variables, and the file on HEAD looks clean:

# config/settings.py — after 5d0c771, "fix: remove hardcoded AWS key from settings.py"
import os
AWS_ACCESS_KEY_ID = os.environ["AWS_ACCESS_KEY_ID"]
AWS_SECRET_ACCESS_KEY = os.environ["AWS_SECRET_ACCESS_KEY"]
S3_EXPORT_BUCKET = os.environ["S3_EXPORT_BUCKET"]

That looks fixed. It isn't. Here's the actual graph, across every branch and ref, not just the commits that are ancestors of HEAD on main:

$ git log --oneline --all --graph
* 9f21b6a (HEAD -> main) chore: switch settings loader to read from environment
* 5d0c771 fix: remove hardcoded AWS key from settings.py
*   e88a204 Merge branch 'feature/reporting-export' into main
|\
| * 2a63f19 (feature/reporting-export) feat: wire S3 export job with working creds
| * 71bd0a0 wip: reporting export scaffold
* | c40de52 docs: note the export job in README
|/
* 1a77003 feat: initial commit — config/settings.py with live AWS key

A teammate branched feature/reporting-export off the very first commit, before 5d0c771's "fix" existed, and copy-pasted the same key into a new file to get an export job working quickly:

# jobs/reporting_export.py — introduced at 2a63f19, on feature/reporting-export
import boto3
# TODO: move this to env vars like settings.py does — same key, just faster to wire up for now
session = boto3.Session(
    aws_access_key_id="AKIAIOSFODNN7EXAMPLE",
    aws_secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
)

That branch merged into main at e88a204, before the "fix" landed at 5d0c771. So the current tip of main is clean, git log -p -- config/settings.py shows the key being removed and looks like a closed loop, and the same secret is nonetheless still sitting, byte for byte, in a file that's fully reachable from main's own history. That gap — checking one file's history instead of the whole repository's — is the exact trap this drill is built to spring.

Get the order right before you touch anything

☺ Like you're 10: Doing the right four things in the wrong order can leave the key just as live as doing nothing at all.

The instinctive response to "a secret is in git history" is to reach for a history rewrite — it feels like undoing the mistake. It's the wrong first move. A history rewrite changes what a fresh clone of your repo will show from now on; it does nothing to the key itself, nothing to any clone or fork that already exists, and nothing to whatever a scraper or a cache captured in the seconds after the push. The only action that actually stops the key from working is telling the credential's own issuer — AWS, in this drill — to stop honoring it. That has to happen first, and it has to happen before you've even finished figuring out the full scope, because scoping takes minutes and an exposed live key is exploitable for every one of them.

Step 1 — revoke, before you've confirmed anything else

☺ Like you're 10: Change the lock now. Figure out who might have a copy of the old key afterward.

Deactivate the key at its source, immediately, before you've scanned a single commit. AWS lets you flip a key's status to Inactive in one atomic call — a reversible action if it somehow turns out to be a false positive, but functionally dead the instant it runs:

aws iam update-access-key \
  --user-name svc-reporting-export \
  --access-key-id AKIAIOSFODNN7EXAMPLE \
  --status Inactive

That single command is the whole point of this step. You have not yet scanned the history, you don't yet know if there's a second copy on some branch, and you haven't issued a replacement — none of that matters yet. What matters is that the credential a stranger might already have stops working now, not after you've finished being thorough. If deactivating it will break something in production before a replacement is ready, that's a real, valid tradeoff to weigh — but weigh it deliberately, in the moment, against the alternative of a live, publicly exposed key. Don't let "I need to be careful" quietly become "I'll get to it after I've looked around a bit."

Step 2 — scan git history to confirm the real scope

☺ Like you're 10: Checking one file's current history tells you about that one file. Checking the whole repository's history tells you what's actually true.

The key is dead now, so this step is about finding out how bad it was, not about racing a clock. The single most useful technique here is git's pickaxe search — -S — which finds every commit, on every branch, whose diff added or removed an exact string, regardless of which file it was in or whether that file's since been renamed:

git log --all -p -S"AKIAIOSFODNN7EXAMPLE"

Run against the graph above, that one command surfaces both hits: the original introduction at 1a77003, and the independent copy on the merged feature branch at 2a63f19 — the second one git log -p -- config/settings.py alone would never show you, because it lives in a different file entirely. This is the concrete lesson of this step: "it's been removed from the file" and "it's gone from the repository" are different claims, and only the second one is the one that matters.

Round out the picture with a dedicated scanner and the platform's own detection, rather than relying on pickaxe alone for a real incident:

# gitleaks only walks HEAD's ancestry by default — pass --all through --log-opts
gitleaks detect --source . --log-opts="--all" --report-path gitleaks-report.json --no-banner

# trufflehog's flag surface has shifted a lot across v2/v3 — confirm against
# `trufflehog --help` for the version you actually have installed
trufflehog git file://. --since-commit 1a77003

Then close out the scope with two checks that live outside git entirely. First, GitHub's own Secret Scanning (Settings → Security → Secret scanning alerts on the real repo) — for some partner-integrated providers GitHub notifies or even auto-revokes on detection in a public repo, but don't assume AWS is on that list without checking GitHub's current partner roster, and don't treat "GitHub didn't flag it" as proof of anything either. Second, forks: gh api repos/<owner>/reporting-service/forks lists anyone who forked the repo before you did anything — a fork made before a history rewrite keeps the old blob permanently, and there is no command that reaches into someone else's fork and removes it. Whatever you found on this pass, from the very first commit onward, is the exposure window you report and the scope incident response & forensics would want written down, not narrowed after the fact.

Step 3 — rotate: issue the replacement and update every consumer

☺ Like you're 10: A new key does nothing until everyone who was using the old one has actually switched.

Issue the new credential, store it somewhere that isn't a source file, and update every place the old one was configured — in that order, because a replacement nobody's using yet is just an idle key:

# issue a new key pair for the same IAM user
aws iam create-access-key --user-name svc-reporting-export

# store it — HashiCorp Vault, not a source file
vault kv put secret/ci/reporting-export \
  aws_access_key_id=AKIA... \
  aws_secret_access_key=...

# update the CI secret that references it
gh secret set AWS_SECRET_ACCESS_KEY --repo you/reporting-service --body "..."

Consumer	Where the old key lived	What has to change
CI pipeline	GitHub Actions repo secret	`gh secret set` with the new value, then re-run the next job to confirm it authenticates
The reporting-export job itself	`jobs/reporting_export.py`, hardcoded	Rewritten to read from Vault or the environment, matching what `5d0c771` already did for `settings.py` — the fix that never reached this file
Running service	Injected env var at deploy time	Redeploy after the secret store holds the new value, so the running process actually picks it up
Teammates' local `.env` files	Copied by hand at some point, almost certainly	A message in the team channel — you cannot discover these by scanning the repo, only by asking

Once every consumer above is confirmed on the new key — not "should be," confirmed — delete the old one outright rather than leaving it deactivated indefinitely:

aws iam delete-access-key --user-name svc-reporting-export --access-key-id AKIAIOSFODNN7EXAMPLE

Only now is it worth purging the blob from git history — git filter-repo is the actively maintained tool for this (BFG Repo-Cleaner is the older, simpler alternative most teams still reach for first):

# expressions.txt: one line, "literal==>replacement"
echo 'AKIAIOSFODNN7EXAMPLE==>REDACTED-AWS-KEY' > expressions.txt
git filter-repo --replace-text expressions.txt
git push --force --all
git push --force --tags

⚠ History rewriting is cleanup, not the control

By the time you run filter-repo, the key should already be dead — this step exists so a future clone doesn't carry the old value forward and so a secrets scanner stops re-flagging the same historical hit, not because it undoes the exposure. It reaches nothing that was already cloned or forked before you ran it, and it rewrites every commit hash downstream of the change — every collaborator with an existing clone has to re-clone or hard-reset, not pull. Announce that before you force-push, or you'll spend the afternoon untangling merge conflicts nobody expected.

Step 4 — verify the old credential is actually dead

☺ Like you're 10: Don't just believe you changed the lock. Try the old key in the door yourself.

This is the step people skip, and it's the one that turns "I think it's handled" into "I checked." Try to actually authenticate with the old, revoked credential and confirm it fails:

AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE \
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
aws sts get-caller-identity
# expect an auth failure — AWS returns the same InvalidClientTokenId-style error
# for a deleted key and an inactive one, deliberately, so the exact wording isn't
# the point; a successful response is the only result that should worry you here

Then answer the question revocation alone can't: was it actually used while it was live? A dead key going forward doesn't tell you what already happened during the exposure window Step 2 established:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \
  --start-time 2026-08-06T00:00:00Z --end-time 2026-08-17T00:00:00Z

Any event in that window that isn't one of your own is the difference between "we caught a leak" and "we're now handling a confirmed incident" — the second one hands off to incident response & forensics, not this drill. Finally, close the loop the other direction: confirm the new key is the one actually in use — a fresh CI run succeeds, the reporting-export job's next scheduled run shows the new key's access-key ID in its own logs, and nothing anywhere still references the old value. Only once all three checks are clean — old key confirmed dead, exposure window checked for misuse, new key confirmed live — is this drill actually finished.

🎬 At the Shift-Left Squad
🐘Ellie the Elephant: Gitleaks just flagged an AWS key in reporting-service. I already flipped it to Inactive — that took ten seconds. Now I want to know how bad it actually is.
🦊Foxy: Someone already "fixed" it two commits later. Convenient. I don't believe that until I've checked every branch, not just the one file.
🐘Ellie the Elephant: Go ahead and not believe it. I already killed the key regardless — that part doesn't wait on your answer.
🦊Foxy: Good, because there's a second copy. Someone branched before the fix landed and pasted the same key into a different file. The merge carried it right back into main.
🦝Rocky the Raccoon: If I'd found that key first, I wouldn't have cared which file it was in, or whether your commit history looked tidy. I'd have just used it.
🐢Timmy the Turtle: Which is exactly why "we deleted it from the file" was never going to satisfy me. I want the CloudTrail check too — not because I doubt Ellie's timing, because I want proof nobody else used it first.
🦉Professor Owl: Revoke first, scope second, replace third, prove it fourth. Same order every time, no matter which credential it is next time.

✓ Checkpoint

1. Why does revoking have to happen before you've even confirmed how far the exposure goes, rather than after? 2. In the commit graph on this page, why doesn't deleting the key from config/settings.py at 5d0c771 actually remove it from the repository — and where else does it still live? 3. What single command finds every commit, on every branch, that touched one specific secret string, even through a file rename? 4. Why isn't rewriting git history with filter-repo or BFG itself "the fix," and what does it fail to reach? 5. Beyond trying to reuse the old key yourself, what's the second thing "verify it's dead" requires you to check?

Check your answers — and the full worked solution

Because the credential is exploitable for every minute it stays active, and scoping the exposure — checking every branch, every fork, every possible copy — takes real time. Revoking first caps the damage at whatever already happened before you noticed; revoking last means the key stays live through the entire time you spend being thorough about everything else.
Because 5d0c771 only ever touched config/settings.py. The identical key was independently copy-pasted into jobs/reporting_export.py on feature/reporting-export, a branch created from the very first commit — before the "fix" existed — and that branch was merged into main at e88a204. The key is still fully reachable from main's own history; it's just no longer visible if you only check one file's log.
git log --all -p -S"<the exact string>" — the pickaxe search. --all covers every ref, not just HEAD's ancestry, and -S matches on the string itself rather than a file path, so a rename or a second file with the same content doesn't hide it.
A history rewrite only changes what a future clone of the repository will contain. It does nothing to the credential itself, and it can't reach any clone or fork that already exists, or anything a scraper or a cache already captured after the original push. The only action that actually stops the key from working is revoking it at the provider — the rewrite is cleanup that happens after that, not a substitute for it.
Whether it was actually used during the exposure window before you revoked it — checked via the credential provider's own audit trail (AWS CloudTrail's lookup-events for this drill). A key confirmed dead going forward tells you nothing about what already happened while it was live; that's a separate question with a separate answer, and it's the one that decides whether this stays a caught leak or becomes a confirmed incident.

The worked solution, against the exact repo above, in order:

# 1 — REVOKE, immediately, before anything else
aws iam update-access-key --user-name svc-reporting-export \
  --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive

# 2 — SCAN, now that the key is dead and scoping isn't racing a clock
git log --all -p -S"AKIAIOSFODNN7EXAMPLE"
#   -> hit 1: commit 1a77003, config/settings.py (the original introduction)
#   -> hit 2: commit 2a63f19, jobs/reporting_export.py (the branch copy)
gitleaks detect --source . --log-opts="--all" --report-path gitleaks-report.json
gh api repos/you/reporting-service/forks           # check who forked, and when

# 3 — ROTATE, and update every consumer from the table above
aws iam create-access-key --user-name svc-reporting-export
vault kv put secret/ci/reporting-export aws_access_key_id=AKIA... aws_secret_access_key=...
gh secret set AWS_SECRET_ACCESS_KEY --repo you/reporting-service --body "..."
#   -> rewrite jobs/reporting_export.py to read from Vault, same as settings.py already does
#   -> redeploy the running service; confirm the next CI run authenticates
aws iam delete-access-key --user-name svc-reporting-export --access-key-id AKIAIOSFODNN7EXAMPLE
echo 'AKIAIOSFODNN7EXAMPLE==>REDACTED-AWS-KEY' > expressions.txt
git filter-repo --replace-text expressions.txt && git push --force --all

# 4 — VERIFY, don't assume
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
  aws sts get-caller-identity                       # -> must fail
aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \
  --start-time 2026-08-06T00:00:00Z --end-time 2026-08-17T00:00:00Z   # -> check for use before revocation

Same shape, different tool, every time this happens for real: a Stripe key, a GitHub PAT, a database password — revoke, scan, rotate, verify, in that order, never rearranged under pressure. Go deeper on the tools that make Step 2 fast at gitleaks and TruffleHog, and on keeping the replacement out of source files at all at HashiCorp Vault and Workload Identity & Pipeline IAM — the version of this drill where there's no long-lived key to leak in the first place. For the fuller picture of why secrets end up hardcoded to begin with, see Secrets management and Static Analysis & Secrets Detection. Next drill: Reconstruct an Incident Timeline, for when Step 4 above turns up a hit.