Practice & Reference · Case study: securing a pipeline

Case study: securing a pipeline

Ledgerly is a fictional Series A fintech startup — eleven engineers, a Node.js/Postgres monolith, and a payments-reconciliation API that a handful of small-business accounting tools integrate against. This is the story of how one committed API key turned into a five-figure incident, and how the team spent the following six weeks turning nearly every concept covered earlier in this course into something running in their actual pipeline.

The starting point

Ledgerly's deploy process in early June looked like most eleven-person startups': a shared AWS account, a Node.js backend read its database password and a Stripe-adjacent payments-processor API key from a .env file, and that .env file was regenerated from a template whenever someone set up a new laptop. On June 2, an engineer fixing a broken local setup script accidentally committed a real .env file — not the .env.example template — to a feature branch, and pushed it to origin. The repository was private, but a second, older mirror of it, created eight months earlier for a now-abandoned open-source SDK experiment, was still public. GitHub's automated secret scanning on public repos flagged the payments-processor key within eleven minutes of the push. So did an opportunistic scraper: automated bots that continuously crawl public GitHub for credential-shaped strings are a documented, active threat, not a hypothetical one, and in this case the scraper won the race. The key was used to pull transaction metadata for 340 merchant accounts before Ledgerly's own key-rotation alert fired four hours later.

The key itself only had read access to transaction metadata, not funds-movement — Ledgerly's payments processor scoped API keys by capability, which limited the blast radius. But metadata included merchant names, bank routing numbers (last four digits), and transaction volumes, which was enough for the attacker to run a convincing invoice-fraud campaign against nine of the exposed merchants over the following week. Ledgerly's post-incident accounting put the direct cost — processor-mandated forensic review, fraud reimbursement to three affected merchants who lost money before catching the fake invoices, and the engineering time spent on containment — at just under $60,000, plus two merchants who churned during the following quarter citing the incident directly.

Discovery and containment

The four-hour gap between the leak and Ledgerly's own detection is the part of this story that maps most directly onto incident response and forensics: GitHub's scanner caught it faster than Ledgerly did, because Ledgerly had no equivalent scanning of its own, on public mirrors or otherwise. Once the on-call engineer got the alert, the response followed the shape any incident runbook should: identify what the key could access, revoke it at the processor before doing anything else, then reconstruct the timeline from processor-side access logs to figure out which merchants were actually touched. Ledgerly's postmortem — written up two days later, blameless by policy, and shared with the whole engineering team — named the root cause plainly: a single long-lived credential, with no expiration and no scanning between "committed" and "exploited," was the entire failure. Everything the team built over the following six weeks was aimed at making sure no single mistake like that one could reach production again.

Weeks 1-2: a centralized secrets manager and short-lived credentials

The first workstream, started the same week as the postmortem, was the one aimed directly at the root cause: no more secrets living in files that could be committed at all. Ledgerly stood up AWS Secrets Manager for anything already in AWS and adopted HashiCorp Vault for the payments-processor key and two other third-party API credentials, following the same pattern covered on secrets management — the application no longer reads a value from disk or an environment file checked into the repo; it authenticates to the secrets manager at container start via an IAM role and requests what it needs.

The more consequential change was pairing that with short-lived credentials instead of just moving the same static keys into a nicer box. Ledgerly's payments processor supported scoped, expiring API tokens with a configurable TTL, and the pipeline was changed to request a fresh token with a 60-minute TTL at deploy time rather than provisioning one long-lived key that lived for the service's entire lifetime. Database credentials moved to Vault's dynamic secrets engine, generating a new scoped Postgres user per deploy that expires automatically. The math the team put in the rollout doc was the same one from the secrets management page: a leaked static key is valid until someone notices and revokes it — four hours, in their own case — while a leaked 60-minute token is worthless to an attacker well before most humans would even see the alert.

Weeks 2-4: pre-merge scanning so the same class of bug can't ship

In parallel, Ledgerly added scanning gates to the CI pipeline, following the shift-left approach from security in CI/CD and the tool breakdown on SAST, DAST & SCA. Two gates went in as required checks blocking merge, not warnings someone could click past: a secret-scanning step (Gitleaks) that would have caught the original .env commit before it ever left the developer's machine, and an SCA step (Trivy against the dependency manifest) after the team discovered, during the broader audit that followed the incident, that a transitive dependency in the payments-reconciliation service had a known-critical deserialization CVE that had gone unnoticed for five months. A SAST gate (Semgrep, using the default ruleset plus one custom rule specifically matching hardcoded-credential patterns) went in the same week.

The rollout was deliberately staged over two weeks: gates ran in report-only mode against the existing codebase first so the team could triage the backlog of findings — mostly SCA noise from transitive dependencies nobody called directly — before flipping them to build-blocking. Turning on a hard-fail gate against an untriaged backlog on day one would have blocked every open pull request and taught engineers to route around the tool, which is exactly the failure mode called out on the SAST/DAST/SCA page.

Weeks 4-5: IaC scanning for the misconfiguration that made it worse

The incident review surfaced a second finding that had nothing to do with the leaked key directly: the S3 bucket storing Ledgerly's transaction-export logs, defined in Terraform, had a misconfigured bucket policy that made it unintentionally readable without authentication — a leftover from an early debugging session that was never reverted. It wasn't the initial entry point, but it meant that once the attacker had working credentials, reconnaissance against Ledgerly's broader AWS footprint was easier than it should have been, because part of that footprint was already exposed. That gap maps directly onto IaC security and policy as code: the same category of misconfiguration policy-as-code is built to catch before it's ever applied.

Ledgerly added Checkov as a required CI check against every Terraform pull request, with a hard-fail policy on public-storage and public-security-group findings specifically — the categories directly relevant to what had gone wrong — and a report-only tier for lower-severity findings, using the same staged-rollout logic as the SAST/SCA gates. The public bucket policy was fixed within the first day of scanning going live; it was the first finding the tool produced.

Week 6: a security champions program so the lesson doesn't stay with one team

The last piece addressed a gap the postmortem named honestly: the engineer who committed the .env file wasn't careless by the team's standards, they simply hadn't been part of the conversations where secrets handling had ever come up, because Ledgerly had never had one. Tooling gates catch mistakes at the pipeline level, but they don't build the judgment that prevents a developer from reaching for the wrong pattern in the first place two teams over. Following security culture and champions, Ledgerly named one security champion per squad (three total, at their size) — not a new security team, but an existing engineer on each squad given dedicated time each sprint to review that squad's security-relevant pull requests, triage that squad's share of scanner findings, and act as the first point of contact before something needs to become a full incident review.

The champions program is also what turned this specific incident into a durable practice rather than a one-time fire drill: the three champions co-wrote Ledgerly's first internal secure-coding checklist, modeled on the pattern in this course's own DevSecOps checklist, and walked it through every squad's next sprint planning session.

Before and after

Six weeks after the incident, Ledgerly's engineering lead summarized the shift in risk posture for the board in four comparisons. Before: secrets lived in files that could be committed by mistake, with no automated check to catch it. After: no application secret exists outside the secrets manager, and anything committed by accident is caught by a required pre-merge scan rather than by an outside party. Before: a vulnerable dependency could sit unnoticed for months. After: SCA runs on every pull request and fails the build on critical, reachable findings. Before: Terraform changes applied directly to AWS with no policy check. After: Checkov blocks public-storage and public-security-group misconfigurations before terraform apply ever runs. Before: secure coding knowledge lived in whichever engineer happened to have picked it up. After: three named champions with dedicated review time and a shared checklist. None of these controls make Ledgerly unbreachable — no pipeline is — but each one closes the specific gap that let one committed file become a $60,000 incident.

Check your answers

1. The root cause was a single long-lived credential with no expiration and no scanning between being committed and being exploited — rotating that one key would have addressed that one leak, but not the underlying pattern of static secrets in files with no detection layer, which is why Ledgerly rebuilt around a secrets manager, short-lived credentials, and scanning rather than just revoking the compromised key.
2. Turning on a hard-fail gate against an untriaged backlog of existing findings would have blocked every open pull request at once and taught engineers to bypass or ignore the tool; running in report-only mode first let the team clear the existing backlog before the gate started blocking new work.
3. It wasn't the entry point, but once the attacker had working credentials from the leaked key, the exposed bucket made reconnaissance against Ledgerly's broader AWS footprint easier, because part of that footprint was already unintentionally accessible without authentication.
4. Tooling gates catch mistakes at the pipeline level but don't build the judgment that prevents a developer from reaching for an unsafe pattern in the first place; the champions program puts a trained point of contact on every squad so security knowledge and review capacity aren't concentrated in one team or one person.