Drill — Fix a Broken Pipeline

Two pipelines, two red builds, one skill: reading a CI failure closely enough to find what's actually wrong instead of what the error message wants you to think is wrong. Both scenarios below are self-contained — no cluster, no cloud account, nothing carried over from the six-part capstone. Clone nothing you'll need again after today; build a five-minute throwaway repo, break it exactly the way described, and get it green using only what the logs tell you. Give yourself 25 minutes per scenario before you open its walkthrough. If you're still stuck at 25, read one paragraph of the walkthrough — not the whole thing — and go back to the terminal.

How this drill works

☺ Like you're 10: Two broken pipelines, no hints until you've actually tried, and a clock running on each one.

Each scenario gives you a real, runnable GitHub Actions workflow that is already broken in a way that isn't obvious from the first error you see. Your job is the same both times: reproduce the failure, read the logs closely enough to find the real root cause (not the first plausible-looking one), fix it, and prove the fix holds by making it fail to fail — reruns, not a single lucky green. Neither scenario depends on the other, and neither depends on anything from the capstone's parcel-api world; that's deliberate; a real on-call rotation hands you one broken pipeline at a time with no continuity either.

Set up the scratch repo

☺ Like you're 10: One tiny throwaway project, just big enough to actually break.

Create one empty repo — you'll point it at Scenario 1's workflow first, then swap in Scenario 2's later. Nothing in here is meant to survive past today:

mkdir pipeline-drill && cd pipeline-drill
git init -b main
npm init -y
npm install express
npm install --save-dev jest
mkdir -p src tests .github/workflows
gh repo create pipeline-drill --private --source=. --remote=origin

Set "scripts": { "test": "jest" } in package.json now — every command below assumes npm test runs Jest.

Scenario 1 — the cache that lied

☺ Like you're 10: The pipeline says a book is missing from the shelf. The shelf is fine. The librarian just never checked it.

You're adding a bulk-discount calculator that leans on decimal.js for exact currency math instead of floating-point arithmetic. Commit the files below, push, and watch a completely reasonable-looking pipeline turn red on you.

// src/discount.js
const Decimal = require("decimal.js");

function applyBulkDiscount(quantity, unitPrice) {
  const total = new Decimal(quantity).times(unitPrice);
  if (quantity > 10) {
    return total.times(0.95).toNumber(); // 5% off once you clear the bulk threshold
  }
  return total.toNumber();
}

module.exports = { applyBulkDiscount };

// tests/discount.test.js
const { applyBulkDiscount } = require("../src/discount");

describe("applyBulkDiscount", () => {
  test("no discount under the 10-unit threshold", () => {
    expect(applyBulkDiscount(9, 100)).toBeCloseTo(900);
  });
  test("5% off at exactly the 10-unit threshold", () => {
    expect(applyBulkDiscount(10, 100)).toBeCloseTo(950);
  });
  test("5% off comfortably over threshold", () => {
    expect(applyBulkDiscount(25, 100)).toBeCloseTo(2375);
  });
});

Run npm install decimal.js so it lands in package.json and package-lock.json, then wire the workflow below — it's plausible, it's the kind of thing a real team ships, and it has already been running green on this repo for a while before today's commit:

# .github/workflows/ci.yml — BROKEN, as given
name: CI
on:
  push:
    branches: ["**"]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"

      - name: Cache node_modules
        id: cache-deps
        uses: actions/cache@v4
        with:
          path: node_modules
          key: ${{ runner.os }}-node-modules

      - name: Install dependencies
        if: steps.cache-deps.outputs.cache-hit != 'true'
        run: npm ci

      - run: npm test -- --ci

Commit everything, push, and let it run once before you add decimal.js — that first green run is what plants the stale cache. Then add the discount files and the decimal.js dependency in a second commit and push again:

echo "console.log('placeholder')" > src/index.js   # anything, just to get one green run first
git add -A && git commit -m "chore: scaffold" && git push -u origin main
gh run watch                                        # wait for it to go green — this plants the cache

npm install decimal.js
git add -A && git commit -m "feat: bulk discount calculator with decimal.js"
git push
gh run watch

The second run fails. Not with a discount-math error — with this:

FAIL tests/discount.test.js
  ● Test suite failed to run

    Cannot find module 'decimal.js' from 'src/discount.js'

      1 | const Decimal = require("decimal.js");
        |                 ^

decimal.js is correctly listed in both package.json and the committed package-lock.json — git show HEAD:package.json | grep decimal proves it. The dependency is not missing from your repo. It's missing from the machine that ran the test. Find out why before you touch a single line of application code.

Diagnose it exactly the way you would a real incident — from the run's own evidence, not a guess:

gh run view --log-failed          # look at the failed step's full output
# ...scroll up past the FAIL block to the "Install dependencies" step — it says "Skipped"

gh cache list
# ID       KEY                  SIZE    CREATED         ACCESSED
# 1        Linux-node-modules   41 MB   18 minutes ago  18 minutes ago

"Install dependencies" shows Skipped, not "0 packages installed" or any kind of error — skipped, because cache-hit reported true. And gh cache list shows exactly one cache entry under the exact literal key from the workflow file, Linux-node-modules, with no lockfile hash anywhere in it. That key matches itself on every single run, forever, regardless of what package-lock.json says — which is the entire bug. cache-hit only reports true on an exact key match; it's correctly, precisely doing what a static key tells it to do.

Fix the key, delete the poisoned cache entry so today's run doesn't reuse it one more time, and push:

# .github/workflows/ci.yml — the fix
      - name: Cache node_modules
        id: cache-deps
        uses: actions/cache@v4
        with:
          path: node_modules
          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-node-

gh cache delete 1                 # or: gh cache list --json id -q '.[].id' | xargs -n1 gh cache delete
git add -A && git commit -m "fix: hash the lockfile into the node_modules cache key"
git push
gh run watch

This run still fails — and that's the point, not a sign you broke something new. "Install dependencies" now shows real npm ci output instead of Skipped, so decimal.js installs correctly and the module-not-found error is gone. What's left is the failure that was there all along, finally visible on its own:

FAIL tests/discount.test.js
  ● applyBulkDiscount › 5% off at exactly the 10-unit threshold

    expect(received).toBeCloseTo(expected)

    Expected: 950
    Received: 1000

       6 |   test("5% off at exactly the 10-unit threshold", () => {
    >  7 |     expect(applyBulkDiscount(10, 100)).toBeCloseTo(950);
         |                                         ^

quantity > 10 excludes exactly 10 units from the discount the test (correctly) expects. Fix the boundary, push, go green:

  if (quantity >= 10) {   // was: quantity > 10

Done when: gh run view --log-failed on your latest run returns nothing to show, because there's nothing failed — three green tests, an "Install dependencies" step that actually ran, and a cache key you can point to and explain in one sentence.

Scenario 2 — the door two shards raced through

☺ Like you're 10: Split the work in two to go faster, but only if the two halves aren't reaching for the same thing at the same time.

Speed up the integration suite by sharding it across two parallel matrix jobs — a completely reasonable, widely-used move. The bug isn't in the sharding. It's in what both shards were quietly pointed at underneath it. Swap in this workflow (a fresh ci.yml, don't worry about Scenario 1's leftovers):

# .github/workflows/ci.yml — BROKEN, as given
name: CI
on:
  push:
    branches: ["**"]
jobs:
  integration:
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2]
    runs-on: ubuntu-latest
    env:
      DATABASE_URL: ${{ secrets.STAGING_DATABASE_URL }}   # one shared, always-on DB
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: "npm"
      - run: npm ci
      - run: npx jest --shard=${{ matrix.shard }}/2 --ci

Someone pointed both shards at the team's one shared staging Postgres via a repo secret, instead of giving each job its own throwaway database — reasonable enough the first time anyone wrote it, since it "just worked" for months. Two integration test files, each copy-pasted from the other and each inserting the same hardcoded fixture row:

// tests/customers.integration.test.js
const { Client } = require("pg");

test("creates a customer record", async () => {
  const client = new Client({ connectionString: process.env.DATABASE_URL });
  await client.connect();
  await client.query(
    "INSERT INTO customers (email, name) VALUES ($1, $2)",
    ["test.user@example.com", "Test User"]
  );
  const { rows } = await client.query(
    "SELECT * FROM customers WHERE email = $1", ["test.user@example.com"]
  );
  expect(rows).toHaveLength(1);
  await client.query("DELETE FROM customers WHERE email = $1", ["test.user@example.com"]);
  await client.end();
});

// tests/billing.integration.test.js — nearly identical, same fixture email
const { Client } = require("pg");

test("attaches a billing profile to a customer", async () => {
  const client = new Client({ connectionString: process.env.DATABASE_URL });
  await client.connect();
  await client.query(
    "INSERT INTO customers (email, name) VALUES ($1, $2)",
    ["test.user@example.com", "Billing Test User"]
  );
  // ...billing assertions omitted...
  await client.query("DELETE FROM customers WHERE email = $1", ["test.user@example.com"]);
  await client.end();
});

Jest's default sharding splits by test file, so customers.integration.test.js and billing.integration.test.js usually land on different shards — different matrix jobs, different runners, running at the same time, both INSERTing the identical email into the identical shared table. Push it a handful of times and watch the failure move around:

error: duplicate key value violates unique constraint "customers_email_key"
    at Client._handleErrorMessage ...

The tell isn't the error text — it's which job fails. Run it several times and check:

gh run rerun --failed          # rerun just the failed jobs, several times
gh run view --log-failed       # sometimes shard 1 fails, sometimes shard 2, never both, never neither

A logic bug fails the same way every time. This one moves — that's your signal that two processes, not one function, are fighting over one resource. Reproduce it locally without waiting on GitHub's queue at all, by racing the two shards against each other yourself:

docker run -d --name pg-drill -e POSTGRES_PASSWORD=test -e POSTGRES_DB=app_test -p 5432:5432 postgres:16
# create the customers table once, with the unique constraint the app expects:
docker exec -i pg-drill psql -U postgres -d app_test -c \
  "CREATE TABLE customers (id serial PRIMARY KEY, email text UNIQUE, name text);"

export DATABASE_URL=postgres://postgres:test@localhost:5432/app_test
npx jest tests/customers.integration.test.js & npx jest tests/billing.integration.test.js & wait

Run that last line a handful of times — it won't fail every time, which is exactly consistent with a race: sometimes one insert-then-delete finishes cleanly before the other starts, sometimes they overlap and one loses. Fix it by giving each matrix job its own ephemeral database instead of sharing one — GitHub Actions spins a fresh services: container per job automatically:

# .github/workflows/ci.yml — the fix
jobs:
  integration:
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2]
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: test
          POSTGRES_DB: app_test
        ports: ["5432:5432"]
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    env:
      DATABASE_URL: postgres://postgres:test@localhost:5432/app_test
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: "npm"
      - run: npm ci
      - run: npx jest --shard=${{ matrix.shard }}/2 --ci

Done when: gh run rerun --failed stops having anything to rerun — five, ten pushes in a row, both shards green every time, not just the one you happened to be looking at.

Both scenarios, one transferable habit

☺ Like you're 10: Before you fix the code, ask whether the code you're looking at is even the code that actually ran.

Neither bug today lived in application logic on first read. Scenario 1's first, loudest error was a missing-module message that had nothing to do with the actual bug two commits deep in discount.js. Scenario 2's error text — a unique-constraint violation — was completely accurate and still pointed you at the wrong layer if you went looking for it in the SQL or the test assertions instead of the workflow's services: block. The move that cracks both: before debugging the code, ask whether the thing that ran is the code and dependencies you think it is, and whether it ran alone. A skipped install step and a job whose failure moves between reruns are two different answers to the same question.

Both fixed? Good — that's the whole drill. For the deeper concepts behind each bug, see Testing in the Pipeline and Build & Artifact Management; for how caching and sharding change once dozens of teams share one set of runners, see Scaling CI/CD Across Teams. GitHub Actions covers the platform this drill ran on end to end. Ready for a different single skill? Try Drill — Secure a Vulnerable Pipeline, or step back to Ship It — Start Here for the six-part continuity version.