Testing in the Pipeline

CI/CD pipelines names a stage called "integration test" and another called "smoke test" and moves on — it assumes you already know what belongs in each box and why. This page is that missing layer: an actual test-automation strategy. We'll walk the test pyramid as an economics argument instead of a shape to memorize, draw an exact line between integration and contract testing (the boundary most teams get wrong), go deep on consumer-driven contract testing specifically — because it's the layer microservices teams skip most often, and the layer whose absence produces the exact outage that finally makes them stop skipping it — and close with the discipline of managing flaky tests without quietly deleting the signal they exist to give you.

The test pyramid: an economics argument, not a shape to worship

☺ Like you're 10: Write lots of cheap, fast checks near the bottom and only a few expensive, slow checks near the top — the shape isn't a rule to obey, it's just what happens when you're honest about what each kind of check costs you.

Mike Cohn's 2009 book Succeeding with Agile drew the original test pyramid as three layers — unit, service, UI — and the shape stuck because it's a compact way to state a real cost curve, not because the exact proportions are sacred. As you move up the pyramid, every layer gets slower to run, more expensive to write and maintain, more prone to nondeterminism, and — this is the part teams forget — less precise about telling you what actually broke. A failing unit test points at one function. A failing end-to-end test points at "something, somewhere, in a system with a database, a queue, and four services, is wrong," and someone now spends an hour finding out what. You'll sometimes see a specific ratio quoted, like 70% unit / 20% integration / 10% end-to-end — treat that as folklore, not law. The number that matters is the direction of the curve: cost and fragility should climb as you go up, and if they don't, something about your test suite is upside down.

Two named failure shapes are worth recognizing because they're both common and both painful in different ways. The ice-cream cone is an inverted pyramid — a thin sliver of unit tests under a heavy scoop of slow, manual, or end-to-end checks — and it's what you get when testing is bolted on after the fact by a separate QA function instead of owned by the engineers writing the code. The hourglass is thin in the middle — plenty of unit tests, plenty of end-to-end tests, almost nothing verifying how services actually talk to each other — and it's brittle at exactly the seam where two teams' code meets, which is not a coincidence: that seam is where contract testing lives, and it's the layer this page spends the most time on below.

A stricter, more rigorous version of the same idea comes from Google's internal practice, described in the book Software Engineering at Google: rather than naming layers by architecture ("unit," "integration"), it names test sizes by measurable properties. A small test runs in a single process, touches no network, no disk beyond tmp, and no real time (no sleep). A medium test may talk to localhost and a real filesystem, but not another machine. A large test may span multiple machines and real external dependencies. The pyramid and Google's sizing are describing the same cost curve from two different angles — one by which architectural boundary a test crosses, the other by which environmental guarantees a test needs — and either one gets you to the same conclusion: push as much of your confidence as you honestly can into the cheap, deterministic layer.

Unit tests: cheap enough to run on every save

☺ Like you're 10: Check one small piece by itself, with nothing real plugged in, so it runs in a blink and tells you exactly which piece broke — not "something, somewhere."

A unit test earns its place in the pyramid's base by being isolated (no network, no disk, no real clock, no shared mutable state with other tests), deterministic (same input, same result, every single time, in any order), and fast — a healthy unit suite runs in seconds even at thousands of tests, specifically so it can run on every save, every pre-commit hook, and every pull request without anyone waiting on it. Isolation is usually bought with test doubles, and the taxonomy from Gerard Meszaros's xUnit Test Patterns is worth knowing precisely rather than calling everything "a mock": a dummy is passed around but never used, a stub returns canned answers, a fake is a lightweight working implementation (an in-memory database standing in for a real one), a spy records how it was called so you can assert on that later, and a mock is pre-programmed with expectations and fails the test itself if those expectations aren't met.

Line or branch coverage percentage — the number a tool like SonarQube puts on a quality gate — is a popular but weak proxy for test quality — a suite can hit 100% coverage while asserting nothing meaningful, simply by executing every line without checking what it produced. Mutation testing is the sharper tool: a tool like Stryker (JavaScript/.NET), PIT (Java, branded "pitest"), or mutmut (Python) automatically injects small bugs into your code — flips a < to a <=, deletes a line, changes a constant — reruns your suite against each mutant, and reports the percentage of injected bugs your tests actually caught. A high mutation score is a much stronger claim than a high coverage number, because it measures whether your assertions do anything, not just whether the code ran.

Integration tests: the murky middle

☺ Like you're 10: Check that your code really works with one real outside thing — a real database, not a pretend one — because a fake stand-in can agree with your code even when the real thing wouldn't.

An integration test verifies that your code correctly talks to something outside its own process — a database, a message queue, the filesystem, another team's API — with as much of the real dependency in the loop as is practical. There are two honest styles, and conflating them is where a lot of pipelines go wrong. A narrow integration test puts your code against a real instance of exactly one dependency at a time; a broad integration test wires several real services together and is, in every cost dimension that matters, closer to an end-to-end test than to a unit test — reach for it sparingly, and know that when you do, you've borrowed the top of the pyramid's problems.

Testcontainers is the standard tool for narrow integration testing today: it spins up a real dependency — Postgres, Kafka, Redis, even a whole service — in a throwaway Docker container for the lifetime of a single test run, then tears it down. You get a real database instead of an in-memory fake, without a shared, always-on environment anyone has to babysit.

import pytest
from testcontainers.postgres import PostgresContainer
from orders.repository import OrderRepository

@pytest.fixture(scope="module")
def db():
    with PostgresContainer("postgres:16-alpine") as pg:
        yield pg.get_connection_url()   # a real, throwaway Postgres — not a mock

def test_order_repository_persists_and_reloads(db):
    repo = OrderRepository(db)
    order_id = repo.save(sku="SKU-4471", qty=2)
    reloaded = repo.get(order_id)
    assert reloaded.sku == "SKU-4471"
    assert reloaded.qty == 2   # a fake in-memory dict would never catch a real column-type mismatch

Contract testing: the piece teams most often skip — and most often regret skipping

☺ Like you're 10: Instead of hooking your code up to your friend's real code just to test it, you both agree in writing exactly what you'll send each other — then each side checks itself against that written agreement, so you catch a broken promise before it ever reaches production.

Here is the specific gap unit and integration tests leave open in a microservices architecture. Your unit tests prove your service is internally correct. Your integration tests prove your service talks correctly to a database or a queue. Neither one proves that your service's assumptions about another team's service still hold — that the OrderService still returns a field called totalCents as an integer, not total as a formatted string, after the team that owns it shipped a "small" refactor on Tuesday. As the number of teams and services scales up, the number of these cross-team seams grows roughly with the square of the team count, and each seam is a place a breaking change can hide in plain sight until it reaches staging — or worse, production.

Consumer-driven contract testing (CDC) closes that specific gap without either side needing the other's runtime available at test time. The consumer team writes a test against a mock of the provider that also records exactly what request it sent and what response shape it expects, as a portable contract file. The provider team then replays those recorded interactions against their real implementation in their own CI — no consumer code involved — and either confirms it still honors every consumer's expectations, or fails loudly before merging a breaking change. Pact is the tool most teams reach for; here's the shape of the workflow end to end.

The mechanics, concretely: the consumer's test suite generates a pact file describing each interaction it depends on, publishes it to a shared broker tagged with the consumer's build version, and the provider's own CI pipeline — on every merge — pulls every pact ever published against it, replays each recorded request against its real, running implementation, and publishes a pass or fail back to the broker per consumer version. The step that actually prevents an outage, rather than just documenting one, is can-i-deploy: a query against the broker that answers one question — does a verified, compatible pact exist between the exact consumer version and exact provider version about to be deployed? Both sides' deploy pipelines run this check as a hard gate before shipping.

{
  "consumer": { "name": "CheckoutWeb" },
  "provider": { "name": "OrderService" },
  "interactions": [
    {
      "description": "a request for an existing order",
      "request": { "method": "GET", "path": "/orders/482" },
      "response": {
        "status": 200,
        "body": { "id": 482, "totalCents": 4599, "status": "CONFIRMED" }
      }
    }
  ]
}

# 1. consumer generates + publishes the pact above:
pact-broker publish ./pacts --consumer-app-version=$GIT_SHA \
  --branch=$GIT_BRANCH --broker-base-url=$PACT_BROKER_URL

# 2. provider CI verifies it against the real OrderService, then:
# 3. both deploy pipelines gate on this before shipping anything:
pact-broker can-i-deploy --pacticipant OrderService --version $GIT_SHA \
  --to-environment production

Contract testing doesn't require a full CDC broker to get started. A lighter, provider-driven alternative is schema-based validation: publish an OpenAPI spec for the provider and use a tool like Dredd or Spring Cloud Contract to check that real responses conform to the published schema. It's cheaper to adopt because only one side has to participate, but the guarantee is weaker — schema conformance proves the shape matches, not that the specific values a consumer actually depends on still behave the way that consumer expects. CDC is stricter because the contract is generated from real consumer expectations, not hand-written by the provider and hoped to be complete.

Given all that, why do teams skip it anyway? It requires both sides to actually participate, someone has to run and maintain a broker, and — this is the real reason — it doesn't feel urgent until the week after a breaking change ships silently and three consumer teams find out in production instead of in CI. The predictable regret pattern that follows is almost always the same: instead of adding the contract layer that would have caught the specific class of bug for the cost of a few seconds per pipeline run, teams pile more end-to-end tests onto the top of the pyramid to try to catch the same cross-service breakage — and the e2e suite balloons, slows down, and turns flaky for exactly the reasons the next two sections cover, all to compensate for a gap one layer down that a contract test would have closed far more cheaply.

End-to-end tests: necessary, so keep the tip thin

☺ Like you're 10: Once in a while, check that the whole finished thing actually works front to back — but only for the handful of trips through it that really matter, because checking every possible path this way is slow and breaks for reasons that have nothing to do with your code.

An end-to-end test exercises a real user journey through the real, deployed system — browser automation with a tool like Playwright, Cypress, or Selenium, or an API-level walk through a live staging deployment. Its flakiness is not a tooling failure to be engineered away; it's structural. Every additional real dependency in the loop — network latency, a third-party sandbox API, browser rendering timing, another team's service having its own bad day — adds a source of nondeterminism the test has no control over. More moving parts mechanically means more nondeterminism, no matter how well the test itself is written.

The strategy that follows from that structural fact is to cover only the small number of critical user journeys end to end — sign-up, login, checkout, the handful of paths that represent the actual value the system exists to deliver — and push every edge case, error path, and permutation down into unit, integration, and contract tests where it's cheaper and more stable to verify. An e2e suite that tries to exhaustively cover the application is both the slowest possible way to get that coverage and the least reliable. It also helps to remember that a thin e2e tip isn't your only safety net after deploy — progressive delivery and a well-chosen deployment strategy catch real production issues on a small slice of live traffic before they reach everyone, which is a second line of defense your test suite doesn't have to shoulder alone.

Ownership matters as much as scope. An e2e suite that belongs to nobody in particular — historically "QA's problem" — rots predictably into an hour-long run that's 30% flaky and that every engineer has learned to re-run until green without reading why it failed. Treat it like production code: it has an owning team, a review process for changes, and a budget for how long it's allowed to take.

Shift-left testing: moving the point of failure earlier

☺ Like you're 10: Catch the mistake on your own computer before you even hand your work in, instead of waiting for someone else to catch it for you three steps later.

"Shift left" means literally moving the point where a test can fail earlier on the pipeline's timeline — reading the pipeline's stage list left to right, shift a check as far left as it can honestly run. The mechanisms are concrete, not just an attitude: pre-commit hooks running the fastest unit tests and linters locally before a commit is even made; pull-request CI running the full unit, narrow-integration, and contract-verification layers before a merge is allowed, so trunk-based development's "always releasable trunk" is actually enforced rather than aspirational; and a local dev environment that runs the same Testcontainers setup CI does, so a developer can reproduce a failing integration test on their own machine in the inner loop instead of waiting on a shared environment to free up.

This page's shift-left is about the functional-test feedback loop specifically. The same principle applied to security scanning — SAST and dependency scanning moved from a late pipeline gate into the editor and the pull request — is its own discipline with its own tooling and tradeoffs; see shift-left security for DevOps for that half of the idea.

Flaky tests: the credibility problem, and the fix that isn't @Skip

☺ Like you're 10: A test that sometimes fails for no reason isn't harmless — it teaches everyone to stop trusting red, and once nobody trusts red, the whole safety net is decoration.

A flaky test is one that passes and fails nondeterministically against the exact same code — no code change required to flip the result. The usual causes are specific and mechanical, not mysterious: test-order dependence and shared mutable state between tests, unpinned real time (Date.now() or a wall-clock sleep instead of a condition-based wait), real network calls to something outside your control, resource leaks or port collisions when tests run in parallel, and genuine race conditions in the async code under test surfacing intermittently rather than every time.

The better process treats flakiness as a signal to track, not a nuisance to silence. Detect it automatically: rerun a newly-failing test a handful of times in the same run, and if it flips outcome with no code change, flag it as a flake candidate rather than either trusting the first fail or blindly retrying to green — pytest-rerunfailures and Maven Surefire's rerunFailingTestsCount do this with visible reporting, and commercial tools like BuildPulse and Gradle's Develocity (formerly Gradle Enterprise) build flakiness scoring across an entire test-run history so a test that's 2% flaky doesn't hide among thousands of green runs. Quarantine it with accountability: move a confirmed flaky test into a suite that still runs and is still visibly dashboarded, but doesn't block merges — with a named owner and an expiry date. It has to be fixed or deleted by that date; quarantine is a holding cell, not a retirement home. Software Engineering at Google describes an internal system built on exactly this shape — automatic flake detection feeding an accountable quarantine process — as one of the few things that kept a codebase with hundreds of thousands of tests trustworthy at all; the specific tooling isn't public, but the practice transfers directly to a team of five.

Where each test type actually lives in the pipeline

☺ Like you're 10: Now put every kind of check back onto the actual assembly line from the CI/CD lesson, in the order that makes sense, and decide out loud which ones are allowed to stop the line.

CI/CD pipelines laid out the generic stage list — lint, unit test, build, integration test, security scan, package, deploy staging, smoke test, deploy prod — without saying which test type belongs in which box or whether it's allowed to block anything. That's the strategy decision this page has been building toward, and it's one every team has to make deliberately rather than inherit by accident.

Every "yes" in that blocks-deploy column is a lever on a specific number in the DORA metrics: change failure rate. A test strategy with the right layer catching the right class of bug at the cheapest possible stage is, in DORA terms, the single biggest thing a team controls about how often a deploy actually breaks something — and testing before deploy still has a ceiling. Beyond it lies chaos engineering, which stops asking "does this pass in staging" and starts asking "does this survive in production," on purpose, before reality asks the same question without warning.