Release engineering & progressive delivery
A release is a controlled experiment on production traffic, whether you design it that way or not. Progressive delivery is what happens when you design it that way on purpose: ship to a small cohort, let automation watch the signals that actually matter, and let that same automation decide whether to promote or roll back — instead of a human eyeballing a dashboard for twenty minutes and hoping. By the end of this page you should be able to describe the full canary-to-100% pipeline, explain why it depends on a reproducible build, and know when a release train is the better fit instead.
A restaurant testing a new dish doesn't print it on every menu in the dining room at once. It puts the dish on three tables' menus first, has a server quietly watch how much of it comes back uneaten and whether anyone complains, and only then decides: print it everywhere, or pull it before more kitchens are stocked for it. Progressive delivery is that same test applied to software — a new version goes to a small slice of traffic first, an automated watcher checks whether anything is coming back "uneaten" (errors, latency, burned error budget), and only a version that passes gets handed the rest of the room.
The three pieces, tied into one mechanism
Progressive delivery isn't a single new technique — it's three existing techniques wired together so none of them requires a human in the loop for the common case. On their own, each piece is limited: a canary release without automated analysis is just a smaller blast radius, still watched by a person; a feature flag without a kill switch tied to health signals is just a manual toggle; an automated rollback without a known-good target to roll back to isn't reliably automatable at all. Combined, they form one pipeline:
- Canary release — the new version runs alongside the old one, receiving a deliberately small, controlled slice of real traffic (often selected by request header, user cohort, or a hash of session ID rather than pure randomness, so the same users stay on the same version for the duration).
- Feature flags — a runtime switch that decouples deploying code from activating it. The new binary can be on every host already; the flag is what actually exposes the new path to a percentage of traffic, and flipping it back is a config change, not a redeploy.
- Automated rollback — the piece that removes the human from the loop on the unhappy path. A controller watches the canary's metrics against the old version's baseline and reverts automatically the moment the comparison crosses a threshold, without waiting for someone to notice and act.
Tools like Argo Rollouts and Flagger implement this pattern natively on Kubernetes: you declare a canary strategy (traffic steps, analysis queries, thresholds) as a resource, and the controller drives traffic shifting and rollback itself. See reliability patterns for where canary and blue-green deployment sit among the broader set of resilience patterns this one specializes.
The health check gate: what "automatically watch" actually means
The gate between traffic steps is not a person refreshing a dashboard — it's a query against the same signals covered in monitoring and observability, evaluated on a schedule against an explicit pass/fail threshold. A typical gate checks two kinds of thing at once, and conflating them is a common design mistake:
- Golden-signal deltas — is the canary's error rate, p99 latency, or saturation meaningfully worse than the baseline (the currently-running stable version), not just worse than some fixed number in isolation. Comparing canary against a live baseline, not a static threshold, is what lets the same gate stay correct through a normal daily traffic swing that would otherwise trip a fixed alert.
- SLO burn rate — is the canary's cohort burning its share of the error budget faster than the rate that would exhaust the whole 28-day budget before the window resets. A burn-rate gate catches a canary that's technically "passing" a raw error-rate threshold but consuming budget at 10x the sustainable pace — the same multi-window, multi-burn-rate technique used for paging decisions, applied here to a promotion decision instead.
If the gate passes for the configured bake time (commonly 5-15 minutes, long enough to catch a slow leak or a cache-warming artifact, short enough to keep the pipeline moving), traffic promotes to the next step. If it fails at any step, the controller rolls back to the last known-good version and halts — it does not "try again with less traffic," which would just repeat the same failing comparison. A well-run rollback also pages someone with the specific metric and threshold that tripped, so the failure is diagnosable, not just reverted.
Why rollback needs a reproducible build first
Automated rollback is only as trustworthy as the thing it rolls back to. If "roll back" means recompiling the previous git tag on demand, you've reintroduced exactly the risk progressive delivery exists to remove: a dependency resolver could pick up a different transitive version than it did last time, a base image could have been repatched, or a compiler flag could differ — any of which means the "known-good" version you roll back to isn't actually the one that was known good. This is why progressive delivery presumes the build-once-promote-everywhere discipline covered in the DevOps course's build and artifact management material: an artifact built once, hashed, and stored immutably in a registry, so that rollback means redeploying an exact, previously-verified digest rather than rebuilding anything. A rollback that redeploys a known artifact takes seconds; a rollback that reruns a build pipeline takes as long as the pipeline does, and re-exposes every failure mode a hermetic build was supposed to close off.
Practically, this means a canary and its baseline should reference the artifact by immutable digest (checkout-svc@sha256:...), not by a mutable tag like :latest or even :stable — a tag can be repointed underneath a rollout, but a digest cannot. The rollback target a controller stores at the start of a rollout should be that exact digest, captured before the canary starts receiving traffic.
Automated rollback removes a human from the promote/revert decision, not from owning the outcome. A team that wires up canary analysis and then stops watching entirely can end up with a rollback loop — a bad build that fails the gate, gets reverted, gets redeployed by CI on the next commit, fails again — burning error budget the whole time without ever paging anyone, because "the automation handled it." A rollback should always page or at minimum log loudly enough that a human confirms why the release failed before the next attempt goes out.
Release trains: batching instead of streaming
Progressive delivery assumes each change can ship the moment it's ready, gated only by automated health checks. A release train inverts that: changes accumulate against a fixed cadence — weekly, biweekly, every two weeks on a Tuesday — and whatever has merged and passed CI by the cutoff ships together, on schedule, whether or not any individual change is "urgent." Anything that misses the cutoff waits for the next train rather than forcing an off-cycle release. Mozilla's Firefox release trains and many mobile app release cycles (bound by app-store review latency) are the canonical examples: cadence is the point, not a side effect.
The two models trade different things. Progressive delivery minimizes blast radius per change and shortens time-to-detect for a bad release, at the cost of more moving release infrastructure and constant background rollout activity. Release trains minimize release-process overhead and give downstream teams (QA, docs, customers on a regulated update schedule) a predictable rhythm to plan around, at the cost of batching multiple changes together — when a train breaks something, isolating which of a dozen batched changes caused it is strictly harder than isolating one canary. The two are not mutually exclusive: a mature setup often runs a release train as the outer cadence (what ships this week) with progressive delivery as the inner mechanism (how each of this week's changes gets rolled out safely once it's in the train) — batching decides when, progressive delivery decides how safely.
What this buys an SRE team
The operational payoff is the one named in this page's lead: less human babysitting per release. Without automated gates, "watch the canary" is a task on someone's plate for the 15-30 minutes after every deploy, competing with everything else on-call is doing, and it degrades under load exactly when it matters most — a team shipping ten releases a day cannot have a human stare at each one. With automated gates, the human is paged only when the automation's own decision needs review: a rollback that fired, or a promotion that's ambiguous enough to need judgment. That's the same symptom-based-alerting philosophy from monitoring and observability applied to releases instead of steady-state operation — page on the outcome that needs a human, not on every step of a process that mostly doesn't.
It also changes the cost-benefit of shipping small. Because a bad canary is caught and reverted automatically within one bake-time window instead of riding on 100% of traffic until someone notices, teams can ship smaller, more frequent changes without a proportional increase in incident risk — the exact trade a release train's batching model gives up in exchange for a predictable cadence.
Progressive delivery doesn't reduce the chance that a bad release happens — it reduces the blast radius and detection time when one does. A canary at 1% traffic that fails still fails; the win is that 1% of users saw it for one bake-time window instead of 100% of users seeing it until a human noticed. Treat the mechanism as a faster, smaller failure detector, not a substitute for testing before the canary starts.
1. What are the three components progressive delivery combines, and what does each one contribute that the others can't provide alone? 2. Name the two categories of check a health-check gate typically evaluates, and why comparing against a live baseline matters more than a fixed threshold. 3. Why does automated rollback require a reproducible, immutably-addressed build, and what specifically goes wrong if rollback means rebuilding from source? 4. What trade-off does a release train make relative to continuous progressive delivery, and how can the two models be combined?
Check your answers
- Canary release (a small, controlled traffic slice for the new version), feature flags (a runtime switch that decouples deploy from activation), and automated rollback (a controller that reverts on a failed health check without waiting for a human). Alone, a canary is still human-watched, a flag is a manual toggle, and rollback has no reliable known-good target to revert to.
- Golden-signal deltas (error rate, latency, saturation compared against the currently-running baseline) and SLO burn rate (whether the canary cohort is consuming error budget faster than the window can sustain). A live baseline matters because a fixed threshold can't distinguish a real regression from normal daily traffic variation.
- Because "known-good" only means anything if the artifact rolled back to is provably the exact one that was previously verified. Rebuilding from source on rollback can silently pull a different transitive dependency version, a repatched base image, or a different compiler flag, so the "known-good" rebuild may not match what was actually good — the fix is redeploying an immutable, digest-addressed artifact instead of recompiling.
- A release train trades per-change agility and small blast radius for a predictable, lower-overhead cadence that downstream teams can plan around, at the cost of batching multiple changes together, which makes isolating a bad change harder when the train breaks something. The two combine by using the train as the outer release cadence and progressive delivery as the inner mechanism for rolling out each change in that train safely.