Feature Flags & Progressive Delivery

Deployment strategies decide which infrastructure serves a request — rolling, blue-green, or canary all move traffic between versions of already-deployed code. A feature flag operates one layer up: it decides which code path runs once a request has already landed on an instance, with a config lookup instead of an infrastructure change. That page introduced flags as the mechanism that decouples deploy from release; this page goes past the introduction and into the machinery — the taxonomy of flag types and who should own each one, how a dark launch and a sticky percentage rollout actually route a specific user, the evaluation architecture that makes a flag check cheap enough to run on every request without becoming a new single point of failure, why a flag flip is the fastest rollback you own, and the governance discipline that keeps a flagging system from quietly becoming its own form of technical debt.

Two clocks: deploying and releasing on separate schedules

☺ Like you're 10: "The code is running" and "people can see it" used to be the same moment. Flags split them into two separate switches you can flip on totally different days.

Deploy means new code is running in production. Release means a user can actually reach and use a given capability. Without flags, an organization is structurally forced to make those the same event — the moment a build reaches production, whatever it contains is live for everyone, which is exactly the pressure that historically pushed teams toward big, infrequent, high-ceremony releases: batch enough changes together and the release becomes a project with a date, a checklist, and a room full of people watching dashboards. A feature flag breaks that coupling by letting a code path exist in production, exercised by real requests, while a config value keeps it dark. The build that deploys it and the config change that releases it can now be separated by minutes, or by a full fiscal quarter, and neither one has to know about the other's schedule.

This is what makes elite delivery performance in the DORA sense — see measuring success: the DORA metrics — compatible with features that take months to build. Deploy frequency measures how often code reaches production, not how often users see something new; a team can deploy dozens of times a day while a single large feature ships dark behind a flag across all of them, then release on a date tied to a marketing launch, a contractual commitment, or nothing more than "when it's ready," without that date ever touching the pipeline. The schematic below makes the split literal: the deploy that first shipped a piece of code and the release that finally exposed it can sit weeks apart on two independent timelines.

A taxonomy of flags: not every toggle behaves the same

☺ Like you're 10: A light switch you'll rip out next week and a breaker panel you'll keep forever are both "switches," but you'd never treat them the same way — flags split the same way.

Treating every conditional in the codebase as "a feature flag" is how flag systems rot. The useful split runs along two independent axes: longevity — how long should this flag exist before someone deletes it — and dynamism — does flipping it require a redeploy, or can it change at runtime with no build involved. Cross those two axes and four archetypes fall out, each with a different owner and a different reason to exist.

The taxonomy comes from the same distinction Pete Hodgson drew in Martin Fowler's engineering writing on the subject: flags aren't one thing, and a system that stores all four types in one undifferentiated table with no lifecycle metadata is how a codebase quietly accumulates hundreds of flags nobody remembers the other branch of. Release and experiment toggles have a built-in expiry the moment you understand what they're for; ops and permission toggles don't, and that's fine — the mistake isn't a long-lived flag, it's a long-lived flag with no owner and no record of why it's still there.

Dark launches and sticky percentage rollouts: the routing mechanics

☺ Like you're 10: A dark launch runs the new code in secret and throws the answer away just to see if it would have worked; a percentage rollout is the same trick as picking birthdays out of a hat, except the hat remembers your birthday so you don't get picked twice.

A dark launch ships a code path to 100% of production instances, running against real traffic, while keeping its output completely invisible to users — sometimes by literally executing the new logic in shadow mode: compute the result, log or compare it against the old logic's answer, and discard the response without ever serving it. This is how you load-test a new recommendation engine, a rewritten pricing calculation, or a new database query pattern against genuine production traffic shape and volume before a single real user is exposed to its output — a shadow-mode bug shows up in a diff log, not an incident. The practice is widely credited to Facebook's engineering team, who dark-launched chat infrastructure years before turning it on, specifically to find out whether the backend would survive real load before anyone could tell it existed.

A percentage rollout is the more familiar cousin — expose a feature to some slice of users and grow that slice over time — but the naive implementation is broken: rolling a fresh random number on every request means the same user can flip in and out of a feature from one page load to the next, which is both a jarring user experience and poison for any experiment relying on a stable cohort. The fix is sticky bucketing via consistent hashing: hash a stable key — usually the flag's key concatenated with the user's ID — into a deterministic number, and compare that number against the rollout percentage. The same user always lands in the same bucket for a given flag, every time, on every instance, with no shared state or database lookup required.

# Sticky bucketing: same user, same flag → always the same bucket.
# No database, no shared state — pure function of (flag_key, user_id).

def is_enabled(flag_key: str, user_id: str, rollout_percent: int) -> bool:
    seed = f"{flag_key}:{user_id}".encode("utf-8")
    bucket = zlib.crc32(seed) % 100          # deterministic, 0-99
    return bucket < rollout_percent

# Rolling 10% -> 25% only ever ADDS users to the "on" bucket — nobody who
# was already on at 10% ever gets bumped back off as the percentage grows,
# because the bucket a user lands in never changes for this flag_key.

Real systems layer targeting rules on top of the bucket function, evaluated top-down with the first match winning: an individual-user override (a specific account ID, for support or QA) beats a segment rule (plan = "enterprise", country in ["US","CA"], or a named beta-tester list) which beats the percentage rollout, which falls through to a hardcoded default if nothing else matches. Rule order is not cosmetic — a segment rule placed after the percentage rollout in evaluation order can silently never fire, because the percentage check already returned an answer for that request.

{
  "flag": "new-checkout-flow",
  "rules": [
    { "if": { "userId": "in", "value": ["u_412", "u_918"] }, "serve": true, "note": "internal QA override" },
    { "if": { "attr": "plan", "op": "eq", "value": "enterprise" }, "serve": true, "note": "enterprise gets it first" },
    { "if": { "attr": "country", "op": "in", "value": ["US", "CA"] }, "rollout": 25, "note": "sticky 25% of NA" }
  ],
  "default": false
}

Evaluation architecture: making a flag check cheap enough to run everywhere

☺ Like you're 10: The rulebook lives in one office, but every checkout counter keeps its own copy so it never has to call the office mid-sale — and if the phone line to the office ever goes dead, every counter already knows what to do by default.

A flagging system splits into a control plane — the dashboard and API where rules, rollout percentages, and kill switches are authored and stored — and a data plane: the SDK running inside your application that actually answers is_enabled("new-checkout", user) on the request path, potentially thousands of times a second. How the data plane gets its rules is the architectural decision that matters most, because a flag check that can fail a request is a worse outage than the one it was meant to prevent.

Two delivery models move rules from control plane to data plane. Polling has the SDK fetch the current ruleset on a fixed interval — simple, but every change waits out the polling window before it's live everywhere, and a short interval means a lot of near-identical HTTP calls. Streaming pushes changes over a long-lived connection (server-sent events or a websocket) the moment they happen, so a kill-switch flip propagates to every connected instance in roughly the time it takes to open a TCP packet, not the length of a polling window. Independently, evaluation itself can happen two ways: local evaluation holds the full ruleset in memory and runs the hash-and-match logic in-process — sub-millisecond, no network call per check — versus remote evaluation, where the SDK calls out to the flag service for every single check. Remote evaluation is simpler to build but turns the flag service into a hard dependency of every request that checks a flag, which is precisely the fragility this whole system exists to avoid; production-grade SDKs default to local evaluation with streaming updates for exactly that reason.

Browser and mobile clients need a third adjustment. A client-side SDK cannot hold the full ruleset the way a server-side SDK does — shipping raw targeting rules to a browser leaks the names of unreleased features, the exact rollout percentage, and which segment a given account belongs to, all readable in a network tab. Vendors solve this by resolving flags server-side into a small, per-user bootstrap payload — just the booleans and variants this specific user should see — and shipping only that to the client, often through a lightweight relay or edge proxy that also cuts down the number of long-lived connections a large fleet would otherwise open directly against the vendor's control plane (a real concern once you're running hundreds of pods on Kubernetes, each wanting its own stream).

Kill switches: the fastest rollback you own

☺ Like you're 10: Rolling back is calling the fire truck; a kill switch is a fire extinguisher already sitting on the wall — same fire, but one of them is already in the room.

A deployment rollback — even blue-green's near-instant router swap covered in deployment strategies — still involves an infrastructure action: a traffic-routing change, a previous artifact being re-promoted, or, for a rolling deployment, an entire reverse rollout run instance by instance. A kill switch skips all of that. It's the same running binary, the same deployed version, just re-reading a config value it was already checking on every request — once the evaluation architecture above has propagated the change, the flip takes effect in the time it takes a stream message to arrive, typically well under a second, with nothing rebuilt and nothing redeployed.

The discipline this enables is wrapping risk before it becomes an incident, not after. Any new, expensive, or third-party-dependent call — a new recommendation-engine lookup, an added payment-processor integration, a freshly introduced cache layer — should ship inside an ops toggle from day one, with a documented owner, a one-line runbook entry ("flipping this off stops X, expect Y to happen instead"), and a monitoring signal that confirms the flip actually worked. Built this way, the kill switch is a pre-positioned lever an on-call engineer reaches for during incident management, not something improvised under pressure at 2 a.m. — the difference between "I know exactly which flag to flip" and "let's try reverting the last three deploys and see which one fixes it."

Flags and true trunk-based development

☺ Like you're 10: A short branch keeps the mess small; a flag is what lets you merge a half-built room into the main house without anyone accidentally wandering into it.

Version control & branching already establishes that trunk-based development depends on flags to keep incomplete work invisible in main — the deeper practice is flag-per-PR discipline: for any change that isn't trivially safe, every individual pull request lands behind its own flag, so merge cadence and user-visible change become fully decoupled variables. A multi-week feature stops being one giant, terrifying merge at the end and becomes dozens of small, individually reviewable, individually revertible merges, each dark until the whole thing is ready to switch on together.

The cost is combinatorial: N independent flags imply up to 2^N possible code-path combinations, and no team tests all of them. The pragmatic approach treats "every flag at its current production value" as the one continuously tested golden path, tests each individual flag's two states in isolation — a CI matrix job per flag, not per combination, the same idea testing in the pipeline covers in full — and deliberately caps how many experimental flags can be concurrently active on the same code path, so the combinatorial space stays small enough for a human to reason about. Locally, developers need the ability to force a flag on or off in their own environment without touching the shared production ruleset — most SDKs support a local override file for exactly this, a piece of the broader inner-loop story in the inner loop & developer experience.

Every flag has a birth — the PR that needed it — and should have a scheduled death: a removal PR once it's fully rolled out and stable. Mature flagging platforms report per-flag usage over a trailing window (which variation was served, and how often) specifically so a flag sitting at 100%-on with zero evaluations of the "off" branch for weeks becomes an obvious, automatable cleanup candidate rather than something someone has to remember.

Governance: a flag is not a security boundary

☺ Like you're 10: Hiding the cookie jar's key under the mat isn't the same as locking the door — a flag that only hides a button in the browser can be walked straight around by anyone who skips the browser.

The evaluation-architecture section above already named the first governance failure: a client-side SDK that ships the full ruleset — feature names, rollout percentages, segment membership — to a browser lets anyone with a network tab discover unreleased work and internal targeting logic, even when the feature it describes is correctly gated off for them. The fix is architectural, not a policy reminder: sensitive rules evaluate server-side, and the client only ever receives the small, per-user, already-resolved bootstrap payload described earlier — booleans, not rules.

The second failure is more consequential. A permission or entitlement flag that hides a UI element for users on the wrong plan tier is not access control — it's a rendering decision. If the API endpoint behind that UI doesn't independently enforce the same authorization check server-side, the gate is trivially bypassed: call the endpoint directly, or flip a client-side override, and the "hidden" feature is fully reachable regardless of what the flag says. Flags are for controlling exposure and rollout risk; authorization is a separate, mandatory server-side check that must hold even if every flag in the system evaluates to true for everyone — the same layered-defense instinct shift-left security for DevOps applies earlier in the pipeline.

Third, because a flag flip is now a legitimate production change with real user-facing impact, it needs the same audit trail as a deploy: who changed which flag, to what value, and when, visible in the same timeline an incident review or a change-management process would consult. "Someone flipped a flag" is a root-cause category now, on equal footing with "someone shipped a bad deploy" — treat it with the same rigor, not as a config tweak that happens outside the record.

Choosing and operating a flagging system

☺ Like you're 10: You can build your own switchboard or buy one that's already wired — either way, the wiring behind the wall (streaming rules, a safe default, a paper trail) has to exist, whoever builds it.

A homegrown system — a database table, an in-memory cache, and a small SDK your team writes — is a reasonable starting point at small scale, but it means re-implementing every piece covered above yourself: streaming propagation, local evaluation with a safe fail-open default, a scoped client-side payload, and an audit log. Most teams past a certain size buy instead. LaunchDarkly is the long-standing commercial leader, with broad SDK language coverage and mature targeting and experimentation tooling. Unleash is open-source and self-hostable under an Apache-2.0-family license, a common choice when data residency or cost rules out a hosted vendor. Flagsmith and GrowthBook occupy similar open-source territory, with GrowthBook leaning further toward experimentation analysis specifically. Product names and licensing terms in this space shift — verify current pricing, hosting options, and license terms against each vendor's own site before committing.

OpenFeature is worth knowing regardless of which backend you pick: a vendor-neutral API specification (hosted as a CNCF project — confirm its current maturity stage on the CNCF landscape before citing it as a stability guarantee) that standardizes the client-side call — client.getBooleanValue("new-checkout", false, context) — while letting the actual rule evaluation come from a swappable provider plugin underneath. Adopt the OpenFeature API in application code and switching flagging vendors later becomes a provider-configuration change, not a find-and-replace across every call site in the codebase — the same "depend on the interface, not the implementation" instinct that makes infrastructure as code modules portable across providers.

Every idea on this page composes with what deployment strategies already taught: rolling, blue-green, and canary decide which infrastructure a request lands on; a flag decides which code path executes once it gets there. A mature rollout runs both at once — canary the infrastructure to a small percentage while additionally gating the risky logic behind a flag — so a bad change has two independent ways to be turned off, and the faster one, the flag, doesn't have to wait on the slower one, the deploy.