Dependency Management & Hyrum's Law

Production readiness reviews tell you to enumerate your dependencies and compare each hard one's SLO against your own target before a service goes near a pager. That's the right first move, and it's also incomplete in a specific, predictable way: the dependency graph a PRR reviews is the graph somebody wrote down — the services you meant to call, on purpose, through a documented contract. It says nothing about the graph that forms afterward, the moment real callers start hitting your API and quietly start relying on things you never promised: a response's field order, an error message's exact wording, how fast a call usually returns, what happens on a particular malformed input. That second graph is real, it changes your actual blast radius, and it never shows up on an architecture diagram until something breaks it. This page covers that problem — Hyrum's Law, named for the Google engineer who stated it precisely — alongside its inverse: what happens to your own SLO ceiling when the hard dependencies point outward, to a vendor who owes you a contract but not a fix, and how you turn an accurate map of both directions into a blast-radius estimate you can actually trust.

Hyrum's Law, precisely

☺ Like you're 10: If enough people use your thing, they'll end up relying on parts of it you never meant to be a promise — and once they do, quietly changing that part breaks them, promise or not.

The formulation, attributed to Google engineer Hyrum Wright and popularized publicly at hyrumslaw.com and in the Google-authored book Software Engineering at Google, reads: "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody." Read it twice, because the precise wording carries more weight than the folk-summary version most people repeat. It doesn't say every behavior will be relied upon — it says every observable behavior will, given enough users, and it makes no exception for behavior you explicitly disclaimed in the docs. A response header you never documented, the exact key order a JSON body serializes in, how many milliseconds a call usually takes, the specific wording of an error string, whether a list happens to come back sorted — none of these are contractual, and all of them are, eventually, somebody's contract, discovered the moment you change them and their build breaks.

The mechanism behind the law is not sentimental — it's a straightforward corollary of scale. A caller integrating against your API is solving a real, present problem, on a deadline, by observing what your system does today. If your documented contract under-specifies a behavior your system happens to exhibit consistently, and leaning on that behavior would make their integration marginally simpler, the population of callers over a large enough N does not universally resist the temptation — some fraction will, every time, because from any one caller's local perspective, coding against the contract and coding against the observed behavior look identical right up until the day they diverge. Hyrum's Law isn't a claim about careless engineers; it's a claim about what a large enough population of any engineers will do when a system behaves predictably in a way that's convenient to use, whether or not that predictability was ever promised.

A genuinely public, verifiable instance of the law resolving itself the honest way: Python dictionaries never contractually guaranteed insertion order before Python 3.7. CPython's 3.6 implementation happened to preserve it as a side effect of a memory-layout optimization to the dict internals — and enough real-world code came to depend on that incidental ordering that the language's own maintainers made insertion-order preservation an official, permanent language guarantee in 3.7, rather than fight a losing battle to stop people relying on it. The implementation detail became the contract because Hyrum's Law had already made it one in practice; the language spec just caught up to reality. Section seven of this page returns to that exact move — sometimes keeping the accident is the correct engineering answer.

Why this is an SRE problem, not just an API-design footnote

☺ Like you're 10: A change that passes every test you wrote and looks completely harmless in code review can still take down someone else's production system — because the thing they relied on was never in your tests to begin with.

Most engineering teams first meet Hyrum's Law as an API-design principle — a reason to think harder about what you document versus what you merely exhibit. For SRE, it's something sharper: a live, recurring cause of production incidents that no amount of unit testing, integration testing, or code review against the documented contract will ever catch, because none of those checks are aimed at the thing that actually broke. A team ships a change that's correct against every written spec — a latency optimization, a library upgrade that changes floating-point rounding at the fourteenth decimal, a switch from one JSON serializer to another that happens to reorder keys, a retry policy that makes a flaky call succeed on attempt one instead of attempt three — and somewhere downstream, a system that had quietly come to depend on the old timing, the old ordering, or the old retry count breaks in a way its own owners can't explain from their own change history, because they didn't change anything. The incident shows up as their page, investigated against their deploys, for a root cause that lives entirely in your commit.

This is what makes Hyrum's Law incidents distinctively expensive to diagnose. A conventional regression breaks the thing that changed; a Hyrum's Law regression breaks something that, by every internal metric the team who broke it can see, didn't change at all. Blameless postmortems exist partly for exactly this pattern — the five-whys has to cross a team boundary to land on the real cause, and "nobody did anything wrong, on paper, and the outage still happened" is one of the harder findings for a blameless process to sit with honestly, precisely because it's true. The fix a postmortem produces for this failure mode is rarely a code fix in the team that shipped the change; it's a process fix — a monitoring gap, a missing deprecation window, a change class that should have gone out behind a flag and didn't — which is exactly the territory the rest of this page covers.

Hard dependencies, soft dependencies, and the difference that actually matters

☺ Like you're 10: A hard dependency is one where, if it goes down, you go down with it, no matter what; a soft one is where you can shrug, hand back something slightly worse, and keep going.

Production readiness reviews and reliability patterns already lean on this distinction without always naming it explicitly, so it's worth pinning down precisely here, because the rest of this page is built on it. A hard dependency is one where your system's request cannot succeed without that dependency succeeding — the call is synchronous and on the critical path, and there is no fallback, cache, or default that lets your service return a usable response if the dependency is unavailable. A soft dependency is one your system can survive losing: a cache serving a slightly stale value, a circuit breaker (see reliability patterns) tripping to a sane default, a non-blocking call moved off the critical path entirely so its failure produces a missing nice-to-have instead of a failed request.

The distinction is not a property of the dependency — it's a property of how you call it. The same payment processor is a hard dependency for a checkout flow that can't complete an order without a real-time authorization response, and could be architected as a soft one for a fraud-scoring feature that's allowed to fall back to "skip enhanced scoring, flag for async review" if the scoring API times out. Nothing about the vendor changes between those two designs — only the calling code's contract with failure does. This matters because it's the one place a team has real leverage: you rarely control a hard dependency's own reliability, especially a third-party one, but you almost always control whether a given call is architected as hard or soft in the first place, and every dependency demoted from hard to soft is one fewer link in the AND chain the next section's math runs against.

Vendor SLOs: you can't out-promise your weakest hard dependency

☺ Like you're 10: If the strongest link in your chain is a nine and the weakest is a six, the chain is a six — no amount of polishing the strong links changes that.

SLO windows & composite SLOs works out the arithmetic for a chain of your own services: for N independent hops each succeeding with probability p_i, the end-to-end probability is the product ∏p_i, which is always less than or equal to the smallest p_i in the chain. That math doesn't stop applying the moment the chain crosses your organization's boundary. If your service has a hard, synchronous dependency on a third-party vendor — an identity provider, a payment processor, a managed database, a CDN, an email or SMS gateway — that vendor's own availability becomes a literal multiplicative term in your own ceiling, whether or not it appears on your architecture diagram as one of "your" services.

Work a realistic example. Suppose your checkout flow has three hard, synchronous external dependencies: an identity provider quoting 99.99% monthly uptime, a cloud compute/storage platform quoting 99.99%, and a smaller fraud-scoring vendor quoting only 99.9% — still building out the multi-region redundancy the bigger two vendors have already paid for. Even if your own code and every other dependency were perfectly, theoretically reliable — 100.000% — your composite ceiling is bounded by the product of all three:

ceiling, all three vendors at 99.99%:
    0.9999 × 0.9999 × 0.9999 ≈ 0.999700  →  99.9700%
    30-day budget: (100% − 99.9700%) × 43,200 min ≈ 12.96 min

ceiling, weakest vendor drops to 99.9%:
    0.9999 × 0.9999 × 0.999  ≈ 0.998800  →  99.8800%
    30-day budget: (100% − 99.8800%) × 43,200 min ≈ 51.84 min

One vendor, one tier weaker than the other two, roughly quadruples your allowed-downtime ceiling — from about 13 minutes a month to nearly 52 — and that's before your own code contributes a single millisecond of failure on top of it. You cannot publish an internal SLO tighter than this ceiling and have it mean anything: an SLO promising 99.95% while carrying a hard dependency capped at 99.9% is not an aggressive target, it's a number nobody on the team can hit no matter how well they execute, and a team held to an unhittable number stops trusting the number at all.

An SLA is not the same instrument as an SLO, and the difference matters most exactly here. As this course's glossary defines it, an SLA is an externally facing, contractual commitment, typically set looser than the internal SLO it's compared against, with financial or credit remedies attached for a miss. A vendor's published SLA number is not a claim that their real-world availability sits at that number — it's a floor below which they owe you something, usually a service credit capped at a small percentage of that billing period's fees, and vendor SLAs are, without real exception, careful to exclude consequential and indirect damages: lost revenue, lost customers, reputational harm. Treat a vendor's SLA credit as a modest discount on a bad month's bill, never as insurance that makes your business whole for what their outage actually cost you. (Verify the exact current terms, exclusions, and credit schedule on the vendor's own SLA page before relying on any specific figure — these terms get revised, and the exclusions live in the fine print, not the headline number.)

The 2017 Amazon S3 outage in US-EAST-1 is the canonical illustration of what a hard dependency on a single vendor actually costs when that vendor has a bad afternoon: a large fraction of the consumer internet, including services with no obvious reason to route through object storage at all, went down or degraded simultaneously, because S3 had quietly become an invisible hard dependency for far more than file storage — status pages, image hosting, static asset delivery, even some vendors' own dashboards used to communicate the incident were themselves affected. See the 2017 AWS S3 outage for the full case study; the lesson for this page is that "we use a major, extremely reliable cloud vendor" and "we have no hard-dependency exposure to a vendor outage" are not the same claim, and the gap between them is invisible until the vendor's number, whatever it published, comes due.

Three concrete levers follow from this math, in rough order of effort. First, and cheapest: know the number. Every hard external dependency's published SLA (or, lacking one, its trailing observed uptime from its own public status page) belongs in the same dependency inventory covered next, compared explicitly against whatever SLO you're proposing to publish — a five-minute check that catches an unhittable target before it's promised to anyone. Second: demote what can be demoted — the fraud-scoring example above is exactly this move, turning a hard dependency's ceiling into a soft dependency's non-issue. Third, and most expensive: engineer real redundancy across independent vendors for dependencies that truly can't be made soft — a second identity provider, a second payment processor, a multi-CDN setup — which only pays off if it's genuine fault-domain independence and not two integrations that both happen to route through the same upstream provider underneath. Reliability Economics covers pricing that third option honestly against what the extra nines are actually worth to the business, rather than buying redundancy because it sounds responsible without running the numbers.

Mapping what you actually depend on

☺ Like you're 10: You can't guard a border you haven't drawn — before you can say who gets hurt by a change, someone has to go find out who's actually standing downstream of it.

Everything above assumes you already know what you depend on. In practice that inventory is never complete on the first attempt, and it decays continuously as services evolve — which is exactly why dependency mapping has to be an ongoing discipline, not a diagram drawn once for a design review and never revisited. Three sources, layered, get you closer to ground truth than any one alone.

Self-declared inventories. The starting point is the PRR-style dependency list from production readiness reviews: every outbound call a service makes, hand-catalogued by the team that owns it, each one marked hard or soft, each hard one's own SLO or SLA recorded alongside it. This is cheap, fast, and reviewable in a design doc — and it's also, reliably, incomplete, because it only captures what the team that wrote it remembered to write down, and it says nothing at all about who calls you.

Static analysis. A second, cheap pass scans a service's actual code, container manifests, and infrastructure-as-code for outbound connection strings, SDK client initializations, hardcoded hostnames, and DNS lookups — catching dependencies the self-declared list missed because the person who added them left the team, or added them in a hotfix that never made it back into the design doc. This closes the gap between "what we meant to call" and "what the code actually calls," but it's still entirely about your outbound edges.

Runtime discovery. The only source that captures ground truth on both edges — what you call, and who calls you — is live traffic. OpenTelemetry instrumentation propagated across service boundaries, visualized in a trace explorer like Jaeger, or a service mesh's own telemetry (Istio, or the Envoy proxies underneath it) all build the call graph from observed requests rather than from anyone's memory of the architecture. This is the layer that actually catches Hyrum's Law dependents, because a service that started calling you last quarter, on its own initiative, with no ticket filed and no design doc updated, is invisible to the first two sources by construction — it only exists in the traffic itself.

That last point deserves its own emphasis, because it's where this page's two halves connect. Mapping your outbound hard dependencies tells you your own SLO ceiling, the subject of the previous section. Mapping your inbound callers — who actually calls you, on what endpoints, how often, and how sensitively to exact behavior — is what tells you your real blast radius, and it's the harder half, because unlike your own outbound calls, you didn't write the calling code and can't grep it. The practical technique is API-gateway or load-balancer access logs aggregated by caller identity (API key, service account, mTLS client certificate), cross-referenced against whatever service catalog you maintain; any caller identity appearing in the logs with no corresponding catalog entry is, by definition, either an undocumented consumer or a Hyrum's Law dependent you haven't met yet.

# Illustrative self-declared dependency manifest — the shape a service
# catalog entry (Backstage-style, or your own PRR template) typically takes.
service: checkout-api
owner: team-checkout
dependencies:
  - name: identity-provider
    type: vendor
    call: synchronous
    hard: true
    their_sla: 99.99          # from the vendor's published SLA page — verify current terms
  - name: fraud-scoring-api
    type: vendor
    call: synchronous
    hard: true
    their_sla: 99.9
    note: >
      Demote to soft in Q3 — fall back to async review queue on timeout.
  - name: recommendations-service
    type: internal
    call: synchronous
    hard: false                # circuit breaker + cached fallback already in place
    their_slo: 99.5
known_consumers:                 # populated from access-log analysis, not memory
  - web-frontend
  - mobile-app
  - partner-api-gateway
  # any caller showing up in gateway logs but not listed here
  # is an undocumented or Hyrum's-Law consumer — investigate before changing
  # any observable behavior of this service.

From dependency map to blast radius

☺ Like you're 10: The documented list of who calls you is your best guess at blast radius — the real number is always that guess plus however many people started depending on you quietly, and you only find out by breaking something or by actually going and checking first.

A dependency map turns into a blast-radius estimate through two separate counts, and conflating them is where most estimates go wrong. Fan-out — what you call, and how many of those calls are hard — determines how much of your availability is at the mercy of things outside your control, the subject of the sections above. Fan-in — who calls you, weighted by how much of their own traffic and criticality routes through you — determines how much of the system's availability is at the mercy of a change you make. A service sitting deep in many other teams' critical paths (the high-fan-out "hub" pattern SLO windows & composite SLOs already flags as needing a tighter internal SLO for exactly this reason) has a large blast radius by construction, independent of how reliable it actually is — a perfectly healthy hub service that changes one observable behavior can still take out a dozen downstream teams simultaneously, each of whom did nothing wrong and has no idea why their own dashboards just went red.

The documented fan-in count from the service catalog is a floor on the real number, not the real number itself, and Hyrum's Law is exactly why. Every consumer discovered through access-log analysis but missing from the catalog is a caller whose sensitivity to which specific behaviors they depend on is completely unknown until you either ask them or change something and watch what breaks. That's the honest, uncomfortable conclusion the rest of this page has been building toward: a documented consumer list plus static analysis gives you a lower bound on blast radius, and the gap between that lower bound and the true blast radius is precisely the population of Hyrum's Law dependents nobody has met yet. Treat any pre-change blast-radius estimate as provisional, and treat "we checked the consumer list and nobody's affected" as a claim about the documented graph, not the real one.

The 2021 Meta BGP/DNS outage is a useful illustration of blast radius extending past what anyone would have mapped in advance, in a related but slightly different way: a configuration change withdrew the BGP routes advertising Meta's own DNS servers, and the resulting outage cascaded into internal systems — widely reported to include the physical badge-access systems needed to let engineers into the data centers to fix the problem — because those internal tools shared a hard, undocumented-as-critical dependency on the same internal network the change had just broken. (Treat the badge-reader detail as widely reported rather than independently verified here, but the structural lesson holds regardless: internal dependency graphs hide exactly the same kind of surprises external ones do, and they're easy to under-map precisely because "internal" quietly gets read as "lower risk" — see the worked incident case study for this course's own worked example of that mistake.) An accurate blast-radius estimate is also the specific input incident command for large-scale incidents uses to decide whether a page needs the full IC/Ops Lead/Comms Lead structure — blast radius spanning more than one team's service is one of that page's own stated escalation triggers, which means a dependency map that under-counts blast radius doesn't just risk a bigger incident, it risks the incident being run at the wrong scale from the first minute.

Living with Hyrum's Law: making implicit dependencies survivable

☺ Like you're 10: You can't stop people from depending on things you never promised — so the real game is shrinking how badly it hurts when you eventually have to change one of them.

None of the above argues for freezing a service in place to avoid ever disturbing an implicit dependency — that's not a real option, and even if it were, it wouldn't be a good one. The realistic goal is narrower: make the inevitable moment when an observable behavior changes as survivable as possible for whoever turns out to be depending on it.

Treat every change to observable behavior as a release, not a refactor. The "safe refactor" framing from earlier in this page is the trap — a change with an identical documented contract still needs the staged rollout, canary, and monitoring discipline of any risky release, specifically because the documented contract is not what determines the blast radius. Reliability patterns and release engineering & progressive delivery already give you the mechanism; the point here is applying it to a wider category of change than "looks risky" would normally trigger.

Version and deprecate deliberately, on a clock the consumer can plan around. When a behavior genuinely does need to change, announce it, run the old and new behavior side by side behind a flag or a version header for a fixed window, and communicate a hard sunset date — the IETF's RFC 8594 Sunset HTTP header is a standard, machine-readable way to signal exactly that date to any client that bothers to check it. A deprecation window doesn't find every Hyrum's Law dependent on its own, but it converts an instant, silent break into a bounded, telegraphed one, which is the difference between a page and a ticket for whoever was relying on the old behavior.

Layer contract testing under wide monitoring — don't substitute one for the other. Consumer-driven contract testing (Pact and similar tools formalize this) lets a known consumer publish an executable expectation of your API that runs in your CI before a breaking change ships — genuinely useful, and genuinely limited to consumers who opted in. By definition, a Hyrum's Law dependent never wrote a contract, so contract testing structurally cannot catch them; the layer that does is wide, always-on monitoring of real traffic across every caller, known or not, watched closely enough during the canary window that an unfamiliar caller's error rate spiking gets noticed before it's the only signal anyone gets.

Sometimes the correct engineering answer is to keep the accident. When a genuinely incidental behavior has enough real, costly-to-migrate dependents, the cheapest fix is occasionally to formalize the accident as a permanent guarantee rather than spend the migration effort to remove it — exactly what CPython did with dict ordering in 3.7. This isn't a defeat; it's Hyrum's Law resolved honestly, by promoting the de facto contract to a real one instead of pretending the old "unspecified" language still means anything once enough of the world is relying on the specific behavior anyway.

Deliberately going looking for these dependents, rather than waiting for one to file the incident on your behalf, is also exactly what chaos engineering is good for: a fault or behavior change injected on purpose, at a known and controlled blast radius, is a far better way to meet an undocumented consumer than a real outage is — and it's covered further in Chaos Engineering at Scale.

Where this fits

☺ Like you're 10: This page answers what you're really depending on and who's really depending on you — everything else in this course quietly assumes you already know the answer to both.

Every mechanism elsewhere in this course quietly assumes the dependency graph underneath it is accurate. SLIs, SLOs & error budgets and composite SLOs assume you know which of your calls are hard and what their own numbers are. Incident command for large-scale incidents assumes blast-radius estimates at declaration time are close to real. Chaos engineering is, among other things, a deliberate, controlled way to go find the Hyrum's Law dependents this page describes before an ordinary deploy finds them for you. Production readiness reviews is where the self-declared half of the dependency inventory this page builds on actually gets written down and reviewed, before a service is trusted with production traffic at all. None of these pages will save you from Hyrum's Law outright — nothing can, it's a structural property of scale, not a bug to patch — but an accurate, continuously-updated dependency map is what keeps its cost small and boring instead of large and surprising.