Three Cautionary Tales
Not every platform initiative ends up in a KubeCon talk. Most of the ones that fail never get written up at all — the team quietly reassigns, the budget gets pulled, and the only record is a Slack channel nobody reads anymore. This case file reconstructs three of the shapes that failure takes most often, built from patterns repeated across many real, unnamed platform efforts rather than any single employer. Each tale gets the same treatment as every named case on this site: starting situation, the architecture and team-design choices that were made and why, what changed, what to steal, and what honestly doesn't transfer.
None of the three organisations below is a real, identifiable company. Each is assembled from patterns that recur across many real platform efforts — the kind of thing a consultant, a conference hallway-track conversation, or a post-incident review turns up again and again, but that individual companies rarely publish as a named post-mortem (failure is bad marketing). Read every number here as illustrative, chosen to make a real and common pattern legible — not as a citation to any actual company's actual metrics. This matches the convention already set by the regulated enterprise case file: composites earn their keep by naming a pattern clearly, not by pretending to be evidence about one employer.
Imagine three kids each try to build a treehouse for the whole street. The first kid builds it alone in secret for a year and nobody wanted a treehouse shaped like that. The second kid builds a lovely treehouse but nails the door shut so only one kind of game can be played in it, and everyone else starts building their own treehouses on the side. The third kid is so afraid of the treehouse falling down that they lock it with six locks and a form to fill out — so everyone just climbs in through the window instead, which is far more dangerous than the front door ever was. Same street, same good intentions, three different ways to end up with an empty (or worse, a leaky) treehouse.
How to read three composite failures
☺ Like you’re 10: Each story below is a little mystery with the same four questions: what did they build, why, what happened, and which idea from the rest of this course would have caught it?
These three tales aren't picked at random — they map directly onto the three families of anti-pattern catalogued in Anti-Patterns & Pitfalls: building something nobody wanted, disguising toil as self-service, and operational shortcuts that detonate later. Read each one against that page's rogues' gallery table and you'll find the exact row. What this case file adds is the connective tissue anti-patterns.html doesn't have room for: a full arc — starting pain, the specific decisions, the organisational design, and a plausible, honestly-numbered outcome — so the pattern sticks as a story, not just a bullet point.
All three tales fail one of the two tests from Anti-Patterns & Pitfalls. The pull test: would a team choose this path if it weren't mandated? The self-service test: can a request complete with zero humans in the loop? Watch for which test each tale fails — it's the fastest diagnostic you'll ever run on a struggling platform.
Tale one — the platform nobody asked for
☺ Like you’re 10: Six engineers disappeared into a room for over a year to build the "perfect" platform. They never once came out to ask anyone what they actually needed.
The starting situation
Picture a 300-engineer online retailer, three years past its Series C, still deploying through a small, overworked infrastructure team that provisions everything by hand from a Jira queue: new namespaces, databases, TLS certificates, CI runners. Lead time for "give my team a new environment" runs four to seven business days. Leadership — reasonably — decides this has to stop, and funds a six-person platform team pulled mostly from that same infrastructure group, with a one-line mandate: "fix our infrastructure story." Nobody assigns a product manager. Nobody budgets time for discovery. The team is told to go build.
The architecture and technology decisions
The most senior engineer on the new team previously worked at a much larger company with a mature internal platform, and the team leans hard on that memory rather than on their own developers' actual pain. Over fourteen months, in near-total isolation from the product teams they're meant to serve, they build: a multi-cluster Kubernetes fleet spanning three cloud regions "for resilience we don't yet need," a bespoke internal developer portal built from scratch instead of adopting Backstage, a service mesh, and a sprawling hierarchy of custom resources modelling workload types — "microservice," "batch job," "ML pipeline," "scheduled job," "stateful set" — most of which no team at this company runs. It is, almost note for note, anti-pattern #4, cargo-culting the hyperscalers, compounding anti-pattern #1, the platform nobody asked for: an architecture sized for a company ten times the scale, solving problems this retailer doesn't have yet, while the one problem everyone actually has — a four-day wait for a namespace — goes untouched until the very end of the build.
The organisational choices
The team is run, deliberately, as a project with an end date rather than a product: a "platform v1 launch" is scheduled on the roadmap like a product release, complete with a launch event. Success is tracked internally as "features shipped" — twelve custom resource types, a portal with forty screens — a vanity metric that measures the team's own output, never whether anyone downstream wants it. No developer outside the platform team is interviewed, shadowed, or even shown a wireframe before launch day. The team reports to an infrastructure VP, not to a product organisation, and its roadmap is set entirely by architecture reviews inside the team.
“There was an all-hands demo with confetti emoji in the announcement. It provisions Kafka topics I've never touched and has a beautiful dashboard for a deployment strategy my team doesn't use. What I actually needed — a Postgres database and a staging URL without a four-day ticket — still isn't in there. So a week later I filed the same old Jira ticket, same as always. Nobody ever asked me what I needed. If they had, it would have taken about ten minutes.”
What changed
Six months post-launch, illustrative adoption looks like this: roughly 8% of teams have tried the portal at all, and fewer than 3% use it for a second service. The original Jira-ticket queue for infrastructure requests — the thing the whole initiative existed to kill — is still running at close to its pre-platform volume. A round of cost cuts arrives; the platform team, unable to point to a single team it demonstrably made faster, is the easiest six headcount to reallocate. The multi-cluster fleet is quietly consolidated back to one cluster over the following year by a much smaller team, who keep almost none of the original custom resources.
| Signal (illustrative) | Before | 6 months after launch |
|---|---|---|
| Lead time: new environment | 4–7 days | 4–7 days (ticket queue still primary path) |
| Teams who've tried the platform | — | ~8% |
| Teams still using it after one service | — | ~3% |
| Platform team headcount | 6 (funded) | 0 (reassigned in cost round) |
What to steal
Run discovery before writing a single CRD: interview the developers who file the most tickets, find the single biggest repeated toil, and build the thinnest viable platform that kills only that pain — nothing else, on day one. Fund the effort as an ongoing product with a roadmap driven by measured pain, not a project with a launch date. Track adoption and time-to-first-deploy from week one, not "features shipped" — a metric that rewards the team for building, never for helping. And size the architecture to the company you actually have: a good Helm chart, a template repo, and a clear README can be a legitimate golden path for a 300-engineer company; a three-region mesh is not a badge of seriousness, it's unstaffed complexity.
What doesn't transfer
This composite compounds several anti-patterns at once for narrative clarity — real failures are rarely this cleanly diagnosable, and most struggling platforms have some genuine adoption mixed in with the misses, which makes them much harder to recognise from the inside than from a case study. It's also worth naming the gap in the public record honestly: because failed internal platforms rarely get written up, this pattern is reconstructed from many partial, second-hand accounts rather than one traceable source — treat it as a composite silhouette of a common shape, not a documented incident.
Tale two — the golden path that became a one-lane bridge
☺ Like you’re 10: The road started smooth and everyone loved it. Then the builders welded the guardrails shut, and the only way to add a lane was to ask the builders — who couldn't keep up.
The starting situation
Now picture a mid-size payments company, roughly 500 engineers, in a compliance-heavy industry — this one gets the beginning right. Teams were shipping inconsistent, occasionally insecure Kubernetes manifests by hand, and a genuine platform team ran real discovery: they interviewed two dozen engineers, found that "getting a new stateless HTTP service to production" was everyone's biggest pain, and shipped a golden-path App custom resource that scaffolds a repo, a pipeline, and a secure-by-default deployment in minutes. Adoption in the first two quarters is genuinely excellent — around 85% of new services launch through it. This is the trap that only springs after a real early win, which is exactly why it's so dangerous: nobody on the team believes they're capable of the mistake anymore.
The architecture and technology decisions
To keep the platform "clean" and easy to reason about for compliance audits, the team makes one fateful design call: the App CRD supports exactly one shape — a stateless HTTP service on a fixed set of ports, with a fixed sidecar set, and no field for overriding the generated Deployment. There is no escape hatch: no podTemplateOverride, no documented way to see or hand-edit the rendered manifests, and no raw-Kubernetes lane for the rare team that needs one. This is anti-pattern #9, over-abstraction, in its purest form — the platform picked the wrong altitude on the very first workload type it didn't anticipate, and had no plan for the next one.
# The ONLY shape the platform's App CRD supports — no override, no escape hatch. apiVersion: platform.example.internal/v1 kind: App metadata: name: fraud-scoring spec: image: internal/fraud-scoring:2.3.0 # That's the entire spec. No field exists for: a second container, # a non-HTTP probe, a StatefulSet, a GPU request, or a cron schedule. # A team that needs ANY of those must file a request to the platform # team to hand-author a one-off exception outside the golden path.
The organisational choices
As the platform succeeds, more of the company's genuinely diverse workloads — a gRPC-only fraud-scoring service, a nightly reconciliation batch job, a stateful cache — arrive needing something the App CRD doesn't cover. Because there's no self-service escape hatch, every one of those becomes a change request that only the eight-person platform team can approve and hand-build, exactly anti-pattern #6, the platform team as the bottleneck. The team that built a genuinely good golden path for the 80% has, without meaning to, become the sole approver and sole implementer for the entire remaining 20% — for a company forty times its own size.
What changed
The backlog of "exception requests" grows every quarter as the product organisation grows, because each new exception's fix is itself hand-built rather than turned into a reusable, self-service capability. Illustrative lead time for a request that falls outside the golden path stretches from a few days in quarter one to several weeks by quarter six. Frustrated teams start doing the thing the platform was built to prevent: hand-writing raw Kubernetes manifests and applying them directly for anything the CRD can't express, quietly maintained outside the golden path's guardrails — a shadow platform re-forming inside a company that has a good, popular platform. The platform team, meanwhile, is now spending most of its week on one-off exception work instead of improving the golden path itself, and on-call load rises because those hand-built exceptions don't inherit the golden path's baked-in observability or policy defaults.
| Signal (illustrative) | Quarter 2 (early win) | Quarter 6 |
|---|---|---|
| Services launched via the golden path | ~85% | ~85% of a smaller "fits the mould" segment |
| Open exception requests | ~5 | ~60+, growing every quarter |
| Lead time for an out-of-shape request | 2–3 days | 4–6 weeks |
| Platform team time on new capability vs. one-off exceptions | ~80% / 20% | ~20% / 80% |
What to steal
Build the escape hatch on day one, not after the first exception — expose the generated manifests, and give power users a documented, still-governed way to drop to a lower-level API (a podTemplateOverride-style field, or a clearly-marked "advanced" path that still runs through the same policy admission gate). Track "requests actioned by hand" as a first-class metric and drive it toward zero, the same discipline anti-pattern #6 recommends — every hand-built exception is a missing self-service feature, not a one-off favour. And budget capability growth as an ongoing roadmap item, the way golden paths at the point of work describes: new workload shapes are expected demand, not surprises.
What doesn't transfer
Not every rigid golden path fails this fast. If an organisation's workloads really are homogeneous — nearly everything genuinely is a stateless HTTP service — a strict, override-free path can hold up far longer than six quarters, so the speed of collapse here depends heavily on how diverse the underlying workload population actually is. The lesson to keep is the structural one (escape hatches, and tracking hand-built exceptions), not the specific timeline.
Tale three — the vault with no door
☺ Like you’re 10: After a scare, the grown-ups locked the treehouse with six locks and a form. So the kids just climbed in through the window — which turned out to be far more dangerous than the door ever was.
The starting situation
A composite insurer, around 700 engineers, fails a security audit finding around unrestricted network egress and overly broad cloud IAM roles. The finding is real and the risk is real. In response, a security team — organisationally separate from the platform team, reporting through a compliance chain, measured solely on "zero findings at next audit" — writes a new set of controls and imposes them without involving the engineers who'd have to live inside them: every network egress rule requires a manually-reviewed firewall-change ticket with a five-business-day SLA; every new IAM permission requires a review meeting held once a week; every application secret must be requested through a ticket into a vault operated by yet another team; every container image needs a 48-hour manual scan-and-sign cycle by a central team before it can run anywhere.
The architecture and technology decisions
None of this is encoded as policy-as-code. There's no OPA or Kyverno admission controller evaluating requests automatically, no self-service secrets platform, no pre-approved egress template a team can adopt without a human in the loop — just a stack of manual review gates layered onto a default-deny network posture with zero pre-approved patterns for common, low-risk needs (calling a well-known SaaS API, reaching a standard package registry). Every one of these gates fails the self-service test: nothing here can complete without a human, no matter how routine the request.
# The vault-with-no-door posture, end to end, for one routine request: 1. Engineer needs the "invoicing" service to call a well-known payments SaaS API. 2. Files a firewall-change ticket. wait: up to 5 business days 3. Firewall change approved; now needs an IAM permission to read the SaaS API key from the secrets vault. 4. Waits for the weekly IAM review meeting. wait: up to 7 days 5. IAM approved; now needs the secret itself, via a separate ticket to the vault-operations team. wait: 1–2 business days 6. Needs the updated image scanned + signed before it can deploy anywhere, including to a test namespace. wait: up to 48 hours # Total: two to three weeks for one routine outbound API call. # Every step requires a human; none of it is expressed as policy-as-code.
The organisational choices
The security team's KPI is "findings at next audit," not developer lead time, and it never meets jointly with the platform or product teams that have to route every routine request through this six-step manual gauntlet — the org design guarantees the two groups never reconcile priorities in the same room. Because the security team's success is measured entirely upstream of impact on delivery, the manual gates keep expanding rather than shrinking after each subsequent audit.
What changed
Predictably, and within a couple of quarters, engineers start routing around the wall rather than waiting weeks for it. Illustrative but familiar patterns: a few teams provision personal cloud storage buckets outside the sanctioned account to move files quickly between systems, unsanctioned SaaS API keys get pasted directly into application config instead of going through the vault ticket queue, and one team spins up an unmanaged, unmonitored virtual machine outside the platform entirely just to get a job done on deadline. Nine months in, a real (if contained) incident traces back not to some external attacker defeating the strict controls, but to one of these workarounds — an unmanaged storage bucket, provisioned to dodge the ticket queue, left without the encryption and access logging every sanctioned path would have applied automatically. The restrictive posture didn't prevent the incident; it manufactured the very hole it existed to close, because the "safe" path was slower than the risk of taking a shortcut felt.
This is anti-pattern #11, cluster-admin for everyone, running in reverse but landing in the same place: excessive friction, like excessive permission, still produces an ungoverned, unaudited path — it's just that instead of one big hole, you get dozens of small, invisible ones that nobody centrally tracks. A guardrail that isn't self-service isn't a guardrail; it's a detour sign pointing at the ditch.
What to steal
The fix, laid out in full in Security & Policy, is to make the safe path the fast path: encode the policy as code — Kyverno or Gatekeeper admission rules that auto-approve requests matching pre-cleared patterns — so routine, low-risk requests complete in minutes with zero humans, and reserve manual review for genuine exceptions. Give teams self-service, short-lived secrets (the External Secrets Operator, or a secrets manager issuing dynamic, expiring credentials) instead of a ticket into a vault. Publish a small set of pre-approved egress templates for common SaaS destinations so "call a known-good external API" doesn't require a bespoke firewall review every time. And measure the security function partly on developer lead time for routine requests, not solely on audit findings — least privilege enables self-service precisely because a well-scoped automatic "yes" keeps blast radius small and keeps people on the governed path.
What doesn't transfer
Not every manual gate is wrong, and this tale shouldn't read as "delete all human review." Some approvals — a genuinely novel data-sharing arrangement, a first-of-its-kind third-party integration — legitimately need a human's judgment, and real regulatory regimes do mandate specific human sign-offs that can't be automated away. The failure here isn't that humans were involved at all; it's that every request, including the routine 95%, was forced through the same slow manual lane as the genuine exceptions. The fix is triage — automate the common case, and spend the human attention you saved on the cases that actually deserve it.
The pattern underneath all three
☺ Like you’re 10: All three treehouse builders forgot to ask the kids who'd actually play in it — before, during, or after building.
Line the three tales up and one root cause explains all of them: each platform optimised for something measurable to the builders — architectural elegance, a clean CRD schema, a clean audit report — instead of for the developer's actual, continuously-measured experience. None of the three teams had a working feedback loop back from the people using (or avoiding) what they built. Tale one never opened that loop at all. Tale two opened it once, got a great early answer, and then stopped listening as circumstances changed. Tale three had a KPI that was actively hostile to opening it. In every case, the fix from Platform as a Product — a platform is a product, its users are customers who can leave, and you win them continuously or you lose them — would have caught the drift months before it became a crisis.
Every tale above would have shown a warning sign months in advance on one specific chart: the trend of "requests handled outside the golden path," whether that's the original ticket queue (tale one), hand-built exceptions (tale two), or unsanctioned workarounds (tale three). A platform team that tracks that number weekly — and treats a rising line as an incident, not background noise — gets to intervene while the fix is still a thin slice, not a rebuild.
Mapping the failures to the CNPE blueprint
☺ Like you’re 10: Each tale is really a different exam domain going wrong.
Each tale sits squarely inside one (sometimes two) of the exam's weighted domains, which is exactly why this composite earns a place alongside the six real case files on the case studies hub.
| Tale | Anti-pattern(s) | CNPE domain | Course lesson |
|---|---|---|---|
| The platform nobody asked for | #1, #3, #4 | Platform APIs & Self-Service · 25% | Platform as a Product |
| The golden path that became a one-lane bridge | #6, #9 | Platform APIs & Self-Service · 25% | Developer Experience & the Inner Loop |
| The vault with no door | #11 (in reverse) | Security & Policy Enforcement · 15% | Security & Policy |
Notice that two of the three tales land in the same domain — Platform APIs & Self-Service — despite looking like opposite mistakes (too little structure vs. too much). That's the exam's own emphasis showing through: at 25%, it's the single largest domain, and both "nobody built a paved road" and "the paved road has no exits" are failures of the same underlying skill — designing a self-service API at the right altitude. See Platform APIs & CRDs and Self-Service & the Developer Portal for the constructive version of both fixes, and the CNPE domain breakdown if you want the full weighting.
⚖ CNPA vs CNPE — The domain percentages in the table above are CNPE's blueprint specifically — CNPA is a separate, fully closed-book multiple-choice exam with no lab component and no published per-tool domain weighting to map these tales onto. CNPA readers should mine these three tales for the underlying failure patterns and lessons (discovery before building, escape hatches on golden paths, policy-as-code over manual gates), since that concept-level knowledge still matters for CNPA's closed-book recall — not for the domain-weight mapping itself.
Diagnosing your own platform
☺ Like you’re 10: A short checklist to run on your own treehouse before someone else has to write its cautionary tale.
Run these three questions against your own platform, in order, and stop at the first "no" — that's your tale in progress.
| Question | A "no" here looks like… |
|---|---|
| Can you name the last five developers you interviewed about their pain, and what you shipped because of it? | Tale one: a platform built on assumption, not discovery. |
| Can every team's real workload shape be expressed through your golden path — including the awkward ones — without filing a request to your team? | Tale two: a golden path with no escape hatch, quietly becoming a bottleneck. |
| Does your riskiest, most routine security or compliance request complete in minutes with zero humans in the loop? | Tale three: a guardrail so slow that people build their own doors. |
Pick the platform you know best — yours, or one you've used as a developer. For each of the three questions above, write one honest sentence of evidence, not a guess: a real interview, a real exception-request count, a real request-to-approval time. If you can't produce evidence for a question, that's itself the finding — it means nobody is currently watching the exact signal that would have caught each of these three tales early.
Foxy: Three platforms, three teams that meant well, three failures. What's the one thing I keep tripping over in all three files?
Gizmo: Easy — I was in the room for all three! First I said “skip talking to users, you already know best.” Then I said “lock the CRD down tight, it's cleaner.” Then I said “more locks, more forms, more Thursdays-only meetings, that's what safety looks like.” 🤑
Timmy: And every single time, the fix was the same shape: put a feedback loop back in, and make the safe path the fast path. Discovery instead of assumption. An escape hatch instead of a cage. Policy-as-code instead of a queue.
Dot: Nobody in any of these stories asked me what I needed until it had already gone wrong. I don't want a perfect platform — I want someone to notice when I quietly stop using it, and ask why.
Timmy: That noticing is the job. Track the line that measures "how much work is happening outside the path you built," and you'll see all three of these tales coming, months before they become case studies.
Set these three failures against the six real, named platforms on the case studies hub — Spotify's golden path via Backstage, Mercedes-Benz's fleet-scale GitOps, adidas and Zalando's self-service at retail scale, CERN's scientific-computing scale, Monzo's regulated microservice estate, and Netflix's paved roads for continuous delivery — and the pattern is symmetric: every one of those succeeded by keeping exactly the feedback loop these three composites let go slack. For the constructive mirror of tale three specifically, read the regulated enterprise, a composite built to show guardrails-as-enabler done right.
1. Which single anti-pattern number, from Anti-Patterns & Pitfalls, best explains why the second tale's platform team became overwhelmed? 2. In the third tale, why did the restrictive security posture end up causing the very incident it was meant to prevent? 3. What one metric would have surfaced all three failures months earlier — and why does it work across such different failure shapes? 4. Why is it notable that tale two's platform had a genuinely good first six months?
Check your answers
- Anti-pattern #6, the platform team as the bottleneck — because the CRD had no escape hatch, every workload that didn't fit the one supported shape became a manual exception only the platform team could build, and that queue grew faster than the team could clear it.
- Because the manual gates were slower than the risk of a shortcut felt, engineers routed around them with unmanaged, unmonitored workarounds — and the incident originated in one of those workarounds, which lacked the encryption and logging the sanctioned (but far slower) path would have applied automatically. Excess friction produced an ungoverned path, just like excess permission would have.
- The trend of "requests handled outside the golden path" — the original ticket queue in tale one, hand-built exceptions in tale two, unsanctioned workarounds in tale three. It works across all three because it directly measures the thing every platform failure ultimately is: developers choosing not to use the sanctioned path.
- It matters because it proves these failures aren't only a "bad platform" problem — a good, well-discovered golden path can still curdle later if it never grows an escape hatch as the workload population diversifies. Good intentions and a strong launch don't inoculate a platform against this family of failure; only an ongoing feedback loop does.