A Startup's First Production Cluster
This is a composite: no single real company, assembled instead from the two mistakes that show up in almost every "our first production cluster" retrospective anyone has ever written up. A fourteen-engineer team migrates off a pile of long-lived virtual machines onto one Kubernetes cluster in about six weeks, racing a customer deadline, with one person splitting their time between the migration and their actual job. They ship every workload with no resource requests or limits, and they put every workload — every team, every environment, every Secret — into a single namespace called prod. Neither decision looks wrong on day one. Both get collected, with interest, on the one night they actually get tested. This page walks through that night, and the namespace-quota-NetworkPolicy bundle that made sure it only happened once.
No company named or described below is real. This is assembled from patterns that recur across many small teams' first production Kubernetes rollouts — the kind of story a platform engineer tells in a hallway or a postmortem retro, not something any single startup published with real names attached. Every number here is illustrative, chosen to make a common, real pattern legible — not a citation to any actual company's actual metrics. Read it the way you'd read Anti-Patterns & Pitfalls: as a diagnosis you can hold up against your own cluster, not as a report about someone else's.
Imagine a new apartment building where nobody assigned units yet — everyone just drops their stuff in one big shared room, and nobody wrote down how much space, water, or electricity any one person is allowed to use. It works fine for weeks, because most people are tidy and don't use much. Then one night, one person decides to run a huge load of laundry — nothing against the rules, because there are no rules — and every washing machine in the building locks up. The building manager doesn't even know whose laundry caused it, because there's no way to tell whose stuff is whose in one big shared room. That's a cluster with no resource limits and one namespace: it works right up until the day something ordinary — a batch job, a bigger customer, a busy Tuesday — asks for more than anyone budgeted, and takes down a stranger's apartment along with its own.
The starting situation
☺ Like you're 10: A small team moves fast, under a real deadline, and the two things that get skipped are the two things that feel the most like paperwork.
Picture a fourteen-engineer B2B logistics-scheduling startup, six weeks from a signed enterprise customer's go-live date, still running everything on a handful of long-lived EC2 instances that one engineer SSHes into whenever something needs restarting. Leadership — reasonably — wants off that pattern before the new customer's traffic arrives, and the team lands on a single-cluster Kubernetes migration as the fastest credible path: one managed control plane, one place to point every service, one kubectl apply instead of eleven different SSH sessions. Nobody is hired specifically to run the cluster. One senior engineer picks up "platform" as roughly a third of their week, on top of the feature work the deadline still requires of them, and the other thirteen engineers each write and ship their own service's manifests directly.
The migration itself goes well, by the measure that matters most in week six: every service that ran on the old VMs is running on the new cluster, the customer's go-live date is met, and nothing has broken yet. Two decisions made under that exact time pressure are the ones this case study is about — neither one felt like a decision at the time, which is precisely how both of them survived past the deadline that was supposed to be their excuse.
Mistake one — every Deployment shipped with no resources block
☺ Like you're 10: Nobody wrote down how much food each guest gets at the table, so the scheduler just seats everyone and hopes there's enough to go around.
Under deadline pressure, "figure out real CPU and memory numbers for eleven services" loses to "ship the eleven services" every time it's weighed against a go-live date. The team's actual reasoning, reconstructed from how this plays out almost everywhere it happens: setting an honest requests and limits value means either load-testing each service first or eyeballing a number and hoping, and eyeballing feels worse than just leaving the field out entirely and letting Kubernetes "figure it out." That reasoning treats an empty resources block as neutral. It is not — Scheduling & Resource Management covers this in full, but the short version is the one that matters here: a container with no requests and no limits doesn't get a safe default, it gets assigned to the BestEffort QoS class — the class first in line to be evicted the instant any node on the cluster runs short of memory, regardless of how important that particular Pod actually is.
# The Deployment as it actually shipped — no resources block at all.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nightly-report-generator
namespace: prod
spec:
replicas: 1
selector:
matchLabels: { app: report }
template:
metadata:
labels: { app: report }
spec:
containers:
- name: report
image: internal/report-generator:latest
# no `resources:` block here, or on any of the other ten
# services shipped the same week — every one of them is
# BestEffort QoS by omission, not by anyone's actual choice"I wrote that Deployment in about twenty minutes to hit the demo deadline. Nobody told me leaving out resources: meant anything — it deployed, the Pod went Running, the report generated. I didn't find out 'BestEffort' was a real, chosen-by-omission thing until the night it decided which Pod got killed first, and it wasn't the one I'd have guessed."
Mistake two — one namespace, prod, for the entire company
☺ Like you're 10: One big shared room for everyone's stuff, instead of a labeled cubby for each person — fine, right up until two people's boxes look identical and someone grabs the wrong one.
The second shortcut compounds the first. kubectl create namespace prod gets run once, in week one, and nobody revisits it — every team ships every service into that same namespace for the rest of the six-week sprint, because creating a new namespace per team was one more decision nobody had explicitly assigned to anyone, and the default path (just use the one that's already there) required zero conversations. Best Practices & Operating Model names exactly what this costs: a namespace is Kubernetes' unit of tenancy, the boundary you'd hang RBAC, a ResourceQuota, and a default-deny NetworkPolicy off of — and none of that boundary exists until someone actually draws it.
$ kubectl get ns
NAME STATUS AGE
default Active 9d
kube-node-lease Active 9d
kube-public Active 9d
kube-system Active 9d
prod Active 6d # every team, every service, one namespace
$ kubectl get deploy -n prod -o custom-columns=NAME:.metadata.name | wc -l
11
# eleven services, three CronJobs, one migrations Job, every team's
# Secrets — all reachable from each other, all covered by one shared
# RBAC Role, all sharing one Quota: none, because none was ever set.The night both mistakes collided
☺ Like you're 10: The missing food rule and the missing name-tag rule turn out to be the same story, told twice in one night.
Three weeks after go-live, the new customer's first end-of-month export runs through nightly-report-generator at 2 a.m. — a bigger job than any test run, because it's the first one processing a full production month of real data instead of a demo dataset. With no limits set, nothing stops its memory usage from climbing well past what anyone tested against. Node A runs short of memory, the kubelet starts evicting, and because every Pod in prod is BestEffort, eviction order comes down to usage-over-request — a comparison that's meaningless when nobody set a request. payments-api, mid-checkout for a live customer, gets evicted. Not the batch job that caused the pressure.
$ kubectl get events -n prod --sort-by=.lastTimestamp | tail -4 2m Warning Evicted pod/payments-api-7f9c8d5b6-4kxqz The node was low on resource: memory. 2m Warning OOMKilled pod/payments-api-7f9c8d5b6-4kxqz Container payments exceeded memory 1m Normal Scheduled pod/payments-api-7f9c8d5b6-9j2pw Successfully assigned prod/payments-api... 1m Warning BackOff pod/nightly-report-generator-cbb4f9d7-tqx1n Back-off restarting failed container # payments-api never asked for memory, so it was BestEffort — first in # line when nightly-report-generator (also BestEffort, also unlimited) # grew past whatever the node had spare that night.
The on-call engineer, paged for the checkout errors, reaches for the fastest fix they know: bounce the two payments-retry-worker Pods that handle the retry queue, in case they're the ones stuck. That's where the second mistake compounds the first. Nothing in prod ever forced any two teams to use distinguishable labels, and the payments team's retry workers and the analytics team's ETL workers had both, independently and for unrelated reasons, been labeled app: worker.
$ kubectl get pods -n prod -l app=worker NAME READY STATUS RESTARTS payments-retry-worker-8f7c9-x2z 1/1 Running 0 payments-retry-worker-8f7c9-z9k 1/1 Running 0 analytics-etl-worker-6b4d1-k3p 1/1 Running 0 analytics-etl-worker-6b4d1-m7q 1/1 Running 0 # two different teams, two unrelated jobs — both labeled app: worker, # because nothing in "prod" ever required either team to be more specific $ kubectl delete pod -n prod -l app=worker pod "payments-retry-worker-8f7c9-x2z" deleted pod "payments-retry-worker-8f7c9-z9k" deleted pod "analytics-etl-worker-6b4d1-k3p" deleted pod "analytics-etl-worker-6b4d1-m7q" deleted # the on-call engineer meant to bounce two Pods. All four came down — # including analytics' month-end ETL run, mid-batch, with nobody from # that team paged, because "worker" in a shared namespace matched # more than anyone typing the command that night had reason to expect.
Neither half of that night is exotic. A label selector in Kubernetes does exactly what it's told — -l app=worker matches every Pod carrying that exact label, full stop, with zero awareness of which team wrote which Pod. The eviction is equally mechanical: BestEffort is first in line, always, by design. Both mechanisms worked precisely as documented. What failed was the governance layer that was supposed to sit on top of them and never got built.
What they fixed
☺ Like you're 10: Instead of remembering the rules every time, they built a template that comes with the rules already attached.
A namespace per team, with a quota and a closed network boundary from day one
The fix isn't "add a ResourceQuota to prod" — a shared namespace with a quota is still one blast radius, one Role, one flat label space. The fix is splitting tenancy along the same line the org chart already draws, and bundling every governance object into the namespace's own creation, before a single workload is allowed to land in it:
# One namespace per team — created with its quota, its safe defaults,
# its closed network boundary, and its scoped Role already attached.
apiVersion: v1
kind: Namespace
metadata:
name: payments-prod
labels: { team: payments, env: prod }
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: payments-prod-quota
namespace: payments-prod
spec:
hard:
requests.cpu: "8"
requests.memory: 16Gi
limits.cpu: "16"
limits.memory: 32Gi
pods: "40"
---
apiVersion: v1
kind: LimitRange
metadata:
name: payments-prod-defaults
namespace: payments-prod
spec:
limits:
- type: Container
defaultRequest: { cpu: 100m, memory: 128Mi }
default: { cpu: 500m, memory: 512Mi }
# a forgotten `resources:` block now lands HERE — Burstable QoS
# with a real ceiling — instead of silently falling to BestEffort
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: payments-prod
spec:
podSelector: {}
policyTypes: ["Ingress", "Egress"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: payments-deployer
namespace: payments-prod
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "patch", "update"]Notice what the LimitRange does that a ResourceQuota alone can't: a ResourceQuota caps the namespace's total consumption, but it doesn't stop a single forgetful Deployment from landing as BestEffort inside that cap. The LimitRange is what turns "someone forgot the resources block" from a silent, dangerous default into a safe, visible one. Best Practices & Operating Model covers this same pairing, plus priority classes, in full.
Real requests and limits — measured, not guessed
The team's second fix is slower and less glamorous: before setting a single number, they let metrics-server and a week of kubectl top pod samples tell them what nightly-report-generator actually uses, instead of guessing high "to be safe" or low "to save cost." That's Sol the Sloth's entire method, applied literally — the same discipline Scheduling & Resource Management walks through for reading a workload's real 95th-percentile usage before committing it to a manifest.
# nightly-report-generator, after a week of real usage samples —
# not the guess that shipped in week one.
resources:
requests:
cpu: 250m
memory: 512Mi # ~95th percentile of a week of real overnight runs
limits:
cpu: "1"
memory: 768Mi # real headroom above that, not an open-ended ceilingA request is a promise to the scheduler — don't place this Pod somewhere that can't spare this much. A limit is a promise to the kubelet — kill or throttle this Pod if it ever exceeds this. Skip either one and the decision doesn't disappear, it just transfers to whichever Pod happens to be sharing that node the day the gap gets tested. Anti-Patterns & Pitfalls, mistake two walks through this exact failure shape in full.
What changed, in numbers
☺ Like you're 10: The same team, the same eleven services, six new labeled cubbies instead of one shared room.
| Signal (illustrative) | Before | Two quarters after the split |
|---|---|---|
| Namespaces holding production workloads | 1 (prod) | 6 — one per team, plus platform-system |
Pods with no resources block | ~70% of the fleet | 0 — the LimitRange backstops any omission |
| Unplanned evictions per month tied to a noisy neighbor | 4–6 | 0 |
Blast radius of one broad kubectl delete -l | any team's workload in prod | exactly one namespace |
| Time to stand up a new team's first namespace | ad hoc — whatever the on-call remembered | ~10 minutes, one templated manifest bundle |
The two tests that would have caught it early
☺ Like you're 10: Two short questions catch both mistakes in this story before either one ever gets a chance to matter.
Line the two mistakes up and each one fails a short, mechanical test — the same two tests worth running against any cluster, not just this composite one. The neighbor test: if this Pod has no requests or limits, what happens to the Pod sharing its node the day this one gets busy? If the honest answer is "I don't know," that Pod is BestEffort by omission, and Requests and limits: what actually enforces them is the page that turns "I don't know" into a real number. The boundary test: if this label, this RBAC Role, or this NetworkPolicy is one typo or one coincidence away from matching more than intended, what else does it touch? If the honest answer is "more than one team's workloads," that's a namespace boundary that was never actually drawn — see Namespaces as the tenancy boundary and RBAC & Admission Control for the fix in full.
Pick one Deployment you're responsible for right now. Open its manifest and find the resources block — if there isn't one, that's the neighbor test failing on the spot. Then run kubectl get pods -n <your-namespace> -l <one of your own labels> and actually read every Pod it returns — if anything you don't recognize shows up, that's the boundary test failing. Neither check takes longer than making a cup of tea, and both are exactly the checks this composite team didn't run until after the night that made them start.
What doesn't transfer
☺ Like you're 10: Not every small team needs six namespaces on day one — the lesson to keep is the habit, not the exact org chart.
A genuinely tiny team — three or four engineers, one product, no regulatory scope — can reasonably run a single namespace for a while without courting disaster, as long as requests and limits are set honestly from the start; the namespace split matters most in direct proportion to how many independent teams are sharing one cluster, because that's what turns "no boundary" from a minor tidiness gap into an actual blast-radius problem. This composite also compounds both mistakes for narrative clarity in one incident — real first-cluster stories are messier, usually spread across several smaller near-misses rather than one clean night. And the specific numbers throughout this page are illustrative, chosen to make the pattern legible, not measurements from any real deployment. What should transfer regardless of team size: set requests and limits before the first production Pod ships, not after the first eviction, and decide the namespace boundary deliberately, even if the deliberate decision for a three-person team is "one namespace, revisited at ten engineers" rather than "six namespaces now."
If your interest here is the exam-relevant version of this story rather than the narrative one, both mistakes map directly onto the CKA blueprint: resource requests and limits sit inside Workloads & Scheduling, and namespace-scoped RBAC sits inside Cluster Architecture, Installation & Configuration. For the two companion composites in this course's case-study set, see A Fintech's Multi-Tenant Platform and A Media Company's Multi-Cluster Migration — or return to the case studies hub for the full set.
Sol: I finally pulled a full week of kubectl top data on nightly-report-generator. It peaks around 430Mi, not the 2Gi somebody assumed it might need. One assumption, never checked, cost us a payments outage.
Gizmo: Or you could've just thrown a bigger node at it. Cheaper than writing four new YAML files a namespace. 🤑
Timmy: A bigger node buys you one more incident's worth of headroom. A LimitRange means the next team's forgotten resources block lands somewhere safe by default — forever, not just until the node fills up again.
Benny: I wrote that original Deployment in twenty minutes to hit the demo deadline. I genuinely didn't know "no resources block" meant "first Pod evicted." Nobody told me BestEffort was a real, chosen-by-omission thing.
Dot: Nobody told me either, and I'm the one who got paged at 2 a.m. I don't want to memorize QoS classes. I want the platform to make the safe default the thing that happens automatically when I forget.
Sol: That's exactly what the LimitRange is for. You don't have to remember. It remembers for you — the same way the namespace boundary means you don't have to hope your label never collides with someone else's.
1. Why did payments-api get evicted before nightly-report-generator, even though the batch job was the one actually consuming the memory? 2. What's the difference between what a ResourceQuota fixes and what a LimitRange fixes — and why did this composite team need both? 3. Why did kubectl delete pod -n prod -l app=worker remove Pods belonging to two unrelated teams, and what specifically about the fix stops it from happening again? 4. Name two governance objects, beyond the Namespace itself, this case study's fix bundled in from day one — and explain why "from day one" matters more than "eventually." 5. What are the neighbor test and the boundary test, and which of this case study's two mistakes does each one catch?
Check your answers
payments-apihad norequestsorlimits, so it defaulted to the BestEffort QoS class — the class the kubelet evicts first under memory pressure, regardless of which Pod actually caused that pressure.nightly-report-generatorwas also BestEffort, but it was still consuming memory rather than sitting idle, and QoS-based eviction order doesn't distinguish "cause" from "victim."- A
ResourceQuotacaps the namespace's total resource consumption across every Pod in it. ALimitRangesets what happens to any individual container that doesn't specify its ownresourcesblock — giving it a real default request and limit instead of falling to BestEffort by omission. The team needed both: the Quota stops any one namespace from starving the cluster, and the LimitRange stops any one forgotten manifest from starving its own neighbors inside that namespace. - Both teams had independently labeled unrelated Pods
app: worker, and a Kubernetes label selector matches on exact string equality with zero awareness of which team owns which Pod — so-l app=workercorrectly, mechanically matched all four. The namespace split stops this because each team's workloads now live in a separate namespace; the same broad selector run insidepayments-prodcan no longer reach anything inanalytics-prod. - Any two of: the
ResourceQuota(a hard ceiling on the namespace's total resource use), theLimitRange(a safe default for any container that omits its own resources block), the default-denyNetworkPolicy(no cross-namespace traffic without an explicit allow rule), or the scoped RBACRole(permissions limited to that one namespace). "From day one" matters because a namespace that opens permissive and gets locked down later almost never actually gets locked down — by the time anyone circles back, something is already depending on the gap. - The neighbor test asks whether a Pod's missing requests or limits could let it starve a Pod sharing its node — it catches mistake one, the missing resources block. The boundary test asks whether a label, Role, or NetworkPolicy is one coincidence away from matching more than intended — it catches mistake two, the single shared namespace.