Scaling, Scheduling & Performance

A platform has two questions to answer for every workload it runs: where should this run, and how much of it should there be? The first is scheduling — the moment a Pod is matched to a node. The second is scaling — the continuous decision to add or remove replicas, and to add or remove the nodes those replicas need. This page goes past the exam blueprint and into the machinery: the scheduler’s filter-and-score cycle, the placement controls you use to steer it, the four kinds of autoscaler and when each one wins, event-driven and scale-to-zero patterns, the QoS rules that decide who gets evicted when a node runs out of room, the hard ceilings that make a single cluster stop scaling, and the thread that runs through all of it — every scaling knob is, underneath, a cost knob.

The Kubernetes scheduler

☺ Like you’re 10: A brand-new order has no table yet. The host first crosses off every station that can’t take it, then picks the best of the ones that can, then writes the table number on the ticket.

When you create a Pod, its spec.nodeName is empty — it is unscheduled. The kube-scheduler watches for these pods and, one at a time, decides which node each should run on. It is worth being precise here, because a lot of platform behaviour (why a pod is Pending, why it landed on the “wrong” node, why a spread didn’t happen) traces straight back to this one component. Recall from the Kubernetes substrate that the scheduler only decides — it never starts a container. That’s the kubelet’s job. The scheduler’s entire output is a single write.

The scheduling cycle — filter, then score

For each pending pod the scheduler runs a two-phase scheduling cycle. Phase one is filtering (historically called predicates): it walks the nodes and eliminates every one that cannot feasibly run this pod. A node is filtered out if it lacks the pod’s requested CPU/memory, doesn’t match a nodeSelector or node affinity, carries a taint the pod doesn’t tolerate, has no free host ports the pod needs, or can’t attach the pod’s volumes. What survives is the set of feasible nodes. If that set is empty, the pod stays Pending and the scheduler tries again later — this is the exact signal the Cluster Autoscaler and Karpenter watch for.

Phase two is scoring (historically priorities): each feasible node is given a score from 0 to 100 by a set of scoring plugins, the scores are weighted and summed, and the highest-scoring node wins (ties broken at random). Scoring is where “feasible” becomes “best” — it balances things like spreading pods of the same Service apart, honouring soft affinity preferences, preferring nodes that already have the container image cached, and, importantly, how full a node is. The default NodeResourcesFit plugin uses a LeastAllocated strategy (spread load across nodes), but you can switch it to MostAllocated to bin-pack — pile pods onto the fewest nodes — which, as Sol will keep reminding you, is a direct lever on cost.

Binding — the decision becomes real

Once a winning node is chosen, the scheduler binds the pod: it issues a write to the pods/binding subresource that sets spec.nodeName. That single field change is the whole handoff. The kubelet on that node is watching the API for pods assigned to it; the moment it sees the binding, it pulls images and starts containers. The scheduler splits its work into a fast, synchronous scheduling cycle (filter + score, one pod at a time, so decisions are consistent) and an asynchronous binding cycle (which may wait on slower work like volume provisioning) so that a slow bind doesn’t stall placement decisions for every other pending pod.

The scheduling framework & plugins

Modern Kubernetes doesn’t hard-code those phases — it exposes them as the scheduling framework, a set of extension points that plugins register against. The main ones, in order, are QueueSort (order the pending queue), PreFilter/Filter (feasibility), PostFilter (runs when filtering found nothing — this is where preemption lives), PreScore/Score/NormalizeScore (ranking), Reserve/Permit (claim resources, optionally delay), and PreBind/Bind/PostBind. The default behaviours you rely on — NodeResourcesFit, NodeAffinity, TaintToleration, PodTopologySpread, InterPodAffinity, VolumeBinding — are all just plugins wired into these points. That architecture is why you can run a second scheduler with a different profile (say, one tuned to bin-pack GPU jobs) alongside the default, or write a plugin for a placement rule Kubernetes doesn’t ship. For a platform, the practical takeaway is that “the scheduler’s behaviour” is configurable policy, not a black box.

Controlling placement

☺ Like you’re 10: These are the notes you pin to an order to steer the host — “must be a station with a pizza oven,” “keep the two dessert cooks apart,” “this station is reserved,” “spread the salads evenly.”

Out of the box the scheduler does a sensible job, but a platform almost always needs to steer placement: keep replicas in different zones for availability, pin GPU workloads to GPU nodes, isolate noisy tenants, or reserve expensive hardware. Kubernetes gives you a layered set of controls, from a blunt label match up to preemption. Knowing which one to reach for — and which are hard rules versus soft preferences — is core platform-design work.

nodeSelector & node affinity

The simplest control is nodeSelector: a map of labels a node must have for the pod to land there. It’s an all-or-nothing hard filter. Node affinity is its richer successor, with two flavours that you must not confuse. requiredDuringSchedulingIgnoredDuringExecution is a hard rule — it acts as a filter, and a pod that can’t satisfy it stays Pending. preferredDuringSchedulingIgnoredDuringExecution is a soft rule with a weight — it acts as a score, nudging the pod toward matching nodes but never blocking it. The clumsy IgnoredDuringExecution suffix carries real meaning: the rule is evaluated only at scheduling time. If a node’s labels change after the pod is placed, the pod is not evicted. Node affinity uses expressive operators (In, NotIn, Exists, Gt, Lt) over node labels, which is how you express “an arm64 node in eu-west-1 with an NVMe disk.”

Pod affinity & anti-affinity

Where node affinity relates a pod to node labels, pod affinity/anti-affinity relates a pod to other pods. Affinity co-locates: “schedule this cache near a pod of the web tier” to cut latency. Anti-affinity repels: “never put two replicas of this database on the same node (or in the same zone)” for availability. The key field is topologyKey — the node label that defines what “together” means (kubernetes.io/hostname for same-node, topology.kubernetes.io/zone for same-zone). Both come in the same hard (required…) and soft (preferred…) forms. One caution from the field: pod affinity is computationally expensive, because scoring a node means examining the pods on every other node in the relevant topology. On large clusters, heavy use of required pod anti-affinity can measurably slow scheduling — which is exactly why topology spread constraints were introduced as a cheaper way to get even distribution.

Taints, tolerations & topology spread

Taints and tolerations are the inverse mechanism: instead of a pod choosing nodes, a node repels pods. You taint a node with a key/value and an effect — NoSchedule (don’t place pods that don’t tolerate it), PreferNoSchedule (a soft version), or NoExecute (also evict already-running pods that don’t tolerate it, after an optional tolerationSeconds). A pod opts in to a tainted node by carrying a matching toleration. This is how dedicated node pools work: taint the GPU nodes nvidia.com/gpu=true:NoSchedule and only GPU workloads (which tolerate it) land there, keeping your expensive silicon from being squatted on by a batch job. Kubernetes itself uses NoExecute taints — node.kubernetes.io/not-ready and node.kubernetes.io/unreachable — to evict pods off a failing node.

Topology spread constraints are the modern, first-class way to distribute replicas evenly across failure domains. You declare a maxSkew (the maximum allowed imbalance between domains), a topologyKey (zone, node, …), a labelSelector (which pods count), and whenUnsatisfiable (DoNotSchedule = hard, ScheduleAnyway = soft). The scheduler then keeps the count of matching pods within maxSkew across every domain — giving you “spread these 6 replicas 2-2-2 across three zones” cleanly, without the O(pods²) cost of anti-affinity. For most HA-spreading needs on a real platform, topology spread is the tool to standardise on.

spec:
  # Hard rule: only nodes in these two zones are feasible (a FILTER).
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - { key: topology.kubernetes.io/zone, operator: In, values: [us-east-1a, us-east-1b] }
  # Even spread: never let one zone hold more than 1 extra replica.
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels: { app: checkout }
  # This workload tolerates (and thus is allowed onto) the dedicated GPU pool.
  tolerations:
    - { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }
  priorityClassName: business-critical      # see preemption, next

PriorityClass & preemption

What happens when an important pod can’t schedule because the cluster is full of unimportant ones? That’s what PriorityClass and preemption solve. A PriorityClass is a cluster-scoped object mapping a name to an integer value (higher = more important); pods reference it via priorityClassName. When a high-priority pod fails filtering because of insufficient resources, the scheduler’s PostFilter step runs preemption: it looks for lower-priority “victim” pods it could evict to make room, and if evicting them would let the high-priority pod fit, it deletes them (respecting their graceful termination) and schedules the important pod in their place. The evicted pods go back to Pending and reschedule elsewhere — or trigger a node scale-up. You can opt a class out of causing evictions with preemptionPolicy: Never (it still jumps the queue, it just won’t kick anyone out). Priority also feeds node-pressure eviction order, which we’ll meet later.

Pod autoscaling

☺ Like you’re 10: Two ways to handle a rush. Horizontal = call in more identical cooks. Vertical = give each cook a bigger station. One adds copies; the other resizes each copy.

Placement decides where a fixed number of pods go. Autoscaling decides how many pods there are, and how big each is, in response to load. Kubernetes ships two pod-level autoscalers that answer those two different questions — and a classic trap when you run both against the same signal.

The Horizontal Pod Autoscaler & its algorithm

The HorizontalPodAutoscaler (HPA) changes the replica count of a Deployment (or StatefulSet, or any scale subresource) to keep an observed metric near a target. Its core algorithm is a single, understandable formula:

desiredReplicas = ceil( currentReplicas × ( currentMetricValue / desiredMetricValue ) )

# Example: 4 replicas, pods averaging 90% CPU, target 60%:
#   ceil( 4 × (90 / 60) ) = ceil(6) = 6 replicas
# Next tick they average 55%:  ceil( 6 × (55/60) ) = ceil(5.5) = 6  → stays

Two details keep it stable. First, a built-in tolerance of 10% (0.1): if the ratio of current to desired sits within ±10% of 1.0, the HPA does nothing, which stops it twitching on noise. Second, a stabilization window: scale-down is deliberately damped (the default window is 300 seconds — it uses the highest recommendation over the last 5 minutes) so a brief dip doesn’t rip out capacity you’ll need again in a minute, while scale-up reacts fast (window 0). You can shape both directions further with the behavior block — policies that cap how many pods or what percentage may be added/removed per interval. Crucially, HPA computes utilization as a percentage of the pod’s resource request, so the HPA is useless without requests set — no request, no denominator.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: checkout, namespace: shop }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: checkout }
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Resource                       # built-in: from metrics-server
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 60 }
    - type: External                       # e.g. queue depth from a broker
      external:
        metric: { name: messages_in_queue }
        target: { type: AverageValue, averageValue: "30" }   # ~30 msgs per pod
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300      # damp flapping on the way down
      policies: [ { type: Percent, value: 50, periodSeconds: 60 } ]

Custom & external metrics

Scaling on CPU is the beginner’s move; it’s often the wrong signal. A web service’s real constraint might be requests-per-second or p99 latency; a worker’s might be queue depth. The HPA supports four metric types: Resource (CPU/memory, via metrics-server), Pods (a custom per-pod metric, averaged — e.g. active connections per pod), Object (a metric describing a single object — e.g. an Ingress’s RPS), and External (a metric from outside the cluster entirely — e.g. the length of a cloud queue). Custom and external metrics don’t come for free: they arrive through the custom.metrics.k8s.io and external.metrics.k8s.io aggregated APIs, which you must supply with an adapter (the Prometheus Adapter is the common one; KEDA, below, is another). The rule of thumb: scale on the metric that actually represents your bottleneck, and lean on your observability stack to know what that is.

The Vertical Pod Autoscaler — right-sizing requests

The VerticalPodAutoscaler (VPA) answers the other question: not “how many pods?” but “how big should each pod’s requests be?” It watches a workload’s actual CPU/memory usage over time and recommends (or applies) requests that fit reality. It has three parts: a recommender (computes target requests from history), an updater (evicts pods whose requests are too far off), and an admission controller (rewrites the requests on the recreated pod). Its updateMode matters enormously: Off only publishes recommendations (superb as a right-sizing advisor you read but don’t automate), Initial sets requests only at pod creation, and Auto/Recreate actively evicts and recreates running pods to resize them — which is disruptive. Historically VPA had to kill a pod to change its requests; the newer in-place pod resize feature is starting to let requests change without a restart, softening that sharp edge. VPA is the natural engine behind “stop over-requesting” — a theme Sol returns to under cost.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata: { name: worker, namespace: shop }
spec:
  targetRef: { apiVersion: apps/v1, kind: Deployment, name: worker }
  updatePolicy: { updateMode: "Off" }      # RECOMMEND only — read it, don't let it evict
  resourcePolicy:
    containerPolicies:
      - containerName: "*"
        minAllowed: { cpu: 50m,  memory: 64Mi }
        maxAllowed: { cpu: "2",  memory: 2Gi }

When HPA and VPA collide

Here is the trap the exam and real life both set. Do not point HPA and VPA at the same resource metric. If HPA scales replicas on CPU utilization and VPA rewrites CPU requests, they fight: VPA lowers the request, which raises measured utilization (same usage ÷ smaller request), which makes HPA add replicas, which lowers per-pod usage, which makes VPA lower requests again — a feedback loop that oscillates and never settles. The safe patterns are: use HPA on a metric VPA doesn’t touch (HPA on custom/external throughput, VPA on memory), or use VPA in Off mode purely as an advisor while HPA does the live scaling, or reach for a purpose-built multidimensional autoscaler that coordinates both. The one thing you must never do is let both automate the same signal.

Node autoscaling

☺ Like you’re 10: If every cooking station is full and orders still can’t be seated, rent more kitchen space. When space sits empty, give it back so you stop paying rent.

Pod autoscaling is pointless if there’s nowhere to put the new pods. When the scheduler can’t find a feasible node, a pod goes Pending — and a node autoscaler reacts by adding capacity, then removes it when it’s no longer needed. Two tools dominate, and the shift from one to the other is one of the more consequential recent changes in how platforms manage compute.

Cluster Autoscaler — scaling node groups

The Cluster Autoscaler (CA) is the long-standing default. It watches for pods that are Pending because of insufficient resources and, when it finds them, grows a node group — a cloud-provider construct like an AWS Auto Scaling Group or a GCP managed instance group — by incrementing its desired count. To scale down, it looks for nodes that have been underutilized (below a threshold, 50% by default) for a sustained period (10 minutes by default) and whose pods could be rescheduled elsewhere, then drains and removes them. Its defining constraint is that it works in terms of pre-defined, homogeneous node groups: every node in a group is the same instance type, and CA only picks which group to grow. If your workloads have varied shapes, you end up hand-maintaining many node groups, and CA’s “simulate whether this group’s instance type would fit the pending pod” logic gets fiddly.

Karpenter — just-in-time, groupless nodes

Karpenter reframes the problem. Instead of choosing among fixed node groups, it looks at the actual resource shape of the pending pods and provisions a node right-sized to fit them, launched directly from the cloud provider — no node groups at all. You give it a NodePool (constraints: which instance families, architectures, zones, capacity types it may use, and limits) plus a provider-specific NodeClass (AMI, subnets, security groups), and Karpenter does the bin-packing math: it considers the pending pods together, picks an instance type (or a diverse set) that fits them most cheaply, and boots it in seconds. Because it isn’t boxed into one instance type per group, it can select from dozens of types on each decision and naturally exploit price differences.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: default }
spec:
  template:
    spec:
      requirements:
        - { key: kubernetes.io/arch, operator: In, values: [amd64, arm64] }
        - { key: karpenter.sh/capacity-type, operator: In, values: [spot, on-demand] }
        - { key: karpenter.k8s.aws/instance-category, operator: In, values: [c, m, r] }
      nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: default }
  limits: { cpu: "1000" }                 # a hard ceiling on this pool's total vCPU
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized   # actively repack to cut cost
    consolidateAfter: 30s

Consolidation, bin-packing & spot

Karpenter’s second act is consolidation: it doesn’t just add nodes, it continuously asks “could this same set of pods run on fewer or cheaper nodes?” If yes, it drains and replaces nodes to tighten the packing — e.g. moving pods off three half-empty nodes onto two, or swapping an on-demand node for a cheaper instance type. This is bin-packing applied as an ongoing cost optimisation, not a one-time placement. It pairs beautifully with Spot capacity: Karpenter can prefer interruptible spot instances (far cheaper) and handle their downside by listening to the cloud’s interruption/rebalance signals, cordoning and draining a node before it’s reclaimed so pods reschedule gracefully. The trade-off is churn: aggressive consolidation and spot mean pods get rescheduled more often, so your workloads must be disruption-tolerant — which is exactly what PodDisruptionBudgets exist to bound. Karpenter honours PDBs and do-not-disrupt annotations so it can’t consolidate a workload into an outage.

Event-driven & scale-to-zero

☺ Like you’re 10: Some kitchens only staff a station when a special order actually arrives — nobody stands there waiting. When ten of those orders queue up, ten cooks appear; when the queue empties, they all go home.

CPU-based HPA is a poor fit for a lot of modern work: queue consumers, cron-like jobs, and services with long idle stretches. Their load isn’t “CPU%,” it’s “how many messages are waiting” or “is anyone calling right now?” — and their ideal floor is often zero replicas. Two projects specialise here.

KEDA — scalers & ScaledObjects

KEDA (Kubernetes Event-Driven Autoscaling, a CNCF-graduated project) lets you scale a workload on the depth of an external event source — and, uniquely for the built-in machinery, all the way down to zero. You attach a ScaledObject to a Deployment (or a ScaledJob to run Jobs) and pick from 70-plus scalers: Kafka lag, RabbitMQ/SQS queue length, Redis list depth, a Prometheus query, a cron window, cloud-native queues, and more. Under the hood KEDA is elegant: for a scaled-to-N workload it creates and manages an HPA for you, feeding it the external metric — so you get all the HPA stabilization behaviour for free. The one thing HPA can’t do, KEDA adds: an activation path from 0→1. When the workload is at zero, KEDA’s agent watches the source directly, and the instant a message appears it scales you to one, after which the HPA takes over.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: image-worker, namespace: media }
spec:
  scaleTargetRef: { name: image-worker }
  minReplicaCount: 0          # scale-to-zero when the queue is empty (free!)
  maxReplicaCount: 100
  cooldownPeriod: 120         # wait before scaling 1 → 0
  triggers:
    - type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.us-east-1.amazonaws.com/1234/thumbnails
        queueLength: "20"     # aim for ~20 in-flight messages per replica
        awsRegion: us-east-1

Queue-depth scaling in practice

The mental model for a worker pool is a target of backlog per replica. You’re saying “I want each pod to be responsible for roughly 20 queued messages; if the queue holds 200, run 10 pods.” It’s just the HPA formula with queue length as the metric: desired = ceil(queueLength / targetPerReplica). This decouples scaling from CPU entirely, which is what you want when a worker spends its time waiting on I/O rather than burning cores. Two cautions: pick a targetPerReplica that reflects real throughput (measure it), and mind the consumer contract — scaling to zero only works if your source buffers while you’re gone (a queue does; a raw synchronous HTTP call does not), and messages must be processed idempotently because pods will come and go mid-flight.

Knative — request-driven scale-to-zero

Where KEDA scales on a queue, Knative Serving scales on requests — it’s the serverless-style path for HTTP services. Its default autoscaler (the KPA) scales on concurrency (how many requests a pod is handling at once) or RPS, and it too can go to zero. The clever piece is the activator: when a service is scaled to zero and a request arrives, the request doesn’t fail — it’s routed to the activator, which buffers it, triggers a scale-up from zero, waits for the first pod to become ready, then forwards the held request. That’s how you get “no pods when idle, real responses when called.” The cost you pay is cold-start latency — the caller of the first request waits for a pod to boot — which is why scale-to-zero suits spiky, latency-tolerant, or internal endpoints far better than a hot user-facing path. Knative also versions each deploy as a Revision, giving you traffic-splitting for canaries, which ties into progressive delivery.

Requests, limits, QoS & the descheduler

☺ Like you’re 10: Every cook reserves a bit of counter space (a request) and promises not to sprawl past a line (a limit). When the station gets dangerously crowded, the ones who reserved nothing get shooed away first.

Under all the autoscaling sits one humble pair of numbers per container — requests and limits — that drives scheduling, isolation, eviction, and, as Sol insists, the bill. Getting them right is the least glamorous and most impactful scaling skill on a platform.

Requests vs limits

A request is what the scheduler reserves: it’s the amount of CPU/memory subtracted from a node’s allocatable capacity when the pod lands, and it’s the number the whole feasibility calculation runs on. A limit is the hard ceiling the kubelet enforces at runtime via cgroups. The two resources behave very differently at the limit, and this is a favourite exam point: CPU is compressible — exceed your CPU limit and you’re merely throttled (slowed), never killed. Memory is incompressible — exceed your memory limit and the kernel OOM-kills the container. So an under-set CPU limit shows up as mysterious latency; an under-set memory limit shows up as crash loops. Requests, meanwhile, are pure reservation: set them too high and you strand capacity (nodes look full while sitting idle); set them too low and you over-pack nodes and invite eviction.

QoS classes & eviction order

From the relationship between a pod’s requests and limits, Kubernetes derives its Quality of Service class, which decides who dies first when a node runs out of memory. Guaranteed: every container sets requests equal to limits for both CPU and memory — the safest tier, evicted last. Burstable: at least one container has a request but it’s not fully Guaranteed — the common middle. BestEffort: no requests or limits anywhere — the cheapest and the first to be sacrificed. When a node hits memory pressure the kubelet evicts in that order — BestEffort first, then Burstable pods most over their requests, Guaranteed last — with pod priority factored in as a tiebreaker. Understanding this ordering is how you make sure a batch job, not your payment service, is the thing that gets evicted when a node gets tight.

containers:
  - name: api
    image: registry.acme.io/api:2.3.1
    resources:
      requests: { cpu: 250m, memory: 256Mi }   # reserved at schedule time
      limits:   { cpu: 250m, memory: 256Mi }   # requests == limits → Guaranteed QoS
# Guaranteed: CPU throttles at 250m but is never killed for it; memory OOM-kills only
# if the container itself exceeds 256Mi. Evicted LAST when the node is under pressure.

Right-sizing from real usage

The single most common waste in a Kubernetes fleet is over-requesting: teams copy a generous request from a template, the scheduler dutifully reserves it, and nodes sit half-idle while the cluster autoscaler keeps buying machines to satisfy reservations that no one uses. The fix is to right-size requests to observed usage plus a sane headroom — exactly what the VPA recommender (in Off mode) computes for you, and what usage dashboards in your observability stack reveal. The healthy target is a request that sits a little above your steady-state usage so the scheduler’s reservations track reality, with limits set to absorb legitimate bursts. Do this across a fleet and node count — and the bill — often drops sharply without touching a single line of application code.

The descheduler — rebalancing over time

The scheduler makes a one-shot decision at placement time and never revisits it. But clusters drift: nodes are added, pods die and respawn, priorities change, and yesterday’s good placement becomes today’s lopsided one — one node jammed, three nearly empty. The descheduler is the counterweight. It runs periodically (as a Deployment or CronJob) and evicts pods that violate the balance you want, so the scheduler places them afresh somewhere better. Its strategies read like a list of drift problems: LowNodeUtilization (move pods off jammed nodes onto empty ones), HighNodeUtilization (the opposite — consolidate for bin-packing/cost), RemoveDuplicates (don’t stack replicas of one workload on a node), RemovePodsViolatingTopologySpreadConstraint, RemovePodsViolatingNodeAffinity, and PodLifeTime. Critically, it respects PodDisruptionBudgets and never evicts more than you’ve allowed — it rebalances within your safety limits. Think of the scheduler and descheduler as a matched pair: one places, the other keeps placement honest as the world changes.

Performance & scale limits

☺ Like you’re 10: Even a magic kitchen has walls. Past a certain number of stations and orders, the one manager’s notebook gets too big and too slow — and the smart move is to open a second kitchen, not cram more into the first.

A single Kubernetes cluster is not infinitely scalable, and a senior platform engineer knows where the walls are — because hitting them looks like mysterious, cluster-wide slowness rather than a clean error. These ceilings come from the control-plane machinery: one datastore, one API surface, one scheduler.

The official scale ceilings

The Kubernetes project publishes and tests against explicit thresholds. In a supported configuration a single cluster should hold no more than 5,000 nodes, no more than 150,000 total pods, no more than 300,000 total containers, and no more than 110 pods per node. These aren’t hard-coded caps (the pods-per-node default is a kubelet flag you can raise; people run bigger clusters with heroics) but they mark where the tested SLOs — API responsiveness, scheduling throughput — start to fray. Treat them as design guidance: if your growth curve is heading toward any of these numbers, plan the next architecture before you arrive, not after.

etcd & the API server

Almost every scale limit traces back to two components. etcd is the single source of truth, and it is latency-sensitive and size-bounded: its default storage quota is 2 GiB and the practical recommended maximum is around 8 GB, beyond which compaction and defragmentation pauses start to hurt. Because it uses Raft consensus, every write must reach a quorum across members, so etcd demands fast disks and low inter-member network latency — a slow disk under etcd manifests as slow everything, since no write completes until etcd commits it. The API server is the other pinch point: it fans every change out to thousands of watchers, and a burst of expensive list calls or a controller gone rogue can saturate it. Kubernetes defends itself here with API Priority and Fairness (APF), which classifies and rate-limits requests so a noisy client can’t starve the scheduler or kubelets of API time. When people say a cluster “feels slow,” the cause is almost always one of these two under strain.

When to shard into more clusters

Past a point, the right answer to “this cluster is getting too big” is not a bigger cluster — it’s more clusters. Sharding into multiple clusters buys you things a single giant cluster can’t: a smaller blast radius (an etcd or apiserver problem takes down one cell, not everything), independent upgrades (roll Kubernetes versions cluster by cluster), hard tenant/regulatory isolation, and a control plane that stays comfortably inside the tested limits. The cost is real — you now have a fleet to keep consistent, which is its own discipline: fleet-wide GitOps, a shared platform API, and cross-cluster networking and identity. That whole problem space — how to run many clusters as one platform without the operational tax exploding — is the subject of Multi-Cluster Platforms. The scaling instinct to internalise: grow a cluster until it’s comfortably productive, then shard rather than chase ever-larger single clusters.

Cost-aware scaling

☺ Like you’re 10: Here’s the secret the whole page has been building to — every time you decide how many cooks and how much kitchen, you’re deciding how big the bill is. Scaling is spending.

Sol has been muttering it in every section; now we say it plainly. Every scaling and scheduling knob is a cost knob. Requests set how much capacity you reserve (and pay for) whether you use it or not. Replica counts multiply that. Bin-packing density, spot usage, consolidation, and scale-to-zero all pull the bill down; over-requesting, generous headroom, anti-affinity spread, and “just in case” minimums push it up. A platform that treats scaling and cost as separate concerns will always overspend, because the decisions are literally the same decisions.

Every scaling knob is a cost knob

Walk back through the page with a price tag in hand. A request reserves node capacity, so over-requesting is money spent on air — right-sizing (VPA/usage) is often the single biggest saving available. Bin-packing (the MostAllocated scoring strategy, Karpenter consolidation, the descheduler’s HighNodeUtilization) fits the same pods on fewer nodes. Spot capacity is a large discount for interruptibility. Scale-to-zero (KEDA, Knative) drops idle workloads to nothing. Conversely, required anti-affinity and aggressive topology spread forbid tight packing and thus cost more — a reliability decision that is also a spending decision, and should be made with both in view. Making these trade-offs deliberately, with the numbers in front of you, is the heart of FinOps.

Overprovisioning vs headroom

There’s a genuine tension between cost and responsiveness, and it has a name: headroom. If you pack every node to the brim and run the autoscaler tight, a traffic spike waits for a brand-new node to boot (a minute or more) before its pods can schedule — a latency hit at the worst moment. The standard remedy is deliberate overprovisioning: run low-priority “balloon” or pause pods that reserve spare capacity and do nothing useful. When a real, higher-priority pod arrives, it preempts a balloon pod and schedules instantly onto the already-warm node, while the node autoscaler quietly replaces the balloon in the background. You’re paying for a little idle capacity to buy fast burst response. How much headroom is a dial, not a default: too little and spikes stall, too much and you’re back to overspending. Set it from your actual burst profile.

Sol’s cost-aware autoscaling playbook

Pulling it together into defaults a platform can ship: right-size requests to real usage (VPA-informed) so reservations track reality; scale stateless services with HPA on the metric that is your bottleneck, not reflexive CPU; push queue and event workers onto KEDA with scale-to-zero where the source buffers; let Karpenter provision just-in-time and consolidate, with spot for anything disruption-tolerant and PodDisruptionBudgets to bound the churn; reserve Guaranteed QoS and high priority for the truly critical path and let everything else be Burstable; and keep a measured slice of headroom via overprovisioning for latency-sensitive bursts. None of these is exotic — together they’re the difference between a platform that scales and stays affordable and one that does neither. The cost side of this story continues in FinOps for Platforms, and the reliability side in Reliability & Incidents.

Scheduling decides where; autoscaling decides how much; QoS and priority decide who wins when it’s tight; the scale limits decide when to split; and cost sits underneath all of it. Bring those together into per-workload defaults on a golden path and you’ve turned a pile of intricate knobs into something Dot never has to see — which is exactly what a platform is for. Next, follow the money in FinOps for Platforms, or the safety net in Reliability & Incidents.