Scheduling & Resource Management

Kubernetes Architecture named kube-scheduler as one box on a diagram: it filters, it scores, it writes a Binding. This page opens that box, and goes past what Workloads & Scheduling's CKA-level treatment of affinity, taints, and requests requires into territory the exam doesn't test but production clusters live or die on. The scheduler is a pluggable framework with named extension points, not a monolith; a Pod's resource requests and limits aren't abstract policy but numbers translated directly into Linux cgroup settings the kernel enforces; QoS class determines the exact order the kubelet kills processes under memory pressure, computed as a real number rather than a label; and affinity, taints, topology spread, and priority all act as independent, sometimes competing, placement signals the scheduler has to reconcile at once, every time. If Workloads & Scheduling taught you the vocabulary, this page teaches you the mechanism underneath each word.

The scheduler's two-phase algorithm, as a plugin pipeline

☺ Like you're 10: "Filter, then score" is the headline, but underneath it's really a checklist of small named steps running one after another, and any one of them can be swapped out or added to.

Modern kube-scheduler isn't a single filter-then-score function — it's the scheduling framework, a pipeline of named extension points that built-in plugins (and, if you write one, your own) hook into. For one Pod pulled off the scheduling queue, the scheduling cycle runs serially, one Pod at a time: PreFilter does cheap up-front checks and can reject the whole cycle early (a Pod requesting a hostPort already claimed cluster-wide, say); Filter is where plugins like NodeResourcesFit, NodeAffinity, TaintToleration, and PodTopologySpread each get a veto over every node — a single "no" from any plugin removes that node from consideration; PostFilter only runs if zero nodes survive filtering, and its default implementation is what triggers preemption; PreScore and Score then rank the nodes that did survive, and Reserve/Permit provisionally hold that node for the Pod before anything is written back to the API. Only after Permit does the binding cycle run — PreBind, Bind (which is what actually writes the Binding object), PostBind — and this second cycle is allowed to run asynchronously, in parallel across several Pods, precisely because the scheduler already made its irreversible node decision back in the (serial) scheduling cycle.

Which node wins the Score phase is itself configurable. The NodeResourcesFit plugin, which scores nodes on how well the Pod's requests fit their allocatable capacity, supports two opposite strategies: LeastAllocated (the default) favors the least-loaded node, spreading Pods out and leaving headroom on every node — good for absorbing spikes, worse for bin-packing efficiency; MostAllocated favors the most-loaded node that can still fit, packing Pods tightly onto as few nodes as possible so the rest can scale to zero — the strategy a cost-conscious cluster running the Cluster Autoscaler or Karpenter usually wants, since an empty node is a node that can be removed. Both are set per-cluster in a KubeSchedulerConfiguration, not per-Pod.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: default-scheduler
  pluginConfig:
  - name: NodeResourcesFit
    args:
      scoringStrategy:
        type: MostAllocated      # pack tight; default is LeastAllocated
        resources:
        - name: cpu
          weight: 1
        - name: memory
          weight: 1

Requests and limits: what actually enforces them

☺ Like you're 10: A request is the honest promise the seating chart is built from; a limit is a hard wall the kernel itself builds around you once you're seated.

Requests and limits aren't Kubernetes-level policy enforced by some Kubernetes process watching over your shoulder — they're translated straight into Linux cgroup settings on the node, the same primitive that gives every container its isolation in the first place (see DevOps' containers & orchestration page for cgroups and namespaces from the container side). And CPU and memory are enforced completely differently, because one is compressible and the other isn't. A CPU request becomes a proportional cpu.weight (cgroup v2; cpu.shares on the older v1 hierarchy) — under contention, the kernel's CFS scheduler divides spare CPU time proportionally to weight, but a container can still burst past its request any time the node has slack. A CPU limit becomes a hard ceiling, cpu.max (v2) or cpu.cfs_quota_us/cpu.cfs_period_us (v1) — cross it and the kernel simply throttles the container's processes for the rest of that period; nothing dies, it just gets slower, which is exactly why a CPU-throttled Pod can look "fine" in kubectl get pods while p99 latency quietly falls apart.

Memory has no such compressible middle ground. A memory request sets no cgroup limit at all by default — it exists purely for the scheduler's bin-packing math and, as the next section covers, for computing how aggressively the kubelet will kill this Pod under pressure. A memory limit becomes a real, hard cap, memory.max (v2) or memory.limit_in_bytes (v1). Cross it and the kernel's own OOM killer fires inside that cgroup, sending SIGKILL to the container's process — the container goes OOMKilled, and the kubelet restarts it per the Pod's restartPolicy, same as any other crash. There's no throttling equivalent for memory, because you can't "slow down" an allocation that's already happened.

# On a cgroup v2 node, every container gets its own leaf cgroup —
# these are the actual files kubelet's translation of requests/limits lands in
CGROUP=/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod.slice/cri-containerd-.scope

cat $CGROUP/cpu.max      # "50000 100000"  → 500m CPU limit (50ms compute per 100ms period)
cat $CGROUP/cpu.weight   # derived from the request — a relative share, not a ceiling
cat $CGROUP/memory.max   # the container's memory limit in bytes, or "max" if unset

# When something in this cgroup gets OOM-killed, the kernel logs it here first
dmesg -T | grep -i "oom-kill\|out of memory"

QoS classes and the kubelet's eviction order, precisely

☺ Like you're 10: Every process on the node gets a "how much do I mind being sacrificed" number, and under real pressure the kernel just kills whoever volunteered the most.

Workloads & Scheduling already covers what determines a Pod's QoS class — Guaranteed when every container's requests equal its limits, Burstable when at least one doesn't, BestEffort when none are set at all. What that page doesn't need to go into is how the kubelet turns that class into an actual kill decision: each container's Linux oom_score_adj value, which the kernel's OOM killer consults directly whenever it needs to pick a victim under real memory pressure. BestEffort containers get oom_score_adj pinned to 1000, the maximum — first in line, always. Guaranteed containers get -997, deep in negative territory — last in line, effectively protected unless nothing else is left. Burstable sits in between, scaled by how large the Pod's memory request is relative to the node's allocatable memory: a Burstable Pod that requested almost nothing gets a score near 999 (barely better than BestEffort); one that requested most of the node's capacity gets a score near 2 (barely worse than Guaranteed). Requesting more, even without raising your limit, buys real protection.

This only fires once the kubelet detects actual node pressure — it polls a fixed set of eviction signals (memory.available, nodefs.available, nodefs.inodesFree, imagefs.available, imagefs.inodesFree, pid.available) against configured thresholds, hard or soft. A hard threshold evicts immediately, no grace period, best-effort graceful termination only. A soft threshold instead starts a grace-period timer — if the signal is still past threshold when the timer expires, eviction proceeds; if it recovers first, nothing happens. Crossing a hard threshold doesn't reach for the OOM killer directly, either — the kubelet's own eviction manager tries to reclaim first (deleting dead containers, unused images) and, failing that, evicts whole Pods proactively, ranked by the same QoS-driven logic: BestEffort Pods over their requests go first, then Burstable Pods exceeding their requests, then Guaranteed only as an absolute last resort. The raw kernel OOM killer, using the oom_score_adj values above, is the backstop for the case the kubelet doesn't catch in time — a sudden memory spike inside a single container, rather than gradual node-wide pressure.

# Typical kubelet eviction flags — hard thresholds are on by default;
# soft thresholds are opt-in and need a paired grace period
--eviction-hard=memory.available<100Mi,nodefs.available<10%,imagefs.available<15%
--eviction-soft=memory.available<300Mi
--eviction-soft-grace-period=memory.available=1m30s
--eviction-max-pod-grace-period=30

Affinity and anti-affinity, past required vs. preferred

☺ Like you're 10: "Preferred" isn't a coin flip — every preference carries a weight, and the scheduler adds all of them up like a scorecard before picking a winner.

The mechanical difference between requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution is filter versus score, not a stronger versus weaker version of the same check. A required rule runs in the Filter extension point — fail it and the node is gone, full stop. A preferred rule runs in Score: each term carries a weight from 1 to 100, and for every node that passes filtering, the scheduler sums the weights of every preferred term that node happens to satisfy, then adds that sum into the node's overall score alongside NodeResourcesFit and the rest. A node can fail every single preferred term and still win, if its resource-fit score is high enough — "preferred" is a vote, not a gate. The IgnoredDuringExecution half of both names is doing real work too: neither kind of rule is re-checked once the Pod is running, so a node's labels changing out from under an already-placed Pod is silently ignored — Kubernetes will not evict a Pod just because its affinity stopped matching after the fact.

Inter-pod affinity and anti-affinity (podAffinity/podAntiAffinity) cost more than node affinity for a structural reason: a node-affinity check only reads one node's own labels, an O(1) lookup, while a pod-affinity check has to find and count other Pods matching a label selector across the whole topology domain named by topologyKey — genuinely more expensive at scheduler scale, which is exactly why the CKA blueprint's canonical anti-affinity example (spread replicas one-per-node with topologyKey: kubernetes.io/hostname) is a required rule kept deliberately small in scope. A newer field, matchLabelKeys, fixes a real rolling-update bug: without it, a Deployment's anti-affinity rule matching on app: checkout sees both the old and new ReplicaSet's Pods as "the same app" during a rollout and can refuse to schedule the new revision anywhere the old one still runs. Adding pod-template-hash to matchLabelKeys makes the affinity comparison implicitly scope itself to Pods from the same ReplicaSet revision only.

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 80                          # summed into node score if satisfied
      podAffinityTerm:
        labelSelector:
          matchLabels: {app: checkout}
        matchLabelKeys: [pod-template-hash]   # don't compare across rollout revisions
        topologyKey: topology.kubernetes.io/zone
    - weight: 20
      podAffinityTerm:
        labelSelector:
          matchLabels: {app: checkout}
        topologyKey: kubernetes.io/hostname

Taints and tolerations the cluster writes for you

☺ Like you're 10: Not every taint comes from a human typing kubectl taint — the cluster itself slaps a taint on a node the moment something's visibly wrong with it.

Workloads & Scheduling covers the three effects — NoSchedule, PreferNoSchedule, NoExecute — from the perspective of a taint you set on purpose. In practice, most of the taints doing real work in a running cluster are ones nobody typed: the node controller and kubelet apply a fixed set of node.kubernetes.io/-prefixed taints automatically the instant a condition is detected — not-ready and unreachable (both NoExecute, applied when a Node's heartbeat Lease stops renewing), disk-pressure, memory-pressure, pid-pressure, and network-unavailable (all NoSchedule), plus unschedulable the moment you run kubectl cordon. A newer one, node.kubernetes.io/out-of-service, exists specifically for a node that was shut down ungracefully — hardware failure, force-deleted VM — so StatefulSet Pods bound to it via PersistentVolumeClaims can be forcibly detached and rescheduled instead of waiting forever for a node that's never coming back.

The NoExecute effect carries one more knob most required-rule reading skips: tolerationSeconds. A Pod can tolerate not-ready or unreachable for a bounded window rather than forever — long enough to survive a brief network blip without every Pod on that node being evicted and rescheduled, but not so long that a genuinely dead node holds workloads hostage.

tolerations:
- key: node.kubernetes.io/not-ready
  operator: Exists
  effect: NoExecute
  tolerationSeconds: 60     # ride out a short blip; evict after 60s if still not-ready
- key: node.kubernetes.io/unreachable
  operator: Exists
  effect: NoExecute
  tolerationSeconds: 60

Every kubelet-managed Pod, and every DaemonSet Pod created through the default admission plugin, already carries built-in tolerations for most of these with a short tolerationSeconds — which is precisely how a log-collector or CNI DaemonSet keeps running on a node that just went NotReady, riding it out instead of getting evicted the instant the taint lands. Writing your own NoExecute toleration on an ordinary Deployment Pod without thinking hard about it is a genuine anti-pattern: it's the one place a small YAML addition can silently turn "this node is unreachable" into "this Pod just keeps not restarting anywhere else."

Topology spread constraints: skew, not just presence

☺ Like you're 10: Anti-affinity asks "is there already one of me here, yes or no" — topology spread asks "how lopsided would placing me here make things," and answers with a number.

topologySpreadConstraints solves a problem pod anti-affinity answers only crudely: keeping replicas evenly spread across a topology domain, not just merely present in more than one. Each constraint names a topologyKey (commonly topology.kubernetes.io/zone or kubernetes.io/hostname), a maxSkew — the maximum allowed difference between the domain with the most matching Pods and the domain with the fewest — and a whenUnsatisfiable policy: DoNotSchedule makes the constraint a hard filter, same weight as a required affinity rule; ScheduleAnyway demotes it to a scoring preference, same weight-based logic as preferred affinity. Unlike anti-affinity's binary "matches or doesn't," skew is recalculated for every candidate placement — the scheduler is directly answering "if I put this Pod in zone B, does max(counts) - min(counts) across all zones stay within maxSkew?"

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels: {app: checkout}
- maxSkew: 1
  topologyKey: kubernetes.io/hostname
  whenUnsatisfiable: ScheduleAnyway   # nice-to-have within a zone; not worth blocking on
  labelSelector:
    matchLabels: {app: checkout}

Priority and preemption

☺ Like you're 10: A higher-priority latecomer can bump a lower-priority Pod out of its seat — but only politely, by asking it to leave, never by throwing it overboard instantly.

Every Pod carries a priority, an int32 resolved at admission time from its priorityClassName — a vocabulary Kubernetes inherited straight from Google's internal Borg system, mentioned back in What Is Kubernetes, and Why as one of the ideas Borg battle-tested first. A PriorityClass object's value can be anything up to 1,000,000,000 for ordinary workloads; the range above that is reserved for Kubernetes' own two built-in classes, system-cluster-critical (2,000,000,000) and system-node-critical (2,000,001,000), which only core cluster add-ons should ever use. At most one PriorityClass in the cluster may set globalDefault: true — every Pod that doesn't name one explicitly gets that value, and if none is marked default, unset Pods fall back to priority 0.

Preemption only activates as the PostFilter fallback from the pipeline earlier on this page — the scheduler always tries to place a Pod on genuinely free capacity first, and only reaches for preemption once zero nodes survive ordinary filtering. When it does, the DefaultPreemption plugin evaluates each infeasible node by asking "if I removed some lower-priority Pods here, would this Pod then fit?", and picks victims to minimize disruption — fewest Pods removed, lowest priority first, and it will not choose a set of victims that would violate a PodDisruptionBudget if any non-violating option exists. The chosen node is recorded on the pending Pod's status.nominatedNodeName, and victims are terminated gracefully, respecting their own terminationGracePeriodSeconds — which means preemption is not instant. Nothing stops a different, equal-or-higher-priority Pod from grabbing that freed capacity in the gap between eviction and rebind; the nomination is a strong hint to the scheduler on retry, not a lock.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: checkout-critical
value: 100000
globalDefault: false
preemptionPolicy: PreemptLowerPriority   # set to "Never" to keep the priority
description: "Customer-facing checkout path — may preempt lower-priority batch work."

Beyond the default scheduler

☺ Like you're 10: When one-Pod-at-a-time placement isn't the right shape for the job, you don't fight the built-in scheduler — you run a second one, side by side with it, just for that job.

Every Pod's spec.schedulerName defaults to default-scheduler, but nothing requires that — you can deploy a second scheduler binary into the cluster under a different name, and any Pod naming it in schedulerName is picked up by that scheduler instead, completely independently of the default one. This is how batch and ML workload schedulers like Volcano and Kueue plug in: the default scheduler's one-Pod-at-a-time model has no concept of gang scheduling — placing all N Pods of a distributed training job together, or none at all, so you never end up with 7 of 8 workers running and burning GPU-hours while the eighth sits Pending forever. Kueue in particular layers job-level queueing and quota management on top of the existing scheduler rather than replacing it outright, and belongs in the same conversation as everything else CNCF's ecosystem builds around Kubernetes' extension points — see The CNCF Project Landscape for where projects like it sit relative to the core project.