Karpenter

Karpenter is a node autoscaler that answers a question the Cluster Autoscaler never asks: not "which of my pre-built node groups should I grow," but "what is the single cheapest machine, chosen from the entire cloud catalogue, that fits the Pods stuck Pending right now?" It watches for unschedulable Pods, works out an exact-fit instance in seconds, and calls the cloud provider's API directly to launch it — no Auto Scaling Group or VM scale set standing in between. Then it keeps working after the Pods land: a continuous consolidation loop keeps asking whether the cluster's existing Pods would fit on fewer or cheaper nodes, and repacks them when they would. This page covers what changes when you adopt that model: the philosophy shift itself, the architecture behind it, the NodePool and NodeClass objects you write by hand, the four reasons Karpenter will ever touch a running node, day-to-day commands, the gotchas that bite in production, and how it stacks up against the Cluster Autoscaler and the alternatives around it.

The philosophy shift: pre-provisioned groups vs. just-in-time capacity

☺ Like you're 10: One approach keeps a shelf of ready-made box sizes and picks the closest; the other custom-orders the exact size every single time.

Autoscaling: HPA, VPA & Cluster Autoscaler and the Cluster Autoscaler tool guide cover the older model in depth, so only the shape of it belongs here: the Cluster Autoscaler scales node groups — an AWS Auto Scaling Group, a GCP managed instance group, an Azure VM scale set — each one homogeneous by construction, one instance type per group. To cover a real workload mix, platform teams end up hand-maintaining a small zoo of groups (on-demand-small, spot-compute, spot-memory, arm64-general, one set per zone, times every environment), and because the autoscaler can only grow whichever group's shape happens to fit, a Pod asking for 2 CPU and 3 GiB routinely lands a node with a dozen spare cores nobody will ever use.

Karpenter, donated to the Kubernetes project by AWS and now a provider-neutral core under kubernetes-sigs/karpenter with separate cloud providers layered on top (AWS is the most mature; Azure's provider is what powers AKS Node Auto Provisioning), removes the intermediate group entirely. You declare constraints — a NodePool — and for each batch of unschedulable Pods, Karpenter evaluates the whole permitted instance-type catalogue and launches whichever specific machine (or small set of machines) fits that batch most cheaply. It reached a stable v1 API in 2024.

Karpenter deliberately stays out of two neighboring jobs. It does not decide how many Pods you need — that's the HPA and VPA's territory, both covered on the autoscaling deep-dive — Karpenter only reacts once those decisions leave a Pod stuck Pending. And it does not bind Pods to nodes either; kube-scheduler still does that. Karpenter's entire job is changing what capacity exists for the scheduler to choose from.

Architecture: one controller, two loops

☺ Like you're 10: One program runs two errands forever — "does anyone need a locker?" and "can I return a locker nobody's using anymore?"

Karpenter runs as a Deployment — conventionally in kube-system, two or three replicas with leader election — watching Pods, Nodes, and its own custom resources, and talking directly to the cloud provider's fleet API. There's a genuine chicken-and-egg problem worth knowing up front: Karpenter cannot provision the node it itself runs on, so it normally sits on a small, statically-sized node group (or on Fargate, where that's available) that exists outside anything Karpenter manages.

Three custom resources carry the whole model, and only two of them are yours to write.

The split is deliberate: a NodePool reads the same way on every cloud, while a NodeClass is where the provider-specific mess is quarantined — swap the cloud, swap the NodeClass, keep the NodePool. And because a NodeClaim is a real, inspectable object rather than an opaque cloud-console entry, debugging a failed launch means kubectl describe, not a support ticket.

NodePool and NodeClass: the objects you actually write

☺ Like you're 10: One file says "here's what kinds of computers you're allowed to rent," a second says "and here's exactly how to set one up on this particular cloud."

Treat a NodePool's requirements as a fence, not a preference — they say what Karpenter may choose, and inside that fence it always picks the cheapest fit. Excluding tiny sizes and old instance generations is standard practice: a nano or micro node wastes most of its capacity on DaemonSets alone, and older generations are almost always worse price-for-performance than their replacement.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  weight: 10                    # higher weight wins when several NodePools could serve a Pod
  limits:
    cpu: "1000"                 # hard ceiling for this pool — your blast radius on cost
    memory: 4000Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized   # the setting that actually saves money
    consolidateAfter: 1m        # wait this long before acting on a consolidation opportunity
    budgets:
      - nodes: "10%"            # never disrupt more than 10% of this pool at once
      - nodes: "0"               # ...and nothing at all during the weekly deploy window
        schedule: "0 14 * * tue"
        duration: 2h
        reasons: [Drifted, Underutilized]
  template:
    metadata:
      labels:
        team.example.com/pool: general
    spec:
      expireAfter: 720h          # rotate every 30 days so nodes keep getting patched
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]      # spot preferred; on-demand is the fallback
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["4"]                       # nothing older than generation 5
        - key: karpenter.k8s.aws/instance-size
          operator: NotIn
          values: ["nano", "micro", "small"]   # DaemonSets alone would eat these

The NodeClass is where the AWS-specific (or Azure-, or GCP-specific) plumbing lives — three selectors do the heavy lifting, and all three are the usual cause of a launch that never happens.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  role: "KarpenterNodeRole-example"     # node IAM role; must already be mapped into the cluster
  amiSelectorTerms:
    - alias: al2023@latest              # pin a published version to opt out of automatic drift
  subnetSelectorTerms:
    - tags: { karpenter.sh/discovery: "example-prod" }   # subnets MUST carry this tag
  securityGroupSelectorTerms:
    - tags: { karpenter.sh/discovery: "example-prod" }
  metadataOptions:
    httpTokens: required                # IMDSv2 only
    httpPutResponseHopLimit: 1          # keeps the instance metadata service out of reach of Pods

What your own workloads write is smaller: nothing about capacity, only how much disruption they can tolerate. A PodDisruptionBudget is the brake Karpenter always honors during consolidation, and karpenter.sh/do-not-disrupt: "true" on a Pod's template stops Karpenter from voluntarily touching the node it's on at all — reach for it on a singleton job or a long migration, never as a blanket default, because a fleet where every Pod opts out is a fleet that has quietly stopped consolidating.

Consolidation and the four disruption reasons

☺ Like you're 10: Karpenter never touches a running node for no reason — it always has exactly one of four named excuses, and each behaves a little differently.

Alongside those four sits interruption handling: given a stream of provider events (on AWS, typically an SQS queue fed by EventBridge), Karpenter receives a spot two-minute warning, a rebalance recommendation, or a scheduled-maintenance notice, and proactively cordons and drains the doomed node before the machine actually disappears — without that plumbing, a spot reclaim is an abrupt kill instead of a graceful one.

Day-to-day commands

☺ Like you're 10: Mostly you're asking "what did it decide, and why?" — and the honest answer is always in one log stream and one list of NodeClaims.

# what exists, and what Karpenter actually chose
$ kubectl get nodepool
$ kubectl get nodeclaim -o wide                          # the real audit trail: type, zone, capacity, node
$ kubectl describe nodeclaim <name>                       # launch errors live in conditions and events
$ kubectl get nodes -L karpenter.sh/nodepool,karpenter.sh/capacity-type,node.kubernetes.io/instance-type

# is a NodePool near its cost ceiling?
$ kubectl get nodepool general-purpose -o jsonpath='{.status.resources}{"\n"}'

# read the decisions — the controller log states in plain text what it considered and why
$ kubectl -n kube-system logs deploy/karpenter -f
$ kubectl -n kube-system logs deploy/karpenter | grep -i 'launched\|disrupt\|drift'

# why is this Pod still Pending? Karpenter writes events onto the Pod itself
$ kubectl describe pod <pod> | grep -A5 Events
$ kubectl get events --field-selector reason=FailedScheduling -A

# the correct way to force a node out — drains, respects PDBs, then terminates the instance
$ kubectl delete nodeclaim <name>                         # never terminate it from the cloud console

# prove a NodePool actually works
$ kubectl create deploy inflate --image=registry.k8s.io/pause:3.9
$ kubectl set resources deploy inflate --requests=cpu=1
$ kubectl scale deploy inflate --replicas=12 && kubectl get nodeclaim -w

Gotchas and failure modes

☺ Like you're 10: Most Karpenter surprises aren't bugs — they're the fence you drew, or forgot to draw, doing exactly what it says.

• Node sizing follows requests, never usage. A fleet of Pods requesting four cores and actually using half of one will make Karpenter faithfully buy an enormous, expensive, idle cluster. Right-sizing requests — the subject of Scheduling & Resource Management — is a prerequisite for Karpenter saving you anything, not an optional follow-up; Cost & FinOps on Kubernetes covers exactly this gap.
• An unsatisfiable PDB, a long terminationGracePeriodSeconds, or a blanket do-not-disrupt silently stops consolidation. A minAvailable equal to the replica count means no eviction is ever permitted; a workload with a one-hour grace period turns every repack into an hour-long stall; and do-not-disrupt copy-pasted into a shared base template propagates to every Deployment built from it. The controller log names the blocking Pod every time — read it before assuming Karpenter is broken.
• Missing discovery tags make a whole zone invisible. Subnet and security-group selectors match on tags, not on names — a new subnet added without the karpenter.sh/discovery tag is simply never considered, and Karpenter keeps packing the zones it can see while one sits permanently empty.
• Over-broad requirements pick surprising instances. A NodePool that permits every family and every size is "maximum flexibility" on paper and a trap in practice — Karpenter optimizes purely for price-per-fit, so it will happily launch a burstable instance whose CPU credits run out mid-load, or a network-optimized shape your CNI has never been tested against. Fence the categories, exclude the sizes you don't want, and keep specialized hardware in its own tainted NodePool.
• It cannot bootstrap itself. Karpenter can't provision the node its own controller runs on — that node has to come from somewhere static and outside its management, or the very first failure mode you hit is Karpenter itself stuck Pending.

Karpenter vs. the alternatives

☺ Like you're 10: A few other ways exist to make sure there's always a computer waiting — some simpler, some that hand the whole job to someone else entirely.

A practical rule: if your cluster runs on AWS or AKS, your workloads are mostly stateless, and anyone has ever questioned the compute bill, Karpenter is the reasonable default — start with a spot-first pool and a conservative on-demand pool, fence the requirements tightly, set limits, and only then turn on WhenEmptyOrUnderutilized. If you're on-prem or on a provider Karpenter doesn't support well, the Cluster Autoscaler remains a perfectly respectable choice. Karpenter itself doesn't sit on the core CKA Workloads & Scheduling blueprint, but the concepts underneath it — Pending Pods, bin-packing, spot capacity, request-driven sizing — do; and if you're studying toward Platform Engineering's own certification track, its Karpenter page goes to CNPE implementation depth this page deliberately stops short of, with the wider ladder of certifications mapped on the sibling Golden Astronaut course.