Karpenter

Karpenter is a node autoscaler that throws away the idea of node groups. It watches the pods your scheduler could not place, works out the cheapest machine in the entire cloud catalogue that would fit them, and launches that exact machine straight from the provider API — usually in under a minute. Then it keeps going: it continuously asks whether the same pods could run on fewer or cheaper nodes, and repacks them if they can. It solves the platform problem of “we are paying for half-empty nodes of the wrong shape, and new capacity takes five minutes to arrive.”

What Karpenter is and the problem it solves

☺ Like you’re 10: It’s a robot that looks at the pods nobody could find room for, and rents the exact right computer to hold them.

Karpenter was built at AWS, open-sourced in 2021, and donated to the Kubernetes project in 2023 — it now lives as a provider-neutral core in kubernetes-sigs/karpenter with separate cloud providers layered on top. It reached a stable v1 API in 2024. The AWS provider is by far the most mature; the Azure provider is what powers AKS Node Auto Provisioning; providers for Cluster API and several other clouds exist at varying maturity.

The node-group problem

The classic Cluster Autoscaler does not really scale nodes. It scales node groups — an AWS Auto Scaling Group, a GCP managed instance group, an Azure VM scale set — each of which is homogeneous by construction: one instance type, one AMI, one set of labels and taints. To make that model flexible, platform teams end up hand-maintaining a zoo of groups: on-demand-small, on-demand-large, spot-compute, spot-memory, gpu-a10, arm64-general, one per availability zone, times three environments. Every new workload shape means a new group and a Terraform change. And because the Cluster Autoscaler only knows “add one more of this,” a pod requesting 2 CPU and 30 GiB of memory will happily boot a node with 16 spare cores that nobody will ever use.

The latency hurts too. The Cluster Autoscaler simulates scheduling, decides which group to grow, changes the group’s desired count, then waits for the group’s own machinery to launch an instance — typically several minutes, in a loop that runs on a fixed interval. For a batch queue that spikes at 09:00, that is a long time to watch a Pending pod.

How Karpenter reframes it

Karpenter deletes the intermediate abstraction. There are no groups; there is a NodePool that expresses constraints — “anything in the c, m or r families, generation 5 or newer, amd64 or arm64, spot or on-demand, in these three zones” — and Karpenter chooses within them, per decision, based on what is actually pending. Given a batch of unschedulable pods it does the bin-packing arithmetic itself: which single instance type, or which small set of types, holds this batch most cheaply while satisfying every pod’s requests, node selectors, affinities, topology spread constraints and tolerations. Then it creates a NodeClaim, calls the provider’s fleet API directly, and the node registers and starts taking pods — commonly in 30 to 60 seconds.

The second half is what earns the money. Karpenter does not stop after provisioning; it runs a continuous disruption loop that looks for nodes to remove or replace. Empty nodes get deleted. Underutilised nodes get their pods repacked onto fewer nodes. An expensive node gets swapped for a cheaper type that still fits. This is bin-packing as an ongoing cost control rather than a one-off placement decision, and it is why the tool shows up in every FinOps conversation.

What Karpenter deliberately is not

Karpenter does not scale pods — that is the HPA, the VPA and KEDA, which decide how much pod you need while Karpenter makes sure a node exists for it. It is not a scheduler either: kube-scheduler still binds every pod, and Karpenter only reads the pending queue and changes the capacity underneath it. Nor is it a cluster lifecycle tool — clusters, control planes and networks are Cluster API and Crossplane territory. And it is cloud-specific by design: the core is generic, but you always run a provider that knows how to buy machines somewhere particular.

Where Karpenter fits in a platform

☺ Like you’re 10: It lives in the basement of the platform. Nobody using the platform sees it — they just never wait for a computer.

In the reference architecture Karpenter sits in the compute substrate, alongside the Kubernetes control plane itself. It is unambiguously platform-team property: a single misconfigured NodePool can bankrupt a budget or boot a node with the wrong IAM role, so tenants should never be able to write one. What tenants do get is the effect — capacity that appears when they need it — plus, at most, the ability to select a NodePool by label or toleration. That split is a textbook example of the self-service boundary: expose the outcome, own the mechanism.

Its neighbours, and who does what

Karpenter’s manifests are ordinary Kubernetes objects delivered through GitOps like anything else — and a NodePool change being a pull request matters enormously, because that change can silently replace every node in the cluster. Prometheus scrapes Karpenter’s own metrics. OpenCost reads the karpenter.sh/capacity-type node label to price spot correctly, and the idle line OpenCost exposes is exactly what consolidation attacks. KEDA is the natural partner upstream: KEDA scales a queue consumer from zero to fifty pods, Karpenter materialises the nodes they need, and a queue that is empty overnight costs almost nothing. And PodDisruptionBudgets — the reliability primitive — are the brake that stops consolidation turning into an outage.

CNPE relevance

Be clear-eyed here: Karpenter is not on the official CNPE tool list. You will not be asked to install it or to recite its API. But the concepts it embodies are squarely in the exam’s Platform Architecture and cost material — node autoscaling, bin-packing, spot capacity, right-sizing, the relationship between pod autoscaling and node autoscaling. The likely shape of an exam encounter is conceptual: distinguishing pod scaling from node scaling, explaining what makes a pod stay Pending, or reasoning about why tight packing saves money and costs resilience. The primary lesson is Scaling & Scheduling; treat this page as the depth behind one paragraph of it, and see the tool landscape for where it sits among the named projects.

How it works — architecture, components, CRDs

☺ Like you’re 10: One program runs two loops forever: “does anything need a computer?” and “can I get rid of a computer?”

Karpenter is a single controller Deployment, conventionally in kube-system, running two or three replicas with leader election. It watches Pods, Nodes and its own custom resources, and talks to the cloud provider’s API. One chicken-and-egg problem is worth knowing: Karpenter cannot provision the node it runs on, so it normally sits on a small managed node group or on Fargate.

The three custom resources

Karpenter’s API is unusually small, which is a large part of its appeal. Two objects you write, one the controller writes for you.

The split matters conceptually. A NodePool is portable across clouds; a NodeClass is where the provider-specific mess is quarantined. Note also that a NodeClaim is a real object you can inspect — which makes debugging Karpenter far more pleasant than debugging an Auto Scaling Group.

Requirements, well-known labels and scheduling

A NodePool’s requirements are written in the same grammar as node affinity — a key, an operator (In, NotIn, Exists, DoesNotExist, Gt, Lt — Gt and Lt take exactly one numeric value) and values. The keys are standard Kubernetes labels plus Karpenter’s own: kubernetes.io/arch, kubernetes.io/os, topology.kubernetes.io/zone, node.kubernetes.io/instance-type, karpenter.sh/capacity-type, and provider keys such as karpenter.k8s.aws/instance-category, instance-family, instance-generation, instance-cpu and instance-memory. Crucially, the pod’s own constraints intersect with the NodePool’s: a pod that demands arm64 will only ever be provisioned onto a node from a NodePool whose requirements permit arm64. If the intersection is empty, no node is created and the pod stays Pending — which is the single most common “why is nothing happening?” cause.

The four disruption reasons

Everything Karpenter does to an existing node has a named reason, and it is worth memorising all four because they behave differently.

Alongside these sits interruption handling: given a queue of provider events (on AWS, an SQS queue fed by EventBridge), Karpenter receives the spot two-minute warning, rebalance recommendations, instance state changes and scheduled maintenance notices, and proactively cordons and drains the doomed node so pods move before the machine vanishes. Without that plumbing configured, spot reclamation is an abrupt kill. Consolidation is also always replacement-first for non-empty nodes: the new node is up and ready before the old one is drained, so capacity never dips.

The resources you will actually write

☺ Like you’re 10: Two files say “here’s what kinds of computers you may rent” and “here’s how to set one up”. A third bit is what app teams write to say “please don’t move me.”

A general-purpose, spot-first NodePool

This is the object you tune most. Read the requirements as a fence, not a preference: they say what Karpenter may choose, and it picks the cheapest fit inside the fence. Excluding small sizes and old generations is standard practice — tiny nodes waste capacity on daemonsets, and old generations are usually worse price-performance.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  weight: 10                  # higher weight wins when several NodePools could serve a pod
  limits:
    cpu: "2000"               # HARD CEILING for this pool — your blast radius on cost
    memory: 8000Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized   # the money setting
    consolidateAfter: 1m      # required in v1: wait this long before acting
    budgets:
      - nodes: "10%"          # never disrupt more than 10% of this pool at once
      - nodes: "0"            # ...and nothing at all during the Monday release window
        schedule: "0 9 * * mon"
        duration: 4h
        reasons: [Drifted, Underutilized]
  template:
    metadata:
      labels:
        platform.acme.dev/pool: general
    spec:
      expireAfter: 720h       # rotate every 30 days so nodes stay patched
      terminationGracePeriod: 1h   # HARD cap: after this, pods go regardless of PDBs
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]     # spot preferred; on-demand is the fallback
        - key: kubernetes.io/arch
          operator: In
          values: ["arm64", "amd64"]        # let Graviton win on price when it can
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]                     # nothing older than gen 6
        - key: karpenter.k8s.aws/instance-size
          operator: NotIn
          values: ["nano", "micro", "small"]   # daemonsets would eat the whole node
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["eu-west-1a", "eu-west-1b", "eu-west-1c"]

The EC2NodeClass — where the cloud specifics live

Three selectors do the heavy lifting, and all three are the usual source of a failed launch. Subnets and security groups are discovered by tag, which is why the cluster’s networking must carry the discovery tag; the role is the IAM role the node assumes, and it must already be permitted to join the cluster.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  role: "KarpenterNodeRole-acme-prod"   # node IAM role; must be mapped into the cluster
  amiSelectorTerms:           # required in v1 — an alias tracks an EKS-optimised AMI family
    - alias: al2023@latest    # Pin a published version to opt out of AMI drift:
                              #   alias: al2023@<published-ami-version>
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "acme-prod"   # THIS TAG MUST EXIST on the subnets
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "acme-prod"
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        encrypted: true
        deleteOnTermination: true
  metadataOptions:
    httpTokens: required      # IMDSv2 only — see the security-policy lesson
    httpPutResponseHopLimit: 1   # keeps IMDS out of reach of ordinary (non-hostNetwork) pods
  tags:
    team: platform
    cost-centre: "4471"       # so the cost tooling can attribute these instances

What application teams write: protecting a workload

Tenants never touch a NodePool. What they can and should do is tell Karpenter what their workload tolerates. Three mechanisms cover almost everything, and the exam-adjacent point is that the workload owns its own disruption tolerance.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 6
  selector:
    matchLabels: { app: checkout }   # required — and it must match the template labels
  template:
    metadata:
      labels: { app: checkout }      # what the PDB and the spread constraint select on
      annotations:
        # Karpenter will not voluntarily disrupt a node running this pod.
        # Use it for singleton jobs and long migrations — NOT as a blanket default.
        karpenter.sh/do-not-disrupt: "true"
    spec:
      terminationGracePeriodSeconds: 30   # long values here stall every repack
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway   # Karpenter honours DoNotSchedule too, but a
                                              # spread it cannot satisfy leaves pods Pending
          labelSelector:
            matchLabels: { app: checkout }
      containers:
        - name: app
          image: ghcr.io/acme/checkout:1.4.3
          resources:
            requests: { cpu: 500m, memory: 512Mi }   # Karpenter sizes nodes from THESE
---
apiVersion: policy/v1
kind: PodDisruptionBudget       # the brake: Karpenter honours this during consolidation
metadata:
  name: checkout
spec:
  minAvailable: 4               # never let a repack take us below four ready pods
  selector:
    matchLabels: { app: checkout }

One line in there deserves emphasis: Karpenter sizes nodes from resource requests, not from usage. A fleet of pods requesting four cores and using half of one will make Karpenter faithfully buy an enormous, expensive, idle cluster. Right-sizing requests is therefore a prerequisite for Karpenter saving you anything, not an optional follow-up — which is precisely the gap OpenCost exists to show you.

Day-to-day commands

☺ Like you’re 10: Mostly you ask “what did it decide, and why?” The answer is in one log stream and one list of NodeClaims.

Seeing what exists and what it chose

kubectl get nodepool
kubectl get ec2nodeclass
kubectl get nodeclaim -o wide        # the real audit trail: type, zone, capacity, node
kubectl describe nodeclaim <name>    # launch errors live in the conditions and events

# What did Karpenter actually buy, and how much of it is spot?
kubectl get nodes -L karpenter.sh/nodepool,karpenter.sh/capacity-type,node.kubernetes.io/instance-type

# Are we near the NodePool ceiling? (limits vs resources in the status)
kubectl get nodepool general -o jsonpath='{.status.resources}{"\n"}'

# Which nodes are Karpenter's at all — its nodes carry an ownership label.
kubectl get nodes -l karpenter.sh/nodepool --no-headers | wc -l

Reading the decisions

Karpenter’s controller log is unusually good: it states in plain text which pods it considered, which instance type it chose, and — when it declines — why. Ninety percent of debugging is this one command.

kubectl -n kube-system logs deploy/karpenter -f
kubectl -n kube-system logs deploy/karpenter | grep -i 'launched\|disrupt\|drift\|error'

# Why is this pod still Pending? Karpenter emits events onto the POD itself.
kubectl describe pod <pod> | sed -n '/Events/,$p'
kubectl get events --field-selector reason=FailedScheduling -A

# Force a node out (drains, respects PDBs, then terminates the instance).
kubectl delete nodeclaim <name>      # correct way — never terminate in the cloud console

# Temporarily stop all voluntary disruption on one node.
kubectl annotate node <node> karpenter.sh/do-not-disrupt=true

# Prove a NodePool works: scale something oversized and watch a node appear.
kubectl create deploy inflate --image=public.ecr.aws/eks-distro/kubernetes/pause:3.7
kubectl set resources deploy inflate --requests=cpu=1
kubectl scale deploy inflate --replicas=12 && kubectl get nodeclaim -w

Karpenter’s controller also serves a Prometheus /metrics endpoint — the port, the Service and the optional ServiceMonitor are set by its Helm chart — covering node and NodeClaim counts broken down by instance type and capacity type, scheduling and provisioning durations, voluntary disruption decisions by reason, and pod scheduling state. Alerting on “pods pending for longer than five minutes while NodeClaims are failing” is the alert that catches most real incidents; the wider pattern is in the command reference and Prometheus.

Gotchas and failure modes

☺ Like you’re 10: Here are the ways it quietly does nothing, or noisily does far too much.

Nothing is provisioned and the pod just sits there

Work the chain in order. Is the pod actually unschedulable for a capacity reason, or is it blocked by something Karpenter cannot fix — an unbound PVC, a missing image pull secret, a node affinity no NodePool can satisfy? Does any NodePool’s requirements intersect the pod’s own nodeSelector, affinity and tolerations? If your NodePool stamps a taint, the pod needs the matching toleration or it will never land there. Has the NodePool hit its limits (check .status.resources against .spec.limits)? Does the pod request more of anything than the largest permitted instance type provides — a 400 GiB memory request against a family capped at 128 GiB will simply never be satisfiable. Finally, read the events on the pod: Karpenter writes its refusal there in words. Workload triage walks the same ladder for pods that never go ready for other reasons.

Consolidation never happens

Three blockers account for nearly all of it. First, PodDisruptionBudgets that can never be satisfied — a minAvailable equal to the replica count, or a PDB on a single-replica Deployment, means no eviction is ever permitted and the node it sits on becomes immortal. Second, long terminationGracePeriodSeconds: a workload with a one-hour grace period turns every repack into an hour-long stall, and a fleet of them means consolidation effectively never converges. Third, karpenter.sh/do-not-disrupt used as a default — one team adds it to a base template, it propagates to forty deployments, and your cluster quietly stops consolidating while everyone congratulates themselves on the annotation. Diagnose by looking for the disruption decisions in the controller log; it says which pod blocked the eviction.

Infrastructure prerequisites that fail silently-ish

Two setup mistakes dominate. Subnet and security-group tags: the selectors match on tags, so a new subnet added without the karpenter.sh/discovery tag is invisible, and Karpenter keeps packing the zones it can see while one stays permanently empty. IAM: the controller needs permission to run and terminate instances, pass the node role and read pricing; the node role needs cluster-join permissions and must be mapped into the cluster’s auth configuration, or the instance boots, costs money, and never registers — the worst failure mode, because you pay for nodes that do nothing. Graceful spot handling additionally needs the interruption queue and its EventBridge rules. Check the troubleshooting playbook for the wider decision tree, and security & policy for why httpPutResponseHopLimit: 1 is not optional.

Churn as a reliability problem

Consolidation plus spot plus expiry means pods move — often. That is fine for a stateless web service and hostile to a long-running batch job, a stateful set with slow-attaching volumes, or anything holding a lease. The mitigations are ordinary Kubernetes: PDBs sized honestly, graceful shutdown that actually works, topology spread, and a separate conservative NodePool (on-demand only, long consolidateAfter) for workloads that genuinely cannot be moved. Aim the aggressive pool at the eighty percent that shrug.

Alternatives and when to choose it

☺ Like you’re 10: Other ways to get more computers exist. Some are simpler, some are dumber, one is “don’t have computers at all.”

The honest framing is that node autoscaling is a spectrum from “I manage the machines” to “I have no idea what a machine is,” and Karpenter sits nearer the far end without going all the way.

The comparison that decides it

A practical rule

If your cluster is on AWS or AKS, your workloads are stateless-ish, and someone has ever complained about the compute bill, Karpenter is the default answer — start with two NodePools (one aggressive spot pool, one conservative on-demand pool), fence the requirements tightly, set limits, and only then turn on WhenEmptyOrUnderutilized. If you are on-prem, on an unsupported provider, or your workloads are long-running and move badly, the Cluster Autoscaler remains a perfectly respectable choice and nobody should be embarrassed by it. And remember the ordering that platform architecture insists on: right-size your requests first. An autoscaler faithfully provisioning capacity for requests nobody uses is an expensive way to automate waste.

Exam relevance and going further

☺ Like you’re 10: Karpenter itself isn’t on the exam list, but why it exists definitely is — and you can’t look up its website on the day.

Karpenter is not on the official CNPE tool list, so do not spend memorisation budget on its YAML the way you would for a GitOps or observability manifest. What is examinable is the surrounding concept space: node autoscaling versus pod autoscaling, why a pod is Pending, what bin-packing buys and costs, spot capacity trade-offs, and how cost decisions and reliability decisions are the same decision viewed from two sides. Karpenter is the clearest worked example of all of those, which is why it earns a page.

The documentation allowlist — read this twice

⚖ CNPA vs CNPE — That allowlist is a CNPE mechanic — a performance exam can afford to let you open a narrow set of docs mid-task. CNPA offers no such allowance: it is fully closed-book multiple-choice, with zero external lookups of any kind, on kubernetes.io or otherwise. Karpenter itself won’t appear by name, but the concept-level knowledge this page drills — node autoscaling versus pod autoscaling, why bin-packing has trade-offs, why sizing follows requests — is still worth holding in your head for CNPA’s closed-book recall.

What to be able to do without notes

Explain in two sentences how Karpenter differs from the Cluster Autoscaler: groupless just-in-time provisioning of right-sized nodes chosen from the whole catalogue, plus a consolidation loop that repacks onto fewer or cheaper nodes — against fixed homogeneous node groups that only grow and shrink. Name the two objects you write (NodePool, NodeClass) and the one Karpenter writes (NodeClaim). List the four disruption reasons: empty, underutilised, drifted, expired. State the three brakes: PodDisruptionBudgets, the karpenter.sh/do-not-disrupt annotation, and NodePool disruption budgets. Say why an unsatisfiable PDB or a long terminationGracePeriodSeconds stalls consolidation. And be able to say, without hedging, that node sizing follows resource requests, not usage — the sentence that connects this page to FinOps.

Official resources for after the exam

Outside the exam the canonical sources are karpenter.sh/docs (the Concepts and Troubleshooting sections are short and unusually honest), the provider-neutral core at github.com/kubernetes-sigs/karpenter, the AWS provider at aws/karpenter-provider-aws, the Azure provider behind AKS Node Auto Provisioning at Azure/karpenter-provider-azure, and — for everything that is actually examinable — kubernetes.io on scheduling and eviction. Pair this page with Scaling & Scheduling for the full autoscaling picture, FinOps and OpenCost for the money, Reliability & Incidents for the disruption side, The Tool Landscape for where it sits among the named projects, and the glossary when a term stops making sense.