Cluster Autoscaler

Cluster Autoscaler answers exactly one question: is there a Pod sitting Pending because nothing in the cluster has room for it, and if so, which node group should grow to fix that. It does not watch CPU or memory utilization to decide when to add a node — a cluster running at 95% average utilization with every Pod successfully placed triggers nothing, while a cluster running at 5% utilization with one unschedulable Pod triggers a scale-up within one scan interval. This course's autoscaling deep-dive already covers that theory — the Pending signal, the scaling formula for its HPA/VPA neighbors, and the expander strategies. This page is the operator's page: the Deployment, IAM policy, and RBAC you actually write to run it against AWS, GCP, or Azure; the tags that wire up node-group auto-discovery, including the extra ones scale-from-zero needs; the PodDisruptionBudgets, affinity rules, and annotations that decide whether a quiet node is genuinely safe to remove; the commands you reach for when a scale-up stalls; and the version-skew, IAM, and priority-expander mistakes that quietly cost real money in production.

What triggers a scale-up, and what deliberately doesn't

☺ Like you're 10: One specific kind of "this car has nowhere to park" — never "the garage feels busy."

The trigger is narrow on purpose: a Pod stuck in Pending because kube-scheduler's filtering phase eliminated every existing node on resource or constraint grounds. Cluster Autoscaler discovers this by polling the API server on a fixed interval — --scan-interval, 10 seconds by default — for Pods without a bound node. It never inspects raw CPU or memory utilization to decide whether to grow anything; a Pod that fails to schedule because it wants a GPU nobody has, a toleration nobody offers, or a topology spread constraint nothing currently satisfies is just as much a trigger as one that simply wants more CPU than any node has free. That single-signal design is also its main limitation: Cluster Autoscaler is purely reactive. It never provisions ahead of demand or holds idle headroom for a spike it hasn't seen yet — a pattern GitOps/CI, load-testing, or the HPA behavior policies covered on the deep-dive page have to compensate for separately if instant capacity actually matters.

Architecture: the scan loop, the cloud-provider interface, and node groups

☺ Like you're 10: One program on the outside of the cluster, talking to two different worlds at once: the API server on one side, the cloud's own scaling knobs on the other.

Cluster Autoscaler runs as a single controller Pod — usually a Deployment with one replica in kube-system, protected by the same leader-election Lease mechanism as any other Kubernetes controller. It has no in-cluster server component and no CRD of its own; everything it needs comes from two sources it bridges together. On one side, it lists and watches Pods and Nodes through the ordinary Kubernetes API — exactly the same client-go machinery the controller pattern already covers. On the other side, it talks to your cloud through a cloud-provider interface: a small Go abstraction — NodeGroup, with methods like TargetSize(), IncreaseSize(), and TemplateNodeInfo() — that each supported cloud (AWS, GCP, Azure, and a dozen smaller ones) implements against its own primitive: an Auto Scaling Group on AWS, a Managed Instance Group on GCP, a Virtual Machine Scale Set on Azure.

That simulation step is the reason Cluster Autoscaler isn't just "increment the ASG by one." For every node group it's allowed to grow, it builds a synthetic Node from that group's launch template and re-runs the scheduler's own filtering predicates against it, in-process — checking resource fit, taints/tolerations, node affinity, and topology constraints exactly as kube-scheduler would. Only a node group whose template actually passes that simulation is a candidate at all; when more than one qualifies, the --expander flag (covered on the deep-dive page — least-waste, priority, most-pods, price, or random) picks the winner.

The Deployment, RBAC, and IAM you actually write

☺ Like you're 10: The attendant needs a badge that lets them see the whole garage, and a phone number that actually reaches the lot next door.

Deploying Cluster Autoscaler yourself — as opposed to a cloud's managed add-on, covered further down — means writing three things: the controller Deployment itself, a ClusterRole broad enough to read Nodes/Pods/PDBs/PVCs/StorageClasses cluster-wide and patch Node taints, and cloud-side credentials scoped to exactly the Auto Scaling actions it needs. On AWS, that last part is almost always IAM Roles for Service Accounts (IRSA), the same pattern RBAC & Admission Control covers for workload identity generally — never a static access key baked into a Secret.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: cluster-autoscaler
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/cluster-autoscaler
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
  labels: { app: cluster-autoscaler }
spec:
  replicas: 1
  selector: { matchLabels: { app: cluster-autoscaler } }
  template:
    metadata:
      labels: { app: cluster-autoscaler }
      # CA should never evict its own single replica while deciding what ELSE to evict
      annotations: { cluster-autoscaler.kubernetes.io/safe-to-evict: "false" }
    spec:
      serviceAccountName: cluster-autoscaler
      containers:
      - name: cluster-autoscaler
        # pin the MINOR version to match your control plane — see gotchas below
        image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.30.1
        command:
        - ./cluster-autoscaler
        - --cloud-provider=aws
        - --namespace=kube-system
        - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-cluster
        - --balance-similar-node-groups
        - --skip-nodes-with-system-pods=true
        - --scale-down-utilization-threshold=0.5
        - --scale-down-unneeded-time=10m
        - --expander=priority
        resources:
          requests: { cpu: 100m, memory: 300Mi }
        volumeMounts:
        - { name: ssl-certs, mountPath: /etc/ssl/certs/ca-certificates.crt, readOnly: true }
      volumes:
      - name: ssl-certs
        hostPath: { path: /etc/ssl/certs/ca-bundle.crt }

The IAM side is the piece worth writing out separately, because scoping it correctly is what stands between "Cluster Autoscaler manages my node groups" and "Cluster Autoscaler can resize any Auto Scaling Group in the account." Read-only discovery calls stay unscoped (there's no per-ASG resource-level permission for Describe* calls); the two mutating actions get scoped down with a tag condition matching the same k8s.io/cluster-autoscaler/<cluster-name> tag used for auto-discovery:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeAutoScalingInstances",
        "autoscaling:DescribeLaunchConfigurations",
        "autoscaling:DescribeScalingActivities",
        "autoscaling:DescribeTags",
        "ec2:DescribeLaunchTemplateVersions",
        "ec2:DescribeInstanceTypes"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:SetDesiredCapacity",
        "autoscaling:TerminateInstanceInAutoScalingGroup"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "autoscaling:ResourceTag/k8s.io/cluster-autoscaler/my-cluster": "owned"
        }
      }
    }
  ]
}

Wiring it to your cloud: node-group auto-discovery and scale-from-zero tags

☺ Like you're 10: Before the attendant can rent a row, every row in the whole garage needs a little tag that says "yes, this one's mine to manage."

Auto-discovery means Cluster Autoscaler finds its node groups by tag instead of a hand-maintained, per-group flag list — the practical difference between "add a node pool and it's just picked up" and "add a node pool and edit a Deployment spec." How that wiring looks, and how commonly you'd run this Deployment yourself at all rather than flip a managed switch, differs sharply by cloud:

Scale-from-zero has an extra requirement worth calling out on its own, because it fails silently otherwise. When a node group's desired size is 0, there is no running instance for Cluster Autoscaler to introspect for labels, taints, or resource capacity — so on AWS it reads that information from extra ASG tags instead, and skipping them either blocks the scale-up entirely or, worse, scales up a group that doesn't actually match what the Pending Pod needs:

k8s.io/cluster-autoscaler/node-template/label/node.kubernetes.io/instance-type = m5.xlarge
k8s.io/cluster-autoscaler/node-template/taint/dedicated = gpu:NoSchedule
k8s.io/cluster-autoscaler/node-template/resources/ephemeral-storage = 100Gi

Scale-down safety: PodDisruptionBudgets, node affinity, and the annotations that override them

☺ Like you're 10: A quiet row only gets given back once every car in it could genuinely park elsewhere — and a few explicit rules can make that permanently untrue, on purpose.

A node becomes a scale-down candidate only after its utilization has sat below --scale-down-utilization-threshold (50% by default) continuously for --scale-down-unneeded-time (10 minutes by default). Crossing that threshold isn't enough on its own — before removing the node, Cluster Autoscaler re-simulates whether every Pod currently on it could actually be rescheduled somewhere else in the cluster, and any of the following blocks the removal outright, indefinitely, re-checked on every scan loop rather than failing once and giving up:

• A PodDisruptionBudget the eviction would violate. The same guardrail VPA's Updater respects — a PDB with minAvailable equal to the workload's replica count makes every one of its Pods permanently non-evictable, which pins the node under it forever.
• Node affinity, pod anti-affinity, or a topology spread constraint that only this node satisfies. The re-simulation runs the real scheduler predicates, so a Pod that can provably only run here — a strict requiredDuringSchedulingIgnoredDuringExecution rule, most often — blocks the node exactly like a PDB would, for the same underlying reason: there's nowhere else for it to go.
• A Pod with no controller behind it (a bare Pod, not backed by a Deployment/ReplicaSet/StatefulSet/Job) — Cluster Autoscaler won't delete something nothing will recreate.
• A Pod using local storage (an emptyDir with real data, a hostPath) unless it carries cluster-autoscaler.kubernetes.io/safe-to-evict: "true" — the opposite of the annotation on the controller's own Pod above, which forces it the other way.
• A kube-system Pod without an eviction annotation, when --skip-nodes-with-system-pods=true (the default) — set cluster-autoscaler.kubernetes.io/enable-ds-eviction: "true" on a DaemonSet Pod to explicitly allow it.
• The node itself carries cluster-autoscaler.kubernetes.io/scale-down-disabled: "true". A blanket, node-level opt-out — useful for a sticky GPU node or a manually pinned node you never want touched, regardless of utilization.

Put those six checks together and scale-down is one repeatable decision loop, run fresh on every scan rather than decided once and forgotten:

When a node clears every check, Cluster Autoscaler cordons it, evicts its Pods respecting terminationGracePeriodSeconds and any applicable PDBs (a normal, budget-aware eviction — not a forced delete), waits for the drain to finish, then calls the cloud provider to terminate the underlying instance and shrink the node group's desired size by one.

Day-to-day commands

☺ Like you're 10: The attendant keeps a logbook — read it before guessing why a car's still circling.

# the single richest source of truth: CA publishes full loop state here every run
$ kubectl -n kube-system get configmap cluster-autoscaler-status -o yaml

$ kubectl -n kube-system logs deployment/cluster-autoscaler --tail=200 -f

# why is THIS pod still Pending? read the Events at the bottom
$ kubectl describe pod checkout-worker-7f9d8 -n checkout

# what does CA think happened, cluster-wide?
$ kubectl get events -A --field-selector reason=TriggeredScaleUp
$ kubectl get events -A --field-selector reason=ScaleDown

# manual overrides — same annotations CA itself reads
$ kubectl annotate node ip-10-0-4-112 cluster-autoscaler.kubernetes.io/scale-down-disabled=true
$ kubectl annotate pod cache-warmer-0 cluster-autoscaler.kubernetes.io/safe-to-evict=true --overwrite

# CA respects a manual cordon in its simulation — it won't schedule new capacity onto it
$ kubectl cordon ip-10-0-4-112

Gotchas that bite in production

☺ Like you're 10: Most of these don't look like a Cluster Autoscaler problem at first — they look like "nothing's happening," which is the hardest kind of bug to notice.

• Version skew. Cluster Autoscaler's minor version is tied to the Kubernetes minor version it was built against and tested for compatibility — running one more than a version or two behind (or ahead of) your control plane is unsupported and can silently miss newer scheduling predicates or CRD fields. Pin the image tag deliberately, and bump it as part of every cluster upgrade, not as an afterthought.
• An incomplete IAM policy fails quietly. A missing autoscaling:DescribeScalingActivities or similar doesn't surface in kubectl get events — it shows up only as an access-denied or throttling line in the controller's own logs, while Pods just keep sitting Pending. Check kubectl -n kube-system logs deployment/cluster-autoscaler before assuming the cluster itself is out of capacity.
• --balance-similar-node-groups needs groups that are actually similar. It only balances instance counts across node groups CA judges near-identical — same instance type, labels, and taints, typically one group per zone. Point it at genuinely different-shaped groups (different instance families, different label sets) and nothing balances, because CA correctly refuses to treat them as interchangeable.
• Running more than one replica needs leader election on, always. It's on by default (--leader-elect=true), but a hand-edited Deployment that turns it off and scales to 2+ replicas gets two controllers racing to resize the same ASG — a genuinely dangerous, self-inflicted failure mode.
• The scheduler simulation is conservative, and that's deliberate. If Cluster Autoscaler can't confidently determine that a node group's template will fit a Pending Pod's request — an exotic resource type without proper node-template tags is the classic case — it declines to scale that group rather than guess, leaving the Pod Pending with no obvious error anywhere.

Alternatives: Karpenter and cloud-native provisioners

☺ Like you're 10: Other options either pick from a wider catalog than "one of my three fixed rows," or hand the whole garage-management job to someone else entirely.

None of this replaces pod-level autoscaling — HPA, VPA, or event-driven scale-to-zero with KEDA — covered in full on the autoscaling deep-dive. Cluster Autoscaler is exam-adjacent rather than an explicit CKA domain item; Workloads & Scheduling expects hands-on HPA fluency, and node autoscaling tends to show up in real interview and production contexts more than on the exam itself — verify current weighting on the CNCF's own curriculum page before planning study time specifically around it.