Drill — Scale Under Load

One skill, start to finish: stand up a HorizontalPodAutoscaler, hit its target with real CPU load, and watch it react — first under Kubernetes' own defaults, then under numbers you picked yourself. You'll deploy a small CPU-hungry app, wire an HPA to it, generate load with a throwaway Pod, and time two things with a clock in your hand: how fast replicas climb when load appears, and how long it takes them to come back down once load disappears. Then you'll swap the default behavior block for a tuned one and run the identical test again, so the difference isn't something you read about — it's something you measured twice, yourself, thirty minutes apart. This drill is fully self-contained: no capstone cluster, no cloud account, nothing carried over from the six-part capstone. Budget 25–40 minutes, most of it spent watching a terminal and a clock.

How this drill works

☺ Like you're 10: Break it into four short acts — set up, load it up, watch the default reaction, then change one setting and watch it react differently.

The Autoscaling deep dive already covers the HPA's formula, its metrics pipeline, and its behavior block in detail — read that first if any term below is unfamiliar. This drill doesn't re-derive any of it; it makes you run it, twice, against a target that actually burns CPU when asked, and forces you to sit through the exact timing the deep dive only describes in prose. You'll deploy php-apache — the same small, deliberately expensive-per-request image the upstream Kubernetes docs use for this exact walkthrough — wire a HorizontalPodAutoscaler to it, and drive load with a busybox Pod in a tight request loop. First under the HPA's out-of-the-box behavior defaults, then under a behavior block you write yourself. CKA's Workloads & Scheduling domain expects you to stand up a basic HPA with kubectl autoscale; this drill goes one step further and asks you to actually watch what it does, on the clock.

Set up the scratch cluster and metrics-server

☺ Like you're 10: Build one throwaway kitchen and hook up its scale before anyone tries to weigh anything.

A default single-node kind cluster is all this drill needs. Bring your own metrics-server the same way the metrics-server tool page and the autoscaling deep dive's own "Try it" both do — kind's kubelets don't ship certificates metrics-server trusts out of the box, so the insecure-TLS escape hatch is expected here, not a mistake:

kind create cluster --name scale-drill --image kindest/node:v1.31.0
kubectl create namespace scale-drill
kubectl config set-context --current --namespace=scale-drill

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl -n kube-system patch deployment metrics-server --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
kubectl -n kube-system rollout status deployment/metrics-server

Give it a full scrape cycle before trusting it — metrics-server's first snapshot can take up to a minute to populate:

kubectl top nodes
# NAME                       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
# scale-drill-control-plane  187m         4%     612Mi           15%

Done when: kubectl top nodes returns real numbers, not an error. If it still errors after a minute, re-check the patch landed with kubectl -n kube-system get deployment metrics-server -o jsonpath='{.spec.template.spec.containers[0].args}' — --kubelet-insecure-tls should be the last entry in the list.

Deploy the target and a baseline HPA

☺ Like you're 10: One small app that visibly works harder the more you ask of it, plus a rule that watches how hard it's working.

php-apache does one deliberately wasteful thing per request — a tight square-root loop — specifically so a handful of concurrent requests moves its CPU usage a lot, fast, without needing real production traffic to prove the point:

# target.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: php-apache
  namespace: scale-drill
spec:
  replicas: 1
  selector: { matchLabels: { run: php-apache } }
  template:
    metadata: { labels: { run: php-apache } }
    spec:
      containers:
        - name: php-apache
          image: registry.k8s.io/hpa-example
          ports: [{ containerPort: 80 }]
          resources:
            requests: { cpu: 200m }
            limits: { cpu: 500m }
---
apiVersion: v1
kind: Service
metadata:
  name: php-apache
  namespace: scale-drill
  labels: { run: php-apache }
spec:
  selector: { run: php-apache }
  ports: [{ port: 80 }]

kubectl apply -f target.yaml
kubectl wait --for=condition=Available deployment/php-apache --timeout=60s
kubectl top pod -l run=php-apache          # confirm metrics-server sees this Pod specifically

kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
kubectl get hpa php-apache
# NAME         REFERENCE               TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
# php-apache   Deployment/php-apache   /50%   1         10        1          8s

Push it under load and watch the default reaction

☺ Like you're 10: Open a second terminal, start the rush, and actually watch the numbers instead of checking back later.

Open a second terminal now and leave this running for the rest of the drill — you want to see the replica count change in real time, not reconstruct it afterward from memory:

# terminal 2 — leave this running
kubectl get hpa php-apache --watch

Back in terminal 1, start the load generator — a plain busybox loop hitting the Service as fast as it can, the same load pattern the upstream HPA walkthrough uses:

# terminal 1 — note the wall-clock time you run this
date
kubectl run -i --tty load-generator --rm --image=busybox:1.28 --restart=Never \
  -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"

Watch terminal 2. Within one or two 15-second sync ticks, TARGETS should read well over 50% — often several hundred percent, since one loop can generate more requests than a single replica can absorb — and REPLICAS should start climbing. Exactly how high it climbs depends on your machine's real CPU speed, so don't expect a specific number; expect it to climb fast and land somewhere between 4 and the max=10 ceiling.

Stop the load and time the default scale-down

☺ Like you're 10: Send the extra cooks home — but watch how long the manager makes them wait before actually letting them go.

Kill the load generator with Ctrl+C in terminal 1 — the --rm flag deletes the Pod for you the moment the shell exits. Note the wall-clock time again, then watch terminal 2 closely:

# press Ctrl+C in terminal 1, then immediately:
date

TARGETS should drop toward 0%/50% within the next sync tick or two — the metric itself updates fast. REPLICAS is a different story: it should stay put at whatever it climbed to, not falling at all, for what will feel like a genuinely long wait. That's the default scaleDown.stabilizationWindowSeconds: 300 at work — the HPA looks back across the whole rolling 5-minute window and keeps whichever replica recommendation was least aggressive anywhere inside it, so one clean drop in load doesn't immediately shrink anything. Let it run and actually time it:

# once REPLICAS finally drops back toward 1:
date

Done when: you have two timestamps — load stopped, and replicas returned to 1 — and the gap between them is somewhere around five minutes, not thirty seconds. That number, measured on your own clock instead of read off a page, is the entire point of this act.

Tune the behavior block and shorten the window

☺ Like you're 10: Same caution, a shorter timer — and a couple of rules for how eagerly it's allowed to add cooks in the first place.

Delete the imperative HPA and replace it with a declarative autoscaling/v2 object carrying your own behavior block. Sixty seconds is still a real, deliberate waiting period — not 0, which would just trade "too cautious" for "too flappy" — it's simply short enough to actually observe twice inside this drill's time budget:

kubectl delete hpa php-apache

# hpa-tuned.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache
  namespace: scale-drill
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 50 }
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - { type: Percent, value: 100, periodSeconds: 15 }
        - { type: Pods,    value: 4,   periodSeconds: 15 }
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 60
      policies:
        - { type: Percent, value: 50, periodSeconds: 60 }
      selectPolicy: Min

kubectl apply -f hpa-tuned.yaml
kubectl get hpa php-apache

Two scale-up policies means the HPA computes a desired count from each one independently and, because selectPolicy: Max, keeps whichever adds more — a plain Percent policy struggles to grow a Pod count of 1 quickly (100% of 1 is still just 1 more), so the Pods: 4 policy is what actually lets this HPA jump straight from 1 toward 5 on the very first tick instead of climbing one replica at a time. scaleDown keeps selectPolicy: Min — the conservative choice, taking whichever policy removes fewer — the same asymmetry the autoscaling deep dive covers: aggressive going up, cautious coming down, just with the caution dial turned to a number you chose instead of the untouched default.

Reset, re-run the identical load, and compare

☺ Like you're 10: Same rush, same food truck — only the manager's patience changed, so time it again and see the difference for real.

Scale back to a clean baseline before repeating the exact same test — you want this run to start from the same place the first one did:

kubectl scale deployment php-apache --replicas=1
kubectl get hpa php-apache --watch   # keep terminal 2 on this

# terminal 1 — the identical command from before
date
kubectl run -i --tty load-generator --rm --image=busybox:1.28 --restart=Never \
  -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"

Watch the scale-up shape this time, not just the endpoint — with the Pods: 4 policy in play, the first jump can be noticeably chunkier than the first run's climb was. Once replicas are well above 1 and holding, kill the load generator with Ctrl+C again, note the time, and watch terminal 2 for the drop:

# Ctrl+C in terminal 1, then:
date
# ...watch terminal 2 until REPLICAS returns to 1, then:
date

Done when: your second scale-down timestamp gap lands somewhere around one to two minutes — close to the 60s window you set, plus a sync tick or two — instead of the roughly five minutes you measured on the first, untouched run. Same load, same target, same machine; the only thing that changed between the two timings is the number you put in behavior.scaleDown.stabilizationWindowSeconds.

Clean up

☺ Like you're 10: Nothing here needs to outlive the drill — delete the whole kitchen.

kind delete cluster --name scale-drill

Both timed runs done? That's the whole drill. For the formula, the metrics pipeline, and how VPA and Cluster Autoscaler fit around the HPA, see the Autoscaling deep dive; for the CKA-paced version of the same imperative command, see the blueprint's Workloads & Scheduling domain. For the mechanics of requests, limits, and QoS classes this whole drill's math rested on, see Scheduling & Resource Management, and for metrics-server's own architecture and failure modes, see the metrics-server tool page. Ready for a different single skill? Try Drill — Harden an RBAC Configuration or Drill — Troubleshoot a Failed Upgrade, or step back to the six-part capstone for the full continuity version.