Cost & FinOps on Kubernetes

A production Kubernetes cluster runs Pods from many teams bin-packed onto a much smaller number of shared nodes — and that single design choice, the one that makes Kubernetes efficient, is also the one that makes its bill almost impossible to read. A cloud invoice prices node-hours, not Pods; it has no idea that half of node prod-worker-14 belonged to Payments and the other half to Search. This page is deliberately scoped to the Kubernetes-specific mechanics of that problem, not the full FinOps operating model, which Platform Engineering's FinOps & Cost Optimization already covers end to end — Inform/Optimize/Operate, unit economics, GreenOps. What's here instead: why shared, bin-packed nodes erase the natural per-team line item a dedicated VM would have; the requests-vs-usage gap that is Kubernetes' single largest source of quiet waste; how node-shape decisions change what you pay for capacity nobody uses; how to run real workloads on spot and preemptible capacity without losing them mid-incident; and the minimum viable version of showback and chargeback once you can finally see the number. None of it is on the CKA or CKAD blueprint — it's the material that shows up the week after the exam, once a real bill lands in someone's inbox.

Why a shared cluster's bill doesn't have an owner

☺ Like you're 10: One invoice arrives for the whole node — and nothing on it says which team's Pods actually made the number that big.

Cloud billing was designed for a world where cost and ownership line up automatically: one VM, one application, one team, one line item. Kubernetes exists specifically to break that assumption on purpose. The scheduler's whole job is bin-packing — cramming Pods of varying sizes from unrelated teams onto a much smaller number of nodes than "one workload, one machine" would ever need, because idle, unshared capacity is exactly the waste Kubernetes was built to eliminate. That's a genuine efficiency win. It's also the reason the invoice for node prod-worker-14 says one dollar figure and nothing else — not which namespace, not which team, not how much of that node sat reserved and unused. The cloud provider bills the node; it has never heard of a Pod.

Closing that gap is a measurement problem before it's anything else, and Kubernetes doesn't solve it alone — a tool has to watch the scheduler's own decisions and turn them into dollars. OpenCost, the CNCF-hosted specification and reference implementation for this exact problem (see The CNCF Project Landscape for where it sits among the rest of the ecosystem), watches every container through Prometheus and computes what each one actually cost against the real hourly rate of the node it ran on. This page assumes that measurement layer exists and focuses on the five decisions — right-sizing, node shape, spot capacity, and allocation — that determine what the number says once you can see it.

The requests-vs-usage gap: where the real waste lives

☺ Like you're 10: You're billed for what you reserved, not what you actually used — and the gap between those two numbers is where most Kubernetes waste hides.

Scheduling & Resource Management already covers the mechanics in full — how a Pod's resources.requests becomes a real Linux cgroup setting, and how QoS class determines eviction order under memory pressure. The cost angle is narrower but sits on the exact same field: a request isn't just what the scheduler checks at bind time, it's what a cost tool charges you for, whether or not the container ever touches it. A Pod that requests 4 CPUs and averages 0.4 reserved four CPU-hours of node capacity every hour, full stop — the other 3.6 sat idle, unusable by anyone else scheduled on that node, and billed anyway. Multiply that gap across a few thousand Pods sized by a developer's confident guess eighteen months ago, and it is routinely the single largest line item a cost report turns up.

Closing it starts with looking, not guessing. kubectl top reads real usage straight from the metrics pipeline Autoscaling: HPA, VPA & Cluster Autoscaler covers in depth — set it next to the requests already on the Pod spec, and the gap is right there in two columns:

# Actual usage, from metrics-server
kubectl top pod -n payments --no-headers
# NAME                CPU(cores)   MEMORY(bytes)
# checkout-7d9f-abcde  180m         210Mi

# What was actually requested, from the spec
kubectl get pod checkout-7d9f-abcde -n payments \
  -o jsonpath='{.spec.containers[0].resources.requests}'
# {"cpu":"2","memory":"2Gi"}
#   ^ requested 2 full cores, using 180m — an 11x gap, billed at the 2-core rate

Doing that Pod by Pod doesn't scale, which is exactly what the VerticalPodAutoscaler's Recommender is for outside of its auto-apply role. Run it in updateMode: "Off" and it does nothing but watch real usage over time and publish a recommendation on the object — no restarts, no risk, just the honest number Sol would have arrived at by hand, days sooner:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-recommender
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout
  updateMode: "Off"   # recommend only — never patches requests automatically
---
# kubectl describe vpa checkout-recommender then shows, per container:
#   Target: cpu: 210m, memory: 340Mi   (down from a requested 2 / 2Gi)

Node-shape strategy: what the fleet's shape costs you

☺ Like you're 10: The size and mix of vans you rent changes how much empty space you pay for — not just how much stuff each one can carry.

Right-sizing controls what each Pod asks for; node shape controls how well those requests actually fit together once they're bin-packed. Every node also carries fixed overhead regardless of size — kube-reserved and system-reserved capacity held back from Allocatable, plus every DaemonSet the platform team runs: a CNI agent, a log shipper, a CSI node plugin, a metrics exporter. On a large node that tax is a rounding error. On a small one it can be a genuinely large slice of what you're paying for per node — the DaemonSet tax scales per-node, not per-core, so shrinking the node doesn't shrink it. Fewer, larger nodes amortize that fixed cost across more usable capacity and bin-pack more tightly, because a scheduler choosing among fewer, bigger buckets strands less space per Pod placed; more, smaller nodes pack worse but fail smaller — losing one node loses less of the cluster.

This is exactly the problem a modern node autoscaler earns its keep on. Autoscaling: HPA, VPA & Cluster Autoscaler covers Cluster Autoscaler and Karpenter in full; the cost-relevant part of that mechanism is Karpenter's continuous consolidation loop, which doesn't just scale down an idle node — it actively asks whether the Pods spread across several under-packed nodes would fit on fewer or cheaper ones, and replaces nodes to make that true even while the cluster looks otherwise healthy. Platform Engineering's Scaling, Scheduling & Performance goes further into consolidation policy and disruption budgets at implementation depth this page doesn't repeat.

Spot and preemptible node pools, without losing the workload

☺ Like you're 10: A spot node is dramatically cheaper right up until the cloud takes it back with about two minutes' notice — plan for that notice, don't plan around never getting one.

Spot (AWS, Azure) and preemptible (GCP) capacity is spare data-center capacity a cloud provider sells at a steep discount — commonly somewhere in the 60-90% range off on-demand, though the exact figure moves with instance type, region, and market demand, so read it off your provider's current pricing rather than treating any number here as fixed. The trade is real: that capacity can be reclaimed with short notice, typically around two minutes on AWS and roughly thirty seconds on GCP's preemptible tier. Kubernetes doesn't get any special exemption from that notice — it just has machinery for reacting to it gracefully instead of losing Pods abruptly. A node termination handler (AWS's aws-node-termination-handler, or Karpenter's own built-in interruption handling watching an SQS queue of EC2 Spot events) watches for the reclaim signal and cordons and drains the node ahead of the actual termination, using the Eviction API so a PodDisruptionBudget gets a real say in the order and pace of that drain.

Which workloads belong on that capacity follows directly from the timeline above. Stateless, horizontally-scaled services with several replicas and a fast, cheap restart are a strong fit — losing one of six replicas for the roughly one minute a reschedule takes is invisible to users if a PDB keeps the others up. Batch jobs, CI runners, and checkpointed ML training tolerate interruption by design. A poor fit is anything with a single replica, slow startup relative to the notice window, or state that can't fail over fast — exactly the workloads QoS and priority already argue should sit at the protected end of the spectrum regardless of cost. Karpenter's documented pattern is to make spot opt-in: a custom taint on spot nodes that only workloads with a matching toleration will land on, rather than letting anything land there by default.

# NodePool: spot-eligible capacity, tainted so nothing lands here by accident
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-batch
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
      taints:
        - key: workload-class
          value: interruptible
          effect: NoSchedule
      nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: default }
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
---
# Workload: opts in explicitly, and a PDB caps how many replicas can drain at once
apiVersion: apps/v1
kind: Deployment
metadata: { name: checkout-worker }
spec:
  replicas: 6
  template:
    spec:
      tolerations:
        - { key: workload-class, operator: Equal, value: interruptible, effect: NoSchedule }
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - { key: karpenter.sh/capacity-type, operator: In, values: ["spot"] }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: checkout-worker-pdb }
spec:
  minAvailable: 4
  selector: { matchLabels: { app: checkout-worker } }

Showback and chargeback: turning the bill into a number teams trust

☺ Like you're 10: Publish the number first and just let people see it — only start moving actual budgets once nobody's arguing that the number is wrong.

Platform Engineering's FinOps & Cost Optimization covers the full allocation taxonomy — showback, chargeback, unit economics — in depth; the summary that matters here is short. Showback publishes each team's share of the shared bill with no budget attached: low political cost, and it's the fastest way to surface the attribution gaps that always exist at the start (unlabeled shared workloads, a runner pool three teams quietly share). Chargeback bills that number against a real budget, and it only holds up once the underlying attribution is accurate enough to survive being disputed — start with showback, and only graduate a namespace or team to chargeback once its numbers have gone unchallenged for a while.

Either model depends on the same prerequisite: every namespace and workload consistently labeled with who owns it, because OpenCost and Kubecost can only attribute cost against labels that actually exist. A convention teams are merely asked to follow decays the moment someone's under deadline pressure; the reliable version enforces the label at admission, rejecting a write instead of hoping for one. RBAC & Admission Control walks through exactly this scenario end to end — a ValidatingWebhookConfiguration named require-cost-center-label that blocks a Deployment write missing the label entirely — and in practice most teams reach for Kyverno or Gatekeeper to express that same policy declaratively instead of hand-rolling the webhook. Whatever labels the platform can't attribute to a specific team — the DaemonSets in the schematic above, a shared ingress controller, the control plane itself if it's self-managed — belongs in its own honestly-named platform or shared bucket, reported as its own line rather than smeared silently across everyone else's number.

Guardrails that keep the number from drifting back up

☺ Like you're 10: A cap nobody enforces isn't a cap — set a ceiling every namespace has to live inside, and clean up the things nobody remembers to turn off.

The levers above are decisions made once; guardrails are what stops the number creeping back afterward. A ResourceQuota scoped per namespace caps total requested CPU and memory before a namespace can quietly grow past what it was ever sized for, and a LimitRange sets a sane default request for any container that ships without one — closing off the cheapest way the requests-vs-usage gap reopens, a Pod deployed with no request field set at all. The other recurring source of drift is capacity nobody's actively using but nobody's actively watching either: a PersistentVolume left behind after its claim is deleted under a Retain reclaim policy, a LoadBalancer Service nobody tore down after the feature it fronted shipped and moved on, a non-production namespace running its full replica count at 2 a.m. on a Saturday for no one. None of these show up as a spike — they show up as a bill that's a little higher every month than the workloads running on it can explain, which is precisely the kind of drift a showback report catches early and a chargeback dispute catches late.