Best Practices & Operating Model

A Kubernetes cluster left ungoverned fails in the same handful of ways every time: one namespace's runaway Deployment starves every other tenant on the node, a control-plane upgrade nobody scheduled turns into an unplanned outage, and a lost etcd volume becomes a company-wide incident because the restore procedure was never actually tested. None of that is really a Kubernetes problem — Kubernetes just makes the consequences visible faster than most systems do. See Anti-Patterns & Pitfalls for that failure catalog in full; this page is the constructive half. Five habits carry most of the weight: a namespace convention that gives every team, environment, and cost center a clean boundary; resource governance that turns "someone else's Deployment ate the node" into a ResourceQuota doing its job automatically; an upgrade cadence that keeps the control plane inside its supported version window without a surprise; a backup and disaster-recovery plan that treats etcd snapshots and workload backups as two separate, both-tested disciplines; and an on-call rotation that already has the runbooks and alerts it needs before the pager ever goes off.

Namespaces as the tenancy boundary

☺ Like you're 10: Namespaces are the separate apartments in the shared building — everyone splits the plumbing and the front door, but each unit gets its own lock and its own rules.

A namespace is Kubernetes' unit of tenancy: the boundary along which you name things, quota things, and — combined with other controls — secure things. It is not, on its own, a security boundary; two Pods in different namespaces can still reach each other over the network and a ClusterRoleBinding still spans every namespace in the cluster unless something stops it. A namespace becomes a real boundary only once you stack RBAC, a default-deny NetworkPolicy, and admission control on top of it — see RBAC & Admission Control and Security: Defense in Depth for that stack in full, and Networking & the CNI for how a NetworkPolicy is actually enforced at the packet level. What a namespace convention buys you on its own is something more mundane and just as important: every resource that exists has an obvious owner, an obvious environment, and an obvious place to look when something in it breaks.

Pick one naming scheme and hold every team to it — <team>-<environment> (checkout-prod, checkout-staging) reads cleanly in kubectl get ns, sorts predictably, and survives a team growing to own more than one service. Reserve kube-system, kube-public, and kube-node-lease for the platform itself, and give shared platform tooling — ingress controllers, cert-manager, your GitOps operator — its own platform-system-style namespace rather than letting it drift into whichever namespace someone happened to be in when they ran helm install. Since Kubernetes 1.22 every namespace is automatically labeled kubernetes.io/metadata.name, which is useful for NetworkPolicy selectors — but don't stop there. Add your own labels for automation and cost attribution, because a label is queryable and a name embedded in a string is not:

apiVersion: v1
kind: Namespace
metadata:
  name: checkout-prod
  labels:
    team: checkout
    env: prod
    cost-center: "cc-4471"
    managed-by: platform-team
---
# A default-deny baseline: every namespace starts closed, and a team
# opens exactly the paths its workloads actually need on top of this.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: checkout-prod
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

That default-deny policy is deliberately the first object created in every new namespace, before a single workload lands in it — a namespace that opens permissive-by-default and gets locked down later almost never actually gets locked down, because by then something is already depending on the hole.

Resource governance: quotas, limits, and priority

☺ Like you're 10: A ResourceQuota is the water meter on the whole apartment; a LimitRange is the rule that every faucet in it has to have a normal-sized handle, even if nobody remembered to say so.

Workloads & Scheduling and Scheduling & Resource Management already cover what a container's requests and limits mean to the scheduler and the kubelet; this section is about making sure they're actually set, on every container, without relying on every engineer remembering. Two objects do that work at the namespace level. A ResourceQuota puts a hard ceiling on total consumption in a namespace — CPU, memory, Pod count, PVC count, even how many LoadBalancer-type Services it can create — so one namespace's runaway Deployment fails with an obvious, immediate error instead of quietly starving every other tenant on the node. A LimitRange works one level down, at the container: it supplies a default request and limit for any container manifest that omits them, and enforces a min/max so nothing can request 64Mi or demand 64Gi by accident. The two are complementary, not redundant — a ResourceQuota with no LimitRange still lets one enormous unbounded container eat the entire namespace's quota in one Pod.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: checkout-prod-quota
  namespace: checkout-prod
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 80Gi
    limits.cpu: "80"
    limits.memory: 160Gi
    pods: "150"
    persistentvolumeclaims: "20"
    services.loadbalancers: "2"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: checkout-prod-defaults
  namespace: checkout-prod
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: 100m, memory: 128Mi }
      default:        { cpu: 500m, memory: 512Mi }
      min:            { cpu: 50m,  memory: 64Mi }
      max:            { cpu: "4",  memory: 8Gi }

Quota and limits stop accidents; they don't stop a manifest that deliberately requests 0m to dodge scheduling pressure, or an image tagged :latest shipping to production. That's a job for an admission controller — Kubernetes' built-in Pod Security Admission covers the security baseline, and a policy engine like OPA Gatekeeper or Kyverno covers everything org-specific, including "every container must declare requests and limits" as a mandatory rule rather than a convention someone can forget. DevSecOps' IaC Security & Policy as Code covers writing those rules in depth, and Platform Engineering's Kyverno tool guide covers one concrete engine for enforcing them at admission time.

Upgrade cadence: staying inside the support window

☺ Like you're 10: Elevator maintenance happens one elevator at a time, on a posted schedule — never all three at once, and never as a surprise.

Upstream Kubernetes ships a new minor version roughly three times a year, and only supports a limited number of the most recent minor versions at any given time — check the current Kubernetes release and support policy for the exact window, since it has widened over the project's history and this page won't age well if it hard-codes a number. Falling further behind than that means running a control plane that no longer receives security patches, which is exactly the gap a CKS-level audit or a serious incident tends to find first. The fix isn't heroics twice a year — it's a standing cadence: pick a target skew behind the latest stable release (most teams land on "one minor version behind," sometimes called N-1), and treat every release that pushes you further than that as a scheduled piece of work, not an emergency.

Two rules govern the mechanics, and both are exam-relevant — Cluster Architecture, Installation & Configuration is where the CKA curriculum tests this directly. First, never skip a minor version on the control plane; kubeadm upgrade enforces this and will refuse to jump from, say, 1.30 straight to 1.32 — you upgrade to 1.31 first, confirm health, then 1.32. Second, respect the version skew policy: kube-apiserver instances in an HA control plane must stay within one minor version of each other, and kubelet is allowed to run some number of minor versions older than the API server — again, confirm the current allowance in the skew policy rather than trusting a fixed number, since it too has changed. In practice that means the control plane always upgrades first, nodes catch up afterward, and a cluster is never caught with newer nodes than control plane.

# Control plane, one node at a time, one minor version at a time
kubeadm upgrade plan                       # shows what's available and what's blocked
kubeadm upgrade apply v1.31.4               # first control-plane node only
apt-get update && apt-get install -y kubelet=1.31.4-* kubeadm=1.31.4-*
systemctl daemon-reload && systemctl restart kubelet
# remaining control-plane nodes: kubeadm upgrade node, then the same kubelet bump

# Each worker node — drain before touching anything on it
kubectl drain node-worker-07 --ignore-daemonsets --delete-emptydir-data
apt-get update && apt-get install -y kubelet=1.31.4-* kubeadm=1.31.4-*
systemctl daemon-reload && systemctl restart kubelet
kubectl uncordon node-worker-07             # only after health checks pass

The drain step is where a PodDisruptionBudget earns its keep — without one, drain can evict every replica of a Deployment simultaneously if they all happen to land on the node being upgraded; with a minAvailable or maxUnavailable PDB set, drain respects it and waits. Roll nodes in small batches per pool or availability zone rather than all at once, so a control-plane compatibility surprise shows up on five nodes instead of five hundred, and rehearse the whole sequence on a staging cluster — or a canary node pool inside production — before it touches anything customer-facing. The kubeadm tool guide and the kubectl tool guide cover the individual commands in more depth than this page needs to.

Backup & DR: etcd snapshots and Velero are not the same job

☺ Like you're 10: One plan saves the building's blueprint; the other saves everyone's actual furniture. You need both, and neither one substitutes for the other.

Teams that back up a cluster "just in case" often discover, mid-incident, that they backed up only half of it. etcd holds the entire declarative state of a self-managed cluster's control plane — every object, every secret, the works — and an etcdctl snapshot save captures exactly that, as a point-in-time image of the control-plane database. It says nothing about whether a PersistentVolume's actual data survives, and on a managed control plane (EKS, GKE, AKS) you don't get to run it at all — the cloud provider owns etcd, not you. Velero covers the other half from user space: it talks to the Kubernetes API like any other client, so it captures namespaced API objects (Deployments, Secrets, ConfigMaps, CRs) and, with a snapshot or file-system backup plugin attached, the volume data those objects point to. Velero works identically on a managed or self-managed cluster; etcd snapshots only apply if you run etcd yourself. Neither backs up what the other one owns — Velero does not capture the raw etcd database, and an etcd snapshot restores cluster state but not a namespace's PersistentVolume contents on its own.

# etcd — control-plane state, self-managed clusters only
ETCDCTL_API=3 etcdctl snapshot save /opt/backup/etcd-$(date +%Y%m%d%H%M).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

etcdctl snapshot status /opt/backup/etcd-202608270200.db --write-out=table

# Velero — namespaced workloads and their volume data, any cluster
velero schedule create checkout-nightly \
  --schedule="0 2 * * *" \
  --include-namespaces checkout-prod \
  --ttl 720h0m0s

velero backup describe checkout-nightly-20260827020000 --details
velero restore create --from-backup checkout-nightly-20260827020000 --wait

Match the schedule and retention to what the workload can actually afford to lose — this is the same RPO/RTO conversation Stateful Workloads & Database Operators has in more depth for anything running on top of a database operator.

On-call readiness for a cluster

☺ Like you're 10: The super needs the master key list and the maintenance log before the pager goes off at 2am — not while standing outside a locked door trying to remember where they put it.

"Ready" for cluster on-call means three concrete things exist before the first page arrives, not three things someone plans to build eventually. First, alerts on the control plane itself, not just on the workloads it runs — etcd leader elections and disk-fsync latency, kube-apiserver request latency and error rate, node NotReady conditions, and certificate expiry on the kubeadm-issued certs, which default to a roughly one-year lifetime and fail silently until the API server starts rejecting every authenticated request at once. Second, runbooks for the failure modes that actually recur — a stuck Pending PVC, a CrashLoopBackOff storm from a bad rollout, DNS resolution failures inside the cluster, a node stuck NotReady — written down before the incident, not improvised during it; A Troubleshooting Methodology and Troubleshooting (CKA Domain 5, worth 30% of the exam on its own) are where those methods live in full. Third, capacity headroom: run at least one extra node's worth of spare capacity across every pool so a single node failure evicts workloads onto existing capacity instead of triggering a scheduling pile-up, and back every workload that can't tolerate an unplanned eviction with a PodDisruptionBudget, the same object the upgrade section above relies on.

None of this is Kubernetes-specific once you're past the cluster's own signals — SRE's Monitoring & Observability covers the golden-signals framework this page assumes, and its postmortems & blameless culture lesson covers what happens after the page, once the incident is actually resolved. Inside this course, Observability on Kubernetes covers wiring up the metrics, logs, and traces this section assumes are already flowing.

The operating-model scorecard

☺ Like you're 10: A short report card — score each row 0, 1, or 2, and the lowest scores are next quarter's real to-do list.

Score your own cluster against these six rows, 0 (not really), 1 (partly), or 2 (solidly true), out of 12 total. The lowest-scoring rows, not the highest, are the ones worth doing something about.