The Kubernetes API & the Controller Pattern
Kubernetes has no back door. Every command you run, every Helm chart you install, and every controller running inside the cluster all reach the exact same place: a REST API served by kube-apiserver, laid out as a self-describing map of API groups and versions. This page pulls that API apart. You'll see kubectl for what it actually is — one HTTP client among many, no more privileged than a curl request carrying the right token — and how apiVersion and kind resolve to one specific URL on that server. Then it turns to what actually does the work once a request lands: controllers, small independent loops that read what you asked for, compare it to what's really running, and nudge one toward the other. ReplicaSet and Deployment are the two built-in controllers every later page assumes you understand — and the loop they run is the exact pattern CRDs and operators reuse to extend Kubernetes with entirely new nouns.
Picture a restaurant where every single order — yours, the delivery app's, even the manager's own lunch — goes through one ticket rail by the kitchen door. Nobody ever walks into the kitchen and starts cooking themselves, not even the manager. You just clip a ticket to the rail that says what you want: "table 4, three burgers." Different cooks watch the rail for tickets in their own category — the grill cook only cares about burger tickets, the salad cook only cares about salad tickets — and each one keeps glancing at the rail and the counter, cooking until what's on the counter matches what the ticket says. Kubernetes' API server is that ticket rail. kubectl is just one more waiter clipping tickets to it, same as everyone else.
kubectl Is Just an API Client
☺ Like you're 10: kubectl doesn't secretly do anything special — it just turns what you typed into a web request and sends it to the same front desk every other tool uses.
kube-apiserver is an HTTPS REST server. Every object type — Pods, Deployments, Secrets, your own future CRDs — has a URL path, and standard HTTP verbs map onto the familiar CRUD operations: GET to read or list, POST to create, PUT/PATCH to update, DELETE to remove, and a long-lived GET with ?watch=true to stream changes as they happen. kubectl reads your kubeconfig for the cluster's address and your credentials, turns the verb and resource you typed into the matching HTTP request, sends it, and pretty-prints whatever JSON comes back. Bump the verbosity and you can watch it happen:
# -v=6 prints the HTTP request line for every call kubectl makes $ kubectl get pods -n shop -v=6 I0827 09:12:04 GET https://10.0.4.12:6443/api/v1/namespaces/shop/pods?limit=500 I0827 09:12:04 200 OK in 14 milliseconds $ kubectl scale deployment/checkout --replicas=5 -n shop -v=6 I0827 09:12:41 PATCH https://10.0.4.12:6443/apis/apps/v1/namespaces/shop/deployments/checkout I0827 09:12:41 200 OK in 21 milliseconds
Nothing in that request requires the kubectl binary specifically. Helm, ArgoCD, your own Python script using the official client library, and a bare curl carrying a bearer token all hit the identical endpoint and get treated identically:
$ TOKEN=$(kubectl create token deploy-bot -n shop)
$ curl -sk -H "Authorization: Bearer $TOKEN" \
https://10.0.4.12:6443/apis/apps/v1/namespaces/shop/deployments/checkout \
| jq '.spec.replicas'
5kubectl carries zero special privilege. Authentication, RBAC authorization, and admission control all sit at the API server boundary and apply identically no matter which client sent the request — which is exactly why a service account with the right RoleBinding can do from a CI pipeline everything a human can do from a laptop, and nothing more.
API Groups, Versions & the Object Skeleton
☺ Like you're 10: Resources live in named drawers so the API can keep growing without two teams ever fighting over the same filing space.
Resource types are organized into API groups so the surface can expand without collisions. The oldest resources — Pod, Service, ConfigMap, Namespace — belong to the core group, which has no name at all and lives under /api/v1. Everything added since has its own named group and hangs off /apis/<group>/<version> instead: Deployments and ReplicaSets live in apps, Jobs and CronJobs in batch, Ingress and NetworkPolicy in networking.k8s.io, RBAC objects in rbac.authorization.k8s.io. Each group versions independently, and the version string tells you how much to trust it: v1alpha1 can change or vanish without warning, v1beta1 is well-tested but not yet frozen, and v1 is stable and forward-compatible. A cluster can serve several versions of one Kind at once, converting between them on request while persisting only one storage version in etcd — which is how the API evolves without forcing you to rewrite every manifest on every upgrade.
Whatever the group, every object you write shares the same five top-level fields:
apiVersion: apps/v1 # which group + version this object belongs to kind: Deployment # which resource type within that group metadata: # name, namespace, labels, annotations, UID name: checkout namespace: shop spec: # the desired state — what YOU wrote replicas: 3 # ... status: # the observed state — what a CONTROLLER wrote back readyReplicas: 3 # ...
Learn that shape once and you can read a resource you've never seen before — including one from a CRD nobody on your team wrote. Two discovery endpoints make the whole surface machine-readable: kubectl api-resources lists every Kind the cluster currently serves and which group it lives in, and kubectl explain reads the server's published OpenAPI schema field by field.
$ kubectl api-resources | grep -iE 'name|deployment|replicaset' NAME SHORTNAMES APIVERSION NAMESPACED KIND deployments deploy apps/v1 true Deployment replicasets rs apps/v1 true ReplicaSet $ kubectl explain deployment.spec.strategy KIND: Deployment VERSION: apps/v1 FIELD: strategyDESCRIPTION: The deployment strategy to use to replace existing pods with new ones.
The full mechanics of translating what you write (apiVersion + kind, called the GVK) into the URL the server actually exposes (the GVR) belong to a deeper architecture treatment — see Kubernetes as the Platform Substrate in the Platform Engineering course for the full RESTMapper walkthrough, and the kubectl tool guide in this course for day-to-day command fluency.
The Reconcile Loop: ReplicaSet
☺ Like you're 10: A ReplicaSet's whole job is counting — it keeps checking how many matching Pods actually exist and starts or stops Pods until that count matches the number you asked for.
As introduced in The Object Model, everything you write to the API is desired state, and a controller's job is closing the gap between that and actual state. A ReplicaSet is the simplest controller in the cluster to see this in — its entire spec is three fields: how many replicas you want, a label selector describing which Pods count toward that number, and a Pod template to stamp out new ones from.
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: checkout-rs
namespace: shop
spec:
replicas: 3
selector:
matchLabels:
app: checkout # must match the template's labels below
template:
metadata:
labels:
app: checkout
spec:
containers:
- name: checkout
image: registry.internal/checkout:1.4.2
ports:
- containerPort: 8080The ReplicaSet controller runs one loop, forever: list the Pods matching spec.selector, count them, compare that count to spec.replicas, and act — create new Pods from the template if the count is low, delete the newest first if it's high. Critically, the loop is level-triggered, not event-triggered: it doesn't remember "a Pod just died," it only ever asks "how many matching Pods exist right now." That's what makes it self-healing without any special-cased crash-recovery logic — a node dying, a Pod being evicted, and someone running kubectl delete pod by hand all look identical to the loop: the count dropped, so it creates a replacement. Note too that the selector has to match the template's own labels, or the apiserver rejects the object outright — a ReplicaSet that can never see its own children is a contradiction the API refuses to store.
kubectl edit pod checkout-rs-x7k2p feels like it fixed something, but it's a Pod, not the ReplicaSet — the edit only survives until that specific Pod is replaced for any reason, at which point the fresh Pod comes straight from spec.template again and your one-off change is gone with no warning. If a change needs to survive, it belongs in the template (or the Deployment above it), not in a running Pod.
Deployment: a Controller of Controllers
☺ Like you're 10: A Deployment never touches a Pod directly — it manages ReplicaSets, and lets each ReplicaSet manage the Pods, which is how you get rollouts and rollback for free.
A bare ReplicaSet has no concept of a rollout: change its Pod template and the existing Pods simply keep running the old image forever, because the controller only counts, it never diffs the template against what's live. Deployment is what most teams actually write, and it works by managing ReplicaSets rather than Pods. Change a Deployment's spec.template, and its controller creates a new ReplicaSet carrying the new template, then scales that one up and the old one down according to strategy.rollingUpdate — maxSurge caps how far over the target count it can go, maxUnavailable caps how far under. The old ReplicaSet isn't deleted; it's kept at zero replicas as revision history, which is the entire mechanism behind rollback.
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout
namespace: shop
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app: checkout
template:
metadata:
labels:
app: checkout
spec:
containers:
- name: checkout
image: registry.internal/checkout:1.4.2$ kubectl set image deployment/checkout checkout=registry.internal/checkout:1.5.0 -n shop $ kubectl rollout status deployment/checkout -n shop Waiting for deployment "checkout" rollout to finish: 2 of 3 new replicas updated... deployment "checkout" successfully rolled out $ kubectl rollout history deployment/checkout -n shop REVISION CHANGE-CAUSE 12 checkout=registry.internal/checkout:1.5.0 $ kubectl rollout undo deployment/checkout -n shop --to-revision=1
This is a three-generation ownership chain — Deployment owns ReplicaSets, ReplicaSets own Pods — and Kubernetes tracks each link explicitly with an ownerReferences field on the child object, which is also what lets kubectl delete deployment checkout cascade cleanly through both ReplicaSets and Pods by default. It's the same shape you'll meet again at Scheduling & Resource Management and across the CKA blueprint's Workloads & Scheduling domain, which assumes this chain as background.
On any cluster you can reach, run kubectl get rs -n <ns> -w in one terminal and kubectl set image deployment/<name> ... in another. Watch the old ReplicaSet's DESIRED count count down while the new one counts up, one step at a time, and match what you see against maxSurge/maxUnavailable on the Deployment. Then add -v=6 to the kubectl set image call and confirm for yourself that it's a single PATCH to /apis/apps/v1/.../deployments/<name> — the Deployment controller does everything else.
The Generic Pattern: Why CRDs & Operators Reuse It
☺ Like you're 10: Every controller — the ones Kubernetes ships and the ones you write yourself — runs the exact same three-step loop, which is exactly why anyone can add a new one without ever touching the core.
kube-controller-manager runs dozens of these loops as one process, one per built-in resource type — a Node controller, an Endpoints controller, a Job controller, ReplicaSet's and Deployment's, and more — and not one of them talks to another directly. Each watches the API for the objects it owns, reconciles, and writes results back to the API for someone else to watch if they care. That isolation is the whole reason the pattern scales: adding a capability means adding another independent loop, never patching a monolith. The internals of that scheduling — informers caching the watch stream, a work queue de-duplicating and rate-limiting reconciles — are covered in Control Plane Internals.
Which is exactly what makes CRDs possible. A CustomResourceDefinition registers a brand-new Kind with the apiserver — new group, new schema, full CRUD and kubectl get support, no code required. On its own, though, a CRD is only storage for a new noun; nothing reconciles it. Pair it with a controller you write, running the identical watch → diff → act loop against your new Kind instead of a built-in one, and you have an operator — a native, self-healing Kubernetes API for a concept Kubernetes never shipped. This course goes deep on building and running one in Operators & Custom Resource Definitions; the Platform Engineering course takes the same mechanism further into building whole self-service platforms on top of it.
Kubernetes core is not one program — it's dozens of independent controllers that agree to only ever talk to each other through the API. A CRD plus a custom controller is not a workaround or a hack; it is a first-class member of that same federation, indistinguishable from the built-ins once it's running.
"I don't know what created me and I don't care what created you. I watch one thing: does what's actually running match what's written down. When it doesn't, I act — create, delete, patch, whatever closes the gap. When it does, I go quiet until the next watch event wakes me up. That's the entire job, and it doesn't change one bit whether I'm a ReplicaSet controller that's been in Kubernetes core since 2016 or a CRD controller someone on your platform team deployed yesterday afternoon."
Professor Owl: Every request in this cluster, no matter who sends it, ends up at the same front desk — the apiserver. kubectl has no side entrance.
Gizmo: Boring. Watch this instead — a Pod's crash-looping, so I just kubectl edit the Pod and fix the env var straight in the running container. Skips the whole ReplicaSet song and dance.
Recon: It'll work for exactly as long as that one Pod survives. The next time it's replaced — a node drain, an eviction, plain bad luck — I build the replacement straight from spec.template, which never learned about your fix.
Gizmo: ...it survived a whole afternoon, that's basically forever in incident time.
Benny the Beaver: I edit the template instead now. Costs me thirty extra seconds and a rollout. Nobody's ever paged me at 2 a.m. over a Deployment that did exactly what its own YAML said.
Professor Owl: Which is the whole point of a controller — it doesn't remember what you did, it only ever checks what you wrote. Write the fix where a controller will find it again.
1. Why does kubectl carry no special privilege compared to a raw curl request with a valid bearer token? 2. What are the three fields in a ReplicaSet's spec, and what has to be true about two of them for the apiserver to accept the object at all? 3. A Deployment's rollout doesn't touch Pods directly — what does it actually manage, and how does that give you rollback for free? 4. What does "level-triggered, not event-triggered" mean for a controller, and why does it make self-healing free rather than a special feature? 5. What two ingredients combine to turn a CRD, which is just storage for a new Kind, into a working operator?
Check your answers
- Because authentication, RBAC authorization, and admission control all run at the apiserver boundary and apply identically to every client — kubectl is just one more caller presenting credentials, with no privilege of its own.
replicas,selector, andtemplate. The selector'smatchLabelsmust match the labels on the template's own metadata, or the apiserver rejects the object — a ReplicaSet that couldn't see its own children would be a contradiction.- It manages ReplicaSets, not Pods. Each new template version gets its own new ReplicaSet, and the previous one is scaled to zero rather than deleted, so rolling back to an earlier revision is just scaling the old ReplicaSet back up and the new one down.
- A level-triggered controller reconciles based on the current state ("how many matching Pods exist right now"), not on remembering a specific event ("a Pod just died"). That means a node dying, an eviction, and a manual delete all look identical and get fixed the same way — self-healing falls out of the model instead of needing dedicated crash-handling logic.
- A schema (from the CRD, giving the new Kind storage, validation, and CRUD via the API) plus a custom controller running the same watch → diff → act loop against that Kind. The CRD alone is inert; the controller is what makes it reconcile.