Foundations · The object model & reconciliation

The Object Model: Declarative APIs & Reconciliation

Everything you will ever create in Kubernetes — a Pod, a Deployment, a Service, even a custom type a platform team invents next year — is written down the same way: a name, a wish, and a report card. That one shared shape is the whole reason Kubernetes can be summed up in a single sentence and still explain almost everything it does: you declare the state you want, and a swarm of small independent loops keeps re-checking the cluster and nudging it toward that state, forever, without you issuing another command. This page is the conceptual seed for the rest of the course — the API and controller mechanics in the next lesson, RBAC and admission control later, most of troubleshooting — because nearly every one of those topics reduces to some version of "read the spec, compare it to reality, close the gap." By the end you should be able to state the difference between spec and status without hesitating, describe what a controller does in its loop, say exactly what a label selector matches against, and know what a namespace does and doesn't isolate.

☺ Explain it like I'm 10

Picture a smart fish tank with an automatic top-off system. You don't refill it by hand every time water evaporates — you set one rule and walk away: "keep the water at the 9-gallon line." A sensor checks the level every few seconds; if it's dropped from evaporation, a splash, or the cat knocking a rock in, a little pump tops it back up. If it drifts too high, a valve drains a little out. You never told the system what would cause the water to drop — you didn't have to. You wrote the goal down once. Every object in Kubernetes works exactly like that tank: you write down what you want ("three Pods running," "a Service listening on port 80") and a robot exactly like your host for this page keeps checking, forever, and quietly fixes whatever's wrong — no matter what caused it.

🤖Your host for this topic: Recon the Robot — the reconciler, who lives entirely inside the watch-diff-act loop this page is about, so nobody on the Squad explains it better.

Every object shares one shape: metadata, spec & status

☺ Like you're 10: A name tag, a wish list, and a report card — every object, built-in or invented next year, is made of exactly those three parts.

Because the Kubernetes API is uniform, every object — a Pod, a Deployment, a StorageClass, a custom resource nobody has written yet — is built from the same skeleton: apiVersion, kind, metadata, spec, and status. Learn this shape once and you can read an object type you've never seen before.

metadata carries identity and bookkeeping: name and namespace, a server-assigned uid, labels and annotations (more on those below), and a resourceVersion the API server bumps on every write — kubectl and controllers use it for optimistic concurrency, so an update built from a stale read gets rejected instead of silently clobbering someone else's change. spec is the desired state, and you (or a higher-level controller acting on your behalf) are the only one who writes it. status is the observed state, and the controller that owns the object is the only one who writes it — never you, and never by hand. A controller's entire existence is: read spec, look at the world, do work, write status.

apiVersion: apps/v1          # which API group/version defines this shape
kind: Deployment              # the type
metadata:                     # identity + bookkeeping
  name: orders-api
  namespace: shop
  labels: { app: orders-api, tier: backend }
spec:                         # DESIRED state — you write this
  replicas: 3
  selector:
    matchLabels: { app: orders-api }
  template:
    metadata:
      labels: { app: orders-api, tier: backend }
    spec:
      containers:
        - name: orders-api
          image: registry.internal/orders-api:2.3.0
          ports: [{ containerPort: 8080 }]
status:                       # OBSERVED state — the Deployment controller writes this
  replicas: 3
  readyReplicas: 3
  observedGeneration: 5       # "I have reconciled up through generation 5"

That last field is worth pausing on. Every edit to spec bumps metadata.generation by one; status.observedGeneration is the controller reporting back "this is the generation I've actually reconciled." If the two numbers differ, the controller hasn't caught up yet — a fact kubectl rollout status leans on directly, and one worth checking before you assume a change has taken effect. The full mechanics of how the API server serves this shape — API groups, versions, discovery, the REST verbs behind kubectl — belong to the next lesson, the Kubernetes API & the controller pattern; if you want the senior-level tour of the same ground in one sitting, Platform Engineering's Kubernetes as the Platform Substrate covers it end to end.

◆ Key idea

spec is a wish, status is a report card. You write one, a controller writes the other, and mixing the two up — editing status by hand, or expecting spec to tell you what's actually running — is the single most common beginner confusion in this whole system.

Desired state vs actual state: the watch/reconcile loop

☺ Like you're 10: A controller doesn't wait to be told something broke. It keeps re-checking the whole picture on a loop, so it fixes things correctly even if it missed the exact moment they broke.

A controller is a small program running one loop, responsible for one kind of object: observe the desired state in spec, observe the actual state of the cluster, compute the difference, and act to shrink it. Kubernetes isn't one big program doing this — it's a federation of dozens of these loops running independently. A ReplicaSet controller makes the running Pod count match spec.replicas. A Node controller notices a node has stopped reporting. An Endpoints controller keeps a Service's list of healthy Pods current. None of them command each other; they all just watch the API and react to what they see.

The mechanism behind "watch" matters because it explains why this scales. list returns the current set of objects plus a resourceVersion cursor; watch opens a long-lived stream and pushes every change since that cursor. A controller lists once to build a local picture, then watches forever to keep it fresh, instead of polling the API in a loop of its own. But watch is only ever an optimization — a way to wake the loop up sooner. The correctness comes from being level-triggered, not edge-triggered: the loop always re-reads the full current level ("desired is 3, actual is 2") rather than trusting a stream of past events ("a Pod was deleted"). Miss an event because you restarted or a message dropped, and an edge-triggered system stays wrong forever. A level-triggered one just re-derives what to do on its very next pass. That's why a controller that was offline for an hour doesn't need to replay a backlog — it reconciles reality the moment it comes back, and "self-healing" isn't a feature bolted on top of Kubernetes, it's what falls out of this model for free.

spec desired state (you write this) 🤖 controller observe · diff · act cluster actual state (controller writes status) watch observe act to converge level-triggered: re-reads the whole picture every pass, so it self-heals no matter what it missed
# Watch the loop happen instead of just reading about it:
kubectl create deployment orders-api --image=nginx --replicas=3
kubectl get pods -l app=orders-api -w &

# In another terminal, kill one Pod by hand:
kubectl delete pod orders-api-7d9f4c-x8k2q

# The ReplicaSet controller notices actual (2) no longer matches
# desired (3) on its very next pass, and creates a replacement —
# usually inside a couple of seconds. Nobody paged anyone.
⚠ Watch out

A successful kubectl apply only means the API server accepted and stored your new spec — it does not mean the cluster already matches it. Reconciliation is asynchronous. Check status and its conditions (or run kubectl rollout status deployment/orders-api) before you assume a change has actually landed, especially in a script or a CI pipeline that moves on to the next step immediately after apply returns.

Labels and selectors: how objects find each other

☺ Like you're 10: A label is a sticky note on a box. A selector is "grab every box whose note says peaches" — it's never about names, only about matching notes.

Labels are key/value pairs in metadata.labels meant for exactly one purpose: being selected on. They're small, indexed, and used to group objects — app=orders-api, tier=backend, env=prod. Annotations live in the same spot in metadata but exist for the opposite reason: non-identifying information for tools and humans — a build SHA, a last-applied-config blob, a controller's own bookkeeping. You never select on an annotation, and unlike labels they can be arbitrarily large. A selector is a query over labels, either equality-based (app=orders-api) or set-based (tier in (backend, worker), env notin (dev), track exists).

This is the glue holding the whole object model together. A Service finds the Pods it load-balances to by label selector, never by IP. A ReplicaSet finds the Pods it owns by label selector. A NetworkPolicy targets the Pods it applies to by label selector. None of these objects know or care about individual Pod names — they care whether a Pod's labels satisfy their selector, full stop.

Service: orders-api selector: app=orders-api Pod A app=orders-api tier=backend Pod B app=orders-api tier=backend Pod C app=orders-api-canary tier=backend endpoint endpoint not selected — label doesn't match exactly
# Selectors are how you scope almost every kubectl read too:
kubectl get pods -l app=orders-api               # equality-based
kubectl get pods -l 'tier in (backend, worker)'   # set-based
kubectl get pods -l app=orders-api,tier=backend   # AND of two clauses (comma = AND)
⚠ Watch out

A Deployment's spec.selector is immutable after creation — you cannot edit it in place. If a Pod template's labels ever drift so they no longer satisfy the selector, the Deployment simply stops seeing "its" Pods and creates fresh ones instead of adopting the orphans, which is how teams end up with silently doubled Pod counts. Changing which Pods a Deployment manages means deleting and recreating the Deployment, not patching the selector.

🦆 Dot's-eye view

"I don't spend my day thinking about selectors. I just add tier: backend to a Pod template because a teammate told me to, and somewhere a Service starts routing traffic to it, a NetworkPolicy starts allowing it, and a Deployment starts counting it as one of its three. One typo in that label and none of those three things happen — and the error I get back isn't 'wrong label,' it's 'nothing showed up.' That's the part worth remembering: label mismatches fail silently, not loudly."

Namespaces: one cluster, many rooms

☺ Like you're 10: A namespace is a floor in the same office building — different doors, same building, same wiring, same landlord.

A namespace is a way to divide one cluster's objects into non-overlapping groups, mostly so two teams — or two environments — can both have a Deployment named orders-api without colliding. Most of what you'll create is namespace-scoped: Pods, Deployments, Services, ConfigMaps, Secrets. A smaller set of objects is cluster-scoped because they describe the cluster itself rather than belong to any one team: Nodes, PersistentVolumes, StorageClasses, and ClusterRoles among them — you can't put a Node "in" a namespace any more than you can put a floor's electrical panel on one floor of the building.

Every cluster starts with four namespaces already there: default (where an object lands if you don't specify one — fine for a demo, a trap in anything real, because it invites unrelated workloads to pile up together), kube-system (the cluster's own control-plane-adjacent objects — CoreDNS, kube-proxy, and friends), kube-public (readable by every authenticated user, rarely used for much), and kube-node-lease (one lightweight Lease object per node, the heartbeat the Node controller watches to decide a node has gone quiet). Beyond those, namespaces are yours to create for whatever grouping makes sense — per team, per environment, per application.

kubectl get namespaces
kubectl get pods -n kube-system                              # -n scopes a single command
kubectl config set-context --current --namespace=shop        # scope every command that follows
kubectl create namespace shop

# A Service is reachable from another namespace by its full DNS name:
#   ..svc.cluster.local
curl http://orders-api.shop.svc.cluster.local:8080/health

Two things namespaces are not, because the assumption trips up almost everyone at some point. A namespace is not a network boundary — by default, a Pod in shop can reach a Pod in any other namespace over the network exactly as freely as it can reach one next to it, until you write a NetworkPolicy to say otherwise. And a namespace is not a security boundary on its own — nothing stops a user from acting across namespaces unless a RBAC Role and RoleBinding scoped to that namespace says otherwise. A namespace is a grouping and a naming boundary; isolation is something you build on top of it with policy, not something it hands you for free. The layered defense this implies — namespaces plus RBAC plus NetworkPolicy plus admission control — gets the full treatment in security: defense in depth.

Put the four ideas on this page together and you have the lens for the entire rest of the course: an object is a spec you wrote plus a status a controller reports, a controller is a level-triggered loop closing the gap between the two, a label selector is how one object finds another without ever knowing its name, and a namespace is how a cluster stays organized without those objects colliding. The next lesson, the Kubernetes API & the controller pattern, opens the API server back up and shows exactly how a write travels through it; control plane internals goes further still, into where these loops physically run. If you'd rather see the same objects from the side of someone shipping an application day to day, DevOps's containers & orchestration covers the same Pod/Deployment/Service trio from that angle, and the CKA blueprint's own Workloads & Scheduling domain is where labels and selectors turn into exam-day muscle memory.

🎬 At the Pod Squad
👺

Gizmo: I wanted to change one label on a Deployment's Pods, so I just deleted the whole thing and reapplied a fresh copy. Way simpler than fighting with selectors.

🤖

Recon: That's not simpler, that's an outage on all three Pods where zero were required. spec.selector is immutable, sure — but spec.template.metadata.labels isn't. Edit that, and I roll the new labels out one Pod at a time. Nothing goes down.

🐿️

Nutty the Squirrel: And don't confuse the two! Labels are for matching — Services, selectors, NetworkPolicies all read them. Annotations are just notes to yourself. I keep a little filing system in my head for which is which.

🦊

Foxy: Okay but how do I actually know reconciliation finished, versus Recon just... hasn't gotten to it yet?

🤖

Recon: Compare metadata.generation to status.observedGeneration. Equal means I've seen your latest wish. Then check the actual status fields — readyReplicas, conditions — because "I've seen it" and "I've finished it" aren't the same claim.

🐢

Timmy the Turtle: And one more thing before anyone asks — putting something in its own namespace doesn't lock the door. That's an RBAC job, and a NetworkPolicy job. A namespace just keeps the name tags from colliding.

🐢 Timmy's checkpoint

1. What's the difference between spec and status on a Kubernetes object, and who is allowed to write each one? 2. Why does a level-triggered controller recover correctly even after missing several watch events, when an edge-triggered one wouldn't? 3. A Service isn't routing traffic to a Pod you just created — what's the first thing to check? 4. What's the practical difference between a label and an annotation? 5. Name one thing a namespace does NOT give you for free, and what you'd add to get it. 6. Why is spec.selector on a Deployment described as immutable, and what should you edit instead if you want to change a Pod's labels?

Check your answers
  1. spec is the desired state, written by you (or a higher controller acting for you); status is the observed state, written only by the controller that owns the object — never edited by hand.
  2. Because it never trusts the stream of past events as its source of truth — every pass it re-reads the full current desired state and full current actual state and re-derives the difference from scratch, so missed events just mean it acts one pass later, not that it acts wrong.
  3. Whether the Pod's labels actually satisfy the Service's selector — a typo or a missing label is the most common reason a Pod exists and runs fine but never appears as an endpoint.
  4. A label is identifying metadata meant to be selected on (small, indexed, used by Services/ReplicaSets/NetworkPolicies to find objects); an annotation is non-identifying metadata for tools and humans (can be large, is never selected on).
  5. Network isolation between Pods in different namespaces (needs a NetworkPolicy) or access isolation between users (needs RBAC Roles/RoleBindings scoped to that namespace) — a namespace alone only prevents naming collisions.
  6. Because changing which Pods a Deployment considers "its own" mid-flight would be dangerous to reconcile safely; instead you edit spec.template.metadata.labels, which the Deployment rolls out through its normal, safe, one-Pod-at-a-time update mechanism.