Platform Engineering in Depth · Kubernetes as the Platform Substrate

Kubernetes as the Platform Substrate

Almost every internal developer platform is built on Kubernetes — not because Kubernetes is a platform (it isn’t, and that’s the whole point of this course), but because it gives you the one thing a platform needs underneath it: a general-purpose, declarative, self-healing control plane you can extend into your own APIs. This page goes past the exam blueprint and into the machinery. We’ll take Kubernetes apart — the reconciliation model, the control plane, the API server, the object model, the nodes, the workload controllers, and the extension points — so that when you assemble a platform on top, you understand exactly what the bedrock is doing and why it behaves the way it does.

☺ Explain it like I’m 10

Imagine a giant magic notebook. You write down how you want the world to look — “I want three copies of my game server running, a door for people to reach it, and a box to save the scores.” You never do the work yourself. Instead, a swarm of tireless little helpers reads the notebook every few seconds, looks at how the world actually is, and quietly fixes anything that doesn’t match — if a server crashes, a helper starts a new one before you even notice. Kubernetes is that notebook plus the swarm of helpers. It doesn’t care what you write in the notebook, which is exactly why clever people can teach it brand-new words — like “give me a database” — and turn it into a platform.

🦉☁️Your hosts for this topic: Professor Owl & Nimbus — Owl is the Architect, drawing the blueprint and explaining why Kubernetes is shaped the way it is; Nimbus is the Kubernetes/cloud bedrock itself, the resource plane every platform stands on. Owl teaches the mechanisms; Nimbus is the mechanism.

The declarative model & control loops

☺ Like you’re 10: You say what you want, not the steps to get there. Little helpers keep checking and fixing until the world matches your wish — forever, not just once.

Everything else in Kubernetes is a consequence of one idea, so it’s worth getting exactly right. Kubernetes is declarative: you submit a description of the end state you want, and the system figures out the steps to reach and maintain it. This is the opposite of the imperative world most engineers grew up in — ssh to a box, run a command, hope it worked, and have no memory of what “correct” even meant afterward.

Desired state vs actual state

Two phrases run through the entire system. Desired state is what you asked for, recorded durably in the API (a Deployment that wants three replicas). Actual state is what is really running in the cluster right now (two replicas, because a node died). The gap between them is the system’s entire reason to exist, and closing that gap — continuously — is what a controller does. You’ll meet this exact pattern again in GitOps, where Git holds the desired state and a reconciler closes the gap; GitOps is simply this Kubernetes idea pushed one level out, to the repo.

Controllers everywhere

A controller is a small program running one loop: observe the desired state, observe the actual state, compute the difference, and act to shrink it. Kubernetes is not one big program — it is a federation of dozens of these loops, each responsible for one kind of object. A ReplicaSet controller makes the number of Pods match; a Node controller notices dead nodes; an Endpoints controller keeps a Service’s member list current. None of them command each other. They all watch the API and react. This is why the design scales conceptually: to add a new capability, you add another loop, not a special case in a monolith. When you extend Kubernetes with your own controller later, you’re joining this federation as a first-class citizen.

◆ Key idea

Kubernetes = a declarative API + a swarm of independent control loops that reconcile actual state toward the desired state stored in that API. Master this one sentence and every component below is just a detail of how.

Level-triggered, not edge-triggered

Here is the subtlety that makes the whole thing robust. Controllers are level-triggered, not edge-triggered. An edge-triggered system reacts to events (“a Pod was deleted!”) and is fragile: miss the event — because you restarted, or the message was dropped — and you’re permanently wrong. A level-triggered system reacts to the current level (“the desired count is 3, the actual count is 2”), so it converges no matter how many events it missed. Kubernetes uses events (watches) only as an optimization to wake the loop up sooner; the loop’s correctness comes from always re-reading the full desired-and-actual level and re-deriving what to do. That’s why a controller that was offline for an hour simply reconciles reality on its next pass instead of replaying a backlog. It’s also why “self-healing” isn’t a bolt-on feature — it falls out of the model for free.

The control plane

☺ Like you’re 10: The “brain” of the cluster — a front desk that takes all requests, a memory that never forgets, a planner that decides where things go, and the helpers that do the fixing.

The control plane is the set of components that make global decisions and drive the cluster. On a real cluster these run as static pods or managed services (in EKS/GKE/AKS you don’t even see them). Four components form the core; a fifth handles cloud integration.

CONTROL PLANE kube-apiserver REST API · authn/z · admission etcd state store · Raft kube-scheduler kube-controller-manager cloud-controller-manager only door watch + act WORKER NODE ( ×N ) kubelet node agent containerd / CRI-O runtime, via CRI → runc kube-proxy Service rules Pod Pod Pod kubelet watches its pods networking via CNI · storage via CSI

kube-apiserver — the only door

The kube-apiserver is the front desk and the single most important component to understand. It exposes the Kubernetes REST API, and every read and write — from kubectl, from controllers, from the kubelet — goes through it. It is also the only component that talks to etcd; nothing else touches the datastore directly. For each request it runs a fixed pipeline: authentication (who are you?), authorization (are you allowed? — usually RBAC), admission control (should this specific object be allowed or modified? — see webhooks), then schema validation, then persistence to etcd. Because it’s stateless, you run several replicas behind a load balancer for availability. When people say “the API is the platform,” this is the process they mean.

etcd — the cluster’s memory

etcd is a distributed, strongly-consistent key-value store, and it holds the entire desired-and-actual state of the cluster — every object you’ve ever created. It uses the Raft consensus algorithm, so it needs a quorum (majority) of members to accept writes; that’s why production clusters run 3 or 5 etcd members, never an even number and never one. If etcd is lost and unbacked-up, the cluster’s memory is gone. Treat it as the crown jewels: encrypt it at rest, restrict access to it to the apiserver alone, and back it up on a schedule. Every “desired state” you’ve read about lives, ultimately, as a row in etcd.

⚠ Watch out

Because etcd is the source of truth, two operational rules are non-negotiable. First, back it up — an etcd snapshot is your cluster’s restore point; without it a disk failure is a total loss. Second, protect it — anyone who can read etcd can read every Secret in the cluster (Secrets are only base64-encoded in there unless you enable encryption-at-rest). On managed control planes the provider does this for you; on self-managed clusters it’s your job.

kube-scheduler — deciding where

When you create a Pod, its spec.nodeName is empty — it’s unscheduled. The kube-scheduler watches for these and assigns each one to a node in two phases: filtering (which nodes are even feasible? — enough CPU/memory, satisfies node selectors, tolerates the node’s taints, has the requested volumes) and scoring (of the feasible nodes, which is best? — spread, affinity, least-loaded). It then “binds” the Pod by writing the chosen node back to the API. Crucially, the scheduler only decides; it never starts a container. That’s the kubelet’s job. The knobs you use to influence it — resource requests, affinity/anti-affinity, taints and tolerations, topology spread — are a topic in their own right; see Scheduling & Scaling.

controller-manager & cloud-controller-manager

The kube-controller-manager is a single binary that runs most of the built-in control loops as goroutines: the Deployment, ReplicaSet, Job, Node, Namespace, ServiceAccount, and Endpoints controllers, among many others. Packaging them together is an operational convenience; conceptually each is still an independent reconciler watching the API. The cloud-controller-manager splits out the loops that must talk to a specific cloud provider so the core stays provider-neutral: the node controller (ask the cloud whether a missing node was really deleted), the route controller (program cloud network routes), and the service controller (provision a cloud load balancer when you create a Service of type LoadBalancer). This split is why the same Kubernetes runs identically on AWS, GCP, Azure, and bare metal — the cloud-specific glue is quarantined in one place.

ComponentRuns whereIts one job
kube-apiserverControl plane (replicated)Serve the REST API; gate every request (authn/authz/admission); the only writer to etcd.
etcdControl plane (3–5 members)Durably store all cluster state with Raft consensus — the single source of truth.
kube-schedulerControl planeAssign each unscheduled Pod to a feasible, well-scored node.
kube-controller-managerControl planeRun the built-in reconciliation loops (ReplicaSet, Node, Job, Endpoints…).
cloud-controller-managerControl planeRun the cloud-specific loops (LB provisioning, routes, node lifecycle).
kubeletEvery nodeMake its node’s Pods real via the runtime; report health back.

The API machinery

☺ Like you’re 10: Kubernetes speaks one tidy language for everything. Learn how the words are spelled once, and you can read and write any object — even ones that don’t exist yet.

The reason Kubernetes can become a platform is that its API is uniform and self-describing. Pods, Services, Deployments, and your own custom types are all handled by the same request machinery. Understanding that machinery is what separates someone who copies YAML from someone who can extend the system.

Resources, objects, and Kinds

Three words get muddled constantly, so let’s pin them down. A Kind is a type (Pod, Deployment) — it’s what you put in the kind: field. An object is a specific persisted instance of a Kind (the Deployment named checkout). A resource is the REST endpoint that stores a collection of objects (deployments, the lowercase-plural URL segment). One Kind is usually served by one resource, but not always — the same Pod Kind is exposed both as the pods resource and as the pods/status and pods/log subresources.

GVK vs GVR

Two nearly-identical triples cause endless confusion. GVK — Group/Version/Kind — is what you write: apiVersion: apps/v1 gives the group (apps) and version (v1), and kind: Deployment gives the Kind. GVR — Group/Version/Resource — is what the server exposes as a URL path: /apis/apps/v1/namespaces/shop/deployments. A component called the RESTMapper translates the GVK you wrote into the GVR the server needs. The core group is special: it has an empty group name, so its objects use a bare apiVersion: v1 and live under /api/v1/… rather than /apis/….

# GVK (Group/Version/Kind) is what you write in YAML:
#     apiVersion: apps/v1   +   kind: Deployment
# GVR (Group/Version/Resource) is the REST path the server serves:
#     /apis/apps/v1/namespaces/shop/deployments

kubectl get deployments.v1.apps -n shop      # a fully-qualified GVR
kubectl api-resources | grep -iE 'name|deployment'
#   NAME          SHORTNAMES   APIVERSION   NAMESPACED   KIND
#   deployments   deploy       apps/v1      true         Deployment

kubectl api-versions                    # every Group/Version the server serves
kubectl explain deployment.spec.strategy   # reads the OpenAPI schema live

The REST surface: list, watch & informers

Every resource supports the same small set of verbs: get, list, watch, create, update, patch, delete, and deletecollection. Two of these power the entire control-loop architecture. list returns the current set of objects plus a resourceVersion — a cursor. watch opens a long-lived stream and delivers every change (added/modified/deleted) since that resourceVersion. Together they let a client build a perfectly accurate local mirror of the cluster: list once to seed it, then watch forever to keep it fresh.

Doing that raw would hammer the apiserver, so client libraries wrap it in an informer: a shared, in-memory cache backed by a single list+watch, which hands cached reads to controllers and fires callbacks on changes. This is why a busy cluster with hundreds of controllers doesn’t melt the apiserver — almost every read is served from an informer cache, not a fresh API call. When you write a controller, you don’t poll; you register with an informer and get woken when something you care about changes.

API groups, versioning & discovery

Resources are organized into API groups (apps, batch, networking.k8s.io, and the empty core group) so the surface can grow without collisions. Each group has independently-evolving versions that signal stability: v1alpha1 (experimental, may vanish), v1beta1 (well-tested, may still change), and v1 (stable, forward-compatible). The apiserver can serve several versions of one Kind at once and converts between them on the fly, persisting just one “storage version” in etcd — which is how a cluster upgrades an API without you rewriting your manifests. Finally, two self-describing endpoints make all of this machine-readable: the discovery API (/apis) lists every group, version, and resource available, and the OpenAPI schema (/openapi/v3) publishes the exact shape of every field. kubectl explain, client-side validation, and server-side apply all read that schema — the API literally documents itself.

🦆 Dot’s-eye view

“I don’t know or care what a ‘GVR’ is. But I love that kubectl explain database.spec tells me exactly what fields my platform’s ‘Database’ thing accepts, the same way it does for a built-in Pod. When the platform team added their own types, my tools just… understood them. That’s the whole trick, isn’t it — new nouns that feel native.”

The Kubernetes object model

☺ Like you’re 10: Every single thing in Kubernetes is written the same way: a name tag, a wish, and a report card. Once you know that shape, you can read anything.

Because the API is uniform, every object — built-in or custom — shares the same skeleton: apiVersion, kind, metadata, spec, and status. Learn this shape once and you can read a resource you’ve never seen before.

metadata, spec & status

The separation of spec from status is the object model’s most important design choice. spec is the desired state, written by you (or a higher controller). status is the observed state, written by the controller that owns the object, never by you. A controller’s whole life is “read spec, look at the world, do work, write status.” The metadata block carries the object’s identity and bookkeeping: name and namespace, a server-assigned uid, a resourceVersion (used for optimistic concurrency — your update is rejected if someone changed the object since you read it), a generation counter, plus labels, annotations, owner references, and finalizers.

apiVersion: apps/v1          # GROUP/VERSION → group "apps", version "v1"
kind: Deployment             # KIND → the object type
metadata:                    # identity + bookkeeping
  name: checkout
  namespace: shop
  labels: { app: checkout, tier: web }
spec:                        # DESIRED state — you write this
  replicas: 3
  selector:
    matchLabels: { app: checkout }
  template:
    metadata:
      labels: { app: checkout }
    spec:
      containers:
        - name: server
          image: registry.acme.io/checkout:1.8.2
          ports: [{ containerPort: 8080 }]
status:                      # OBSERVED state — the controller writes this
  replicas: 3
  readyReplicas: 3
  observedGeneration: 4      # "I have reconciled up to generation 4"

Labels vs annotations & selectors

Both are key/value maps in metadata, but they exist for opposite reasons. Labels are identifying metadata meant for selection: they’re indexed, kept short, and used to group objects. Annotations are non-identifying metadata meant for tools and humans: build SHAs, checksums, last-applied config, ingress controller settings. You never select on annotations, and they can be large. A selector is a query over labels — equality-based (env=prod) or set-based (tier in (web, api), track notin (canary), app exists). This is the glue of the whole system: a Service finds its Pods by label selector, a ReplicaSet owns Pods by label selector, a NetworkPolicy targets Pods by label selector. Get your labels wrong and a Service silently routes to the wrong Pods.

LabelsAnnotations
PurposeIdentify & group objectsAttach arbitrary metadata for tools/humans
Selectable?Yes — the basis of selectorsNo
Size / shapeSmall, constrained, indexedCan be large & structured
Typical useapp=checkout, tier=web, env=prodBuild SHA, kubectl.kubernetes.io/last-applied…, controller config

Owner references & garbage collection

How does deleting a Deployment also delete its Pods, when the Deployment controller never explicitly commands it? Through owner references. Every dependent object carries an ownerReferences entry pointing at its owner: Pods reference their ReplicaSet, the ReplicaSet references its Deployment. A dedicated garbage collector controller watches for owners that no longer exist and deletes their orphaned dependents. You choose the cascade behavior at delete time: background (delete the owner now, clean up dependents async — the default), foreground (delete dependents first, then the owner), or orphan (delete only the owner, leaving dependents behind). This is pure emergent behavior — no controller has a hard-coded “also delete the Pods” line; ownership plus GC produces it.

Finalizers

Sometimes an object needs cleanup that happens elsewhere before it can truly vanish — deleting a cloud load balancer, releasing an external DNS record, tearing down a database. A finalizer makes that safe. When you request deletion of an object carrying finalizers, the apiserver does not remove it; it merely sets a metadata.deletionTimestamp. The object now lingers in a “terminating” state until every controller that registered a finalizer does its cleanup and removes its own finalizer key. Only when the finalizer list is empty does the object actually disappear. This is why a stuck namespace sits in Terminating forever — some resource inside it has a finalizer whose controller can’t complete. When you build an operator that provisions real infrastructure, finalizers are how you guarantee you don’t leak cloud resources when someone deletes the custom object.

# A ReplicaSet owns its Pods via ownerReferences (this is what enables GC):
kubectl get pod checkout-7d9f-x8 -o jsonpath='{.metadata.ownerReferences}'
#   [{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"checkout-7d9f",
#     "controller":true,"blockOwnerDeletion":true,"uid":"..."}]

# Finalizers pause deletion until cleanup runs; a stuck one = stuck "Terminating":
kubectl get ns shop -o jsonpath='{.spec.finalizers}'   # ["kubernetes"]
kubectl delete deployment checkout    # cascades: Deployment → ReplicaSet → Pods

The node / data plane

☺ Like you’re 10: The brain decides; the hands do. On every worker machine, one worker reads its orders and actually starts the containers.

The control plane decides; the data plane — the worker nodes — does the actual running of containers. Each node runs three things.

kubelet — the node agent

The kubelet is the agent on every node and the bridge between the abstract API and real Linux processes. It watches the apiserver for Pods bound to its node, then makes them real: pulling images, asking the runtime to start containers, mounting volumes/ConfigMaps/Secrets, and running the health probes — liveness (restart if unhealthy), readiness (remove from Service endpoints until ready), and startup (protect slow starters). It continuously reports Pod and Node status back to the API. One boundary matters: the kubelet only manages containers Kubernetes created. A container you start by hand on the node is invisible to it — nothing reconciles it, nothing restarts it — which is exactly the shortcut you must never take.

The Container Runtime Interface

The kubelet does not itself know how to run a container; it speaks the Container Runtime Interface (CRI), a gRPC contract, to a pluggable runtime. The mainstream implementations are containerd and CRI-O; both ultimately call a low-level OCI runtime (usually runc, or sandboxed options like gVisor and Kata Containers) that creates the Linux namespaces and cgroups a container actually is. This is why the removal of “dockershim” in Kubernetes 1.24 changed so little for users: Docker Engine never spoke CRI itself — the kubelet reached it through a built-in shim (dockershim), and once that shim was dropped, containerd (which Docker uses underneath anyway) simply became the direct CRI runtime. CRI is the first of the great pluggable contracts — CRI for runtime, CNI for networking, CSI for storage — that let Kubernetes define a shape and let the ecosystem supply the implementation.

The Pod & its lifecycle

The Pod is the smallest deployable unit — not a container, but a small group of co-located containers that share a network namespace (one IP, reachable on localhost between them) and can share volumes. A hidden “pause” (sandbox) container holds those shared namespaces open. A Pod moves through phases: Pending (accepted, not yet all-running — pulling images or waiting to schedule), Running, then a terminal Succeeded or Failed, with Unknown if the node stops reporting. Finer-grained conditions (PodScheduled, Initialized, ContainersReady, Ready) track progress, and init containers run to completion in order before the app containers start. Critically, a bare Pod is not self-healing — if its node dies, it’s gone. Durability comes from wrapping Pods in a controller, which is the next section.

Static pods & kube-proxy

Two node-level details close out the data plane. A static pod is one the kubelet runs directly from a manifest file on disk (typically /etc/kubernetes/manifests), with no controller and often no apiserver involved — the apiserver only sees a read-only “mirror pod.” This is the bootstrap trick: on a kubeadm cluster the apiserver, etcd, scheduler, and controller-manager themselves run as static pods, because you need something to start the control plane before the control plane exists. Meanwhile kube-proxy implements the Service abstraction on each node: it watches Services and their Endpoints and programs kernel rules (iptables or IPVS) so that traffic to a Service’s stable ClusterIP is load-balanced to the current healthy Pods. Modern eBPF dataplanes (Cilium) can replace kube-proxy entirely — another pluggable contract at work. The full story of Services, DNS, and CNI lives in Networking.

Workload controllers

☺ Like you’re 10: Different helpers for different jobs — one keeps N copies alive, one gives each copy a name and a locker, one puts a copy on every machine, and one runs a task until it’s done.

You rarely create Pods directly. Instead you declare a workload controller whose spec describes Pods and whose loop keeps them alive. Choosing the right one is a core platform-design skill, because each makes a different guarantee.

Deployment & ReplicaSet

A ReplicaSet has one job: keep exactly N Pods matching its selector alive, creating or deleting Pods to converge. You almost never write one directly, because a Deployment manages ReplicaSets for you and adds rollouts. When you change a Deployment’s Pod template (say, a new image), the Deployment controller creates a new ReplicaSet and shifts replicas from old to new gradually, honoring maxSurge (how many extra Pods may exist during the roll) and maxUnavailable (how many may be missing). It keeps old ReplicaSets around as revision history so kubectl rollout undo is instant. Deployments are the default home for stateless services; layering safe progressive rollouts (canary, blue-green) on top is covered in CI/CD & Progressive Delivery.

StatefulSet

A StatefulSet is for workloads where identity matters — databases, message brokers, quorum systems. Unlike a Deployment’s interchangeable Pods, it gives each Pod three guarantees: a stable, ordinal name (db-0, db-1, …) that survives restarts; stable storage via volumeClaimTemplates, which mint one PersistentVolumeClaim per Pod that reattaches to the same Pod on reschedule; and ordered creation, scaling, and rolling updates (db-0 is ready before db-1 starts). That ordering and sticky storage is exactly what a database replica needs and exactly what a stateless web app doesn’t. The deeper story of PVs, storage classes, and stateful data lives in Storage & State.

DaemonSet

A DaemonSet ensures one copy of a Pod runs on every node (or every node matching a selector). It’s the pattern for node-level infrastructure: log collectors, metrics agents (node-exporter), the CNI plugin itself, security sensors. As nodes join the cluster the DaemonSet controller automatically schedules its Pod onto them; as nodes leave, those Pods go with them. Because these are platform components rather than app workloads, DaemonSets are usually installed by the platform team and are invisible to developers — the plumbing under the golden path.

Job & CronJob

The controllers above keep Pods running forever; a Job runs Pods to completion. It tracks completions (how many successful runs you need) and parallelism (how many at once), retries failures up to backoffLimit, and finishes when the target count of successes is reached — ideal for migrations, batch processing, and one-shot tasks. A CronJob wraps a Job with a schedule, creating a fresh Job on a cron expression, with controls for concurrencyPolicy (may runs overlap?), startingDeadlineSeconds (skip a missed run rather than pile up), and history limits. Together they turn Kubernetes into a scheduler for finite work, not just long-running services.

API (spec) desired state 🦉 controller observe · diff · act Cluster actual state watch observe act to converge level-triggered: re-reads the full state every pass, so it self-heals no matter what it missed
ControllerGuaranteeReach for it when…
DeploymentN interchangeable replicas + rolling updates & rollbackStateless services (web, API) — the default.
StatefulSetStable identity, sticky per-Pod storage, ordered rolloutDatabases, brokers, quorum systems.
DaemonSetExactly one Pod per (matching) nodeNode agents: logging, metrics, CNI, security.
JobRun Pods to successful completion, with retriesMigrations, batch, one-shot tasks.
CronJobCreate a Job on a scheduleRecurring batch: backups, reports, cleanups.
🦉 Professor Owl’s workshop · 20 min

On a throwaway cluster (kind or minikube), run kubectl create deploy web --image=nginx --replicas=3, then in one terminal watch kubectl get pods -w and in another delete a Pod with kubectl delete pod <name>. Watch the ReplicaSet controller replace it in seconds — that’s the level-triggered loop defeating reality. Now run kubectl get rs and note the ReplicaSet the Deployment created for you; edit the Deployment’s image and watch a second ReplicaSet appear as the rollout shifts replicas across. Three commands, and the whole control-loop model becomes something you’ve seen, not just read.

Extending the platform

☺ Like you’re 10: The best part — you can teach the magic notebook brand-new words. Add “Database” to its vocabulary and it treats your invention exactly like a built-in.

Here is where Kubernetes stops being a container tool and becomes, in the CNCF’s own framing, “a platform for building platforms.” Three extension points let you add your own APIs and behaviors without forking anything.

Custom Resource Definitions

A CustomResourceDefinition (CRD) registers a brand-new Kind with the apiserver. Once applied, the apiserver serves full CRUD, list/watch, validation, and RBAC for your type exactly as if it were built in — it’s stored in etcd, shows up in kubectl get, and obeys an OpenAPI v3 structural schema you supply. On its own a CRD is just storage for a new noun. Pair it with a custom controller that watches those objects and does real work — provisioning a cloud database, wiring a DNS record — and you have the operator pattern: your domain concept becomes a native, reconciled Kubernetes API. This is the literal mechanism behind self-service “give me a database” buttons; the full treatment is in Platform APIs, CRDs & Operators, and the tools that turn CRDs into cloud resources live in IaC Control Planes.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.platform.acme.io      # <plural>.<group>
spec:
  group: platform.acme.io
  scope: Namespaced
  names: { kind: Database, plural: databases, shortNames: [db] }
  versions:
    - name: v1alpha1
      served: true
      storage: true                     # exactly one version is the storage version
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [engine, sizeGiB]
              properties:
                engine:  { type: string, enum: [postgres, mysql] }
                sizeGiB: { type: integer, minimum: 10 }
# Now `kubectl get databases` works, and a controller can reconcile them into real DBs.

The API aggregation layer

CRDs are the easy path, but they inherit etcd as their storage and the apiserver’s request handling. When you need more — custom storage, computed/virtual resources that aren’t persisted, or high-performance protobuf — you use the API aggregation layer. You run your own extension apiserver and register it with an APIService object; the kube-apiserver then transparently proxies requests for that group/version to your server. The canonical example is metrics.k8s.io, served by the metrics-server: kubectl top reads live, un-stored, computed metrics through an aggregated API. Reach for aggregation only when a CRD genuinely can’t express what you need — it’s far more work to build and operate.

Admission webhooks

The third extension point intercepts writes at the moment they enter the apiserver, after authn/authz but before persistence. Mutating admission webhooks can modify an incoming object — this is how a service mesh injects a sidecar, or a platform stamps default labels and resource limits onto every Pod. Validating admission webhooks can only accept or reject — “no container may run as root,” “every workload must carry a cost-center label.” The chain runs mutating webhooks first, then schema validation, then validating webhooks, then the write. These are the enforcement teeth behind policy-as-code and admission controllers like Kyverno and OPA Gatekeeper (and the newer in-tree ValidatingAdmissionPolicy, which expresses rules in CEL with no webhook server at all).

kubectl Authn who? Authz RBAC Mutating webhooks Schema validate Validating policy etcd every write runs this gauntlet inside the kube-apiserver
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata: { name: image-policy.acme.io }
webhooks:
  - name: images.acme.io
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
    clientConfig:
      service: { name: image-policy, namespace: platform, path: /validate }
    admissionReviewVersions: ["v1"]
    sideEffects: None
    failurePolicy: Fail       # "closed": if the webhook is down, REJECT the write
⚠ Watch out

failurePolicy: Fail makes a webhook a hard dependency of every matching write. If your policy service is down, creating Pods stops cluster-wide — a self-inflicted outage. Scope webhook rules as narrowly as possible, exclude the kube-system namespace, and think hard before making the control plane depend on a Pod you deployed. Admission is powerful precisely because it sits in the critical path — respect that.

What Kubernetes is — and is not

☺ Like you’re 10: Kubernetes is a super-strong engine and chassis — not a finished car. Someone still has to add the steering wheel, seats, and dashboard so a normal person can drive. That “someone” is you.

We’ve now seen the whole machine, so we can state plainly what it is. Kubernetes is a declarative, extensible control plane for infrastructure — a uniform API plus a swarm of reconciling controllers. That is enormously powerful, and it is also the reason Kubernetes is not, by itself, something you can hand to a developer.

A resource orchestrator, not a PaaS

Out of the box, Kubernetes gives you no source-to-URL build, no CI/CD, no application catalog, no opinionated “right way” to ship a service, no real secrets management (Secrets are base64, not encrypted, until you add encryption), and no golden path. It gives you primitives — Pods, Services, controllers, an extensible API — and expects you to compose them. A Heroku-style PaaS (git push and your app is live) is a product built on top; Kubernetes is the substrate that product would run on. Confusing the two is the classic mistake: handing raw kubectl and YAML to product developers and calling it a platform.

Batteries not included — the pluggable contracts

Kubernetes’ deliberate design is to define contracts and let you choose implementations. Networking is a contract (CNI) — you install Calico, Cilium, or a cloud plugin; Kubernetes ships none. Storage is a contract (CSI) — you install a driver for EBS, or Ceph, or your NAS. The runtime is a contract (CRI). Ingress needs a controller you supply; even in-cluster DNS is an add-on (CoreDNS). This modularity is a feature — it’s why Kubernetes runs everywhere — but it means a “vanilla” cluster is a kit of parts, not a finished environment. Someone has to choose, install, integrate, secure, and operate all of it consistently. Which brings us to the point of the entire course.

The gap platform engineering fills

Between “raw Kubernetes primitives” and “a developer who just wants to ship” lies a wide gap, and closing it is platform engineering. The platform team assembles the pluggable pieces into a coherent whole, wraps the sharp primitives in self-service APIs and golden paths, bakes in policy guardrails so the easy way is the safe way, wires up observability by default, and manages it all with GitOps — so that Dot the Duck files no tickets and touches no YAML, yet everything underneath is the same reconciled Kubernetes we just dissected. The platform reference architecture is exactly the blueprint for that assembly, and What & Why We Platform is the argument for doing it at all. Kubernetes is the substrate; the platform is what you build on it.

◆ Key idea

Kubernetes gives you a declarative API and control loops for infrastructure — and stops there, on purpose. It is a superb substrate and a poor product. Platform engineering is the discipline of turning that substrate into a product: golden paths, self-service APIs, and guardrails that make the safe path the easy path.

🎬 At the Platform Guild
🦊

Foxy: So Kubernetes is basically a fancy way to run Docker containers, right?

🦉

Professor Owl: That’s the sliver you see, not the shape underneath. Kubernetes is a declarative API and a swarm of control loops. Running containers is just one thing it reconciles — the same machinery can reconcile load balancers, storage, or nouns you invent.

☁️

Nimbus: I’m the bedrock. Hand me a spec and I keep reality matching it — a node dies, I reschedule; drift appears, a controller erases it. I don’t care whether the spec says “Pod” or “Database.” Teach me the word and I’ll keep it true.

👺

Gizmo: Ugh, so much machinery. Just SSH into a node and docker run the container yourself — instant, no YAML, no controllers! 🤑

🐢

Timmy: And the second that node reboots, your container’s gone forever — no controller is watching it, Gizmo. A hand-run container is a snowflake. Let the reconciler own it, or don’t run it.

🦆

Dot: Honestly? I don’t want to know what a kubelet is. I just want “give me a database” to work. Make the substrate someone else’s job — that’s literally why the platform exists.

You now understand the bedrock every later lesson stands on: a declarative API, reconciling controllers, a control plane and data plane, an object model, and the extension points that let you grow your own platform APIs. From here, the course builds upwardturning CRDs into self-service, wiring the network, handling state, and assembling the whole platform — but it all reconciles down to what you just took apart.

🐢 Timmy’s checkpoint

1. In one sentence, what is the difference between desired and actual state, and what closes the gap? 2. Why does “level-triggered” make controllers self-healing where “edge-triggered” would be fragile? 3. Name the four core control-plane components and one job of each — and which one is the only thing that talks to etcd. 4. What’s the difference between a GVK and a GVR? 5. Labels vs annotations — which can you select on, and name one use for each. 6. Give two ways to extend the Kubernetes API, and explain why we say “Kubernetes is a platform for building platforms.”

Check your answers
  1. Desired state is what you declared in the API (e.g. 3 replicas); actual state is what’s really running (e.g. 2). A controller continuously reconciles actual toward desired to close the gap.
  2. A level-triggered loop re-reads the full desired-and-actual state every pass and re-derives what to do, so it converges no matter how many events it missed; an edge-triggered system reacts to one-off events and is permanently wrong if it drops or misses one.
  3. kube-apiserver (serves the REST API, gates every request, the only component that talks to etcd), etcd (the consistent state store / source of truth), kube-scheduler (assigns unscheduled Pods to nodes), kube-controller-manager (runs the built-in reconciliation loops). The apiserver is the sole writer to etcd.
  4. GVK = Group/Version/Kind, what you write in YAML (apps/v1 + Deployment); GVR = Group/Version/Resource, the lowercase-plural REST path the server exposes (/apis/apps/v1/…/deployments). A RESTMapper translates one to the other.
  5. You select on labels, never annotations. Labels identify/group (app=checkout) and drive Services, ReplicaSets, and NetworkPolicies; annotations hold non-identifying metadata for tools/humans (build SHA, last-applied config, controller settings).
  6. Any two of: CRDs (register a new Kind the apiserver serves natively), the API aggregation layer (proxy a group/version to your own extension apiserver), and admission webhooks (mutate/validate writes). We call it “a platform for building platforms” because the same uniform API, storage, watch, and RBAC that serve built-in types serve your types too — so you can add first-class platform APIs (like “Database”) without forking Kubernetes.