Platform APIs, CRDs & Operators

Kubernetes isn’t just a place to run containers — it’s a platform for building platforms. Its API is extensible: teach it brand-new nouns like kind: Database or kind: Environment, then write a tireless program that makes them come true. This lesson is the first half of the exam’s Platform APIs domain: designing a Custom Resource Definition, how the operator/controller pattern turns one line into real infrastructure, and choosing a framework. The caterpillar-to-butterfly move: ugly YAML in, clean self-service API out.

Why extend the Kubernetes API at all

☺ Like you’re 10: Instead of a 200-page manual, you hand your friend one button: “make me a sandwich.” The platform is the buttons you invent.

The mental shift that unlocks the domain: the platform you’re building isn’t a website or a tool — it’s a set of APIs. When Dot wants a database, the truth is a pile of tightly-coupled Kubernetes objects: a StatefulSet, a headless Service, a PersistentVolumeClaim, a Secret, a ConfigMap, a backup CronJob, maybe a NetworkPolicy. Two hundred lines of YAML Dot must copy, keep consistent, and never get wrong — a caterpillar: functional, but not something you’d hand a developer.

Extending the API hides all that behind a new resource type developers write. Kubernetes was built for this: the API server treats custom types as first-class citizens, so they get the same kubectl get, RBAC, audit log, and GitOps reconciliation as a built-in Pod. You’re not bolting a portal onto Kubernetes — you’re teaching it your platform’s vocabulary.

CRDs: teaching Kubernetes a new noun

☺ Like you’re 10: A CRD is the dictionary entry: it tells Kubernetes “a Database is a real word now, and here’s what a valid one looks like.”

A CustomResourceDefinition (CRD) is itself a Kubernetes object: kubectl apply it once and the API server serves a new endpoint. Register Database and you instantly get kubectl get databases, create/update/delete, watch, and RBAC verbs — no code required to store and validate objects. The CRD is the schema and front door; the operator gives the noun behaviour.

Anatomy of a CustomResourceDefinition

The CRD names your type and its shape. A few exam rules: metadata.name must be exactly <plural>.<group>; scope is Namespaced or Cluster (global, like a Node); and every served version carries its own schema. Here’s a real Database, with the pieces the exam loves — an OpenAPI schema, additionalPrinterColumns, the /status subresource, shortNames, and categories:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.platform.acme.io        # MUST be <plural>.<group>
spec:
  group: platform.acme.io
  scope: Namespaced                        # or Cluster
  names:
    kind: Database
    plural: databases
    singular: database
    shortNames: [db]                       # kubectl get db
    categories: [platform, acme]           # kubectl get platform → shows all our CRs
  versions:
    - name: v1alpha1
      served: true                         # this version is reachable via the API
      storage: true                        # exactly ONE version is the stored one
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [engine, size]
              properties:
                engine:  { type: string, enum: [postgres, mysql] }
                version: { type: string, default: "16" }
                size:    { type: string, enum: [small, medium, large], default: small }
                highAvailability: { type: boolean, default: false }
            status:                        # the operator writes here, not the user
              type: object
              properties:
                phase:    { type: string }
                endpoint: { type: string }
                conditions:                # the standard metav1.Condition array
                  type: array
                  x-kubernetes-list-type: map
                  x-kubernetes-list-map-keys: [type]
                  items:
                    type: object
                    required: [type, status, lastTransitionTime, reason, message]
                    properties:
                      type:               { type: string }
                      status:             { type: string, enum: ["True", "False", "Unknown"] }
                      reason:             { type: string }
                      message:            { type: string }
                      lastTransitionTime: { type: string, format: date-time }
      subresources:
        status: {}                         # status gets its own endpoint; main writes ignore it
      additionalPrinterColumns:
        - { name: Engine, type: string, jsonPath: .spec.engine }
        - { name: Size,   type: string, jsonPath: .spec.size }
        - { name: Phase,  type: string, jsonPath: .status.phase }
        - { name: Age,    type: date,   jsonPath: .metadata.creationTimestamp }

The OpenAPI schema — validation & defaulting for free

☺ Like you’re 10: The schema is bouncer and helper: it turns away nonsense (“engine: banana”) and fills in blanks you left empty (“no size? here’s ‘small’”).

The openAPIV3Schema earns a platform API its keep. In apiextensions.k8s.io/v1 it must be a structural schema (every field typed), enforced before your operator runs. Two powerful things for zero code: validation (enum, required, minimum/maximum, pattern, and CEL x-kubernetes-validations rules reject a bad Database at apply time) and defaulting (a default: is filled in on write, so version and size get sane values when Dot omits them). Bad requests bounce with a clear error, so your controller only sees well-formed objects. A structural schema also prunes: any field you didn’t declare is silently dropped on write — which is why every field your operator writes into status (conditions included) has to appear in the schema.

Printer columns, subresources, shortNames & categories

☺ Like you’re 10: Polish that makes your new brick feel store-bought: a nice table when you list them, and a safe “report card” only the robot writes on.

Four touches separate a toy CRD from a real platform API:

• additionalPrinterColumns — control what kubectl get databases shows. Surface .status.phase and the endpoint, and an operator’s progress becomes visible at a glance instead of buried in -o yaml.
• The /status subresource — split status onto its own endpoint so users edit spec (desired) while the operator owns status (observed). It also prevents a status write from bumping metadata.generation, which is how a controller cheaply detects “did the spec actually change?”
• The /scale subresource — declare a scale subresource mapping a replicas path, and suddenly kubectl scale and even the HorizontalPodAutoscaler can target your custom resource, just like a Deployment.
• shortNames & categories — db saves keystrokes; putting your type in a platform category means kubectl get platform lists every custom resource your team ships, one tidy inventory.

Versions & conversion

☺ Like you’re 10: Improve the brick’s design and old bricks still have to fit — a little translator converts between old and new shapes so nothing breaks.

APIs evolve. A CRD can serve multiple versions (say v1alpha1 and v1) at once, but exactly one is storage: true — the shape written to etcd. When the shapes differ, set spec.conversion.strategy: Webhook and run a conversion webhook that translates between versions on the fly, so old and new clients both work. That’s how you graduate an API from experimental to stable without a flag day. (If versions are identical, use strategy: None.)

The custom resource: the butterfly Dot writes

☺ Like you’re 10: The CRD was the dictionary entry. A custom resource is an actual sentence in that new word — one real request.

Once registered, a custom resource (CR) is an instance of it — the friendly object a developer authors, the butterfly. Compare the eight lines below to the two hundred lines of StatefulSet-plus-friends they replace:

apiVersion: platform.acme.io/v1alpha1
kind: Database
metadata:
  name: orders-db
  namespace: checkout
spec:
  engine: postgres
  version: "16"
  size: medium
  highAvailability: true

Dot commits that to the config repo, Argo CD or Flux syncs it, and a minute later kubectl get db orders-db shows Phase: Ready with a connection endpoint. Dot never touched a StatefulSet, picked a storage class, or wrote a backup schedule — the platform absorbed it all.

The operator: giving your noun a verb

☺ Like you’re 10: The CRD lets you write down “I want a database.” The operator reads it, builds the database, and rebuilds any piece that breaks — forever.

A CRD alone stores and validates objects; nothing happens. An operator — a specialised controller — supplies the behaviour: the same reconcile loop that powers GitOps, aimed at your custom resource. It packages the know-how a human expert would apply, run in software day and night — “an SRE for one application, encoded as code.”

Recon’s loop, again: observe → diff → act

You met this loop in GitOps; it’s the same shape here — Recon doesn’t care whether desired state came from Git or a custom resource. It watches Database objects (desired), sees what exists (actual), diffs, and closes the gap. Crucially it is level-triggered, not edge-triggered: instead of firing once on “created,” it repeatedly drives actual toward desired. Delete the Service or lose a node, and the next reconcile quietly rebuilds it — self-healing comes free.

Watches, informers & the work queue

☺ Like you’re 10: The robot doesn’t poll “anything new?” all day. It keeps a live notebook of every Database and gets tapped the instant one changes.

An operator doesn’t hammer the API server: it sets up watches fed by informers — a shared in-memory cache from a list-then-watch stream, so reads are cheap. When a Database or child changes, the informer drops a key onto a work queue that is rate-limited and de-duplicating: fifty rapid edits collapse into one reconcile; a failure re-queues with backoff. It also watches what it owns, so deleting the child Service re-enqueues the parent Database. That’s why operators scale to thousands of objects.

Owner references & garbage collection

☺ Like you’re 10: Each part the robot builds gets a tag: “I belong to orders-db.” Throw away orders-db and Kubernetes sweeps up everything wearing that tag.

When the operator creates the StatefulSet, Service, and Secret, it stamps each with an owner reference back to the Database. That wires them into Kubernetes’ built-in garbage collection: delete the Database and the cluster cascades to every owned child — no orphaned volumes, no leaked Services. The operator never hand-deletes; it just sets ownership and lets the platform clean up. Owner refs also tell it which objects are “its” to reconcile.

Finalizers & status conditions

☺ Like you’re 10: A finalizer is a “don’t toss this yet — one last chore” sticky note. Conditions are the report card the robot fills in: Ready? Yes/No, and why.

Owner-ref cleanup handles resources inside the cluster — but a real cloud database, or a final backup? That’s what finalizers are for. A finalizer is a string in metadata.finalizers; while it’s present, a delete doesn’t remove the object — it just sets a deletionTimestamp. The operator notices, runs cleanup (snapshot data, deprovision the external resource), removes its finalizer, and only then does the object disappear. Skip finalizers and you leak external state on kubectl delete.

Finally, the operator reports through status conditions — the standard type/status/reason/message array (Ready=True, Progressing=True, Degraded=False) that tools, dashboards, and kubectl wait --for=condition=Ready all understand. That’s how a platform tells Dot “provisioning… now ready,” machine-readably. Here’s the loop in controller-runtime shape, scaffolded by Kubebuilder:

func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var db platformv1.Database
    if err := r.Get(ctx, req.NamespacedName, &db); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)   // gone — owned children GC’d via ownerRefs
    }

    // Deleting? run finalizer cleanup, then let the object go.
    if !db.DeletionTimestamp.IsZero() {
        if err := r.takeFinalBackupAndDeprovision(ctx, &db); err != nil {
            return ctrl.Result{}, err                       // retry — object stays until we succeed
        }
        controllerutil.RemoveFinalizer(&db, dbFinalizer)
        return ctrl.Result{}, r.Update(ctx, &db)
    }
    // Finalizers live in metadata — adding one only counts once it is persisted.
    if controllerutil.AddFinalizer(&db, dbFinalizer) {
        if err := r.Update(ctx, &db); err != nil {
            return ctrl.Result{}, err
        }
    }

    // observe → diff → act: build desired children and apply them.
    sts := desiredStatefulSet(&db)
    controllerutil.SetControllerReference(&db, sts, r.Scheme)   // ownerRef → cascade delete
    if err := r.Patch(ctx, sts, client.Apply,
        client.FieldOwner("database-controller"), client.ForceOwnership); err != nil {
        return ctrl.Result{}, err                               // server-side apply the child
    }

    // report observed state back through .status conditions.
    meta.SetStatusCondition(&db.Status.Conditions, metav1.Condition{
        Type: "Ready", Status: metav1.ConditionTrue, Reason: "Provisioned",
        Message: "database is serving connections",
    })
    db.Status.Phase = "Ready"
    return ctrl.Result{}, r.Status().Update(ctx, &db)       // level-triggered: re-runs on every change
}

Operator frameworks — what to build one with

☺ Like you’re 10: You don’t carve the robot from scratch. Use kits — some for robot-building experts, some where you write only the clever part.

You rarely hand-write the watch/informer/queue plumbing — you use a framework. The exam won’t quiz you on SDK flags, but you should know the landscape and, more importantly, when to reach for each:

The rough rule: reach for Helm/Ansible to package an existing deployment with light lifecycle, Kubebuilder/Go when the logic is genuinely stateful and complex, and Kopf or Metacontroller for smaller automations where full Go is overkill.

Operator Capability Levels

☺ Like you’re 10: Robots come in five belts, from “can only install it” to “runs everything itself.”

The community Operator Capability Levels give you a maturity ladder for scoping how much operational knowledge to encode:

1. Basic Install — provisions the app and its config.
2. Seamless Upgrades — upgrades versions safely, in order.
3. Full Lifecycle — backups, restores, failover, scaling.
4. Deep Insights — emits metrics, alerts, and workload analysis.
5. Auto Pilot — auto-scales, auto-tunes, auto-remediates without a human.

Serving the API: admission & the request path

☺ Like you’re 10: Before your Database request is saved, it goes through a short security line: who are you, what may you do, tidy the form, is it valid?

Extending the API means understanding how a request is served. When Dot runs kubectl apply -f orders-db.yaml, the object travels a fixed path through the kube-apiserver before it’s stored — where a platform enforces its rules:

Mutating webhooks run first and can change the object — inject a sidecar, set a default the schema can’t express, add a label. Then the OpenAPI schema validates structure. Then validating webhooks give a final yes/no — where policy engines like OPA/Gatekeeper and Kyverno live, enforcing rules like “every Database must set an owner label.” Only then is the object stored and your operator woken. Admission control is part of serving a platform API.

CRDs and operators are the machinery under every self-service promise: the platform team encodes a hard operational job once — an API plus a loop — and hands developers a friendly noun. Next, Mira and Nutty add a storefront — self-service provisioning with Crossplane (just CRDs-and-controllers over cloud infrastructure) and a Backstage portal, so Dot picks a template instead of writing even eight lines. For how the planes fit together, keep the reference architecture and glossary close.