Operators & Custom Resource Definitions

A CustomResourceDefinition lets you register a brand-new noun with the Kubernetes API — a PostgresCluster, a CertificateRequest, a KafkaTopic — and the apiserver will store it, validate it, version it, and serve it through kubectl exactly as if it shipped in Kubernetes core. On its own, though, a CRD is inert: it's a shape for data, not behavior. The operator pattern is what turns that shape into something alive, by pairing it with a controller that runs the same watch-diff-act reconcile loop covered in The Kubernetes API & the Controller Pattern — except now the loop runs against your Kind instead of a built-in one, encoding operational knowledge that used to live in a wiki page and an on-call human. This page goes underneath the pattern you've already seen applied to ReplicaSet and Deployment: how a CRD's OpenAPI schema actually gets enforced, what a minimal controller's reconcile function really does, when writing one is the right call versus installing an operator someone else already wrote and hardened, and a complete, runnable-shaped worked example connecting every piece.

A CRD is API extension, not automation

☺ Like you're 10: Registering the new form gets it accepted and filed correctly — it does not hire anyone to act on it.

A CustomResourceDefinition is itself a Kubernetes object, applied to the cluster like anything else, that tells kube-apiserver about a new API group, version, and Kind. Once it's accepted, the apiserver immediately gains a new REST endpoint — full GET/LIST/WATCH/CREATE/UPDATE/DELETE, storage in etcd, kubectl get/describe/edit support, RBAC scoping by verb and resource name exactly like Pods or Deployments get, and — critically — schema validation against an OpenAPI v3 structural schema you supply. The apiserver will reject a custom object that violates that schema (wrong type, missing required field, a string where an enum was declared) before it's ever written to etcd, the same way it rejects a Pod spec with a typo'd field name. None of that requires a single line of Go code running anywhere; it's pure API-server configuration.

# A CRD registering a new Kind: ScheduledBackup, in group backups.example.com
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: scheduledbackups.backups.example.com   # must be <plural>.<group>
spec:
  group: backups.example.com
  names:
    kind: ScheduledBackup
    plural: scheduledbackups
    singular: scheduledbackup
    shortNames: ["sbk"]
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: ["target", "schedule", "retentionDays"]
              properties:
                target:
                  type: string
                  description: "PVC name to snapshot"
                schedule:
                  type: string
                  pattern: '^(\*|[0-9,/*-]+) (\*|[0-9,/*-]+) (\*|[0-9,/*-]+) (\*|[0-9,/*-]+) (\*|[0-9,/*-]+)$'
                retentionDays:
                  type: integer
                  minimum: 1
                  maximum: 365
            status:
              type: object
              properties:
                lastBackupTime: { type: string, format: date-time }
                conditions:
                  type: array
                  items: { type: object, x-kubernetes-preserve-unknown-fields: true }
      subresources:
        status: {}   # separates spec writes (users) from status writes (the controller)

Two details in that manifest carry the whole design. First, the schema block is what makes a CRD's validation genuinely comparable to a built-in type's — required, minimum/maximum, pattern, and enums are all enforced server-side, so a malformed ScheduledBackup never even reaches etcd. Second, the status subresource exists to separate two writers that must never race: a user or GitOps tool writes spec to declare desired state, while only the controller writes status to report what actually happened — kubectl apply against the main resource can't accidentally clobber a status field, and RBAC can grant "edit the object" without also granting "lie about its own status."

The operator pattern: a reconcile loop over your own Kind

☺ Like you're 10: The operator is the employee whose only job is to keep re-reading the new form and doing whatever it says, forever, without ever being told to check again.

An operator is a CRD plus a controller that watches objects of that Kind and runs the identical watch → diff → act loop that ReplicaSet and Deployment run against Pods — level-triggered, not event-triggered, meaning it never trusts "I remember doing this already" and instead re-derives the right action from current state every time it wakes up. Concretely, a reconciler is a function with one job: given the name of one object, read its current spec and status from the API, read whatever real-world state it's responsible for (a running Pod, a cloud resource, a file on a snapshot volume), compare the two, and take exactly the actions needed to close the gap — then return. It is never handed the object's full history, only its name; everything it needs to decide what to do, it re-reads fresh on every call. That's the property that makes an operator crash-safe: kill the controller mid-reconcile, restart it anywhere, and the very next reconcile for that object starts from the same place a healthy one would have, because nothing depended on in-memory state surviving the crash.

In practice a reconciler doesn't poll the API in a naive loop — it's driven by an informer, a client-side cache that maintains a long-lived watch connection and de-duplicates events into a work queue, which is the same machinery kube-controller-manager's built-in controllers run on and is covered in more depth in Control Plane Internals. What's specific to operators is only the reconcile function's body — what "real-world state" it compares against, and what actions closing the gap requires. For a ReplicaSet that's counting Pods; for the ScheduledBackup example above it might mean creating a CronJob that snapshots a PVC on schedule, then writing lastBackupTime back to status once a snapshot succeeds. Both are the exact same shape of loop, at different points on Kubernetes' extension surface.

Build vs. adopt: when writing your own operator is the right call

☺ Like you're 10: Before inventing a brand-new form and hiring someone to process it, check whether another department already has a form — and an employee — that does almost exactly what you need.

Writing an operator is real, ongoing software engineering: a reconcile loop that's wrong in a subtle way doesn't throw a stack trace, it silently does the wrong thing to real infrastructure, possibly at 3 a.m., possibly on every object in the cluster at once if the bug is in a shared code path. That cost is easy to underestimate the first time, which is exactly why "should we build one" deserves a real answer instead of defaulting to yes because CRDs feel cheap to declare.

When building genuinely is the right call, don't hand-roll the informer, work queue, and leader-election machinery — Kubebuilder and the Operator SDK scaffold a real Go project on top of controller-runtime, the same library kube-controller-manager's own loops are conceptually aligned with, and generate the CRD's OpenAPI schema straight from your Go struct's tags via controller-gen. Platform Engineering's Kubebuilder & the Operator SDKs covers that scaffolding in full, and its Platform APIs, CRDs & Operators takes the pattern all the way to self-service platform APIs — this page stays deliberately narrower, at the mechanism itself. If what you actually need is cloud infrastructure reconciled as Kubernetes objects rather than a wholly custom Kind, Crossplane is very often the "adopt" answer instead of the "build" one: it's an operator for infrastructure, already written.

A worked minimal example: from CRD to running controller

☺ Like you're 10: Here is the whole toy version, start to finish — the new form, the employee who reads it, and what "doing the job" actually looks like in code small enough to fit on one page.

Put the pieces together end to end with a deliberately small example: a ScheduledBackup operator whose entire job is "ensure a CronJob exists that runs a snapshot on the declared schedule, and report the last time it ran." The CRD from earlier already defines the shape; here's the reconciler, written in the idiomatic controller-runtime shape that Kubebuilder scaffolds (trimmed to the parts that matter — imports and boilerplate omitted):

// Reconcile is called with only a NamespacedName — never a payload,
// never "what changed." Everything needed is re-read from the API.
func (r *ScheduledBackupReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var backup backupsv1.ScheduledBackup
    if err := r.Get(ctx, req.NamespacedName, &backup); err != nil {
        // Object gone — Kubernetes' own garbage collection (owner refs)
        // already cleaned up anything we created. Nothing to do.
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Deletion in progress: run cleanup, then release the finalizer.
    if !backup.DeletionTimestamp.IsZero() {
        if controllerutil.ContainsFinalizer(&backup, backupFinalizer) {
            if err := r.cleanupSnapshots(ctx, &backup); err != nil {
                return ctrl.Result{}, err // retry cleanup, don't drop the finalizer yet
            }
            controllerutil.RemoveFinalizer(&backup, backupFinalizer)
            return ctrl.Result{}, r.Update(ctx, &backup)
        }
        return ctrl.Result{}, nil
    }
    if !controllerutil.ContainsFinalizer(&backup, backupFinalizer) {
        controllerutil.AddFinalizer(&backup, backupFinalizer)
        if err := r.Update(ctx, &backup); err != nil {
            return ctrl.Result{}, err
        }
    }

    // Desired state, derived fresh from spec every single pass.
    desired := buildCronJob(&backup)

    var existing batchv1.CronJob
    err := r.Get(ctx, client.ObjectKeyFromObject(desired), &existing)
    switch {
    case apierrors.IsNotFound(err):
        if err := ctrl.SetControllerReference(&backup, desired, r.Scheme); err != nil {
            return ctrl.Result{}, err
        }
        if err := r.Create(ctx, desired); err != nil {
            return ctrl.Result{}, err // requeued automatically on error
        }
    case err == nil:
        if existing.Spec.Schedule != desired.Spec.Schedule {
            existing.Spec.Schedule = desired.Spec.Schedule
            if err := r.Update(ctx, &existing); err != nil {
                return ctrl.Result{}, err
            }
        }
    default:
        return ctrl.Result{}, err
    }

    // Only the controller writes status — never the user.
    backup.Status.LastReconciled = metav1.Now()
    if err := r.Status().Update(ctx, &backup); err != nil {
        return ctrl.Result{}, err
    }
    return ctrl.Result{}, nil
}

Trace what makes this a correct reconciler rather than merely working code. It never assumes anything about why it was called — the same function runs whether this is the object's first-ever reconcile or its ten-thousandth, because everything it needs is re-read from the API at the top. It handles deletion explicitly via the finalizer, so cleanup can't be skipped by a crash between "object deleted" and "cleanup ran." It only writes to the CronJob when the desired and actual schedule actually differ, not unconditionally every pass — an operator that blindly re-applies on every reconcile generates pointless API traffic and, worse, can stomp on fields another controller legitimately owns. And it uses ctrl.SetControllerReference to set an owner reference on the CronJob it creates, which is what lets Kubernetes' own garbage collector delete that CronJob automatically if the ScheduledBackup is ever force-deleted without the finalizer path running — a safety net under the safety net.

Everything on this page is the exact same shape you already met in The Kubernetes API & the Controller Pattern applied one level further out — a CRD is a Kind, a reconciler is a controller, and the loop doesn't know or care that neither one shipped with Kubernetes core. Where those controllers wire together into always-converging pipelines is GitOps on Kubernetes; where an operator manages genuinely stateful workloads like databases specifically, Stateful Workloads & Database Operators picks the pattern back up. For the security side of running someone else's controller with cluster-wide write access, DevSecOps' Kubernetes security deep dive and this course's own RBAC & Admission Control cover exactly how much you should trust an operator's ServiceAccount before you grant it. And for the reliability angle on a controller that's now a single point of failure for its whole Kind, SRE's Kubernetes reliability patterns is the natural next stop.