Kubebuilder & the Operator SDKs

Kubebuilder is the scaffolding toolkit and opinionated framework for building your own Kubernetes APIs — it generates a Go project, a CustomResourceDefinition, and a working controller skeleton, and hands you the controller-runtime library that does the hard concurrency, caching and retry work for you. It solves the platform problem of “our operational knowledge lives in a wiki page and a human,” by letting you encode that knowledge once as a reconcile loop behind a friendly new noun that developers can simply kubectl apply.

What Kubebuilder is and the problem it solves

☺ Like you’re 10: It’s a starter kit for teaching Kubernetes a brand-new word, plus a robot that gives the word a meaning.

Kubebuilder is a kubernetes-sigs project — the same organisation that builds Kubernetes itself — and it is the reference way to build an operator: a custom resource definition plus a controller that acts on it. You have already met the concept on Platform APIs & Operators; this page is about the tool that makes actually shipping one a day’s work instead of a quarter’s.

The problem before frameworks

Writing a Kubernetes controller from raw client-go is genuinely difficult. You must build informers and shared caches so you are not hammering the API server, wire a work queue with rate limiting and exponential backoff, write DeepCopy functions for every type by hand, register your types into a runtime Scheme, hand-author hundreds of lines of OpenAPI schema in your CRD YAML, generate RBAC that matches what your code actually does, set up leader election so two replicas do not fight, and expose health and metrics endpoints. None of that is your business logic. Every operator author was rebuilding the same scaffolding, slightly wrong, in a slightly different way.

What Kubebuilder actually gives you

Kubebuilder is really three things bundled. The CLI (kubebuilder init, kubebuilder create api) scaffolds a Go module, a Makefile, a cmd/main.go, a config/ tree of Kustomize bases, and a Dockerfile. The controller-runtime library supplies the Manager, the client, the cached reader, the work queue and the reconciler interface. And controller-gen reads special // +kubebuilder: comment markers in your Go source and generates the CRD YAML, the RBAC rules, and the DeepCopy code — so your Go types are the single source of truth and the YAML is a build artefact, never something you edit.

What it is not

Kubebuilder is not a runtime — nothing called “Kubebuilder” runs in your cluster. What runs is a plain Go binary you built, deployed as a Deployment, that links controller-runtime. It is not a packaging or distribution system either: it will not publish your operator to a catalogue or manage upgrades for users (that is OLM, from the Operator Framework). And it is not a way to avoid learning Go — everything below assumes you are writing Go. If that is a blocker, the alternatives section has Python, YAML and shell answers.

Where it fits in a platform

☺ Like you’re 10: It lives in the part of the platform that invents the buttons, not the part that presses them.

Kubebuilder sits squarely in the control plane / platform API layer of the reference architecture. It is the tool platform engineers use to manufacture the abstractions everything else consumes. A developer never installs Kubebuilder; they only ever see its output — a new kind they can write eight lines of YAML against.

Its neighbours

Crossplane is the “don’t write Go” answer to the same problem: it is itself a set of Kubebuilder-style controllers, and its Compositions let you assemble new APIs declaratively. Reach for Crossplane when you are composing existing resources; reach for Kubebuilder when your logic is genuinely imperative — talking to a legacy system, sequencing a migration, reacting to an external event. Cluster API is one of the largest Kubebuilder projects in existence and a superb code-reading exercise. Argo CD and Flux deliver your operator and your custom resources like any other manifest. Backstage puts a form in front of the noun you invented. And Kyverno or Gatekeeper guard the same API surface with policy.

Why platform teams reach for it

Because a reconcile loop is the only durable place to put operational knowledge. A runbook rots, a Helm chart cannot recover from drift, and a CI job only runs when someone triggers it. A controller runs forever, is level-triggered, and turns “the four things you must remember to do when a tenant onboards” into code with tests. That is the mechanism behind almost every self-service capability worth having.

CNPE domain relevance

Kubebuilder itself is not on the CNPE exam tool list — but CRDs and operators absolutely are an exam competency, under platform APIs and extending Kubernetes. You will not be asked to write Go. You will be expected to read a CRD, understand what a controller does, explain reconciliation and owner references, and reason about why a custom resource is stuck. Treat this page as the deep background that makes those questions obvious, and Platform APIs & Operators as the exam-shaped lesson.

How it works — architecture and the reconcile contract

☺ Like you’re 10: One boss (the Manager) holds a shared notebook of everything happening, and hands little jobs to your robot one at a time.

The Manager

Everything hangs off a Manager, created in cmd/main.go. It owns the shared cache (one set of informers, however many controllers you run), the client (which reads from the cache and writes straight to the API server), the Scheme (the registry mapping Go types to Group/Version/Kind), leader election, the metrics endpoint, and the /healthz and /readyz probes. You register controllers on it and call mgr.Start(ctx), which blocks until the context is cancelled.

The Reconcile contract

Your entire controller is one method: Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error). The req contains only a namespace and a name — deliberately. It does not tell you what changed or whether it was a create, update or delete. It says: “something about this object may have moved; go look.” You then fetch the object, work out the whole desired world, make it so, and return. Three return shapes matter: ctrl.Result{}, nil means done, forget it; ctrl.Result{}, err means failed, requeue with exponential backoff; ctrl.Result{RequeueAfter: d}, nil means succeeded but check again in d — the right way to poll an external system.

Watches, Owns and predicates

SetupWithManager declares what puts keys on the queue. For(&Database{}) is the primary type. Owns(&appsv1.StatefulSet{}) means “when a StatefulSet I own changes, enqueue its owner” — this is what makes a child crash-looping wake the parent up, and it works by reading the child’s controller owner reference. Watches(...) plus a map function handles arbitrary relationships (“when this Secret changes, reconcile every Database referencing it”). Predicates filter events before they reach the queue; the common one is predicate.GenerationChangedPredicate{}, which drops updates that only touched status or annotations — because metadata.generation only increments when spec changes. Without it, a controller that writes its own status can spin in a tight self-triggered loop.

Markers and controller-gen

Markers are comments that begin // +kubebuilder: (or // + for core controller-tools markers). controller-gen parses them and emits YAML and Go. These are the ones worth recognising on sight:

The resources you will actually write

☺ Like you’re 10: You write the menu card (a Go struct), the recipe (one function), and the machine prints the rest.

The API type — where the CRD comes from

This is api/v1alpha1/database_types.go after kubebuilder create api, filled in. Note that Spec is the user’s contract and Status is written only by the controller — never mix the two.

package v1alpha1

import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

// DatabaseSpec is the developer-facing contract: the only part a human writes.
type DatabaseSpec struct {
	// +kubebuilder:validation:Enum=postgres;mysql
	// +kubebuilder:default=postgres
	Engine string `json:"engine"`

	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=1024
	StorageGB int32 `json:"storageGB"`

	// +optional
	// +kubebuilder:validation:Pattern=`^[a-z0-9-]+$`
	Team string `json:"team,omitempty"`
}

// DatabaseStatus is written ONLY by the controller.
type DatabaseStatus struct {
	// ObservedGeneration is the .metadata.generation this status reflects.
	// +optional
	ObservedGeneration int64 `json:"observedGeneration,omitempty"`

	// +optional
	// +listType=map
	// +listMapKey=type
	Conditions []metav1.Condition `json:"conditions,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=db,categories=platform
// +kubebuilder:printcolumn:name="Engine",type=string,JSONPath=`.spec.engine`
// +kubebuilder:printcolumn:name="Ready",type=string,JSONPath=`.status.conditions[?(@.type=="Ready")].status`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
type Database struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   DatabaseSpec   `json:"spec,omitempty"`
	Status DatabaseStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true
type DatabaseList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []Database `json:"items"`
}

func init() { SchemeBuilder.Register(&Database{}, &DatabaseList{}) }

The reconciler — finalizers, ownership, conditions

Every important pattern in operator writing appears in this one function. Read the comments; they are the lesson.

// +kubebuilder:rbac:groups=platform.acme.io,resources=databases,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=platform.acme.io,resources=databases/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=platform.acme.io,resources=databases/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=statefulsets,verbs=get;list;watch;create;update;patch;delete

const finalizerName = "platform.acme.io/cleanup"

func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	var db platformv1alpha1.Database
	if err := r.Get(ctx, req.NamespacedName, &db); err != nil {
		// NotFound means it is already gone. Never requeue: there is nothing to do.
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	// --- deletion path: the object is not gone until we drop the finalizer ---
	if !db.DeletionTimestamp.IsZero() {
		if controllerutil.ContainsFinalizer(&db, finalizerName) {
			if err := r.deleteCloudDatabase(ctx, &db); err != nil {
				return ctrl.Result{}, err // retried with backoff; must be idempotent
			}
			controllerutil.RemoveFinalizer(&db, finalizerName)
			return ctrl.Result{}, r.Update(ctx, &db)
		}
		return ctrl.Result{}, nil
	}
	if controllerutil.AddFinalizer(&db, finalizerName) {
		if err := r.Update(ctx, &db); err != nil {
			return ctrl.Result{}, err
		}
	}

	// --- desired state: rebuild it from scratch every pass, never diff by hand ---
	sts := r.desiredStatefulSet(&db)
	// Owner reference: when the Database is deleted, GC deletes this child for free.
	if err := ctrl.SetControllerReference(&db, sts, r.Scheme); err != nil {
		return ctrl.Result{}, err
	}
	// Server-side apply: sts must carry TypeMeta (APIVersion + Kind) or the
	// API server rejects the patch — the #1 surprise when moving off Update.
	if err := r.Patch(ctx, sts, client.Apply,
		client.ForceOwnership, client.FieldOwner("database-controller")); err != nil {
		return ctrl.Result{}, err
	}

	// --- status: conditions + the observedGeneration pattern ---
	// Shown True for brevity; real code derives it from sts.Status.ReadyReplicas.
	meta.SetStatusCondition(&db.Status.Conditions, metav1.Condition{
		Type:               "Ready",
		Status:             metav1.ConditionTrue,
		Reason:             "StatefulSetAvailable", // must be a CamelCase machine token
		Message:            "database is accepting connections",
		ObservedGeneration: db.Generation,
	})
	db.Status.ObservedGeneration = db.Generation
	if err := r.Status().Update(ctx, &db); err != nil {
		return ctrl.Result{}, err // a Conflict here is normal — requeue and re-read
	}

	// Succeeded, but re-check the external system in five minutes.
	return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
}

func (r *DatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		// Ignore our own status writes: only spec changes bump .metadata.generation.
		For(&platformv1alpha1.Database{},
			builder.WithPredicates(predicate.GenerationChangedPredicate{})).
		Owns(&appsv1.StatefulSet{}). // child changes enqueue the owner
		Watches(&corev1.Secret{},
			handler.EnqueueRequestsFromMapFunc(r.secretToDatabases)).
		WithOptions(controller.Options{MaxConcurrentReconciles: 4}).
		Complete(r)
}

What controller-gen emits, and what a developer writes

make manifests turns the markers above into config/crd/bases/platform.acme.io_databases.yaml. You never edit this file — but you should be able to read it, because the exam will show you one:

# GENERATED by controller-gen from database_types.go — do not edit
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.platform.acme.io          # must be <plural>.<group>
spec:
  group: platform.acme.io
  names:
    kind: Database
    listKind: DatabaseList
    plural: databases
    singular: database
    shortNames: [db]
    categories: [platform]
  scope: Namespaced
  versions:
    - name: v1alpha1
      served: true                           # reachable over the API
      storage: true                          # exactly ONE version may be true
      subresources:
        status: {}                           # from +kubebuilder:subresource:status
      additionalPrinterColumns:
        - name: Engine
          type: string
          jsonPath: .spec.engine
        - name: Ready
          type: string
          jsonPath: .status.conditions[?(@.type=="Ready")].status
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [engine, storageGB]
              properties:
                engine:  { type: string, enum: [postgres, mysql], default: postgres }
                storageGB: { type: integer, format: int32, minimum: 1, maximum: 1024 }
                # quoted — a bare ^[a-z0-9-]+$ is not a legal scalar inside { }
                team:    { type: string, pattern: "^[a-z0-9-]+$" }
            # status MUST be in the schema too: with pruning on, anything the
            # schema does not describe is silently dropped on write.
            status:
              type: object
              properties:
                observedGeneration: { type: integer, format: int64 }
                conditions:
                  type: array
                  x-kubernetes-list-type: map
                  x-kubernetes-list-map-keys: [type]
                  items:
                    type: object
                    required: [type, status, reason, lastTransitionTime]
                    properties:
                      type:               { type: string }
                      # quoted — otherwise YAML turns these into booleans
                      status:             { type: string, enum: ["True", "False", "Unknown"] }
                      reason:             { type: string }
                      message:            { type: string }
                      observedGeneration: { type: integer, format: int64 }
                      lastTransitionTime: { type: string, format: date-time }
---
# ...and this is all a developer ever has to write:
apiVersion: platform.acme.io/v1alpha1
kind: Database
metadata:
  name: checkout-db
  namespace: checkout
spec:
  engine: postgres
  storageGB: 50
  team: payments

Day-to-day commands

☺ Like you’re 10: Two commands to make the project, then a handful of make shortcuts you type all day.

Scaffolding a new operator

# 1. Create the project skeleton in an EMPTY directory.
mkdir database-operator && cd database-operator
kubebuilder init \
  --domain acme.io \
  --repo github.com/acme/database-operator \
  --owner "Acme Platform Team"

# 2. Add an API + a controller for it. Answers both prompts with --resource/--controller.
kubebuilder create api \
  --group platform --version v1alpha1 --kind Database \
  --resource --controller

# 3. (Optional) add a defaulting/validating admission webhook for the same kind.
kubebuilder create webhook \
  --group platform --version v1alpha1 --kind Database \
  --defaulting --programmatic-validation

kubebuilder version          # CLI version, build date, Go version

# Optional plugins are applied to an existing project by their FULL key
# (name.domain/version), e.g. the Grafana dashboards plugin:
kubebuilder edit --plugins=grafana.kubebuilder.io/v1-alpha

The Makefile targets you will type a hundred times

make generate     # controller-gen object: regenerate zz_generated.deepcopy.go
make manifests    # controller-gen crd,rbac,webhook: regenerate config/**/*.yaml
                  # RUN BOTH after ANY change to types or markers.

make fmt vet      # gofmt + go vet
make test         # unit + envtest suites (downloads control-plane binaries)
make build        # compile bin/manager

make install      # kubectl apply the CRDs only
make run          # run the controller LOCALLY against your kubeconfig — fastest loop
                  # note: webhooks and in-cluster RBAC are NOT exercised by `make run`

make docker-build docker-push IMG=ghcr.io/acme/database-operator:v0.1.0
make deploy       IMG=ghcr.io/acme/database-operator:v0.1.0   # CRDs + RBAC + Deployment
make undeploy     # tear it all down again
make uninstall    # remove the CRDs (this DELETES every custom resource — see gotchas)

# Then, as a user of your new API:
kubectl explain database.spec.engine     # the OpenAPI schema, served by the API server
kubectl get databases -o wide
kubectl describe database checkout-db    # conditions and events live here
kubectl logs -n database-operator-system deploy/database-operator-controller-manager -f

Testing with envtest

Kubebuilder scaffolds an envtest suite, and it is the framework’s best-kept secret. setup-envtest downloads real kube-apiserver and etcd binaries and runs them locally as processes — so your tests exercise genuine API-server validation, defaulting, admission and status subresources, with no cluster and no Docker. The crucial caveat: there is no kubelet and no scheduler. Pods you create will never run, never become Ready, and never get an IP. Assert on the objects you created and on your own status conditions, never on a workload actually starting; for that you need a real kind cluster and an end-to-end test.

Gotchas and failure modes

☺ Like you’re 10: Almost every operator bug is one of five mistakes, and four of them come from forgetting the robot might be woken up at any moment.

Assuming a single, ordered reconcile

Reconcile is not a workflow step. It can fire twice in the same millisecond (two events, two workers if MaxConcurrentReconciles > 1), or be interrupted halfway by a pod eviction. Code like “create the Secret, then create the Deployment, then mark done” breaks the moment the process dies between steps. Write each pass as: read everything, compute everything, apply everything, and let repeated application be harmless. Server-side apply (client.Apply with a stable FieldOwner) makes this dramatically easier than get-then-update.

Conflict errors are normal — requeue, do not panic

Kubernetes uses optimistic concurrency: your write carries a resourceVersion, and if anything changed since you read it, you get a 409 Conflict. Treat this as routine. Return the error so the object is requeued with backoff and re-read on the next pass — never retry in a tight in-function loop, and never blank the resourceVersion to force the write through, which silently clobbers someone else’s change. If conflicts dominate your logs, you are probably writing status on every pass even when nothing changed; meta.SetStatusCondition plus a “did anything actually change?” check fixes it.

RBAC markers that do not match the code

The generated ClusterRole comes only from your +kubebuilder:rbac markers. Add a call to create a ConfigMap without adding the marker and it works perfectly under make run (you are using your own admin kubeconfig) and fails in-cluster with is forbidden: User "system:serviceaccount:..." cannot create resource "configmaps". The fix is always: add the marker, make manifests, redeploy. Grant the narrowest verbs that work — an operator with cluster-wide * on * is a privilege-escalation path, as security & policy explains.

Ownership, finalizers and deletion traps

Four sharp edges cluster here. Owner references cannot cross namespaces: if a namespaced dependent names a namespaced owner in a different namespace, the reference is treated as absent and the garbage collector deletes the dependent. Conversely, a cluster-scoped object may only be owned by another cluster-scoped object — point it at a namespaced owner and the reference is unresolvable, so that child is never collected at all. (ctrl.SetControllerReference returns an error rather than let you build the cross-namespace case.) A stuck finalizer means a stuck delete: if your controller is down or its cleanup keeps erroring, the object sits in Terminating forever, and so does its namespace; diagnose with kubectl get -o yaml and look at metadata.finalizers. Deleting a CRD deletes every custom resource of that kind, cascading through owner references to their children — so make uninstall on a shared cluster is a genuinely destructive act. And a blanket WithEventFilter(predicate.GenerationChangedPredicate{}) applies to all watches, including Owns(), which silences the child status updates you actually wanted; scope predicates per-source with builder.WithPredicates instead.

Alternatives and when to choose it

☺ Like you’re 10: There are several kits. Some want Go, one wants Python, one wants only YAML, and one wants a shell script.

“Operator SDK” deserves a clarification first, because the naming confuses everyone: the Operator SDK’s Go plugin is built on Kubebuilder. They are not rivals so much as the same engine in two chassis. Operator SDK adds Ansible-based and Helm-based operators (no Go at all), integration with OLM (the Operator Lifecycle Manager, for catalogues, dependency resolution and upgrades), make bundle packaging, and scorecard testing. It also popularised the Capability Levels — a five-rung maturity ladder from 1: Basic Install, through 2: Seamless Upgrades, 3: Full Lifecycle (backup, restore, failover), 4: Deep Insights (metrics, alerts, logs) to 5: Auto Pilot (auto-scaling, auto-tuning, auto-remediation) — a genuinely useful vocabulary for arguing about how finished an operator is.

The comparison that decides it

A practical rule

Ask one question: is my new API just a bundle of resources that already exist? If yes, use Crossplane Compositions (or a chart) and write no code — see anti-patterns on inventing controllers you do not need. If no — if you must call an external system, sequence a migration, react to a webhook, or make a judgement — then you need a real reconcile loop, and Kubebuilder is the default. Choose Operator SDK over bare Kubebuilder only when you actually need OLM packaging or the non-Go languages.

Exam relevance and going further

☺ Like you’re 10: Nobody will ask you to write Go — but they will ask you what the robot does and why the thing is stuck.

What to be able to do without notes

Explain what an operator is in one sentence (a CRD plus a controller that reconciles it). Read a CustomResourceDefinition and name group, names.plural, scope, versions[].served and storage (exactly one storage version), subresources.status, and where the validation lives. State the reconcile contract and why it is level-triggered. Explain ownerReferences and cascading garbage collection, what a finalizer is for and how it wedges a delete, and what status.conditions plus observedGeneration tell you. Know that the CRD is the noun and the controller is the verb — a CRD alone does nothing at all. Drill the CRD and custom-resource manifests on Know Cold, and keep the command reference for kubectl explain, get crd and api-resources.

The documentation allowlist — read this twice

During the CNPE exam the only permitted documentation is kubernetes.io/docs, kubernetes.io/blog, task-specific documentation linked from the exam’s Quick Reference box, and local man pages and /usr/share docs. The Kubebuilder Book, the controller-runtime godoc and the Operator SDK site are not available to you. What is available is the Kubernetes documentation on Custom Resources and CRDs — and that is genuinely enough, because the exam tests the Kubernetes-side concepts, not the Go framework. Practise finding “Extend the Kubernetes API with CustomResourceDefinitions” on kubernetes.io quickly, and remember kubectl explain works offline against any CRD installed in the cluster.

⚖ CNPA vs CNPE — That allowlist is a CNPE-specific mechanic — CNPE is hands-on, so it grants a narrow set of live lookups. CNPA has none of it: a fully closed-book, multiple-choice exam with zero external resources and zero lookups of any kind, which makes it stricter than CNPE, not looser. Even so, the concept layer above — what a CRD and controller are, why reconcile is level-triggered, what a finalizer does — is exactly the kind of concept-level knowledge CNPA's closed-book recall draws on.

Official resources for after the exam

The canonical sources are the Kubebuilder Book (work the CronJob tutorial end to end — it is the best few hours you can spend on this topic), the controller-runtime repository and its godoc, sdk.operatorframework.io for Operator SDK and the Capability Levels, Kopf, Metacontroller, and the Kubernetes Custom Resources concept page. Pair this page with Platform APIs & Operators for the exam-shaped lesson, Crossplane for the no-code alternative, the tool landscape for where it all sits, and the glossary whenever a term stops making sense.