The Capstone · Part 3 of 6 · pairs with Platform APIs, CRDs & Operators (D3)

Capstone Part 3 — a new noun for ledger

Every part so far has deployed ledger using nouns Kubernetes already knows — Deployment, Service, Rollout. This part teaches the platform a noun it has never heard: kind: LedgerDatabase. You will scaffold a real Kubebuilder operator, write its reconciler by hand, deploy it into the cluster you already have, wire it into the same GitOps repo Benny built in Part 1, and then break things on purpose to prove the controller reconciles and self-heals. This is the hands-on twin of Platform APIs, CRDs & Operators — that lesson explains why a CRD-plus-controller beats two hundred lines of copy-pasted YAML; this part makes you build one.

⚖ CNPA vs CNPE — Everything in this part — scaffolding a Kubebuilder project, hand-writing a Go reconciler, proving self-heal on a real cluster — is CNPE-only; CNPA is closed-book multiple-choice with no lab component, so nobody sits down and builds an operator for it. But the concepts underneath (what a CRD is, what a controller/operator does, why owner references and finalizers exist) still matter for CNPA's closed-book recall, just tested as recognition rather than something you build.

☺ Explain it like I’m 10

So far, ledger has only used bricks Kubernetes already sells in the store — Pod bricks, Service bricks. Today you invent a brand-new brick and teach the store to sell it: a LedgerDatabase brick. You also build the little robot that watches for that brick and actually builds the real database behind it — and if someone snaps a piece off, the robot notices and glues it back on. By the end, asking for a database is eight lines of YAML, not two hundred.

🦋🤖Your hosts for this part: Mira the Butterfly & Recon the Robot — Mira designs the new noun (the CRD) developers get to write, and Recon is the reconcile loop you are about to hand-build, waking up forever to make that noun come true and stay true.

Arriving from Part 2, leaving for Part 4

☺ Like you’re 10: Here’s exactly what’s already built when you sit down, and exactly what you’ll hand off when you’re done.

If you’ve worked the capstone in order, your platform-dev kind cluster already looks like this. From Part 1 (Foundation): Argo CD is running in the platform namespace, reconciling an apps/ App-of-Apps from your platform-capstone Git repo, with prune and selfHeal on. From Part 2 (Delivery): a Kubernetes-native pipeline builds ledger:TAG images, and an Argo Rollouts canary in the ledger namespace ships them 10% → 50% → 100%, auto-aborting bad versions. ledger is up, healthy, and serving traffic — but it has no real datastore of its own; whatever it’s been using so far is a placeholder.

# confirm you're picking up where Part 2 left off
kubectl -n platform get applications
kubectl -n ledger get rollout ledger
kubectl -n ledger get pods -l app=ledger

By the end of this part, the platform will understand a new word: kind: LedgerDatabase. A ledger-db-operator Deployment will be running in the platform namespace, reconciling a LedgerDatabase named ledger-db in the ledger namespace into a real (if tiny) Postgres StatefulSet, headless Service, and credentials Secret — all owned, all self-healing, all committed to the same Git repo Argo CD already watches. Part 4 (Self-Service) picks this up directly: Mira and Nutty put a Backstage form in front of this exact CRD, so scaffolding a brand-new service’s database becomes one click instead of eight lines of YAML.

◆ Key idea

Nothing here talks to a real cloud provider — that’s deliberately out of scope for this part (Crossplane’s job, covered on the Crossplane page). Your operator provisions an in-cluster Postgres via a StatefulSet, which is exactly the “toy but real” scope this capstone needs: every mechanism — CRD, reconcile loop, owner refs, finalizers, self-heal — is identical to what a cloud-backed operator does, just aimed at a cheaper child resource.

Design the noun: the `LedgerDatabase` CRD

☺ Like you’re 10: Before you build the robot, you write the dictionary entry — what counts as a valid “LedgerDatabase,” and what a developer is and isn’t allowed to say.

Following the pattern from Platform APIs & Operators, the API group is platform.acme.io — the same fictional group used across this course — and the kind is LedgerDatabase. Kubebuilder inverts the usual order: you write a Go struct decorated with markers, and controller-gen emits the CRD YAML. Here is the developer-facing contract you’re designing:

Field	Type	Meaning
`spec.engine`	string, enum	Only `postgres` for this capstone — the schema still declares it, so growing to a second engine later is additive.
`spec.storageGB`	int, 1–100	How large a volume to claim. Defaults to `5` if Dot omits it.
`spec.highAvailability`	bool	`true` asks for 3 replicas instead of 1 — the operator decides what that means, Dot just flips a switch.
`spec.backupSchedule`	string, optional	A cron expression — wired up in this part’s schema, left for a future part’s operator logic to actually act on.
`status.phase`, `status.endpoint`, `status.conditions`	written by the operator	What Dot reads back — never written by hand.

The Go type — where the CRD comes from

This is api/v1alpha1/ledgerdatabase_types.go, filled in after kubebuilder create api. Spec is the part a human writes; Status is written only by the controller — mixing the two is the single most common CRD-design mistake.

package v1alpha1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// LedgerDatabaseSpec is the developer-facing contract — the only part a human writes.
type LedgerDatabaseSpec struct {
	// +kubebuilder:validation:Enum=postgres
	// +kubebuilder:default=postgres
	Engine string `json:"engine,omitempty"`

	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=100
	// +kubebuilder:default=5
	StorageGB int32 `json:"storageGB,omitempty"`

	// +kubebuilder:default=false
	HighAvailability bool `json:"highAvailability,omitempty"`

	// +optional
	// +kubebuilder:validation:Pattern=`^[0-9*/,-]+ [0-9*/,-]+ [0-9*/,-]+ [0-9*/,-]+ [0-9*/,-]+$`
	BackupSchedule string `json:"backupSchedule,omitempty"`
}

// LedgerDatabaseStatus is written ONLY by the controller. Never hand-edit these fields.
type LedgerDatabaseStatus struct {
	Phase              string `json:"phase,omitempty"`
	Endpoint           string `json:"endpoint,omitempty"`
	ObservedGeneration int64  `json:"observedGeneration,omitempty"`

	// +optional
	// +listType=map
	// +listMapKey=type
	Conditions []metav1.Condition `json:"conditions,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=ldb,categories=platform
// +kubebuilder:printcolumn:name="Engine",type=string,JSONPath=`.spec.engine`
// +kubebuilder:printcolumn:name="Storage",type=string,JSONPath=`.spec.storageGB`
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
// +kubebuilder:printcolumn:name="Ready",type=string,JSONPath=`.status.conditions[?(@.type=="Ready")].status`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
type LedgerDatabase struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   LedgerDatabaseSpec   `json:"spec,omitempty"`
	Status LedgerDatabaseStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true
type LedgerDatabaseList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []LedgerDatabase `json:"items"`
}

func init() {
	SchemeBuilder.Register(&LedgerDatabase{}, &LedgerDatabaseList{})
}

☺ Like you’re 10: That block of comments above each field isn’t decoration — it’s the actual dictionary entry the store reads to reject bad orders before your robot ever sees them.

⚠ A CRD is a promise you keep forever

Once LedgerDatabase ships, developers write manifests against it and it becomes a public API you support. Notice how thin the schema is: no field for “which Postgres extensions,” no field for “custom postgresql.conf.” That’s deliberate — per the anti-pattern Mira warns about, over-abstraction recreates the cognitive load you set out to remove. Ship the thin API; widen it later, additively, when a real need shows up.

Scaffold the operator with Kubebuilder

☺ Like you’re 10: You don’t carve the robot’s body from scratch — a kit builds the arms, legs and wiring, and you write only the recipe.

Kubebuilder generates a Go module, a Makefile, a config/ tree of Kustomize bases, and a Dockerfile, plus the controller-runtime library that supplies the manager, client, cache and work queue. Run this next to (not inside) your platform-capstone Git repo — it’s its own module:

mkdir ledger-db-operator && cd ledger-db-operator
kubebuilder init --domain acme.io --repo github.com/acme/ledger-db-operator
kubebuilder create api --group platform --version v1alpha1 --kind LedgerDatabase --resource --controller

That leaves you with a project shaped like this — fill in api/v1alpha1/ledgerdatabase_types.go with the struct above, and internal/controller/ledgerdatabase_controller.go with the reconciler below:

ledger-db-operator/
├── PROJECT                                   # kubebuilder's own bookkeeping file
├── go.mod
├── Makefile                                  # make manifests / install / run / docker-build
├── cmd/
│   └── main.go                               # wires up the Manager and registers the controller
├── api/
│   └── v1alpha1/
│       ├── ledgerdatabase_types.go           # the Go struct — the single source of truth
│       └── groupversion_info.go              # generated: Scheme + GroupVersion
├── internal/
│   └── controller/
│       └── ledgerdatabase_controller.go      # the Reconcile function you write by hand
└── config/
    ├── crd/bases/platform.acme.io_ledgerdatabases.yaml   # generated by `make manifests`
    ├── rbac/role.yaml                                     # generated from +kubebuilder:rbac markers
    ├── manager/manager.yaml                                # the operator's own Deployment
    └── samples/platform_v1alpha1_ledgerdatabase.yaml       # a starter LedgerDatabase to edit

Run make manifests now — it reads every +kubebuilder: marker in your Go source and regenerates the CRD YAML, the RBAC role, and the DeepCopy code. Never hand-patch the generated CRD; if the YAML and the struct disagree, the struct wins and the YAML is stale.

make manifests
cat config/crd/bases/platform.acme.io_ledgerdatabases.yaml

The generated CRD is exactly the shape you’d hand-write from the Platform APIs lesson — here it is, trimmed to what matters for this capstone:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: ledgerdatabases.platform.acme.io          # MUST be <plural>.<group>
spec:
  group: platform.acme.io
  scope: Namespaced
  names:
    kind: LedgerDatabase
    plural: ledgerdatabases
    singular: ledgerdatabase
    shortNames: [ldb]
    categories: [platform]
  versions:
    - name: v1alpha1
      served: true
      storage: true
      subresources:
        status: {}
      additionalPrinterColumns:
        - { name: Engine,  type: string, jsonPath: .spec.engine }
        - { name: Storage, type: string, jsonPath: .spec.storageGB }
        - { name: Phase,   type: string, jsonPath: .status.phase }
        - { name: Ready,   type: string, jsonPath: '.status.conditions[?(@.type=="Ready")].status' }
        - { name: Age,     type: date,   jsonPath: .metadata.creationTimestamp }
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                engine:            { type: string, enum: [postgres], default: postgres }
                storageGB:         { type: integer, minimum: 1, maximum: 100, default: 5 }
                highAvailability:  { type: boolean, default: false }
                backupSchedule:    { type: string }
            status:
              type: object
              properties:
                phase:              { type: string }
                endpoint:           { type: string }
                observedGeneration: { type: integer, format: int64 }
                conditions:
                  type: array
                  x-kubernetes-list-type: map
                  x-kubernetes-list-map-keys: [type]
                  items:
                    type: object
                    required: [type, status, lastTransitionTime, reason, message]
                    properties:
                      type:               { type: string }
                      status:             { type: string, enum: ["True", "False", "Unknown"] }
                      reason:             { type: string }
                      message:            { type: string }
                      lastTransitionTime: { type: string, format: date-time }

Write the reconciler

☺ Like you’re 10: This one function is the robot. It looks, compares, and fixes — over and over, forever.

The reconciler is level-triggered, exactly like Recon’s loop in GitOps: it never asks “what changed?” — it asks “what should be true right now?” and drives the cluster there. Three things must happen in the right order: add a finalizer before creating anything external, stamp every child with an owner reference so garbage collection is automatic, and report through status conditions so kubectl get ldb tells the truth at a glance.

package controller

import (
	"context"
	"fmt"
	"time"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	"k8s.io/apimachinery/pkg/api/meta"
	"k8s.io/apimachinery/pkg/api/resource"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/types"
	"k8s.io/apimachinery/pkg/util/rand"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"

	platformv1alpha1 "github.com/acme/ledger-db-operator/api/v1alpha1"
)

const ledgerDBFinalizer = "platform.acme.io/ledgerdatabase-cleanup"

// +kubebuilder:rbac:groups=platform.acme.io,resources=ledgerdatabases,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=platform.acme.io,resources=ledgerdatabases/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=platform.acme.io,resources=ledgerdatabases/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=statefulsets,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=services;secrets,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=events,verbs=create;patch

type LedgerDatabaseReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

func (r *LedgerDatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	log := ctrl.LoggerFrom(ctx)

	var db platformv1alpha1.LedgerDatabase
	if err := r.Get(ctx, req.NamespacedName, &db); err != nil {
		// gone — any owned children are already garbage-collected via ownerRefs.
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	// Deleting? Run the finalizer's cleanup exactly once, then let the object go.
	if !db.DeletionTimestamp.IsZero() {
		if controllerutil.ContainsFinalizer(&db, ledgerDBFinalizer) {
			log.Info("taking final backup before deprovisioning", "ledgerdatabase", db.Name)
			if err := r.takeFinalBackup(ctx, &db); err != nil {
				return ctrl.Result{}, err // retry — the object stays until this succeeds
			}
			controllerutil.RemoveFinalizer(&db, ledgerDBFinalizer)
			if err := r.Update(ctx, &db); err != nil {
				return ctrl.Result{}, err
			}
		}
		return ctrl.Result{}, nil
	}

	// Not deleting: make sure our finalizer is present before we create anything external.
	if controllerutil.AddFinalizer(&db, ledgerDBFinalizer) {
		if err := r.Update(ctx, &db); err != nil {
			return ctrl.Result{}, err
		}
	}

	// Secret: created once, never overwritten — we don't want to rotate a password
	// out from under a running Postgres just because the reconciler ran again.
	secret := &corev1.Secret{}
	secretKey := types.NamespacedName{Name: db.Name + "-credentials", Namespace: db.Namespace}
	if err := r.Get(ctx, secretKey, secret); apierrors.IsNotFound(err) {
		secret = desiredSecret(&db)
		if err := controllerutil.SetControllerReference(&db, secret, r.Scheme); err != nil {
			return ctrl.Result{}, err
		}
		if err := r.Create(ctx, secret); err != nil {
			return ctrl.Result{}, err
		}
		log.Info("issued new credentials", "secret", secret.Name)
	} else if err != nil {
		return ctrl.Result{}, err
	}

	// StatefulSet and Service: server-side apply on every reconcile, so both
	// hand-edited drift and a deleted child heal on the very next pass.
	sts := desiredStatefulSet(&db)
	if err := controllerutil.SetControllerReference(&db, sts, r.Scheme); err != nil {
		return ctrl.Result{}, err
	}
	if err := r.Patch(ctx, sts, client.Apply,
		client.FieldOwner("ledgerdatabase-controller"), client.ForceOwnership); err != nil {
		return ctrl.Result{}, err
	}

	svc := desiredService(&db)
	if err := controllerutil.SetControllerReference(&db, svc, r.Scheme); err != nil {
		return ctrl.Result{}, err
	}
	if err := r.Patch(ctx, svc, client.Apply,
		client.FieldOwner("ledgerdatabase-controller"), client.ForceOwnership); err != nil {
		return ctrl.Result{}, err
	}

	// Read the StatefulSet back to decide Ready vs Progressing.
	var live appsv1.StatefulSet
	ready := false
	if err := r.Get(ctx, types.NamespacedName{Name: sts.Name, Namespace: sts.Namespace}, &live); err == nil {
		ready = live.Spec.Replicas != nil && live.Status.ReadyReplicas == *live.Spec.Replicas && live.Status.ReadyReplicas > 0
	}

	db.Status.ObservedGeneration = db.Generation
	db.Status.Endpoint = fmt.Sprintf("%s.%s.svc.cluster.local:5432", svc.Name, db.Namespace)
	if ready {
		db.Status.Phase = "Ready"
		meta.SetStatusCondition(&db.Status.Conditions, metav1.Condition{
			Type: "Ready", Status: metav1.ConditionTrue, Reason: "Provisioned",
			Message: "StatefulSet is fully ready",
		})
	} else {
		db.Status.Phase = "Provisioning"
		meta.SetStatusCondition(&db.Status.Conditions, metav1.Condition{
			Type: "Ready", Status: metav1.ConditionFalse, Reason: "WaitingForPods",
			Message: "waiting for the StatefulSet's pods to become ready",
		})
	}
	if err := r.Status().Update(ctx, &db); err != nil {
		return ctrl.Result{}, err
	}

	if !ready {
		return ctrl.Result{RequeueAfter: 5 * time.Second}, nil // poll until the pods catch up
	}
	return ctrl.Result{}, nil // level-triggered: we'll run again on the next watch event regardless
}

func (r *LedgerDatabaseReconciler) takeFinalBackup(ctx context.Context, db *platformv1alpha1.LedgerDatabase) error {
	// In this capstone the "backup" is a log line — a real operator would snapshot
	// the PVC or shell out to pg_dump before letting the object disappear.
	ctrl.LoggerFrom(ctx).Info("simulated final backup complete", "ledgerdatabase", db.Name)
	return nil
}

func desiredSecret(db *platformv1alpha1.LedgerDatabase) *corev1.Secret {
	return &corev1.Secret{
		ObjectMeta: metav1.ObjectMeta{Name: db.Name + "-credentials", Namespace: db.Namespace},
		StringData: map[string]string{
			"POSTGRES_USER":     "ledger",
			"POSTGRES_PASSWORD": rand.String(20),
			"POSTGRES_DB":       "ledger",
		},
	}
}

func desiredStatefulSet(db *platformv1alpha1.LedgerDatabase) *appsv1.StatefulSet {
	replicas := int32(1)
	if db.Spec.HighAvailability {
		replicas = 3
	}
	labels := map[string]string{"app": db.Name, "platform.acme.io/owner": db.Name}
	return &appsv1.StatefulSet{
		TypeMeta:   metav1.TypeMeta{APIVersion: "apps/v1", Kind: "StatefulSet"},
		ObjectMeta: metav1.ObjectMeta{Name: db.Name, Namespace: db.Namespace, Labels: labels},
		Spec: appsv1.StatefulSetSpec{
			Replicas:    &replicas,
			ServiceName: db.Name,
			Selector:    &metav1.LabelSelector{MatchLabels: labels},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{Labels: labels},
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{{
						Name:  "postgres",
						Image: "postgres:16-alpine",
						Ports: []corev1.ContainerPort{{ContainerPort: 5432, Name: "postgres"}},
						EnvFrom: []corev1.EnvFromSource{{
							SecretRef: &corev1.SecretEnvSource{
								LocalObjectReference: corev1.LocalObjectReference{Name: db.Name + "-credentials"},
							},
						}},
						VolumeMounts: []corev1.VolumeMount{{
							Name: "data", MountPath: "/var/lib/postgresql/data", SubPath: "pgdata",
						}},
						Resources: corev1.ResourceRequirements{
							Requests: corev1.ResourceList{corev1.ResourceCPU: resource.MustParse("100m")},
						},
					}},
				},
			},
			VolumeClaimTemplates: []corev1.PersistentVolumeClaim{{
				ObjectMeta: metav1.ObjectMeta{Name: "data"},
				Spec: corev1.PersistentVolumeClaimSpec{
					AccessModes: []corev1.PersistentVolumeAccessMode{corev1.ReadWriteOnce},
					Resources: corev1.VolumeResourceRequirements{
						Requests: corev1.ResourceList{
							corev1.ResourceStorage: resource.MustParse(fmt.Sprintf("%dGi", db.Spec.StorageGB)),
						},
					},
				},
			}},
		},
	}
}

func desiredService(db *platformv1alpha1.LedgerDatabase) *corev1.Service {
	labels := map[string]string{"app": db.Name}
	return &corev1.Service{
		TypeMeta:   metav1.TypeMeta{APIVersion: "v1", Kind: "Service"},
		ObjectMeta: metav1.ObjectMeta{Name: db.Name, Namespace: db.Namespace},
		Spec: corev1.ServiceSpec{
			ClusterIP: "None", // headless — stable per-pod DNS, the standard pairing for a StatefulSet
			Selector:  labels,
			Ports:     []corev1.ServicePort{{Port: 5432, Name: "postgres"}},
		},
	}
}

func (r *LedgerDatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&platformv1alpha1.LedgerDatabase{}).
		Owns(&appsv1.StatefulSet{}).
		Owns(&corev1.Service{}).
		Owns(&corev1.Secret{}).
		Complete(r)
}

◆ Key idea — Owns() is what makes self-heal possible

Owns(&appsv1.StatefulSet{}) tells controller-runtime: “when a StatefulSet carrying my owner reference changes — including a delete — enqueue the owning LedgerDatabase for reconciliation.” That single line is the entire mechanism behind Milestone 9 below. Without it, deleting the child would sit unnoticed until the next unrelated event.

⚠ Level-triggered means idempotent

This Reconcile may run ten times for one change, once for ten changes, or be re-run from scratch after a crash. Never keep counters in memory, never assume “this call is the create.” Every invocation reads the world fresh via Get and drives it toward the target with client.Apply — that’s exactly what makes it safe to re-run forever, which is exactly what a reconciler does.

Build, load and deploy the operator

☺ Like you’re 10: First you test the robot on a leash from your own laptop. Only once it behaves do you put it in a box (a container) and let it live in the cluster full-time.

Iterate locally against the real cluster first

Before containerizing anything, install the CRD and run the controller binary on your laptop, pointed at platform-dev via your normal kubeconfig. This is the fastest edit-compile-run loop you will get all capstone:

make install                 # applies config/crd/bases/*.yaml to platform-dev
kubectl get crd ledgerdatabases.platform.acme.io
make run                     # runs the manager binary locally, logs to your terminal
# leave it running in this terminal; open a second one for the next steps

Containerize and load it into `platform-dev`

Once local runs behave, build the real image. On a single-node kind cluster, kind load docker-image is the fastest path — no registry required. If you already stood up a local registry in Part 2 for registry.local/ledger, pushing there instead works identically; either way the image reference in the Deployment below must match.

make docker-build IMG=registry.local/ledger-db-operator:v0.1.0
kind load docker-image registry.local/ledger-db-operator:v0.1.0 --name platform-dev
# (only if you're using a real local registry instead of kind load:)
# docker push registry.local/ledger-db-operator:v0.1.0

RBAC and the operator's own Deployment

Stop the local make run (Ctrl-C) and deploy the containerized operator into the platform namespace — the same namespace hosting Argo CD, so add-ons stay in one place:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ledger-db-operator
  namespace: platform
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ledger-db-operator
rules:
  - apiGroups: [platform.acme.io]
    resources: [ledgerdatabases]
    verbs: [get, list, watch, create, update, patch, delete]
  - apiGroups: [platform.acme.io]
    resources: [ledgerdatabases/status, ledgerdatabases/finalizers]
    verbs: [get, update, patch]
  - apiGroups: [apps]
    resources: [statefulsets]
    verbs: [get, list, watch, create, update, patch, delete]
  - apiGroups: [""]
    resources: [services, secrets]
    verbs: [get, list, watch, create, update, patch, delete]
  - apiGroups: [""]
    resources: [events]
    verbs: [create, patch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ledger-db-operator
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: ledger-db-operator }
subjects:
  - kind: ServiceAccount
    name: ledger-db-operator
    namespace: platform
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ledger-db-operator
  namespace: platform
  labels: { app: ledger-db-operator }
spec:
  replicas: 1
  selector: { matchLabels: { app: ledger-db-operator } }
  template:
    metadata: { labels: { app: ledger-db-operator } }
    spec:
      serviceAccountName: ledger-db-operator
      containers:
        - name: manager
          image: registry.local/ledger-db-operator:v0.1.0
          args: ["--leader-elect"]
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits: { memory: 256Mi }
          livenessProbe:  { httpGet: { path: /healthz, port: 8081 }, initialDelaySeconds: 5 }
          readinessProbe: { httpGet: { path: /readyz,  port: 8081 }, initialDelaySeconds: 5 }

⚠ Apply the CRD before the Deployment

If the CustomResourceDefinition isn’t registered yet, the operator’s manager will fail its /readyz check trying to start an informer against a type the API server doesn’t know. Always apply order: CRD → RBAC → Deployment. In Git this is naturally enforced by putting them in one manifest set that Argo CD applies together, or with sync waves if you want to be explicit.

Wire it into GitOps

☺ Like you’re 10: Once the robot works by hand, you hand its blueprint to Recon so it lives in the poster too — no more manual applying, ever again.

Everything you just kubectl apply’d by hand now needs to move into the same platform-capstone Git repo Argo CD already watches from Part 1. Add one new top-level folder, platform/, alongside the apps/ and ledger/ folders the whole capstone canon relies on:

platform-capstone/                    # the one Git repo the whole capstone shares
├── apps/                             # Argo CD Application manifests — App-of-Apps root points here
│   ├── ledger.yaml                   # from Part 1/2 — the ledger app itself
│   └── ledger-db-operator.yaml       # NEW this part — the operator as its own Argo Application
├── platform/                         # NEW this part — platform add-on manifests (mirrors the "platform" namespace)
│   └── ledger-db-operator/
│       ├── crd.yaml                  # config/crd/bases/platform.acme.io_ledgerdatabases.yaml
│       ├── rbac.yaml                 # ServiceAccount + ClusterRole + ClusterRoleBinding
│       └── deployment.yaml           # the operator's own Deployment
└── ledger/                           # the ledger service's own Kubernetes manifests
    ├── deployment.yaml / rollout.yaml    # from Part 1/2
    ├── service.yaml
    └── ledger-db.yaml                # NEW this part — the LedgerDatabase custom resource

The custom resource itself — the eight lines a developer actually writes — lands in ledger/, right beside the app it belongs to:

# ledger/ledger-db.yaml
apiVersion: platform.acme.io/v1alpha1
kind: LedgerDatabase
metadata:
  name: ledger-db
  namespace: ledger
spec:
  engine: postgres
  storageGB: 10
  highAvailability: false
  backupSchedule: "0 3 * * *"          # 3am daily — a future part's operator work reads this

And a second Argo CD Application in apps/ reconciles the operator itself, separate from the app, so the two lifecycles don’t block each other:

# apps/ledger-db-operator.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ledger-db-operator
  namespace: platform
spec:
  project: default
  source:
    repoURL: https://github.com/YOU/platform-capstone.git
    targetRevision: main
    path: platform/ledger-db-operator
  destination:
    server: https://kubernetes.default.svc
    namespace: platform
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [ CreateNamespace=true ]

Commit all of it, push, and let the App-of-Apps root from Part 1 pick up the new child Application automatically. From this point on, you never kubectl apply the operator, its RBAC, or the LedgerDatabase by hand again — Recon does it, the same reconciler you met in GitOps, just now watching a folder that happens to describe your own operator.

Do the reps — 14 milestones

☺ Like you’re 10: Reading about a robot doesn’t prove it works. Build it, then try to trick it, three different ways.

Work these in order on your existing platform-dev cluster. Each should take five to twenty minutes. Tick a box and your progress saves in this browser.

0 / 14 milestones complete

🦆 Dot’s-eye view

“Before this part, ‘I need a database’ meant a Slack message to platform and a day of waiting. Now it’s eight lines in the same PR as my Deployment change. I don’t know or care that there’s a Go reconciler behind kind: LedgerDatabase — I just know it shows up Ready a minute after I merge, and if I kubectl delete the Secret by accident, it doesn’t stay broken.”

What you’ll have built

☺ Like you’re 10: By the end, the platform speaks a brand-new word, there’s a tireless robot behind it, and the poster (Git) is back in charge of both.

Your platform-dev cluster now understands kind: LedgerDatabase: a hand-written CRD with a real OpenAPI schema, printer columns and a /status subresource; a Kubebuilder-scaffolded Go operator running in the platform namespace that provisions a Postgres StatefulSet, headless Service and credentials Secret for every LedgerDatabase; owner references that cascade-delete children automatically; a finalizer that runs cleanup before an object is allowed to disappear; status conditions that make kubectl get ldb tell the truth; and the whole thing — CRD, RBAC, operator Deployment, and the ledger-db custom resource itself — reconciled from the same Git repo Argo CD has watched since Part 1. Reconciliation and self-heal are proven twice: once by deletion (Milestone 9) and once by drift (Milestone 10).

🎬 At the Platform Guild
🦊Foxy: Wait, why build a whole Go operator for one Postgres StatefulSet? Couldn’t Benny just commit the StatefulSet YAML straight into ledger/?
🦋Mira: He could — for one database. But the CRD is the noun every future service reuses. When Nutty’s team needs their own database next month, they write eight lines, not two hundred, and they get the same backups and self-heal ledger gets automatically.
🤖Recon: BEEP. And I don’t care if desired state came from a plain Deployment or a custom resource — I watch, diff, act, forever, either way. Delete my Service, I rebuild it. Every time.
👺Gizmo: Ooh, CRDs! Let’s make one for every tiny knob — kind: Cache, kind: Coffee, kind: Whatever — and wrap a Bash script behind each. Ship it! 🤑
🐢Timmy: A CRD with no reconcile loop is a database table wearing a costume, Gizmo. And every noun we ship is an API we support forever — thin schema, real controller, or don’t bother.
🦆Dot: I just want ledger-db to show Ready: True and stay that way. Which — it did, even after you two deleted my Secret twice trying to prove a point.

🐢 Timmy’s checkpoint

1. Why does the reconciler add a finalizer before creating the StatefulSet, Service and Secret, rather than after? 2. What single line in SetupWithManager makes a deleted child StatefulSet wake the parent LedgerDatabase back up? 3. Why does the reconciler check for an existing Secret before creating one, instead of applying it every pass like it does the StatefulSet and Service? 4. What two Git changes prove Argo CD — not you — now owns the operator and the database?

Check your answers

Because once the StatefulSet, Service and Secret exist, deleting the LedgerDatabase without a finalizer would let Kubernetes remove the object (and cascade-delete the children) before the operator ever got a chance to run cleanup — the finalizer must be in place before there is anything worth cleaning up.
Owns(&appsv1.StatefulSet{}) (and the matching lines for Service and Secret) — it tells controller-runtime to enqueue the owning LedgerDatabase whenever an owned child changes, including a delete.
The Secret holds a randomly generated password. Applying it every pass the way the StatefulSet and Service are applied would silently rotate the credentials on every reconcile, breaking a Postgres instance that's already running with the old password — so it's created once, on first sight, and left alone afterward.
Deleting ledger/ledger-db.yaml from Git and watching prune remove the live object (proving Git deletions are real deletions, not orphans); and reverting that deletion and watching the whole stack — CRD, operator, and the custom resource — come back from one commit with zero manual kubectl apply.

Next: hand this exact CRD to a developer portal so requesting a database stops being “write eight lines of YAML” and becomes “fill in one form” — Capstone Part 4 — Self-Service, pairing with Self-Service & Developer Portals. Deep-dive the concepts behind everything you just built on Platform APIs, CRDs & Operators and Kubebuilder & the Operator SDKs, compare against the declarative alternative on Crossplane, pressure-test with the Platform APIs labs and practice tasks, and see the whole six-part build in context on the Capstone Hub.

Capstone Part 3 — a new noun for ledger

Arriving from Part 2, leaving for Part 4

Design the noun: the `LedgerDatabase` CRD

The Go type — where the CRD comes from

Scaffold the operator with Kubebuilder

Write the reconciler

Build, load and deploy the operator

Iterate locally against the real cluster first

Containerize and load it into `platform-dev`

RBAC and the operator's own Deployment

Wire it into GitOps

Do the reps — 14 milestones

Part A · Design and scaffold

Part B · Build and deploy

Part C · Prove reconciliation, self-heal, and safe deletion

What you’ll have built

Capstone Part 3 — a new noun for ledger

Arriving from Part 2, leaving for Part 4

Design the noun: the LedgerDatabase CRD

The Go type — where the CRD comes from

Scaffold the operator with Kubebuilder

Write the reconciler

Build, load and deploy the operator

Iterate locally against the real cluster first

Containerize and load it into platform-dev

RBAC and the operator's own Deployment

Wire it into GitOps

Do the reps — 14 milestones

Part A · Design and scaffold

Part B · Build and deploy

Part C · Prove reconciliation, self-heal, and safe deletion

What you’ll have built

Design the noun: the `LedgerDatabase` CRD

Containerize and load it into `platform-dev`