Control Plane Internals

Kubernetes Architecture named the five control-plane components and what each one is responsible for. This page opens three of them back up and looks at the actual machinery running inside: how etcd gets a cluster of independent processes to agree on one truth using the Raft consensus algorithm, how kube-apiserver streams live state to thousands of clients through a watch mechanism instead of making them poll, how the reflector → informer → work-queue pipeline in client-go turns that watch stream into the reconcile loop every controller runs, and how Lease-based leader election keeps exactly one kube-scheduler and one kube-controller-manager doing real work across a highly-available control plane, no matter how many replicas are running. None of this is CKA, CKAD, or CKS blueprint material — it's the layer underneath the blueprint, useful the moment you're operating etcd yourself, writing an operator, or debugging a control plane that's behaving strangely under load.

Why this page goes past the exam blueprint

☺ Like you're 10: Kubernetes Architecture told you the five control-plane pieces exist and roughly what job each one does — this page is what's actually humming inside three of them, at a level no exam question will ever ask you to reproduce.

Kubernetes Architecture is the map: it names kube-apiserver, etcd, kube-scheduler, kube-controller-manager, and cloud-controller-manager, and traces one Pod's request through all five. The Kubernetes API & the Controller Pattern goes one layer deeper and shows you the reconcile loop itself — desired state, actual state, and the gap a controller closes between them — while explicitly deferring two things to this page: "the internals of that scheduling — informers caching the watch stream, a work queue de-duplicating and rate-limiting reconciles." This page pays that off, plus two things neither earlier page touches at all: how etcd's own consistency actually works, and how an HA control plane avoids running the same controller twice.

If you only need this material to pass an exam, you can stop reading now — none of the five sections below appear on the CKA, CKAD, or CKS blueprint. If you're planning to run etcd yourself, write an operator with kubebuilder, or diagnose a control plane that's degrading under load rather than one that's simply down, keep going. And if what you actually want is the API-shape side of this machinery — how apiVersion plus kind resolves to a URL, what a RESTMapper does — that lives in Platform Engineering's Kubernetes as the Platform Substrate, which this page deliberately doesn't re-derive.

etcd and Raft: how a distributed log agrees with itself

☺ Like you're 10: Three copies of the same logbook, run by three separate desks, and a new entry doesn't count as real until at least two of the three desks have it written down.

etcd doesn't stay consistent across its members by luck — it runs the Raft consensus algorithm, and every etcd cluster you'll ever operate is a live instance of it. Raft's job is simple to state and genuinely tricky to build correctly: given N independent processes that can each crash, restart, or fall behind, keep them agreeing on one ordered log of entries, even while that's happening. Members elect one of themselves leader for a fixed term (a monotonically increasing counter), and only the leader accepts new writes — a client write becomes a proposed log entry, the leader replicates it to every follower, and the entry is only committed, and only then visible to reads, once a majority of members (a quorum) have durably appended it to their own log. That's the entire reason etcd clusters run with an odd member count: three members tolerate the loss of one and still have a quorum of two; five tolerate the loss of two; four members buy you nothing over three, because you still only tolerate losing one before quorum breaks, while carrying an extra member's replication cost for free.

If the leader stops sending heartbeats — a crash, a network partition, a slow disk that makes it look dead — followers wait out a randomized election timeout (each one picks a slightly different wait, specifically so they don't all call an election in the same instant), then one of them proposes itself as candidate for the next term and asks the others for votes. Whichever candidate reaches a majority first becomes the new leader for that term, and every future write and heartbeat carries that new term number. You can watch this directly:

$ etcdctl --endpoints=https://10.0.4.11:2379,https://10.0.4.12:2379,https://10.0.4.13:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key endpoint status --cluster -w table

+-------------------------+------------------+---------+-----------+-----------+
|        ENDPOINT         |        ID        | VERSION | IS LEADER | RAFT TERM |
+-------------------------+------------------+---------+-----------+-----------+
| https://10.0.4.11:2379  | 8211f1d0f64f3269 | 3.5.9   |   true    |     4     |
| https://10.0.4.12:2379  | 91bc3c398fb3c146 | 3.5.9   |   false   |     4     |
| https://10.0.4.13:2379  | fd422379fda50e48 | 3.5.9   |   false   |     4     |
+-------------------------+------------------+---------+-----------+-----------+
# Same RAFT TERM on every healthy member — that agreement IS consensus.
# Kill the leader and re-run this: expect a higher term and a new IS LEADER=true row.

Underneath the log, etcd stores the cluster's entire keyspace with MVCC (multi-version concurrency control): every write bumps one global, monotonically increasing revision number for the whole keyspace, not per key, and old revisions are retained until periodic compaction discards them. That global revision is where a Kubernetes object's resourceVersion ultimately comes from — apiserver exposes it to clients as an opaque string, and it's the mechanism the next section builds on directly. Because every write needs a majority round-trip and a disk fsync before it's durable, etcd is unusually intolerant of slow storage — a few hundred milliseconds of added disk latency is enough to trigger missed heartbeats and unnecessary re-elections, which is exactly the failure mode below.

Watch: how kube-apiserver streams state without polling

☺ Like you're 10: Instead of a reporter calling the newsroom every ten seconds to ask "anything new?", the newsroom keeps the phone line open and just talks the moment something changes.

Every controller, kubelet, and long-running client in a cluster needs to know about changes to some slice of objects the instant they happen, and asking "did anything change?" on a fixed interval scales terribly — thousands of clients each polling thousands of objects would flatten kube-apiserver on its own. Kubernetes avoids that entirely with watch: a client sends a normal GET with ?watch=true and a starting resourceVersion, and instead of closing the connection after one response, apiserver holds it open and streams a sequence of ADDED, MODIFIED, and DELETED events as they happen, each one carrying the object's new resourceVersion. The client just keeps that one HTTP connection open and reads events off it as they arrive — no re-request, no polling loop.

apiserver doesn't turn around and open a fresh etcd watch for every client watch it serves. Each apiserver replica keeps its own in-memory watch cache per resource type, seeded with an initial List and kept current by exactly one etcd watch per type, and every client watch for that type is served out of that shared cache rather than hitting etcd directly. That's what lets a busy cluster support thousands of concurrent client watches without etcd ever seeing thousands of concurrent watch connections — the fan-out happens in apiserver's memory, once per replica, not once per client.

From a watch to a reconcile: informers, listers, and work queues

☺ Like you're 10: A controller doesn't sprint to do work the instant a ticker event arrives — it jots down "something changed here" on a to-do list, and a separate pool of workers works through that list at a steady, controlled pace.

A raw watch stream is a firehose, and no controller author wants to hand-write "reconnect on disconnect, replay missed events, keep a local cache in sync" from scratch for every controller. client-go's informer machinery is the shared plumbing that does it once, correctly, for everyone — and it's the exact thing the API & controller pattern page pointed here for. Four pieces, in order:

• A Reflector runs one List (to seed initial state) followed by a continuous Watch against exactly one resource type, and pushes every change into a DeltaFIFO — an ordered queue of "here's what changed" deltas.
• A SharedInformer drains that FIFO, updates a local, thread-safe cache called the Indexer (indexed by namespace and whatever else you configure), and then fires the delta out to every registered event handler. It's shared deliberately — one informer per resource type serves every controller in the process that cares about that type, so N controllers watching Pods cost one watch connection, not N.
• Registered event handlers (OnAdd/OnUpdate/OnDelete) are meant to do almost nothing — their entire job is computing a stable namespace/name key and enqueuing it.
• A work queue (rate-limited, and automatically de-duplicating a key that's already queued) is what a fixed pool of worker goroutines actually pop from, one key at a time, to run the real reconcile logic.

queue := workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter())

informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
    AddFunc: func(obj interface{}) {
        key, _ := cache.MetaNamespaceKeyFunc(obj)
        queue.Add(key) // just the key — never the object itself
    },
    UpdateFunc: func(old, new interface{}) {
        key, _ := cache.MetaNamespaceKeyFunc(new)
        queue.Add(key)
    },
    DeleteFunc: func(obj interface{}) {
        key, _ := cache.DeletionHandlingMetaNamespaceKeyFunc(obj)
        queue.Add(key)
    },
})

// One of N worker goroutines, running forever:
for {
    key, shutdown := queue.Get()
    if shutdown {
        return
    }
    if err := reconcile(key.(string), lister); err != nil { // reads the Indexer, no network call
        queue.AddRateLimited(key) // retry, with exponential backoff
    } else {
        queue.Forget(key)
    }
    queue.Done(key)
}

Two design choices here matter more than the code makes them look. First, reconcile reads current state from the Lister — a thin read-only wrapper over the Indexer's local cache — not from a live API call, which is exactly why a busy controller doesn't hammer kube-apiserver on every reconcile; it only touches the API to write results back. Second, enqueuing a key instead of the object itself means the queue naturally coalesces rapid-fire updates: if the same object changes five times before a worker gets to it, that's still one key, reconciled once against whatever the Indexer holds by the time the worker actually runs — which is exactly the level-triggered behavior the controller pattern page described for ReplicaSet, now visible as a direct consequence of this queue design rather than a special case. A separate, periodic resync (informers default to roughly every 30 seconds, tunable) re-enqueues every key in the cache regardless of whether anything changed, as a safety net against any watch event that silently never arrived.

This exact pipeline is what Operators & Custom Resource Definitions covers you writing directly, and what higher-level frameworks like kubebuilder and controller-runtime wrap for you so you write a Reconcile(ctx, req) function and never touch a Reflector by hand. It's also worth knowing this is the same shape kube-scheduler and kube-controller-manager themselves are built from internally — they aren't special-cased, they run the identical reflector-informer-workqueue plumbing against the resource types they care about.

Leader election: one active scheduler, one active controller-manager

☺ Like you're 10: The editor's badge — only one bureau chief wears it at a time, it gets handed back in a few seconds if that chief goes quiet, and a backup chief is always standing by, ready to put it on.

An HA control plane runs multiple replicas of kube-scheduler and kube-controller-manager for availability — but unlike kube-apiserver, running two of them actively reconciling at the same instant isn't just wasteful, it's a correctness bug. Two active schedulers could both filter-and-score the same Pending Pod and both write a Binding for it, to two different nodes; two active controller-managers could both act on the same Node going unhealthy. So both components default to --leader-elect=true in any HA deployment, and use a Kubernetes-native mechanism to guarantee exactly one active replica at a time: a Lease object, in the coordination.k8s.io/v1 API group.

Every replica races to acquire the same named Lease (kube-scheduler, kube-controller-manager, both in kube-system). Whichever one wins writes its own identity into holderIdentity and starts running its actual reconcile loops; the losers block, doing nothing, continuously watching that same Lease. The winner must keep renewing it well before it expires — controlled by three tunables, defaulting to a lease duration of 15 seconds, a renew deadline of 10 seconds (the active replica gives up and steps down if it can't renew within this window), and a retry period of 2 seconds between acquisition attempts by standbys. Miss that renewal — a crash, a network partition, a node dying — and within roughly the lease duration, a standby wins the next race and takes over. No process needs to be restarted and no operator needs to intervene; the standby was already running, just idle.

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: kube-scheduler
  namespace: kube-system
spec:
  holderIdentity: kind-control-plane-2_a3f9c1e2-8b7d-4e11-9a2f-0c9e5f7d1b44
  leaseDurationSeconds: 15
  acquireTime: "2026-08-27T09:41:03.000000Z"
  renewTime: "2026-08-27T10:15:47.128441Z"
  leaseTransitions: 1

Worth keeping separate in your head: the same Lease API type also backs the per-node heartbeat mechanism kubelet renews and the Node controller watches — covered in Kubernetes Architecture — but that's a different Lease, in a different namespace (kube-node-lease), doing a different job. One API shape, two unrelated uses: "is this node still alive" and "which replica of this component is allowed to act."

Putting it together: a scheduler failover, end to end

☺ Like you're 10: Three separate layers of "who's in charge right now" are all running at once in a healthy cluster — and they almost never fail together, which is exactly the point.

Zoom out and a highly-available control plane is really three independent leadership questions being answered simultaneously, at three different layers, none of them aware of the other two: which etcd member is the current Raft leader (an etcd-internal question, answered by etcd, invisible to the rest of the cluster), whether kube-apiserver has a leader at all (it doesn't — every replica stays active), and which single kube-scheduler and which single kube-controller-manager currently hold their respective Lease. A scheduler crash affects exactly one of these three layers: the standby schedulers, who were already running and already watching the Lease, see the active replica stop renewing, race for it once the lease genuinely expires, and the winner picks up scheduling within roughly leaseDurationSeconds — while etcd's Raft leader and every kube-apiserver replica never notice anything happened at all.

The scheduling algorithm itself — filtering, scoring, resource requests versus node capacity — isn't this page's territory; that's Scheduling & Resource Management. This page only cares about which single scheduler process gets to run that algorithm at any given moment, and how the cluster notices and recovers when it stops. Similarly, if a real control-plane incident is what brought you here, the systematic approach to diagnosing one — not just this page's internals — belongs to A Troubleshooting Methodology, and the operational side of etcd specifically (snapshot, restore, and the exact commands CKA expects) lives in Cluster Architecture, Installation & Configuration.