containerd

containerd is the daemon that actually runs your containers — the layer directly beneath kubelet, reached over the Container Runtime Interface (CRI), that pulls images, sets up a Pod's sandbox, and hands each container off to a shim and runc to start. Every other page in this course treats it as a fact of the architecture; this page treats it as a tool with its own daemon, its own config file, its own CLI clients, and its own failure modes worth knowing by name. That means the plugin architecture containerd is actually built from, why Docker Engine and Kubernetes parted ways when "dockershim" was removed, the handful of settings in /etc/containerd/config.toml that quietly break a node when they're wrong, a genuinely surprising gotcha in how containerd isolates its own state (it has "namespaces," and they are not Kubernetes namespaces), the three-way split between ctr, crictl, and nerdctl, and how to use crictl to debug a node directly when kubectl itself has nothing to say.

What containerd actually is

☺ Like you're 10: It's not one big program — it's a small core plus a set of plug-in pieces, each one owning exactly one job: fetching images, storing layers, or starting processes.

containerd began life as the piece of Docker Engine that actually managed the container lifecycle, was spun out and donated to the CNCF in 2017, and graduated as an independent, top-level CNCF project in 2019 — the same governance track Kubernetes itself went through. Architecturally it's a single daemon (containerd, usually running as a systemd unit) exposing one gRPC API over a Unix socket, normally /run/containerd/containerd.sock, with almost everything behind that API implemented as a plugin: a content store (the shared, content-addressed blob store every pulled image layer lands in), one or more snapshotters (usually overlayfs on Linux, which turns those immutable layers into a writable root filesystem for a container without copying them), a runtime v2 shim manager, an image service, an events subsystem other plugins can subscribe to, and — the one this whole course cares about — the CRI plugin, which is what lets kubelet talk to containerd at all.

Every one of the four clients at the top of that diagram is a peer speaking to the same daemon over the same socket — kubectl and the API server are nowhere in this picture at all, which is exactly why a node's containerd keeps running fine through an API server outage and is precisely what makes it useful for debugging one.

Why Docker Engine and Kubernetes parted ways

☺ Like you're 10: Docker Engine and containerd were never really two competitors — Docker Engine always ran on containerd underneath. Kubernetes just stopped needing the extra layer on top.

Before Kubernetes 1.20, kubelet talked to full Docker Engine through a built-in translation layer called dockershim, because Docker Engine itself never spoke CRI — it predates CRI entirely. That shim was deprecated in 1.20 and removed outright in Kubernetes 1.24 (2022). The framing that briefly went around — "Kubernetes drops Docker support" — overstated it: containerd was Docker Engine's own runtime the whole time, so once the shim was gone, kubelet simply talked to the same containerd directly instead of routing through dockerd first. Nothing about the image format changed; an image built with docker build is a standard OCI image and runs identically whether containerd is fronted by Docker Engine or not. This course's DevOps sibling covers the full Docker Engine → containerd → runc pipeline, including the shim-survives-a-restart detail, from the build side — see the DevOps Docker tool guide — and the Platform Engineering course frames CRI as one of three pluggable contracts (alongside CNI and CSI) in Kubernetes as the Substrate. This course's own Cluster Architecture, Installation & Configuration blueprint page introduces all three extension interfaces at CKA depth; this page goes one layer deeper on containerd specifically.

The CRI plugin: how kubelet actually talks to containerd

☺ Like you're 10: kubelet doesn't call containerd's private API — it sends one standard, documented request format, and any runtime that understands that format can sit behind it.

CRI is a gRPC contract with two services: RuntimeService (start/stop/list Pods and containers, exec into one, fetch logs) and ImageService (pull, list, and remove images). containerd's CRI plugin implements both against the same socket every other client uses. kubelet is configured with --container-runtime-endpoint=unix:///run/containerd/containerd.sock (the default on nodes provisioned by kubeadm) and, from that point on, CRI is the only way kubelet talks to a runtime — there is no other supported code path since dockershim's removal.

One CRI concept worth knowing by name: every Pod gets a PodSandbox before any of its containers start — in practice, on Linux, an actual container running the tiny pause image, which does nothing but hold open the network namespace (and IPC/UTS namespaces) that every container in that Pod then joins. That's the mechanism, not an implementation detail — it's the entire reason containers in one Pod share localhost and can see each other's ports without any Service in between. The pause image containerd uses for this is configured, not hardcoded, which is the first line of the next section.

The config you actually write: /etc/containerd/config.toml

☺ Like you're 10: One text file decides which cgroup style containerd uses, which tiny "pause" image every Pod gets built on top of, and where it's allowed to pull images from.

containerd's config is TOML, versioned with version = 2 at the top of the file. Most nodes start from a generated default and edit a handful of keys rather than writing one from scratch:

# generate a default config, then edit it — never hand-write from a blank file
$ containerd config default | sudo tee /etc/containerd/config.toml
$ sudo systemctl restart containerd

# /etc/containerd/config.toml — the handful of keys that matter most on a Kubernetes node
version = 2

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.9"   # must match what kubelet/kubeadm expects

  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
    runtime_type = "io.containerd.runc.v2"

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
      SystemdCgroup = true   # must match kubelet's --cgroup-driver

  [plugins."io.containerd.grpc.v1.cri".registry]
    config_path = "/etc/containerd/certs.d"   # preferred over the older inline [registry.mirrors]

Two of those keys are the ones that actually break nodes when they're wrong, and both are covered in depth in the gotchas below: SystemdCgroup has to agree with whatever cgroup driver kubelet itself is using, and sandbox_image has to be an image that's actually reachable and matches what the rest of the control plane expects a Pod's sandbox to be running. The registry block controls where image pulls actually go — mirrors, insecure registries, and pull credentials for a private registry all live here (or, in the newer and now-preferred form, in per-host hosts.toml files under the config_path directory, one subdirectory per registry hostname).

containerd's own namespaces — not the Kubernetes kind

☺ Like you're 10: The word "namespace" means something completely different here than it does everywhere else in this course — mixing the two up is how ctr images ls shows you an empty list on a node that's clearly running dozens of Pods.

containerd multiplexes multiple independent clients over one daemon using its own concept of a namespace — a way to keep one client's images, containers, and content store entries from colliding with another's. It has nothing to do with a Kubernetes Namespace object. By convention, the CRI plugin puts absolutely everything it manages — every image kubelet pulls, every container kubelet starts — into one containerd namespace called k8s.io. crictl always talks CRI, so it's automatically scoped to k8s.io and this never comes up. ctr, the low-level client, defaults to a namespace called default instead — which is why ctr images ls on a perfectly healthy Kubernetes node routinely prints nothing at all.

$ ctr namespace ls                       # see every containerd namespace that has anything in it
$ ctr images ls                          # empty on a k8s node — wrong (default) namespace
$ ctr -n k8s.io images ls                # this is the one kubelet actually populated
$ ctr -n k8s.io containers ls
$ ctr -n k8s.io tasks ls                 # running processes, one per container

Day-to-day commands: ctr, crictl, and nerdctl

☺ Like you're 10: Three different remote controls for the same TV, built for three different people — containerd's own developers, a Kubernetes operator debugging a node, and someone who just wants Docker-shaped muscle memory to keep working.

# crictl config once, so you don't retype the endpoint every time
$ sudo tee /etc/crictl.yaml <<'EOF'
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
EOF

# nerdctl, for comparison — docker-shaped commands, no docker.sock anywhere
$ nerdctl run -d --name web -p 8080:80 nginx:1.27
$ nerdctl compose up -d

Note what's genuinely absent from that table: kubectl. It isn't a fourth option here because it doesn't talk to containerd at all — it talks to the API server, which talks to kubelet on the right node, which talks to containerd over CRI. Every hop in that chain is a place something can be unreachable while containerd itself is perfectly healthy, which is exactly the situation the next section is built for.

crictl for direct node-level debugging

☺ Like you're 10: When the phone lines to the control room are down, you don't give up — you walk straight into the engine room and ask the engine what it's actually doing.

Cluster Architecture, Installation & Configuration and Troubleshooting already introduce crictl as the tool for exactly one scenario: the API server itself is down, so nothing routed through it — including kubectl — can tell you anything. This section is the fuller node-level workflow those pages point back to. SSH onto the node in question, then work from the runtime's own view of the world instead of the cluster's:

# the read loop, entirely local to this one node
$ crictl pods                                        # every PodSandbox this node's runtime knows about
$ crictl pods --name checkout                         # find the sandbox for a specific Pod by name
$ crictl ps -a                                        # every container, any state — Exited included
$ crictl images                                       # what's actually cached locally, right now

# once you've found the container you care about
$ crictl inspect  | grep -A4 '"pid"'    # the container's PID on this host, from the runtime's own record
$ crictl logs --tail=100 -f              # stdout/stderr, no kube-apiserver in the path at all
$ crictl exec -it  sh                    # a shell inside it, same caveat as kubectl exec: needs one to exist
$ crictl stats                           # live CPU/memory, straight from the runtime, no metrics-server needed

# cleanup, when a stopped container or a stale image is the actual problem
$ crictl rm 
$ crictl rmi 

The correlating step people skip is tying a Pod you already know about (from a stale kubectl describe pod output, or a UID you copied down before the API server went away) back to a specific sandbox: crictl inspectp prints the sandbox's own metadata, including the Pod's UID and namespace, which is how you confirm you're looking at the right one before you start reading logs or attaching a shell.

Gotchas and failure modes

☺ Like you're 10: Most containerd problems on a Kubernetes node trace back to one of a small handful of named causes — worth memorizing the names so you recognize the symptom instantly next time.

Cgroup driver mismatch. kubelet and containerd both need to agree on whether they're managing cgroups through systemd or the raw cgroupfs — kubeadm-provisioned clusters default both to systemd since 1.22 and configure them to match automatically, but a manually assembled node, an old runbook, or a copy-pasted config from a pre-1.22 guide can still leave them disagreeing. The result isn't subtle: kubelet fails to start, or Pods on that node behave inconsistently under resource pressure because two different cgroup managers are fighting over the same hierarchy. Fix it in one place — SystemdCgroup in config.toml — and confirm kubelet's own --cgroup-driver (or its config file equivalent) agrees.

Pause/sandbox image drift. sandbox_image in config.toml has to point at an image kubelet and containerd both expect and that's actually reachable from the node. A stale pin, a registry migration (the default moved from k8s.gcr.io to registry.k8s.io around the 1.25 era), or a firewalled registry shows up as Pods stuck indefinitely at sandbox creation — before any of your own containers are even attempted, which makes it a confusing one to diagnose from kubectl describe pod alone. crictl pull <the configured sandbox_image> run directly on the node isolates the problem to exactly this cause in seconds.

Registry config v1 vs. v2 confusion. Older guides show private-registry auth as an inline [plugins."io.containerd.grpc.v1.cri".registry.mirrors] block; newer containerd versions deprecate that path in favor of per-host hosts.toml files under config_path. Mixing the two forms, or following a tutorial written for the other one, produces pulls that silently succeed against the public registry while quietly ignoring the mirror or credential you thought you configured.

containerd surviving a restart doesn't mean containers survive a reboot. Because each container's containerd-shim-runc-v2 is its own independent process, sudo systemctl restart containerd to pick up a config change does not kill running containers — the shims keep the containers alive and simply reconnect once containerd comes back. A full node reboot is a different story: nothing about that shim survives the kernel itself going down, so every container on that node restarts along with it.