Docker

Docker is the toolchain that turned Linux containers from a kernel feature experts could wield into a workflow anyone could learn in an afternoon: one file format for describing an image (the Dockerfile), one layered, content-addressed format for storing it, one CLI for building, running, and shipping it, and a registry protocol for moving it between machines. Containers & orchestration covers why containers beat VMs at isolation speed and density at all — this page assumes that context and goes deep on the tool most people actually mean when they say "containers": how an image is really a stack of layers, why the build cache invalidates the way it does, the Dockerfile habits that separate a lean production image from a bloated one, and where Docker Engine itself sits relative to containerd, runc, and the rest of the runtime landscape it helped standardize.

What Docker is and the problem it solves

☺ Like you're 10: Linux could already fence off processes before Docker existed — Docker's real invention was making that fencing easy to describe, easy to build, and easy to hand to someone else.

The isolation primitives a container actually runs on — namespaces and cgroups — are Linux kernel features that predate Docker by years, and tools like LXC already exposed them before 2013. What LXC didn't have was a workflow: a standard way to describe an image as a sequence of build steps, a standard on-disk format for that image, a registry protocol for distributing it, and one CLI command to go from source code to a running container. Docker, released in March 2013 by a company then called dotCloud (soon renamed Docker, Inc.), supplied all four at once, and that combination — not the isolation itself — is what made containers a mainstream default rather than a specialist's tool.

"Docker" today names a family of components, not one program, and conflating them is the source of a lot of confusion later on this page: Docker Engine is the local build-and-run stack (covered in the architecture section below); the Docker CLI (docker) is the client you actually type commands into; Docker Compose defines and runs a multi-container application from one YAML file, the standard tool for the local inner loop; Buildx/BuildKit is the modern build engine (its own section below); Docker Desktop packages all of the above plus a Linux VM for macOS and Windows developers; and Docker Hub is Docker, Inc.'s public image registry, one of many registries an image can live in alongside Amazon ECR, Google Artifact Registry, or GitHub Container Registry.

Docker's other lasting contribution was giving that format away. In 2015 Docker co-founded the Open Container Initiative (OCI) under the Linux Foundation and donated the image and runtime specifications that became its foundation, which is why an image built with docker build today runs unmodified under Podman, containerd, or a Kubernetes node that has never had Docker Engine installed on it — the format is a public standard, not a Docker-only file.

Where Docker fits in the delivery pipeline

☺ Like you're 10: Docker's job happens right after code is tested and right before something else takes the finished box and puts it to work.

In the CI/CD pipeline, a build stage runs docker build (or docker buildx build) against a Dockerfile checked into the repository, tags the resulting image, and pushes it to a registry — the same artifact-storage role covered generally in build & artifact management. Whatever CI product is running that stage — Jenkins, GitHub Actions, GitLab CI/CD, or CircleCI — is invoking the same docker CLI a developer would type locally; nothing about Docker itself changes between a laptop and a CI runner except who's typing the command. Increasingly that stage also runs an image scanner (Docker Scout, Trivy, or Grype) before the push, catching known-vulnerable base-image packages ahead of the registry rather than after — see Shift-Left Security and Supply-Chain Security & SBOM for the fuller pattern, including generating a software bill of materials for the image at the same step.

Downstream, whatever pulls that pushed image and actually runs it is a separate concern from Docker entirely — a single docker run on one host, a Compose stack for local development, or, at real scale, an orchestrator that schedules many copies across many machines, which is exactly the territory containers & orchestration and Kubernetes cover. Docker builds and ships the box; it does not decide how many copies of the box run or where.

Architecture: Docker Engine, containerd, and runc

☺ Like you're 10: Typing one Docker command actually wakes up a small chain of programs, each handing the job to a more specialized one below it, until the last one just starts a process.

The docker CLI is a thin client. It talks to a background daemon, dockerd, over a REST API — by default a Unix socket at /var/run/docker.sock on Linux — and it's dockerd that owns networking, volumes, Compose orchestration, and driving image builds through BuildKit. But dockerd doesn't create containers itself either: it delegates that to containerd, a smaller, CRI-compliant daemon Docker spun out and donated to the CNCF in 2017 (graduated as an independent CNCF project in 2019), which handles image pulling and storage and the container lifecycle over its own gRPC API. For each running container, containerd launches a small, dedicated containerd-shim-runc-v2 process, whose entire job is to keep that one container's process alive and reportable even if containerd or dockerd restarts — and the shim in turn calls runc, the low-level OCI runtime that does the actual work of creating namespaces and cgroups and executing the container's process inside them, per the OCI runtime-spec Docker helped write.

That last branch in the diagram matters more than it looks: Kubernetes never actually needed Docker Engine, only a compatible way to run OCI images. From 2016 (Kubernetes 1.5) the Container Runtime Interface (CRI) let kubelet talk to any compliant runtime, and Kubernetes shipped a translation shim, "dockershim," so it could keep talking to full Docker Engine installs through that interface. That shim was deprecated starting in Kubernetes 1.20 and removed outright in Kubernetes 1.24 (2022) — headlines at the time read as "Kubernetes drops Docker support," which overstated it: what was dropped was Docker Engine as the node-level runtime, not the image format. A container built with docker build is an OCI image either way, and it runs identically under containerd or CRI-O, which is exactly what most Kubernetes nodes use directly today.

Images and layers: how the build cache actually invalidates

☺ Like you're 10: Docker decides whether to reuse an old sheet or redraw a new one by checking, in order, every sheet before it — one different sheet and every sheet after it has to be redrawn too.

Every instruction that changes the filesystem — RUN, COPY, ADD — produces a new, immutable, content-addressed layer: a tar diff identified by a SHA-256 digest. Instructions like ENV, LABEL, and CMD only change image metadata and don't add a filesystem layer at all. At run time, a union filesystem — overlay2 on Linux — stacks every read-only image layer plus one thin, writable "container layer" on top, and presents the result as one merged filesystem. That writable layer is where anything the running container writes actually lands; it's discarded (unless committed) the moment the container is removed, which is why containers are meant to be disposable and any state worth keeping belongs in a volume.

Docker's classic build cache decides whether to reuse or rebuild each layer by checking, per instruction, whether the parent layer's digest and the instruction text (with any build args substituted) match a cached result exactly — and for COPY/ADD specifically, whether the checksum of the files being copied also matches. The moment one instruction misses the cache, every instruction after it misses too, because each layer's cache key includes its parent's digest as an input. That single fact is the entire reason instruction order in a Dockerfile matters as much as it does: copying a dependency manifest and installing dependencies before copying the rest of the source means an application-code change only invalidates the last couple of layers, not the whole build.

BuildKit — the build engine Docker Engine has used by default since version 23.0 (2023), and what docker buildx build always uses — improves on that model rather than replacing it: it executes independent stages of a multi-stage build in parallel, skips stages nothing in the final image depends on, and supports importing and exporting the build cache to a remote registry (--cache-from / --cache-to), which is what makes fast incremental builds possible on stateless, ephemeral CI runners that start with no local cache at all. BuildKit also introduces cache mounts (--mount=type=cache) — a directory, like a package manager's download cache, that persists across builds in BuildKit's own cache store without ever being committed into an image layer.

Multi-architecture images extend the same layer model one level up: a docker buildx build --platform linux/amd64,linux/arm64 produces one image per architecture, then publishes them under a single tag as an OCI image index (a "manifest list") — the client pulling myimage:1.0 automatically gets the manifest that matches its own CPU architecture, with no separate tag to remember per platform.

Dockerfile best practices: multi-stage builds and fewer layers

☺ Like you're 10: Build with the messy toolbox in one room, then carry only the finished part into a clean room for the final photo — and clean up a mess in the same step you made it, not later.

A multi-stage build uses more than one FROM in a single Dockerfile, each starting a new, independent stage, so a heavy build toolchain — compilers, package managers, source code — never has to ship in the image that actually runs in production. Only files explicitly copied forward with COPY --from= survive into the next stage.

# Dockerfile — multi-stage build
FROM golang:1.22-alpine AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download                    # cached separately from source — see the caching section above
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o /out/checkout ./cmd/checkout

FROM gcr.io/distroless/static-debian12:nonroot AS final
COPY --from=builder /out/checkout /checkout
USER nonroot:nonroot                   # never run as root by default
ENTRYPOINT ["/checkout"]

The final stage here has no shell, no package manager, and no compiler in it at all — just the statically linked binary and a non-root user, which shrinks both the image (megabytes instead of the builder stage's several hundred) and the attack surface a compromised process could exploit. This is the same principle covered generally in Immutable Infrastructure & Golden Images: ship exactly what's needed to run, nothing that was only needed to build.

Minimizing layer count matters for a subtler reason than tidiness: layers are purely additive. A file removed in a later layer isn't actually gone from the image — the image still contains the earlier layer with the file in it, plus a "whiteout" marker in the later layer that hides it from the merged view at run time. Both layers still get pulled, pushed, and stored. The fix is to install, use, and clean up within the same RUN instruction:

# Bad — three layers; the apt cache from layer 1 is still in the image after layer 3 "deletes" it
RUN apt-get update
RUN apt-get install -y curl
RUN rm -rf /var/lib/apt/lists/*

# Good — one layer; the cache never becomes a permanent part of any layer at all
RUN apt-get update && apt-get install -y --no-install-recommends curl \
    && rm -rf /var/lib/apt/lists/*

A handful of smaller habits round out a Dockerfile that survives review: keep a .dockerignore next to it so the build context — everything sent to the daemon before the first instruction even runs — doesn't include .git, node_modules, or a stray .env file; pin base images to a specific tag, or better, a digest (node:20.11.1-slim@sha256:...), since an unpinned :latest can silently change what a "reproducible" build actually produces from one day to the next; and prefer COPY over ADD unless you specifically need ADD's automatic tar-extraction or its ability to fetch a remote URL, both of which are easy to trigger by accident.

# .dockerignore — keeps the build context small and keeps secrets out of it entirely
.git
node_modules
dist/
*.md
.env
Dockerfile
.dockerignore

Day-to-day commands

☺ Like you're 10: A handful of verbs cover almost everything: build it, look at it, run it, watch it, ship it, clean it up.

# build and inspect
$ docker build -t registry.internal/checkout:1.4.2 .
$ docker buildx build --platform linux/amd64,linux/arm64 \
    --cache-from type=registry,ref=registry.internal/checkout:buildcache \
    --cache-to type=registry,ref=registry.internal/checkout:buildcache,mode=max \
    -t registry.internal/checkout:1.4.2 --push .
$ docker history registry.internal/checkout:1.4.2       # every layer, in build order, with its size
$ docker inspect registry.internal/checkout:1.4.2        # full image config as JSON

# run and interact
$ docker run -d --name checkout -p 3000:3000 --restart unless-stopped registry.internal/checkout:1.4.2
$ docker exec -it checkout sh                             # a shell inside a running container
$ docker logs -f checkout                                 # follow its stdout/stderr
$ docker ps                                                # what's running right now
$ docker ps -a                                             # including stopped containers

# ship
$ docker tag registry.internal/checkout:1.4.2 registry.internal/checkout:latest
$ docker push registry.internal/checkout:1.4.2
$ docker pull registry.internal/checkout:1.4.2

# clean up
$ docker system df                                         # how much disk images/containers/cache are using
$ docker image prune                                       # remove dangling (untagged) images only
$ docker system prune -a --volumes                         # remove everything unused — read the warning first

# local multi-service dev — see The Inner Loop & Developer Experience
$ docker compose up -d
$ docker compose logs -f api

Gotchas and failure modes

☺ Like you're 10: A few habits that look harmless on day one — mounting the wrong socket, never cleaning up, trusting a tag instead of a specific version — turn into real trouble by month three.

Disk fills up quietly. Every build leaves behind intermediate layers, and every stopped-but-not-removed container, dangling image, and unreferenced build-cache entry sits on disk until something prunes it. docker system df shows where the space actually went; docker system prune and BuildKit's own cache garbage collection (configurable via buildkitd.toml) are the fix — CI runners with no scheduled prune step are the most common place this bites, because they build constantly and nobody's watching disk usage until a build fails with "no space left on device."

An unpinned base tag makes "reproducible" builds lie. FROM node:20 can resolve to a different actual image next month as upstream ships patch releases under that same tag, so a rebuild from identical source code can produce a genuinely different image — different base-layer CVEs, different behavior — without a single line of your own Dockerfile changing. Pin to a digest for anything that needs to be reproducible on purpose, and re-pin deliberately, on a schedule, rather than floating forever.

The container runtime landscape: Docker Engine vs. the alternatives

☺ Like you're 10: Other tools can build or run the exact same kind of box Docker builds — they just trade away different pieces of Docker's convenience for something else they care about more.

Because the image format is an open OCI standard, "which tool built this image" stopped being a lock-in question years ago. The real question is which tool you want doing the building and running day to day, and that comes down to a genuine trade between convenience and what each option gives up.

The practical rule most teams land on: Docker Engine for local development and CI, because it's the one tool that both builds and runs with the least ceremony; containerd or CRI-O directly on Kubernetes nodes, because that's what kubelet already talks to and Docker Engine there would just be an unused extra hop; and Podman specifically when a root daemon is the objection someone can't get past any other way. None of that changes what a Dockerfile looks like or how the build cache behaves — the concepts on this page travel with the image regardless of which of these actually runs it.

Put the build-stage habits from this page into practice in Capstone Part 1 — Pipeline Foundation, then go find and fix a scanning gap in Drill — Secure a Vulnerable Pipeline. If you're weighing a vendor credential on top of this material, Mirantis's Docker Certified Associate exam covers this exact ground — but Mirantis discontinued new DCA registrations some time ago, so confirm current availability on Mirantis's own site before planning around it rather than trusting a fixed date printed here.