Release Engineering, Images & Artifacts

Between “the code compiles” and “it’s serving real users” sits a whole discipline most tutorials skip: how you turn source into an artifact, pack that artifact into an image, store it in a registry, give it a name you can trust, promote the very same bits from dev to prod, and prove — cryptographically — that what’s running is exactly what you built. That is release engineering. This page goes deep on the mechanics: what a container image really is at the byte level, how to build one well (and how to build one without a Docker daemon), how to make it tiny and hard to attack, where OCI registries hold far more than images now, how versioning and promotion actually work, how to separate deploying code from releasing it, and how to sign the whole supply chain so nothing unverified ever runs.

Container images from first principles

☺ Like you’re 10: A container image isn’t one magic file — it’s a stack of transparent sheets. Each sheet adds a few files, and stacking them makes the whole picture.

Before you can build images well, you have to know what one is. Strip away the branding and a container image is a boringly simple thing: an ordered stack of filesystem layers plus a small config that says how to run them, all tied together by a manifest and addressed by cryptographic hashes. The Open Container Initiative (OCI) standardised this format, which is why an image built by Docker runs under containerd, CRI-O, Podman, or any conformant runtime — the on-disk shape is a spec, not a vendor’s secret.

Layers & the union filesystem

Each layer is a tarball of filesystem changes — “these files were added, these modified, these whitespace-marked as deleted” — relative to the layer beneath it. At runtime the container engine stacks them into a single view using a union filesystem (typically overlayfs on Linux): the image’s layers become read-only lower directories, and the engine adds one thin writable layer on top for the running container. When your process reads a file, the union picks the topmost layer that has it; when it writes, the engine does copy-on-write — copies the file up into the writable layer and edits the copy, leaving the shared image layers untouched.

Two consequences fall out of this design and they shape everything below. First, layers are shared and deduplicated: if ten images all sit on the same debian-slim base layer, the node stores and pulls that layer once. Second, the writable layer is ephemeral — delete the container and everything written there is gone, which is exactly why persistent data belongs in a volume, never in the container’s own filesystem.

The image manifest & config

Two small JSON documents tie the stack together. The manifest lists the image’s config descriptor and its ordered layers, each referenced by mediaType, size, and — crucially — a content digest (a sha256 hash of the blob). The config object holds the runtime instructions the OCI runtime needs: the Entrypoint and Cmd, environment variables, the User to run as, the working directory, exposed ports, and a rootfs section listing each layer’s diff_id plus a history of how the image was built. There’s also a manifest list (a.k.a. image index) — a manifest of manifests — which is how a single tag like python:3.12 can transparently serve an amd64 image to your laptop and an arm64 image to a Graviton node.

# peek at the layers and config that actually make up an image
$ crane manifest ghcr.io/acme/checkout:1.4.3 | jq '{config: .config.digest, layers: [.layers[].digest]}'
{
  "config": "sha256:1b8e…c07",
  "layers": [
    "sha256:2d4f…a91",     # base layer — shared across many images
    "sha256:9a1c…5be"      # your app-binary layer — unique to this build
  ]
}

# the image config carries how to RUN it (entrypoint, user, env)
$ crane config ghcr.io/acme/checkout:1.4.3 | jq '.config | {User, Entrypoint, Env}'

Tags vs digests & immutability

Here is the distinction that quietly causes more production incidents than any other on this page. A tag (checkout:1.4.3) is a mutable, human-friendly pointer — a sticky note on a manifest. Nothing stops someone from re-pushing a different image to the same tag tomorrow, and :latest does this by design. A digest (checkout@sha256:5e0b…) is the immutable, content-derived name of an exact manifest; change one byte anywhere in the image and the digest changes too. Pull by tag and you get “whatever that note points at right now”; pull by digest and you get the image, byte-for-byte, forever — or an error if it no longer exists.

Dockerfile craftsmanship

☺ Like you’re 10: A Dockerfile is a recipe. Write the steps in a smart order and you reuse most of yesterday’s cooking; write them badly and you re-bake the whole cake every time you change the sprinkles.

Most images are still described by a Dockerfile — a sequence of instructions, each of which produces a layer. Writing one that works is easy; writing one that is small, fast to rebuild, secure, and reproducible is the craft. Four ideas carry almost all the value.

Multi-stage builds

The single most important technique is the multi-stage build: use one stage with the full toolchain to compile, and a separate, minimal stage that contains only the finished artifact. Your compiler, build caches, package managers, and source tree stay in the throwaway build stage and never reach production. The result is dramatic — a Go service can drop from a ~900 MB image (with the whole Go toolchain inside) to well under 20 MB that contains just the static binary and a handful of certificates.

# syntax=docker/dockerfile:1

# ---- build stage: full toolchain, never shipped ----
FROM golang:1.22 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download                       # cached until go.mod / go.sum change
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/checkout ./cmd/checkout

# ---- runtime stage: distroless, non-root, tiny ----
FROM gcr.io/distroless/static:nonroot
COPY --from=build /out/checkout /checkout
USER 65532:65532                          # nonroot — never uid 0
EXPOSE 8080
ENTRYPOINT ["/checkout"]

Layer caching & instruction ordering

Each instruction is a cache entry keyed on the instruction text and its inputs. The builder reuses a cached layer until something changes; then that layer and every layer after it are rebuilt. The rule that follows is order from least- to most-frequently-changing. Copy your dependency manifests (go.mod, package.json, requirements.txt) and install dependencies before copying your source, because your source changes on every commit but your dependency list rarely does — so the expensive install step stays cached across hundreds of code changes. BuildKit sharpens this further with --mount=type=cache, which persists a package or compiler cache across builds without baking it into a layer.

Small base images, non-root & least privilege

Two defaults quietly determine your image’s attack surface. The base image decides how much OS you carry: a full ubuntu ships hundreds of packages (each a potential CVE); debian-slim or alpine trim that; distroless or scratch carry almost nothing (covered in the next section). The user decides blast radius: images default to running as root (uid 0), so a container escape starts with root. Add a USER that isn’t 0, and pair it with a Pod securityContext that sets runAsNonRoot: true, drops all Linux capabilities, and mounts the root filesystem read-only. The image and the platform then reinforce each other: the image ships non-root, and the cluster refuses to run it any other way.

Reproducibility

A reproducible build produces the identical image (same digest) from the same source — which is what lets a signature actually mean something. The enemies of reproducibility are hidden non-determinism: unpinned base tags (FROM node:latest drifts daily), unpinned dependencies (apt-get install curl grabs whatever version is current), embedded build timestamps, and network fetches whose result changes over time. The fixes are mechanical: pin the base by digest, pin package versions and use lockfiles, set SOURCE_DATE_EPOCH so timestamps are stable, and prefer builders (Buildpacks, ko, Jib, Nix) that are reproducible by construction. You don’t need bit-for-bit reproducibility to ship — but the closer you get, the more confidently you can say “the thing I signed is the thing running.”

Daemonless & higher-level builders

☺ Like you’re 10: The old way needs a big powerful robot (the Docker daemon) that basically has the keys to the whole house. Newer builders do the same job without borrowing the keys — safer, especially inside a shared cluster.

Classic docker build talks to the Docker daemon, a long-running root process. Doing that inside CI usually means either mounting the host’s /var/run/docker.sock or running Docker-in-Docker in a privileged container — and both effectively hand root-on-the-node to whatever your pipeline runs. On a shared build cluster that is a serious escalation path. A whole family of builders exists to remove the daemon, and another family removes the Dockerfile entirely.

BuildKit & Kaniko — building in the cluster

BuildKit is the modern build engine (now the default behind docker build): it builds a dependency graph of the Dockerfile, runs independent steps concurrently, has first-class cache mounts and remote cache import/export, and can run rootless. Kaniko (from Google) and Buildah take a complementary angle: they execute a Dockerfile in userspace, inside an ordinary unprivileged container, and push the result straight to a registry — no daemon, no privileged socket. That makes them the natural fit for building images as a Kubernetes Job on the same cluster you deploy to, without punching a hole in its security model.

# Kaniko — build inside the cluster from a Job, no Docker socket, no privilege
executor --dockerfile=Dockerfile \
         --context=git://github.com/acme/checkout.git \
         --destination=ghcr.io/acme/checkout:1.4.3 \
         --cache=true

# BuildKit (buildctl) — rootless, with an exported layer cache
buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=. \
  --output type=image,name=ghcr.io/acme/checkout:1.4.3,push=true \
  --export-cache type=registry,ref=ghcr.io/acme/checkout:buildcache

Cloud Native Buildpacks — no Dockerfile at all

Cloud Native Buildpacks (a CNCF project, with Paketo and Google as major implementations) delete the Dockerfile from the developer’s life. You point the pack CLI (or kpack, its in-cluster controller) at your source; a detect phase figures out the language and framework, a build phase assembles an optimised, layered OCI image, and you never write build instructions. The killer feature is rebase: because Buildpacks keep the OS/base in distinct layers from your app, when a base image CVE is fixed the platform can swap the base layer across thousands of images without rebuilding the apps. For a platform team, that turns “patch every service’s base image” from a fleet-wide code change into a metadata operation.

Language-native builders — ko & Jib

For single-language shops, language-native builders are often the sweet spot. ko builds and pushes a container image straight from Go source — no Dockerfile, no daemon — defaulting to a distroless base and producing reproducible images with an SBOM attached; it’s a one-liner and integrates cleanly with Kubernetes manifests. Jib (a Maven/Gradle plugin from Google) does the same for Java: it separates dependencies, resources, and your classes into distinct layers for excellent caching, builds without a Docker daemon, and pushes directly to a registry from your normal build. Both trade generality for a frictionless, secure-by-default experience in their language.

# Buildpacks — from source to image, no Dockerfile
$ pack build ghcr.io/acme/checkout:1.4.3 --builder paketobuildpacks/builder-jammy-base

# ko — build & push a Go image, no Dockerfile, no daemon, distroless base
$ KO_DOCKER_REPO=ghcr.io/acme ko build ./cmd/checkout

# Jib — build & push a Java image from the normal build, no daemon
$ ./gradlew jib --image=ghcr.io/acme/checkout:1.4.3

Minimal & secure images

☺ Like you’re 10: The fewer things you pack in the box, the fewer things can break or be tampered with. A box with just your toy is safer than one stuffed with tools a thief could use.

Every file in an image is attack surface and a potential CVE. A shell, a package manager, curl, and a full libc are convenient — and they are exactly what an attacker uses to pivot after a break-in. The modern discipline is to ship the least that will run your app, so that both the vulnerability count and the post-exploit toolkit shrink toward zero.

distroless & scratch

scratch is the empty base: nothing at all. It’s perfect for a fully static binary (a Go or Rust program with no dynamic dependencies) — the image is only your binary. Distroless (Google’s images) is one careful step up: it contains your app’s runtime dependencies (glibc, CA certificates, timezone data) but no shell and no package manager. That “no shell” property is a genuine security control — an attacker who lands remote code execution can’t just drop into sh, and many exploit chains simply stall. The trade-off is debugging: you can’t kubectl exec into a shell that doesn’t exist, so you lean on ephemeral debug containers and good observability instead.

Chainguard & Wolfi

Wolfi is a Linux “undistro” designed specifically for containers: no kernel of its own, a fully declared build graph, and packages built to be minimal and reproducible. Chainguard Images are built on Wolfi and aim for a striking goal — zero known CVEs at publish time — with each image shipping a signature and an SBOM out of the box, and rebuilt continuously as fixes land. Where distroless gives you “no shell,” Chainguard/Wolfi give you “minimal and continuously patched and signed,” which is why they’ve become a popular hardened base for regulated platforms. Most publish both a runtime variant and a -dev variant (with a shell/apk) so you can build with the tools and ship without them.

Smaller surface = fewer CVEs

This isn’t a vibe; it’s arithmetic. A scanner (Trivy, Grype) reports vulnerabilities against the packages present. Remove the packages and you remove their CVEs — a scratch or distroless image can have single-digit or zero findings where the same app on a full base shows dozens, most in software your service never even calls. Smaller images also pull faster (less to move, better layer sharing) and start faster, so the security win comes with a performance and cost win. Feed those slimmer images into your scanning and policy gates and the whole pipeline gets quieter and greener.

Registries & OCI artifacts

☺ Like you’re 10: A registry is the warehouse where wrapped boxes live. And it turns out the same warehouse can store more than toy boxes — it can hold the instruction sheets, the receipts, and the safety seals too.

A registry is the content-addressed store your images live in — Docker Hub, GitHub Container Registry, the cloud providers’ registries, and self-hosted options. For a platform team, an internal registry isn’t optional plumbing; it’s a control point where you enforce access, scanning, retention, and provenance.

Harbor, registries & pull-through caches

Harbor (CNCF-graduated) is the reference open-source registry for platforms: projects with RBAC, built-in vulnerability scanning (Trivy), signing/verification, quotas, replication between registries, and tag immutability and retention rules. A key feature is the proxy cache (pull-through cache): point it at an upstream like Docker Hub, and the first pull fetches and caches the image while subsequent pulls serve locally. That kills Docker Hub rate-limit failures in your builds, speeds pulls, and means an upstream outage doesn’t stop your deploys — the cluster pulls the cached copy.

Retention & garbage collection

Registries grow without bound if you let them — every CI run can push an image, and untagged manifests pile up behind moved tags. Two mechanisms keep it sane. Retention policies decide which tags to keep (“keep the last 10 per repo, plus anything tagged release-*, plus whatever’s deployed”) and which to expire. Garbage collection then reclaims the storage of blobs no manifest references anymore. The subtlety worth internalising: because layers are shared, GC must be careful — a blob is only safe to delete when no manifest points at it, which is why registries run GC as a deliberate, often read-only-windowed operation rather than a casual cron.

OCI artifacts beyond images

Here’s the shift that reframes the registry. The OCI spec generalised “manifest + config + blobs” so a registry can store any content type, not just runnable images — these are OCI artifacts. Today your registry is a universal artifact store: Helm charts push and pull over oci://; SBOMs, signatures, and provenance attestations live as artifacts linked to an image via the Referrers API (so “what’s attached to this digest?” is a single query); WASM modules ship as OCI artifacts for edge runtimes; and increasingly ML models and datasets are distributed the same way. One store, one access model, one set of retention and signing rules — for config, code, and everything about them.

# the registry is a universal artifact store, not just an image host
$ helm push checkout-1.4.3.tgz oci://ghcr.io/acme/charts    # a Helm chart, as an OCI artifact

# an SBOM and a signature attached to an image digest, discoverable via the Referrers API
$ oras discover ghcr.io/acme/checkout@sha256:5e0b…
  └── application/spdx+json          sha256:aa11…    # the SBOM
  └── application/vnd.dev.cosign…    sha256:bb22…    # the signature

Versioning & promotion

☺ Like you’re 10: Build the toy once, then carry that exact toy from your room to the classroom to the science fair. Don’t rebuild a “new” toy at each stop and hope it’s the same.

An image needs a name humans can reason about and a promotion path that guarantees the bits you tested are the bits you ship. Get versioning and promotion right and rollbacks become trivial and audits become boring — which is exactly what you want.

Semantic versioning & the tag contract

Semantic Versioning — MAJOR.MINOR.PATCH — is a contract with your consumers: bump MAJOR for a breaking change, MINOR for backwards-compatible features, PATCH for backwards-compatible fixes. It matters most for things others depend on (base images, libraries, platform CRDs) so a consumer can pin 1.x and safely receive patches. Many teams add build metadata — a Git SHA or a date — so a tag like 1.4.3-a1b2c3d ties an image to the exact commit that produced it. But remember the earlier lesson: even a perfectly semantic tag is still a mutable pointer. The version is for humans; the digest is the truth.

Digest-pinned, immutable deploys

Deploying by digest closes the loop between “what we approved” and “what runs.” When your Deployment references checkout@sha256:5e0b… rather than checkout:1.4.3, three good things follow: the image can’t change under you, a rollback is a precise revert to a known digest, and your admission policy can require digests (rejecting any floating tag) and verify a signature on that exact digest. The ergonomics — nobody wants to type hashes — are solved by tooling: your CI resolves the tag to a digest and writes it into the manifest, so humans review a readable diff (“bump to 1.4.3”) while the cluster gets an unforgeable digest.

# resolve a human tag to its immutable digest, then deploy the digest
$ crane digest ghcr.io/acme/checkout:1.4.3
sha256:5e0b3c…f92

# what the running manifest actually pins — the tag comment is just for humans
image: ghcr.io/acme/checkout@sha256:5e0b3c…f92   # 1.4.3

Build once, promote many

The cardinal rule of release engineering: build the artifact exactly once, then promote that same artifact across environments. The anti-pattern is rebuild-per-environment — running a fresh build for dev, another for staging, another for prod. Even with identical source, separate builds can pull a newer transitive dependency, a patched base layer, or a different toolchain, so the prod image is subtly not the thing you tested. Build-once means promotion is just moving a digest forward: a small, auditable change to the next environment’s GitOps config, gated by whatever tests and approvals you require. Same bits, dev to prod — no surprises, and a rollback is “point the environment back at the previous digest.”

Release strategies & feature management

☺ Like you’re 10: There’s a difference between bringing a new game to the party (deploying) and letting everyone play it (releasing). You can quietly put the game in the cupboard first, then open it to a few friends, then to everyone.

Getting bits onto servers is only half of shipping. The other half is exposing them to users without a scary big-bang moment. This is where Benny hands off to Pip: Benny’s digest is built, signed, and promoted — now Pip shifts real traffic onto it a sip at a time, and feature flags let you separate “it’s deployed” from “it’s on.”

Recap: canary & blue-green

Two progressive-delivery patterns (covered in depth in CI/CD & Progressive Delivery) do the traffic side. Blue-green runs two complete environments — the live “blue” and the new “green” — and cuts traffic over all at once when green looks healthy, with instant rollback by cutting back. Canary shifts a small percentage of traffic to the new version, watches the golden signals, and ramps up (or rolls back automatically) based on what it sees. Both are about limiting the blast radius of a bad release — Pip’s whole reason for existing — and both operate on the artifact side of the story: the image is already built and promoted; these decide who reaches it.

Decoupling deploy from release — feature flags

The most powerful idea in this section: deploying code and releasing a feature are different events, and you should be able to do one without the other. A feature flag is a runtime switch that gates a code path, so you can deploy new code “dark” (present but off), then turn it on for internal users, then 1%, then everyone — with no redeploy. That decoupling is transformative: it lets you merge to main continuously (trunk-based development), ship the image once, and control exposure as a config change measured in seconds, not a build-and-deploy cycle measured in minutes.

OpenFeature (a CNCF project) standardises this: a vendor-neutral SDK API plus a provider model, so your code calls one getBooleanValue("new-checkout-flow", …) and the provider behind it can be the open-source flagd, or a SaaS like LaunchDarkly or Flagsmith, without rewriting your app. You avoid lock-in at the API boundary and let the platform choose (or swap) the flag backend.

// flagd — an OpenFeature provider evaluates this flag definition.
// "deployed" was Benny's job; this dial is the actual "release".
{
  "flags": {
    "new-checkout-flow": {
      "state": "ENABLED",
      "variants": { "on": true, "off": false },
      "defaultVariant": "off",
      "targeting": {
        "fractional": [
          { "var": "sessionId" },
          [ "on", 5 ],          // 5% of sessions get the new flow…
          [ "off", 95 ]         // …95% stay on the old one
        ]
      }
    }
  }
}

Kill switches, progressive rollout & experimentation

Once exposure is a dial, three capabilities come almost for free. A kill switch is a flag you flip off to instantly disable a misbehaving feature — a rollback measured in seconds and without touching the deployment, which is a far calmer 2 a.m. than reverting an image. Progressive rollout is ramping a flag’s percentage up while watching metrics — the same shape as a canary, but at the feature granularity rather than the whole service. And experimentation (A/B testing) uses flag variants to serve different cohorts and compare outcomes, so a “release” becomes a measured hypothesis rather than a leap of faith. The through-line: the artifact is immutable and already deployed; the flag decides who experiences what, when.

Securing the build & supply chain

☺ Like you’re 10: Put a list of every ingredient on the box (so you can check for anything nasty), then add a tamper-proof seal signed by the real baker, and have the door only open boxes with a matching seal.

The last mile of release engineering is trust: proving that the artifact was built from the source you think, by the pipeline you think, and hasn’t been swapped since. The security page covers admission and policy in full; here we focus on the build-time and artifact-time controls that make that policy possible. The mantra is sign at build, verify at admission — and nothing unsigned runs.

SBOMs — knowing what’s inside

A Software Bill of Materials is a machine-readable inventory of every component in your image — packages, versions, licences — in a standard format (SPDX or CycloneDX). Syft generates one from an image or filesystem; you then attach it to the image as an OCI artifact. The payoff is answering the question every team dreads during a zero-day: “are we affected by CVE-X?” Instead of frantically rebuilding and rescanning, you query the SBOMs you already published — grype sbom:./sbom.json — and know in seconds which images across the fleet contain the vulnerable version. No SBOM, and that question takes days.

Signing & provenance — cosign & SLSA

cosign (from the Sigstore project) signs images and artifacts by digest. Its modern mode is keyless: instead of a long-lived private key you have to guard, cosign gets a short-lived certificate from Fulcio tied to your CI’s OIDC identity (e.g. “this GitHub Actions workflow in the acme org”), signs, and records the signature in the Rekor transparency log. Verification then checks not just “is this signed?” but “was it signed by the identity we expect?” On top of signatures, SLSA (Supply-chain Levels for Software Artifacts) defines provenance — a signed statement of how and where an artifact was built (the builder, the source commit, the parameters) — so you can require that images were produced by your trusted pipeline and not on someone’s laptop. Signatures, SBOMs, and provenance are all stored as attestations (in-toto statements) linked to the image digest.

# 1) inventory what's inside — generate and attach an SBOM
$ syft ghcr.io/acme/checkout@sha256:5e0b… -o spdx-json > sbom.spdx.json

# 2) sign the image BY DIGEST — keyless: identity from CI's OIDC, logged in Rekor
$ cosign sign ghcr.io/acme/checkout@sha256:5e0b…

# 3) attach the SBOM as a signed attestation on that digest
$ cosign attest --predicate sbom.spdx.json --type spdxjson \
    ghcr.io/acme/checkout@sha256:5e0b…

# 4) verify the signer identity — this is what admission runs before letting it run
$ cosign verify \
    --certificate-identity-regexp '.*@acme\.com$' \
    --certificate-oidc-issuer https://token.actions.githubusercontent.com \
    ghcr.io/acme/checkout@sha256:5e0b…

Attestations verified at admission

Signing is only useful if something checks it before code runs. That check happens at the cluster door: an admission controller (Sigstore’s policy-controller, or Kyverno / OPA Gatekeeper) intercepts every Pod and refuses to admit an image unless it satisfies policy — signed by the expected identity, carrying valid SLSA provenance, referencing an allowed registry, and pinned by digest. This is the enforcement end of everything above: digests make identity unforgeable, cosign proves the signer, SLSA proves the builder, and admission turns those proofs into a hard gate. An unsigned image, or one built off your golden path, simply never schedules.

Release engineering is the quiet spine of the whole platform: it turns source into a small, signed, immutable artifact, stores it once, promotes those exact bits forward, and proves at the door that nothing was swapped. Master it and the layers above — GitOps reconciliation, progressive delivery, and self-service golden paths — inherit a foundation of trust. Skip it and you inherit mystery images, drifting tags, and unanswerable questions at the worst possible moment. Next, watch Pip take these artifacts the last mile in CI/CD & Progressive Delivery, and see how the door is guarded in Security & Policy.