Immutable Infrastructure & Golden Images

Every server or container running your code eventually needs to change — a security patch, a new library, a config tweak — and there are exactly two disciplined ways to make that change stick. One is to reach into the running thing and edit it. The other is to build a brand-new one from a known-good recipe and throw the old one away. This page is about the second approach: baking a golden image once with a tool like Packer, then shipping changes by replacing running servers or containers wholesale instead of patching them where they stand. It closes with the part people usually get wrong — why none of this makes configuration management obsolete. It just moves where, and how often, that discipline runs.

Two philosophies for keeping a fleet correct

☺ Like you're 10: Mutable means you fix the same server again and again for as long as it lives; immutable means a server is never fixed at all — it's replaced by a fresh one built from scratch.

A mutable server is provisioned once and then modified in place for the rest of its life: patched, upgraded, hotfixed, occasionally SSH'd into during an incident. Its identity persists across every one of those changes — it's still "web-04" a year later, just a year's worth of edits deep. This is the model most infrastructure ran on by default for decades, and it's the model configuration management was built to make safe: Ansible, Puppet, and Chef exist specifically to make those repeated in-place edits idempotent and declarative rather than ad hoc.

An immutable server is never modified after it boots. Once launched from an image, its filesystem is treated as read-only in spirit if not in enforcement — if something needs to change, you don't edit the running instance, you build a new image with the change baked in and replace the instance entirely. The term traces to a widely cited 2013 post by engineer Chad Fowler, "Trash Your Servers and Burn Your Code," and the pattern spread quickly through companies running large, homogeneous fleets — Netflix's early image-bake tooling (Aminator) is one of the better-known production examples — because it converts "is this specific server correct?" from an open question you have to go check into a closed one: it's correct if and only if it was launched from the golden image, full stop.

The community shorthand for this shift is pets vs. cattle: a pet server has a name, gets individually nursed back to health when it's sick, and is irreplaceable. A cattle server has a number, gets replaced without ceremony the moment something's wrong with it, and no one server is special. Immutable infrastructure is the cattle model taken to its logical conclusion — replacement isn't just what happens when a server is unhealthy, it's the only mechanism by which a healthy server's software ever changes at all.

Why patch-in-place decays: config drift and the snowflake server

☺ Like you're 10: Even a well-behaved patching tool only fixes the parts of a server it was told to watch — anything outside that list still quietly wanders off on its own.

Configuration management already covers config drift — the gap that opens between a host's declared state and its real state whenever something outside the config tool touches it. Applied to a long-lived, patched-in-place fleet, drift is not a rare failure, it's the default trajectory: every manual incident fix, every OS auto-update the config tool never declared an opinion on, every one-off yum install a debugging session left behind, nudges that specific host further from its siblings. Enough of that, on enough hosts, over enough years, and you get a snowflake server — one nobody fully trusts to describe, replace, or even reboot with confidence, because nobody actually knows everything currently keeping it alive.

Here's the sharper version of that problem: even a pull-based config agent reconciling every 30 minutes only closes the gap for whatever's in its manifest. A Puppet catalog that manages nginx's config file, the deploy user, and three packages has nothing to say about the kernel patch level, the stray debugging tool someone apt install'd last Tuesday, or the log file quietly filling a disk — none of that is drift from the tool's point of view, because it was never declared one way or the other. Config management corrects drift in the properties it was told to manage; it can't correct drift in the properties nobody thought to write down. Immutable infrastructure sidesteps the whole category: there is no "everything else," because nothing is ever modified after boot in the first place. You're not detecting and correcting drift faster — you're removing the mechanism by which drift could occur at all.

Baking the image: Packer and the image-as-artifact pipeline

☺ Like you're 10: Packer boots a throwaway practice server, runs your setup steps on it once, takes a snapshot of the result, and then throws the practice server away — the snapshot is the golden image.

Packer, from HashiCorp, is the tool most teams reach for to build golden images across cloud providers and container runtimes from one shared workflow. A Packer template — written in HCL2, the same configuration language family as Terraform — declares two things: a source (which builder plugin to use, and what to launch it from — amazon-ebs, googlecompute, azure-arm, docker, qemu) and a build block that lists provisioners to run against a temporary instance launched from that source. When you run packer build, Packer launches a throwaway builder instance from a base image, executes every provisioner against it in order, then snapshots the result into a new, versioned artifact — an AMI, a machine image, a container image — and tears the builder instance down. Nothing about that builder instance persists except the artifact it produced.

packer {
  required_plugins {
    amazon = { version = ">= 1.2.0", source = "github.com/hashicorp/amazon" }
  }
}

source "amazon-ebs" "web" {
  ami_name      = "web-golden-{{timestamp}}"
  instance_type = "t3.medium"
  region        = "us-east-1"
  source_ami_filter {
    filters     = { name = "al2023-ami-*-x86_64", virtualization-type = "hvm" }
    owners      = ["amazon"]
    most_recent = true
  }
  ssh_username = "ec2-user"
}

build {
  sources = ["source.amazon-ebs.web"]

  provisioner "shell" {
    inline = ["sudo dnf -y update", "sudo dnf -y install amazon-cloudwatch-agent"]
  }

  # the SAME playbook style covered in configuration-management.html — it just
  # runs once, against this one temporary builder instance, instead of forever
  # against a fleet of long-lived hosts.
  provisioner "ansible" {
    playbook_file = "../playbooks/webserver.yml"
  }

  post-processor "manifest" {
    output = "manifest.json"   # records the exact AMI ID this build produced
  }
}

Two habits separate a golden-image pipeline that's trustworthy from one that just looks like it is. First, validate before you publish: run packer validate for syntax, then test the baked artifact itself before anything launches from it in production — tools like Chef InSpec or Goss can assert "nginx is installed, listening on 443, and the deploy user exists" against the built image directly, the same kind of check Testing in the Pipeline runs against application code. Second, scan before you publish: a vulnerability scanner like Trivy or Grype run against the finished image, wired as a required CI step rather than an optional one, catches an OS-level CVE the same commit-gate way Shift-Left Security for DevOps catches an application dependency CVE — see also Supply-Chain Security & SBOM for generating a software bill of materials for the image itself. A golden image that skips both checks is just a mutable server's bugs, baked in and copied a thousand times.

The same discipline on containers: golden container images

☺ Like you're 10: A Dockerfile is Packer's idea in miniature — build once, tag the result so it can never quietly change, and never edit a running container, ever.

Container images are immutable infrastructure's other, arguably more common, substrate — and the discipline is the same one Packer applies to a VM. A multi-stage Dockerfile is the container-world equivalent of a Packer template: a build stage compiles or installs dependencies, a slim runtime stage copies over only what's needed to run, and the finished image is pushed to a registry as a single, content-addressed artifact — see containers & orchestration for the registry mechanics themselves.

# ---- build stage ----
FROM node:20.11.1-alpine@sha256:c37b9a4c1a2b3f4d5e6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# ---- runtime stage: the golden image ----
FROM node:20.11.1-alpine@sha256:c37b9a4c1a2b3f4d5e6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
USER node
CMD ["node", "dist/server.js"]

The @sha256:… digest pin, not just the 20.11.1-alpine tag, is what makes this genuinely immutable rather than immutable-in-name-only: a tag can be silently re-pushed to point at a different image later, but a digest is the content itself — the same digest always resolves to the exact same bytes, forever. Build & artifact management covers the general case of this rule; here it's the difference between a golden image and a moving target with a golden-sounding name.

Once built and pushed, that image is never mutated again. A container that needs a fix doesn't get docker exec'd into and patched live — that's the exact same anti-pattern as SSHing into a VM to hand-patch it, with an extra twist: the fix isn't just undocumented, it's not durable. The moment that container restarts, it starts fresh from the original image, and the live patch is gone as if it never happened. The correct fix is the same one covered above: change the Dockerfile, rebuild, get a new digest, and roll that new image out — the exact same digest gets promoted unchanged from dev through staging to production, which is the "build once, promote everywhere" principle CI/CD pipelines and deployment strategies both depend on.

Shipping the change: replace, don't patch

☺ Like you're 10: Instead of fixing the machines that are already running, you turn on new ones built from the new recipe, wait for them to prove they're healthy, and then turn the old ones off.

Baking a golden image is only half the pattern — the other half is how a fleet actually moves from running the old image to running the new one, and every mainstream mechanism does it the same conceptual way: launch new, verify healthy, retire old, never touch what's still running mid-flight.

On an AWS Auto Scaling Group, the mechanism is instance refresh: you publish a new AMI, point a new launch template version at it, and trigger a refresh with a min_healthy_percentage that bounds how much capacity can disappear at once. The ASG launches new instances from the new template, waits for them to pass health checks, terminates a batch of old ones, and repeats until every instance in the group is running the new image — no instance is ever edited, only replaced.

resource "aws_launch_template" "web" {
  name_prefix   = "web-"
  image_id      = data.aws_ami.golden.id   # points at the new AMI the Packer build published
  instance_type = "t3.medium"

  lifecycle {
    create_before_destroy = true           # the new template version exists before the old one retires
  }
}

resource "aws_autoscaling_group" "web" {
  # ... vpc_zone_identifier, min_size, max_size, etc.
  launch_template { id = aws_launch_template.web.id, version = "$Latest" }

  instance_refresh {
    strategy = "Rolling"
    preferences { min_healthy_percentage = 90 }   # never drop below 90% capacity mid-replace
  }
}

That Terraform snippet's create_before_destroy and the plan-then-apply workflow covered earlier in this course are the same idea applied to this exact change: a human reviews the plan — "replace launch template, trigger instance refresh across 12 instances" — before anything is torn down. On Kubernetes, the identical logic runs one layer up: changing a Deployment's Pod template image never patches a running Pod in place — the Deployment controller creates new Pods from the new spec and terminates old ones according to maxSurge/maxUnavailable, exactly the rolling-replace mechanics covered in containers & orchestration. Managed Kubernetes node pools push immutability down one more layer still: an EKS or GKE node-image upgrade cordons and drains old nodes and replaces them with nodes booted from a new node image, rather than SSH-ing in and running a package manager against a live node.

Where configuration management still fits

☺ Like you're 10: The same playbook still runs — it just runs once, on a practice server, before anything ships, instead of forever on servers people are actually using.

This is the point worth being precise about, because it's the most commonly missed one: immutable infrastructure changes when configuration is applied, not whether configuration management is still doing the work. Look back at the Packer template above — the ansible provisioner is running the exact same kind of idempotent, desired-state playbook that configuration management covers, written in the exact same declarative style ("ensure nginx is installed," not "install nginx"). Nothing about baking a golden image removes Ansible, Puppet, or Chef from the picture — it relocates their job from "reconcile 200 live production hosts, every 30 minutes, forever" to "reconcile one disposable builder instance, once, before anything ships." Same tool, same idioms, radically different blast radius and frequency.

What immutable infrastructure genuinely can't bake in ahead of time is anything only knowable at the moment a specific instance boots: which availability zone it landed in, which secrets-broker endpoint to talk to, which IAM role or Kubernetes service account it should assume. That's handled by a thin, deliberately minimal boot-time injection layer — cloud-init user-data, EC2 instance metadata, a Kubernetes ConfigMap or environment variable mounted at Pod start — that sets identity and non-secret, environment-specific parameters without touching the baked filesystem layers underneath. Crucially, that injection runs exactly once, at first boot, and never again; it is not a standing agent quietly re-applying itself against production traffic on its own schedule the way a pull-based CM agent does. For anything genuinely secret at that boot step, the identity-first pattern from Secrets & Credential Management — trust the platform-issued identity, fetch a short-lived credential on demand — is what replaces a standing bootstrap secret baked into the image itself.

The operational tax: image sprawl and rebuild cadence

☺ Like you're 10: A thousand identical copies of a server are only safe if the pattern they're copied from gets refreshed regularly — otherwise you've just made a thousand copies of the same mistake.

Immutable infrastructure isn't free, and the two costs that catch teams off guard both stem from the same root fact: a golden image is a full artifact, and artifacts pile up if nobody prunes them. Every AMI is a full EBS snapshot; every container image is a set of registry layers. Left unmanaged, a fleet accumulates hundreds of image versions nobody launches from anymore, quietly compounding storage spend — the same kind of unwatched-by-default cost FinOps for Delivery Pipelines covers for CI compute. The fix is a lifecycle policy that expires old, unreferenced versions automatically rather than by memory:

{
  "rules": [
    {
      "rulePriority": 1,
      "description": "Expire untagged images older than 14 days",
      "selection": {
        "tagStatus": "untagged",
        "countType": "sinceImagePushed",
        "countUnit": "days",
        "countNumber": 14
      },
      "action": { "type": "expire" }
    }
  ]
}

The second cost is subtler and matters more: a golden image freezes its OS packages, libraries, and base image at the moment it was baked. That's the whole point when it comes to a running fleet — every instance is bit-for-bit identical, and identical to something you can inspect and re-derive at will. But it means a golden image built 90 days ago is, silently, 90 days stale against every CVE published since, and nothing about the image itself will ever tell you that — it isn't drift from its own baked state, which never changes, but it is drift relative to the outside world. Unlike a pull-based CM agent that re-evaluates its manifest every 30 minutes and would at least have a chance to pick up a patched package, an unrebuilt golden image just sits there, correct by its own definition, quietly falling further behind.

The fix is a scheduled rebuild cadence: rerun the same Packer template against a fresh upstream base image on a fixed schedule — nightly or weekly, tightened for anything internet-facing — plus an out-of-band rebuild triggered the moment a high-severity CVE lands in something the image ships, using the same scanners (Trivy, Grype) from the bake pipeline run continuously against images already deployed, not just at build time. AWS's EC2 Image Builder is a managed alternative to running this cadence yourself, if you'd rather not operate the scheduling and pipeline plumbing directly. Either way, the rebuild cadence isn't optional hygiene sitting outside the pattern — it's the mechanism that makes immutability's core promise, consistency, actually worth having.

Immutable infrastructure isn't a rejection of everything configuration management and infrastructure as code already taught you — it's those same declarative, idempotent, plan-before-you-apply habits, aimed at a disposable builder instance instead of a fleet of long-lived ones. Practice the replace-not-patch mechanics hands-on in Drill — Write a Reusable IaC Module and Capstone Part 2 — Infrastructure as Code; Packer, Docker, Ansible, and Terraform cover the individual tools referenced throughout this page in more depth, and Resilient Cloud Solutions covers where a self-healing, replace-on-failure fleet fits into the broader resiliency picture.