Kustomize

Kustomize is configuration customization for Kubernetes with no templating language at all — it lives right inside kubectl, and every file you write is ordinary, valid, schema-checkable YAML. You keep one canonical set of manifests as a base, then declare small overlays that patch it per environment: six replicas and a real image tag in prod, one replica and debug logging in dev, nothing else different. It solves the specific mess that shows up the moment an app needs to run in more than one place — three folders of copy-pasted YAML that quietly stop matching each other — without asking you to learn Go templates or ship a package. It's built into every kubectl binary since 1.14, so if you've typed kubectl apply -k, you've already used it.

What Kustomize is, and the problem it solves

☺ Like you're 10: It keeps one copy of your YAML and lets you stick small "change just this bit" notes on top, one set of notes per environment.

Kustomize is maintained under kubernetes-sigs/kustomize and has shipped embedded inside kubectl since v1.14 — no separate install required to get started, though the standalone kustomize binary exists too and usually runs a newer version. Its whole design fits in one word from its own tagline: it is template-free. There is no {{ .Values.replicaCount }}, no rendering engine turning almost-YAML into real YAML, no step where a typo in a template produces something the API server rejects for reasons that don't point back at your mistake. Every input file is a complete, valid Kubernetes manifest — or close to it — and the one extra file Kustomize needs, kustomization.yaml, just says which files to load and what to change about them.

The problem it's solving is the one every team hits the moment an app runs anywhere besides a single cluster: the object model gives you one Deployment, one Service, one ConfigMap — but dev, staging, and prod need three copies of each that are 95% identical and 5% deliberately different. Copy-pasting three folders looks fine on day one. By the end of the quarter, prod has a security context dev never got, staging's liveness probe is stale, and nobody can say with confidence what the actual difference between any two environments is anymore, which is exactly the drift operating Kubernetes well is trying to prevent. Whatever fixes this has to keep the shared 95% living in exactly one place and make the 5% that's genuinely different easy to read at a glance.

Bases and overlays: the pattern that carries the whole tool

☺ Like you're 10: One folder holds the real drawing. One small folder per environment holds only the sticky notes for what's different there.

A base is a directory of ordinary manifests plus a kustomization.yaml that lists them. You could kubectl apply -f the base directly and it would work — that's the whole promise of template-free configuration, nothing in it is provisional or half-finished. An overlay is a second, much smaller directory whose own kustomization.yaml points back at the base as a resources entry and layers a handful of changes on top: a namespace, a name prefix, a replica count, a patch file or two. Promotion from staging to prod stops being "diff two nearly-identical folders and hope you spot everything" and becomes "diff one small overlay file" — reviewable by a human in well under a minute, which is exactly the property GitOps tooling like GitOps on Kubernetes depends on to make a pull request an honest description of what's about to change.

There's no controller running any of this and nothing installed into the cluster on Kustomize's own behalf. It defines exactly one API object that never gets applied anywhere — the Kustomization file kind itself (apiVersion: kustomize.config.k8s.io/v1beta1) — plus an optional Component kind (v1alpha1) for a reusable fragment of one. Kustomize is a pure function: directory in, YAML out, nothing left running afterward.

The kustomization.yaml fields you'll actually reach for

☺ Like you're 10: One small file lists what to pick up and what to change about it — running the build does the picking-up and the changing, in a fixed order every time.

A build runs in two phases, and the order matters for reasoning about what a given file will see. First, generators run — configMapGenerator and secretGenerator manufacture brand-new objects from literals and files. Then transformers run over everything, generated objects included, in a fixed sequence: patches first, then the naming and metadata transformers (namespace, namePrefix/nameSuffix, labels, commonAnnotations), then replicas and images, and finally a name-reference pass that walks the whole object graph and fixes up every place one object refers to another by name. That last pass is the quiet workhorse: rename a ConfigMap with namePrefix: prod- and it also rewrites the configMapKeyRef inside the Deployment consuming it — something a text-substitution templating engine has no way to do, because it has no idea the two files are related.

Put together, a base and a production overlay look like this — a Deployment, Service, and ServiceAccount in the base, and an overlay that namespaces everything, prefixes the names, scales it up, and patches in real resource limits:

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:                        # ordinary manifests, each valid on its own
  - deployment.yaml
  - service.yaml
  - serviceaccount.yaml

labels:                           # modern replacement for commonLabels
  - pairs:
      app.kubernetes.io/name: checkout
      app.kubernetes.io/part-of: storefront
    includeSelectors: false       # do NOT write these into the Deployment's selector

images:
  - name: checkout                # matches spec.containers[].image "checkout:..."
    newName: ghcr.io/acme/checkout
    newTag: 1.4.2                 # CI bumps this line, or uses `kustomize edit`

# overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: storefront-prod
namePrefix: prod-

resources:
  - ../../base                    # a relative path — or a Git URL pinned to a tag/SHA

replicas:
  - name: checkout                 # the base name, BEFORE namePrefix is applied
    count: 6

images:
  - name: ghcr.io/acme/checkout
    newTag: 1.4.2

patches:
  - path: resources-patch.yaml     # strategic merge; target inferred from the file itself

  - target:                        # JSON 6902, inline, matched by an explicit selector
      kind: Deployment
      labelSelector: app.kubernetes.io/part-of=storefront
    patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/imagePullPolicy
        value: IfNotPresent

Two patch dialects: strategic merge and JSON 6902

☺ Like you're 10: One patch style looks like the object with only the changed bits filled in. The other gives step-by-step surgical instructions instead.

A strategic merge patch is a partial Kubernetes object, written in the same shape as its target, with only the fields you want changed filled in. Kubernetes' merge-key rules do the rest — containers merge by name, ports merge by containerPort — so the patch file reads almost like documentation of the one thing that's different. It's the right default, and it's the one people find pleasant to review, because it looks exactly like the object it's touching, minus everything that isn't changing.

# overlays/prod/resources-patch.yaml — a strategic merge patch
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout                   # identifies the target — the pre-prefix name
spec:
  template:
    spec:
      containers:
        - name: checkout            # the merge key for the containers list
          resources:
            requests: { cpu: 500m, memory: 512Mi }
            limits:   { memory: 512Mi }

A JSON 6902 patch (RFC 6902) is a list of surgical operations instead — add, replace, remove, copy, move, test — each one addressed by a JSON Pointer path such as /spec/template/spec/containers/0/env/1. Reach for it when a strategic merge can't say what you mean: deleting an element outright, or writing to a precise index in a list. It's less readable at a glance but it's the only dialect that can remove something rather than just override it.

# a JSON 6902 patch, removing an env var and adding a toleration
- op: remove
  path: /spec/template/spec/containers/0/env/1
- op: add
  path: /spec/template/spec/tolerations
  value:
    - key: workload-class
      operator: Equal
      value: batch
      effect: NoSchedule

Generators: configMapGenerator and secretGenerator

☺ Like you're 10: These build a whole ConfigMap or Secret from a short list of settings, and quietly rename it every time the settings change.

configMapGenerator and secretGenerator manufacture entire objects from literals, a .env file, or referenced files, rather than requiring you to hand-write the ConfigMap or Secret YAML yourself. By default they append a hash of the generated content to the object's name — checkout-config becomes something like checkout-config-9fh72tk4bd — and that hash is doing real work, not decoration.

# base/kustomization.yaml (excerpt)
configMapGenerator:
  - name: checkout-config
    literals:
      - LOG_LEVEL=info
      - FEATURE_GIFTWRAP=false
    # → emits ConfigMap "checkout-config-9fh72tk4bd" (content-hash suffix)

secretGenerator:
  - name: checkout-db
    literals:
      - DB_USER=checkout_svc
    # → base64-encoded Secret; the VALUE still has to come from somewhere real, see below

Because the object's name changes whenever its content changes, and the name-reference transformer rewrites every consumer's reference to match, a Deployment that reads that ConfigMap via envFrom or a mounted volume gets a genuinely new pod template the moment the config changes — which triggers a real rolling update instead of a mutated ConfigMap sitting there unread until some unrelated restart happens to pick it up. This is the single cleverest thing Kustomize does with almost no code, and it's the exact mechanism behind the well-known "restart on ConfigMap change" trick that Helm charts have to fake by hand with a checksum annotation.

Day-to-day commands

☺ Like you're 10: A handful of commands cover almost everything: show me the result, show me what would change, then actually do it.

The single most important habit is to render before you apply. kubectl kustomize prints the finished YAML to stdout without touching the cluster at all, which turns "I hope my patch did what I meant" into "I can read the exact lines it produced."

# --- render and inspect: nothing touches the cluster ---
$ kubectl kustomize overlays/prod                    # print the built manifests
$ kustomize build overlays/prod                       # same, via the standalone binary (often newer)
$ kubectl kustomize overlays/prod | kubectl apply --dry-run=server -f -
$ kubectl diff -k overlays/prod                       # what WOULD change in the live cluster

# --- apply and remove ---
$ kubectl apply -k overlays/prod                      # build, then apply
$ kubectl delete -k overlays/prod                     # build, then delete those objects

# --- edit the kustomization file from a script or CI, no manual YAML surgery ---
$ cd overlays/prod
$ kustomize edit set image ghcr.io/acme/checkout=ghcr.io/acme/checkout:1.4.3
$ kustomize edit set replicas checkout=8
$ kustomize edit set namespace storefront-prod
$ kustomize edit add resource networkpolicy.yaml
$ kustomize edit fix                                  # migrate deprecated fields in place

# --- worth wiring into CI ---
$ diff <(kustomize build overlays/staging) <(kustomize build overlays/prod)
$ kubectl version --client                            # check the EMBEDDED kustomize version

In production you'll rarely run kustomize build by hand at all. An Argo CD Application whose source.path contains a kustomization.yaml is detected and rendered automatically by the repo-server, and Flux ships a controller named for the tool — kustomize-controller, reconciling a Kustomization custom resource that is not the same thing as the kustomization.yaml file, a naming collision that trips up nearly everyone the first time. GitOps on Kubernetes walks through that reconciliation loop end to end.

Gotchas and failure modes

☺ Like you're 10: A few things bite almost everyone once: a name that keeps changing, labels that break a running app, and patches that quietly do nothing.

The ConfigMap name hash has consequences beyond triggering a rollout. Old hashed ConfigMaps are not garbage-collected by Kustomize itself, so they accumulate in the cluster unless something else prunes them — a GitOps reconciler with pruning enabled, typically. Setting disableNameSuffixHash: true gets rid of the accumulation but brings back the exact stale-config problem the hash exists to solve. And an object that references the ConfigMap somewhere Kustomize's name-reference transformer can't see — buried in an annotation, or inside a custom resource it has no schema for — won't get rewritten, and will quietly point at a name that no longer exists.

commonLabels and immutable selectors is the classic outage. commonLabels writes its labels into spec.selector.matchLabels as well as into pod templates — but a Deployment's selector is immutable once created, so adding a common label to an app that's already running fails on apply with field is immutable, which shows up as an Argo CD sync that fails forever or a Flux reconciliation stuck in the same loop. The fix going forward is the newer labels field with includeSelectors: false for anything cosmetic. If you're already stuck, the only way out is deleting and recreating the Deployment — with the downtime that implies — which is exactly the kind of self-inflicted incident anti-patterns & pitfalls catalogues.

A patch that matches nothing can fail silently. A strategic-merge patch naming a resource that doesn't exist fails loudly, with no matches for Id .... But a patches entry whose target uses a labelSelector or annotationSelector that matches zero objects is, in the general case, a silent no-op: the build succeeds, the manifest comes out unchanged, and nobody notices until an environment quietly behaves like the one before it in the promotion chain. A golden-file test in CI — render the overlay and diff it against a committed expected output — catches this before a human has to.

Version skew is real and worth naming. The kustomize embedded in kubectl lags the standalone binary, sometimes by more than a year, and Argo CD and Flux each bundle their own version too. A build that renders cleanly on your laptop can fail in a pipeline running an older embedded version that predates the field you used. Pin deliberately where it matters, and be conservative about reaching for a feature that only recently stabilized. Ordering bites in a related way: because transformers run in a fixed sequence, a patch targeting a resource by name sees the pre-prefix name — which is why the earlier prod overlay example targets checkout, not prod-checkout, even though namePrefix is set in the very same file.

Kustomize vs. Helm: when to reach for which

☺ Like you're 10: Kustomize is for the stuff you own and change a lot. Helm is for packaging something up to hand to somebody else. Most real clusters end up using both.

The honest framing is rarely "which tool is better" — it's "what are you actually doing right now." Distributing software to strangers who need a stable, documented set of knobs is a packaging problem, and packages want Helm. Operating workloads that you own, across a handful of environments you control, is a customization problem, and customization wants Kustomize.

Most mature platforms use both at once, deliberately: a vendor ships a chart, and you need one change the chart author never exposed as a value. Kustomize's helmCharts field can inflate the chart during the build (with --enable-helm) and then apply your patches to the rendered output — or you run helm template in CI, commit the result, and treat it as a plain Kustomize base, which has the side benefit of making every future diff reviewable as ordinary YAML instead of a template render nobody can eyeball. Either way, the rule of thumb holds up well in practice: Helm for what you consume from someone else, Kustomize for what you customize yourself. Kustomize isn't called out by name on the official CKA or CKAD exam blueprints, but kubectl apply -k, bases, and patches show up naturally the moment a scenario needs the same app running across more than one namespace or environment — this course's own Workloads & Scheduling domain page is the place that pattern gets exercised directly, and Platform Engineering's own Kustomize page goes further into the GitOps-specific angle if that's the direction you're headed next.