External Secrets & the secrets patterns

The External Secrets Operator (ESO) is a Kubernetes controller that reads secret values out of a real secret manager — HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, 1Password — and writes them into ordinary Kubernetes Secret objects, continuously. It exists to solve the one problem that GitOps cannot solve on its own: your whole desired state is supposed to live in Git, but the database password absolutely must not. ESO's answer is that only a reference ever goes in Git — a pointer saying “fetch prod/checkout/db from Vault into a Secret called checkout-db” — while the value itself lives, rotates, and is audited somewhere Git can never see.

What it is and the problem it solves

☺ Like you’re 10: It’s a helper that fetches real passwords from a locked safe and puts them where your app can reach them — so the password never has to be written in the public notebook.

Kubernetes has a Secret object, and at first glance the problem looks solved. It isn’t, for two independent reasons. First, a native Secret’s data field is base64-encoded, not encrypted — base64 is a transport encoding a ten-year-old can reverse, so anyone with read access to the object, or to an etcd backup, has the plaintext. Second, and more sharply for platform teams: a Secret is a YAML manifest, and GitOps says all manifests live in Git. Commit it and the password is now in permanent, cloneable, un-deletable history.

Why “just don’t commit it” fails

The naive workaround is to keep Secrets out of Git and have a human kubectl apply them by hand. This breaks everything good about reconciliation: the cluster now contains state nobody declared, a rebuilt cluster comes back subtly broken, and “who set this value and when?” has no answer. You have traded a leak risk for an availability and auditability risk. The real requirement is a way for a secret to be fully declared in Git while its value is not.

The reference/value split

That is the idea the whole page turns on. Split a secret into two halves: the reference (which secret, from where, into what Kubernetes object, refreshed how often) and the value (the actual bytes). The reference is boring metadata — commit it, review it, diff it, roll it back. The value lives in a purpose-built store that does access control, versioning, audit logging and rotation properly. ESO is the machinery that joins the two at runtime, inside the cluster, where nobody is watching.

Where it fits in a platform

☺ Like you’re 10: It sits in the “keep things safe” part of the platform, and it hands its results to almost everything else.

ESO is a control-plane add-on in the security and configuration domain — cluster infrastructure, not application code. It is installed the same way as any other platform add-on (a Helm chart, reconciled from the config repo by Argo CD or Flux), and it produces something every workload already understands: a native Secret. That output contract is why it integrates with everything without anything having to know it exists.

Neighbouring tools

Think of it as sitting beside cert-manager, which solves the same shape of problem for a different asset — cert-manager materialises certificates into Secrets on a renewal schedule; ESO materialises credentials into Secrets on a refresh schedule. Both are consumed by ordinary Deployments. Upstream of ESO sit the stores themselves and, increasingly, the infrastructure control planes that create them: a Crossplane claim that provisions an RDS instance writes its connection secret somewhere, and ESO is often what carries it into the app namespace. Alongside sit the policy engines — Kyverno or Gatekeeper — which you use to forbid the anti-pattern (a bare committed Secret) that ESO makes unnecessary.

CNPE domain relevance

External Secrets is not on the CNPE exam tool list, and neither is Vault. But secrets handling underpins the GitOps & Continuous Delivery domain and sits squarely inside the Security & Policy domain, so the exam can absolutely ask you conceptually which pattern keeps a secret out of Git, or why a committed Secret is a finding. Treat this page as the practical companion to the lesson at Secrets & Workload Identity: know the patterns cold, and know that only native Kubernetes Secret manifests are the sort of thing you might be asked to write.

How it works — architecture and CRDs

☺ Like you’re 10: There are two kinds of note: one says which safe and which badge, the other says which drawer and which box to put it in.

ESO installs as a small set of Deployments (usually in the external-secrets namespace): the controller itself, a validating webhook, and a cert-controller that keeps the webhook’s certificates fresh. It introduces five custom resources you will meet day to day, and the split between them is the thing worth understanding.

SecretStore and ClusterSecretStore

A SecretStore answers “where do I fetch from, and how do I authenticate?” It names a provider (vault, aws, gcpsm, azurekv, onepassword, and many more) and the auth method to use with it. It is namespaced, so a team can own the connection to their own path. A ClusterSecretStore is the identical spec at cluster scope — one platform-owned connection that any namespace can reference. Note the sharp edge: a ClusterSecretStore is a shared credential, so whatever it can read, every namespace that may reference it can read. Scope its permissions in the store, not just in Kubernetes.

ExternalSecret and ClusterExternalSecret

An ExternalSecret answers “which keys, into which Secret, how often?” It points at a store, lists what to fetch, names the Kubernetes Secret to create, and sets a refreshInterval. Fetching comes in two flavours: data maps individual remote keys to individual Secret keys (precise, explicit), while dataFrom pulls an entire remote object at once (with extract) or every secret matching a pattern (with find). A ClusterExternalSecret is a template that fans a single ExternalSecret out into many namespaces selected by label — the standard way to push a shared image-pull secret or a common CA bundle everywhere at once.

PushSecret — the arrow reversed

PushSecret goes the other direction: it takes a Secret that already exists in the cluster and writes it into the external store. That sounds backwards until you meet the case it serves — an operator generated a credential in-cluster (a database operator minting a password, say) and you want the canonical store to hold the authoritative copy for other consumers and for humans. Use it deliberately and sparingly; the default direction of travel should be inward.

The resources you will actually write

☺ Like you’re 10: Two short notes and you’re done: one describing the safe, one describing what to fetch from it.

Three realistic examples. Read them as a pair-plus-one: the store, the secret, and the fan-out.

A ClusterSecretStore for Vault (Kubernetes auth)

This is the shape you will write most often in production: Vault’s Kubernetes auth method, where ESO presents its own ServiceAccount token and Vault validates it against the cluster’s token reviewer — so there is no static Vault token stored anywhere. version: v2 selects Vault’s KV version 2 engine, which is versioned and soft-deletes rather than overwrites. One portability note: current ESO serves these types under external-secrets.io/v1, but older installations still serve the same resources under the deprecated v1beta1 — run kubectl api-resources --api-group=external-secrets.io before copying a manifest between clusters.

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-prod
spec:
  provider:
    vault:
      server: https://vault.acme.internal:8200
      path: kv                     # the KV mount point
      version: v2                  # KV v2 — versioned, soft-delete
      auth:
        kubernetes:
          mountPath: kubernetes    # Vault auth mount
          role: eso-prod           # Vault role bound to the SA below
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets

The equivalent for AWS Secrets Manager swaps the provider block for aws: with service: SecretsManager (or ParameterStore), a region, and a jwt auth stanza pointing at a ServiceAccount annotated for IRSA — which means, again, no stored access key.

An ExternalSecret with data, dataFrom and templating

Now the consumer side. Note four things: refreshInterval controls how often ESO re-reads the store; target.template lets you compose a value (here, a full DSN) rather than passing raw fields to the app; dataFrom.extract flattens an entire remote JSON object into Secret keys; and the easily-missed mergePolicy — a template defaults to Replace, meaning the rendered keys become the only keys in the resulting Secret, so if you want the fetched keys to survive alongside the composed one you must ask for Merge.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: checkout-db
  namespace: checkout
spec:
  refreshInterval: 1h              # re-read the store on this cadence
  secretStoreRef:
    name: vault-prod
    kind: ClusterSecretStore
  target:
    name: checkout-db              # the Secret ESO creates & owns
    creationPolicy: Owner          # ESO owns it; deleting the ES deletes it
    template:
      engineVersion: v2
      mergePolicy: Merge           # keep the fetched keys AND add the templated one
      data:
        # build a connection string from the fetched fields
        DATABASE_URL: "postgres://{{ .username }}:{{ .password }}@db.prod:5432/checkout"
  data:
    - secretKey: username          # key inside the Kubernetes Secret
      remoteRef:
        key: prod/checkout/db      # path inside the store
        property: username         # field within that object
    - secretKey: password
      remoteRef:
        key: prod/checkout/db
        property: password
  dataFrom:
    - extract:
        key: prod/checkout/stripe  # pull every field of this object in

A ClusterExternalSecret fanned out by namespace label

One object, many namespaces — the platform-team pattern for something every tenant needs. Add the label to a namespace and the Secret appears; remove it and the Secret goes away.

apiVersion: external-secrets.io/v1
kind: ClusterExternalSecret
metadata:
  name: registry-pull
spec:
  externalSecretName: registry-pull
  refreshTime: 1h
  namespaceSelectors:
    - matchLabels:
        acme.io/tenant: "true"     # every tenant namespace gets it
  externalSecretSpec:
    refreshInterval: 1h
    secretStoreRef: { name: vault-prod, kind: ClusterSecretStore }
    target:
      name: registry-pull
      template:
        type: kubernetes.io/dockerconfigjson
        data:
          .dockerconfigjson: "{{ .dockerconfig | toString }}"
    data:
      - secretKey: dockerconfig
        remoteRef: { key: platform/registry, property: dockerconfigjson }

Day-to-day commands

☺ Like you’re 10: There’s no special app to learn — you check on it with the same kubectl you already use.

ESO has no CLI of its own. Everything is kubectl against its CRDs, which is a genuine design virtue: it means the tool is debuggable with the skills the exam already tests.

Installing and inspecting

# install (in real life, do this from your config repo via Argo CD / Flux)
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace

# what CRDs did it bring?
kubectl api-resources --api-group=external-secrets.io

# is the store reachable and authenticated?
kubectl get clustersecretstore vault-prod -o wide
kubectl describe clustersecretstore vault-prod   # look for Ready / ValidationFailed

Debugging a secret that will not appear

# the status and its conditions hold the real error
kubectl -n checkout get externalsecret checkout-db
kubectl -n checkout describe externalsecret checkout-db

# did the Secret land, and does it have the keys you expect?
kubectl -n checkout get secret checkout-db -o jsonpath='{.data}' | tr ',' '\n'

# force an immediate re-sync without waiting for refreshInterval
kubectl -n checkout annotate externalsecret checkout-db \
  force-sync="$(date +%s)" --overwrite

# operator logs — auth failures and provider errors surface here
kubectl -n external-secrets logs deploy/external-secrets --tail=100

The Vault side

# assumes a KV v2 engine mounted at kv/ — matching path: kv in the store above
vault kv put   kv/prod/checkout/db username=checkout password='s3cr3t'
vault kv get   kv/prod/checkout/db     # KV v2 keeps versions
vault kv get -version=3 kv/prod/checkout/db

# dynamic credentials: Vault mints a short-lived DB user with a lease
vault read database/creds/checkout-role
vault lease revoke -prefix database/creds/

That last pair is the point of Vault that people miss. A dynamic secret is not stored at all — Vault creates a fresh database user on demand, hands it back with a lease, and revokes it when the lease expires. There is no long-lived password to steal, and “rotation” becomes automatic rather than an event.

Gotchas and failure modes

☺ Like you’re 10: The helper fetches the new key — but nobody tells your app to go and look in the box again.

Rotation does not restart your pods

This is the single most common surprise. ESO updates the Secret; it does not restart anything. A Secret consumed via env or envFrom is read once at container start and is frozen for the life of the process — the pod keeps using the old password forever. (A Secret mounted as a volume does get updated in place by the kubelet — after a delay of roughly the kubelet sync period plus its cache TTL, so on the order of a minute or two — but only if the app re-reads the file, and never at all if the volume was mounted with subPath, which pins the file at its original contents.) The two standard fixes: run Reloader, a controller that watches Secrets and rolls the Deployments that reference them; or have your rendering put a checksum of the secret into a pod-template annotation so the change itself triggers a rollout. Symptoms of getting this wrong look exactly like an application bug — see Workload triage.

Auth, scope and blast radius

A ClusterSecretStore that can read the whole Vault mount is a cluster-wide skeleton key: any namespace allowed to reference it can pull any path. Scope the Vault role or IAM policy to a prefix, prefer per-team namespaced SecretStore objects for anything sensitive, and remember that RBAC on the resulting Secret is what actually protects the value once it lands. If a Secret is missing entirely, check the ExternalSecret’s conditions first, then the store’s — the failure is far more often an auth or path error than a bug. Delivery-shaped symptoms belong in Delivery triage.

Templating and type mistakes

Two small ones with outsized annoyance value. A Secret meant for an image pull must have type: kubernetes.io/dockerconfigjson and exactly the key .dockerconfigjson; get either wrong and the kubelet silently ignores it. And values pulled from a store frequently carry a trailing newline from however a human pasted them in — which breaks tokens in ways that are very hard to see. Template deliberately rather than hoping the raw bytes are clean.

Alternatives and when to choose it

☺ Like you’re 10: Three different ways to keep the key out of the public notebook — scrambled in the notebook, scrambled a different way, or not in the notebook at all.

ESO is one of three GitOps-safe patterns, and a mature platform sometimes runs two. The distinction that matters is where the secret material physically is.

The three patterns compared

Read them as a ladder. Sealed Secrets is the cheapest honest answer: kubeseal encrypts a Secret to a public key that only the target cluster’s controller can undo, so the resulting SealedSecret is safe in a public repo — but it is bound to that cluster and rotation means a commit. SOPS encrypts individual values in place, keeping the file readable and diffable, and Flux decrypts it during reconciliation; excellent ergonomics, same rotation problem. ESO removes secret material from the repo altogether, at the cost of a runtime dependency on the store.

Vault delivery: ESO vs Agent Injector vs the CSI driver

The endgame: no stored credential at all

Every pattern above is a way of moving a stored secret around more safely. The destination is not to move it but to abolish it. With workload identity — SPIFFE/SPIRE issuing short-lived cryptographic identities, or IRSA and its GKE/Azure equivalents federating a pod’s ServiceAccount token to a cloud role — a workload proves who it is and receives a credential valid for minutes. There is nothing to rotate, nothing to leak, nothing to commit. Vault’s dynamic secrets are the same idea for databases. Treat ESO as the pragmatic middle of the ladder for the many systems that still demand a static password, and push everything you can to identity. The full ladder is laid out in Secrets & Workload Identity.

Exam relevance and going further

☺ Like you’re 10: You won’t be asked to run this tool in the exam — but you must be able to say why a password in Git is wrong, and what to do instead.

What to be able to do cold

Four things. One: say in one sentence why a native Secret is not secure by itself (base64 is encoding; you need encryption at rest plus RBAC). Two: name the three GitOps-safe patterns and the one-line distinction between them — Sealed Secrets and SOPS put ciphertext in Git, ESO puts only a reference. Three: write a plain Kubernetes Secret and consume it from a Deployment via envFrom and via a volume, from memory — that is the examinable manifest, drilled on Know Cold. Four: explain why updating a Secret does not update a running pod.

The documentation allowlist

Say this plainly, because it changes how you should study: during the CNPE exam the only documentation you may open is kubernetes.io/docs, kubernetes.io/blog, whatever task-specific links appear in that question’s Quick Reference box, and local man pages and /usr/share docs. external-secrets.io and HashiCorp’s Vault docs are not available to you. Neither tool is on the CNPE tool list, so no task should require either — but you cannot look up an ExternalSecret field mid-exam, and you can look up a native Secret on kubernetes.io in seconds. Spend your memorisation budget accordingly, and check the docs map for what is actually reachable.

⚖ CNPA vs CNPE — That allowlist is a CNPE-only mechanic: CNPE is hands-on, so it permits those narrow live lookups mid-task. CNPA is stricter, not looser — a fully closed-book, multiple-choice exam with zero external resources and zero lookups of any kind, so neither external-secrets.io nor kubernetes.io would be reachable there. Even so, the concept-level knowledge above — why a native Secret isn't secure by itself, and the ciphertext-in-Git vs reference-only distinction between Sealed Secrets/SOPS and ESO — is exactly the kind of thing CNPA's closed-book recall draws on.

Official resources for after the exam

The canonical sources are the ESO docs at external-secrets.io (the API and Provider sections are the ones you will live in), the source at github.com/external-secrets/external-secrets, Kubernetes’ own Secret concept page and its encrypting-data-at-rest task — both of which are on the exam allowlist and worth reading now. For the alternatives: Sealed Secrets, SOPS, and SPIFFE. Then come back to the tool landscape to see how this piece sits beside the rest, and to Security & Policy for the enforcement half of the story.