Secrets & Workload Identity

Every platform eventually trips over the same rock: real apps need database passwords, API tokens, and cloud credentials, yet the two systems the platform is built on — a public-ish Git repo and a shared multi-tenant cluster — are both terrible places to keep a secret. This deep dive walks the whole arc, from the uncomfortable truth that native Kubernetes Secrets barely protect anything, through the tools that let you keep secrets near Git without keeping them in Git (Sealed Secrets, the External Secrets Operator, SOPS), and on to the real destination senior teams are chasing: making most secrets short-lived — or making them disappear entirely and replacing them with cryptographic workload identity. The best secret is the one you never have to store.

Why native Kubernetes Secrets aren’t enough

☺ Like you’re 10: A Kubernetes “Secret” sounds locked, but out of the box it’s more like writing your password in a slightly weird alphabet — anyone allowed to look can still read it.

The very first misconception to demolish is that a Kubernetes Secret object is a secure object. It is a convenience object: a place to keep small blobs separate from your ConfigMaps and mount them as files or environment variables. What it is not, by default, is encrypted, tightly access-controlled, or safe to treat as a vault. Understanding exactly how thin the protection is tells you why the rest of this page exists.

Base64 is encoding, not encryption

The data fields of a Secret are base64-encoded, and engineers routinely mistake that for security. Base64 is a reversible transport encoding with no key — it turns bytes into an ASCII-safe alphabet so YAML doesn’t choke on binary. Anyone who can read the Secret can decode it in one line:

$ kubectl get secret db-creds -n payments -o jsonpath='{.data.password}' | base64 -d
S3up3r$ecret-Pr0d-Pa55w0rd      # ← the real plaintext, no key required

So base64 buys you exactly nothing against anyone with read access. The obfuscation is accidental, never a control. Treat the contents of any Secret as plaintext to every principal — human or workload — that the API and the underlying store expose it to.

Encryption at rest: etcd and KMS

☺ Like you’re 10: Kubernetes writes every Secret into one big shared notebook (etcd). Unless you turn on a lock, that notebook is written in plain pencil — steal the notebook and you’ve got everything.

Every Secret is persisted in etcd, the cluster’s key-value database. In a default cluster those values sit in etcd unencrypted — so an etcd disk snapshot, a backup left in an S3 bucket, or a compromised control-plane node hands the attacker every secret at once. The fix is the API server’s EncryptionConfiguration, and the strong form uses envelope encryption backed by an external KMS (AWS KMS, GCP Cloud KMS, Azure Key Vault): the API server encrypts each Secret with a local data-encryption key (DEK), and that DEK is itself encrypted by a key-encryption key (KEK) that never leaves the KMS. A stolen etcd backup is then useless without live access to the KMS.

# EncryptionConfiguration passed to kube-apiserver via --encryption-provider-config
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets                         # encrypt Secret objects at rest
    providers:
      - kms:                            # first provider = used to ENCRYPT new writes
          apiVersion: v2                # KMS v2: better key rotation & performance
          name: cloud-kms
          endpoint: unix:///var/run/kmsplugin/socket.sock
      - identity: {}                    # fallback so existing plaintext can still be READ

RBAC on Secrets and the blast radius

☺ Like you’re 10: If someone’s allowed to “list all the keys in the room,” they don’t just get their key — they get everyone’s. So be stingy about who can list keys.

Encryption at rest protects the disk; RBAC protects the API. And this is where teams quietly bleed: a Role with get/list on secrets grants read of every secret it scopes — one over-broad grant and a single compromised pod can exfiltrate the whole namespace’s credentials. Three habits shrink the blast radius. First, never grant list secrets broadly; scope by resourceNames to the exact Secret a workload needs. Second, give every workload its own ServiceAccount and set automountServiceAccountToken: false where a pod needs no API access, so a popped container can’t use the mounted token to read the API at all. Third, remember that anyone who can create a pod in a namespace can usually mount and read the Secrets there — so “deploy” permission is effectively “read secrets” permission unless policy stops it.

The GitOps secrets problem

☺ Like you’re 10: GitOps says “write everything down in the shared notebook and let a robot copy it into the cluster.” Great — until you realise a password would go in that notebook too, and the notebook remembers forever.

The moment you adopt GitOps, a beautiful principle — the whole desired state lives in Git — collides with an ugly requirement: some of that state is secret. Naively, the checkout app’s manifest wants a Secret with the database password right beside the Deployment that uses it. Commit that, and you have created a problem that is far worse than it looks.

Git history is forever

Deleting a secret from a file does not remove it from Git — every prior commit still contains it, and git log -p hands it right back. Worse, Git is distributed: by the time you notice, the secret is in every fork, every clone on every laptop, every CI cache, and quite possibly a mirror on the public internet. History-rewriting tools like git filter-repo can scrub the primary repo, but they can never recall the copies. The only safe operational assumption is stark: a secret that has ever touched a Git repository is a burned secret and must be rotated, not un-committed.

The reference belongs in Git; the value never does

Once you internalise that rule, the design space clarifies. What lives in the repo is either (a) a reference that names a secret held in an external store, or (b) ciphertext that is safe to publish because only an in-cluster key can decrypt it. Both keep the reconciler’s promise — the full desired state is still declared in Git and auditable — while the actual sensitive bytes stay outside the readable history. Your reviewers can diff a pull request, your auditor can trace every change, and yet no human reading the repo ever sees a live credential.

Two families of answer

Everything that follows falls into two families. Encrypt-and-commit tools (Sealed Secrets, SOPS) let you put ciphertext directly in Git; the cluster decrypts it on the way in. Sync-in tools (the External Secrets Operator) keep the value in an external store and put only a reference in Git; an operator pulls the value into a native Secret at runtime. Which family fits depends on whether you want Git to be self-contained (encrypt-and-commit) or whether you want a single external source of truth you can rotate independently (sync-in). We’ll meet the store first, then all three tools.

External secret stores — the source of truth outside the cluster

☺ Like you’re 10: Instead of scattering copies of the key everywhere, you put the one real key in a bank vault, and everything else just asks the vault when it needs to get in.

A dedicated secret store is a purpose-built vault: strong encryption, fine-grained access policy, a full audit trail of every read, and — crucially — first-class support for rotation and short leases. On a platform it becomes the single source of truth for secrets, exactly as Git is the source of truth for config. The cluster stops owning secrets and becomes a mere consumer of them.

HashiCorp Vault

HashiCorp Vault is the reference implementation of the category. Its KV engine stores static secrets, but its real power is the surrounding machinery: pluggable auth methods (a workload authenticates with its Kubernetes ServiceAccount token via the Kubernetes auth method, rather than holding a Vault token), fine-grained policies, a tamper-evident audit log, the transit engine for encryption-as-a-service, and — the crown jewels — dynamic secrets engines that generate credentials on demand with a lease and automatic revocation (covered next section). A pod proves it is checkout using its own SA token; Vault maps that to a policy and returns only the secrets that identity is allowed.

Cloud secret managers

If you live in one cloud, the managed option is often the pragmatic choice: AWS Secrets Manager (and the cheaper SSM Parameter Store), GCP Secret Manager, and Azure Key Vault. Their appeal is integration — access is governed by the same cloud IAM you already run, secrets encrypt under the cloud KMS automatically, and rotation can be wired to a managed function. Their limit is portability and depth: you get solid static-secret storage and basic rotation, but rarely Vault’s breadth of dynamic engines. Many platforms run both — a cloud manager for cloud-native credentials, Vault where they need dynamic issuance or multi-cloud neutrality.

Why “outside the cluster” changes everything

☺ Like you’re 10: When the real key lives in the bank — not in the house — losing the house doesn’t lose the key.

Moving the source of truth out of the cluster buys four things at once. Blast radius: a fully compromised cluster leaks only the secrets it had pulled recently, not a central archive. Rotation: you change the value in one place and every consumer picks it up, instead of hunting copies. Audit: the store logs every single read — “who fetched the payments DB password, and when?” is a query, feeding directly into governance & compliance evidence. Reach: many clusters, plus CI and VMs, can share one governed source. The cluster becomes a cache of short-lived material, not the crown vault.

Three ways to bridge Git and the cluster

☺ Like you’re 10: Three popular tools, three different tricks, all obeying the same rule: never let a readable password sit in the notebook.

Here are the three patterns the exam and the real world both name. Two encrypt-and-commit (Sealed Secrets, SOPS) and one syncs-in (the External Secrets Operator). They are not rivals so much as fits for different constraints; a mature platform often uses more than one.

Sealed Secrets — encrypt to a cluster key, commit the ciphertext

Sealed Secrets (Bitnami) is the simplest to reason about. A controller in the cluster holds an asymmetric key pair and publishes the public half. You run kubeseal locally to encrypt a normal Secret with that public key, producing a SealedSecret custom resource that is safe to commit — only the controller’s private key can open it. On apply, the controller reconciles each SealedSecret into a real Secret. It is asymmetric on purpose: developers can seal without ever holding a decryption key.

# encrypt locally with the controller's PUBLIC key — output is safe to commit
$ kubeseal --format yaml --controller-namespace kube-system \
    < db-secret.yaml > sealed-db-secret.yaml

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-creds
  namespace: payments               # scope is baked in: sealed for THIS name+namespace
spec:
  encryptedData:
    password: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEQAx...   # opaque ciphertext, safe in Git
  template:
    metadata: { name: db-creds, namespace: payments }
    type: Opaque

Two senior caveats. By default sealing is strict-scoped to one name and namespace, so ciphertext can’t be copied to another namespace to smuggle a value out — good, but it means renaming requires re-sealing. And the controller’s private key is now the crown jewel: back it up (losing it means every SealedSecret is unrecoverable) and rotate it on a schedule. Sealed Secrets keeps Git fully self-contained but gives you no central rotation — the value still originates from a human sealing it.

External Secrets Operator — sync from the store into native Secrets

☺ Like you’re 10: A little robot inside the cluster that keeps asking the bank vault “what’s the current password?” and quietly writes today’s answer into a Secret for the app.

The External Secrets Operator (ESO) implements the sync-in family and is the most common choice on cloud platforms. You declare a SecretStore (or cluster-wide ClusterSecretStore) describing where secrets live and how to authenticate, and an ExternalSecret that maps external keys to a native Secret ESO creates and keeps refreshed. Only references live in Git; the values never do. Because ESO re-reads on refreshInterval, rotating the value in the store propagates automatically.

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: payments
spec:
  provider:
    vault:
      server: "https://vault.acme.internal"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:                    # the app proves identity with its SA token —
          mountPath: "kubernetes"      # no Vault token stored anywhere
          role: "payments-app"
          serviceAccountRef: { name: "checkout" }
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-creds
  namespace: payments
spec:
  refreshInterval: "1h"               # re-pull hourly → rotation propagates for free
  secretStoreRef: { name: vault-backend, kind: SecretStore }
  target:
    name: db-creds                    # the native Secret ESO will create & own
    creationPolicy: Owner
  data:
    - secretKey: password             # key in the resulting Secret
      remoteRef:
        key: payments/db              # path in Vault
        property: password

Note what just happened: Git contains no secret at all — only the fact that checkout needs payments/db#password. The store is the source of truth; ESO is the reconciler between it and Kubernetes. This is the pattern that composes best with dynamic secrets and the maturity ladder we’re climbing toward.

SOPS — encrypt values in the repo, decrypt at apply

SOPS (Secrets OPerationS, Mozilla) takes the other encrypt-and-commit path. Unlike Sealed Secrets it encrypts only the values of a YAML/JSON file, leaving keys and structure readable — so pull-request diffs still show which fields changed, just not their plaintext. Encryption keys can be age, a cloud KMS, or PGP. Its killer integration is with Flux: a Kustomization with a decryption block decrypts SOPS files on the fly at apply time, using a private key stored in the cluster.

# .sops.yaml — encrypt only data/stringData fields, with an age recipient
creation_rules:
  - path_regex: .*\.enc\.yaml$
    encrypted_regex: ^(data|stringData)$
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8j
---
# Flux decrypts SOPS files during reconciliation using an in-cluster age key
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: { name: apps, namespace: flux-system }
spec:
  interval: 10m
  path: ./apps/prod
  sourceRef: { kind: GitRepository, name: platform-config }
  decryption:
    provider: sops
    secretRef: { name: sops-age }     # holds the age PRIVATE key

SOPS shines when you want secrets self-contained in Git and reviewable field-by-field, especially with Flux. It’s weaker on rotation (still human-originated, like Sealed Secrets) and its decryption key is, again, a jewel to guard. Here is the trade matrix:

Dynamic secrets & rotation — shrink the standing footprint

☺ Like you’re 10: The safest password is one that only works for the next hour. Steal it at midnight and by 1am it’s already garbage.

Every tool so far still deals in standing secrets — long-lived values that exist whether or not anyone is using them, and stay dangerous until someone rotates them. The next leap is to make secrets ephemeral: issued on demand, valid briefly, then revoked. A leaked ephemeral credential is a small, self-healing incident instead of a catastrophe.

Dynamic, short-lived credentials

Vault’s dynamic secrets engines are the canonical example. Instead of storing a database password, Vault holds admin access to the database and mints a brand-new, unique user with a short lease (TTL) each time a workload asks. When the lease expires — or the workload’s session ends — Vault revokes it by dropping the database role. No two pods share a credential, nothing long-lived sits in etcd, and a compromised value dies on its own within the hour.

# one-time setup: let Vault manage a Postgres and define a short-lived role
$ vault secrets enable database
$ vault write database/config/payments-pg \
    plugin_name=postgresql-database-plugin \
    allowed_roles="payments-app" \
    connection_url="postgresql://{{username}}:{{password}}@pg.acme.internal:5432/app" \
    username="vault-admin" password="$ADMIN_PW"

$ vault write database/roles/payments-app \
    db_name=payments-pg \
    creation_statements="CREATE ROLE \"{{name}}\" LOGIN PASSWORD '{{password}}' \
      VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" max_ttl="24h"

# runtime: each read returns a FRESH user, auto-revoked when the lease ends
$ vault read database/creds/payments-app
# → username: v-token-payments-app-x9f2...   password: A1a-...   lease_duration: 1h

Automatic rotation of the static ones that remain

☺ Like you’re 10: For the keys you can’t make melt, at least change the locks on a timer instead of “never.”

Not everything can be dynamic — some third-party API keys are simply long-lived. For those, the goal is automatic rotation on a schedule, so no credential ages indefinitely. Cloud secret managers offer managed rotation (a function swaps the value and updates the store on a cadence); Vault can rotate the root credential it uses so even the bootstrap password isn’t static. The trap is the rotation window: if you swap a value but consumers cache the old one, they break. The clean design overlaps validity — the new credential works before the old is revoked — and lets consumers (via ESO’s refreshInterval, or a sidecar) pick up the new value with no restart.

The maturity ladder: toward zero standing secrets

Putting it together gives a ladder every platform can be placed on. The higher the rung, the smaller the window in which a leaked secret is useful — and the top rung has no secret to leak at all, which is the subject of the next section.

Workload identity — the endgame

☺ Like you’re 10: The face scanner. The app doesn’t carry a key at all — it just is itself, provably, and doors open because they recognise it.

The deepest fix isn’t a better place to hide a secret — it’s not having one. Workload identity gives each workload a cryptographically-verifiable identity, and services and clouds trade that for short-lived access. There is no API key to store, rotate, or leak. This is where Pip lights up: identity lives on the wire, not in a vault.

SPIFFE, SPIRE, and SVIDs

SPIFFE (Secure Production Identity Framework For Everyone) is the open standard for this. It defines a SPIFFE ID — a URI like spiffe://acme.io/ns/payments/sa/checkout — as a universal name for a workload, and an SVID (SPIFFE Verifiable Identity Document) as the credential proving it, delivered as either an X.509 certificate or a JWT. SPIRE is the runtime that issues them: a server anchors a trust domain, and node agents attest each workload (“this really is the checkout pod, on this node, with this SA”) before handing it a short-lived SVID through the local Workload API. The workload never stores a long-lived key; it fetches a fresh SVID and presents it — and any peer can verify it against the trust domain. This is exactly the identity a service mesh uses for the mTLS you met in security & policy.

Cloud workload identity (IRSA, GKE, Azure)

☺ Like you’re 10: Instead of giving your app a cloud password, the cloud learns to trust the cluster’s “I vouch for this pod” note and hands back a one-hour pass.

Every major cloud now offers the same trick under different names: bind a Kubernetes ServiceAccount to a cloud identity, and let the pod exchange a projected SA token for short-lived cloud credentials — no static access key ever stored in the cluster. On AWS this is IRSA (IAM Roles for Service Accounts; newer clusters can also use EKS Pod Identity); on Google it’s GKE Workload Identity; on Azure, Azure Workload Identity.

# AWS IRSA — annotate the ServiceAccount with the IAM role to assume
apiVersion: v1
kind: ServiceAccount
metadata:
  name: checkout
  namespace: payments
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/payments-checkout
---
# the IAM role's trust policy — only THIS SA in THIS namespace may assume it
# {
#   "Effect": "Allow",
#   "Principal": { "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED..." },
#   "Action": "sts:AssumeRoleWithWebIdentity",
#   "Condition": { "StringEquals": {
#     "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED...:sub": "system:serviceaccount:payments:checkout"
#   }}
# }

OIDC federation: the trust chain

The magic underneath all three is OIDC federation. The Kubernetes API server is itself an OIDC issuer: it signs projected ServiceAccount tokens (short-lived JWTs, audience-scoped) and publishes a public JWKS the cloud can fetch. The cloud’s STS is configured to trust that issuer for a specific role, with conditions pinning the exact sub (namespace + ServiceAccount). At runtime the pod presents its projected token; STS validates the signature against the cluster’s public keys, checks the conditions, and returns credentials that live for minutes. Trust flows one way, over public keys — no shared secret in sight.

cert-manager: short-lived identity for the wire

☺ Like you’re 10: A tireless robot that hands out ID cards which expire fast — and quietly prints new ones before the old expire, so no door ever slams shut.

Identity on the network needs a steady supply of fresh certificates, and hand-rotated ones always expire at 2am. cert-manager is the Kubernetes-native certificate robot: you declare an Issuer/ClusterIssuer (backed by an internal CA, Vault, or an ACME provider like Let’s Encrypt) and a Certificate, and it obtains the cert, stores it in a Secret, and renews before expiry. It secures ingress TLS, and via istio-csr can back a mesh’s workload SVIDs — the same short-lived-identity philosophy, applied to TLS. No expired-cert outages, no long-lived key sitting around.

Secrets in CI/CD — kill the long-lived cloud key

☺ Like you’re 10: The pipeline used to keep a master key taped inside the drawer. Now it shows its work badge each run and borrows a key that expires when the job ends.

The single most-leaked secret in the industry is the CI/CD credential — a static cloud access key stored in a pipeline’s secret settings, with broad permissions and no expiry. It ends up in logs, in forked-PR runs, in screenshots. The same OIDC federation that frees workloads frees pipelines too: the runner proves its identity per run and borrows short-lived cloud credentials — the static key simply ceases to exist.

OIDC from the pipeline to the cloud

Modern CI systems (GitHub Actions, GitLab CI) are OIDC issuers. A workflow mints a signed token describing exactly which repo, branch, and job is running; the cloud’s trust policy accepts that token for a scoped role and returns temporary credentials. No AWS_ACCESS_KEY_ID secret in the repo at all.

# GitHub Actions → AWS with OIDC — zero static keys
permissions:
  id-token: write        # let the job mint an OIDC token
  contents: read
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/ci-deployer
          aws-region: us-east-1
          # note: NO aws-access-key-id / aws-secret-access-key —
          # the runner exchanges its OIDC token for temporary STS creds

# AWS trust policy pins the exact repo + branch so a fork can't assume the role:
#   "Condition": { "StringEquals": {
#     "token.actions.githubusercontent.com:sub": "repo:acme/checkout:ref:refs/heads/main"
#   }}

What still needs a secret, and how to hold it

Some pipeline secrets remain — a third-party SaaS token, a signing key. Keep them out of the repo and in the CI provider’s encrypted secret store (or better, fetch them from Vault at run time with a short lease so the pipeline holds them only for the job). The principle mirrors the maturity ladder: prefer OIDC (no secret) > dynamic fetch (short-lived) > encrypted CI secret (static but hidden) — and never a plaintext value in the workflow file.

Pipeline blast radius: masking, scoping, least privilege

☺ Like you’re 10: Give the robot only the keys for this job, hide anything it does print, and don’t lend those keys to strangers who open pull requests.

Three controls contain what a pipeline can do with whatever it holds. Masking redacts known secret values from logs (defence-in-depth, not a guarantee — a base64 or split value can slip through). Environment scoping gates production credentials behind protected environments and required reviewers, so a random branch can’t touch prod. And least privilege means the borrowed role can do exactly this deploy and nothing more — a compromised job can’t pivot. Pull-request builds from forks should never receive privileged credentials at all. This is the secrets half of CI/CD & progressive delivery: the pipeline shifts scanning and signing left, but must itself hold as little power, for as short a time, as possible.

Detection & hygiene — assume something will leak

☺ Like you’re 10: Even careful people drop keys sometimes. So you also keep a metal detector at the door and a plan for the day a key goes missing.

No matter how good your patterns are, someone will paste a token into a commit, a log, or a config. Mature platforms plan for it: they detect leaks fast and respond assuming compromise. Prevention and detection are different jobs — you need both.

Secret scanning: gitleaks, trufflehog, push protection

Automated scanners hunt for credential-shaped strings. gitleaks and trufflehog scan commits, history, and diffs against pattern and entropy rules; the strongest place to run them is a pre-commit hook (stop the secret before it’s ever committed) plus a CI gate (catch what slips through) plus platform push protection (the forge rejects a push containing a known token type). trufflehog can even verify a found credential against its provider, cutting false positives.

# .pre-commit-config.yaml — block secrets locally, before they reach history
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.0
    hooks:
      - id: gitleaks           # fails the commit if a secret-shaped string is found

# and as a CI backstop on every PR:
#   gitleaks detect --source . --redact --exit-code 1

Leak response: a committed secret is a burned secret

When a scanner (or a bad day) surfaces a leaked credential, the response is not “delete the commit.” As we saw, history and clones make deletion meaningless. The correct, non-negotiable sequence is rotate → revoke → audit: rotate the credential so the leaked value is worthless, revoke the old one at the source, then audit the store and API logs for any use during the exposure window. This is precisely where the earlier rungs pay off — if the leaked thing was a fifteen-minute token or a Vault lease, step one already happened by itself.

Hygiene as a platform default

The senior move is to make the secure path the only path, so no developer has to remember any of this. When Mira’s self-service template scaffolds a service, it should ship already wired: its own ServiceAccount bound to a scoped cloud identity (no static key), an ExternalSecret pointing at the team’s store, a pre-commit scanner and CI gate, and admission policy that refuses a pod that mounts a raw plaintext Secret. Because it’s all declarative, it flows through GitOps — references and policies reconcile from Git, with one audit trail and one-commit rollback — and the read-log evidence lands in governance & compliance for free. Guardrails baked into the golden path, exactly as the best-practices playbook prescribes.

That’s the whole journey in one arc: from “key under the doormat” to a face scanner. Native Secrets are a convenience, not a vault; keep values out of Git; put the source of truth in a real store; make what remains short-lived; and wherever you can, replace the secret with an identity nobody can steal. Next, Timmy carries the guardrail mindset into the broader security & policy layer, and Dot’s clean golden path continues in self-service.