HashiCorp Vault

Vault is what "a centralized secrets manager" from secrets management looks like once you actually install it: a policy-gated API that authenticates a client, hands back a token scoped to exactly what that client is allowed to touch, and then either reads a stored value or — more interestingly — generates one on the spot and forgets it again a short while later. This page skips the concepts you already have and goes straight to the mechanism: how the database secrets engine mints and revokes a Postgres credential per request, how the transit engine turns encryption into an API call instead of a library import, and how a CI job or a Kubernetes pod actually proves who it is without ever holding a long-lived password.

What Vault is, and the four things it makes you name

☺ Like you're 10: Vault has exactly four moving parts worth learning — who's asking, what they're allowed to do, what kind of secret they want, and how long they get to keep it.

Vault is an open-source (and separately, commercially licensed) secrets management system originally built at HashiCorp, run either self-hosted as a cluster you operate or as the managed HCP Vault offering. Structurally it's simpler than the ecosystem around it suggests: every single request to Vault reduces to four objects working together.

• Auth methods — pluggable ways to prove identity. Token, AppRole, Kubernetes, LDAP, OIDC, and AWS/GCP/Azure IAM are the common ones; a client authenticates through exactly one of these and gets back a Vault token in exchange.
• Policies — HCL documents that grant capabilities (read, create, update, delete, list, sudo) on specific API paths. Vault is default-deny: a path with no matching policy statement is simply unreachable, not a warning.
• Secrets engines — pluggable backends mounted at a path, each speaking its own dialect of "give me a secret." KV v2 stores and versions a value you wrote yourself; Database, PKI, SSH, and the AWS/GCP/Azure engines generate a credential on request; Transit generates nothing and stores nothing — it just performs cryptographic operations on your behalf.
• Tokens and leases — every authenticated request carries a token, and every engine-issued secret that isn't permanent carries a lease with a TTL. Vault's own internal expiration manager — not the client, not a cron job — is what revokes a lease when its TTL runs out.

Underneath all four sits one more piece worth naming once: the barrier, an AES-256-GCM encryption layer that wraps every byte before it ever reaches the storage backend (Integrated Storage/Raft, or Consul in older deployments). Vault starts sealed — the encryption key needed to read that storage isn't in memory — and stays that way until a threshold of unseal keys (Shamir's Secret Sharing, default 3-of-5) is supplied, or an auto-unseal mechanism backed by a cloud KMS does it automatically on boot. The storage backend itself never holds anything Vault hasn't already encrypted; steal the Raft snapshot and you still have nothing readable without the unseal keys.

Static secrets versus dynamic secrets: what a lease expiry actually does

☺ Like you're 10: A static secret is a password Vault remembers for you; a dynamic secret is a brand-new password Vault's engine tells the database to create right now, and to delete again in an hour.

The KV v2 engine (vault kv put secret/orders/config username=svc password=hunter2) is what most people picture first — Vault as a versioned, access-controlled key-value store. It's a real improvement over a checked-in .env file, but it's still a static secret: a value someone or something wrote once, that stays valid until someone remembers to rotate it. The far more consequential capability is the database secrets engine, which doesn't store a credential at all — it holds a root connection to your database and mints a brand-new one, scoped and time-boxed, for every single request.

# one-time setup: point the engine at Postgres with a privileged root connection
$ vault secrets enable database
$ vault write database/config/orders-postgres \
    plugin_name=postgresql-database-plugin \
    connection_url="postgresql://{{username}}:{{password}}@postgres.internal:5432/orders?sslmode=require" \
    allowed_roles="readonly,readwrite" \
    username="vault-admin" \
    password="$(cat /tmp/vault-admin-pw)"

# a ROLE defines what SQL runs to create — and to destroy — a credential
$ vault write database/roles/readonly \
    db_name=orders-postgres \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
      GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

# every read to this path runs the creation_statements FRESH — no two callers ever get the same username
$ vault read database/creds/readonly
Key                Value
---                -----
lease_id            database/creds/readonly/2n8Fh3k9s...
lease_duration       1h
lease_renewable      true
username             v-token-readonly-a1b2c3d4e5f6
password             A1b2C3d4-generated-by-vault

That last read is the whole point: Vault connects to Postgres as its own privileged root user, runs creation_statements to literally CREATE ROLE with a password only that request will ever see, and hands back a lease alongside it. Nothing about that username existed a moment before the read, and nothing about it will exist an hour later — Vault's internal expiration manager runs a matching revocation_statements (a default DROP ROLE if you don't override it) the instant the lease's TTL is up, with no human, no cron job, and no application code involved.

$ vault lease renew database/creds/readonly/2n8Fh3k9s...   # extend, up to max_ttl
$ vault lease revoke database/creds/readonly/2n8Fh3k9s...  # kill this one lease right now
$ vault lease revoke -prefix database/creds/readonly        # incident response: kill EVERY outstanding lease for this role
$ vault write -f database/rotate-root/orders-postgres        # rotate Vault's own root password — now not even an operator can read it back out

That last command is worth sitting with: after rotating the root credential, the plaintext root password Vault uses to connect to Postgres is not retrievable by anyone, including a Vault administrator with full access — it exists only inside the encrypted barrier. The same generation-plus-revocation pattern extends to the AWS, GCP, and Azure secrets engines (mint a scoped, short-lived IAM credential per request instead of a standing access key) and to the PKI engine (issue a short-lived TLS certificate instead of managing a long-lived one by hand) — see cryptography & key management for PKI specifically.

The transit engine: encryption as a service

☺ Like you're 10: Instead of your app carrying a key around and doing the locking itself, it hands the thing to lock over to Vault, and Vault hands back the locked version — the key never leaves the room.

Transit is the engine most people underuse, because it doesn't fit the "Vault stores secrets" mental model at all. It stores nothing you give it and generates no credential — it holds a named encryption key and performs encrypt/decrypt/sign/verify operations against it on request, so an application never has direct access to raw key material, and a database compromise of the application's own storage exposes only ciphertext.

$ vault secrets enable transit
$ vault write -f transit/keys/orders-pii-key            # default: aes256-gcm96; also chacha20-poly1305, rsa-4096, ecdsa-p256, ed25519

$ vault write transit/encrypt/orders-pii-key plaintext=$(base64 <<< "4111-1111-1111-1111")
Key            Value
---            -----
ciphertext     vault:v1:8SDd3WHDOjf7mq69CyCqYjBXAiQQAVZRkgUyckLmN2p...

$ vault write transit/decrypt/orders-pii-key ciphertext="vault:v1:8SDd3WHDOjf7mq69CyCqYjBXAiQQAVZ..."
Key          Value
---          -----
plaintext    NDExMS0xMTExLTExMTEtMTExMQ==             # base64 — your app decodes it locally

Your application stores vault:v1:8SDd3W... next to the record it belongs to — Vault's own versioned key never leaves the barrier, and that v1 in the ciphertext isn't decoration. Rotating the key is one command, and it doesn't break anything already encrypted:

$ vault write -f transit/keys/orders-pii-key/rotate      # new key version 2; version 1 ciphertexts still decrypt fine
$ vault read transit/keys/orders-pii-key                  # inspect min_decryption_version / min_encryption_version
$ vault write transit/rewrap/orders-pii-key ciphertext="vault:v1:8SDd3W..."   # re-encrypt under v2, WITHOUT ever exposing plaintext

rewrap is the operation that makes routine key rotation actually practical: a scheduled job can upgrade every stored ciphertext to the latest key version without the application, or a human, ever seeing the underlying plaintext at any point in the process. For data too large to send through the API on every call — a file, a big JSON blob — transit also does envelope encryption via transit/datakey/plaintext/<key>: Vault generates a one-time data-encryption key, returns both the plaintext DEK (used once, in memory, then discarded) and that DEK wrapped under the named transit key (stored alongside the ciphertext it protects). The app does the bulk AES work locally at full speed; only the small wrapped key ever has to round-trip through Vault to be decrypted again later.

Auth methods: how a CI job or a pod actually proves who it is

☺ Like you're 10: A CI job shows a name-tag plus a one-time password it was just handed; a pod shows the ID badge Kubernetes already stapled to it — neither one is carrying a permanent key.

Everything above assumes a caller already has a Vault token. Getting one is where the "no long-lived credential anywhere" principle gets tested hardest, because something has to be the first secret that gets a machine in the door. Vault's answer is different for a CI pipeline than for a Kubernetes pod, and the difference is worth knowing cold.

AppRole — built for CI

AppRole splits authentication into two halves deliberately so that neither one, alone, is a usable credential. RoleID is like a username — it identifies which role is logging in, it isn't secret, and it's fine to bake straight into a pipeline's YAML. SecretID is the actual credential, generated fresh, typically limited to one use, and handed to the pipeline through a wrapping mechanism so that even the CI orchestrator distributing it never sees the plaintext value.

$ vault auth enable approle
$ vault write auth/approle/role/ci-pipeline \
    token_policies="ci-policy" \
    token_ttl=15m \
    token_max_ttl=30m \
    secret_id_ttl=10m \
    secret_id_num_uses=1

$ vault read auth/approle/role/ci-pipeline/role-id
role_id   58b4c650-8a5f-4b1e-9c7d-3f2e1a0b9c8d          # not secret — lives in the pipeline definition

# a trusted controller (not the job itself) mints a wrapped, single-use secret_id
$ vault write -f -wrap-ttl=60s auth/approle/role/ci-pipeline/secret-id
wrapping_token: hvs.CAESIP1...                            # the job receives THIS, not the secret_id

# the pipeline unwraps once — if it's ever unwrapped a second time, that's proof of interception
$ vault unwrap -field=secret_id hvs.CAESIP1...
$ vault write auth/approle/login role_id="58b4c650-..." secret_id="..."
# → a token scoped to ci-policy, dead in 15 minutes even if nobody revokes it

Response wrapping is what keeps this honest: the wrapping token is single-use by design, so if the pipeline's unwrap call ever fails with "wrapping token already used," that's a hard signal something else read the SecretID first — a much stronger detection signal than a static credential ever gives you.

Kubernetes auth — built for pods

A pod already carries an identity Kubernetes itself vouches for: its projected ServiceAccount token, a short-lived JWT mounted automatically at /var/run/secrets/kubernetes.io/serviceaccount/token. The Kubernetes auth method lets Vault validate that JWT directly against the cluster's TokenReview API, so a pod authenticates using an identity it was handed for free, not a secret anyone had to provision.

$ vault auth enable kubernetes
$ vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_SERVICE_HOST:443" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token

$ vault write auth/kubernetes/role/orders-service \
    bound_service_account_names="orders-sa" \
    bound_service_account_namespaces="orders" \
    token_policies="orders-policy" \
    ttl=15m

# from inside the pod — this is what the Vault Agent Injector sidecar automates
$ vault write auth/kubernetes/login \
    role="orders-service" \
    jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token

The binding is what does the real work: this role only issues a token to a JWT claiming to be ServiceAccount orders-sa in namespace orders — steal that JWT from a different pod's filesystem and it authenticates as that pod's identity, scoped to whatever policy that role grants, not an open door to everything. In practice, application teams rarely run vault write auth/kubernetes/login by hand at all: the Vault Agent Injector mutating webhook reads pod annotations and sidecar-injects an agent that logs in, fetches secrets, and writes them to a shared volume before the main container even starts.

# pod annotations — the injector does the login/fetch/render, the app just reads a file
annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "orders-service"
  vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/readonly"

Both auth methods land in the same place — a policy-scoped, TTL-bound token — which is exactly the point: workload identity & pipeline IAM and zero trust for pipelines both build on this "identity Vault already trusts, not a secret someone had to hand out" pattern, and Kubernetes security deep dive covers the ServiceAccount token projection mechanics this auth method depends on.

Day-to-day commands

☺ Like you're 10: Turn it on, prove who you are, read or write a path — almost every session boils down to those three moves.

# cluster lifecycle
$ vault operator init                              # ONE TIME: prints unseal keys + the initial root token
$ vault operator unseal                        # run 3 times (default threshold), by 3 different key-holders
$ vault status                                      # sealed? which storage backend? HA leader?

$ vault login                                # or: vault login -method=approle role_id=... secret_id=...
$ vault token lookup                                 # what am I, and how long have I got left

# KV v2 — note the /data/ and /metadata/ path split this engine adds under the hood
$ vault kv put secret/orders/config username=svc password=hunter2
$ vault kv get secret/orders/config
$ vault kv get -version=2 secret/orders/config       # a prior version — soft-deleted data, still versioned
$ vault kv metadata delete secret/orders/config       # ACTUALLY destroy every version — soft delete alone doesn't

# policy and audit
$ vault policy write ci-policy ci-policy.hcl
$ vault policy read ci-policy
$ vault audit enable file file_path=/var/log/vault_audit.log   # do this before anything else in a real cluster

# ci-policy.hcl — default-deny; only what's listed here is reachable
path "database/creds/readonly" {
  capabilities = ["read"]
}
path "secret/data/orders/*" {          # KV v2 read/write goes through /data/
  capabilities = ["read", "list"]
}
path "secret/metadata/orders/*" {      # `vault kv list` needs THIS path too — a common gap
  capabilities = ["list"]
}

Gotchas and failure modes

☺ Like you're 10: Most of the pain comes from forgetting that Vault says no by default, and that a lease is a promise with a deadline, not a favor.

• Default-deny bites on the KV v2 path split. KV v2 silently namespaces every operation under data/ (read/write the value) and metadata/ (list, version history, permanent destroy) beneath the mount. A policy granting only secret/data/orders/* lets an app read its own secret but makes vault kv list secret/orders/ fail with a permission error that looks like a bug rather than a missing policy line.
• Soft delete isn't gone. vault kv delete removes the current version from view but keeps it recoverable; only vault kv metadata delete (or destroy) actually erases version data. A right-to-erasure obligation needs the latter, not the former.
• A lease nobody renews is a scheduled outage. An application that fetches a database credential once at startup and never renews or re-fetches it will work fine for exactly one TTL, then start failing every query the moment it expires — silently, at 3am, on a schedule nobody wrote down. The Vault Agent (sidecar or template mode) exists specifically to own renewal so application code doesn't have to.
• Sealed means sealed, cluster-wide, and it forgets nothing about why. A restart without auto-unseal configured re-seals every node; someone has to physically supply a threshold of unseal keys before anything is served again. This is the top reason teams configure cloud-KMS auto-unseal for anything running unattended.
• The root token is a break-glass credential, not a login. It bypasses every policy check. The correct lifecycle is: generate it once during vault operator init, use it only to bootstrap the first real auth method and policies, then revoke it — not hand it to a CI system or leave it sitting in a password manager as "the Vault login."
• Dynamic secrets add real load to the backend. A database engine issuing a fresh role per pod restart, under a rolling deploy or an autoscaling event, means real CREATE ROLE/DROP ROLE churn against production Postgres. Watch connection limits and role-creation rate the same way you'd watch any other database load, especially right after a cluster-wide restart.

Vault versus the managed cloud secrets services

☺ Like you're 10: The cloud-native options are easier to turn on if you only live in one cloud; Vault earns its keep the moment you don't, or the moment you want secrets that generate themselves.

Multi-cloud or on-prem databases, an SSH fleet, or a real need for encryption-as-a-service push the decision toward Vault, where the operational cost buys a capability the managed stores don't have at all. A single-cloud shop storing a handful of API keys usually gets less total operational burden from the native managed service. Many mature platforms run both: Vault as the one place a dynamic Postgres or SSH credential gets minted, cloud-native stores for that cloud's own service-to-service secrets. One more thing worth verifying directly rather than assuming: Vault's license moved from MPL 2.0 to the Business Source License in 2023, which prompted the community fork OpenBao under the Linux Foundation, and HashiCorp itself was acquired by IBM (announced 2024) — check current licensing terms and product naming on HashiCorp's own site rather than an older source, including this one.