HashiCorp Vault

Vault is the dedicated tool behind everything Secrets & Credential Management covers conceptually: static vs. dynamic credentials, rotation, least-privilege scoping, and encryption. That page stayed vendor-neutral on purpose. This one picks a specific vendor and goes to the metal — how Vault actually stores data, what "sealed" and "unsealed" really mean, how a request proves who it is, how a policy decides what that identity can touch, and where a secret engine turns a policy-approved request into an actual credential, certificate, or ciphertext. Vault is the reference implementation cited throughout the earlier page for a reason: most other secrets managers converge on some version of the same three ideas — an encrypted store nobody can read directly, identity-based access instead of a shared password, and secrets minted on demand instead of stockpiled in advance.

What Vault is, and the problem it solves

☺ Like you're 10: Vault is a server that locks secrets away, only opens for someone who proves who they are, and usually hands out a temporary pass instead of the permanent key.

HashiCorp released Vault in 2015 to solve a problem that predates it by decades: every nontrivial system accumulates credentials — database passwords, API keys, TLS private keys, cloud access keys — and the default place they end up is wherever it was easiest to put them at 5 p.m. on a Friday: a config file, an environment variable, a wiki page, a Slack DM. Vault is a standalone server (not a library, not a sidecar-only pattern, though sidecars exist on top of it) whose entire job is to be the one place secrets live, gated behind authentication and policy, with every access logged.

Three properties separate it from "an encrypted database with an API," which is the reductive but common misreading: it does identity-based access instead of a single shared credential to the vault itself (see auth methods below); it can generate secrets on demand rather than only storing ones a human created (see secret engines); and it treats encryption as an operation, not just a storage property — an application can ask Vault to encrypt or decrypt a value without Vault ever handing over the key that did it. All three ideas point the same direction: minimize how much any single credential, anywhere in your system, is worth if it leaks.

Vault is one piece of a broader HashiCorp toolchain: Terraform provisions infrastructure and needs credentials to talk to a cloud API, Consul handles service networking and can double as a storage backend for Vault itself, and Packer builds images that may need to fetch build-time secrets. This page covers Vault alone; where it overlaps with another tool's territory, that boundary is called out explicitly rather than re-explained.

Architecture: the storage backend and the seal/unseal barrier

☺ Like you're 10: Everything Vault ever writes to disk is scrambled first, and the key that unscrambles it only exists in memory, rebuilt by combining pieces held by different people.

Vault's server process separates into two things that are easy to conflate: the Vault core — the running process that handles requests, enforces policy, and does encryption — and the storage backend, which is just where encrypted bytes live. Vault never trusts the storage backend to keep anything secret. Whatever engine is behind it — Raft (Vault's own built-in "integrated storage," the default and recommended choice since Vault 1.4), Consul (the original and still-supported backend, useful if you're already running Consul for service discovery), or a handful of others (file storage for single-node non-HA setups, cloud object stores) — it only ever holds ciphertext. Someone with raw filesystem access to a Raft data directory, or raw KV access to Consul, gets nothing readable.

The thing that makes that true is called the barrier. Every value Vault writes is encrypted with an AES-256-GCM encryption key before it ever reaches storage, and that encryption key is itself encrypted at rest with a root key. On a fresh install, vault operator init generates the root key once and immediately splits it using Shamir's Secret Sharing into N key shares, of which any threshold number (a common default is 5 shares, threshold 3) can reconstruct it — no smaller subset can, and no single share reveals anything about the key on its own. When a Vault server process starts (or restarts), it comes up sealed: the encrypted data is right there in storage, but the root key doesn't exist anywhere except in the memory of an already-unsealed node. vault operator unseal, run once per key share up to the threshold, feeds shares in one at a time; once enough have been submitted, Vault reconstructs the root key in memory (never written to disk), uses it to decrypt the encryption key, and the barrier opens.

Auto-unseal replaces the human-held key shares with an external key management service: on startup Vault calls out to AWS KMS, Azure Key Vault, GCP Cloud KMS, or another Vault cluster's Transit engine, which unwraps the root key automatically. This is what makes Vault viable in autoscaling groups and disaster-recovery failover — nobody has to be paged at 3 a.m. to type in a key share when a node reboots — at the cost of moving trust from "three humans in different time zones" to "whoever controls that one KMS key." Auto-unsealed clusters still generate recovery keys (the Shamir-style equivalent, used for operations like generate-root or rekey rather than routine startup) so the human-quorum option isn't lost entirely, just moved off the startup path.

High availability and the Raft/Consul choice

Both Raft and Consul support running Vault as a cluster: one node is active and serves all writes, the rest are standby and either redirect or forward requests to the active node (Vault Enterprise's Performance Standby nodes can also serve reads locally). Raft implements leader election itself, using the same Raft consensus algorithm Consul and etcd are built on, and Vault's own autopilot feature manages node health and safe removal — which is why Raft has become the default recommendation: one fewer system (Consul) to run and reason about. Consul remains a legitimate choice when you're already operating a Consul cluster for service discovery and don't want a second consensus system. Either way, cluster size should be an odd number (3 or 5 is typical) so quorum math has no ties.

Secret engines: where the actual secrets come from

☺ Like you're 10: Vault doesn't have one kind of drawer — it has different drawers for different jobs: one just stores things, one hands out fresh database logins, one hands out short-lived certificates, and one only ever encrypts and decrypts for you without showing you the key.

A secret engine is mounted at a path (vault secrets enable -path=secret kv-v2) and defines what happens when something reads or writes under that path. Different engines behave completely differently — some just store what you give them, some generate something new on every request — and understanding which kind you're using is most of what separates "Vault as a fancy encrypted file" from "Vault as a credential factory."

KV v2 — static secrets, versioned

The Key/Value version 2 engine is the closest thing to "just store a secret." It's a straightforward path-addressed store, but with real version history baked in: every write creates a new numbered version rather than overwriting the old one, deletes are soft (recoverable with vault kv undelete) until an explicit destroy, and writes support check-and-set (-cas) so two concurrent writers can't silently clobber each other. This is the engine most teams reach for first, and it's the right default for values that genuinely are static — a third-party API key with no rotation API, a webhook signing secret.

$ vault secrets enable -path=secret kv-v2
$ vault kv put secret/checkout/prod db_host=prod-db.internal api_key=sk_live_...
$ vault kv get secret/checkout/prod
$ vault kv get -version=3 secret/checkout/prod     # read an older version
$ vault kv metadata get secret/checkout/prod         # version history, without the values
$ vault kv delete secret/checkout/prod               # soft delete — recoverable
$ vault kv undelete -versions=4 secret/checkout/prod
$ vault kv destroy -versions=1,2 secret/checkout/prod  # permanent, for those specific versions

Database secrets engine — dynamic, per-request credentials

This is the engine Secrets & Credential Management used as its reference example for dynamic secrets, and it's worth seeing configured for real. You give Vault one privileged "root" connection to the database once; from then on, every read against a configured role creates a brand-new database user with a randomly generated password and a bounded lifetime, and drops that user automatically when the lease expires or is explicitly revoked.

$ vault secrets enable database
$ vault write database/config/checkout-postgres \
    plugin_name=postgresql-database-plugin \
    connection_url="postgresql://{{username}}:{{password}}@prod-db.internal:5432/checkout" \
    allowed_roles="checkout-readonly" \
    username="vault-admin" password="…rotate this immediately after setup…"

$ vault write database/roles/checkout-readonly \
    db_name=checkout-postgres \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
      GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" max_ttl="24h"

$ vault read database/creds/checkout-readonly
# Key                Value
# lease_id            database/creds/checkout-readonly/2f3a9c...
# lease_duration      1h
# username            v-token-checkout-r-a1b2c3d4e5
# password            A1b2-generated-randomly-9f8e

Two requests against checkout-readonly yield two different usernames — which means an audit trail down to the individual request, not "someone with the shared password." Postgres, MySQL, MSSQL, MongoDB, and several other databases have first-party plugins; the shape (a privileged root connection, a role, a creation-statement template) is the same across all of them.

PKI secrets engine — certificates instead of a certificate you forgot to renew

The PKI engine turns Vault into a certificate authority: mount it once, either generate or import a root (or intermediate) CA, define a role constraining what a certificate request is allowed to ask for (allowed domains, max TTL, key type), and issue short-lived TLS certificates on demand — often minutes to hours old rather than the year-plus lifetime a manually issued cert usually gets. The point isn't just automation; it's that a certificate valid for six hours makes "the private key leaked three weeks ago" a non-event by the time anyone notices.

$ vault secrets enable pki
$ vault write pki/root/generate/internal common_name="acme.internal" ttl=87600h
$ vault write pki/roles/checkout-svc \
    allowed_domains="checkout.acme.internal" allow_subdomains=true \
    max_ttl="24h"
$ vault write pki/issue/checkout-svc common_name="checkout.acme.internal" ttl="4h"
# returns certificate, private_key, and ca_chain in one response

Transit — encryption as a service

The Transit engine is the one genuinely different idea on this list: it stores nothing of the caller's. An application sends plaintext to Vault, Vault encrypts it with a named key it manages entirely server-side, and hands back ciphertext — the raw key material never leaves Vault, ever, in either direction. The same engine handles decryption, HMACs, signing/verification, and key rotation (new encryption uses the newest key version; old versions stay available to decrypt data encrypted under them, and a min_decryption_version setting can retire old versions once nothing needs them).

$ vault secrets enable transit
$ vault write -f transit/keys/checkout-pii

$ vault write transit/encrypt/checkout-pii plaintext=$(base64 <<< "4111-1111-1111-1111")
# ciphertext: vault:v1:8SDd3WHDOjf7mq69CyCVsQ...

$ vault write transit/decrypt/checkout-pii ciphertext="vault:v1:8SDd3WHDOjf7mq69CyCVsQ..."
# plaintext (base64-decode to recover the original)

$ vault write transit/rotate/checkout-pii        # new writes use v2; v1 still decrypts old data

This is the pattern usually called encryption as a service: application code gets to "just encrypt this field" without ever managing a key, rotating one, or being trusted with key material at all — which is a meaningfully smaller blast radius than an app holding its own AES key in an environment variable. It overlaps with what a cloud KMS does (AWS KMS, Azure Key Vault's key operations, GCP Cloud KMS); the case for Transit specifically is a single, cloud-agnostic API and audit trail across a multi-cloud estate, or simply keeping encryption operations inside the same system already handling every other secret.

Cloud and infrastructure engines

The AWS, Azure, and GCP secrets engines apply the database engine's exact pattern to cloud IAM: Vault holds a privileged credential (or an assumable role) once, and mints short-lived, scoped access keys or tokens per request — an sts:AssumeRole-backed temporary AWS credential instead of a long-lived IAM access key pair sitting in a CI variable forever. The SSH engine issues signed, short-lived SSH certificates or one-time passwords in place of distributing a static private key to every engineer who might need shell access to a box.

Auth methods and identity: proving who's asking

☺ Like you're 10: Before Vault hands anything to anyone, it needs proof of who's asking — a robot proves it with a certificate its platform already gave it, a human usually proves it by logging into their company account.

Every request to Vault must first authenticate, and authentication always ends the same way regardless of method: Vault issues a token, bound to one or more policies and a TTL. Everything after that — reading a secret, requesting a dynamic credential, calling Transit — is authorized against that token's policies, not against whatever credential was used to get it. The auth method's only job is answering "is this really who it claims to be," and different callers need genuinely different proof.

AppRole is worth walking through concretely because it's the default answer for "how does a CI pipeline authenticate to Vault," and its two-part design exists specifically to solve a bootstrap problem: the RoleID can be committed to a pipeline config without much risk (it identifies a role, it doesn't grant anything by itself), while the SecretID is the actual credential and is usually delivered separately — pulled by a trusted orchestrator, generated as single-use, or wrapped.

$ vault auth enable approle
$ vault write auth/approle/role/checkout-ci \
    token_policies="checkout-prod-deploy" \
    token_ttl=15m token_max_ttl=1h \
    secret_id_ttl=10m secret_id_num_uses=1

$ vault read auth/approle/role/checkout-ci/role-id
# role_id   f3b1c9d2-...    — fine to store in the pipeline config

$ vault write -f auth/approle/role/checkout-ci/secret-id
# secret_id   8a2e91f4-...  — the actual credential; deliver this narrowly, not as a variable

Response wrapping: getting "secret zero" somewhere safely

Every one of these methods still has a bootstrap problem: something has to hand the SecretID, the initial token, or another piece of "secret zero" to the caller in the first place, over some channel. Response wrapping is Vault's answer — instead of returning a value directly, an operation can return a single-use wrapping token, which the real recipient exchanges (vault unwrap) for the actual value exactly once. If a wrapping token has already been unwrapped by someone else — an intermediate system, an attacker who intercepted a build log — the legitimate recipient's unwrap attempt fails loudly instead of silently handing over a value someone already read, which turns "was this ever intercepted" from an unanswerable question into a hard, immediate signal.

Policies: what an identity is allowed to do

☺ Like you're 10: A policy is a list of doors an identity is allowed to open, written down explicitly — anything not on the list stays locked by default.

A policy is an HCL document naming paths and the capabilities (create, read, update, delete, list, plus the special sudo and explicit deny) allowed on each. Policies attach to tokens — directly, or via the auth method that issued the token, as in the AppRole example above where checkout-ci maps to token_policies="checkout-prod-deploy". The engine is deny-by-default: a path with no matching policy statement is unreachable, not silently wide open, which is the opposite failure direction from IAM-style systems where an overly broad wildcard can silently grant more than intended.

# checkout-prod-deploy.hcl
path "secret/data/checkout/prod/*" {
  capabilities = ["read"]
}

path "database/creds/checkout-readonly" {
  capabilities = ["read"]        # mints a fresh, TTL-bound DB user on every read
}

path "transit/encrypt/checkout-pii" {
  capabilities = ["update"]      # Transit's write-shaped API uses "update" for encrypt/decrypt calls
}

# everything else: denied, by omission

Policy paths can also be templated against the caller's own identity — path "secret/data/{{identity.entity.aliases.auth_kubernetes.metadata.service_account_name}}/*" lets one policy correctly scope a thousand different pods to their own namespace's secrets without writing a thousand policies. That relies on Vault's identity subsystem, which links aliases from different auth methods (a Kubernetes service account, an LDAP username) back to one canonical entity, so "who is this, really" has one consistent answer no matter which door they came through.

$ vault policy write checkout-prod-deploy checkout-prod-deploy.hcl
$ vault policy read checkout-prod-deploy
$ vault policy list
$ vault token create -policy=checkout-prod-deploy -ttl=1h    # for testing a policy by hand

Day-to-day commands

☺ Like you're 10: A handful of commands cover almost everything: start it, unlock it, log in, read or write a secret, and check on a lease.

# lifecycle
$ vault operator init                     # ONCE, on a fresh cluster — prints unseal keys + initial root token
$ vault operator unseal                   # run once per key share, up to the threshold
$ vault status                            # sealed?, HA mode, active node, storage type
$ vault operator rekey                    # rotate the unseal keys / recovery keys
$ vault operator rotate                   # rotate the encryption key used for NEW writes (old data still decrypts)

# auth
$ vault login -method=oidc                # human SSO login, opens a browser
$ vault login -method=approle role_id=... secret_id=...

# secrets
$ vault secrets list                      # every mounted engine and its path
$ vault kv get secret/checkout/prod
$ vault read database/creds/checkout-readonly

# leases — the thing dynamic secrets are actually built on
$ vault lease renew database/creds/checkout-readonly/2f3a9c...
$ vault lease revoke database/creds/checkout-readonly/2f3a9c...
$ vault lease revoke -prefix database/creds/checkout-readonly   # kill EVERY lease under a role, now

# audit — turn this on before you need it, not after
$ vault audit enable file file_path=/var/log/vault_audit.log

Gotchas and failure modes

☺ Like you're 10: Most Vault incidents aren't Vault being wrong — they're dev mode left running, a root token nobody revoked, or a lease clock nobody was watching.

Dev mode is not a smaller version of production — it's a different thing entirely

vault server -dev runs a single node, in-memory, permanently unsealed, with an all-powerful root token printed straight to the terminal. It exists for exactly the fifteen-minute workshop above, and it's genuinely useful for that. It becomes an incident the moment someone leaves it running past that — in-memory storage means a restart silently discards every secret in it, and a permanently unsealed root-token server reachable on the network is one of the more avoidable ways to lose an entire secrets estate at once.

Root tokens are a liability the moment setup is done

The initial root token from vault operator init can do anything, including reading and rewriting policy — it's meant for bootstrapping, not daily operation. Best practice is to use it immediately to create a properly scoped admin policy and a token or auth method tied to a real human identity, then revoke the root token (vault token revoke) rather than let it sit active. A standing root token discovered during an audit, months later, unable to be tied to a specific person's login, is a finding every serious security review flags.

Lease expiry storms

If Vault itself is unreachable for a stretch — an outage, a botched upgrade — dynamic leases keep counting down regardless, because the TTL clock isn't Vault checking in, it's just time passing. When Vault comes back, everything that expired while it was down needs re-issuing at once, and every consumer that was quietly relying on auto-renewal discovers it simultaneously: a stampede of new database user creation, new certificate issuance, all at once, against systems that weren't sized for it. The mitigation is the same shape as any thundering-herd problem — jittered renewal instead of everyone renewing at the same TTL boundary, monitoring on "leases expiring in the next N minutes" as a leading indicator rather than finding out from the stampede itself, and consumers built to tolerate a renewal failing occasionally rather than treating one missed renewal as fatal.

Forgetting that KV v2 needs "data" in the API path

KV v2's actual HTTP API nests reads and writes under a data/ segment (and metadata under metadata/) that the vault kv CLI subcommand quietly adds for you — which is exactly why a hand-written policy path of secret/checkout/* (missing data/) silently matches nothing against KV v2, while the same policy would have been correct against the older KV v1 engine. This single missing path segment is one of the most common "why is my policy not working" reports; when a policy denies something it should allow, check the mount's KV version before anything else.

Transit is not a substitute for real secrets management, and vice versa

Transit encrypts and decrypts; it does not remember what it encrypted, does not version application data, and isn't where you'd store the plaintext at all — that's still the application's or KV's job, with Transit sitting in front of the write. Conversely, using KV to hand an application its own raw AES key so the application can encrypt things itself reintroduces exactly the key-management burden Transit exists to remove. Pick one job per engine; using KV where Transit belongs (or the reverse) is a common early design mistake.

Vault vs. its alternatives

☺ Like you're 10: Other secrets managers do similar jobs — the real choice is usually about which cloud you're already living in versus how many clouds you need one system to cover.

The practical rule mirrors the one Terraform reaches for its own comparison: pick the cloud-native option when you're genuinely single-cloud and want the smallest number of moving parts, and pick Vault when you're multi-cloud, need dynamic secrets beyond what a cloud vendor's rotation templates cover, or want one consistent audit trail and policy language across every environment you run. Plenty of real organizations run both — a cloud-native secrets manager for that cloud's own resource credentials, Vault for everything that has to work the same way across clouds.

Vault is the concrete version of nearly everything Secrets & Credential Management argued for in the abstract, and it shows up as a supporting character across the rest of this course: Ansible can pull secrets from it instead of a static encrypted vault file, Jenkins and CircleCI both integrate with it to avoid storing long-lived pipeline credentials, Terraform can inject secrets at apply time instead of writing them into HCL or state, and Kubernetes workloads use it as the real encryption layer a bare Secret object doesn't provide on its own. Practice the mechanics hands-on in Drill — Secure a Vulnerable Pipeline and Capstone Part 6 — Security Hardening. If you're building toward a credential specifically for this tool, see Vault Associate — as with any vendor exam, verify the current format and pricing on HashiCorp's own certification page before committing to a study plan, since both shift over time.