Secrets & Credential Management

A pipeline is a machine for turning identity into access, over and over, at every stage from a developer's laptop to a production database. That means it is also a machine for creating secrets — passwords, API keys, private keys, tokens — faster than any human can track them by hand. This page goes past "put it in a vault instead of a config file" and into the mechanics: the difference between a static credential and a dynamic one, what actually happens during an automated rotation, how to scope access so a leaked credential is a shrug instead of an incident, why secrets multiply across CI runners and repos no matter how careful anyone is, and the real tradeoffs between a dedicated secrets manager and encrypted config in git. This is deliberately vendor-neutral — the mechanics here are what HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and GCP Secret Manager all converge on, whichever one your org happens to run.

The shape of a secret — and why it sprawls

☺ Like you're 10: A secret is anything that opens a door — a password, a key, a token — and a pipeline has dozens of doors, so it ends up with dozens of keys sitting around in places nobody's watching.

A secret, precisely, is any value whose disclosure grants access: passwords, API tokens, private keys and TLS certificates, database connection strings, cloud access keys, SSH keys, webhook signing secrets, and the encryption keys that protect other secrets. What makes secrets a distinct engineering problem — not just "configuration, but sensitive" — is that a leaked secret doesn't degrade gracefully. A leaked feature flag causes a bug. A leaked database password causes an incident report.

Sprawl is not a discipline failure so much as a structural one: every stage of a pipeline is a different door, needing a different key, with a different owner. Source control needs deploy keys and personal access tokens. CI runners need container registry credentials, cloud provider credentials, and third-party API tokens for whatever SaaS the build integrates with. Build artifacts can end up with configuration baked into an image layer. Runtime needs database passwords and downstream service credentials. Observability tooling needs its own API keys. Chat and paging integrations need webhook URLs that are themselves bearer secrets. Each of these has a different owner, a different blast radius, and — left unmanaged — a different, undocumented rotation cadence of "never."

Where secrets actually leak from

In practice, the same handful of vectors account for almost every real-world exposure: a credential hardcoded in source and committed (often to fix a "quick" merge conflict at 11 p.m.); a value that made it into git history and was later "removed" from the latest commit, which does nothing — the blob is still reachable from an old commit hash; a secret baked into a container image layer, recoverable by anyone who can pull and inspect the image, even from a layer that was later overwritten; a plain env or printenv command in a CI job that dumps every environment variable, secrets included, straight into a build log; and a secret pasted into a chat message or ticket "just for now" during an incident, which outlives the incident by years. None of these require an attacker — they're all things a well-meaning engineer does under time pressure, which is exactly why prevention has to be structural (see containing sprawl below), not a reminder in a wiki page nobody reads under deadline.

The chicken-and-egg version of this problem is worth naming on its own: something has to authenticate to the secrets store itself before it can fetch anything, so how do you avoid that bootstrap credential becoming just another static secret sitting in a config file? The answer nearly every mature setup converges on is platform-provided identity — an EC2 instance profile, a Kubernetes service account token, a GitHub Actions OIDC token — something the runtime environment hands the workload for free, that a secrets broker can verify without the workload ever holding a standing credential for the broker itself. That single move, trusting an identity instead of distributing a bootstrap secret, is the foundation the rest of this page builds on.

Static vs. dynamic credentials

☺ Like you're 10: A static credential is the same key handed out again and again until someone remembers to change the locks. A dynamic one is a fresh key cut on the spot that stops working on its own.

A static credential is a fixed value — a password, an API key, a long-lived IAM access key pair — that stays valid, unchanged, from the moment it's issued until something explicitly rotates or revokes it. It's simple to reason about and simple to cache, which is exactly why it's the default every tool reaches for first. It's also why static credentials are the majority of what shows up in breach postmortems: the value is valid indefinitely, so the exposure window is however long it takes someone to notice, which for an unmonitored service account can be measured in years, not hours.

A dynamic credential is generated on demand, scoped to one requester, and time-bound by a lease with a TTL (time-to-live) — after which it simply stops working, no revocation step required. HashiCorp Vault's database secrets engine is the reference implementation: instead of storing a shared Postgres password, Vault holds a privileged connection to the database and, on each request, creates a brand-new database user with a randomly generated password and a short TTL, then drops that user automatically when the lease expires or is revoked. Two workloads that both ask for "read access to the checkout database" get two different usernames, which means an audit trail down to the individual request, not just "someone with the shared password."

Dynamic doesn't win automatically. It costs you a live dependency on the broker at request time, credentials that can expire mid-use if a consumer holds one longer than expected, and more moving parts than "read an environment variable." Static wins when a workload genuinely needs a value to be stable for a long time — a signing key whose public counterpart is pinned elsewhere, for instance — or when the consuming system simply can't participate in a lease-renewal handshake. The practical rule: default to dynamic wherever the consumer can support it, and treat every remaining static credential as a thing that needs an explicit, scheduled rotation plan precisely because nothing is going to expire it for you.

The secrets lifecycle, as one loop

☺ Like you're 10: A secret isn't just "stored" — it's born, kept safe, handed out carefully, used quietly, replaced on a schedule, and killed instantly if something goes wrong. Miss any one step and the rest don't matter.

Treating "secrets management" as a single verb hides six distinct jobs, each with its own failure mode. Generate — how strong is the value, and where was it actually created (never hand-typed by a human who now also remembers it). Store — encrypted at rest, in a system built for this and not a general-purpose database or a spreadsheet. Distribute — scoped to exactly the identity that needs it, ideally short-lived. Use — held in memory only, never written to disk or a log. Rotate — replaced on a schedule or on suspicion, without breaking every consumer at once. Revoke — killed immediately, out of band, the moment compromise is suspected. An audit thread runs underneath all six: every request, grant, and expiry timestamped, because "who had access to this credential on the day it was misused" is a question you either can answer instantly or can't answer at all.

Notice what's missing from most teams' mental model: generate and revoke are usually the two weakest links. Generation is weak because a human typing "a memorable but strong password" into a form is worse at entropy than a machine-generated 32-byte random value, every time, and yet it's still the default for anything created outside a secrets tool. Revocation is weak because most systems are built to rotate on a schedule, not to kill a specific credential right now — which is precisely the capability you need in the ten minutes after a leak is discovered, and precisely the one teams discover they don't have while they're discovering the leak.

Automated rotation without breaking everything downstream

☺ Like you're 10: If you swap the lock the instant you cut the new key, everyone still walking to the door with the old key gets locked out. Good rotation makes both keys work for a little while, so nobody's caught mid-walk.

Rotation sounds simple — generate a new value, throw away the old one — until you count how many places the old value is cached: a running process's memory, a CI variable, a config map, a colleague's local .env file. A hard cutover rotates the source of truth and instantly breaks every consumer that hasn't picked up the new value yet, which is how "we rotated the database password" turns into a self-inflicted outage. The fix nearly every rotation system converges on — this is the general shape behind AWS Secrets Manager's rotation Lambda contract, and it's worth knowing in its vendor-neutral form even if you never write an AWS-specific one — is a four-step, dual-validity rotation:

1. createSecret  — generate a new credential value/version. The OLD value
                    stays fully valid. Nothing downstream has changed yet.

2. setSecret     — push the new value to the system it authenticates to
                    (e.g. ALTER USER ... PASSWORD on the database). Both
                    old and new credentials now authenticate successfully.

3. testSecret    — connect using the NEW value end-to-end (not just "did
                    the API call succeed" — an actual login/query) before
                    telling anything to depend on it.

4. finishSecret  — mark the new version as current. Consumers that ask
                    for "the current secret" now get the new one. The old
                    version is kept, still valid, for a grace period —
                    then explicitly invalidated once nothing is using it.

The property that makes this safe is the window in step 2 through the grace period after step 4, where both values authenticate successfully. A consumer that cached the old value at 9:00 a.m. and one that fetched the new value at 9:05 a.m. both keep working, right up until the grace period ends — at which point anything still holding the old value has had a real, bounded amount of time to notice and re-fetch, not zero. Skipping straight from "generate" to "the old value is now dead" is the single most common way a well-intentioned rotation causes the incident it was trying to prevent.

Cadence is its own tradeoff. Rotate too rarely and you're back to a de facto static credential with a long exposure window if it leaks between rotations. Rotate too aggressively and every rotation becomes a small, recurring chance to break a consumer that wasn't tested against the dual-validity window — cost and fragility, for a security benefit that's already mostly captured by the first rotation interval you chose. A common, defensible default for credentials that can't go fully dynamic is 30–90 days, tightened for anything with a wide blast radius (a root-equivalent cloud credential) and relaxed for anything low-sensitivity. The cleanest way out of the whole tradeoff is the one from the previous section: a dynamic credential with a 15-minute lease doesn't need a rotation schedule at all — every single request is already, functionally, a rotation.

Least-privilege scoping, stage by stage

☺ Like you're 10: The person who tests the recipe shouldn't have a key to the whole restaurant — give every job exactly the drawer it needs, nothing more.

Rotation limits how long a leaked credential works. Scoping limits what it can do while it's valid — and the two together are what actually shrink blast radius, because a perfectly rotated credential that still grants admin on every environment is still a very bad leak. Scoping has four independent axes, and a mature secrets setup answers all four for every credential it issues, not just one:

Applied to a pipeline, this means every stage gets its own answer, not one shared "CI credential" that happens to work everywhere because nobody scoped it down. A build stage needs to read source and write build/test logs — it does not need deploy access. A test stage needs a throwaway, TTL-scoped database it can freely mutate. A deploy stage needs write access to exactly one target environment, and for cross-account or cross-cluster deploys, that's usually implemented as a short-lived assumed role rather than a standing credential to every environment at once. A running service needs only its own secrets, not its neighbors'. Security & Compliance covers this same idea from the identity-and-policy side — permissions boundaries, resource policies, condition keys — for teams running on AWS specifically; this page is the vendor-neutral version that applies whichever secrets backend and cloud you're on.

Scoping in a policy language

Whatever backend you use, scoping usually resolves to a path- or namespace-based access policy attached to an identity. A Vault policy for a deploy role that can only read production secrets for one service — and only mint read-only database credentials for it — looks like this:

# checkout-prod-deploy.hcl — attached to the checkout service's deploy identity only
path "secret/data/checkout/prod/*" {
  capabilities = ["read"]
}

path "database/creds/checkout-readonly" {
  capabilities = ["read"]     # mints a fresh, TTL-bound DB user on every read
}

# everything not explicitly listed is denied by default — the policy language
# is allow-list only, which is what makes "forgot to scope this" fail closed

The load-bearing detail is the last comment: a well-designed policy language is deny-by-default, so a path nobody thought to add is unreachable rather than silently wide open. That's the opposite failure mode from IAM-style systems where an overly broad wildcard silently grants far more than intended — worth checking explicitly whichever backend you're on, because the two failure directions require opposite habits to catch.

Dedicated secrets manager vs. encrypted config

☺ Like you're 10: You can lock your valuables in a bank vault with a guard who logs every visit, or in a home safe you keep in the closet — both work, but they trade convenience for oversight in opposite directions.

Two broad approaches solve "where does the secret actually live," and they are not the same tool wearing different logos — they make genuinely different tradeoffs. Encrypted config keeps secrets as ciphertext committed alongside code — SOPS (backed by a cloud KMS, PGP, or age key), git-crypt, Ansible Vault, or Kubernetes' Sealed Secrets controller, which encrypts a Secret client-side so only the controller running in-cluster can decrypt it. The secret travels through the exact same git history, pull-request review, and versioning as everything else, and there is nothing new to stand up or keep available. A dedicated secrets manager — HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager — is a purpose-built service: centralized policy, per-request audit logging, native dynamic secrets, and built-in rotation integrations, at the cost of being a live runtime dependency your workloads now need a network path and an identity to reach.

The two are not mutually exclusive, and the strongest real-world setups usually layer them rather than picking one. A common pattern: use encrypted config for a small number of low-sensitivity bootstrap values — the ones needed before anything else can start, like the auto-unseal configuration that lets a Vault cluster reach its own cloud KMS key on boot — and a dedicated secrets manager for everything downstream of that bootstrap. That way the "master key" problem — whoever holds the decryption key for your encrypted config can read everything it protects — is scoped to the smallest possible set of values, and the bulk of your actual application secrets live somewhere with per-request audit logging and native rotation.

Containing sprawl: detection, ephemeral runners, and the leak playbook

☺ Like you're 10: You can't stop every accidental slip, so build the safety net instead — a robot that reads every commit before it lands, machines that forget everything the second the job ends, and a fire drill for the day something still gets out.

Given how many ordinary, well-intentioned actions leak a secret (see the vectors above), prevention has to be automated and structural, not a policy people are expected to remember under deadline pressure. Four practices carry most of the real-world weight:

Scan before it lands, not after. Tools like gitleaks, TruffleHog, and detect-secrets pattern-match against known credential shapes (AWS access key prefixes, private key headers, high-entropy strings) and run in two places for two different jobs: a pre-commit hook catches an accidental commit on the developer's own machine before it ever leaves their laptop, and a required CI check — scanning the full diff, not just the latest commit — catches whatever the pre-commit hook was skipped or bypassed for. Wire the CI check to fail the build, not just warn; a warning that nobody reads is not a control. This is the specific mechanic Shift-Left Security for DevOps covers in the broader context of catching problems before merge rather than after deploy.

Make CI runners forget. A long-lived, self-hosted runner that executes job after job on the same disk accumulates cached credentials, leftover environment files, and build artifacts across jobs that were never supposed to share state. Ephemeral, per-job runners — a fresh container or VM torn down the instant the job finishes — bound the lifetime of anything that leaked into that environment to the length of a single job, and they mean a compromised job can't read what the previous job's credentials left behind.

Assume every log is public. Most CI systems auto-mask a value once it's registered as a secret, replacing it with *** in log output — but that masking only works for values the system knows about. A raw env dump, a set -x trace around a command that embeds a credential in its arguments, or a secret concatenated into a longer string before printing all defeat it, because the platform never sees the exact substring to redact. Treat build logs as a public artifact by default: never print environment variables wholesale, and prefer passing secrets as masked/secret-typed pipeline variables or short-lived files over plain environment variables wherever the CI platform supports the distinction.

Prefer proving identity over shipping a credential at all. The OIDC federation pattern — a CI platform's short-lived job token, exchanged for a cloud credential via a trust policy scoped to a specific repo and branch — eliminates an entire category of standing secret from ever existing in CI in the first place, because there's no long-lived cloud access key sitting in a repo secret for anyone to leak. This is the same identity-first move from the start of this page, applied specifically to the CI-to-cloud hop.

When prevention fails anyway — and eventually it will — the response is the same regardless of which vector caused it, and speed matters more than diagnosis: revoke or rotate the credential immediately, before spending time figuring out exactly how it leaked; check the access audit trail for the exposure window, looking specifically for use from unfamiliar identities, IPs, or times; scrub git history if the leak was a commit, as a cleanup step, not the fix itself; and route it through incident response like any other production event rather than a quiet, undocumented fix — see incident management for the blameless-postmortem process this should actually go through. A leaked secret that gets rotated in ten minutes and never discussed again teaches the org nothing about why it happened; the same leak run through a real postmortem is how the scanning rule that would have caught it earlier actually gets written.

Secrets and credential management is not a bolt-on to the rest of this course — it's a thread running underneath nearly every page you've already read. It borrows the desired-state reconciliation you saw in configuration management and infrastructure as code (a provisioning tool itself needs credentials to talk to a cloud API, and that bootstrap credential deserves the same scrutiny as anything it provisions), it's the specific mechanism behind the "shared responsibility" identity story in Security & Compliance, and it feeds directly into Supply-Chain Security & SBOM the moment you start signing artifacts with keys that are, themselves, secrets to manage. Practice the mechanics hands-on in Drill — Secure a Vulnerable Pipeline and Capstone Part 6 — Security Hardening; if you want to go deep on one backend specifically, HashiCorp Vault covers the tool, and the Vault Associate certification page has the exam-track version of everything on this page (verify current exam format and pricing on HashiCorp's own certification page before you commit to a study plan).