Secrets management

Every pipeline runs on secrets — database passwords, cloud API keys, signing tokens — and how you store and hand out those secrets determines how bad a single leaked credential actually is. This page covers why hardcoded secrets are a standing liability, how a centralized secrets manager and short-lived credentials shrink the blast radius of a leak, and why rotation and pre-commit scanning have to run continuously, not once.

Why hardcoded secrets are a standing liability

A secret pasted directly into source code or a config file doesn't just risk exposure once — it becomes part of the codebase's permanent record. Git stores every commit as an immutable object; deleting a secret in a later commit and even force-pushing over the branch tip does not remove it from local clones, forks, CI caches, or any mirror that already pulled the history. The only reliable fix after a leak is to treat the credential as compromised and rotate it — scrubbing history with tools like git filter-repo is a cleanup step, not a substitute for rotation.

Hardcoded secrets leak through more channels than a careless git add. Verbose application logs and unhandled exceptions routinely dump connection strings or auth headers into log aggregators that far more people can read than the source repo itself. CI/CD systems are a second major leak surface: build logs echo environment variables by default unless the platform explicitly masks them, and a misconfigured CI/CD pipeline step can print a secret to stdout on every run. Once a secret is baked into an artifact — a container image layer, a compiled binary, a public npm package — it ships to everyone who pulls that artifact, including anyone who mirrors it after you've "fixed" it upstream.

The operational cost compounds the security cost. A hardcoded secret is usually referenced in more than one place — a dozen microservices reading the same .env value, or a key copy-pasted into three different repos — so rotating it means finding every reference and redeploying every consumer in lockstep. Teams routinely skip rotation because it's manual and risky, which is exactly the incentive a centralized secrets manager exists to remove.

Centralized secrets managers

A secrets manager — HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and Google Secret Manager are the tools you'll see most in production — is a dedicated service that stores secrets encrypted at rest, controls access through policy, and injects them into applications and pipelines at runtime instead of at build time. The application never has the secret's value on disk in source control; it authenticates to the secrets manager (via an IAM role, a Kubernetes service account, or a short-lived token) and requests the value it needs when it starts up or when a request requires it.

This buys you three things a flat file or environment file checked into a repo cannot: centralized access policy (who and what can read which secret, enforced in one place instead of scattered file permissions), a full audit log of every read (who accessed which secret and when, which is often the first thing an incident responder pulls during incident response), and encryption plus versioning managed by infrastructure built for exactly that job rather than bolted on. Kubernetes-native tools like External Secrets Operator or the Vault Agent Injector extend this pattern into clusters, syncing secrets from the manager into pods without ever writing them into a Kubernetes Secret object in plaintext YAML.

Dynamic, short-lived credentials

The deeper shift a secrets manager enables is moving from static credentials to dynamic ones. A static credential — a database password you set once and use for a year — is valid until someone remembers to change it, so a leak from six months ago is still exploitable today. A dynamic credential is generated on demand with a time-to-live (TTL) measured in minutes or hours: Vault's database secrets engine, for example, can create a brand-new PostgreSQL user with scoped permissions for each application instance, then automatically revoke it when the lease expires.

The security payoff is blast radius. If a static API key leaks, it is valid until someone notices and manually revokes it — which, per Verizon's annual Data Breach Investigations Report, regularly takes weeks. If a short-lived credential with a 15-minute TTL leaks, an attacker has, at most, a 15-minute window before it expires on its own, and the exposure window is bounded by design rather than by how fast a human responds. Cloud providers apply the same idea at the infrastructure layer: AWS STS AssumeRole and workload identity federation (GCP, Azure AD) issue temporary credentials to a workload's identity instead of embedding a long-lived access key, which is why modern cloud guidance treats a hardcoded IAM access key as something close to a code smell.

// BAD — secret hardcoded directly in application config,
// now permanently readable in git history and this file
const dbConfig = {
  host: "prod-db.internal",
  user: "app_service",
  password: "Tr0ub4dor&3-prod-2024",   // <- real value, checked into git
  apiKey: "sk_live_51H8x9K2mN7pQrStUvWx"
};

// GOOD — value comes from an environment variable populated
// at runtime by the secrets manager (Vault agent, ESO, etc.);
// nothing sensitive ever touches the repo
const dbConfig = {
  host: process.env.DB_HOST,
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,   // injected at container start
  apiKey: process.env.API_KEY          // short-lived, auto-rotated
};

Rotation as policy, not cleanup

Rotation is the practice of periodically replacing a secret's value even when there's no known leak, and it needs to be a scheduled, automated policy rather than something that only happens after an incident. Treating rotation as a one-time cleanup — "we rotated it after the breach, we're done" — misses the point: the goal is to keep the exposure window small on every secret, all the time, so a leak nobody has detected yet still expires on schedule.

• Scheduled rotation — secrets managers can auto-rotate database credentials, API keys, and TLS certificates on a fixed interval (commonly 30-90 days for static credentials) without human involvement, updating both the credential store and the target system atomically.
• Rotation on personnel change — any secret an engineer had direct access to gets rotated when that engineer leaves the team, independent of the regular schedule.
• Immediate rotation on suspected exposure — a secret flagged by a scanner (see below), found in a log, or referenced in a bug report gets rotated immediately, not queued for the next scheduled cycle.

Combined with dynamic, short-lived credentials, rotation policy is what actually closes the gap that hardcoded secrets open: instead of one long-lived value that's dangerous for as long as it exists, every credential has a bounded useful life by construction.

Detecting leaked secrets as a safety net

Policy and tooling reduce how often a secret gets hardcoded, but they don't reach zero — someone will eventually paste a token into a commit by accident. Detection tooling exists to catch that fast, before the commit reaches a shared branch or a build log. Pre-commit hooks running tools like Gitleaks, TruffleHog, or detect-secrets scan a diff for credential-shaped strings (recognizable patterns like AKIA AWS key prefixes, sk_live_ Stripe keys, or high-entropy strings) before the commit is even created locally, which is the cheapest possible place to stop a leak.

That local check is a courtesy, not a control — a developer can skip a hook with --no-verify, so the same scanning has to run again as a mandatory gate later in the pipeline: a dedicated secret-scanning stage in CI, GitHub's native push protection and secret scanning on the remote repository, and periodic full-history scans that catch anything that slipped through both earlier layers. Findings should fail the build, not just log a warning, and should trigger the same "assume compromised, rotate immediately" response as any other confirmed leak — detection without a rotation follow-through just produces alerts nobody acts on.