Cryptography & Key Management

Secrets management covered where a credential lives and how long it lives there. This page covers the layer underneath that: what actually happens when something gets "encrypted," why almost nobody encrypts real data directly with a KMS API call, what changes in practice between a cloud KMS, a dedicated HSM, and a key sitting in an environment variable, and how a TLS certificate renews itself without a human remembering to do it. The thread running through all of it is one discipline pipeline engineers skip constantly: a sentence like "the config is encrypted" is not a security claim until you can say encrypted with what, encrypted where, and encrypted against whom.

"We encrypted it" is three unanswered questions

☺ Like you're 10: Before you believe a lock keeps you safe, you have to ask what kind of lock, which door it's on, and who it's actually meant to stop.

Three questions turn "we encrypted it" from a slogan into an engineering claim. Encrypted with what — a named algorithm and key length, not a vague "it's encrypted," because "encrypted" covers everything from AES-256-GCM to a home-rolled XOR cipher a contractor wrote in 2014. Encrypted where — at rest (sitting on a disk, in a database, in a backup snapshot), in transit (moving across a network, almost always TLS), or, less commonly outside confidential-computing setups, in use (decrypted in memory while a process is actively working with it — the state that's true for nearly every byte of "encrypted" data most of the time it exists). Encrypted against whom — which specific attacker or failure mode does this actually stop, and which does it not touch at all.

That third question is the one teams skip, and it's the one that matters most, because the same ciphertext can be simultaneously "secure" against one attacker and worthless against another.

Notice the pattern in the last column: every layer's blind spot is "an attacker who already holds the legitimate credential to ask for the plaintext." That's not a flaw unique to any one of these — it's the boundary of what encryption, as a category, can do. Encryption protects data from someone who doesn't have the right to see it. It cannot protect data from someone who does, which is precisely why access control, key policy, and the rest of this page matter as much as the cipher itself.

The primitives a pipeline engineer actually touches

☺ Like you're 10: A padlock, a wax seal, and a fingerprint check are three different tools for three different jobs — mixing them up is how people get hurt.

This isn't a full cryptography course — it's the short list of primitives that actually show up in a CI/CD pipeline, plus the confusion that costs teams the most: treating hashing, message authentication, and password storage as interchangeable when they solve three different problems.

Symmetric: AES-256-GCM

AES-256-GCM is the default choice for encrypting data with a single shared key — it's an AEAD (Authenticated Encryption with Associated Data) construction, meaning one operation gives you confidentiality and integrity together: tampered ciphertext fails to decrypt rather than silently decrypting into garbage. The one operational rule that actually matters: never reuse a nonce (IV) with the same key. GCM's security proof depends on nonce uniqueness, and reusing one is not a minor weakness — it can leak the authentication key outright and let an attacker forge ciphertext that decrypts successfully. Every library that does this correctly generates the nonce for you; the failure mode is almost always a hand-rolled implementation that hardcodes or increments an IV badly.

Asymmetric: RSA, ECDSA, and Ed25519

Asymmetric (public-key) cryptography is what you reach for when the encrypting and decrypting parties can't share a secret in advance — TLS handshakes and digital signatures both depend on it. RSA is the oldest and still the most universally supported; NIST guidance has been pushing the floor from 2048-bit toward 3072-bit for keys expected to stay secure past 2030, and RSA operations are noticeably slower and produce larger signatures than the alternatives below. ECDSA (commonly on the P-256 curve) gives roughly equivalent security to RSA-3072 at a fraction of the key and signature size, which is why most TLS certificates issued today are ECDSA rather than RSA. Ed25519 (EdDSA over Curve25519) is newer still, deterministic (no per-signature random nonce to get wrong — the exact class of bug that has broken ECDSA implementations in the wild), and fast enough that it's become the default for SSH keys, Git commit signing, and Sigstore/cosign artifact signatures; CA support for issuing Ed25519 leaf certificates in the public Web PKI is still catching up, so verify current support before standardizing on it for a public-facing TLS certificate.

Hashing, HMAC, and password hashing are three different jobs

All three produce a fixed-length string from an input, which is exactly why they get confused. A plain hash (SHA-256) is for integrity — proving a file wasn't altered, fingerprinting a build artifact — and carries no secret at all. HMAC (HMAC-SHA256) adds a shared secret key to that hash, turning it into a message-authentication primitive: it proves the message came from someone who holds the key, not just that the bytes are unchanged. Password hashing is a different job entirely — you're not fingerprinting data, you're deliberately making an attacker's brute-force guessing slow and memory-expensive — which is why SHA-256 is the wrong tool here despite being a hash: it's fast, and fast is exactly what you don't want when the input space is "passwords humans pick." Argon2id (winner of the 2015 Password Hashing Competition) or bcrypt/scrypt are the correct choice, because they're deliberately slow and memory-hard. A fourth, separate job — deriving several keys from one master secret — belongs to a KDF like HKDF (RFC 5869), which is neither a password hasher nor a MAC.

# Symmetric, authenticated: AES-256-GCM (confidentiality + integrity in one pass)
openssl enc -aes-256-gcm -salt -pbkdf2 -in plaintext.txt -out ciphertext.bin -pass pass:demo-only-do-not-use

# Asymmetric: an ECDSA P-256 key, what most TLS leaf certs use today
openssl ecparam -name prime256v1 -genkey -noout -out ec-key.pem

# Ed25519: deterministic, no per-signature randomness to mess up — Sigstore/cosign and modern SSH default here
openssl genpkey -algorithm ed25519 -out ed25519-key.pem

# Three different jobs — reaching for the wrong one is the actual bug:
sha256sum build-output.tar.gz                                    # integrity fingerprint, no secret involved
openssl dgst -sha256 -hmac "$SHARED_SECRET" payload.json         # message authentication, needs a shared key
# Passwords need Argon2id/bcrypt (slow + memory-hard) — never a fast general-purpose hash like SHA-256

Envelope encryption: why nobody hands a database to a KMS API

☺ Like you're 10: The bank vault door never leaves the bank. What actually travels is a small key that only works because the vault made it, and the vault will happily make you a new one.

Every cloud KMS enforces a hard size limit on what it will directly encrypt or decrypt — AWS KMS's direct Encrypt/Decrypt calls cap plaintext at 4KB, and equivalent limits exist on Google Cloud KMS and Azure Key Vault (check each provider's current documentation, since limits do get revised). That's not an arbitrary restriction; it reflects what a KMS is actually for. A KMS or HSM exists to protect one thing extremely well — a small number of long-lived master keys, called KEKs (Key Encryption Keys) — and to never let that key material leave the hardware boundary, not to serve as a general-purpose bulk-encryption endpoint that every byte of application data round-trips through.

The pattern that makes this practical for data of any size is envelope encryption. Instead of encrypting your data with the KEK directly, you ask the KMS to generate a fresh, random DEK (Data Encryption Key). The KMS hands back two things: the DEK in plaintext, and the same DEK encrypted ("wrapped") under the KEK — the KEK itself never leaves the KMS/HSM boundary at any point in this exchange. You use the plaintext DEK to encrypt your actual data locally, at full disk or network speed, with no size limit and no per-byte KMS round-trip. Then you immediately discard the plaintext DEK from memory and store only the wrapped DEK, right alongside the ciphertext it protects. To decrypt later, you send the small wrapped DEK — never the data — back to the KMS, which unwraps it using the KEK and returns the plaintext DEK; you use that to decrypt locally, then discard it again.

This is also why key rotation doesn't mean re-encrypting every object your organization has ever stored. Rotating the KEK only changes how future DEKs get wrapped; a KMS with automatic rotation (AWS KMS rotates the backing key material of a symmetric customer-managed key annually by default) keeps prior key-material versions available internally purely so old wrapped DEKs still unwrap correctly — the DEK/KEK split is precisely what makes that painless.

# 1. Ask KMS for a data key wrapped under the KEK "prod-app-kek"
aws kms generate-data-key --key-id alias/prod-app-kek --key-spec AES_256 > dk.json
PLAINTEXT_DEK=$(jq -r .Plaintext dk.json | base64 -d)      # used once, in memory only
WRAPPED_DEK=$(jq -r .CiphertextBlob dk.json)                 # this is what gets stored

# 2. Encrypt the real data locally — any size, no further KMS calls
openssl enc -aes-256-gcm -K "$PLAINTEXT_DEK" -iv "$IV" -in database-export.sql -out export.enc
unset PLAINTEXT_DEK                                            # discard immediately

# --- later, to decrypt ---
# 3. Send only the small wrapped DEK back — never the ciphertext itself
aws kms decrypt --ciphertext-blob fileb://wrapped-dek.bin \
  --output text --query Plaintext | base64 -d > dek.raw
openssl enc -d -aes-256-gcm -K "$(xxd -p dek.raw)" -iv "$IV" -in export.enc -out database-export.sql

KMS vs HSM vs a key sitting in code

☺ Like you're 10: A bank vault, a home safe, and a key taped under the doormat all "keep something safe" — but they don't defend against the same burglar.

All three approaches let a program "have" a key. The difference is where the raw key material can ever exist, and therefore which attacker each one actually stops.

A key policy — the JSON document attached to a KMS key that names exactly which principals may use it, and for which operations — is where the theoretical boundary becomes a real one. Scoping kms:Decrypt to one specific IAM role, rather than to a broad wildcard, is the difference between "only the checkout service can read this" and "anyone with any KMS permission in the account can read this." Encryption context tightens it further: an authenticated (but not itself encrypted) key-value pair supplied on both encrypt and decrypt, which the KMS refuses to unwrap unless the context matches exactly — a cheap way to bind a specific ciphertext to a specific purpose so it can't be decrypted somewhere it was never meant to be used.

{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "AllowKeyAdmins", "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::111122223333:role/kms-admins" },
      "Action": ["kms:Create*", "kms:Describe*", "kms:ScheduleKeyDeletion"],
      "Resource": "*" },
    { "Sid": "AllowOnlyCheckoutServiceToDecrypt", "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::111122223333:role/checkout-service-runtime" },
      "Action": "kms:Decrypt",
      "Resource": "*",
      "Condition": { "StringEquals": { "kms:EncryptionContext:app": "checkout" } } }
  ]
}

Key lifecycle: rotation, versioning, and crypto-shredding

☺ Like you're 10: Changing a lock on a schedule keeps a copied key from working forever. Destroying a lock on purpose is how you make everything behind it unreadable, instantly, without touching what's behind it at all.

Scheduled rotation replaces a KEK's backing material periodically — cloud KMS providers typically automate this annually for symmetric keys — without changing the key's identifier (its ARN or resource name), so every existing wrapped DEK still decrypts correctly against the same key reference; the provider keeps prior key-material versions internally purely to make that transparent. Manual rotation (creating an entirely new KEK and re-wrapping DEKs under it over time) is reserved for cases where policy requires a hard key-identity change, not just fresh material under the same identity.

Multi-party control matters most for the root of the whole system. HashiCorp Vault's default unseal mechanism uses Shamir's Secret Sharing: the master key is split into a configurable number of shares (commonly five), with a threshold (commonly three) required to reconstruct it — no single operator, and no single stolen laptop, can unseal Vault alone. In production, most teams delegate this to auto-unseal, wrapping the master key with a cloud KMS key instead of running a manual quorum ceremony on every restart — trading the Shamir ceremony for a boundary the team already trusts, not eliminating multi-party control, just relocating it.

# Shamir: 5 shares exist; any 3 reconstruct the master key — no single operator can unseal alone
vault operator init -key-shares=5 -key-threshold=3
vault operator unseal   # run 3 times total, once per share, by 3 different holders

# Auto-unseal: delegate the wrapping key to a cloud KMS instead of a manual ceremony
seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "alias/vault-unseal-key"
}

Crypto-shredding is the sharpest tool this whole model produces, and it falls directly out of the envelope-encryption pattern: destroy the key that wraps a piece of data, and every byte that key ever protected becomes permanently unrecoverable — without touching the ciphertext itself. That's a genuinely useful erasure mechanism for something like a GDPR "right to erasure" request against data replicated across backups, read replicas, and cold storage you may not even be able to enumerate quickly: issue one KEK per tenant or per data subject, and "delete this person's data" becomes "destroy this one key," not a distributed hunt through every copy. Cloud KMS providers build in a deliberate delay — AWS KMS enforces a 7–30 day pending-deletion window — specifically so a single compromised or mistaken deletion request can't instantly and irreversibly destroy production data.

# Destroying the KEK makes every DEK it ever wrapped permanently unreadable —
# without touching a single byte of the (possibly replicated, possibly backed-up) ciphertext
aws kms schedule-key-deletion --key-id alias/tenant-48213-erasure-key --pending-window-in-days 7

TLS certificate automation: the end of yearly renewal panic

☺ Like you're 10: Instead of a sticky note reminding a human to renew a certificate once a year, a small robot checks the expiry date constantly and just quietly gets a new one early, every time, forever.

A TLS certificate is a public key plus an identity claim, signed by a Certificate Authority that vouches for the binding between them. The operational problem for decades was that certificates lived a year or more and renewal was a manual, calendar-driven task — which is exactly the failure mode behind a long, embarrassing list of real production outages caused by an expired cert nobody rotated in time. The fix is the same shift already familiar from short-lived credentials in secrets management: make certificates short-lived by default, and automate the renewal so no human has to remember anything.

The ACME protocol (RFC 8555, Automated Certificate Management Environment) is what makes that automation possible. A client proves control over a domain by solving a challenge the CA issues — HTTP-01 (serve a token at a well-known URL path on that domain) or DNS-01 (publish a specific TXT record; the only challenge type that supports wildcard certificates, since you can't serve an HTTP path for every possible subdomain) — and once the CA validates the challenge, it issues the certificate automatically, no human review step. Let's Encrypt, run by the nonprofit Internet Security Research Group, is the ACME CA most teams default to; its certificates are valid for 90 days by design, specifically to force automation rather than tolerate manual renewal (check Let's Encrypt's current documentation for any newer short-lived-certificate options, since that program continues to evolve).

In Kubernetes, cert-manager is the operator that runs this loop continuously: a Certificate resource declares the desired state — which domain, which issuer, how long the cert should live, how far before expiry to renew — and cert-manager reconciles it exactly the way any other Kubernetes controller reconciles a spec, requesting a new certificate through ACME, solving the challenge automatically, and writing the result into a Kubernetes Secret that workloads mount.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: security@acme.example
    privateKeySecretRef: { name: letsencrypt-prod-account-key }
    solvers:
      - dns01:
          route53: { region: us-east-1 }   # DNS-01 — required for wildcard certs
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-acme-io-tls
  namespace: checkout
spec:
  secretName: api-acme-io-tls
  duration: 2160h      # 90 days
  renewBefore: 720h    # renew ~60 days in — roughly 2/3 of the lifetime, well before expiry
  dnsNames: [api.acme.io]
  issuerRef: { name: letsencrypt-prod, kind: ClusterIssuer }

Renewing at roughly two-thirds of the certificate's lifetime, rather than waiting until just before expiry, is deliberate: it leaves days of runway to retry a failed challenge, rotate a broken DNS integration, or catch a stuck renewal in monitoring before it becomes an outage. The same short-lived philosophy pushed further, automatically, on every internal connection rather than one public endpoint, is what a service mesh's mTLS does — see Kubernetes Security Deep Dive and zero trust for pipelines for that layer.

Putting it together — and naming the threat model out loud

☺ Like you're 10: Locking the box is easy. The real work is being able to say exactly who still can't open it — and who still can.

SOPS (Secrets OPerationS, originally a Mozilla project) is a common way pipeline configs actually apply everything above: it encrypts only the values in a YAML/JSON file, leaving keys and structure readable, and it does it using envelope encryption under the hood — the exact DEK/KEK pattern from earlier in this page, wearing a file-format wrapper. A KMS key ARN (or an age recipient, or a PGP key) in .sops.yaml names which key wraps the file's data key; anyone who can call sops -d is really just calling KMS Decrypt the same way any other consumer would.

# .sops.yaml — which files get encrypted, and under which KMS key
creation_rules:
  - path_regex: \.enc\.yaml$
    kms: arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab
    encrypted_regex: ^(password|api_key|token)$

# Structure stays plaintext; only matched values get wrapped:
# before → password: "Tr0ub4dor&3-prod-2024"
# after `sops -e config.yaml > config.enc.yaml`:
# password: ENC[AES256_GCM,data:Xy2f...,iv:...,tag:...,type:str]

sops -d config.enc.yaml > config.yaml   # decrypt requires calling KMS as a trusted principal

Now name the threat model, the way this whole page has been insisting on. A committed config.enc.yaml defends against a stolen git repository — the ciphertext is useless to anyone who can clone the repo but can't also call kms:Decrypt as a principal the key policy trusts. It does not defend against a compromised CI runner that legitimately holds that same decrypt permission — an attacker who lands there just asks KMS the same way the pipeline does and gets the plaintext back, no cryptographic weakness required. And it does not defend against a key policy that's broader than it should be — that failure was never a cryptography problem at all; it was an access-control problem wearing an encryption costume. Every claim in that paragraph is exactly the discipline this page opened with: not "is it encrypted," but encrypted with what, encrypted where, and encrypted against whom.

This page covered the mechanics; secrets management covers where the wrapped keys and short-lived credentials actually live day to day, and workload identity & pipeline IAM covers exactly who's allowed to call Decrypt in the first place. For what encryption buys you specifically toward a legal erasure obligation, see privacy engineering & data protection; for the FIPS-validation and key-policy evidence an auditor will actually ask for, see compliance as code at scale. And for the signing half of this story — proving an artifact wasn't tampered with, rather than keeping it confidential — see Sigstore & cosign.