cert-manager

cert-manager is the Kubernetes controller that turns X.509 certificates into ordinary declarative objects: you write a Certificate resource saying “I want a valid certificate for shop.acme.io, in this Secret,” and a controller obtains it from a real certificate authority, stores it, watches the clock, and quietly renews it before it expires — forever, without a human. It solves the single most boring and most catastrophic platform problem there is: certificates always expire, expiry is invisible until the exact second it isn’t, and a platform that renews TLS by hand will eventually take its ingress, its service mesh, or its own admission webhooks down at three in the morning.

What cert-manager is and the problem it solves

☺ Like you’re 10: It’s a robot that goes and fetches the “this is really me” stickers your apps need, and swaps them for new ones before the old ones go stale.

cert-manager is a CNCF graduated project — a set of controllers and custom resources you install into a cluster (conventionally the cert-manager namespace). Once it is running, obtaining a certificate stops being a ticket, a shared spreadsheet, or a person with a laptop and an OpenSSL command, and becomes a Kubernetes object that a controller reconciles like any other. That is the whole idea: the same reconciliation loop that keeps your Deployments correct now keeps your TLS correct.

The problem before cert-manager

Traditional certificate management is a manual, calendared, human process: somebody generates a private key and a CSR, sends it to a CA, waits, receives a bundle, pastes it into a Secret or a load-balancer console, and writes “renew shop.acme.io” in a calendar eleven months out. Every step is a place to make a mistake, and the failure mode is uniquely nasty — nothing degrades gradually. The certificate is perfectly valid, and then at a precise timestamp it is not, and every client refuses the connection at once. Meanwhile the private key has been handled by humans and stored who knows where.

What it automates

cert-manager owns the entire lifecycle. It generates the private key inside the cluster (so the key material never travels), builds the CSR, drives whatever proof-of-ownership dance the CA requires, writes the signed certificate and key into a Kubernetes Secret of type kubernetes.io/tls, and then schedules renewal. Because the certificate lands in an ordinary Secret, everything downstream — an Ingress controller, a Gateway, a mounted volume, a webhook server — consumes it with no special knowledge that cert-manager exists.

What it is not

cert-manager is not a general secret manager: it creates and rotates one very specific kind of secret. For database passwords, API tokens and cloud credentials you want the External Secrets Operator or a sealed-secrets approach — see Secrets Management for the full picture and the “reference in Git, plaintext never” rule. It is also not a certificate authority by itself, although it can act as one via a CA or SelfSigned issuer; the trust decision is still yours. And it does not distribute trust bundles to clients — that is the job of its sibling project, trust-manager, which you meet below.

Where it fits in a platform

☺ Like you’re 10: It sits in the basement. Almost nothing talks to it directly, but if it stops, a surprising number of things upstairs break.

In the layered platform model cert-manager is a platform-services capability sitting on top of the Kubernetes substrate, closest to the security and networking domains. It is classic invisible infrastructure: application teams never write an Issuer, they just add one annotation to an Ingress and get HTTPS. That asymmetry — enormous value, near-zero developer surface area — is exactly what a good platform product feels like.

The four jobs it does in a real platform

First, public TLS for ingress: certificates from Let’s Encrypt or a commercial ACME CA for every externally reachable hostname, driven by an annotation. Second, an internal CA for service-to-service traffic: a private root, issued per workload, so east-west traffic is encrypted and mutually authenticated without buying anything. Third, serving certificates for admission webhooks — every validating or mutating webhook and every operator with a conversion webhook needs a TLS server certificate whose CA the API server trusts, and cert-manager plus its cainjector is the usual way operators scaffolded with Kubebuilder get one. Policy controllers such as Kyverno generate and rotate their own self-signed webhook certificates out of the box, but can be pointed at cert-manager instead where a platform wants one PKI. Fourth, the service-mesh trust anchor: Istio and Linkerd both need a root or intermediate to sign workload identities, and both document cert-manager as a supported way to issue and rotate it.

Its neighbours

Upstream, cert-manager is installed and configured by your GitOps controller like any other add-on — its ClusterIssuers live in the infrastructure folder of the config repo described in GitOps Workflows. Sideways, it collaborates with your ingress controller and networking layer (HTTP-01 solving literally creates a temporary Ingress or HTTPRoute), and with secret storage when the issuer is Vault. Downstream, Prometheus scrapes its metrics — certmanager_certificate_expiration_timestamp_seconds is the single most valuable alert on this page — and policy engines can require that every Ingress carries a TLS block.

CNPE domain relevance

Be honest about this one: cert-manager is not on the official CNPE tool list, so you will not be asked “write a ClusterIssuer.” It earns its place here because it underpins Platform Architecture & Infrastructure (15% of the exam — cert-manager appears in essentially every reference architecture as a required cluster add-on) and Security & Policy (15%), where encryption in transit is a stated competency and TLS everywhere is how you achieve it. Treat it as background knowledge that makes the security and observability questions easier, not as a memorisation target. The tool you do need at that depth is catalogued on The Tool Landscape.

How it works — architecture and CRDs

☺ Like you’re 10: One helper does the work, one checks your paperwork is valid before it’s accepted, and one glues the “who to trust” bit into other things that need it.

A standard install lays down three Deployments in the cert-manager namespace, plus a set of CRDs. Knowing which is which makes debugging fast, because the log line you need is nearly always in exactly one of them.

The three components

The controller is the engine: it watches every cert-manager CRD, drives issuance, and schedules renewals. Nearly all interesting log output is here. The webhook is a validating and mutating admission webhook that checks your cert-manager resources are well-formed before the API server stores them — which is why a broken webhook produces the infuriating “Internal error occurred: failed calling webhook” on a perfectly reasonable kubectl apply. The cainjector watches for objects annotated cert-manager.io/inject-ca-from and copies the right CA bundle into their caBundle fields — it is how webhook configurations and CRD conversion endpoints learn to trust cert-manager’s own CA, and cert-manager uses it to bootstrap its own webhook.

The custom resources it introduces

Six resources across two API groups. In cert-manager.io/v1: Issuer and ClusterIssuer (where certificates come from — namespaced and cluster-scoped versions of the same thing), Certificate (the one you write: “I want this”), and CertificateRequest (a single, immutable, one-shot request for one signing operation). In acme.cert-manager.io/v1, used only when the issuer is ACME: Order (one ACME order for a set of names) and Challenge (one proof-of-control test for one name).

The issuance chain — memorise this shape

Every issuance walks the same ladder, and every debugging session walks it in the same order. A Certificate spawns a CertificateRequest; if the issuer is ACME, that spawns an Order, which spawns one Challenge per DNS name; when the challenges pass, the CA signs, and the signed certificate lands in the Secret. When something is wrong, the top of the ladder says something vague (“Issuing”) and the truth is at the bottom.

The resources you will actually write

☺ Like you’re 10: Here is the actual YAML you type — where certificates come from, what you want, and the one-line shortcut most people use instead.

In practice a platform team writes two or three ClusterIssuers once, and application teams write nothing at all — they add an annotation. Here are all of those.

A ClusterIssuer for public certificates (ACME, HTTP-01)

HTTP-01 proves control by serving a token at http://<name>/.well-known/acme-challenge/<token>. cert-manager creates a temporary Pod, Service and Ingress to answer it, then deletes them. It requires the name to resolve publicly and port 80 to be reachable from the internet — which rules it out for internal-only clusters and for wildcards.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging          # ALWAYS build against staging first
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: platform@acme.io          # CA emails you about expiry & policy changes
    privateKeySecretRef:
      name: letsencrypt-staging-account-key   # your ACME ACCOUNT key — not a cert key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx   # which class the temporary solver Ingress uses
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod            # identical, but the real endpoint and rate limits
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform@acme.io
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

A DNS-01 ClusterIssuer for wildcards and private clusters

DNS-01 proves control by writing a _acme-challenge.<name> TXT record. It is slower and needs credentials for your DNS provider, but it is the only way to get a wildcard certificate, and it works for clusters with no inbound internet at all. Note the selector — one issuer can carry several solvers and pick between them by DNS zone or by label.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform@acme.io
    privateKeySecretRef:
      name: letsencrypt-dns-account-key
    solvers:
      - selector:
          dnsZones: ["acme.io"]      # use this solver only for names in this zone
        dns01:
          route53:
            region: eu-west-1
            hostedZoneID: Z0123456789ABCDEFGHIJ
            # best practice: no static keys — use IRSA / Workload Identity on the
            # cert-manager ServiceAccount so no cloud credential exists in a Secret
      - selector:
          dnsZones: ["internal.acme.io"]
        dns01:
          rfc2136:                    # plain dynamic DNS update, for on-prem BIND
            nameserver: 10.0.0.53:53
            tsigKeyName: cert-manager
            tsigAlgorithm: HMACSHA512
            tsigSecretSecretRef:
              name: rfc2136-tsig
              key: tsig-secret-key

A Certificate you write by hand

For internal service-to-service certificates, mesh trust anchors and webhook serving certs, you write the Certificate explicitly. This is the resource worth being able to type from memory.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: checkout-server-tls
  namespace: checkout
spec:
  secretName: checkout-server-tls      # the Secret cert-manager CREATES (need not exist)
  secretTemplate:
    labels:
      app.kubernetes.io/part-of: checkout   # labels/annotations copied onto the Secret
  issuerRef:
    name: internal-ca                  # which Issuer/ClusterIssuer signs it
    kind: ClusterIssuer                # omit and it defaults to a namespaced Issuer!
    group: cert-manager.io
  commonName: checkout.checkout.svc
  dnsNames:                            # SANs — the names this cert is valid for
    - checkout.checkout.svc
    - checkout.checkout.svc.cluster.local
  duration: 2160h                      # 90 days. Default is 90d if omitted
  renewBefore: 720h                    # renew 30 days early. Default is 1/3 of duration
  usages:
    - server auth
    - client auth                      # both, because this service also dials peers (mTLS)
  privateKey:
    algorithm: ECDSA                   # ECDSA/RSA/Ed25519
    size: 256
    rotationPolicy: Always             # generate a FRESH key on every renewal — set this
  revisionHistoryLimit: 3              # keep the last 3 CertificateRequests for forensics

The annotation shortcut — what app teams actually use

Ninety percent of real usage is one annotation on an Ingress or a Gateway. cert-manager watches those objects, reads the hostnames out of the spec, and creates the Certificate for you — a small, beautiful piece of self-service.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  namespace: shop
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod   # use "issuer" for a namespaced one
spec:
  ingressClassName: nginx
  tls:
    - hosts: ["shop.acme.io"]
      secretName: shop-tls          # cert-manager creates and fills this Secret
  rules:
    - host: shop.acme.io
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: { name: shop, port: { number: 80 } }
---
# Gateway API: the SAME annotation, on the Gateway
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: edge
  namespace: infra
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  gatewayClassName: istio
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: shop.acme.io
      tls:
        mode: Terminate
        certificateRefs:
          - name: shop-gw-tls       # cert-manager creates and fills this Secret too

On an Ingress, the tls block must list the hosts and name a secretName that cert-manager will own — normally one that does not exist yet, since cert-manager creates and overwrites it. Miss the tls block entirely and the annotation does nothing at all, silently — the single most common “cert-manager isn’t working” report. The Gateway variant has a second silent failure: cert-manager’s Gateway API integration is off by default and has to be turned on when cert-manager is installed, so on a stock install the annotation on a Gateway is simply ignored. Confirm that before debugging anything else.

Distributing trust with trust-manager

Issuing certificates is half the job; clients also need to trust them. trust-manager is cert-manager’s companion: a controller with a Bundle resource that assembles CA certificates from Secrets, ConfigMaps, inline PEM and the public web PKI roots, then writes the merged bundle into a ConfigMap in every namespace you select. Applications mount that ConfigMap as their CA store.

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: acme-trust
spec:
  sources:
    - useDefaultCAs: true            # the public web PKI roots
    - secret:                        # plus our own internal root
        name: internal-ca-root
        key: ca.crt
  target:
    configMap:
      key: ca-bundle.crt             # key inside the generated ConfigMap
    namespaceSelector:
      matchLabels:
        trust: enabled               # only namespaces opting in

Day-to-day commands

☺ Like you’re 10: Mostly you type describe four times, walking down the ladder until one rung tells you the truth.

cert-manager ships an optional CLI, cmctl, but everything important is reachable with plain kubectl — which matters, because cmctl will not be installed on a machine you did not prepare.

Install and verify

# Helm is the usual install path
helm repo add jetstack https://charts.jetstack.io && helm repo update
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true          # older chart versions called this installCRDs

# Three Deployments must be Ready before ANY cert-manager resource will apply
kubectl -n cert-manager get deploy
kubectl -n cert-manager rollout status deploy/cert-manager-webhook

# Prove the webhook path actually works end to end
cmctl check api --wait=2m
kubectl get clusterissuer                 # READY should be True

Walking the chain when something is wrong

kubectl get certificate -A                        # READY=False is your starting point
kubectl describe certificate shop-tls -n shop     # rung 1 — usually says "Issuing"
kubectl get certificaterequest -n shop            # rung 2 — the newest one is the live attempt
kubectl describe certificaterequest shop-tls-1 -n shop
kubectl get order -n shop                         # rung 3 — ACME only
kubectl describe order shop-tls-1-1234567890 -n shop
kubectl get challenge -n shop                     # rung 4 — THE REAL ERROR LIVES HERE
kubectl describe challenge shop-tls-1-1234567890-987 -n shop

# One command that walks all of it for you, if cmctl is available
cmctl status certificate shop-tls -n shop

# Controller logs, when the resources themselves are silent
kubectl -n cert-manager logs deploy/cert-manager --tail=100

Renewing, inspecting and cleaning up

cmctl renew shop-tls -n shop            # force renewal now (duration/renewBefore unchanged)
cmctl renew --all -n shop               # every Certificate in a namespace
cmctl inspect secret shop-tls -n shop   # decode and pretty-print the issued certificate

# The pure-kubectl equivalents, for when cmctl is not installed
kubectl get secret shop-tls -n shop -o jsonpath='{.data.tls\.crt}' \
  | base64 -d | openssl x509 -noout -text | head -20
kubectl get secret shop-tls -n shop -o jsonpath='{.data.tls\.crt}' \
  | base64 -d | openssl x509 -noout -dates -subject -ext subjectAltName

# Nuclear option: delete the Secret and cert-manager re-issues from scratch.
# Safe for a broken cert, DANGEROUS on a working one under Let's Encrypt rate limits.
kubectl delete secret shop-tls -n shop

Gotchas and failure modes

☺ Like you’re 10: Almost everything that goes wrong is one of four things: the proof failed, DNS was slow, you asked the sticker office too many times, or nobody noticed a sticker peeling.

Debug downwards, never sideways

The number-one time sink is reading the Certificate’s status, seeing “Issuing”, and then guessing. The Certificate is a summary; it rarely knows why. Walk the ladder — Certificate → CertificateRequest → Order → Challenge — and stop at the first rung with a concrete message. HTTP-01 challenges fail with things like 404 or connection refused, which means the temporary solver Ingress is not reachable: the wrong ingressClassName, DNS not yet pointing at your load balancer, port 80 blocked at a firewall, another Ingress with a catch-all rule winning the route, or a redirect that discards the path — sending /.well-known/acme-challenge/<token> to a bare https://host/ so the token is never served. Note the nuance, because people get this wrong: Let’s Encrypt does follow redirects for HTTP-01, including plain HTTP→HTTPS ones, and does not validate the certificate it finds on the redirect target — so a path-preserving redirect is fine. It is the path-losing rewrite, or a CDN or WAF in front answering that path itself, that breaks you. That is a genuinely nasty networking triage case, because the redirect is working exactly as configured and that is the problem.

DNS-01 propagation delays look like hangs

DNS-01 writes a TXT record and then waits for it to become visible. Between provider API latency, negative caching of the previous NXDOMAIN answer, and recursive resolvers, that can take minutes — occasionally far longer with a long zone TTL. cert-manager self-checks propagation before telling the CA to look, so a Challenge sitting in pending with “waiting for DNS-01 challenge propagation” is usually not broken, just slow. Verify from outside the cluster with dig TXT _acme-challenge.shop.acme.io. When it is genuinely stuck it is normally CNAME delegation confusion or the solver writing to the wrong zone — which is what the dnsZones selector exists to prevent.

Let’s Encrypt rate limits will lock you out

Let’s Encrypt enforces hard limits — historically on the order of fifty certificates per registered domain per week, only a handful of duplicate certificates (the exact same set of names) per week, and a much tighter cap on failed validations per hostname per hour. The precise numbers get revised, so treat the linked rate-limits page as the source of truth and the shape as the lesson. A misconfigured issuer in a retry loop, an ApplicationSet stamping out the same Certificate across many clusters, or an engineer repeatedly deleting the Secret to “make it try again” will burn through the allowance in an afternoon, and the lockout lasts days. You cannot clear it on demand.

Expiry is a platform-wide outage, not an app bug

This is the one Timmy actually loses sleep over. cert-manager is often used for certificates that nothing obviously depends on until they lapse. If the CA bundle behind your service mesh expires, every mTLS connection in the cluster fails simultaneously and your dashboards go dark at the same moment. If a webhook serving certificate expires, the API server can no longer call that webhook — and if it is configured failurePolicy: Fail, admission starts rejecting objects, which can mean you cannot deploy the fix. Renewal itself can also silently fail while the old certificate is still valid, giving you a thirty-day window in which everything looks fine and nothing is being renewed. So: alert on certmanager_certificate_expiration_timestamp_seconds and on certmanager_certificate_ready_status, page well before expiry, and rehearse the recovery. Observability and reliability both have a stake here.

The quieter traps

A few more, in the order you meet them. Deleting a Certificate does not delete its Secret by default, so a stale Secret can keep serving an old certificate that nothing is renewing. Copying a TLS Secret between namespaces by hand produces a certificate nobody owns and nobody renews — use trust-manager or a per-namespace Certificate instead. Upgrading cert-manager across several minor versions at once is unsupported; step through them, and mind that CRDs installed by Helm need the chart’s CRD flag or they will not be upgraded. And a Certificate that lives in namespace A cannot use an Issuer in namespace B — that is precisely what ClusterIssuer is for. When you are lost, the troubleshooting playbook and workload triage pick up the thread.

Alternatives and when to choose it

☺ Like you’re 10: Other ways exist to get stickers, but most of them need a human with a calendar. That’s the whole difference.

The real comparison is rarely “cert-manager versus another Kubernetes certificate controller” — it has no serious peer in that niche. It is “cert-manager versus doing it somewhere else, or by hand.”

The comparison that decides it

A practical rule

If TLS terminates at a cloud load balancer for one or two public hostnames and nothing inside the cluster needs a certificate, managed cloud certificates are genuinely simpler and you should use them. The moment you need a wildcard, a private CA, mutual TLS between services, a service mesh, or a webhook that must serve TLS the API server trusts, you need in-cluster issuance and cert-manager is the answer everyone converges on. Most platforms end up running both: the cloud LB certificate for the edge, cert-manager for everything behind it.

Exam relevance and going further

☺ Like you’re 10: This one isn’t on the test — but on the day, you couldn’t look up its website anyway, so learn the shape, not the fields.

What the exam actually expects

cert-manager is not on the official CNPE tool list, so no task will require you to configure it by name. What it supports is conceptual: when a security or architecture question asks how a platform provides encryption in transit, or how workloads get identities, or which add-ons belong in a baseline cluster, cert-manager is the correct answer and knowing its shape makes the reasoning quick. Spend your revision time on the named tools in The Tool Landscape and the manifests on Know Cold; read this page once for understanding rather than drilling it.

The documentation allowlist — read this twice

⚖ CNPA vs CNPE — That allowlist is a CNPE-only mechanic: CNPA hands you no documentation whatsoever, official or otherwise — it's a fully closed-book multiple-choice exam with zero lookups of any kind, which makes it stricter than CNPE, not looser. Even so, the issuance-chain concepts on this page are worth keeping for CNPA prep, since that exam recalls the same platform-engineering ground from memory rather than testing it on a live cluster.

What to carry away regardless

Four things. The issuance chain — Certificate → CertificateRequest → Order → Challenge — and the habit of walking it downward when debugging. The HTTP-01 versus DNS-01 trade-off: HTTP-01 is simple but needs public port-80 reachability and cannot do wildcards; DNS-01 handles wildcards and private clusters but needs zone credentials and patience. The output contract: a plain kubernetes.io/tls Secret, which is why everything integrates. And the operational truth that certificate expiry is a cluster-wide outage class of its own, deserving a real alert. For the wider secret-handling story read Secrets Management; for how add-ons like this get installed and kept correct, GitOps Workflows; and when a term stops making sense, the glossary.

Official resources for after the exam

Outside the exam, the canonical sources are cert-manager.io/docs (the Configuration, ACME and Troubleshooting sections repay careful reading), the trust-manager docs at cert-manager.io/docs/trust/trust-manager, the source at github.com/cert-manager/cert-manager, the project’s CNCF page at cncf.io/projects/cert-manager, and Let’s Encrypt’s own rate-limits page — worth reading once, properly, before you ever point an issuer at production.