Capstone Part 2 — Progressive Delivery

This is the second of five parts building one continuous project: cert-tracker, the exam-tracking service Part 1 got running under Argo CD, self-healing and prune both proven live. Every release since Part 1 has meant one thing: edit an image tag in Git, let the reconciler apply it, and let a plain Deployment's RollingUpdate cycle through every pod at once — whether or not the new build was actually safe. Today that changes. This page swaps cert-tracker's Deployment for an Argo Rollout, wires a real weighted traffic split through the NGINX ingress controller cert-tracker already needs, and gates every step on an AnalysisTemplate reading a real metric — not a guess, not a fixed pause. By the end you will have watched a safe release ramp itself from 20% to 100% on its own, and watched a broken one get caught and reversed automatically, with nobody pressing a button either time.

What this part assumes, and what it produces

☺ Like you're 10: Just the cluster, the two repos, and the app Part 1 already got Synced and Healthy — nothing about cert-tracker's own code changes today, except two small version bumps at the very end.

This part assumes Part 1's cert-tracker Application reports Synced and Healthy — pods Running in the cert-tracker namespace, prune and selfHeal both proven with your own hands, not just configured. You need everything Part 1 needed — kubectl, Docker, git, the argocd CLI — plus the kubectl argo rollouts plugin. Nothing about the real mesh or the real observability stack gets built yet: the ingress controller and the narrow Prometheus this page installs are deliberately the smallest slice of each that a working canary actually needs, not Part 3's mesh or Part 4's dashboards.

Platform bootstrap: the ingress controller, Argo Rollouts, and a narrow Prometheus

☺ Like you're 10: Three pieces of building infrastructure go in once, by hand, before the app itself ever notices anything changed.

Install the NGINX ingress controller first — cert-tracker's canary split rides entirely on its built-in canary annotations, and it is not a service mesh, so installing it doesn't front-run Part 3. The kind-flavored manifest below matches Part 1's assumed local cluster; swap it for minikube addons enable ingress if that's what you're running instead:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
kubectl -n ingress-nginx rollout status deploy/ingress-nginx-controller --timeout=180s

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
kubectl -n argo-rollouts rollout status deploy/argo-rollouts --timeout=180s

curl -fSL -o kubectl-argo-rollouts \
  https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x kubectl-argo-rollouts && sudo mv kubectl-argo-rollouts /usr/local/bin/

Then a Prometheus deliberately narrow enough to answer one query: it scrapes only the ingress controller's own :10254/metrics endpoint, nothing app-level yet.

# platform/prometheus.yaml — applied by hand, not tracked in cert-tracker-config
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    scrape_configs:
      - job_name: ingress-nginx
        static_configs:
          - targets: ["ingress-nginx-controller-metrics.ingress-nginx.svc.cluster.local:10254"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  selector: { matchLabels: { app: prometheus } }
  template:
    metadata: { labels: { app: prometheus } }
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v2.54.1
          args: ["--config.file=/etc/prometheus/prometheus.yml"]
          volumeMounts: [{ name: config, mountPath: /etc/prometheus }]
      volumes:
        - name: config
          configMap: { name: prometheus-config }
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
spec:
  selector: { app: prometheus }
  ports: [{ port: 9090, targetPort: 9090 }]

kubectl create namespace monitoring
kubectl apply -f platform/prometheus.yaml
kubectl -n monitoring rollout status deploy/prometheus --timeout=120s

The Rollout: swapping the workload kind without touching the pod template

☺ Like you're 10: Swapping the label on the box from Deployment to Rollout — everything already inside stays exactly the same, only what happens on the next release changes.

In cert-tracker-config, replace apps/cert-tracker/base/deployment.yaml with rollout.yaml. The pod template is copied verbatim from Part 1 — same image, same probes — and everything new lives in spec.strategy:

# apps/cert-tracker/base/rollout.yaml — replaces deployment.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: cert-tracker
  labels: { app: cert-tracker }
spec:
  replicas: 2
  selector:
    matchLabels: { app: cert-tracker }
  template:                              # identical to Part 1's Deployment — nothing here changes
    metadata:
      labels: { app: cert-tracker }
    spec:
      containers:
        - name: cert-tracker
          image: ghcr.io/kubestronaut/cert-tracker:v0.1.0
          ports: [{ containerPort: 8080 }]
          readinessProbe: { httpGet: { path: /healthz, port: 8080 }, initialDelaySeconds: 3 }
          livenessProbe: { httpGet: { path: /healthz, port: 8080 }, initialDelaySeconds: 10 }
  strategy:
    canary:
      canaryService: cert-tracker-canary   # selectors Argo Rollouts manages for you — see below
      stableService: cert-tracker-stable
      trafficRouting:
        nginx:
          stableIngress: cert-tracker       # the Ingress it clones into a canary variant
      analysis:
        templates:
          - templateName: cert-tracker-success-rate   # namespaced — see why, below
        startingStep: 1
        args:
          - name: canary-service
            value: cert-tracker-canary
      steps:
        - setWeight: 20
        - pause: { duration: 2m }
        - setWeight: 50
        - pause: { duration: 2m }
        - setWeight: 100

The rest of the base directory grows by four files — two Services Argo Rollouts will manage the selectors of, the stable Ingress its NGINX integration clones, and the AnalysisTemplate itself:

# apps/cert-tracker/base/service-stable.yaml
apiVersion: v1
kind: Service
metadata:
  name: cert-tracker-stable
spec:
  selector: { app: cert-tracker }   # Argo Rollouts narrows this with a pod-template-hash label at sync time
  ports: [{ port: 80, targetPort: 8080 }]
---
# apps/cert-tracker/base/service-canary.yaml — identical shape, different name
apiVersion: v1
kind: Service
metadata:
  name: cert-tracker-canary
spec:
  selector: { app: cert-tracker }
  ports: [{ port: 80, targetPort: 8080 }]
---
# apps/cert-tracker/base/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cert-tracker
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: cert-tracker.kubestronaut.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: { name: cert-tracker-stable, port: { number: 80 } }
---
# apps/cert-tracker/base/analysistemplate.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: cert-tracker-success-rate
spec:
  args:
    - name: canary-service
  metrics:
    - name: success-rate
      interval: 30s
      count: 6
      initialDelay: 30s                  # let the canary warm up before judging it
      successCondition: result[0] >= 0.98
      failureLimit: 2                    # tolerate 2 bad readings; the 3rd fails the run
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: |
            sum(rate(nginx_ingress_controller_requests{
                  service="{{args.canary-service}}", status!~"5.."}[2m]))
            /
            sum(rate(nginx_ingress_controller_requests{
                  service="{{args.canary-service}}"}[2m]))

Update base/kustomization.yaml's resources list to swap deployment.yaml for the five files above, and update overlays/dev/patch-replicas.yaml's target.kind from Deployment to Rollout — the patch still targets the same resource name, only the kind changed. Commit both, plus every new file, and push:

cd cert-tracker-config
git rm apps/cert-tracker/base/deployment.yaml
# add rollout.yaml, service-stable.yaml, service-canary.yaml, ingress.yaml, analysistemplate.yaml
# edit base/kustomization.yaml and overlays/dev/patch-replicas.yaml as described above
git add .
git commit -m "feat: cert-tracker as a canary Rollout, gated on a real ingress metric"
git push

argocd app sync cert-tracker      # or wait for the next ~3min poll
argocd app get cert-tracker       # project: kubestronaut, still — nothing cluster-scoped got added

Keeping Argo CD out of the Rollout controller's own field

☺ Like you're 10: Two robots reaching for the same dial — one has to be told "not this one, that one's the other robot's job."

cert-tracker-stable and cert-tracker-canary are plain manifests under Git, applied and continuously reconciled by the same cert-tracker Application as everything else. The moment a rollout starts, the Rollout controller narrows each Service's spec.selector with its own rollouts-pod-template-hash label, live, so each one points at exactly the right ReplicaSet — a runtime edit, not a Git commit. With selfHeal: true still on from Part 1, the very next reconcile pass sees a live selector that no longer matches Git's checked-in { app: cert-tracker } and — this is external-owner drift, the exact shape that page already named, not a mistake to revert — would fight the Rollout controller for it, forever, unless told otherwise. Add the carve-out to bootstrap/application.yaml:

# cert-tracker-config/bootstrap/application.yaml — spec.ignoreDifferences added
spec:
  ignoreDifferences:
    - group: ''
      kind: Service
      name: cert-tracker-stable
      jsonPointers: [/spec/selector]
    - group: ''
      kind: Service
      name: cert-tracker-canary
      jsonPointers: [/spec/selector]

cd cert-tracker-config
git commit -am "fix: ignoreDifferences on the two Services Rollouts now owns"
git push                                          # keep the change in history, same as everything else

kubectl apply -n argocd -f bootstrap/application.yaml   # nothing reconciles bootstrap/ automatically — apply it once, by hand

The cert-tracker-canary Ingress that NGINX-mode Rollouts creates needs no such carve-out at all — it is a resource Argo CD never applied in the first place, generated and owned entirely by the Rollout controller, and it never appears anywhere in cert-tracker-config's rendered manifests. Argo CD can only prune or revert resources it tracks; an object it never created is invisible to its diff, full stop. That's a structural difference from the Istio-based routing Argo Rollouts also documents, where the controller patches weight fields directly onto a single VirtualService Argo CD does track — a genuine two-controller conflict over one object, needing its own ignoreDifferences entry on the weight fields themselves. The NGINX router sidesteps that particular fight structurally; the Service selectors are the one field this setup still has to hand off explicitly.

Proving it: a safe release promotes itself, a broken one catches itself

☺ Like you're 10: Send the safe test capsule through first and watch it get waved further each time it reports clean numbers — then send a flawed one and watch the exact same watchdog say no.

Ship a small, safe change first — v0.1.1 adds a version field to the /healthz response, nothing that touches /exams or /status at all:

docker build -t ghcr.io/kubestronaut/cert-tracker:v0.1.1 .
docker push ghcr.io/kubestronaut/cert-tracker:v0.1.1
# edit the image tag in apps/cert-tracker/base/rollout.yaml
git commit -am "feat: report version in /healthz response (v0.1.1)"
git push

kubectl argo rollouts get rollout cert-tracker -n cert-tracker --watch

# ...a couple minutes in:
# SetWeight  20    Steps  1/5
# AnalysisRun cert-tracker-abc12  Running  (success-rate: measuring...)
# SetWeight  50    Steps  3/5
# AnalysisRun cert-tracker-abc12  Successful  (success-rate: 1.0)
# SetWeight  100   Steps  5/5
# Rollout    cert-tracker  Healthy

Now ship the build worth proving against — not one that crashes outright, since a crashed process fails its readiness probe and never receives a single request, canary or not. v0.2.0 adds pagination to GET /exams behind a new query-parsing helper that throws on a malformed parameter, outside Express's own error handling, and takes the whole process down with it. /healthz never touches that code path, so kubelet keeps calling the pod ready between crash-restarts — but a meaningful slice of real /exams traffic during any given window lands mid-crash and gets a connection reset, which the ingress controller's own metrics record honestly as a 5xx:

kubectl argo rollouts set image cert-tracker cert-tracker=ghcr.io/kubestronaut/cert-tracker:v0.2.0 -n cert-tracker
kubectl argo rollouts get rollout cert-tracker -n cert-tracker --watch

# SetWeight  20    Steps  1/5
# AnalysisRun cert-tracker-def45  Running  (success-rate: measuring...)
# AnalysisRun cert-tracker-def45  Failed   (success-rate: 0.91, want >= 0.98)
# Rollout    cert-tracker  Degraded

kubectl get analysisrun -n cert-tracker
kubectl describe analysisrun cert-tracker-def45 -n cert-tracker
# Message: Metric "success-rate" assessed Failed due to failed (2) > failureLimit (2)

argocd app get cert-tracker
# Sync Status: Synced   -- still. The spec Argo CD applied still matches Git; the abort lives in the Rollout's status.

That last command is the entire point of this section, not a side note. cert-tracker's traffic never left the 20% ceiling this test held it to, the abort happened with nobody watching a dashboard, and the Argo CD Application shows exactly the same Synced it always does — proof that Rollout health and Application sync status are answering two different questions, exactly as Argo Rollouts already warned. Once a real fix ships as v0.2.1, kubectl argo rollouts retry rollout cert-tracker -n cert-tracker clears the abort and re-runs the same steps against it.

What "done" looks like for Part 2, and where Part 3 picks up

☺ Like you're 10: A capsule design that tests itself in small, growing steps and pulls itself back the instant a real reading looks wrong — proven twice, not assumed once.

At the end of this part: cert-tracker runs as a Rollout, gated by a namespaced AnalysisTemplate reading a real metric off the NGINX ingress controller; a weighted split between cert-tracker-stable and cert-tracker-canary moves real traffic in steps; the Application carries the two ignoreDifferences entries that keep selfHeal from fighting the Rollout controller; and you've watched, with your own commands, both a safe release promote itself and a broken one get caught and reversed. Nothing here gets thrown away:

Milestones

☺ Like you're 10: Tick each box only once you've actually watched it happen on your own screen, not because the step "sounds right."

Work these in order — each depends on the cluster and repo state from the one before. Progress saves in this browser.

Part 2 turned a plain Deployment into a release that promotes itself on good numbers and reverses itself on bad ones, with nobody watching a dashboard either way. Continue to Capstone Part 3 — Mesh & Policy, where this same Rollout's traffic moves inside a real mesh. Or step back to Build Your Cert Tracker — Start Here to see how this capstone's five parts fit together, and revisit Argo Rollouts and GitOps Philosophy for the concepts behind what you just built.