CI/CD & Progressive Delivery

Last lesson gave you the destination: a Git repo a controller continuously reconciles — GitOps is the CD reconciler. But something must fill that repo with a tested, scanned, trustworthy image tag, then let real users onto the new version slowly enough that a mistake hurts ten people, not ten thousand. That’s this lesson: the CI pipeline that feeds GitOps, and the progressive-delivery strategies (blue/green and canary) that turn a deploy into a safe, gradual, automatically-judged release.

CI vs CD — and the clean handoff

☺ Like you’re 10: CI is the machine that builds and checks your toy; CD hands it out. This lesson is mostly about building the first machine well — and handing the toy out carefully.

Continuous Integration (CI) merges small changes often and proves each is safe: every push is built, unit-tested, packaged into a container image, and scanned. Continuous Delivery / Deployment (CD) gets it out — and the two letters after the slash matter: continuous delivery keeps every green build ready to release (a human clicks the button); continuous deployment ships it automatically. In a GitOps platform that CD half is last lesson’s reconciler — Argo CD or Flux pulling from Git — so your pipeline’s job is narrower than you’d think.

The pipeline’s real job: end by committing to Git

The single most important idea on this page: a GitOps-friendly CI pipeline does not finish by running kubectl apply or helm upgrade at your cluster. Its final act is two safe writes — push the image to a registry, then commit a config-repo change that bumps the image tag (often a pull request) — then it stops, and the reconciler rolls the new commit out. This is the GitOps mantra made concrete: CI pushes, CD pulls. The pipeline holds no cluster credentials, and every deploy is a reviewable, revertible Git commit.

Shift security left — into the pipeline

☺ Like you’re 10: Check the toy for sharp edges on the conveyor belt, before it reaches a kid — not after someone gets hurt.

The exam’s security domain lives right here too. “Shift left” means running the checks early, as build steps that can fail the build: generate an SBOM (a bill of materials, e.g. with Syft), scan for vulnerabilities (Trivy or Grype) and block on criticals, and sign the image (Sigstore cosign) so the cluster can later verify provenance. A signed, scanned, inventoried image is what makes the rest of security & policy enforceable at admission time — the pipeline is where Benny and Timmy shake hands.

Kubernetes-native pipelines

☺ Like you’re 10: Run the build-machine outside the cluster (a robot you rent) or inside it (a robot made of the same Lego bricks as everything else). Tekton and Argo Workflows are the inside kind.

Two broad choices for where the conveyor belt runs. External runners — GitHub Actions, GitLab CI, Jenkins — execute on managed or self-hosted machines outside your cluster. Huge ecosystems, right next to your source, which is why most teams start there; keep them honest about the handoff — build, push, commit, no cluster admin. Kubernetes-native pipelines run the build as pods inside the cluster, defined as Custom Resources, so the pipeline shares one API, one RBAC model, and one audit log with everything else.

Tekton — pipelines as Kubernetes resources

Tekton is the Kubernetes-native CI engine to know cold — a Continuous Delivery Foundation project (not CNCF, a common trip-up), but pure Kubernetes CRDs end to end. A Task is an ordered list of steps, each a container running a command (clone, build, test). A Pipeline arranges Tasks into a graph — runAfter for ordering, omit it for parallel. A PipelineRun (or TaskRun) is one execution, recording status and logs. Workspaces are shared volumes threaded through Tasks, so source checked out by one reaches the next that builds it. Params feed values in; Results pass small values out. Tekton Triggers turn a webhook into a PipelineRun, closing the loop from “git push” to “pipeline runs.”

apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: build-image
spec:
  params:
    - name: image                       # e.g. registry.acme.io/checkout:1.4.3
      type: string
  workspaces:
    - name: source                      # the checkout from a previous Task
  steps:
    - name: build-and-push
      image: gcr.io/kaniko-project/executor:latest
      args:                             # Kaniko builds in userspace — no Docker daemon
        - --context=$(workspaces.source.path)
        - --dockerfile=$(workspaces.source.path)/Dockerfile
        - --destination=$(params.image)
---
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: build-and-ship
spec:
  params:
    - name: image
    - name: repo-url
  workspaces:
    - name: shared                      # one volume shared across all Tasks
  tasks:
    - name: fetch
      taskRef: { name: git-clone }      # reusable Task from the Tekton catalog
      params:
        - { name: url, value: $(params.repo-url) }
      workspaces:
        - { name: output, workspace: shared }
    - name: scan
      runAfter: [fetch]
      taskRef: { name: trivy-scan }     # fail the run on critical CVEs
      workspaces:
        - { name: source, workspace: shared }
    - name: build
      runAfter: [scan]
      taskRef: { name: build-image }
      params:
        - { name: image, value: $(params.image) }
      workspaces:
        - { name: source, workspace: shared }
    - name: bump-tag
      runAfter: [build]
      taskRef: { name: update-config-repo }   # commit the new tag → GitOps takes over
      params:
        - { name: image, value: $(params.image) }

Argo Workflows — DAGs of containers

Argo Workflows is the other Kubernetes-native engine — a general-purpose workflow controller, sibling to Argo CD. You define a Workflow whose templates are either steps (lists of lists — inner parallel, outer sequential) or a dag (tasks with explicit dependencies). Born on data and ML pipelines, it shines when your CI is a fan-out/fan-in graph. Like Tekton, every step is a pod and the whole pipeline is a CRD — no build server to babysit. The exam just wants you to recognise both as in-cluster, declarative pipelines.

Building images without a Docker daemon

☺ Like you’re 10: Old build machines needed a master key to the whole building to build one toy. Kaniko builds it inside its own sandbox — no master key handed out.

The catch with building inside Kubernetes: the classic docker build needs the Docker daemon — a mounted socket or a privileged pod — handing any pipeline a path to the node and its workloads. So Kubernetes-native builds go daemonless. Kaniko executes each Dockerfile instruction in userspace inside an ordinary pod — no mounted Docker socket, no privileged mode — and pushes straight to a registry. Cloud Native Buildpacks go further — no Dockerfile at all: they detect your language, apply a builder, and produce a reproducible, well-layered OCI image (via the pack CLI, or kpack, the Kubernetes-native buildpacks controller). Rootless BuildKit and Buildah are other daemonless options.

Progressive delivery — deploy is not release

☺ Like you’re 10: Putting the new toy on the shelf (deploy) and handing it to kids (release) are different things. Progressive delivery keeps them separate, so you can undo a mistake before many kids hold it.

Kubernetes’ default Deployment already does a rolling update: it replaces old pods a few at a time (maxSurge/maxUnavailable) so no downtime. But it can’t route traffic by percentage or judge the new version by metrics — the instant a new pod passes its readiness probe the Service adds it to the endpoint pool and it serves real production traffic. Progressive delivery adds the three things it lacks: it decouples deploy from release, controls the traffic split, and makes promotion and rollback automatic and metric-driven, shrinking the blast radius — how many users a bad release touches before you catch it.

Feature flags — the other way to split them

Canary and blue/green split deploy from release at the infrastructure layer (who gets routed to the new binary). Feature flags do it at the code layer: ship the new code “dark” behind an off switch, then flip the flag to release the behaviour — to 5% of users, or a beta cohort — with no redeploy, killing it instantly if it misbehaves. OpenFeature is the CNCF vendor-neutral standard; LaunchDarkly, Unleash, and Flagsmith are implementations. The two compose: a canary picks which users hit the new pods, a flag which users see the new feature. Mature teams use both.

Blue/Green and canary — two rollout shapes

☺ Like you’re 10: Two ways to switch to the new toy: build a whole second playground and flip everyone at once (blue/green), or let a few kids try it and slowly invite more (canary).

Blue/Green — flip the whole thing at once

Run two environments: blue is the current live version, green the new one. Deploy green alongside blue, smoke-test it with zero user traffic, then flip the router (or the Service’s selector) so all traffic goes to green; rollback is just flipping back. Pros: a fast atomic cutover, a clean look at green before anyone sees it, trivial rollback. Cons: two full copies (double resources during the switch), an all-or-nothing flip (a bug past smoke tests hits everyone at once), and any database migration must be backward-compatible so blue and green share data across the flip.

Canary — a sliver at a time

☺ Like you’re 10: Named after the coal-mine canary — send a few travellers down the new road first and watch them, before the crowd.

A canary routes a small slice of live traffic — say 10% — to the new version, keeps the rest on stable, and watches its metrics. Healthy? It ramps: 10% → 25% → 50% → 100%. Not healthy? It aborts, shifting traffic straight back to 0%. The blast radius is far smaller than blue/green — only the current step’s percentage is ever exposed — and you don’t double your whole fleet. The price is complexity: a true split needs real traffic control, and safe promotion real metrics.

The tools — Argo Rollouts & Flagger

☺ Like you’re 10: Two robots do the traffic-cop job. One asks you to swap your Deployment for a fancier version (Argo Rollouts); the other watches your existing Deployment and quietly manages a copy (Flagger).

You don’t hand-code traffic ramps and metric checks — two CNCF-ecosystem projects on the exam’s tool list do it declaratively. Both do canary and blue/green, both drive a mesh or ingress, and both auto-rollback on bad metrics. They differ mainly in where they sit relative to your workload.

Argo Rollouts — a Deployment with a strategy

Argo Rollouts introduces a Rollout resource that replaces your Deployment (same pod template) and adds a strategy.canary or strategy.blueGreen block. A canary strategy is an explicit recipe of steps: setWeight (shift N% to the canary), pause (a fixed duration, or indefinitely as a manual gate), setCanaryScale, and analysis (run an AnalysisTemplate). Under the hood it manages a stable and a canary ReplicaSet and, with a traffic router (Istio, NGINX, ALB, Gateway API, SMI), the real weights. A kubectl argo rollouts plugin and dashboard let you watch or drive a rollout by hand.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
spec:
  replicas: 10
  selector:
    matchLabels: { app: checkout }
  template:                              # identical to a Deployment's pod template
    metadata: { labels: { app: checkout } }
    spec:
      containers:
        - name: checkout
          image: registry.acme.io/checkout:1.4.3
  strategy:
    canary:
      canaryService: checkout-canary     # Services the router splits between
      stableService: checkout-stable
      trafficRouting:
        istio:
          virtualServices:
            - name: checkout
              routes: [primary]
      steps:
        - setWeight: 10                  # 10% to the canary
        - pause: { duration: 5m }        # soak, let metrics accumulate
        - analysis:                      # query Prometheus; abort if unhealthy
            templates:
              - templateName: success-rate
            args:                        # scope the query to the canary Service
              - name: canary-service
                value: checkout-canary.default.svc.cluster.local
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100                 # promote — canary becomes the new stable

Flagger — automate an existing Deployment

Flagger (in the Flux family) automates the same strategies but keeps your normal Deployment. You add a Canary resource pointing at it via targetRef; Flagger creates primary and canary Services, shifts traffic through your mesh/ingress, and runs metric checks each interval — promoting by stepWeight up to maxWeight, or rolling back after threshold failed checks. It has built-in metrics (request-success-rate, request-duration) plus custom MetricTemplates, and webhooks for load and acceptance tests. It watches the pod template and auto-starts a canary whenever the image changes.

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: checkout
spec:
  targetRef:                             # your ordinary Deployment — unchanged
    apiVersion: apps/v1
    kind: Deployment
    name: checkout
  service:
    port: 80
  analysis:
    interval: 1m                         # evaluate every minute
    threshold: 5                         # 5 failed metric checks → roll back
    maxWeight: 50                        # ramp up to 50% before full promotion
    stepWeight: 10                       # +10% each interval: 10,20,30,40,50
    metrics:
      - name: request-success-rate
        thresholdRange: { min: 99 }      # need >= 99% success
        interval: 1m
      - name: request-duration
        thresholdRange: { max: 500 }     # p99 latency <= 500ms
        interval: 1m

Traffic shifting — how Pip routes the canary

☺ Like you’re 10: To send one in ten travellers down the new road, someone must stand at the fork and count. A service mesh or smart ingress is that doorkeeper.

Here’s a subtlety the exam loves. A plain Kubernetes Service load-balances across pods roughly evenly, per connection — so “canary by replica count” (1 new pod among 9) is only a coarse, connection-sticky approximation of 10%, not a true per-request split. For precise, request-level weighting you need weighted routing:

• A service mesh — Istio (a VirtualService splitting weight across subset destinations) or Linkerd — which routes by request and can shift 10/90 exactly.
• An ingress or Gateway — NGINX ingress canary annotations, or the Gateway API HTTPRoute with weighted backendRefs, the emerging standard that both Argo Rollouts and Flagger support.

The mesh does double duty, which is why Pip hosts this section: it shifts the traffic and emits the golden-signal metrics — request success rate and latency per version — that the analysis step reads. Routing and reporting from one layer. (Pip’s other job, mutual-TLS between services, lives in security & policy.)

Automated analysis & rollback

☺ Like you’re 10: The traffic-cop doesn’t guess. She reads the same dashboards the watchtower keeps, and if the new toy is making kids cry, she takes it back herself — no grown-up needed.

This is where delivery meets observability — the payoff of the domain. An analysis step queries a metrics provider — usually Prometheus — for the canary’s golden signals (error rate, p99 latency, saturation) and compares them to a threshold. Define a successCondition, a failureLimit, and — when the analysis runs as a rollout step — a count, so the measurement finishes instead of sampling forever: while the condition holds, the rollout promotes; when the failure limit is breached it aborts automatically — traffic snaps back to stable, the canary ReplicaSet scales to zero, nobody paged at 2am. Bad changes never reach everyone — precisely how your DORA change-failure rate drops.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
    - name: canary-service                 # passed in by the Rollout's analysis step
  metrics:
    - name: success-rate
      interval: 1m
      count: 3                             # 3 measurements, then this step passes
      successCondition: result[0] >= 0.99  # promote only while >= 99% succeed
      failureLimit: 2                      # 2 failing checks → abort the rollout
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(istio_requests_total{reporter="source",destination_service=~"{{args.canary-service}}",response_code!~"5.*"}[2m]))
            /
            sum(rate(istio_requests_total{reporter="source",destination_service=~"{{args.canary-service}}"}[2m]))

Note the query is scoped to the canary via an arg — a query that lumps canary and stable together would drown a 10% canary’s errors in 90% healthy traffic and promote a broken release. Wire that template into the Rollout above (its analysis step already passes the arg) and the canary judges itself; Flagger expresses the same idea with threshold and thresholdRange. Either way, “is this release healthy?” becomes a query the platform runs for you — every minute, at every step.

That completes Domain 2. GitOps gave you the reconciled control panel; CI/CD fills it with trustworthy, signed images, and progressive delivery turns each change into a small, watched, reversible step. Next you climb from deploying apps to offering capabilities — the platform APIs and CRDs and the self-service that let Dot request a database as easily as she ships code. Any unfamiliar term is in the glossary.