Argo Workflows

Argo Workflows is a Kubernetes-native workflow engine: you describe a multi-step job as a custom resource, and the controller runs every step as a pod, passing parameters and files between them, retrying what fails, and cleaning up after itself. It solves the platform problem of “I need to run a sequence of containers with dependencies, on the cluster I already have, without buying a separate orchestration product” — CI builds, batch data pipelines, nightly maintenance, ML training, cluster automation. If Tekton is a CI/CD framework, Argo Workflows is a general-purpose orchestrator that happens to be excellent at CI.

What it is and the problem it solves

☺ Like you’re 10: Kubernetes is great at running one container. It’s bad at running six containers in a particular order. Argo Workflows fills that hole.

Kubernetes gives you a Pod for “run this container” and a Job for “run this container until it succeeds.” Neither gives you “run these six containers, some in parallel, some only after others finish, passing a build artefact from step two into step five, and retry step four if the network flakes.” Teams that need that reach for a CronJob wrapped around a giant shell script, a Jenkins server bolted on the side, or a home-grown Python driver — and every one of those puts orchestration logic outside the cluster’s declarative API, where it can’t be reviewed, versioned, or reconciled.

Everything is a pod, every pipeline is an API object

Argo Workflows takes the opposite bet. A workflow is a Kubernetes custom resource (Workflow, in the argoproj.io/v1alpha1 group). Each step becomes a real pod, scheduled by the real scheduler, subject to your real quotas, network policies, and node affinity — no agent fleet, no build servers to patch. The controller watches Workflow objects, creates pods in dependency order, and records each step’s status back into the object.

Because a workflow is just an API object, everything you already know applies: kubectl get workflows lists them, RBAC governs who can create them, admission policy from Kyverno or Gatekeeper can validate them, and GitOps can deliver them.

Not a CI tool — an orchestration tool

The line to remember: Tekton is CI/CD pipelines; Argo Workflows is general orchestration of Kubernetes resources. Tekton’s vocabulary is built around shipping software — Task, Pipeline, PipelineRun, workspaces, a catalogue of build steps. Argo Workflows’ vocabulary is deliberately generic — templates, DAGs, parameters, artifacts — so the same engine runs a CI build, a genomics pipeline, a nightly database compaction, an ML training sweep, or cluster automation that creates and waits on other Kubernetes objects.

Where it fits in a platform

☺ Like you’re 10: It lives in the “getting work done” part of the platform, not the “keeping websites up” part.

On the platform architecture map, Argo Workflows sits in the delivery and automation plane — the machinery that turns an event or a schedule into work on the Kubernetes substrate. It is not the control plane that keeps services healthy, and it is not a portal. It is the engine behind “something has to happen now, in several stages.”

How it relates to its neighbours

The exam’s tool list simply names “Argo,” so you need the family map straight. Argo CD reconciles desired state from Git — a continuous loop with no end. Argo Rollouts replaces Deployment to give you canary and blue/green promotion. Argo Workflows runs finite jobs that start, do something, and end. Argo Events (the fourth sibling, not on the exam list) turns webhooks and queue messages into workflow submissions.

In a full reference architecture, the pattern is: Argo Workflows (or Tekton) builds and signs the image and writes a new tag into the config repo; Argo CD notices the commit and syncs; Argo Rollouts performs the safe rollout. Three tools, three jobs, one path from commit to production — told end-to-end in CI/CD & Progressive Delivery.

CNPE domain relevance

Workflows show up in more than one domain. Most obviously D2 · GitOps & Continuous Delivery, since “Argo” is on the official tool list. But it also lands in Platform Engineering Core whenever it is the execution engine behind a self-service action (“give me a seeded test database”), and in Observability, because workflow pods emit logs and metrics like everything else. Treat it as the platform’s general-purpose “do a thing” button.

How it works — architecture and CRDs

☺ Like you’re 10: One controller reads your recipe cards and makes pods. A second, optional piece is the website you watch them on.

The runtime is small. The workflow-controller is a single deployment that watches Workflow objects, walks the dependency graph, creates one pod per step, and writes status back. Each workflow pod runs your container alongside an executor — the emissary executor, which is the default in modern releases — that wraps your command, captures its outputs, and copies artifacts in and out. In older versions you had to choose an executor; today you do not, so treat it as an implementation detail rather than a knob. The argo-server is an optional API and web UI; the CLI talks either directly to the Kubernetes API or to argo-server. Artifacts live in an external object store — S3, MinIO, GCS, Azure Blob — because pods are ephemeral and a file written in step one has to survive until step five.

The four custom resources you author

Argo Workflows installs a fair number of CRDs, but only four are ones a human writes. Know these four by name and by purpose — this is the most likely thing to be asked. All four live in the argoproj.io/v1alpha1 API group.

The rest are machinery the controller and executor use among themselves — WorkflowTaskResult is the one you will meet, because every step pod writes one and the RBAC to do so is the single most common installation mistake (see the gotchas below). Argo also ships WorkflowEventBinding for turning API events into submissions. You do not hand-write any of these; you only grant permission on them.

Template types — the real vocabulary

☺ Like you’re 10: A “template” is one instruction on the recipe card. Two kinds of instruction are just “here’s a list of other instructions.”

spec.templates is a list of named templates; spec.entrypoint says which one starts. They come in two families: leaf templates that do something, and orchestration templates that call other templates.

Those six carry almost every real pipeline. A handful of others exist for narrower jobs — containerSet runs several containers inside one pod so they can share a volume without a PVC, and http makes a request without spending a pod at all. Learn the six first; reach for the rest only when the six are genuinely awkward.

Fan-out is a property of the calling task, not a template type: withItems expands a task over a hard-coded list, withParam expands it over a JSON array produced by an earlier step, and withSequence expands it over a numeric range. Each expansion produces its own pod, which is exactly why the parallelism caps in the gotchas section matter.

The resources you will actually write

☺ Like you’re 10: Here are real recipe cards — copy the shapes, not the words.

A DAG workflow with parameters, artifacts and retries

The canonical build pipeline: clone, then test and scan in parallel, then push. Note the parameter plumbing ({{workflow.parameters.*}}, {{inputs.parameters.*}}), the artifact handoff, and the housekeeping fields near the top that stop your cluster filling with dead pods.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: build-           # the API server appends a random suffix per run
  namespace: ci
spec:
  entrypoint: pipeline
  serviceAccountName: workflow-runner    # NOT "default" — see the RBAC gotcha
  arguments:
    parameters:
      - name: repo
        value: https://github.com/acme/checkout.git
      - name: revision
        value: main
  ttlStrategy:
    secondsAfterCompletion: 86400        # delete the Workflow object after 1 day
  podGC:
    strategy: OnWorkflowSuccess          # delete step pods once the run succeeds
  templates:
    - name: pipeline
      dag:
        tasks:
          - name: clone
            template: clone
            arguments:
              parameters:
                - { name: repo, value: "{{workflow.parameters.repo}}" }
                - { name: revision, value: "{{workflow.parameters.revision}}" }
          - name: test
            template: run-make
            dependencies: [clone]        # test and scan both wait on clone…
            arguments:
              parameters: [{ name: target, value: test }]
              artifacts:
                - name: src
                  from: "{{tasks.clone.outputs.artifacts.src}}"
          - name: scan
            template: run-make
            dependencies: [clone]        # …and then run in parallel with each other
            arguments:
              parameters: [{ name: target, value: scan }]
              artifacts:
                - name: src
                  from: "{{tasks.clone.outputs.artifacts.src}}"
          - name: push
            template: run-make
            dependencies: [test, scan]   # push waits for BOTH
            arguments:
              parameters: [{ name: target, value: push }]
              artifacts:
                - name: src
                  from: "{{tasks.clone.outputs.artifacts.src}}"

    - name: clone
      inputs:
        parameters: [{ name: repo }, { name: revision }]
      container:
        image: alpine/git        # pin to a digest in production
        command: [sh, -c]
        args: ["git clone --depth 1 -b {{inputs.parameters.revision}} {{inputs.parameters.repo}} /work/src"]
      outputs:
        artifacts:
          - name: src
            path: /work/src              # tarred and uploaded to the artifact repo
      retryStrategy:
        limit: "3"
        retryPolicy: OnTransientError    # network flakes only, not real failures
        backoff: { duration: "10s", factor: "2", maxDuration: "2m" }

    - name: run-make
      inputs:
        parameters: [{ name: target }]
        artifacts:
          - name: src
            path: /work/src              # downloaded from the artifact repo
      container:
        image: ghcr.io/acme/builder:1.9
        workingDir: /work/src
        command: [make]
        args: ["{{inputs.parameters.target}}"]
        resources:
          requests: { cpu: "500m", memory: 1Gi }
          limits:   { memory: 2Gi }

A reusable WorkflowTemplate driven by a CronWorkflow

Real platforms almost never submit raw Workflow objects. The platform team publishes a WorkflowTemplate (or ClusterWorkflowTemplate), and everything else — a cron schedule, a portal button, an event — references it by name. Below, a nightly report uses a steps template with a script step whose stdout feeds the next step, and a CronWorkflow that fires it. Note the schedule field: recent versions take a list under schedules, while the single schedule string you will still see in older examples is deprecated. Both are accepted today; write the list. If a cluster rejects the list form, it is running an older controller — that alone tells you something useful about the environment.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: nightly-report
  namespace: data
spec:
  entrypoint: main
  serviceAccountName: workflow-runner
  templates:
    - name: main
      steps:                              # outer list = sequential stages
        - - name: pick-partition           # inner list = parallel within a stage
            template: pick
        - - name: crunch-eu
            template: crunch
            arguments:
              parameters:
                - { name: region, value: eu }
                - { name: partition, value: "{{steps.pick-partition.outputs.result}}" }
          - name: crunch-us                # crunch-eu and crunch-us run together
            template: crunch
            arguments:
              parameters:
                - { name: region, value: us }
                - { name: partition, value: "{{steps.pick-partition.outputs.result}}" }
        - - name: approve
            template: hold                 # manual gate before publishing
        - - name: publish
            template: crunch
            arguments:
              parameters:
                - { name: region, value: all }
                - { name: partition, value: "{{steps.pick-partition.outputs.result}}" }

    - name: pick
      script:                              # stdout becomes outputs.result
        image: python:3.12-alpine
        command: [python]
        source: |
          import datetime
          print((datetime.date.today() - datetime.timedelta(days=1)).isoformat())

    - name: hold
      suspend: {}                          # waits forever until `argo resume`

    - name: crunch
      inputs:
        parameters: [{ name: region }, { name: partition }]
      container:
        image: ghcr.io/acme/reporter:2.3
        args: ["--region={{inputs.parameters.region}}", "--date={{inputs.parameters.partition}}"]
---
apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: nightly-report
  namespace: data
spec:
  schedules:                             # a LIST since v3.6; older docs show a single `schedule:` string
    - "17 2 * * *"
  timezone: Europe/London
  concurrencyPolicy: Forbid              # skip if the previous run is still going
  startingDeadlineSeconds: 300           # if the controller was down, don't fire late
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  workflowSpec:
    workflowTemplateRef:
      name: nightly-report               # add clusterScope: true for a ClusterWorkflowTemplate

The resource template — and the RBAC it demands

This is the template that makes Argo Workflows a Kubernetes orchestrator. It creates an arbitrary object, then blocks until a successCondition on that object becomes true. The condition syntax is deliberately simple — a dotted field path compared with ==, or tested with in (…) against a set — evaluated by re-reading the object, so it cannot express arbitrary JSONPath filters over lists. Here a workflow provisions an ephemeral test database through a custom provisioning CRD, waits for it to be ready, and gets the RBAC it needs. A real Crossplane claim reports readiness through its status.conditions array rather than a single status.phase field, which is exactly the kind of shape the simple condition syntax struggles with — check the CRD you are actually waiting on before you write the condition.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata: { name: ephemeral-db, namespace: ci }
spec:
  entrypoint: provision
  serviceAccountName: workflow-runner
  templates:
    - name: provision
      resource:
        action: create
        setOwnerReference: true            # object is GC'd when the Workflow is deleted
        successCondition: status.phase == Ready
        failureCondition: status.phase == Failed
        manifest: |
          apiVersion: acme.io/v1alpha1
          kind: PostgresClaim
          metadata:
            generateName: test-db-
          spec:
            size: small
            version: "16"
      outputs:
        parameters:
          - name: dbname
            valueFrom:
              jsonPath: '{.metadata.name}' # feed the created name to later steps
---
# The ServiceAccount the workflow runs as, and the permissions it needs.
apiVersion: v1
kind: ServiceAccount
metadata: { name: workflow-runner, namespace: ci }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: workflow-runner, namespace: ci }
rules:
  # Required by the executor for EVERY workflow — this is the one people forget:
  - apiGroups: ["argoproj.io"]
    resources: ["workflowtaskresults"]
    verbs: ["create", "patch"]
  # Required only because we use a resource template against PostgresClaim:
  - apiGroups: ["acme.io"]
    resources: ["postgresclaims"]
    verbs: ["get", "list", "watch", "create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: workflow-runner, namespace: ci }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: workflow-runner }
subjects:
  - { kind: ServiceAccount, name: workflow-runner, namespace: ci }

Artifact repository and shared volumes

Artifacts need somewhere to live. The controller reads a default from its own ConfigMap, but the cleanest per-namespace pattern is an artifact-repositories ConfigMap holding one named repository per key. An annotation on that ConfigMap nominates the key to use when nothing else is said; a workflow that wants a different key names both the ConfigMap and the key in spec.artifactRepositoryRef. If your steps only need scratch space within one run and never leave the cluster, a volumeClaimTemplate is simpler and much faster than round-tripping through S3.

apiVersion: v1
kind: ConfigMap
metadata:
  name: artifact-repositories
  namespace: ci
  annotations:
    workflows.argoproj.io/default-artifact-repository: minio-store
data:
  minio-store: |
    s3:
      endpoint: minio.minio.svc:9000
      insecure: true                 # in-cluster MinIO over plain HTTP
      bucket: argo-artifacts
      keyFormat: "{{workflow.namespace}}/{{workflow.name}}/{{pod.name}}"
      accessKeySecret: { name: minio-creds, key: accesskey }
      secretKeySecret: { name: minio-creds, key: secretkey }
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata: { generateName: shared-scratch-, namespace: ci }
spec:
  entrypoint: main
  artifactRepositoryRef:             # optional: pick a key from the ConfigMap above
    configMap: artifact-repositories
    key: minio-store
  volumeClaimTemplates:              # one PVC created per run, deleted with the run
    - metadata: { name: scratch }
      spec:
        accessModes: [ ReadWriteOnce ]
        storageClassName: fast-ssd
        resources: { requests: { storage: 20Gi } }
  templates:
    - name: main
      steps:
        - - { name: generate, template: writer }
        - - { name: consume,  template: reader }
    - name: writer
      container:
        image: busybox:1.36
        command: [sh, -c, "dd if=/dev/urandom of=/scratch/blob bs=1M count=64"]
        volumeMounts: [{ name: scratch, mountPath: /scratch }]
    - name: reader
      container:
        image: busybox:1.36
        command: [sh, -c, "wc -c /scratch/blob"]
        volumeMounts: [{ name: scratch, mountPath: /scratch }]

☺ Like you’re 10: Artifacts are “post it to a locker in the cloud so any later step can fetch it.” A shared volume is “everyone works on the same desk.” The desk is faster; the locker works across nodes and outlives the run.

Day-to-day commands

☺ Like you’re 10: There’s a little program called argo that talks to the robot chef. Six commands cover almost everything.

The argo CLI

# Submit and follow. --watch prints a live tree; -p sets a parameter.
argo submit -n ci build.yaml --watch -p revision=release-2.1

# Submit FROM a stored template (the way real platforms do it)
argo submit -n data --from workflowtemplate/nightly-report --watch
argo submit -n ci   --from clusterworkflowtemplate/golden-build -p repo=https://...

# See what's happening
argo list -n ci                        # all workflows, newest first
argo list -n ci --status Running
argo get  -n ci @latest                # @latest = most recently submitted here
argo watch -n ci build-9k2xp           # live-updating tree view
argo logs -n ci build-9k2xp -f         # stream all step logs
argo logs -n ci build-9k2xp -c main    # one container of one step

# Fix things
argo retry     -n ci build-9k2xp       # re-run ONLY the failed nodes, keep successes
argo resubmit  -n ci build-9k2xp       # brand-new run from the same spec
argo stop      -n ci build-9k2xp       # graceful: exit handlers still run
argo terminate -n ci build-9k2xp       # immediate: kill everything now
argo resume    -n data nightly-report-xxxxx   # release a suspend/approval gate
argo delete    -n ci --completed --older 7d

# Templates and schedules
argo template create wt.yaml -n data
argo template list   -n data
argo cron list       -n data
argo cron suspend nightly-report -n data
argo lint build.yaml                   # validate BEFORE you submit

When the CLI isn’t there

Everything is a CRD, so plain kubectl works and is the fallback you want in an exam terminal or a locked-down cluster. Knowing this pair of equivalences is worth real marks.

kubectl get workflows -n ci                     # short name: wf
kubectl get wf -n ci -o wide
kubectl describe wf build-9k2xp -n ci
kubectl get wf build-9k2xp -n ci -o jsonpath='{.status.phase}'   # Pending|Running|Succeeded|Failed|Error
kubectl create -f build.yaml -n ci              # `create`, not `apply` — generateName
kubectl delete wf --all -n ci

# The controller labels workflows and their pods — useful when the CLI is missing
kubectl get wf -n ci -l workflows.argoproj.io/phase=Failed
kubectl get wf -n ci -l workflows.argoproj.io/completed=true
kubectl get pods -n ci -l workflows.argoproj.io/workflow=build-9k2xp
kubectl logs -n ci build-9k2xp-clone-1234567890 -c main          # `main` = your container

# The controller's own health and config
kubectl -n argo logs deploy/workflow-controller -f
kubectl -n argo get cm workflow-controller-configmap -o yaml
kubectl get workflowtemplates,clusterworkflowtemplates,cronworkflows -A

Note kubectl create rather than apply for a workflow that uses generateName — apply needs a fixed metadata.name and will reject the object. See the command reference for the wider kubectl muscle memory.

Gotchas and failure modes

☺ Like you’re 10: Three things trip up almost everybody: permissions, where files live, and rubbish not getting taken out.

RBAC — the number-one support ticket

Workflow pods run as a ServiceAccount, and by default that is the namespace’s default SA, which has no permissions at all. Two failures follow. First, the executor needs create and patch on workflowtaskresults in argoproj.io to report step outputs — without it, steps run fine but the workflow fails at the end with workflowtaskresults is forbidden. Second, a resource template acts on the cluster as that ServiceAccount, so it needs explicit verbs on the target kind, including get/list/watch if you use a successCondition (Argo has to keep reading the object to evaluate it). Always create a dedicated SA and set serviceAccountName; if a step dies with a 403, that’s your answer. Two refinements worth knowing: serviceAccountName can be set per template as well as on the whole spec, so a single workflow can run its risky resource step under a more privileged identity while every other step stays minimal; and if you enable artifact garbage collection, the controller needs its own permission on the artifact GC task resource — a separate failure with a similar-looking error. General approach: Triage: delivery.

Artifact repository misconfiguration

Artifact failures are late and cryptic. If no repository is configured, workflows that don’t use artifacts run happily and workflows that do fail at upload time — so the problem appears only when someone adds an output. Watch for: the bucket doesn’t exist (Argo does not create it by default), the credentials Secret is in a different namespace from the workflow (it must be in the workflow’s namespace), insecure: true missing for plain-HTTP in-cluster MinIO, and a keyFormat that collides between runs. Whatever store you pick is a real dependency: if S3 is down, your pipelines are down. Background: Storage & State.

Pod and workflow cleanup

By default, completed workflow pods and completed Workflow objects stay. A busy CI namespace accumulates thousands of Completed pods, which bloats etcd, slows every kubectl get pods, and eventually trips node pod limits. Two independent knobs fix it and you need both: podGC.strategy (OnPodCompletion, OnPodSuccess, OnWorkflowCompletion, OnWorkflowSuccess) deletes the pods; ttlStrategy (secondsAfterCompletion, secondsAfterSuccess, secondsAfterFailure) deletes the Workflow object. Note that podGC also takes a label selector, so you can keep the pods of one labelled step for inspection while sweeping the rest. Set both as controller-wide defaults in the controller ConfigMap rather than trusting every author to remember them. But be deliberate: aggressive pod GC deletes the pods whose logs you wanted, so pair it with archiveLogs or log shipping into Loki.

Alternatives and when to choose it

☺ Like you’re 10: Other robot chefs exist. Pick by what you’re cooking.

The field of options

Nothing about Argo Workflows is exotic; every one of its jobs has a competitor that does it differently. What matters for the exam and for real design reviews is being able to say why you picked one, in terms of where the orchestration state lives and who can govern it.

The honest summary: choose Argo Workflows when your orchestration needs are varied and your platform is Kubernetes-first; Tekton when they are narrowly CI/CD; a plain CronJob when there is only one container — a surprisingly common right answer, and one the anti-patterns page will nag you about.

The test that actually decides it

Ask one question: should this pipeline be governed by the cluster? If the answer is yes — you want admission policy to inspect it, RBAC to constrain it, quotas to bound it, and GitOps to deliver it — then the pipeline must be an API object, and that rules out anything whose definition lives in a vendor’s database or a .yml file interpreted by an external runner. If the answer is no, an external CI system you already run is very often the cheaper choice, and adopting a second orchestrator to avoid it is the more expensive mistake.

The second question is about shape, not governance: how much of your work is a genuine graph? A single container is a CronJob. A straight line of build stages is comfortable in either Tekton or Argo Workflows. A real fan-out and fan-in — especially one whose width is computed at runtime — is where Argo Workflows starts earning its keep, because withParam over an earlier step’s output has no clean equivalent in most alternatives.

Running two engines on purpose

Plenty of mature platforms run Tekton for application CI and Argo Workflows for everything else, and that is a defensible choice rather than a failure to decide — the two have different audiences and different blast radii. What is not defensible is running both for the same job because two teams each picked one. If you carry two engines, write down the boundary, put it in the developer experience docs, and make the self-service layer hide the difference. The platform guild exists to settle exactly this kind of argument once instead of per team.

Exam relevance and going further

☺ Like you’re 10: In the exam you can’t look this up on Argo’s website. Learn the shapes now.

What to be able to do cold

“Argo” is on the official CNPE tool list, and the exam is performance-based, so the bar is doing. Be able to: name the four CRDs and their purposes; explain and write both dag and steps; explain what a resource template does and what RBAC it needs; wire an output parameter or artifact from one step into the next; convert a workflow into a WorkflowTemplate and fire it from a CronWorkflow; and read a failing run with argo get, argo logs, and kubectl describe wf. Above all, hold the boundary line: Tekton is CI/CD pipelines; Argo Workflows is general orchestration of Kubernetes resources — and Argo CD, Argo Rollouts, and Argo Workflows are three different projects.

The documentation allowlist — read this twice

⚖ CNPA vs CNPE — That allowlist is a CNPE mechanic: CNPE is hands-on, and permits those narrow lookups mid-task. CNPA is stricter, not looser — a fully closed-book, multiple-choice exam with zero external references of any kind, not even kubernetes.io. Knowing Argo Workflows' CRDs and template types cold is still worth having for CNPA's closed-book recall.

Where to go next on this site — and officially

Continue with CI/CD & Progressive Delivery, the primary lesson this tool serves; compare against Tekton; see how the finished artefact gets deployed in GitOps Workflows and Argo CD; and browse the landscape from the tool hub. Unfamiliar words are in the Glossary; debugging moves in Triage: workloads. Officially — outside the exam — trust the project docs at argo-workflows.readthedocs.io, the examples directory and field reference at github.com/argoproj/argo-workflows, and the Argo home at argoproj.github.io. That examples directory is the fastest way to learn the template vocabulary: a hundred small runnable workflows, one idea each.