Argo Workflows
Where Argo CD never stops — reconciling forever against whatever's currently in Git — Argo Workflows runs jobs that actually finish. It's the Argo family's workflow engine: a Kubernetes-native way to declare a multi-step pipeline as one custom resource, hand it to a controller, and watch it become real pods running in the order — sequential, parallel, or a full dependency graph — the pipeline specifies, then terminate cleanly with a result. It's also the single biggest domain on the CAPA blueprint, at 36% — bigger than Argo CD's own 34%, and the domain most candidates under-study because Argo CD is the famous one. This page goes past the blueprint into the machinery: the controller and executor behind every pod, the six template types that are this project's whole vocabulary, how steps and dag compose them into the fan-out/fan-in shape a real data pipeline needs, how artifacts move files between steps, WorkflowTemplate and CronWorkflow for reuse and scheduling, the CLI, and the gotchas that catch a first pipeline.
Picture a professional kitchen instead of Argo CD's patrolling robot. A head chef reads a recipe card — searing has to happen before plating, but two sous-chefs can chop different vegetables at the very same time. Anything a station finishes gets set on a pass-through shelf so the next station can pick it up without walking over and asking. And once the dish is plated and sent out, that ticket is done — nobody keeps re-cooking it forever the way the patrol robot keeps re-checking its hotel room. Argo Workflows is that kitchen: the recipe card is a Workflow, the stations are pods, the pass-through shelf is an artifact repository, and every ticket eventually closes.
dag template from the inside, and Olly's the same host who carries 36% of the CAPA blueprint and walks the family tree on The Argo Ecosystem.Architecture: the controller, the executor, and where pods actually run
☺ Like you're 10: One piece reads the recipe and decides who cooks next, one piece is the cook standing at the actual stove, and a shelf on the side holds anything a later station needs.
A standard install lays down two Deployments in the argo namespace by default, and neither one is where the actual work happens. workflow-controller watches every Workflow object, walks its dag or steps graph, and creates exactly one Pod per template node the moment that node's dependencies have reported Succeeded — it also ticks every CronWorkflow's schedule and stamps out a new Workflow at each fire. argo-server is the API behind both the web UI and the argo CLI, and optionally backs onto a Postgres or MySQL Workflow Archive so completed runs stay queryable long after their Workflow objects are garbage-collected out of etcd. Neither Deployment runs your pipeline's actual containers — that work happens in the Pod each step gets, wrapped by the configured executor. Emissary has been the default since v3.1: it injects itself as the main container's entrypoint, captures stdout and the exit code without needing access to the container runtime at all, and is the only executor most current installs still ship, now that the older Docker executor — which needed /var/run/docker.sock mounted straight into every step's pod, a real privilege-escalation surface — was removed outright a few releases later.
Argo Workflows has no proprietary execution runtime of its own. Every template that actually runs something becomes exactly one Kubernetes Pod, scheduled and constrained by whatever's already true about the cluster. There's no separate build-agent fleet to size, patch or secure — resource quotas, taints, PodSecurity admission and Kyverno policy already apply to a pipeline step the same way they apply to any other workload, for free.
The Workflow spec and the six template types
☺ Like you're 10: The recipe card has six kinds of instructions — do a thing, run a script, ask Kubernetes to make something, wait for a person, or two ways of ordering the other instructions.
The unit you submit is a Workflow: its spec.entrypoint names one template in spec.templates, and that template is where execution starts. The CAPA curriculum's "Understand Argo Workflow Templates" and "Understand the Argo Workflow Spec" competencies both come down to knowing six template types cold, split into two families. Definitions actually do something: container runs an image with a command; script is the same idea with an inline source block the executor writes to a file and runs, automatically capturing stdout as a default output named result; resource creates, patches, applies, or deletes a Kubernetes object directly, no container involved at all; suspend pauses the workflow, either indefinitely until someone runs argo resume or for a fixed duration. Invocators compose the definitions rather than doing anything themselves: steps is a list of lists — sequential outer, parallel inner — and dag declares each task's dependencies and lets the controller derive both order and parallelism from the graph. Newer releases add containerSet, http, data and plugin on top of these six; useful to recognise, but not what the competency list is pointing at.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: telemetry-etl-
namespace: argo
spec:
entrypoint: pipeline
arguments:
parameters:
- name: orbit-pass
value: "2026-08-27T04-12Z"
templates:
- name: pipeline # dag — the invocator this workflow starts on
dag:
tasks:
- name: ingest
template: ingest-raw
- name: validate
template: validate-quality # script
dependencies: [ingest]
- name: provision
template: provision-scratch-volume # resource
dependencies: [validate]
when: "{{tasks.validate.outputs.result}} != auto-reject"
- name: review
template: hold-for-review # suspend
dependencies: [provision]
- name: finish
template: finish-and-notify # steps
dependencies: [review]
- name: ingest-raw # container
container:
image: kubestronaut/telemetry-ingest:1.3.0
args: ["--pass", "{{workflow.parameters.orbit-pass}}"]
outputs:
artifacts:
- name: raw
path: /out/raw.parquet # written to the artifact repository
- name: validate-quality # script — captures stdout as outputs.result
inputs:
artifacts:
- name: raw
path: /in/raw.parquet
from: "{{tasks.ingest.outputs.artifacts.raw}}"
script:
image: python:3.12-slim
command: [python]
source: |
import sys
# a real check would inspect /in/raw.parquet
print("auto-approved")
- name: provision-scratch-volume # resource — creates + waits, not just applies
resource:
action: create
successCondition: status.phase == Bound
manifest: |
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
generateName: telemetry-scratch-
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 5Gi
- name: hold-for-review # suspend — a human approval gate
suspend: {} # no duration: waits for `argo resume` indefinitely
- name: finish-and-notify # steps — sequential, unlike the dag above
steps:
- - name: archive
template: archive-results
- - name: notify
template: notify-mission-control
- name: archive-results
container:
image: kubestronaut/telemetry-archive:1.1.0
- name: notify-mission-control
container:
image: curlimages/curl:8.7.1
command: ["curl", "-X", "POST", "http://mission-control.internal/notify"]That one manifest touches all six types and shows two composition tricks worth noticing: a dag task can call a steps template as easily as a container one — invocators nest inside invocators — and the when: field on the provision task reads validate's captured result to skip the rest of the pipeline entirely on a bad batch, without a separate conditional-workflow mechanism.
Data-processing DAGs: dynamic fan-out and fan-in
☺ Like you're 10: One station figures out how many helpers are needed and spins up exactly that many, all chopping at once, and the next station won't start plating until every single helper is done.
The example above is a fixed DAG — five named tasks, written by hand. The CAPA curriculum's "Run Data Processing Jobs with Argo Workflows" competency means something more specific: a dynamic fan-out, where the number of parallel tasks isn't known until the pipeline is already running. withItems supplies a literal list written in the manifest; withParam supplies a list computed at runtime — commonly a JSON array in a prior task's captured result, or an explicit outputs.parameters value with valueFrom.path pointing at a file the step wrote. The controller expands one task per array element automatically, each addressable inside its own template via {{item}}, and the downstream fan-in task simply lists the fanned-out task's name once in its own dependencies — the controller already knows that means "every instance," not just one.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: telemetry-batch-etl-
namespace: argo
spec:
entrypoint: pipeline
parallelism: 10 # workflow-wide cap on concurrently running pods
templates:
- name: pipeline
dag:
tasks:
- name: discover
template: list-shards
- name: process-shard
template: process-one-shard
dependencies: [discover]
arguments:
parameters:
- name: shard
value: "{{item}}"
withParam: "{{tasks.discover.outputs.result}}" # dynamic fan-out, N unknown ahead of time
- name: reduce
template: reduce-results
dependencies: [process-shard] # fan-in: waits for every shard, however many N was
- name: list-shards
script:
image: python:3.12-slim
command: [python]
source: |
import json
# in real use: list partitions in the object store for this orbit pass
print(json.dumps(["shard-000", "shard-001", "shard-002", "shard-003"]))
- name: process-one-shard
inputs:
parameters:
- name: shard
retryStrategy:
limit: "3"
retryPolicy: OnTransientError
container:
image: kubestronaut/telemetry-crunch:1.4.0
args: ["--shard", "{{inputs.parameters.shard}}"]
- name: reduce-results
container:
image: kubestronaut/telemetry-reduce:1.0.2Artifacts: passing files between steps through a repository
☺ Like you're 10: Small notes ride along inside the recipe card itself; actual files go on the pass-through shelf instead, and the next station just asks for them by name.
Parameters and artifacts answer the same question — how does data move from one step to the next — for two very different sizes of data. Parameters are small strings, passed via inputs.parameters/outputs.parameters and stored directly inside the Workflow object's own status, visible in plain kubectl get workflow -o yaml. Artifacts are files of any size, declared under outputs.artifacts with a path inside the step's container filesystem, and moved through an external artifact repository rather than through Kubernetes at all — a downstream template consumes one with inputs.artifacts and a from: reference like {{tasks.ingest.outputs.artifacts.raw}}, exactly as the ETL example above does.
None of that works without somewhere to put the files. A cluster-wide default lives in an artifact-repositories ConfigMap, one YAML-in-a-string block per named repo, with an annotation marking which key is the default; a workflow can override it explicitly with spec.artifactRepositoryRef, and a template can override that again.
apiVersion: v1
kind: ConfigMap
metadata:
name: artifact-repositories
namespace: argo
annotations:
workflows.argoproj.io/default-artifact-repository: default-s3-repo
data:
default-s3-repo: |
s3:
bucket: kubestronaut-telemetry-artifacts
endpoint: s3.amazonaws.com
region: ap-south-1
insecure: false
accessKeySecret:
name: telemetry-s3-credentials
key: accessKey
secretKeySecret:
name: telemetry-s3-credentials
key: secretKey
---
# per-workflow override, if this pipeline needs a different bucket than the cluster default
# spec:
# artifactRepositoryRef:
# configMap: artifact-repositories
# key: default-s3-repo"My first pipeline had five steps, and step three's output just quietly went nowhere for two days before anyone noticed. Nobody had ever set up an artifact-repositories ConfigMap on that cluster — every outputs.artifacts block had been failing since the day I wrote it, and the workflow still showed Succeeded, because the step that produced the artifact ran fine; it only choked trying to save it, and the error sat three clicks deep in that one pod's logs instead of anywhere I was looking."
Reuse at scale: WorkflowTemplate, ClusterWorkflowTemplate, and CronWorkflow
☺ Like you're 10: Instead of copying the whole recipe card every time, you keep one master copy on a shelf and just ask for it by name — and a clock can pull a fresh copy down automatically, on schedule.
WorkflowTemplate is a namespaced, reusable collection of templates — write the ETL pipeline above once, save it as a WorkflowTemplate, and every future run references it with workflowTemplateRef instead of re-pasting the whole spec. ClusterWorkflowTemplate is the same idea, cluster-scoped, for steps a platform team publishes once and every namespace can reuse — subject to normal Kubernetes RBAC on who's allowed to submit against it. Individual templates can also be referenced piecemeal with plain templateRef — one dag task pulling in a single named template from a WorkflowTemplate, rather than the whole thing as an entrypoint. CronWorkflow is the scheduling layer on top of either: it mirrors core Kubernetes' own CronJob-from-Job relationship almost field for field — a schedule, a timezone, and a workflowSpec the controller stamps into a brand-new Workflow object at every tick.
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: telemetry-etl-template
namespace: argo
spec:
entrypoint: pipeline
templates:
- name: pipeline
dag:
tasks:
- name: ingest
template: ingest-raw
- name: validate
template: validate-quality
dependencies: [ingest]
- name: ingest-raw
container: { image: "kubestronaut/telemetry-ingest:1.3.0" }
- name: validate-quality
script:
image: python:3.12-slim
command: [python]
source: |
print("auto-approved")
---
apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
name: telemetry-etl-every-orbit-pass
namespace: argo
spec:
schedule: "0 */6 * * *" # every 6 hours — roughly one orbit pass
timezone: "Etc/UTC"
concurrencyPolicy: Forbid # do NOT default this away — see the warning below
startingDeadlineSeconds: 300
successfulJobsHistoryLimit: 5
failedJobsHistoryLimit: 5
workflowSpec:
workflowTemplateRef:
name: telemetry-etl-templateLeave concurrencyPolicy unset on a CronWorkflow and it defaults to Allow: if a run from one tick is still processing when the next tick fires, both run at once, fully independently, with no queue and no warning. For an idempotent read-only report that might be harmless; for a pipeline that writes to a shared table or claims a scratch volume, two overlapping runs racing each other is a real correctness bug, not just wasted compute. Set Forbid to skip a new run outright while one's still in flight, or Replace to terminate the running one and start fresh — the same three-value choice core Kubernetes' CronJob offers, and just as easy to forget here.
Day-to-day commands
☺ Like you're 10: Submit it, watch it, look at the logs, and — if a station's waiting on you — tell it to go ahead.
# submit and watch a workflow run live $ argo submit -n argo --watch telemetry-etl.yaml $ argo submit --from workflowtemplate/telemetry-etl-template -p orbit-pass=2026-08-27T04-12Z $ argo list -n argo # every Workflow, phase, duration $ argo get telemetry-etl-abc12 # full detail: DAG graph, per-node status $ argo logs telemetry-etl-abc12 -f # stream logs across every step's pod $ argo resume telemetry-etl-abc12 # unblock a `suspend` template waiting on a human $ argo retry telemetry-etl-abc12 # re-run only the failed nodes, not the whole DAG $ argo terminate telemetry-etl-abc12 # stop everything now, regardless of phase $ argo delete telemetry-etl-abc12 $ argo lint telemetry-etl.yaml # validate templates and references before submitting $ argo template create telemetry-etl-template.yaml # register a WorkflowTemplate $ argo cron create telemetry-etl-cron.yaml # register a CronWorkflow $ argo cron list -n argo $ kubectl get wf -n argo # `wf` is the built-in short name for Workflow
On a throwaway cluster, submit the ETL example from the templates section above with argo submit --watch. Watch it stop dead at the hold-for-review node — that's the suspend template doing exactly what it's for — then open a second terminal and run argo resume to let it continue. Then delete the artifact-repositories ConfigMap (or never create one) and resubmit: the ingest step still shows Succeeded, and the failure only shows up when you dig into that pod's own logs. That gap between "the step ran" and "the step's output actually landed somewhere" is the single most common first-pipeline surprise. See the data-processing DAG drill for a guided version with a real dynamic fan-out.
Gotchas and failure modes
☺ Like you're 10: Most surprises come from a step reporting "done" a little too early, or from nobody telling old finished tickets to actually leave the counter.
A resource template without successCondition reports success the instant the object exists
An action: create resource template with no successCondition reports Succeeded the moment the Kubernetes API accepts the object — not once it's actually ready. The provision-scratch-volume template in the templates section above sets successCondition: status.phase == Bound deliberately; strip that one line out and the dag would happily move on to review while the PVC is still Pending, and whatever mounts it downstream fails for a reason that has nothing to do with the step that actually broke. It's the same sync-status-versus-health-status trap Argo CD guards against explicitly — except a resource template gives you no built-in health check at all unless you write the condition yourself.
Unbounded withParam fan-out has no default concurrency cap
withParam or withItems generating a hundred items means a hundred Pods requested from the scheduler in the same reconcile pass unless something says otherwise — Argo Workflows sets no default limit. The telemetry-batch-etl- example above sets spec.parallelism: 10 at the workflow level for exactly this reason; a parallelism field on an individual template caps that one node's fan-out instead, for when different stages of the same pipeline need different limits. Skip both and a big dynamic batch can starve every other workload sharing that node the instant it starts.
Skip ttlStrategy and podGC, and finished workflows never leave
A Workflow that reaches Succeeded or Failed just sits there by default — the object in etcd, every one of its pods still on the node — until someone runs argo delete by hand. spec.ttlStrategy.secondsAfterCompletion garbage-collects the Workflow object itself after a delay; spec.podGC.strategy: OnWorkflowSuccess (or OnPodSuccess, for cleanup mid-run rather than only at the end) removes the pods separately, since deleting the Workflow object alone doesn't automatically take its pods with it. Neither is set by default, and a cluster running frequent CronWorkflows without either fills up with completed pods fast enough to matter.
successCondition stops a resource template from lying about readiness. parallelism stops a dynamic fan-out from taking down a node. ttlStrategy and podGC stop finished runs from piling up. None of the three is on by default — all three are one line each.
Argo Workflows vs. the alternatives
☺ Like you're 10: Other kitchens can run a recipe too — they just trade a friendlier menu, a narrower specialty, or living outside the kitchen entirely.
| Option | Model | Best when | Costs you |
|---|---|---|---|
| Argo Workflows | Kubernetes-native DAG/steps engine; every step is a Pod; CRD-based | Already Kubernetes-native, want data-processing DAGs alongside GitOps and progressive delivery in the same family | No built-in event listening on its own — pairs with Argo Events for that; UI is functional, not a dedicated CI product |
| Tekton | Kubernetes-native, CI/CD-focused: Task/Pipeline CRDs, plus Tekton Triggers for events | Building a CI/CD platform specifically, and want a CNCF alternative with its own dedicated Triggers and Dashboard ecosystem | Weaker fit for general-purpose, non-CI data DAGs; more CRDs to learn (Task, TaskRun, Pipeline, PipelineRun) |
| Kubeflow Pipelines | Built on top of Argo Workflows itself, adding a Python SDK and an ML-experiment-tracking UI | The team is doing ML/data-science pipelines specifically and wants first-class experiment tracking on top | A heavier install, and ML-specific abstractions leak through the moment you need one plain step — you're still learning Argo Workflows underneath either way |
| Apache Airflow | Long-running scheduler with DAGs written as Python; not Kubernetes-native by default (a KubernetesExecutor is one deployment option) | The team already lives in Python DAGs, and pipelines mix Kubernetes work with non-Kubernetes systems | Its own scheduler and metadata database to run and back up; a DAG is Python code, not a Kubernetes object other tools in this ecosystem can watch or diff |
| Plain Job / CronJob | Kubernetes' own batch primitives — one Pod (or a fixed count), no dependency graph | One step, no dependencies, and a full workflow engine is genuinely overkill | No template reuse, no artifacts, no DAG — chaining more than one step means hand-rolling init containers or a second Job that polls for the first |
Argo Workflows doesn't appear as a named domain on the core Kubernetes exams — CKA, CKAD and CKS examine the built-in Job/CronJob primitives it builds on top of, not this CRD specifically — but it's 36% of CAPA on its own, and it goes deeper still, in far more hands-on operational depth, on the Platform Engineering course's own Argo Workflows reference for anyone continuing on toward the CNPE afterward.
Olly the Octopus: Fan-out's done — forty shard pods finished clean, reduce just started.
Foxy: Forty pods at once? Doesn't that usually fall over?
Olly: It would have, if I hadn't set parallelism: 10. Argo Workflows doesn't cap a fan-out on its own — I have to say the number.
Gizmo: Or just don't cap it. Bigger burst, faster pipeline, right? 😈
Timmy the Turtle: Wrong — that's forty pods hitting the scheduler in one reconcile with zero limit, starving whatever else is running on that node.
Benny the Beaver: Same instinct as skipping a resource request on a Job. Cheap right up until the one run it isn't.
Olly: And cap or no cap, reduce still waits for every last shard to report Succeeded before it starts. The DAG doesn't move on early either way.
1. Name the six Argo Workflows template types and split them into "definitions" and "invocators." 2. What's the practical difference between steps and dag, and why does a data-processing fan-out/fan-in usually reach for dag? 3. What happens to a step's outputs.artifacts if no artifact repository is configured, and when does that failure actually surface? 4. What does a resource template's successCondition protect against, and what happens if you leave it out? 5. Why can an unbounded withParam fan-out threaten the rest of the cluster, and what setting fixes it? 6. What's the difference between a WorkflowTemplate and a ClusterWorkflowTemplate, and between workflowTemplateRef and plain templateRef? 7. What does a CronWorkflow actually create at each scheduled tick, and which core Kubernetes object does that mirror?
Check your answers
- Definitions (they run something or act):
container(runs an image),script(runs an inline script, capturing stdout asresult),resource(creates/patches/deletes a Kubernetes object directly),suspend(pauses for a human or a duration). Invocators (they compose the others):steps(a list of lists — sequential outer, parallel inner) anddag(tasks withdependencies, order and parallelism derived automatically). stepsis fixed at write time — you hand-write every sequential group and every parallel item.dagderives execution order from each task's declareddependencies, which is what lets it combine withwithParamfor a fan-out whose size isn't known until runtime; a fixedstepslist can't expand itself the same way.- The artifact save fails silently from the pipeline's point of view — the step that produced it can still report
Succeeded, because running the step and saving its output are two separate operations. The failure surfaces only if you open that specific pod's own logs, or later, when a downstream step that consumes the missing artifact fails for what looks like an unrelated reason. - It protects against the resource template reporting success the instant the Kubernetes API merely accepts the object, regardless of whether it ever becomes healthy. Without it, a
dagproceeds to the next task while, say, a PersistentVolumeClaim is stillPending, and the real failure shows up downstream, disguised as some other step's problem. - Argo Workflows sets no default cap on how many pods a
withParam/withItemsfan-out can request at once, so a large dynamic list can flood the scheduler and starve other workloads on the same nodes.spec.parallelism(workflow-wide) or a per-templateparallelismfield caps concurrent pods for that fan-out. - WorkflowTemplate is namespaced; ClusterWorkflowTemplate is cluster-scoped and reusable across namespaces, subject to RBAC.
workflowTemplateRefruns the referenced template's wholeentrypointas if it were pasted in; plaintemplateRefpulls in just one named template from it, for use as a single node inside a different workflow's own graph. - It creates a brand-new
Workflowobject, stamped fromspec.workflowSpec, at every scheduled tick — mirroring core Kubernetes'CronJob, which creates a newJobobject at every tick the same way.