Drill — Build a Data-Processing DAG

Argo Workflows' biggest exam domain isn't Argo CD's job — it's Argo Workflows itself, and the single competency worth the most marks on the CAPA is "Work with DAG (Directed-Acyclic Graphs)" paired with "Run Data Processing Jobs." Today you write exactly that, from a blank file: one Workflow that pulls a batch of sales data, fans it out into two segments that process in parallel, and fans back in to a single load step that can't start until both finish. The part most first attempts get wrong isn't the fan-out or the fan-in — it's passing real artifacts (files) between tasks instead of quietly falling back to small string parameters, which is the distinction this drill is built to drill. It's fully self-contained: a throwaway kind cluster and Argo Workflows' own quick-start install, no dependency on the capstone. Budget 25-40 minutes.

Why dag instead of steps

☺ Like you're 10: steps is a to-do list you read top to bottom. dag is a family tree — you just say who's whose parent, and the robot figures out who can go at the same time.

Argo Workflows gives you two ways to compose templates into a pipeline. steps is a list of lists: the outer list runs in order, and anything grouped inside one inner list runs in parallel. It's easy to read but the concurrency is implicit in how you nest brackets — reordering a pipeline means physically moving JSON around. dag is the other shape: you declare tasks and, for each one, a dependencies list naming which other tasks must finish first. The workflow-controller derives the execution order and the maximum safe parallelism from that graph itself. Nothing runs before its declared dependencies finish; anything with no undone dependencies left runs immediately, concurrently, with no extra syntax for "at the same time." That's exactly the shape a real ETL pipeline has — extract once, transform several segments independently, load once everything is ready — so dag is what the CAPA curriculum means by "Run Data Processing Jobs," and it's what you're building today.

Stand up the scratch world

☺ Like you're 10: One cluster, one install command — and that one command quietly sets up a tiny toy filing cabinet for your artifacts too.

Stand up a fresh kind cluster, then install Argo Workflows with its own quick-start manifest — not the minimal one. The full quick-start bundles the workflow-controller, the argo-server UI/API, and a throwaway MinIO instance already wired up as the default artifact repository. That last piece is exactly what today's drill needs: somewhere real for artifacts to actually live between tasks.

kind create cluster --name dag-drill

kubectl apply -f https://github.com/argoproj/argo-workflows/releases/latest/download/quick-start.yaml
kubectl -n argo wait --for=condition=Available deploy/workflow-controller --timeout=180s
kubectl -n argo wait --for=condition=Available deploy/argo-server --timeout=180s
kubectl -n argo wait --for=condition=Available deploy/minio --timeout=180s

argo version --short

The quick-start manifest creates its own argo namespace, so there's no separate kubectl create namespace step. It also sets argo-server's auth mode to server — meaning the UI needs no bearer token — which is convenient for a scratch cluster and exactly the kind of shortcut you'd never want in production. Optionally, open the UI to watch the graph render live as you submit:

kubectl -n argo port-forward svc/argo-server 2746:2746 >/tmp/argo-pf.log 2>&1 &
# then open https://localhost:2746 — accept the self-signed cert warning, throwaway cluster only

Template one: extract, and declare it as an artifact

☺ Like you're 10: Write the file to disk, then tell Argo "the thing at this path is the box — hand it to whoever asks."

Start a new file, dag.yaml. Every template that needs to hand off real data does two things: it writes a file inside its own container, and it declares that file path under outputs.artifacts. Argo's artifact-repository sidecar logic does the rest — on that path existing when the container exits, it uploads it to MinIO automatically, no extra code in your script.

# dag.yaml — part 1: the Workflow shell and the extract template
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: sales-etl-
spec:
  entrypoint: pipeline
  arguments:
    parameters:
      - name: batch-date
        value: "2026-08-27"
  templates:

    - name: extract
      script:
        image: alpine:3.19
        command: [sh]
        source: |
          mkdir -p /tmp/out
          cat <<EOF > /tmp/out/sales.csv
          segment,amount
          electronics,140
          groceries,310
          electronics,90
          groceries,55
          electronics,205
          EOF
          echo "extract: wrote $(wc -l < /tmp/out/sales.csv) lines for batch {{workflow.parameters.batch-date}}"
      outputs:
        artifacts:
          - name: raw
            path: /tmp/out/sales.csv

That {{workflow.parameters.batch-date}} reference is a parameter — a small string, substituted inline, good for things like a batch date or a segment name. The outputs.artifacts block is the other channel entirely: raw is now a named artifact any later task can reference as {{tasks.extract.outputs.artifacts.raw}}.

Fan out: two independent transforms, one template

☺ Like you're 10: Write the sorting instructions once, then use them twice with a different label each time — that's what makes it reusable instead of copy-pasted.

You need two transform tasks — one per sales segment — and they should not need two separate templates. Write transform once, parameterized on which segment it's filtering for, and reference it from two different dag tasks that each pass a different value. Because neither transform task depends on the other, only on extract, the controller runs them concurrently the instant extract finishes.

# dag.yaml — part 2: the reusable transform template (append inside templates:)
    - name: transform
      inputs:
        parameters:
          - name: segment
        artifacts:
          - name: raw
            path: /tmp/in/sales.csv
      retryStrategy:
        limit: "2"
        retryPolicy: OnTransientError
      script:
        image: alpine:3.19
        command: [sh]
        source: |
          mkdir -p /tmp/out
          grep "^{{inputs.parameters.segment}}," /tmp/in/sales.csv > /tmp/out/rows.csv || true
          COUNT=$(wc -l < /tmp/out/rows.csv)
          TOTAL=$(awk -F, '{s+=$2} END {print s+0}' /tmp/out/rows.csv)
          echo "{{inputs.parameters.segment}},$COUNT,$TOTAL" > /tmp/out/summary.csv
          echo "transform[{{inputs.parameters.segment}}]: $COUNT rows, total $TOTAL"
      outputs:
        artifacts:
          - name: processed
            path: /tmp/out/summary.csv

Two things to notice. First, inputs.artifacts is the mirror image of outputs.artifacts — Argo downloads whatever artifact the caller wires in and stages it at /tmp/in/sales.csv before the script even starts; the container never sees MinIO, an endpoint, or a bucket name. Second, the retryStrategy is free realism: transient failures are common enough in real data pipelines that "run it, and retry a flaky step a couple of times" belongs on any task that isn't purely idempotent-and-cheap already.

Fan in: load can't start until both branches land

☺ Like you're 10: The fourth kid doesn't touch anything until both sorted piles are on the table — not one, not "close enough," both.

The dag block itself is what ties extract, the two transforms, and load together — and it's the only place fan-out and fan-in are actually declared. load lists two dependencies, so the controller will not schedule it until both transform-electronics and transform-groceries report Succeeded:

# dag.yaml — part 3: the dag template and the load template
    - name: pipeline
      dag:
        tasks:
          - name: extract
            template: extract

          - name: transform-electronics
            template: transform
            dependencies: [extract]
            arguments:
              parameters:
                - name: segment
                  value: electronics
              artifacts:
                - name: raw
                  from: "{{tasks.extract.outputs.artifacts.raw}}"

          - name: transform-groceries
            template: transform
            dependencies: [extract]
            arguments:
              parameters:
                - name: segment
                  value: groceries
              artifacts:
                - name: raw
                  from: "{{tasks.extract.outputs.artifacts.raw}}"

          - name: load
            template: load
            dependencies: [transform-electronics, transform-groceries]
            arguments:
              artifacts:
                - name: electronics
                  from: "{{tasks.transform-electronics.outputs.artifacts.processed}}"
                - name: groceries
                  from: "{{tasks.transform-groceries.outputs.artifacts.processed}}"

    - name: load
      inputs:
        artifacts:
          - name: electronics
            path: /tmp/in/electronics.csv
          - name: groceries
            path: /tmp/in/groceries.csv
      script:
        image: alpine:3.19
        command: [sh]
        source: |
          echo "== combined load, batch {{workflow.parameters.batch-date}} =="
          cat /tmp/in/electronics.csv /tmp/in/groceries.csv

Set spec.entrypoint: pipeline at the top of the file (in part 1) so the workflow actually starts at the dag template — that's the one field that ties the whole file to this specific graph instead of leaving it just a bag of reusable templates.

Submit it and watch the graph resolve

☺ Like you're 10: Hit send, then watch the family tree light up node by node instead of top to bottom.

Submit with --watch to see the shape of execution in your terminal, not just the final result:

argo submit -n argo dag.yaml --watch

Name:                sales-etl-abc12
Namespace:           argo
Status:              Succeeded
Progress:            4/4

STEP                          TEMPLATE   PODNAME                    DURATION
✔ sales-etl-abc12             pipeline
├─✔ extract                   extract    sales-etl-abc12-...        4s
├─✔ transform-electronics     transform  sales-etl-abc12-...        3s
├─✔ transform-groceries       transform  sales-etl-abc12-...        3s
└─✔ load                      load       sales-etl-abc12-...        2s

Two things confirm the shape actually behaved: the two transform-* rows have overlapping, not sequential, timestamps if you watch it live or check argo get -o json for their startedAt fields — and load's startedAt is never earlier than the later of the two transforms' finishedAt. If you opened the UI, the graph view draws exactly the diamond shape from the schematic above: one node, splitting to two, rejoining to one.

Prove the artifacts actually moved

☺ Like you're 10: Don't just trust the green checkmarks — read what each step actually printed, and make sure the numbers add up.

Green checkmarks confirm the containers exited zero; they don't confirm the right data crossed the wire. Pull the logs and check the arithmetic yourself:

argo logs -n argo @latest | grep -E "extract:|transform\[|combined load" 

extract: wrote 6 lines for batch 2026-08-27
transform[electronics]: 3 rows, total 435
transform[groceries]: 2 rows, total 365
== combined load, batch 2026-08-27 ==
electronics,3,435
groceries,2,365

extract counted a header line plus five data rows. Each transform saw only the rows matching its own segment — never the other one's — because each got a fresh, independently-staged copy of the same raw artifact, filtered inside its own container. And load printed both summaries, meaning the two artifacts named in its inputs.artifacts really did arrive as files on disk, not as a note claiming they existed.

The transferable habit

☺ Like you're 10: The shape — one thing splits, several things run at once, one thing waits for all of them — shows up everywhere once you know to look for it.

Nothing about today's pipeline needed exotic YAML — four templates, one dag block, a handful of dependency arrows. The skill worth keeping is recognizing the shape: any job with an independent-work phase in the middle is a fan-out/fan-in candidate, whether that's transforming data segments, running the same test suite against five environments, building three container images from one source checkout, or fanning a notification out to every downstream team and waiting for all of their acks before closing a ticket. And the artifact-versus-parameter line generalizes just as cleanly past Argo Workflows entirely — it's the same question a CI/CD pipeline in any tool asks about a build cache, a test report, or a compiled binary: is this small enough to be a string, or does it need to actually be a file?

Pipeline succeeded and the arithmetic checked out? That's the whole drill. For the concepts underneath it, see The Argo Ecosystem and the Argo Workflows tool reference; for where this exact skill is examined, see the CAPA blueprint, whose largest single domain — Argo Workflows at 36% — is built almost entirely from DAGs, templates, and artifacts like the ones you just wrote. Ready for a different single skill? Try Drill — Diagnose a Stuck Argo CD Sync, or go build the full loop in Build Your Cert Tracker — Start Here.