Drill — Diagnose a Stuck Argo CD Sync

You ship a canary release. Argo CD says Synced — the cluster matches git, exactly what you asked for. And yet nothing is actually working: traffic never shifts, the rollout never promotes, and the Application card sits there glowing an angry Degraded red underneath that reassuring "Synced" label. This drill hands you that exact contradiction, self-inflicted by one missing block in one YAML file, and asks you to find it the way an on-call engineer would — from the evidence the tools already give you, not from re-reading the manifests until something looks wrong. It's fully self-contained: a throwaway kind cluster, Argo CD, Argo Rollouts, and Istio, no dependency on the capstone. Budget 25-40 minutes before you open the walkthrough below.

What "stuck" actually means here

☺ Like you're 10: Argo CD keeps two separate scoreboards, not one — and today only one of them is red.

Every Argo CD Application carries two independent status fields, and conflating them is the single most common reason a "stuck sync" takes longer to find than it should. Sync Status answers one question only: does the live state of every resource in the cluster match what's declared in git, right now? It's a diff, nothing more — it has no opinion on whether the resources it's comparing are actually functioning. Health Status answers a completely different question: is each resource, once applied, actually doing its job? For a Deployment, health means the right number of replicas are ready. For an Argo Rollouts Rollout — which Argo CD ships a built-in health check for — health means the rollout's own status.phase, which can be Healthy, Progressing, Paused, or Degraded. Today you'll watch an Application sit at Synced and Degraded at the same time, and that combination is not a contradiction in Argo CD's model — it's two honest, independent answers to two different questions.

Build the scratch world

☺ Like you're 10: One cluster, four control planes, one tiny throwaway git repo — set up once, used for the whole drill.

Stand up a fresh kind cluster and the three pieces this drill needs on top of plain Kubernetes: Argo CD (the reconciler), Argo Rollouts (the canary controller), and Istio (the mesh that actually moves traffic).

kind create cluster --name stuck-sync-drill

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl -n argocd rollout status deploy/argocd-server --timeout=180s

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

istioctl install --set profile=demo -y
kubectl label namespace default istio-injection=enabled --overwrite

Log in to the Argo CD CLI against the in-cluster server:

kubectl -n argocd port-forward svc/argocd-server 8080:443 >/tmp/argocd-pf.log 2>&1 &
ARGO_PWD=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)
argocd login localhost:8080 --username admin --password "$ARGO_PWD" --insecure

Now the scratch repo. This is a real, tiny GitHub repo — Argo CD's repo-server runs inside the cluster and can't reach a plain local filesystem path on your laptop, so a throwaway remote is the fastest thing that actually works:

mkdir stuck-sync-drill && cd stuck-sync-drill
git init -b main
mkdir manifests
gh repo create stuck-sync-drill --private --source=. --remote=origin

Ship the broken canary

☺ Like you're 10: Four files, three of them perfectly correct, one of them missing exactly one paragraph.

You're rolling out checkout as an Argo Rollouts canary, traffic-shifted through Istio instead of raw pod counts. Four manifests, all in manifests/. The Service is unremarkable:

# manifests/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: checkout
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: checkout

The Rollout replaces what would otherwise be a Deployment, and hands its traffic shifting to Istio by naming a VirtualService and a DestinationRule it expects to already exist:

# manifests/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
spec:
  replicas: 3
  selector:
    matchLabels: { app: checkout }
  template:
    metadata:
      labels: { app: checkout }
    spec:
      containers:
        - name: checkout
          image: ghcr.io/example/checkout:v2
          ports:
            - containerPort: 8080
  strategy:
    canary:
      steps:
        - setWeight: 20
        - pause: { duration: 2m }
        - setWeight: 50
        - pause: { duration: 2m }
        - setWeight: 100
      trafficRouting:
        istio:
          virtualService:
            name: checkout-vs
            routes:
              - primary
          destinationRule:
            name: checkout-destrule
            canarySubsetName: canary
            stableSubsetName: stable

The VirtualService splits traffic between a stable subset and a canary subset by weight — Argo Rollouts will own that weight going forward, stepping it through 0 → 20 → 50 → 100 as the canary steps advance:

# manifests/virtualservice.yaml
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: checkout-vs
spec:
  hosts: ["checkout"]
  http:
    - name: primary
      route:
        - destination: { host: checkout, subset: stable }
          weight: 100
        - destination: { host: checkout, subset: canary }
          weight: 0

And here's the file that shipped incomplete. Someone copied the DestinationRule from before checkout ever had a canary path, added the traffic-routing block to the Rollout above, and never came back to give this file its second subset:

# manifests/destinationrule.yaml — BROKEN, as shipped
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: checkout-destrule
spec:
  host: checkout
  subsets:
    - name: stable
      labels: { rollouts-pod-template-hash: placeholder-stable }
    # canary subset — missing

Commit, push, and create the Argo CD Application pointing at this repo, automated and self-healing like any real GitOps setup would be:

git add -A && git commit -m "canary release for checkout, take one"
git push -u origin main

GH_USER=$(gh api user -q .login)
argocd app create checkout \
  --repo "https://github.com/$GH_USER/stuck-sync-drill.git" \
  --path manifests \
  --revision main \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace default \
  --sync-policy automated \
  --self-heal \
  --auto-prune

argocd app sync checkout

Give it a minute, then look:

$ argocd app get checkout
Name:               argocd/checkout
Sync Status:        Synced to main (a1b2c3d)
Health Status:      Degraded

GROUP                 KIND             NAMESPACE  NAME               STATUS  HEALTH    MESSAGE
                       Service          default    checkout           Synced  Healthy   service/checkout created
networking.istio.io    VirtualService   default    checkout-vs        Synced  Healthy
networking.istio.io    DestinationRule  default    checkout-destrule  Synced  Healthy
argoproj.io            Rollout          default    checkout           Synced  Degraded  rollout is invalid: see kubectl describe

Every single resource is Synced — git and cluster agree on all four objects, no drift anywhere. And the Rollout is Degraded, which is why the Application as a whole shows Degraded too: an Application's health is the aggregate of every child resource's health, and one Degraded child is enough to sink the whole thing.

Diagnose it like an on-call, not a re-read

☺ Like you're 10: Don't reread the file hunting for the typo. Ask the Rollout itself what it's stuck on — it already knows.

Start with the tool that watches the rollout specifically:

kubectl argo rollouts get rollout checkout --watch

Name:            checkout
Namespace:       default
Status:          ✖ Degraded
Message:         InvalidSpec: The Rollout "checkout" is invalid
Strategy:        Canary
  Step:          0/5
  SetWeight:     20
  ActualWeight:  0
Images:          ghcr.io/example/checkout:v2 (stable)

NAME                              KIND        STATUS        AGE  INFO
⟳ checkout                        Rollout     ✖ Degraded    3m
└──# revision:1
   └──⧉ checkout-58f7d9c8b4       ReplicaSet  ✔ Healthy     3m   stable
      ├──□ checkout-...-abc12     Pod         ✔ Running     3m
      ├──□ checkout-...-def34     Pod         ✔ Running     3m
      └──□ checkout-...-ghi56     Pod         ✔ Running     3m

ActualWeight: 0 — three healthy pods, zero traffic ever shifted, Step 0/5 never even started. That's your "stuck sync" from the outside: a canary that looks alive but is frozen at the very first step. Now ask why, from the object's own conditions, not a guess:

kubectl describe rollout checkout

Conditions:
  Type          Status  Reason
  ----          ------  ------
  InvalidSpec   True    InvalidSpec
Events:
  Type     Reason        Age   From                Message
  ----     ------        ----  ----                -------
  Warning  InvalidSpec   4m    rollout-controller  The Rollout "checkout" is invalid:
                                spec.strategy.canary.trafficRouting.istio.destinationRule:
                                Invalid value: "checkout-destrule": the DestinationRule
                                "checkout-destrule" does not have the subset(s) "canary"

There's the exact claim, in one line, from the one controller that actually tried to use the missing subset: the Rollout named canarySubsetName: canary, went looking for a subset called canary inside the checkout-destrule DestinationRule, and didn't find one. Confirm it directly against the object itself, not the controller's report of it:

kubectl get destinationrule checkout-destrule -o yaml

spec:
  host: checkout
  subsets:
  - labels:
      rollouts-pod-template-hash: placeholder-stable
    name: stable
# that's it — no second entry

One subset. Exactly what kubectl describe already told you, now confirmed straight from the resource's own spec.

Fix it, and prove it holds

☺ Like you're 10: Add the missing paragraph, push it, and watch the frozen robot start moving on its own.

The fix is exactly the one missing block — add the canary subset the Rollout was already asking for:

# manifests/destinationrule.yaml — the fix
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: checkout-destrule
spec:
  host: checkout
  subsets:
    - name: stable
      labels: { rollouts-pod-template-hash: placeholder-stable }
    - name: canary
      labels: { rollouts-pod-template-hash: placeholder-canary }

Commit and push — selfHeal: true means Argo CD will pick this up on its own within its next poll interval, or force it immediately:

git add -A && git commit -m "fix: add missing canary subset to checkout DestinationRule"
git push
argocd app sync checkout

Watch the rollout directly — the InvalidSpec condition should clear and the canary should actually start stepping:

kubectl describe rollout checkout | grep -A2 Conditions
kubectl argo rollouts get rollout checkout --watch

ActualWeight should now be climbing off zero, and Step should read 1/5 or further, no longer parked at 0/5. Each pause: { duration: 2m } step really does wait two minutes by design — fine for a real release, slow for a 25-40 minute drill, so promote past both pauses by hand to see the whole thing land:

kubectl argo rollouts promote checkout
# wait for the next pause, then:
kubectl argo rollouts promote checkout --full

Done when: argocd app get checkout shows Sync Status: Synced and Health Status: Healthy together, kubectl describe rollout checkout has no InvalidSpec condition left, and kubectl argo rollouts get rollout checkout shows Step 5/5, ActualWeight: 100, and the old stable ReplicaSet scaled to zero.

The transferable habit

☺ Like you're 10: When something's "stuck," ask which scoreboard is actually red before you start reading code.

Nothing about today's fix required deep Istio expertise — it was one missing YAML block, four lines. The skill worth keeping is the sequence you used to find it: notice that Sync and Health disagreed, trust that disagreement instead of assuming "Synced" means "fine," ask the one controller that actually owns the broken behavior what it's stuck on (kubectl describe, not a guess), and confirm its claim directly against the resource it's pointing at. That sequence generalizes past Istio subsets completely — a Certificate that's Synced but never issues, a HorizontalPodAutoscaler that's Synced but can't read a metric, an Ingress that's Synced but 404s. Every one of those is the same shape: the git-to-cluster diff is clean, and the actual failure is one layer beneath it, in a controller with its own opinion about whether things are working.

Fixed and holding? Good — that's the whole drill. For the concepts underneath it, see GitOps Philosophy, Service Mesh Architecture, and The Argo Ecosystem; for the certifications this exact bug class shows up on, see CGOA, CAPA, and ICA — whose own blueprint calls a missing-subset reference "the single most common self-inflicted 503" on that exam. Tool references: Argo CD and Argo Rollouts. Ready for a different single skill? Try Drill — Lock Down a Mesh Namespace, or go build the full loop in Build Your Cert Tracker — Start Here.