GitOps on Kubernetes
Every principle GitOps is built on — declare desired state, store it in Git, apply it automatically, keep checking that the live system still matches — describes something Kubernetes was already doing internally before "GitOps" had a name. The Kubernetes API & the Controller Pattern showed a ReplicaSet controller watching the API, diffing desired state against actual state, and acting to close the gap, forever, without anyone asking it to. GitOps takes that identical loop and runs it one layer further out: instead of only reconciling what's declared in etcd against what's really running, an agent living inside the cluster also reconciles what's committed to a Git repository against what's declared in etcd. Two reconcile loops, stacked, both level-triggered, both self-healing, both indifferent to whether the last change came from a person, a crash, or another controller entirely. This page is about that stack specifically — why it fits Kubernetes so naturally, what changes about credentials and trust once the puller lives on the inside, and what actually happens when two reconcilers both decide they own the same field.
Imagine your bedroom has to always match a photo pinned to the corkboard — not tidied once when the photo went up, but kept matching all day, every day. A robot lives in the room and checks the photo every few minutes: toys back in the bin, bed remade, whatever's out of place gets fixed, no questions asked. If you clean it yourself first, great — the robot checks, sees it already matches, and does nothing. But here's the part that matters: the robot never asks anyone's permission to fix a mismatch, and it never says "I already checked an hour ago, I'll trust that." It looks again every single time. That's the whole idea — the photo (kept in Git) is the truth, and something that already lives inside the room, not a person walking in from the hallway with a mop, is what keeps reality matching it.
GitOps's four rules, in Kubernetes' own vocabulary
☺ Like you're 10: GitOps has four rules; Kubernetes' own controllers were already following three of them, long before anyone called it GitOps.
The community around OpenGitOps formalized GitOps as four principles: the system's desired state is described declaratively; that description is stored somewhere versioned and immutable, so every change has a full history and an author; approved changes are applied automatically, by software rather than a person running a command; and the live system is continuously reconciled against that description, with drift detected and, where configured, corrected. None of that is Kubernetes-specific — but lay each one next to a plain Kubernetes object and the overlap is almost exact.
| GitOps principle | Kubernetes' own analog | What GitOps specifically adds |
|---|---|---|
| Declarative | Every object's spec is already declarative — you say what you want, never the steps to get there (see the object skeleton) | Nothing — this one Kubernetes already had from day one |
| Versioned & immutable | A Deployment keeps its old ReplicaSet at zero replicas as revision history, so kubectl rollout undo has something to roll back to | A human-readable history with commit messages, diffs, and git blame — etcd's own revision history isn't meant for people to read |
| Applied automatically | Controllers already apply changes the instant they land in etcd — no person runs kubectl apply a second time for a controller to notice | Moves the point where "automatic" starts one hop further back — from "already in etcd" to "merged to main" |
| Continuously reconciled | The exact watch → diff → act loop every built-in controller already runs | Runs that same loop with a Git ref as the desired-state input instead of only an in-cluster object |
Nothing about GitOps required Kubernetes to add a new capability. A GitOps controller has no special access Kubernetes doesn't already grant any other controller — same apiserver, same RBAC, same watch mechanism. GitOps isn't a Kubernetes feature; it's an ordinary operator, in the exact sense Operators & CRDs defined one, whose one watched "resource" happens to be a Git ref instead of another API object.
Two reconcile loops, stacked
☺ Like you're 10: Git aims at the objects in the cluster the same way those objects aim at the real world — one loop's output is the next loop's desired state.
Follow one change all the way through and the stack becomes concrete. A commit lands on main. A GitOps controller — Flux's source-controller and kustomize-controller pair, or Argo CD's application-controller, covered in full tool-specific depth in DevOps's Argo CD page — notices on its next poll (or an instant webhook), clones the repo, renders the manifests, and diffs them against what's currently declared in the API. Where they differ, it applies the difference: a PATCH to kube-apiserver, landing in etcd, exactly the same request shape any other API client would send. That's Loop 1: Git versus declared state.
From there, Kubernetes doesn't know or care that the change arrived via GitOps rather than a person's kubectl apply. The built-in controllers in kube-controller-manager — ReplicaSet, Deployment, and the rest — see the new declared state and reconcile the real world to match: creating Pods, which kubelet turns into running containers, and — for a Service of type LoadBalancer — cloud-controller-manager provisioning an actual load balancer somewhere outside the cluster entirely. That's Loop 2: declared state versus the real world, the identical loop the controller pattern page already walked through in detail.
What makes this genuinely fit Kubernetes rather than just tolerate it: the GitOps controller reconciling Loop 1 is itself watched by the identical client-go reflector/informer/work-queue pipeline that Control Plane Internals covers for every built-in controller — a long-lived watch on the objects it's responsible for, not a naive poll of the whole cluster. Kill it mid-sync and restart it anywhere, and it re-reads Git's current HEAD and the API's current state from scratch, reaching the same conclusion a healthy run would have — the exact crash-safety property Operators & CRDs described for any reconciler, inherited here for free.
Pull, not push: what actually changes about trust
☺ Like you're 10: The robot living in your room needs no key to your parents' house — only its own eyes on the corkboard. A cleaner who comes from outside needs a key to get in.
A pipeline running helm upgrade --install at the end of a CI job is push-based: something outside the cluster authenticates in and writes, which means a cluster-admin-capable credential has to live somewhere in CI — a system that, on most teams, also runs untrusted code from every pull request. A GitOps controller inverts that: it already runs inside the cluster it's changing, as a Deployment with its own ServiceAccount, and the only credential it needs pointed outward is read-only access to a Git repository. No inbound cluster credential has to exist anywhere outside the cluster at all.
Flux implements Loop 1 as two ordinary CRDs plus their controllers — the same "CRD plus reconciler" shape Operators & CRDs covered generically, here applied to Git itself as the watched resource:
# source-controller reconciles this against the remote Git ref, on an interval —
# no different in shape from any other controller watching any other Kind
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: checkout-manifests
namespace: flux-system
spec:
interval: 1m
url: https://github.com/acme/checkout-manifests.git
ref:
branch: main
---
# kustomize-controller reconciles this: build the given path, apply it,
# and — if prune is on — delete anything the API has that Git no longer does
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: checkout
namespace: flux-system
spec:
interval: 5m
path: "./overlays/prod"
prune: true
sourceRef:
kind: GitRepository
name: checkout-manifests
targetNamespace: checkoutNotice what's absent: nothing here grants an outside system write access to the cluster. GitRepository only needs read access to the remote — a deploy key scoped to one repo, not a kubeconfig. That single inversion is why Kubernetes courses across this platform keep returning to GitOps whenever cluster-credential exposure comes up: it's a design that removes an entire class of exposed secret, not a policy that has to be enforced by convention. The credential the GitOps controller's own ServiceAccount holds is the one that now matters most — it typically needs broad write access across whatever it manages, which is exactly the kind of privileged, cluster-wide identity RBAC & Admission Control and DevSecOps's Kubernetes security deep dive cover how to scope down rather than leave at cluster-admin by default.
Drift: how it's found, and what "correcting" it actually means
☺ Like you're 10: A pillow out of place because a guest moved it, and a pillow out of place because the cat did — the robot can't tell the difference, and it doesn't need to.
Drift is any field on a live object that no longer matches what the GitOps controller would render from Git's current HEAD — full stop. It doesn't matter whether the cause was a person's kubectl edit, an admission webhook mutating a field on the way in, or an entirely separate controller writing to the same object for its own reasons; from the reconciler's point of view, level-triggered means it only ever asks "does live match desired, right now" and never "why doesn't it." That's the same property the ReplicaSet controller already has for Pod counts, applied here to arbitrary fields on arbitrary objects.
# Someone "fixes" a bug live, straight against a GitOps-managed Deployment
$ kubectl scale deployment/checkout -n checkout --replicas=8
# Within one reconcile interval, self-heal is on, and the controller reverts it —
# no error, no alert unless you're watching the sync/reconciliation events yourself
$ kubectl get deployment checkout -n checkout -w
NAME READY UP-TO-DATE AVAILABLE AGE
checkout 8/8 8 8 2h
checkout 3/8 3 8 2h13m # reconciler patched replicas back to 3
checkout 3/3 3 3 2h13mWhat "correcting" drift means is a policy choice, not a fixed behavior. Detection-only mode reports OutOfSync (Argo CD's term) or a failing Kustomization reconciliation (Flux's) without touching the object — useful the moment you want a human to review before anything reverts. Self-heal mode, as above, reverts automatically on the next reconcile. Neither is universally correct: a hotfix that genuinely needed to ship faster than a PR review belongs in Git within minutes regardless, precisely because self-heal will otherwise erase it silently and without explanation the next time the loop runs.
A GitOps controller's diff only ever compares two renderings of the same object — desired-from-Git and live-from-API. It has no concept of "who changed this" or "why," and no memory of the object's history beyond what it needs to compute the current diff. That's precisely why it can't distinguish a well-intentioned hotfix from an attacker's tampering — both look identical: a live field that doesn't match Git. Anything that needs to tell those apart is a job for audit logging and admission control, not for the reconciler.
When two reconcilers want the same field
☺ Like you're 10: Two people trying to set the same thermostat to two different numbers, each one convinced they're the only one allowed to touch the dial.
Kubernetes lets more than one controller legitimately write to the same object by tracking ownership per field, not per object — the mechanism is called Server-Side Apply, and the record it keeps is metadata.managedFields. Every write identifies itself with a field manager name, and the apiserver remembers exactly which manager last set which field, so a GitOps controller can own a Deployment's image and env vars while the HorizontalPodAutoscaler separately owns spec.replicas — right up until both managers believe they own the identical field.
$ kubectl get deployment checkout -n checkout -o json | jq '.metadata.managedFields[].manager'
"argocd-controller"
"horizontal-pod-autoscaler"
$ kubectl get deployment checkout -n checkout -o json \
| jq '.metadata.managedFields[] | select(.manager=="horizontal-pod-autoscaler") | .fieldsV1'
{
"f:spec": { "f:replicas": {} }
}If the manifest rendered from Git still specifies spec.replicas: 3 while an HPA also targets that Deployment, both tools are correct by their own lights and the outcome is a livelock, not a one-time mistake. Every reconcile, the GitOps controller sees live replicas (say, 7 — the HPA's own last write) differ from Git's declared 3 and, with self-heal on, patches it back to 3. Minutes later the HPA's own loop sees CPU utilization still above target and scales back up. Neither tool logs an error, because from each one's own perspective, nothing went wrong — it simply corrected a difference it's entitled to correct. This is not a GitOps-specific bug; it's the general "two writers, one field" problem every Kubernetes object already has, made permanent instead of a single overwrite because both writers now run forever. The fix is to give the field exactly one owner: omit replicas from the Git-tracked manifest entirely, or use the tool's own escape hatch — Argo CD's ignoreDifferences (see the Argo CD page for its exact syntax) or Flux's equivalent field exclusion.
"People ask if it bothers me, sharing an object with another reconciler. It doesn't — I only ever look at the fields I'm told are mine, and managedFields tells me exactly which ones that is. What does bother me is being handed a manifest that claims a field somebody else is already reconciling. I'm not being stubborn when I patch it back for the third time in ten minutes; I genuinely cannot tell that I'm in a fight, only that the live value doesn't match what I was told to want. Take the field out of my manifest, and the fight ends instantly — not because I got smarter, but because there's nothing left for me to disagree about."
On a kind cluster, install Flux or Argo CD, point it at a small repo with one Deployment and syncPolicy.automated.selfHeal (or Flux's default reconciliation) turned on. First, run kubectl scale against the managed Deployment by hand and watch kubectl get deploy -w until the reconciler reverts it — time how long that takes against the tool's configured interval. Then add an HPA targeting the same Deployment while replicas is still present in your Git-tracked manifest, and watch kubectl get deploy -w flap between two values every reconcile. Fix it by deleting replicas from the manifest and committing — that one line is the entire remedy.
Everything past this point is tool-specific mechanics rather than the pattern itself: Argo CD's Application CRD, sync waves and hooks, app-of-apps and ApplicationSets live in full depth on DevOps's Argo CD page, and the CNCF's own project landscape page in this course places both Argo and Flux among the wider ecosystem. The Kubestronaut ladder this course follows stops at CKA/CKAD/CKS; the GitOps-specific credential, CGOA — the Certified GitOps Associate, along with Argo's and Flux's own project certifications, lives one rung over on Platform Engineering's CGOA page and the sibling Golden Astronaut course, which covers the rest of the Golden Kubestronaut ladder end to end. And for what happens when the reconciler itself becomes a single point of failure for everything it manages, SRE's Kubernetes reliability patterns picks up exactly that thread.
Nutty the Squirrel: I catalogued the field managers on every Deployment in checkout. Two of them have both argocd-controller and horizontal-pod-autoscaler listed as owning spec.replicas at once.
Recon the Robot: That's not a typo, that's the flap you'd see in the events if anyone were watching. I keep correcting to 3. The HPA keeps correcting to whatever CPU wants. We are both right and the Deployment never once stabilizes.
Gizmo the Gremlin: Easy — turn off self-heal everywhere. Nothing fights if nothing corrects anything! 🤷
Timmy the Turtle: That's not a fix, that's giving up the entire reason we run a reconciler. You lose drift correction on every field to dodge a conflict on one.
Benny the Beaver: I'll just delete replicas out of the manifest and commit. The HPA's already the one deciding that number for real — Git shouldn't have been claiming it in the first place.
Professor Owl: Which is the actual lesson under all of it: two reconcile loops stacked cleanly only when every field has exactly one owner. Where that's true, nobody notices the machinery. Where it isn't, everybody does.
1. Name the two reconcile loops this page describes, and say precisely what each one compares. 2. Why does a pull-based GitOps controller remove the need for a cluster-write credential inside CI — and what credential does the cluster still reach outward for? 3. What counts as "drift," and why can't a reconciler tell a well-intentioned hotfix apart from an attacker's tampering? 4. What is a field manager, and what does Server-Side Apply's per-field ownership tracking let two different controllers do to the same object? 5. Concretely, what goes wrong when both a GitOps manifest and an HPA declare spec.replicas with self-heal on — and why doesn't either tool log an error? 6. Where do Argo CD/Flux's own tool-specific mechanics and the CGOA exam live, if not on this page?
Check your answers
- Loop 1 compares Git's rendered manifests against the API's declared objects (run by the GitOps controller). Loop 2 compares the API's declared objects against the real world — actual Pods, containers, cloud resources (run by Kubernetes' built-in controllers and kubelet).
- Because the agent applying changes already runs inside the cluster and only needs read access to Git — no external system needs a credential capable of writing to the cluster. The cluster still needs an outward credential for read access to the Git repository (e.g. a scoped deploy key).
- Drift is any live field that no longer matches what the GitOps controller would render from Git's current HEAD, regardless of cause. The reconciler is level-triggered — it only ever compares current desired vs. current live state, with no memory of who changed what or why, so a hotfix and tampering look identical to it.
- A field manager is the name a client identifies itself with when it writes to an object; Server-Side Apply records, per field, which manager last set it. That per-field tracking lets two controllers (e.g. a GitOps controller and an HPA) legitimately co-own different fields of the same object without one silently overwriting the other's work.
- Both tools keep correcting the same field back to what each believes is correct — the GitOps controller reverts replicas to Git's declared value, then the HPA scales it back up based on live CPU, forever. Neither logs an error because each write is a legitimate, intentional correction from that tool's own point of view — nothing is malfunctioning, they simply disagree about who owns the field.
- Argo CD's and Flux's own CRDs, sync policies, and CLI mechanics live on DevOps's Argo CD page; CGOA and the rest of the CNCF project-associate ladder live on Platform Engineering's CGOA page and the sibling Golden Astronaut course.