Flux — Helm Releases

This is the half of Flux that runs real Helm releases. Two controllers do the work: source-controller finds and packages a chart, and helm-controller drives the Helm SDK to install, upgrade, test, roll back or uninstall it — all from one HelmRelease object under version control. It solves the platform problem that a pipeline running helm upgrade --install can never solve: nobody holds cluster credentials, nothing drifts silently between pipeline runs, a failed upgrade repairs itself without a human, and “which chart version, with which values, is actually released in prod?” has one authoritative answer that lives in Git.

What declarative Helm is and the problem it solves

☺ Like you’re 10: Instead of a person typing helm upgrade, you write down what you want and a robot inside the cluster keeps the release matching it.

Almost every Kubernetes platform ends up consuming charts. Ingress controllers, cert-manager, Prometheus and the kube-prometheus-stack, External Secrets, databases, your own applications — all shipped as charts, because a chart is how the ecosystem packages “a component plus its knobs”. The question is not whether you use Helm; it is who runs it, and what happens when a run goes wrong.

The three problems with running helm from a pipeline

The default answer — a CI job with a kubeconfig running helm upgrade --install — creates three durable headaches, and they are the same three that GitOps exists to fix, sharpened by Helm’s particular quirks.

Credentials. Helm is a client-side tool with no server-side component to delegate to, so that job needs a long-lived, usually cluster-admin-ish kubeconfig reachable from wherever CI runs — which makes CI the most valuable target in the estate.

Values amnesia. Helm does not remember the flags you did not pass. If Tuesday’s deploy ran --set replicaCount=4 and Wednesday’s did not, Wednesday silently resets it to the chart default. The real values in force are the union of a values file in one repo, some --set flags in a pipeline definition in another, and whatever the person who ran the emergency deploy typed. No single artefact answers “what values are in force?”.

Nothing between runs. Nothing corrects drift between deploys: a Deployment hand-edited at 2am stays edited until the next deploy silently reverts it, at a moment nobody chose. And an upgrade that fails halfway leaves the release in a pending-upgrade state that blocks every future run until a human notices.

What helm-controller changes

A HelmRelease is a Kubernetes object that says: this chart, from this source, in this version range, with these values, released here, and here is what to do if it goes wrong. The controller runs inside the cluster with its own ServiceAccount, so no external system holds credentials. The values are one merged, readable, reviewable structure that lives in Git. And the failure behaviour — retry, roll back, uninstall, or freeze and shout — is a field you set deliberately rather than an incident you improvise.

What it is not

helm-controller does not author charts — that is your job, and Helm’s page covers it. It does not build or push images; CI still does that, with Tekton or a hosted runner. It does not do progressive delivery: a HelmRelease upgrade is an all-at-once rollout of whatever the chart renders, and canaries need Flagger or Argo Rollouts layered on the workload afterwards. It does not decrypt SOPS — kustomize-controller does that, which is a distinction with real consequences covered later. And it is not a substitute for Kustomization: the two reconcilers coexist, and choosing between them for a given component is a design decision, not a preference.

Vocabulary: the Helm Operator is gone

Flux v1 shipped Helm support as a separate project called the Helm Operator, whose HelmRelease lived in the helm.fluxcd.io API group. It is end-of-life. Today’s HelmRelease belongs to helm.toolkit.fluxcd.io and is reconciled by helm-controller, one of the GitOps Toolkit controllers described on the Flux page. If a blog post tells you to install “the Helm Operator”, or shows an apiVersion without toolkit in it, it is describing something that no longer exists. Within Flux v2 itself the HelmRelease API has moved through v2beta1 and v2beta2 to v2, and a handful of fields moved during that journey — always check what the cluster in front of you actually serves.

Where HelmRelease fits in a platform

☺ Like you’re 10: It sits in the “install the thing” part of the platform — after the box is built, before anyone watches how it behaves.

On the plane diagram in Platform Architecture, HelmRelease lives in the delivery control plane alongside Kustomization. Upstream of it sit chart authors — the ecosystem’s, or your own platform team’s charts published to an OCI registry. Downstream sit the things that observe and shape what it deployed.

Its neighbours on the paved road

Helm is the packaging format and the semantics; helm-controller is the thing that runs those semantics on a schedule. Kustomize appears twice, in two different roles that are easy to confuse: as the engine behind Flux’s Kustomization resource, and as the post-renderer that can patch a chart’s rendered output — the second is covered in depth on this page. External Secrets and SOPS supply the values you cannot commit in plaintext. Sigstore/Cosign verifies chart artefacts pulled from an OCI registry before they are ever unpacked. Flagger takes the Deployment your chart just created and rolls the next version of it out progressively. Prometheus scrapes helm-controller’s own metrics so you can alert on releases stuck not-Ready. And Flux’s image automation can rewrite an image tag inside a HelmRelease’s values block in Git, which is how a new build reaches a chart-deployed app without a human editing YAML.

Kustomization or HelmRelease? — the decision you will actually make

For every add-on you install, you choose one of two reconcilers. The honest split is about where the configuration surface comes from. If upstream ships a chart with a well-designed values.yaml that already exposes what you need to change, a HelmRelease is less code and gives you rollback for free. If you need to change something the chart does not expose, you are choosing between fighting the chart with post-renderers and simply rendering it once and managing plain manifests with a Kustomization.

A common and defensible pattern is to use HelmRelease for third-party components and Kustomization for your own applications, whose manifests you already control. Another is the rendered manifests pattern: CI runs helm template, commits the plain YAML to an environment branch, and a Kustomization applies it — trading a build step for pull-request diffs that a human can actually read. That trade-off is discussed further on the Helm page and in Configuration Management.

CNPE domain relevance

Flux is on the official CNPE tool list, and Domain 2 — GitOps & Continuous Delivery carries 25% of the exam blueprint, tied for the heaviest weighting. Chart-driven installs are also how most of the tools in the other domains get onto a cluster in the first place, so a scenario that says “install the monitoring stack and make it survive a failed upgrade” is a Helm question wearing an observability costume. Expect to be asked to write a working HelmRelease, to make one release wait for another, to explain why a release is stuck, or to point a release at a chart in an OCI registry.

How it works — helm-controller and source-controller together

☺ Like you’re 10: One robot fetches and checks the box; the other robot reads your sheet, builds the thing, and writes down what it built.

The most common misconception about Flux’s Helm support is that HelmRelease talks to a chart repository. It does not. helm-controller never fetches a chart. It asks source-controller for one, by creating a HelmChart object on your behalf, and waits for an artifact to appear. Understanding that indirection explains most of the confusing objects you will find in a real cluster.

The four objects and who owns them

GitRepository and Bucket can also act as chart sources — in that case source-controller packages the chart directory itself rather than downloading a published .tgz. Their own shapes are on the Flux page.

What one reconciliation actually does

Follow one change end to end, because every troubleshooting question is really “which of these seven steps stopped?”

1. Chart resolution. helm-controller reads spec.chart and creates or updates the generated HelmChart. source-controller reconciles it: for an HTTP HelmRepository it consults the cached index.yaml, picks the highest version satisfying the range, downloads that .tgz and verifies its digest; for an OCI source it pulls the artifact; for a Git or Bucket source it packages the chart directory. The result is an artifact with a revision, and for charts the revision is the chart version.

2. Values composition. helm-controller reads every entry in valuesFrom in list order, merging each into an accumulating map, then merges the inline spec.values on top. Missing references are a hard failure unless marked optional.

3. The change decision. This is the step people miss. The controller hashes the chart revision, the composed values, the post-renderers and the relevant spec fields into a digest and compares it with what it recorded for the last release. If nothing changed, it does nothing — no helm upgrade, no re-apply, no correction of hand edits. Unlike kustomize-controller, which re-applies on every interval, helm-controller is change-triggered. The interval is how often it looks, not how often it acts.

4. The operation. No release in storage means install; a changed digest means upgrade; a release found in a failed or pending state means remediate first. The chosen operation runs through the Helm SDK against the Kubernetes API, using helm-controller’s own ServiceAccount or the one named in spec.serviceAccountName.

5. Waiting and testing. By default the controller waits for the applied objects to become ready, bounded by timeout (five minutes unless you say otherwise); disableWait turns that off and disableWaitForJobs narrows it. Then, if spec.test.enable is true, the chart’s helm.sh/hook: test pods run — and a failing test counts as a failed release unless ignoreFailures is set, which means a test failure can trigger a rollback. That is usually what you want.

6. Status and remediation. Success writes Ready=True and a new entry in the release history. Failure runs the remediation configured for that operation, then updates Ready, Released, TestSuccess, Remediated and, when retries run out, Stalled. Kubernetes events are emitted throughout, which is what notification-controller forwards to Slack.

Where the release actually lives

Helm has always stored release state in the cluster, in Secrets named sh.helm.release.v1.<release>.v<revision>, base64-encoded and gzipped. helm-controller uses the same storage, which is what makes helm history work. Two fields decide where things land and they default differently, which trips up nearly everyone once:

• spec.targetNamespace — where the chart’s objects go. Defaults to the HelmRelease’s own namespace.
• spec.storageNamespace — where the Helm history Secrets go. Also defaults to the HelmRelease’s own namespace, not to targetNamespace.

So a HelmRelease in flux-system with targetNamespace: monitoring deploys into monitoring but keeps its history in flux-system. helm list -n monitoring then shows nothing at all, and people conclude Flux “isn’t really using Helm”. Use helm list -A, or set storageNamespace to match targetNamespace so the release is where a human would look for it.

The resources you will actually write

☺ Like you’re 10: One object that says where the box comes from, one object that says how to build it. Everything else is detail.

The baseline: a chart repository and a release

This is ninety percent of real usage. Read it twice — every later example is this shape with more fields.

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 1h                     # how often source-controller re-downloads index.yaml
  url: https://stefanprodan.github.io/podinfo
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 10m                    # how often helm-controller re-evaluates this release
  releaseName: podinfo             # explicit — see the naming gotcha below
  targetNamespace: web             # where the chart's OBJECTS land
  storageNamespace: web            # where the Helm HISTORY lands (defaults to the HelmRelease’s own namespace)
  chart:
    spec:
      chart: podinfo               # the chart NAME in the repository index
      version: "6.x"               # a semver range, resolved by source-controller
      sourceRef:
        kind: HelmRepository
        name: podinfo
        namespace: flux-system
      interval: 1h                 # how often to look for a newer version matching "6.x"
  install:
    createNamespace: true
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
  values:
    replicaCount: 2
    ingress:
      enabled: true
      className: nginx

Three intervals appear in that manifest and they do genuinely different things. HelmRepository.spec.interval is how often the repository’s index.yaml is re-downloaded — if this is an hour, a chart published five minutes ago is invisible for up to an hour no matter what else you set. chart.spec.interval is how often the version range is re-evaluated against that cached index; leave it out and it inherits the HelmRelease’s interval. HelmRelease.spec.interval is how often the controller re-evaluates the release as a whole. The worst-case delay for a new chart version to reach the cluster is the sum of the first two.

OCI charts, two ways

Publishing charts as OCI artifacts to the same registry as your images has become the default for platform teams: one credential, one retention policy, one place to sign things. Flux supports it through two different shapes, and knowing why both exist is worth an exam point.

# Option A — treat the registry namespace as a chart repository.
# Use this when one registry path holds many charts.
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: acme-charts
  namespace: flux-system
spec:
  type: oci                        # the field that changes everything
  url: oci://ghcr.io/acme/charts
  interval: 10m
  secretRef:
    name: ghcr-auth                # a kubernetes.io/dockerconfigjson Secret
  # provider: aws | azure | gcp    # ...or use the cloud's workload identity instead
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: checkout
  namespace: flux-system
spec:
  interval: 10m
  chart:
    spec:
      chart: checkout              # the repository (image name) under that path
      version: ">=1.4.0 <2.0.0"
      sourceRef:
        kind: HelmRepository
        name: acme-charts

With type: oci the HelmRepository behaves differently from the HTTP kind: there is no index.yaml to download, so the object is essentially a URL plus credentials rather than something that produces an artifact of its own. Version resolution happens by listing tags in the registry when the HelmChart reconciles.

# Option B — the chart as a first-class, individually verifiable OCI artifact.
# Use this when you want signature verification, digest pinning, or a chart
# that other objects also reference.
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: checkout-chart
  namespace: flux-system
spec:
  interval: 10m
  url: oci://ghcr.io/acme/charts/checkout
  ref:
    semver: ">=1.4.0 <2.0.0"       # or: tag: 1.4.7   or: digest: sha256:...
  secretRef:
    name: ghcr-auth
  verify:
    provider: cosign               # refuse to unpack an unsigned chart
    secretRef:
      name: cosign-public-key
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: checkout
  namespace: flux-system
spec:
  interval: 10m
  chartRef:                        # mutually exclusive with spec.chart
    kind: OCIRepository
    name: checkout-chart
  values:
    replicaCount: 3

chartRef is the newer of the two shapes and is not present in the oldest HelmRelease API versions; it accepts an OCIRepository or a HelmChart you manage yourself. Its real advantage is that supply-chain verification becomes a property of the source, so a chart that fails Cosign verification never becomes an artifact and therefore never reaches a release. Setting both chart and chartRef is a validation error, not a merge.

A chart that lives in a Git repository

During development, or for charts you have not bothered to publish, a GitRepository can be the source. There is one field here that causes more confusion than any other in the whole API.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: platform-charts
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/acme/platform-charts.git
  ref:
    branch: main
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: internal-api
  namespace: flux-system
spec:
  interval: 10m
  chart:
    spec:
      chart: ./charts/internal-api      # a PATH in the repo, not a chart name
      sourceRef:
        kind: GitRepository
        name: platform-charts
      reconcileStrategy: Revision       # new commit => new chart artifact => upgrade
      valuesFiles:                      # paths relative to the REPOSITORY root
        - ./charts/internal-api/values.yaml
        - ./charts/internal-api/values-prod.yaml

Also note that for a Git source, valuesFiles paths are relative to the repository root, whereas for a chart pulled from a repository they must be paths inside the chart. And valuesFiles replaces the default behaviour rather than adding to it: if you list files, the chart’s own values.yaml is only used if you list it. When valuesFiles is set, source-controller repackages the chart with those values baked in, which is why the generated HelmChart becomes specific to that one release.

Values composition — the part you will get wrong first

Values can come from four places, and the merge order is not negotiable. Every entry in valuesFrom is merged in list order, each overriding the last; then spec.values is merged over all of them. Inline always wins.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: checkout
  namespace: apps
spec:
  interval: 10m
  chart:
    spec:
      chart: checkout
      version: "1.4.x"
      sourceRef:
        kind: HelmRepository
        name: acme-charts
        namespace: flux-system         # cross-namespace source reference
  valuesFrom:
    # merged first, in list order — later entries beat earlier ones
    - kind: ConfigMap
      name: cluster-defaults           # whole document at key values.yaml
    - kind: ConfigMap
      name: region-overrides
      valuesKey: values-eu-west-1.yaml # a different key, same YAML-document shape
    - kind: Secret
      name: checkout-db
      valuesKey: password              # a single scalar value...
      targetPath: postgresql.auth.password   # ...injected at this dot-path
    - kind: ConfigMap
      name: experiment-tuning
      optional: true                   # missing => skipped, not a failure
  values:
    # inline values are merged LAST, so these beat everything above
    replicaCount: 3
    ingress:
      enabled: true
      host: checkout.acme.example

Two habits keep this maintainable: put anything a human might tune during an incident inline where the PR diff shows it, and resist layering five ConfigMaps — composition is a feature right up until nobody can answer “where does resources.limits.memory come from?”, at which point helm get values checkout -n apps is the only honest answer.

Remediation — what actually happens when a release fails

This is the block that earns HelmRelease its keep, and the one whose semantics exam questions like to probe. Each operation has its own settings, and install and upgrade remediate differently because there is nothing to roll back to on a first install.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: checkout
  namespace: apps
spec:
  interval: 10m
  timeout: 8m                      # ceiling for any single Helm operation; inherited below
  maxHistory: 10                   # Helm revisions retained (controller default is 5)
  chart:
    spec:
      chart: checkout
      version: "1.4.x"
      sourceRef:
        kind: HelmRepository
        name: acme-charts
        namespace: flux-system
  install:
    createNamespace: true
    crds: Create                   # install CRDs from the chart's crds/ dir if absent
    remediation:
      retries: 2                   # retry the failed install twice; -1 means forever
      remediateLastFailure: false  # leave the wreckage for a human to look at
  upgrade:
    crds: CreateReplace            # DO update chart CRDs on upgrade (default is Skip)
    cleanupOnFail: true            # delete resources the failed upgrade created
    remediation:
      retries: 2
      strategy: rollback           # or: uninstall
      remediateLastFailure: true   # roll back after the final retry fails
      ignoreTestFailures: false    # a failed chart test counts as a failed upgrade
  rollback:
    timeout: 5m
    cleanupOnFail: true
    recreate: false                # true = delete and recreate pods during rollback
  uninstall:
    keepHistory: false
    deletionPropagation: background
  test:
    enable: true                   # run the chart's helm.sh/hook: test pods
    ignoreFailures: false

Once retries are exhausted and remediation has run, the controller marks the release Stalled and stops trying. It will not attempt another upgrade until something meaningful changes — the chart version, the composed values, the HelmRelease generation — or until you explicitly reset it. That is deliberate: a controller retrying a broken upgrade every ten minutes forever is how you turn one bad release into a cluster-wide outage. It is also the single most common reason someone says “Flux stopped working”, and the fix is in the commands section below.

Drift detection and correction

Because helm-controller only acts when the desired state changes, a hand-edited Deployment stays hand-edited indefinitely — until the next real upgrade quietly wipes it out. Drift detection closes that gap by comparing the objects recorded in the Helm release against what is live.

spec:
  driftDetection:
    mode: enabled                  # disabled (default) | warn | enabled
    ignore:
      # the HorizontalPodAutoscaler owns this field, not the chart
      - paths: ["/spec/replicas"]
        target:
          kind: Deployment
      # the vertical autoscaler rewrites this on one specific workload
      - paths: ["/spec/template/spec/containers/0/resources"]
        target:
          kind: Deployment
          name: checkout

warn detects and reports drift as an event and a log line without changing anything — the right first step when adopting this on an existing cluster, because it tells you how much drift you have before it starts correcting it. enabled detects and corrects, by triggering an upgrade of the release so the recorded manifests are re-applied.

The ignore list takes JSON pointers (note /spec/replicas, not spec.replicas) with an optional Kustomize-style target selector. You will need it, because plenty of legitimate actors mutate objects after Helm creates them: autoscalers own replica counts, mutating admission webhooks inject sidecars and volumes, service meshes rewrite pod specs, and Flagger deliberately scales the chart’s Deployment down while it manages a -primary copy. Turning drift correction on without an ignore list in front of any of those produces a controller and a webhook taking turns undoing each other, forever, at whatever your interval is.

Ordering releases with dependsOn

Charts have prerequisites: a database before the app, cert-manager before anything requesting a Certificate, the CRDs of an operator before the custom resources that use them.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: checkout
  namespace: apps
spec:
  interval: 10m
  dependsOn:
    - name: redis                  # same namespace as this HelmRelease
    - name: cert-manager
      namespace: cert-manager      # cross-namespace dependency
  chart:
    spec:
      chart: checkout
      version: "1.4.x"
      sourceRef:
        kind: HelmRepository
        name: acme-charts
        namespace: flux-system

dependsOn means “do not install or upgrade me until every named HelmRelease reports Ready.” Three things about it are worth committing to memory. It only references other HelmRelease objects — you cannot depend on a Kustomization from here, and if you need that ordering you express it between the Kustomizations that contain them, as described on the Flux page. Because helm-controller waits for applied objects by default, a dependency reporting Ready genuinely does mean its workloads came up — which is a stronger guarantee than Kustomization.dependsOn gives you when wait is off. And a dependency cycle simply stalls: both releases sit waiting for each other with no error that names the loop, so if two releases are inexplicably not progressing, draw the graph. Recent controller versions also let a dependency define a custom readiness expression rather than relying on the plain Ready condition; check kubectl explain helmrelease.spec.dependsOn for what your cluster supports.

Post-renderers — patching a chart you do not control

Every platform team eventually meets a chart that will not expose the one field they need: a priorityClassName, a topology spread constraint, an annotation your policy engine requires. The options are to fork the chart, to abandon it for plain manifests, or to patch its output. Post-renderers are the third option: Kustomize applied to the rendered manifests after Helm templating and before apply.

spec:
  postRenderers:
    - kustomize:
        patches:
          # strategic-merge patch: metadata.name must match the RENDERED object
          - target:
              kind: Deployment
              name: checkout
            patch: |
              apiVersion: apps/v1
              kind: Deployment
              metadata:
                name: checkout
                annotations:
                  acme.example/owner: payments-team
              spec:
                template:
                  spec:
                    priorityClassName: platform-critical
                    topologySpreadConstraints:
                      - maxSkew: 1
                        topologyKey: topology.kubernetes.io/zone
                        whenUnsatisfiable: ScheduleAnyway
                        labelSelector:
                          matchLabels:
                            app.kubernetes.io/name: checkout
          # JSON 6902 patch against the same object
          - target:
              kind: Deployment
              labelSelector: "app.kubernetes.io/name=checkout"
            patch: |
              - op: replace
                path: /spec/template/spec/containers/0/imagePullPolicy
                value: IfNotPresent
        images:
          - name: ghcr.io/acme/checkout
            newTag: 1.4.7-hotfix

The mechanics are ordinary Kustomize, with the usual sharp edges: a strategic-merge patch must name an object that actually exists in the rendered output, list-index paths like containers/0 break the moment the chart adds an init container ahead of yours, and ~1 escapes a literal / inside a JSON-pointer key. Because the patched result is what goes into the Helm release, helm get manifest shows the patched version and drift detection compares against the patched version — which is exactly what you want.

Running a release as somebody else

By default helm-controller acts with its own, fairly powerful ServiceAccount. In a shared cluster that is exactly the privilege escalation you were trying to avoid: a tenant who can write a HelmRelease can deploy a chart that creates a ClusterRoleBinding. Two fields fix it.

spec:
  serviceAccountName: team-payments   # impersonate this SA for all release operations
  targetNamespace: payments
  storageNamespace: payments
  kubeConfig:                         # optional: release onto a DIFFERENT cluster
    secretRef:
      name: prod-eu-kubeconfig
      key: value

serviceAccountName makes every Kubernetes call the release performs run as that ServiceAccount, so the tenant’s release can only do what the tenant’s RBAC allows. kubeConfig points the release at a remote cluster entirely, which is one way to run a hub-and-spoke fleet. Both belong to a wider design — cross-namespace reference rules, tenant isolation, and how you stop one team’s HelmRelease from reading another team’s Secrets — which is Flux — Multi-Tenancy’s subject, not this page’s.

Day-to-day commands

☺ Like you’re 10: The flux command asks and nudges; the helm command is for looking, never for touching.

Two CLIs are in play and the division of labour is strict. flux reads and nudges the CRDs. helm is a read-only forensic tool against the release storage — running helm upgrade behind the controller’s back creates a revision the controller did not author, and the next reconcile will overwrite it while producing a very confusing history.

Look first

# The one command to start every investigation:
flux get all -A                          # sources, kustomizations, helmreleases

flux get helmreleases -A                 # Ready / Suspended / last applied revision
flux get sources helm -A                 # HelmRepository objects
flux get sources chart -A                # the generated HelmChart objects
flux get sources oci -A                  # OCIRepository objects

# Narrow to what is broken:
flux get helmreleases -A --status-selector ready=false

# Everything Kubernetes knows about one release:
kubectl describe helmrelease checkout -n apps
kubectl get helmrelease checkout -n apps -o yaml | less   # read .status carefully

The .status block is where the answer usually is. lastAttemptedRevision versus the last successful one tells you whether the failure is new. The condition list distinguishes could not get a chart (a source problem) from upgrade failed (a chart or values problem) from Stalled (retries exhausted, and it is now waiting for you). Newer controller versions also record a history array summarising recent releases, which saves a trip to helm history.

Then nudge

# Stop waiting for the interval. --with-source re-fetches the chart first,
# which is what you want when a new chart version was just published.
flux reconcile helmrelease checkout -n apps --with-source

# Refresh the repository index without touching the release:
flux reconcile source helm acme-charts -n flux-system

# Pause and resume — the safe way to hand-edit during an incident:
flux suspend helmrelease checkout -n apps
flux resume  helmrelease checkout -n apps

# Force an upgrade even though nothing changed (re-run of the same release),
# and reset the failure counters on a stalled release. Recent CLI versions
# expose these as flags; older ones need the equivalent annotations, so check:
flux reconcile helmrelease --help
flux reconcile helmrelease checkout -n apps --force
flux reconcile helmrelease checkout -n apps --reset

Understand what those last two do differently, because reaching for the wrong one wastes an outage. --force makes the controller run an upgrade even though the digest is unchanged — the fix when the release is fine on paper but the cluster objects are wrong. --reset clears the accumulated failure counts on a release that has stalled after exhausting its retries — the fix when the controller has given up and you have since fixed the underlying problem. Both are implemented as annotations on the object (a force at and a reset at timestamp), so if your CLI predates the flags you can still do it with kubectl annotate; check the annotation names with kubectl explain or the controller’s logs rather than guessing.

Forensics with the helm CLI

# Where IS the release? Remember storageNamespace defaults to the
# HelmRelease's own namespace, not targetNamespace — so search everywhere:
helm list -A

# The revision history, including which revision is deployed and which failed:
helm history checkout -n apps

# The values actually in force — the merged result, not what you wrote:
helm get values checkout -n apps            # user-supplied values only
helm get values checkout -n apps --all      # merged with the chart defaults

# The manifests the release owns, post-render included:
helm get manifest checkout -n apps

# The raw storage, when you need to prove a release exists:
kubectl get secret -n apps -l owner=helm

Generate manifests instead of typing them

# --export prints YAML instead of applying it, so you can commit it.
flux create source helm acme-charts \
  --url=https://charts.acme.example \
  --interval=1h --export > acme-charts-source.yaml

flux create helmrelease checkout \
  --namespace=apps \
  --source=HelmRepository/acme-charts.flux-system \
  --chart=checkout \
  --chart-version="1.4.x" \
  --values=./values-prod.yaml \
  --target-namespace=payments \
  --interval=10m --export > checkout-release.yaml

# Trace a live object back to the release that created it:
flux trace deployment/checkout -n payments

# Events and logs for one release:
flux events --for HelmRelease/checkout -n apps
flux logs --kind=HelmRelease --name=checkout --namespace=apps --follow
flux logs --level=error -A

Two more habits worth building. flux tree helmrelease checkout -n apps lists every object the release owns, which answers “did the chart actually create the ServiceMonitor?” without reading rendered YAML. And for a dry run, render locally with the same values — helm template checkout acme/checkout --version 1.4.7 -f values-prod.yaml — then diff against helm get manifest. Recent flux versions have been growing a first-party diff for HelmReleases; check flux diff --help on the cluster in front of you rather than assuming either way.

Gotchas and failure modes

☺ Like you’re 10: Most “Flux broke my Helm release” moments are the robot waiting, the robot having given up, or the robot looking in a different cupboard than you are.

“Nothing happens” — the change-triggered surprise

Symptom: the HelmRelease is Ready=True, the interval ticks by, and a manual edit someone made to a Deployment is still there. Or you fixed something in the cluster and expected the controller to restore it. Cause: helm-controller is change-triggered. With an unchanged chart version, unchanged values and drift detection off, the correct behaviour is to do nothing. Fix: enable driftDetection.mode, or accept the model and treat hand edits as temporary by construction. For a one-off correction, flux reconcile helmrelease NAME --force. This is the single biggest conceptual difference from Kustomization, and internalising it prevents half the confusion on this list.

“Another operation (install/upgrade/rollback) is in progress”

Symptom: the release is wedged; helm history shows the newest revision as pending-upgrade or pending-install. Cause: the process performing the operation died mid-flight — helm-controller was OOM-killed, evicted, or restarted during a rollout. Helm has no lease timeout; a pending revision blocks every subsequent operation forever. Fix: recent helm-controller versions detect a release left in a pending state on startup and remediate it automatically, so the first move is to check whether it recovers on its own. If it does not, the manual repair is a helm rollback to the last deployed revision, or deleting the sh.helm.release.v1.<release>.v<N> Secret for the pending revision — do that with flux suspend on and the release’s history in front of you. Prevention: give helm-controller a generous memory limit; charts with hundreds of objects are the usual OOM trigger.

“Upgrade retries exhausted” / Stalled

Symptom: Ready=False, a Stalled condition, and the message names an exhausted retry budget. Nothing changes no matter how long you wait. Cause: by design — the controller stops after remediating the configured number of failures. Fix: fix the actual problem (usually visible in the previous failure’s message: a bad value, a missing Secret, a chart bug, a timeout on a slow workload), then either push the fix to Git — which changes the generation and un-stalls it — or clear the counters with flux reconcile helmrelease NAME --reset. Do not simply raise retries; if two attempts failed, twenty will too.

The release renamed itself and now collides

Symptom: cannot re-use a name that is still in use, or a duplicate set of objects appears alongside the originals. Cause: when releaseName is not set explicitly, the generated release name depends on targetNamespace — adding or changing targetNamespace later therefore changes the release name, and the controller believes it is installing something new next to an existing release. The same collision happens when a chart was originally installed imperatively with helm install and you now point a HelmRelease at the same name. Fix: always set releaseName explicitly from day one. For an adoption collision, either uninstall the old release cleanly first, or set install.replace: true knowingly. Names are also truncated to 53 characters, so very long namespace-plus-name combinations can collide in ways that look impossible until you count the characters.

helm list shows nothing

Symptom: the workloads are running in monitoring, but helm list -n monitoring is empty and someone concludes Flux does not use real Helm. Cause: storageNamespace defaults to the HelmRelease’s namespace, not targetNamespace, so the history Secrets are in flux-system. Fix: helm list -A, and set storageNamespace equal to targetNamespace in new releases so the release is where a human would look. Changing storageNamespace on an existing release is a migration, not an edit — the controller will not find the old history and will treat it as a fresh install.

The chart upgraded itself at 3am

Symptom: a component you did not touch is suddenly on a new major version. Cause: a loose version range. version: "*" (the default when you omit it) or version: ">=1.0.0" means “whatever is newest”, and source-controller will take it the moment the repository index refreshes. Fix: pin deliberately. A patch-only range (~1.4.0) is defensible for well-behaved charts, a minor range (1.4.x, ^1.4.0) is a judgement call, and an exact pin plus automation that opens a PR is the most reviewable option. Whatever you choose, remember the two-interval delay: a tight chart.spec.interval means nothing if the HelmRepository index only refreshes hourly.

Secrets in values

Symptom: a password ends up in a Git diff, or a security review asks where the database credential is. Cause: the obvious place to put a password is spec.values, and that is stored in plaintext in the HelmRelease object, visible to anyone with get helmrelease. Fix: use valuesFrom with a Secret and targetPath, and get that Secret from External Secrets or from a SOPS-encrypted file in the repo. And note the boundary precisely: helm-controller does not decrypt SOPS. Decryption is kustomize-controller’s job via Kustomization.spec.decryption, so the encrypted Secret must be applied by a Kustomization that the HelmRelease then references. Be honest about the residual exposure too: whatever values are in force end up inside the Helm storage Secret, base64-encoded and gzipped — obfuscated, not encrypted — so Kubernetes RBAC on Secrets, and ideally encryption at rest, is still doing the real work. Secrets Management covers the full picture.

Chart-source authentication

Symptom: 401 Unauthorized, failed to download chart, or an index that fetches but a chart that does not. Causes, in the order you should check them: the Secret is in the wrong namespace (it must be in the same namespace as the HelmRepository, not the HelmRelease); the Secret is the wrong type (OCI sources want kubernetes.io/dockerconfigjson, HTTP sources want username/password keys); TLS material is in the wrong field, since certificate configuration moved out of secretRef into a dedicated certSecretRef and the old form is deprecated; or the index and the chart tarballs live on different hosts, which is common for repositories backed by object storage. That last one is the sneaky one: credentials are not forwarded across a redirect to a different host unless spec.passCredentials: true is set, and that flag is off by default for good reason — turning it on sends your credentials to whatever host the redirect names. For cloud registries, prefer spec.provider: aws|azure|gcp and workload identity over a static Secret you have to rotate.

Three shorter ones, in the order you meet them

Alternatives and when to choose it

☺ Like you’re 10: Everyone runs the same chart. The difference is who runs it, what they remember afterwards, and what happens when it goes wrong.

There are four honest ways to get a chart onto a cluster, and they differ less in capability than in what they leave behind.

The honest comparison with Argo CD

Neither engine is better here; they made a defensible choice in opposite directions. Argo CD decided that a Helm chart is a manifest generator — it renders and then owns the resulting objects itself, which buys a uniform diff view across Helm, Kustomize and plain YAML, and one consistent sync/health model. The costs are real: no Helm release to inspect, Helm hooks reinterpreted through a different hook system, no helm rollback, and charts using lookup or random functions generating permanent meaningless drift. Flux decided a Helm chart is a release, and delegated to the Helm SDK. That buys genuine Helm semantics — hooks, tests, revisions, rollbacks, and remediation you can configure declaratively. Its costs are also real: Helm’s state model is now inside your GitOps loop, so you inherit pending-upgrade wedges, storage Secrets, and a controller that does nothing until something changes.

Two practical tie-breakers. If a chart’s hook ordering or a helm test matters to you — database migrations gated on a pre-upgrade hook, say — Flux preserves it and Argo CD reinterprets it. If your teams live in a dashboard comparing desired versus live all day, Argo CD’s rendering model makes that view uniform, while Flux gives you CLI output and whatever you build in Grafana or Backstage.

Versus a pipeline running helm

A CI job running helm upgrade --install can do everything a HelmRelease can do, once. What it cannot do is remember — which is the whole argument made at the top of this page. The migration itself is usually easy: same chart, same values file, moved into valuesFrom or inline. The part to plan for is adoption, because an existing Helm release with the same name will collide with the controller’s first install unless you handle it, as described in the naming gotcha above.

What it is not an alternative to

Do not let a HelmRelease become a place where non-Helm problems get solved. Progressive delivery belongs to Flagger or Argo Rollouts, layered on the workload the chart created — an upgrade is still all-at-once. Policy belongs in admission control with Kyverno or Gatekeeper, not in a post-renderer that only runs in one delivery path. Secret material belongs in External Secrets or SOPS. Infrastructure belongs in Crossplane or Cluster API resources, which Flux is happy to reconcile — through a Kustomization, usually, because they are plain custom resources and do not need a chart at all.

Exam relevance and going further

☺ Like you’re 10: You have to be able to write a HelmRelease from a blank file — because you cannot look one up during the test.

Flux is one of the officially named CNPE tools, and chart-driven installation is how most of the exam’s other tools would realistically arrive on a cluster. Domain 2 — GitOps & Continuous Delivery carries 25% of the blueprint. Expect practical tasks: point a release at a chart repository, pin a version range, compose values from a ConfigMap, make one release wait for another, or explain why a release is not progressing.

Your on-cluster fallbacks are genuinely good, and rehearsing them is worth more than memorising field names. kubectl api-resources | grep toolkit lists every Flux kind with its group and short name. kubectl explain helmrelease.spec --recursive prints the entire schema straight from the CRD, including every remediation field on this page. flux create helmrelease ... --export generates a valid skeleton you can edit instead of typing YAML from memory. And kubectl get helmrelease -A -o yaml on an existing release gives you a working example of the exact shape the cluster serves.

⚖ CNPA vs CNPE — the allowlist is a CNPE mechanic; CNPA is fully closed-book, with no external resources of any kind. That is stricter, not looser. For CNPA the concepts matter more than the YAML: what a declarative release is, why pull beats push, and why a controller correcting drift is different from a pipeline re-running.

What to be able to do cold

Without notes, you should be able to: write a HelmRepository and a HelmRelease that installs a named chart at a version range into a target namespace; explain the difference between targetNamespace and storageNamespace; compose values from a ConfigMap and a Secret and state the precedence order; configure install and upgrade remediation and say what each does on failure; order two releases with dependsOn; say why a release with unchanged values does not correct a hand edit and what to set if you want it to; force a reconcile with flux reconcile helmrelease --with-source; suspend and resume; and diagnose a not-Ready release from flux get, kubectl describe, flux events and helm history. Rehearse the loop in Practice: GitOps, work the decision tree in Triage: Delivery, and keep the Command Reference next to you while you drill.

Where to go next on this site

Back up one level to Flux for the toolkit as a whole, bootstrap and the Kustomization half of the story. Sideways to Flux — Image Automation for how a new image tag reaches a release’s values without a human, and to Flux — Multi-Tenancy for impersonation, cross-namespace rules and fleet layout. Down a layer to Helm for chart authoring and to Kustomize for the patching engine behind post-renderers. Across to Flagger for what happens after the release lands, and to Argo CD for the other GitOps engine. For theory rather than tooling, GitOps and CI/CD & Progressive Delivery are the lessons this page sits underneath, with Release Engineering for the wider discipline and the Tool Landscape for how Flux sits among the official fifteen.

Official sources — for study time, not exam time

Read these before the exam: the helm-controller documentation and API reference at fluxcd.io/flux/components/helm, the source-controller reference for HelmRepository, HelmChart and OCIRepository at fluxcd.io/flux/components/source, the Helm guide at fluxcd.io/flux/guides/helmreleases, the controller source at github.com/fluxcd/helm-controller, and Helm’s own docs at helm.sh/docs for what the SDK underneath is actually doing. The vendor-neutral principles behind all of it are at opengitops.dev.