Litmus

LitmusChaos is the CNCF chaos-engineering project built specifically for Kubernetes rather than adapted to it: instead of a script that SSHes into a box and kills a process, a chaos experiment in Litmus is a Kubernetes object — a custom resource you kubectl apply, review in a pull request, and let a controller reconcile exactly like a Deployment. It ships with a hub of pre-built fault templates covering the failure modes chaos engineering already taught you to reason about in the abstract — pod kills, network loss, CPU and memory hogs, disk fills, node drains — and because every experiment definition is plain YAML, it slots directly into a GitOps pipeline as a scheduled, version-controlled stage rather than a separate tool a human has to remember to run. This page covers Litmus's architecture, its three core custom resources, the ChaosHub, the commands you'll actually type, and how a team wires it into GitOps so chaos runs itself, safely, on a schedule.

What Litmus is and the problem it solves

☺ Like you're 10: It's chaos engineering rewritten so the "experiment" is a Kubernetes object, not a script somebody has to remember to run by hand.

Litmus began at MayaData (the company behind the storage project OpenEBS) and was donated to the Cloud Native Computing Foundation, entering the CNCF Sandbox in 2020 and later advancing to Incubating status — check the project's current maturity tier on the CNCF project page before citing it, since sandbox-to-incubating-to-graduated moves happen on the foundation's own timeline, not this page's. What made Litmus distinct from the chaos tooling that came before it — Netflix's own Chaos Monkey among them — is that it doesn't bolt fault injection onto Kubernetes from the outside. It is a set of Kubernetes controllers and custom resource definitions (CRDs): a chaos experiment is declared the same way a Deployment or a NetworkPolicy is declared, reconciled by an operator watching the API server, and left as an auditable object in kubectl get output long after the experiment finishes.

That framing solves a specific problem the rest of this course's chaos-engineering material leaves open: chaos engineering tells you an experiment needs a hypothesis, a scope, and an expected steady-state — but says nothing about what actually types the fault into the cluster, records whether the hypothesis held, or makes sure the same experiment runs again next month without a human remembering to. Litmus is one concrete answer to all three: the hypothesis and scope live in a ChaosEngine, the fault logic lives in a ChaosExperiment pulled from a shared library, and the verdict lands in a ChaosResult — three Kubernetes objects that together make an experiment something you can commit, review, and schedule instead of something one engineer runs from memory.

Architecture: the operator, the runner pod, and ChaosCenter

☺ Like you're 10: A controller watches for an experiment request, spins up a short-lived worker to carry it out, and that worker's job pod does the actual breaking — then everyone cleans up and writes down what happened.

Litmus's execution model has two layers, and they can be run independently. The core layer is kubectl-and-CRD only: a chaos-operator deployment watches for ChaosEngine objects, and needs nothing more than the CRDs and RBAC to function — a fully GitOps-compatible, UI-free installation. The optional layer is ChaosCenter (the current name for what earlier releases called Litmus Portal) — a web control plane with its own workflow engine, a GraphQL API, and the litmusctl CLI, which connects to one or more target clusters through a lightweight in-cluster agent over gRPC. Component names have shifted across Litmus's 1.x, 2.x, and 3.x lines, so treat the shape below as durable and verify current naming against Litmus's own docs before you build automation against a specific API surface.

Whichever layer creates it, a ChaosEngine triggers the same reconciliation loop. The chaos-operator notices the new (or updated) ChaosEngine, checks a safety gate called annotation check — by default it refuses to touch a workload unless its pods carry the litmuschaos.io/chaos: "true" annotation, a deliberate opt-in so Litmus can never accidentally target something nobody explicitly exposed to it — and then creates a short-lived runner pod in the application's namespace. The runner pod, running under a tightly scoped chaosServiceAccount, in turn launches the actual experiment job pod: the container that does the fault injection itself, using an image pulled from the ChaosHub (commonly litmuschaos/go-runner, wrapping lower-level primitives like tc for network shaping, stress-ng for CPU/memory pressure, and direct Kubernetes API calls for pod and node actions). When the fault completes, the result is written back as a ChaosResult, and — if the chaos exporter is deployed — surfaced as a Prometheus metric a Grafana dashboard can chart alongside the rest of your telemetry.

The core custom resources: ChaosEngine, ChaosExperiment, ChaosResult

☺ Like you're 10: One object says which app to hit and with what settings, one is the reusable instruction sheet for the fault itself, and one is the report card written after it runs.

Almost everything you write by hand in Litmus is a ChaosEngine. It names the target application, which reusable fault to run against it, and the tuning knobs for that run — an environment-variable list is how Litmus experiments take their configuration, rather than a nested settings block. A ChaosExperiment, by contrast, is a read-only template: you install it once from the ChaosHub, you don't hand-write its business logic, and multiple ChaosEngines across different namespaces can reference the same installed experiment. The ChaosResult is created and updated by the experiment job itself, and is the object you actually query to find out whether the hypothesis held.

# chaosengine-checkout-pod-delete.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: checkout-pod-delete
  namespace: checkout
spec:
  appinfo:
    appns: checkout
    applabel: "app=checkout"          # selects pods carrying this label
    appkind: deployment
  chaosServiceAccount: pod-delete-sa  # scoped to THIS experiment, THIS namespace — see gotchas
  annotationCheck: "true"             # default; refuses to run unless target pods opt in
  engineState: active                 # "stop" halts a running/scheduled engine
  jobCleanUpPolicy: delete            # or "retain" to keep the job pod for post-mortem debugging
  experiments:
    - name: pod-delete               # must match an installed ChaosExperiment of the same name
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"             # seconds the experiment window runs for
            - name: CHAOS_INTERVAL
              value: "10"             # seconds between each pod kill within that window
            - name: FORCE
              value: "false"          # graceful delete, not SIGKILL
            - name: PODS_AFFECTED_PERC
              value: "25"             # blast-radius control — never more than a quarter of pods
        probe:
          - name: checkout-still-serving
            type: httpProbe
            mode: Continuous          # runs the whole time chaos is active, not just before/after
            runProperties:
              probeTimeout: 5
              interval: 5
              retry: 1
            httpProbe/inputs:
              url: "http://checkout.checkout.svc.cluster.local/healthz"
              method:
                get:
                  criteria: "=="
                  responseCode: "200"

The probe block is the direct implementation of the falsifiable steady-state hypothesis chaos engineering insists an experiment needs before it means anything. A httpProbe polls an endpoint and checks a response code; a cmdProbe runs a shell command and checks its output or exit code; a k8sProbe asserts a Kubernetes resource's state (a Deployment still reporting the expected replica count, say); a promProbe queries Prometheus and compares the result against a threshold. Attach one, and "checkout completion stays above 99%" stops being a claim someone eyeballs on a dashboard mid-experiment and becomes a machine-checked pass/fail baked into the ChaosResult itself.

The ChaosHub: a library of fault templates, not a blank page

☺ Like you're 10: Nobody writes the fault-injection code themselves — they pick a pre-built one off the shelf and just fill in the settings.

The ChaosHub is Litmus's catalog of pre-built ChaosExperiment templates, browsable at hub.litmuschaos.io and organized into categories that map cleanly onto the blast-radius ladder from chaos engineering: generic (pod-delete, pod-cpu-hog, pod-memory-hog, pod-network-loss, pod-network-latency, pod-network-corruption, pod-network-duplication, disk-fill, container-kill), node-level (node-drain, node-cpu-hog, node-memory-hog, node-taint, kubelet-service-kill), and cloud-provider experiments scoped to AWS, GCP, and Azure (EC2 instance stop, EBS volume loss, RDS instance reboot, and their equivalents elsewhere) — the closer analog, inside Kubernetes, to what Chaos Monkey and AWS Fault Injection Service do to compute instances directly. Installing one is a plain manifest apply — the hub publishes each experiment's CRD, default RBAC (rbac.yaml), and a starter ChaosEngine you copy and adapt rather than write from nothing.

This is also where the GitOps story starts before an experiment ever runs. A hub isn't limited to the public default — ChaosCenter lets a team add a private, Git-backed hub: point it at your own repository and branch, and every experiment template your organization has vetted, tuned, or written in-house becomes a versioned artifact reviewed through the same pull-request process as application code, not a one-off script living in someone's home directory.

Day-to-day commands

☺ Like you're 10: Install the template, apply the engine, then watch two objects — the runner pod's logs while it runs, and the ChaosResult once it's done.

# install Litmus core (operator + CRDs) — for a full ChaosCenter install, use the
# official Helm chart from the litmuschaos repo instead; chart name/version changes
# across releases, so confirm the current one on Litmus's own getting-started docs
$ helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
$ helm repo update
$ kubectl create ns litmus
$ helm install chaos litmuschaos/litmus-core -n litmus     # verify current chart name

# pull an experiment template down from the ChaosHub into the target namespace
$ kubectl apply -f https://hub.litmuschaos.io/api/chaos/master?file=charts/generic/pod-delete/experiment.yaml -n checkout
$ kubectl apply -f https://hub.litmuschaos.io/api/chaos/master?file=charts/generic/pod-delete/rbac.yaml -n checkout

# opt a workload into chaos, then run the experiment
$ kubectl annotate deployment checkout litmuschaos.io/chaos="true" -n checkout
$ kubectl apply -f chaosengine-checkout-pod-delete.yaml

# watch it happen
$ kubectl get chaosengine checkout-pod-delete -n checkout -w
$ kubectl get pods -n checkout -l name=checkout-pod-delete-runner
$ kubectl logs -n checkout -l name=checkout-pod-delete-runner -c chaos-runner -f

# the verdict — this is the line a CI pipeline actually polls
$ kubectl get chaosresult checkout-pod-delete-pod-delete -n checkout \
    -o jsonpath='{.status.experimentStatus.verdict}'
$ kubectl describe chaosresult checkout-pod-delete-pod-delete -n checkout

# via litmusctl, against ChaosCenter instead of raw kubectl
$ litmusctl config set-account --endpoint https://chaoscenter.acme.internal
$ litmusctl get agent-list
$ litmusctl create chaos-workflow -f chaos-workflow.yaml -p 

# stop an active engine without deleting it (re-set engineState: active to resume)
$ kubectl patch chaosengine checkout-pod-delete -n checkout --type merge -p '{"spec":{"engineState":"stop"}}'

Running chaos as a scheduled, version-controlled GitOps stage

☺ Like you're 10: Because a chaos experiment is just YAML, it can live in the same repo as the app, get reviewed the same way, and run itself on a calendar instead of waiting for someone to click a button.

Every object on this page — ChaosEngine, an installed ChaosExperiment, a custom ChaosHub, even a multi-step workflow — is ordinary Kubernetes YAML, which means it fits into a GitOps repository exactly the way a Deployment or a NetworkPolicy does: a chaos/ directory sitting next to the app's Helm chart or Kustomize overlays, synced by whatever reconciler the platform runs the same way everything else in the cluster is synced. That single move converts chaos engineering from "a thing an engineer remembers to run before a big launch" into a reviewable, diffable, rollback-able part of the deployment pipeline — the exact discipline blameless postmortems and production readiness reviews already apply to everything else that touches production.

Three concrete patterns come out of that:

• Scheduled, recurring runs. A single ChaosEngine applied once doesn't repeat itself on a calendar. ChaosCenter's workflow engine is built on Argo Workflows, and its CronWorkflow support is what turns a chaos experiment (or a chained sequence of several) into a recurring job — the closest in-Kubernetes equivalent to Chaos Monkey's always-on cadence, except declared as a version-controlled manifest with an explicit cron schedule, a named owner in git blame, and a pull-request history, instead of a standalone daemon nobody remembers configuring.
• A pipeline gate, not a side activity. A CI/CD stage applies a ChaosEngine (or triggers a Litmus workflow through litmusctl or ChaosCenter's GraphQL API) against a freshly deployed canary, polls the resulting ChaosResult for its verdict, and blocks promotion to full production traffic if the verdict is Fail — turning a resilience hypothesis into the same kind of automated gate a failing unit test already is, rather than a manual step someone can forget under deadline pressure.
• Reviewed changes to the blast radius. Because PODS_AFFECTED_PERC, TOTAL_CHAOS_DURATION, and the probe thresholds all live in the committed ChaosEngine, widening an experiment's scope — moving from one pod to a percentage of traffic, the same ladder chaos engineering describes — is a pull request with a visible diff, not a UI toggle someone flips without anyone else noticing until the next incident review.

Gotchas and failure modes

☺ Like you're 10: Most Litmus surprises come from RBAC copied too loosely, a version mismatch with an old tutorial, or the experiment's own image failing to download.

RBAC copied from a tutorial is usually over-scoped

Every experiment on the ChaosHub ships a starter rbac.yaml, and the fastest way through a demo is to grab the broadest role in a blog post and move on. In production that habit hands a short-lived runner pod far more permission than a single pod-delete experiment needs — a real gap covered in depth in security's overlap with reliability. Scope chaosServiceAccount to exactly the namespace and the verbs the specific experiment requires, and treat a role that grants more than that as a finding, not a convenience.

Version skew breaks copy-pasted manifests

Litmus's architecture and component names have moved across three major eras — 1.x's standalone CRDs, 2.x's Litmus Portal, 3.x's ChaosCenter — and a manifest copied from an older blog post can reference a CRD (ChaosSchedule from the 1.x line, notably) that a current installation no longer ships. When an applied manifest silently does nothing, check the CRD's apiVersion and confirm it's still installed before assuming the experiment itself is broken.

The experiment's own image has to actually pull

The experiment job pod pulls its execution image — commonly litmuschaos/go-runner — from a public registry at run time. In an air-gapped cluster or during a registry outage, the chaos experiment fails before it injects any fault at all, which is a genuinely confusing first debugging step: the system you're testing looks unaffected not because it survived, but because nothing was ever injected. Mirror the image ahead of time in any environment that can't reach the public registry on demand.

Cleanup policy and stale job accumulation

jobCleanUpPolicy: retain is genuinely useful the first few times you debug a failing experiment — the runner and job pods stay around for kubectl logs and kubectl describe after the fact. Left on permanently across a recurring CronWorkflow, though, it quietly fills the namespace with completed Job objects that nobody prunes. Default to delete once an experiment is trusted, and reserve retain for active debugging.

Litmus vs. the alternatives

☺ Like you're 10: Other tools ask the same "inject a real fault and check the hypothesis" question — the real differences are how Kubernetes-native they are, how far outside Kubernetes they reach, and who's paying for support.

The practical rule most Kubernetes-first teams land on: reach for Litmus (or Chaos Mesh) when the workloads under test already live in the cluster and the deployment pipeline is already GitOps, because the experiment then costs nothing extra to review or schedule. Reach for Gremlin or a cloud-native tool like AWS FIS when chaos needs to reach infrastructure Kubernetes doesn't own. See the SRE toolchain for where the chaos-tooling category sits next to monitoring and paging, and chaos engineering at scale for running a chaos program across many teams and services rather than one experiment at a time.

Going further

☺ Like you're 10: This page is enough to write and schedule a real experiment — Litmus's own site is where component names and exact CRD fields stay current.

The canonical sources are the Litmus documentation at litmuschaos.io, the ChaosHub itself at hub.litmuschaos.io, the source and issue tracker at github.com/litmuschaos/litmus, and the CNCF project page at cncf.io/projects/litmus for current maturity status. Pair this page with chaos engineering for the hypothesis-and-blast-radius discipline Litmus implements in YAML, Kubernetes reliability patterns for the failure modes worth targeting first, chaos engineering at scale for running a program across many services rather than one experiment, and anti-fragility & learning from failure for where this fits the exam blueprint. For hands-on reps designing and running a real experiment end to end, the chaos-experiment-design drill and Capstone Part 6 — chaos engineer it are exactly that exercise.