Velero

Velero is the open-source tool that gives a Kubernetes platform an answer to the three questions nobody wants to be asked live: can we get that namespace back?, can we move this workload to the new cluster?, and what happens if this cluster disappears? It walks the API server, writes the resulting API objects into object storage as a tarball with metadata, and captures the data inside persistent volumes either by asking the storage layer for a CSI snapshot or by reading the filesystem directly with Kopia. It then reverses the whole process on demand — into the same cluster, a different namespace, or an entirely different cluster.

What Velero is and the problem it solves

☺ Like you’re 10: It’s a helper that copies your cluster into a box in a warehouse, and can unpack the box later — here, or somewhere else entirely.

Velero started life at Heptio as Ark, was renamed to Velero in 2019, and is developed in the open under the vmware-tanzu GitHub organisation with a broad contributor community. It occupies a slot almost every other cloud native tool ignores: the GitOps stack can rebuild your configuration from Git in minutes, but it has nothing whatsoever to say about the twelve gigabytes in a PostgreSQL PersistentVolumeClaim, the Secrets a controller generated at runtime, or the twenty custom resources an operator wrote back into the API server. Velero covers exactly that gap.

The problem before Velero

Ask a platform team without a backup tool how they would recover a deleted namespace and you get one of three answers. “We’d re-apply from Git” — restores the manifests, loses the data. “We have volume snapshots in the cloud console” — per-disk, untagged, and no record of which snapshot belonged to which claim in which namespace. Or “we take etcd snapshots” — an all-or-nothing image of the control plane, unusable for restoring one namespace while the cluster keeps running. None of these is a recovery procedure; they are ingredients.

The awkward truth is that most Kubernetes data loss is not a cloud region going dark. It is a kubectl delete namespace in the wrong terminal, an Argo CD app with pruning enabled and a bad path in Git, or a Helm upgrade that recreated a StatefulSet with a different volumeClaimTemplate. Those are human-scale accidents, and they need a namespace-scoped, human-scale undo.

What a Velero backup actually contains

A Velero backup is two things stored side by side. First, a tarball of API objects — the JSON of every resource that matched your inclusion rules, arranged by namespace and resource type, plus cluster-scoped resources that the namespaced items depend on. Second, volume data, captured by whichever mechanism you configured, referenced by metadata that ties each captured volume back to the PersistentVolumeClaim that owned it. Alongside those sit logs, a resource list, and a JSON summary — which is why the CLI can tell you afterwards exactly what went in and what did not.

Everything lands in one object storage bucket under a predictable layout: <bucket>/<prefix>/backups/<backup-name>/. That bucket, not the cluster, is the durable artefact — Velero is stateless enough that you can delete the whole installation, reinstall it against the same bucket, and within a minute every backup reappears as a Backup object ready to restore. That property is what makes cluster migration almost trivial.

What Velero deliberately is not

Velero does not back up etcd. It talks to the API server as a well-behaved client, so it captures what the API exposes — not the raw control-plane database, not the certificates, not the kubeadm configuration. A self-managed control plane still needs etcdctl snapshot save as a separate discipline, as the substrate lesson covers. Velero is also not a database backup tool: it can snapshot the volume under a database, but a consistent logical dump belongs to the engine’s own tooling. And it is not continuous — your recovery granularity is the interval of your Schedule, nothing finer.

Where Velero fits in a platform

☺ Like you’re 10: It sits in the basement with the spare keys. Nobody thinks about it until the day it is the only thing that matters.

In the reference architecture, Velero is a platform-operated capability in the same tier as observability and policy: the platform team installs and owns it, tenants consume it without installing anything, and its outputs are governed centrally. The self-service surface a tenant sees is usually not Velero at all — it is a small platform API or a Backstage action that says “protect this namespace, daily, keep 30 days”, which the platform translates into a Schedule.

The capability it provides, in platform terms

Backup is where an abstract reliability conversation becomes concrete. Reliability and incidents talks about RPO — how much data you can afford to lose — and RTO — how long recovery may take. Velero turns both from aspirations in a document into settings you can point at: schedule: "0 */6 * * *" is a six-hour RPO, and a rehearsed restore that takes forty minutes is a forty-minute RTO. If your compliance obligations name retention periods, ttl is where they land.

GitOps covers config; Velero covers state

This division is worth stating plainly, because teams get it wrong in both directions. If everything you run is stateless and fully described in Git, you may genuinely not need Velero for recovery — you need a working cluster and a sync. The moment you host a database, a durable queue, a registry, or any operator that treats the API server as its own database, Git stops being sufficient. Velero also captures runtime objects Git never had: Secrets created by cert-manager or External Secrets, and CRs written back by controllers. That cuts both ways — a Velero backup therefore contains secret material, so the bucket needs encryption at rest, tight IAM, and the care secrets management demands everywhere else.

CNPE relevance

Velero is not on the official CNPE tool list, and you should not expect a task that says “install Velero”. It is here because backup and disaster recovery are unambiguously platform responsibilities, and the concepts are examinable: stateful workloads and their volume lifecycle in Storage & State, RPO/RTO and blast radius in Reliability, moving workloads between clusters in Multi-cluster, and the recurring anti-pattern of untested recovery. Read this page for the platform judgement, not for a memorised CLI.

How it works — architecture, components, CRDs

☺ Like you’re 10: You write a note asking for a backup. A program in the cluster reads the note, walks around collecting things, and posts the box to the warehouse.

Velero is a controller like any other. The CLI does almost nothing clever: it creates a Backup or Restore object and then watches it. Everything real happens server-side, which means you can drive the entire tool with kubectl apply and GitOps if you prefer, and it means backups triggered by a Schedule follow exactly the same code path as ones you type by hand.

The components

Three pieces, plus plugins. The Velero server is a Deployment in the velero namespace running the backup, restore, schedule, backup-sync and garbage-collection controllers. The node-agent is an optional DaemonSet — only needed for filesystem backup and for snapshot data movement — that mounts the host’s pod volume directory and runs Kopia. The velero CLI is a thin client. Everything provider-specific arrives as a plugin container: an object store plugin for AWS, Azure or GCP that knows how to talk to the bucket, and a volume snapshotter plugin for cloud disks. CSI support is built into the server in current releases rather than shipped as a separate plugin, which is a common source of stale advice on the internet.

Two ways to capture volume data

This choice matters more than any other. CSI snapshots ask the storage driver to create a VolumeSnapshot — fast, storage-native, usually copy-on-write. But the snapshot lives in the storage system, meaning a specific cloud and usually a specific region, and is useless to a cluster elsewhere unless you also enable snapshot data movement, which uploads its contents into your object storage bucket. Filesystem backup instead has the node-agent read the mounted volume file by file into a Kopia repository in the bucket: slower, and it must read a live filesystem, but it works with any storage class — including ones with no snapshot support — and is portable by construction. Kopia replaced Restic as the default uploader; Restic is still selectable with --uploader-type=restic but is the legacy path, and the old ResticRepository resource is now BackupRepository.

The custom resources

All of these live in the velero.io API group, in the velero namespace. Everything is velero.io/v1 except the snapshot data-movement types, which are still velero.io/v2alpha1 — worth knowing before you write a manifest or an RBAC rule against them.

The resources you will actually write

☺ Like you’re 10: Three little YAML files: where the warehouse is, what to pack and how often, and how to unpack it somewhere new.

Storage locations — the foundation everything sits on

Get this wrong and nothing else works. A BackupStorageLocation is usually created for you by velero install, but writing it by hand is how you add a second region, a read-only location for restores from another cluster’s bucket, or per-tenant isolation.

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: primary
  namespace: velero
spec:
  provider: aws                    # matches the installed object store plugin
  default: true                    # used when a Backup omits storageLocation
  objectStorage:
    bucket: acme-platform-backups
    prefix: prod-eu                # lets several clusters share one bucket safely
  credential:                      # a key inside a Secret in the velero namespace
    name: velero-cloud-credentials
    key: cloud
  config:
    region: eu-west-1
    s3ForcePathStyle: "false"      # "true" + s3Url for MinIO / on-prem S3
  accessMode: ReadWrite            # ReadOnly on a DR cluster: restore, never write
  backupSyncPeriod: 1m             # how often it re-reads the bucket's backup list
  validationFrequency: 1m          # marks the location Unavailable if it can't reach it
---
apiVersion: velero.io/v1
kind: VolumeSnapshotLocation        # only needed for NATIVE cloud disk snapshots
metadata:
  name: eu-west-1
  namespace: velero
spec:
  provider: aws
  config:
    region: eu-west-1               # snapshots are created HERE and stay HERE

☺ Like you’re 10: One file says which warehouse, one says which snapshot fridge. The fridge only works in the town it stands in.

Backups and Schedules

A one-off Backup and a Schedule are the same object; the schedule just wraps a template around it. Note ttl: this is not a hint but a hard expiry — when it elapses the garbage collector deletes the backup and its snapshots. The default is 720h0m0s, thirty days, and it is the setting people most often forget to raise before a compliance auditor asks about it.

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: checkout-adhoc-preupgrade
  namespace: velero
spec:
  includedNamespaces: [checkout]
  excludedResources:                # things it is pointless or harmful to restore
    - events
    - events.events.k8s.io
    - pods                          # let the Deployment recreate them
  labelSelector:                    # only objects carrying this label
    matchLabels:
      backup.acme.dev/tier: critical
  includeClusterResources: null     # null = auto: include only what's needed
  snapshotVolumes: true             # take CSI / cloud snapshots of PVs
  snapshotMoveData: true            # ...and upload the CSI snapshot's CONTENTS to the
                                    # bucket, which is what makes them portable across
                                    # regions. Needs the node-agent DaemonSet.
  defaultVolumesToFsBackup: false   # true = use Kopia for every volume instead
  storageLocation: primary
  volumeSnapshotLocations: [eu-west-1]  # only consulted by the NATIVE cloud volume
                                        # snapshotter plugin, not by the CSI path
  ttl: 2160h0m0s                    # 90 days, then garbage-collected
---
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: checkout-nightly
  namespace: velero
spec:
  schedule: "0 1 * * *"             # 01:00 UTC — the server's clock, not yours
  paused: false
  useOwnerReferencesInBackup: false # true = deleting the Schedule deletes its backups
  template:                         # a Backup spec, minus the metadata
    includedNamespaces: [checkout]
    snapshotVolumes: true
    ttl: 720h0m0s
    hooks:
      resources:
        - name: quiesce-postgres
          includedNamespaces: [checkout]
          labelSelector:
            matchLabels: { app: postgres }
          pre:                       # runs BEFORE the volume is captured
            - exec:
                container: postgres
                # dump onto a path on the PVC that is being backed up, so the captured
                # volume holds a consistent logical copy as well as the raw files
                command: ["/bin/sh","-c","pg_dump -U postgres -Fc -f /var/lib/postgresql/data/velero/app.dump app"]
                onError: Fail        # Fail = abort the backup; Continue = carry on
                timeout: 10m
          post:                      # runs AFTER the volume has been captured
            - exec:
                container: postgres
                command: ["/bin/sh","-c","rm -f /var/lib/postgresql/data/velero/app.dump"]
                onError: Continue

Hooks are the difference between a snapshot and a consistent snapshot. A block-level snapshot of a running database is crash-consistent — equivalent to pulling the power cord. Most engines recover from that, but “most” is doing heavy lifting in a sentence about your only copy of production data. A pre-hook that dumps to a file on the volume being captured turns a gamble into a procedure. Note that you generally cannot drive a database’s low-level backup API from a pre/post hook pair: PostgreSQL’s pg_backup_start / pg_backup_stop (named pg_start_backup / pg_stop_backup before PostgreSQL 15) are non-exclusive and bound to the session that called them, so two separate short-lived psql invocations simply cancel each other out. A logical dump, or the database operator’s own backup mechanism, is the workable answer. Hooks can equally be declared as pod annotations (pre.hook.backup.velero.io/command, /container, /timeout, /on-error and the matching post.hook.backup.velero.io/* set), letting an application team own their own quiescing without touching the platform’s Schedule — a good self-service seam.

Restores, namespace mapping, and existing resources

A Restore is where the interesting decisions live. By default Velero skips a resource that already exists rather than overwriting it, which surprises people who expected an undo and got a no-op.

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: checkout-restore-2026-07-20
  namespace: velero
spec:
  backupName: checkout-nightly-20260719010012
  includedNamespaces: [checkout]
  namespaceMapping:
    checkout: checkout-recovery     # restore SIDE BY SIDE, verify, then cut over
  restorePVs: true                  # recreate PVs from snapshots / Kopia data
  existingResourcePolicy: update    # "none" (default) = skip anything that exists
                                    # "update" = patch it to match the backup
  preserveNodePorts: false          # true only if you need the exact NodePorts back
  includedResources: []             # empty = everything that was in the backup
  excludedResources:
    - nodes
    - events
  hooks:
    resources:
      - name: warm-cache
        includedNamespaces: [checkout-recovery]
        labelSelector:
          matchLabels: { app: checkout-api }
        postHooks:
          - exec:
              container: api
              command: ["/bin/sh","-c","/app/bin/rebuild-cache"]
              waitTimeout: 5m       # how long to wait for the container to be ready
              execTimeout: 10m
              onError: Continue

The namespaceMapping line is the single most useful field on this page. Restoring into a new namespace lets you verify the data before touching the live one, turns a terrifying operation into a reversible one, and is exactly how you rehearse a restore in production without a maintenance window. Make it the default habit and “restore drill” stops being scary enough to postpone.

Day-to-day commands

☺ Like you’re 10: Ask for a backup, watch it, read what it did, and — the important one — practise unpacking it.

Creating backups and schedules

# Everything below just creates the CRs shown above — the server does the work.
velero backup create checkout-adhoc \
  --include-namespaces checkout \
  --exclude-resources events,events.events.k8s.io \
  --selector 'backup.acme.dev/tier=critical' \
  --snapshot-volumes \
  --ttl 168h0m0s \
  --wait                                   # block until it finishes

# Filesystem (Kopia) backup instead of storage snapshots — needs the node-agent.
velero backup create checkout-fs \
  --include-namespaces checkout \
  --default-volumes-to-fs-backup

# Portable snapshots: take a CSI snapshot, then move its DATA into the bucket.
velero backup create checkout-portable \
  --include-namespaces checkout --snapshot-move-data

velero schedule create checkout-nightly \
  --schedule="0 1 * * *" --include-namespaces checkout --ttl 720h0m0s
velero schedule pause  checkout-nightly    # e.g. during a planned migration
velero backup create manual-from-sched --from-schedule checkout-nightly

Reading, debugging, and proving what happened

These four commands are the whole diagnostic loop, and the second one is the one nobody runs often enough.

velero get backups                          # phase, errors, warnings, expiry
velero backup describe checkout-nightly-20260719010012 --details
#   ^ resource counts, per-volume snapshot status, hook results, WHY it partially failed
velero backup logs checkout-nightly-20260719010012 | grep -iE 'error|warn'
velero backup download checkout-nightly-20260719010012   # the raw tarball, to inspect

velero restore describe checkout-restore-2026-07-20 --details
velero restore logs     checkout-restore-2026-07-20

# The infrastructure itself.
velero backup-location get                  # Available vs Unavailable — check first
velero snapshot-location get
kubectl -n velero logs deploy/velero
kubectl -n velero logs ds/node-agent        # filesystem backup problems live here
kubectl -n velero get podvolumebackups,datauploads
velero debug --backup checkout-nightly-20260719010012   # support bundle, all of it

Restore and migration drills

# The rehearsal: restore into a scratch namespace, verify, delete. Do this monthly.
velero restore create drill-$(date +%Y%m%d) \
  --from-backup checkout-nightly-20260719010012 \
  --namespace-mappings checkout:checkout-drill \
  --existing-resource-policy=none \
  --wait
kubectl -n checkout-drill get pods,pvc
kubectl -n checkout-drill exec deploy/postgres -- psql -U postgres -c 'select count(*) from orders;'
kubectl delete ns checkout-drill

# Cluster migration, in three steps.
#  1. In the NEW cluster, install Velero against the SAME bucket and prefix.
#     Pin the plugin tag to the release that matches your Velero version.
velero install --provider aws \
  --bucket acme-platform-backups --prefix prod-eu \
  --backup-location-config region=eu-west-1 \
  --plugins velero/velero-plugin-for-aws:vX.Y.Z \
  --secret-file ./credentials \
  --use-node-agent \
  --use-volume-snapshots=false
#     On a restore-only cluster, flip the location to ReadOnly afterwards
#     (velero install has no flag for it; patch the object it created):
kubectl -n velero patch backupstoragelocation default \
  --type=merge -p '{"spec":{"accessMode":"ReadOnly"}}'
#  2. Wait ~1 minute for backupSyncPeriod, then confirm the history appeared.
velero get backups
#  3. Restore.
velero restore create --from-backup checkout-nightly-20260719010012 --wait

Gotchas and failure modes

☺ Like you’re 10: The scary ones aren’t loud errors. They’re backups that say “mostly fine” and restores that quietly skip things.

PartiallyFailed, and the phases nobody reads

Velero’s phases are honest, and that honesty is what makes them dangerous: PartiallyFailed is a green-ish word for “some of your data is not in this backup.” It appears in a list next to Completed entries and looks fine at a glance. Worse, a backup whose filters matched nothing at all still finishes Completed with zero errors — a perfectly successful backup of nothing, usually caused by a typo in a namespace name or a label selector that matches no objects.

Restore ordering, CRDs, and cluster-scoped resources

Restoring Kubernetes objects is not a bulk apply — order matters, and Velero enforces a priority list: CustomResourceDefinitions first, then Namespaces, StorageClasses, VolumeSnapshotClasses and snapshot contents, then PersistentVolumes and PersistentVolumeClaims, then ServiceAccounts, Secrets and ConfigMaps, and only then Pods and controllers. The reason is obvious once stated — a custom resource cannot be created before its CRD is established, and a Pod cannot mount a PVC that does not exist yet. When you restore an operator-managed application to a fresh cluster and it fails, the cause is nearly always this: the CRD was not in the backup, because includeClusterResources defaulted to auto and your namespace filter excluded cluster-scoped objects. Back up the operator’s CRDs explicitly, or restore the operator first and its CRs second. Workload triage is where you land when the restored pods then refuse to become ready.

Two related traps. Cluster-scoped resources such as ClusterRoles, ValidatingWebhookConfigurations and PriorityClasses are only pulled in automatically when a restored namespaced object references them — a namespace-only backup is not a cluster backup. And Velero never deletes: restoring over a live namespace adds and optionally updates objects, but anything created since the backup stays. A restore is not a rollback to a point in time. If you need one, delete the namespace first or restore into a mapped one.

Snapshots are cloud-bound, region-bound, and quietly expensive

A native cloud disk snapshot is a pointer inside one storage system in one region. It cannot be restored into a different cloud, usually not into a different region without an explicit copy step, and never into a cluster whose storage class comes from a different driver. Teams discover this at the worst moment: the DR plan says “fail over to eu-central-1”, and every snapshot is in eu-west-1. The fixes are --snapshot-move-data, so contents land in your replicable bucket, or filesystem backup, which is portable by design. Note too that ttl deletes snapshots as well as metadata, making long retention on large volumes a real FinOps line item — while deleting a Backup with kubectl delete instead of velero backup delete removes the record and orphans the snapshots, which then bill forever.

The golden rule

Alternatives and when to choose it

☺ Like you’re 10: Other boxes and other helpers exist. Some are fancier, some are free, and one of them isn’t really a backup at all.

The honest framing is that these options are mostly complementary layers rather than rivals. Serious platforms run two or three: etcd snapshots for the control plane, Velero for namespaces and volumes, and native database backups for the databases that matter most.

The comparison that decides it

A practical rule

Run GitOps for configuration recovery, Velero for namespaces and volumes, and database-native backups for your handful of tier-one datastores. Choose filesystem backup (Kopia) as the default unless volume sizes make it too slow, because portability is worth more than speed on the day you need it. Put the bucket in a different account and region from the cluster with versioning and object-lock enabled. Then schedule the drill. See The Tool Landscape for how Velero sits alongside the named CNPE projects, and Storage & State for the volume lifecycle it depends on.

Exam relevance and going further

☺ Like you’re 10: Velero itself is unlikely to be on the test — but “how do you get it back?” absolutely is, and you cannot look up the answer.

Velero is not on the official CNPE tool list, so do not spend drill time memorising its flags. What is examinable is the surrounding judgement: what a PersistentVolume’s reclaim policy does when a claim is deleted, why a StatefulSet’s volumes outlive its pods, what RPO and RTO mean and how a schedule maps to them, why GitOps alone does not protect stateful workloads, and why an untested recovery plan is an anti-pattern rather than a plan. Expect those as scenario questions in Storage & State and Reliability framing, and see the architecture practice set for the shape they take.

The documentation allowlist — read this twice

⚖ CNPA vs CNPE — That allowlist is a CNPE-only mechanic — CNPE is hands-on and lets you reference a narrow set of docs during the exam. CNPA is fully closed-book multiple-choice with zero external lookups of any kind, so there is no allowlist to lean on at all. Even so, the backup/restore judgement this page drills is exactly the kind of concept-level knowledge CNPA expects you to recall cold.

What to be able to explain without notes

Say in one sentence what a Kubernetes backup must capture that Git does not: volume data and runtime-generated objects. Explain why restore ordering exists and name two things that come first (CRDs, Namespaces). Explain why a native cloud snapshot does not help a cross-region failover, and the two fixes. State the difference between RPO and RTO. And be ready to say, without hedging, why a backup that has never been restored provides no assurance — that argument is the one a platform engineer is actually paid to make. The wider decision tree for “it came back but it is broken” lives in the troubleshooting playbook, and delivery-side rollbacks in delivery triage.

Official resources for after the exam

Outside the exam, the canonical sources are velero.io/docs (the Restore Reference and Backup Hooks pages repay slow reading), the API types in github.com/vmware-tanzu/velero, kopia.io/docs for the uploader underneath filesystem backup, and the upstream Kubernetes volume snapshot documentation — which, unlike the rest, is on the exam allowlist. Pair this page with Storage & State for volumes and StatefulSets, Multi-cluster for the migration story, Best practices for where the drill belongs in your operating rhythm, and the glossary when a term stops making sense.