Velero

Every cluster eventually needs an answer to three separate emergencies: someone ran kubectl delete namespace against the wrong context, a control-plane host disappeared along with the region it lived in, and a workload needs to move to a brand-new cluster without weeks of manual re-provisioning. GitOps can rebuild the Deployment and ConfigMap objects a workload is made of, but it has nothing to say about the twelve gigabytes inside a PersistentVolumeClaim or the Secret a controller minted at runtime and never committed to Git. Velero is the tool that covers that gap: it walks the Kubernetes API, writes what it finds into an object storage bucket as a portable archive, and — separately — captures the bytes inside every attached volume, either through a CSI snapshot or by reading the filesystem directly. Run in reverse, that same archive rebuilds a namespace, a dead cluster, or an entirely different one.

What a Velero backup actually contains

☺ Like you're 10: Two separate jobs happen every time — one camera-click for the API objects, one truck for what's inside the volumes — and either one can succeed while the other quietly doesn't.

A Backup is two things, captured by two different mechanisms, stored side by side in one object storage bucket. First, a tarball of API objects — the JSON of every resource that matched your namespace, label, and resource filters, plus the cluster-scoped resources those namespaced objects depend on. Second, volume data, captured either by asking the storage driver for a CSI snapshot or by having Velero's own agent read the volume's filesystem directly with Kopia (its default uploader; the older restic uploader still works but is the legacy path). Those two halves fail independently, which is the single most important thing to internalize before you trust a green checkmark.

Velero is deliberately not two other things it gets confused with. It is not an etcd snapshot — it talks to kube-apiserver as an ordinary client, so it captures what the API exposes, never the raw control-plane database, the static Pod manifests, or the certificates a self-managed control plane depends on. Cluster Architecture, Installation & Configuration covers etcdctl snapshot save as its own, separate discipline, and Best Practices & Operating Model already lays out exactly where etcd snapshots and Velero backups each end and why a healthy operating model runs both. And it is not GitOps — GitOps on Kubernetes can rebuild your declared configuration from a Git commit in minutes, but it was never told about the runtime Secret a controller generated last night or the rows inside a database's volume. Velero exists for exactly the state Git never held.

Architecture: the server, the node-agent, and where the bucket fits

☺ Like you're 10: One deployment does the talking to Kubernetes and the cloud; one helper rides along on every node for volumes that need reading byte by byte; and everything either of them produces lands in one shared warehouse.

The velero CLI is a thin client — nearly everything it does is create a Backup, Schedule, or Restore object and then watch it. All the real work happens server-side in the Velero server, a Deployment in the velero namespace that bundles the backup, restore, schedule, backup-sync, and garbage-collection controllers into one binary. That matters in practice: because a Schedule just stamps out ordinary Backup objects on a cron, a backup triggered at 1 a.m. runs through the exact same code path as one you type by hand at your desk, and the whole tool is drivable through plain kubectl apply if that's how your platform prefers to manage it.

Two optional pieces round it out. The node-agent is a DaemonSet, one Pod per node, that mounts each node's Pod-volume directory and runs Kopia against it — it's only needed when you want filesystem backup instead of, or as well as, CSI snapshots, and it's also what moves a CSI snapshot's contents into the bucket when you ask for that. Everything provider-specific — talking to a particular cloud's object store, taking a particular cloud's disk snapshot — arrives as a plugin container selected at install time, which is why the same Backup spec works unmodified against AWS, Azure, GCP, or an on-prem S3-compatible store like MinIO.

The resources you actually write

☺ Like you're 10: Three little files: where the warehouse is, what to pack and how often, and how to unpack it — possibly somewhere new.

A BackupStorageLocation is the foundation everything else sits on; velero install usually creates one for you, but writing it by hand is how you add a read-only location for restores, or point a second cluster at the same bucket for migration.

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: primary
  namespace: velero
spec:
  provider: aws                    # matches the installed object-store plugin
  default: true                    # used whenever a Backup omits storageLocation
  objectStorage:
    bucket: acme-k8s-backups
    prefix: prod-eu                # lets several clusters share one bucket safely
  credential:
    name: velero-cloud-credentials
    key: cloud
  accessMode: ReadWrite             # ReadOnly on a DR-standby cluster: restore, never write
  backupSyncPeriod: 1m              # how often it re-reads the bucket's own backup list
---
apiVersion: velero.io/v1
kind: VolumeSnapshotLocation         # only needed for NATIVE cloud disk snapshots
metadata:
  name: eu-west-1
  namespace: velero
spec:
  provider: aws
  config:
    region: eu-west-1                # the snapshot is created HERE and stays HERE

A Schedule wraps a Backup template around a cron expression — the controller stamps out ordinary backups from it. The ttl field is not a hint; it's a hard expiry that deletes the backup and its snapshots the moment it elapses.

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: checkout-nightly
  namespace: velero
spec:
  schedule: "0 1 * * *"              # 01:00 — the server's clock, not yours
  template:
    includedNamespaces: [checkout]
    snapshotVolumes: true            # take CSI / cloud snapshots of the PVs
    defaultVolumesToFsBackup: false  # true = use Kopia for every volume instead
    ttl: 720h0m0s                    # 30 days, then garbage-collected — snapshots too
    hooks:
      resources:
        - name: quiesce-postgres
          labelSelector:
            matchLabels: { app: postgres }
          pre:                        # runs BEFORE the volume is captured
            - exec:
                container: postgres
                command: ["/bin/sh","-c","pg_dump -U postgres -Fc -f /var/lib/postgresql/data/velero/dump.sql app"]
                onError: Fail          # Fail = abort the backup; Continue = carry on anyway

Hooks are what turns a snapshot into a consistent one — a raw volume snapshot of a running database is crash-consistent, which most engines survive but which is a bad bet on your only copy of production data. A Restore is where the interesting decisions live: by default Velero skips anything that already exists rather than overwriting it, which surprises people expecting a plain undo.

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: checkout-restore-20260827
  namespace: velero
spec:
  backupName: checkout-nightly-20260827010000
  namespaceMapping:
    checkout: checkout-recovery       # restore SIDE BY SIDE, verify, then cut over
  restorePVs: true
  existingResourcePolicy: none        # "none" (default) = skip anything that exists
                                       # "update" = patch it to match the backup instead

Day-to-day commands

☺ Like you're 10: Ask for a backup, watch it, read what it actually did, then — the important one — practice unpacking it somewhere safe.

Velero ships as a Helm chart or via its own CLI installer; either path lands the same server Deployment.

# Install — pin a plugin version and point it at your bucket
$ velero install --provider aws \
    --bucket acme-k8s-backups --prefix prod-eu \
    --backup-location-config region=eu-west-1 \
    --plugins velero/velero-plugin-for-aws:v1.10.0 \
    --secret-file ./credentials-velero \
    --use-node-agent

# Create backups
$ velero backup create checkout-adhoc --include-namespaces checkout --wait
$ velero backup create checkout-fs --include-namespaces checkout --default-volumes-to-fs-backup
$ velero schedule create checkout-nightly --schedule="0 1 * * *" --include-namespaces checkout --ttl 720h0m0s

# Read, debug, prove what happened — the second line is the one nobody runs enough
$ velero get backups                                          # phase, errors, expiry
$ velero backup describe checkout-nightly-20260827010000 --details
$ velero backup logs checkout-nightly-20260827010000 | grep -iE 'error|warn'
$ velero backup-location get                                  # Available vs Unavailable — check first

# Restore, into a mapped namespace so a drill is safe to run in production
$ velero restore create checkout-drill \
    --from-backup checkout-nightly-20260827010000 \
    --namespace-mappings checkout:checkout-drill \
    --wait
$ velero restore describe checkout-drill --details

One mechanism, three jobs

☺ Like you're 10: The exact same box and the exact same truck can rebuild your old room, a matching new room across town, or a totally different apartment in another city — you just decide where to send it.

Velero's restore path never asks whether the destination is the cluster it came from. That single fact is why the same mechanism covers three jobs that feel operationally different but are mechanically identical.

• Accidental-delete recovery. Same cluster, same everything — restore the latest backup into a mapped namespace (checkout → checkout-recovery), verify the data, then cut traffic over and delete the broken original.
• Disaster recovery. The original cluster is gone. Stand up a new one, install Velero pointed at the same bucket and prefix, wait roughly a backupSyncPeriod for the backup history to reappear as Backup objects, and restore. Nothing about this step differs from a routine restore — the bucket, not the cluster, was always the durable artifact.
• Migration. The original cluster is still running fine, but the workload needs to live somewhere else entirely — a version upgrade too risky to do in place, a move to a new region, a switch of cloud providers. Same install-against-the-same-bucket procedure as DR, just planned and rehearsed instead of triggered by an outage, usually followed by a deliberate DNS or traffic cutover rather than an emergency one.

# Migration / DR, on the NEW cluster — same bucket, same prefix
$ velero install --provider aws \
    --bucket acme-k8s-backups --prefix prod-eu \
    --plugins velero/velero-plugin-for-aws:v1.10.0 \
    --secret-file ./credentials-velero --use-node-agent

$ kubectl -n velero patch backupstoragelocation primary \
    --type=merge -p '{"spec":{"accessMode":"ReadOnly"}}'   # restore-only until cutover

$ velero get backups                                          # confirm history synced back
$ velero restore create --from-backup checkout-nightly-20260827010000 --wait

Gotchas

☺ Like you're 10: The scary failures aren't loud errors — they're backups that say "mostly fine" and restores that quietly skip things nobody asked them to skip.

• PartiallyFailed looks fine in a list. It sits right next to Completed entries and means "some of your data isn't in this backup." A backup whose filters matched nothing at all still finishes Completed with zero errors — check the resource and volume counts in describe --details, not the phase word alone.
• A restore is not a rollback. Velero adds missing objects and, with existingResourcePolicy: update, patches existing ones — but it never deletes anything created since the backup. Restoring over a live namespace gives you a merged half-old, half-new state, not a point-in-time revert.
• Restore ordering exists, and CRDs go first. Velero enforces a priority order — CustomResourceDefinitions, then Namespaces, StorageClasses, PersistentVolumes, PersistentVolumeClaims, ServiceAccounts and Secrets, and only then Pods and controllers. Restore an operator-managed app to a fresh cluster and it fails almost every time because the operator's CRD was never in the backup — a namespace-only filter doesn't pull in cluster-scoped resources unless something namespaced still references them.
• Native cloud snapshots are region- and cloud-bound. A CSI-triggered disk snapshot lives inside one storage system in one region and can't be restored into a different cloud or usually even a different region on its own. snapshotMoveData: true (or plain filesystem backup) is what makes a backup actually portable — exactly what a DR or migration plan needs, and exactly the gap that surprises teams the day their "DR region" turns out to hold zero usable snapshots.
• TTL deletes snapshots too, and deleting the wrong way orphans them. A long-retention Backup keeps its underlying volume snapshots alive for that entire window — a real storage cost. And running kubectl delete on a Backup object instead of velero backup delete removes the record but leaves the snapshots behind, quietly billing forever.
• Crash-consistent isn't the same as application-consistent. A plain volume snapshot of a running database is equivalent to pulling the power cord — survivable for many engines, not a guarantee for your only copy of production data. Stateful Workloads & Database Operators covers what a real backup strategy needs once an operator, not a cron job, owns the reconcile loop; Velero's pre/post hooks are how you bridge the gap when it doesn't.

Velero vs. the alternatives

☺ Like you're 10: A few other boxes and helpers exist. Some are free, some come with a bill and a fancy dashboard, and one of them isn't really a backup at all.

These are mostly complementary layers, not rivals — a serious operating model runs more than one at once, which is exactly what Best Practices & Operating Model recommends: etcd snapshots for the control plane, Velero for namespaces and volumes, and the database's own native backup for the handful of datastores where that matters most.

Velero isn't a named domain on the CKA blueprint — but the judgment underneath it is: PersistentVolume reclaim policy, RPO/RTO thinking, and why an untested restore is a hope rather than a plan all sit inside the storage domain and Anti-Patterns & Pitfalls. For the deeper platform-engineering treatment — self-service backup APIs, quiesce-hook nuance, and where backup sits in a platform's reliability story — Platform Engineering's own Velero tool page goes considerably further than this page needs to, and the wider Golden Kubestronaut ladder those platform-engineering certifications belong to is mapped on the sibling Golden Astronaut course.