Volume backups

A volume keeps a database's data safe across a sandbox's life, a hard kill, a redeploy, sleep, and a daemon restart. A backup goes one step further: a point-in-time, consistent copy of a volume that you can restore into a new volume at any time. The same copy primitive also clones a volume, so you can fork a database for a preview or test environment.

crucible volume backup db                          # back up the "db" volume
crucible volume backup ls db                        # list its backups
crucible volume restore --from <id> --to db-copy    # restore into a NEW volume
crucible volume clone db db-preview                 # copy a volume directly

Consistency by state

A backup is only as good as the filesystem image it copies, so crucible picks the safe method automatically from the volume's current state:

For a live volume, crucible tells the guest agent to FIFREEZE the volume's filesystem (only that mount, never the root filesystem the agent runs from), copies the backing file while writes are held, then FITHAWs it. The freeze lasts only as long as the copy, and the agent auto-thaws on a watchdog if the daemon ever fails to, so a running database is backed up with no downtime and no risk of a stuck freeze. Restore and clone follow the same rules: a live source is frozen for the copy.

Live backups need a reflink filesystem

A live backup freezes the guest only for the duration of the copy, so that copy must be O(1). On a reflink filesystem (btrfs or XFS) it is an instant copy-on-write clone. On ext4 there is no reflink, so a copy is a full byte-for-byte read that could hold the freeze for seconds. crucible therefore refuses a live backup when the backup filesystem is not reflink-capable and asks you to sleep the app first (a slept or detached backup works on any filesystem). Put the backup directory on btrfs or XFS to back up live databases.

No downtime, measured

A live backup pauses a database only for the freeze, and only briefly. Measured on postgres:16-alpine on a btrfs volume (1 vCPU / 512 MiB, AMD Ryzen AI 9 HX 370), under 16 concurrent writers sustaining ~2,500 padded-row INSERTs per second, taking 12 live backups back to back:

The whole backup, freeze included, takes about 90 ms, and it barely moves with load because the copy is an O(1) reflink (pushing to 32 writers left it at ~105 ms). The freeze itself is a slice of that: an fsync of the volume plus the reflink. No transaction failed: a query that lands during the freeze waits for it and then completes, so the client sees a brief latency blip, not an error. That worst-case blip grows with write concurrency (more in-flight commits queue behind the freeze, then flush together on thaw), but it stays a pause, not an outage. At 32 concurrent writers the busiest query saw a few hundred milliseconds and still zero failures.

Reproduce with scripts/bench_backup.sh (WRITERS=N SAMPLES=M).

Where backups live

Backups are written under --backup-dir (default <volume-dir>/backups). Point it at a different disk, or at a mounted network or object store, for durability that survives the loss of the volume's own disk:

crucible daemon --volume-dir /var/lib/crucible/volumes \
                --backup-dir  /mnt/backups

Two trade-offs to know:

• A backup directory on the same filesystem as the volumes reflinks (O(1), cheap), but a same-disk backup does not survive that disk dying. It protects against an accidental rm, a bad migration, or application-level corruption you want to roll back from.
• A backup directory on another disk or host survives disk death and lets you move a volume to another machine, but the copy is a full byte copy (reflink cannot cross filesystems), and a live backup there is refused (not reflink-capable).

Native off-host targets (S3, rsync) and incremental backups are planned for a later release; today a --backup-dir on a mounted volume is the way to keep backups off-host.

Restore and clone

• volume restore --from <backup-id> --to <name> materializes a backup into a new volume. It never overwrites an existing volume, so a restore can never clobber live data. Attach the new volume to a sandbox or app as usual.
• volume clone <src> <dst> copies a volume straight into a new one, with no backup record in between. Use it to fork a database for a preview environment or a test run. The clone is fully independent: writing to it does not touch the source.

A restored or cloned volume mounts read-write in a guest and, if its source was not cleanly unmounted, replays its journal on first mount, exactly like any crash-recovered filesystem.

Incremental backups

A plain volume backup is a full whole-image copy. An incremental stores only the blocks that changed since a parent backup, so frequent backups stay cheap:

crucible volume backup db                       # a full (the base)
# … later, after more writes …
crucible volume backup db --incremental         # only the changed blocks, vs the latest backup
crucible volume backup db --parent <backup-id>  # … or against a specific parent

• A backup chain is a base full plus a run of incrementals, each relative to its parent. Restore the tip and the base and every delta are reassembled automatically — volume restore --from <incremental-id> --to <name> just works.
• The whole chain must be present to restore; deleting a backup that an incremental still depends on is refused (delete the children first).
• How it works. With no changed-block tracking in the stack, a delta is computed by hashing each fixed block (1 MiB) of the current image and shipping only those that differ from the parent's manifest of block hashes. On restore the reconstructed image is verified block-for-block against the tip's manifest, so a corrupt or incomplete chain is caught, not silently restored. It works on encrypted volumes too — the blocks are the LUKS ciphertext (AES-XTS is per-block), so nothing decrypts to compute a delta.
• Cadence and retention are yours (or your control plane's): the daemon gives the full/incremental primitives; deciding when to take a full vs an incremental, and how long to keep them, is policy on top.

backup ls shows each backup's KIND (full / incremental) and PARENT. Incrementals also export/import off-host like fulls — export each backup in the chain and re-import them base-first (pass --parent <local-id> for each delta), then restore the tip. See off-host backups.

Command reference

All are available over the REST API (POST /volumes/{name}/backups, GET /backups, POST /volumes/{name}/restore, POST /volumes/{name}/clone) and as MCP tools (volume_backup, volume_restore).

Acceptance: scripts/smoke_backups.sh covers a detached backup (loop-mounted to verify the data), a slept-app backup, a live backup via fsfreeze on a reflink filesystem, a restore into a new volume, and an independent clone; scripts/smoke_incremental_backup.sh covers a full → incremental chain (the delta is far smaller than the full), a tip restore that reassembles base + delta (data intact, plaintext and encrypted), the parent-delete guard, and an off-host chain export/import round-trip. See also volumes.md and serverless.md.

Off-host backups (export / import)

A backup under --backup-dir still dies with the box. To survive host or disk loss, a backup has to leave the host — and that is a job the daemon does not do itself: it stays provider-agnostic (no S3/GCS SDKs, no cloud credentials). Two thin verbs move a backup's bytes across the API so a control plane (or a cron script) can ship them to wherever it keeps backups:

# stream a backup off the host (gzip by default — the image is sparse)
crucible volume backup export <id> -w db.img.gz
crucible volume backup export <id> | aws s3 cp - s3://backups/db.img.gz   # piped

# later, on a fresh host: stream it back on, then restore to a new volume
crucible volume backup import --source db -f db.img.gz     # prints a new backup id
crucible volume restore --from <new-id> --to db-restored

Both are gated by the default-deny volume_backup scoped-token op — they move volume data across the boundary, so a read-only token can't. Export gzips by default (holes in the sparse ext4 image compress away over the wire); --raw streams the backing file uncompressed for a caller that dedups its own way. Import expects gzip unless --raw, and records the backup under this host's id (the backup lives here now). The bytes are a static, already-consistent point-in-time image, so export/import add no freezing — they just move the file v0.6.3 already made consistent.

For a self-hoster with no control plane, pointing --backup-dir at an NFS or S3-fuse mount is the simplest off-host path (no export needed); export/import are what a remote control plane uses to pull backups over the network. Efficient incremental / deduplicated off-host storage is left to that layer (e.g. restic or borg over the exported streams).

Acceptance: scripts/smoke_offhost_backup.sh runs the full loop — back up a volume, export it, delete the volume and its local backup, import the stream back, restore to a new volume, and confirm the data survived — plus the volume_backup-vs-read gate.

Daemon backup (crucible admin backup)

Volume backups protect your data; the daemon backup protects the daemon's knowledge of everything else — without it, a rebuilt host has volumes but no apps, tokens, or pull credentials. One command captures it all, hot, while the daemon keeps serving:

crucible admin backup                       # writes crucible-backup-<ts>.tar.gz
crucible admin backup -w cp.tar.gz          # explicit filename
crucible admin backup | ssh vault 'cat > crucible.tar.gz'   # piped = streamed

The archive holds up to five entries — each present only when its component is configured:

Each store's copy is individually consistent; the archive as a whole is not a cross-store transaction (the stores are independent, and skew between them is harmless). Volume data is deliberately not included — pair this with crucible volume backup, ideally with --backup-dir on off-host storage.

The endpoint (GET /admin/backup) is gated by the default-deny admin_backup scoped-token op: the archive carries usable registry secrets and the full token/policy state, so treat the file like a credential file (it is written 0600).

Restore procedure

Restore is deliberately a documented procedure onto a stopped daemon, not a command — laying store files under a live daemon invites corruption:

1. Stop the daemon (systemctl stop crucible).
2. Extract the archive and copy each entry to its configured path: app.db → --app-db, tokens.json → --token-file, volume-index.db → <volume-dir>/index.db, registry-credentials.json → --registry-store.
3. Restore volume data from your volume backups if this is a fresh host (volume restore needs the daemon up — do it after step 4).
4. Start the daemon.

On startup the reconciler treats the restored app records as desired state and re-creates every app that was running — a restore is not just records coming back, it is the apps healing themselves. Pre-disaster API keys work immediately (the token store is the same file).

Acceptance: scripts/smoke_daemon_backup.sh runs the full disaster: state created, archive taken, every store deleted, archive restored, and the app observed self-healing back to running with the old token still valid.