Benchmarks

Reproducible measurements from crucible-bench, a harness that drives a real daemon through the sdk Go package (the same typed client the CLI, TUI, and MCP server use). It reports latency distributions, fork fan-out scaling, the lazy-memory efficiency of fork, and sandbox density.

Read these as one machine's numbers, not a spec. They come from a single host, one sandbox size — enough to show the shape of the runtime, not a cross-hardware guarantee. Everything you need to re-run is at the bottom.

The filesystem under --work-base is the biggest lever

Fork clones the per-child rootfs copy-on-write. On a reflink filesystem (btrfs / XFS reflink=1) that clone is O(1) — extents are shared, nothing is copied. On ext4 (no reflink — still the most common default) it's a full byte-copy of the rootfs. Same box, same daemon, same 1 GiB rootfs; only --work-base/--chroot-base differ:

If you fork a lot, put --work-base on btrfs or XFS. The rest of this page reports both, so you can see the floor (ext4) and the ceiling (reflink). Note that wake barely cares about the filesystem (2.0×) — it restores against the live rootfs with lazy memory and copies nothing on the timed path — while fork/fan-out care enormously (up to ~18×), because each ext4 child byte-copies a full ~1 GiB rootfs. Marginal RAM per fork is similar on both filesystems (~12–16 MiB) — guest pages are shared via userfaultfd regardless of the disk; reflink's win is wall-time and I/O, not memory.

Setup

Latency

Fork is ~2.6× faster than a cold create even on ext4, and ~8× faster on reflink — pay the ~1 s boot once, then branch cheaply. Wake beats a cold create ~8× (125 ms vs 1.03 s on btrfs) and is nearly storage-independent (see below). Exec overhead is ~2–3 ms (a vsock roundtrip), essentially filesystem-independent. Snapshot's median is ~0.5 s with occasional multi-second tails from writing the 512 MiB memory image through writeback.

Wake (scale-to-zero)

The v0.5.0 headline: a slept app costs ~zero RAM and wakes on the next request. The product number is proxy wake — an HTTP request hits a slept app through the ingress proxy and is served, with the wake (restore + reseed RNG + step clock) happening inline while the proxy holds the request. nginx:alpine, proxywake phase:

Wake is ~8× faster than a cold create (125 ms vs 1.03 s, btrfs) and, unlike fork, only mildly depends on the filesystem (2.0×). That's because wake restores in place against the live rootfs with lazy (userfaultfd) memory — no rootfs clone, no memory copy on the timed path, cost O(working set) not O(guest RAM). The ext4 penalty is only the page-cache pressure left by the untimed sleep-snapshot. Either way it's comfortably sub-second.

There is also a wake phase (--phases wake) that times the bare sandbox-level restore-in-place mechanism; it needs a rootfs whose guest agent has /wake (OCI images used by proxywake always do). Every JSON result carries an env block — host, CPU, kernel, memory, and a live FICLONE reflink probe of --reflink-path — so nobody compares an ext4 number against a btrfs one.

Stateful (volume) wake — v0.6.2

Before v0.6.2 a volume-backed app woke by cold boot: destroy the instance, re-create it, boot the service, run recovery — seconds for a database. v0.6.2 makes a volume app snapshot-sleep and restore in place exactly like a stateless one — the running process comes back from the snapshot with its volume re-attached, nothing copied on the timed path. redis:alpine on a volume, volumewake phase (app sleep → app wake → running):

The volume adds no meaningful overhead over a stateless wake — ~170 ms pure restore on reflink, the same ballpark as the 125 ms non-volume proxy wake — and, like every wake, it barely depends on the filesystem (1.3×), because the restore reads the live rootfs + volume drive with lazy (userfaultfd) memory instead of copying them. So a self-hosted serverless postgres or redis comes back in well under a quarter-second on the first connection after it slept, with no cold boot and no WAL recovery: the database process is already running in the restored memory, attached to its volume.

Fork fan-out

Forking N children from one snapshot in a single call. Per-child cost falls with batch size as fixed per-fork overhead amortizes — and reflink pulls far ahead as N grows:

On ext4 each child copies a full 1 GiB rootfs, so per-child cost plateaus around ~285–340 ms and the 128-way batch moves ~128 GiB of disk — it took 36 s vs 2 s on btrfs. On reflink the clone is O(1), so per-child cost keeps falling as fixed overhead amortizes (173 ms → 16 ms), and throughput climbs to ~63 forks/s at 128-way.

Memory efficiency

The lazy-userfaultfd payoff: guest RAM is served on demand from the snapshot's memory file, so forks page in only their working set instead of each copying 512 MiB. Forking 64 children from one warm snapshot:

Marginal RAM per fork is comparable on both filesystems (~13–16 MiB) — the guest's memory is shared read-only via userfaultfd on either disk, so what each child adds is just its faulted-in working set plus copy-on-write divergence. btrfs is slightly lighter (no page-cache pressure from rootfs byte-copies) and, crucially, ~26× faster wall-time (1.3 s vs 33 s) because ext4 also moves 64 GiB of rootfs I/O that reflink skips entirely.

Density (reflink)

Forking toward a live-sandbox target on btrfs and watching free RAM: this run reached 320 concurrent microVMs — from ~11.6 GiB free at the start to 7.9 GiB free at 320 live, ~11 MiB marginal RAM per running VM (its faulted-in working set) — before the first failure.

The stop was not RAM exhaustion (7.9 GiB was still free) but a per-fork restore-readiness timeout: inside a 64-wide fork batch at that concurrency, one child's guest agent didn't answer /healthz within the restore deadline. So 320 is this run's first-failure point, not a hard ceiling — a longer agent-ready deadline (or a smaller fork batch) would push the peak higher.

On reflink, density is bounded by RAM and restore concurrency. On ext4 it's disk-bound instead — each fork writes a full ~1 GiB rootfs, so 320 would move ~320 GiB — which is why this phase is reflink-only (scripts/bench_reflink.sh FS=ext4 skips density by default). Use reflink for density.

Reproduce

The one-command runner sets up the target filesystem (a btrfs loopback, or a dir on the ext4 root), stages the rootfs + work dirs on it, starts a daemon, runs crucible-bench (including the end-to-end proxywake number), stamps the environment + a live reflink probe into the JSON, and tears down:

make build && make bench                      # builds ./crucible and ./bin/crucible-bench

sudo FIRECRACKER_BIN=… JAILER_BIN=… KERNEL=… ROOTFS=… \
  FS=btrfs scripts/bench_reflink.sh           # reflink numbers → bench-btrfs-*.json
sudo … FS=ext4  scripts/bench_reflink.sh      # ext4 numbers   → bench-ext4-*.json

Knobs: DENSITY, SAMPLES, PHASES, IMG_SIZE, KEEP=1 (keep the loopback for repeat runs). Add volumewake to PHASES for the volume-app snapshot-wake number (the daemon runs with a --volume-dir). To drive an existing daemon by hand instead, point crucible-bench --addr … --reflink-path <daemon work-base> and add --phases proxywake --proxy-addr … --proxy-domain … for the wake number. crucible-bench --help lists every knob.

The end-to-end serverless story — a postgres on a volume, snapshot-wake vs cold-boot, timed from the wake trigger to a query answering — is a separate one-command run: sudo … scripts/bench_serverless.sh (boots its own daemon, needs psql on the host).

Mass wake (the herd)

Scale-to-zero packs more sleeping apps than host RAM, so "everything wakes at once" is a designed-for scenario. sudo … N=20 MEM=256 scripts/bench_masswake.sh boots N scale-to-zero apps, drains them with app sleep --all, then fires N concurrent wakes and reports the wake-latency distribution (p50/p90/p99/max, both client end-to-end and daemon-measured), how many wakes the --wake-min-free-mib floor deferred to a clean 503 + retry (graceful degradation, not failure), and the host MemAvailable low-water mark. It only fails if an app never serves even after a sequential retry. Knobs: N, MEM, WAKE_MIN_FREE, KEEP=1.

Encryption overhead (v0.8.0)

Volume encryption at rest is per-volume LUKS (AES-256-XTS). What does it cost? scripts/bench_encryption.sh A/Bs a plaintext vs. an encrypted volume across the paths that have a real cost, on a reflink btrfs --work-base (so wake isn't byte-copy-inflated) and an AES-NI host. Single box, same caveat as the rest of this page.

The shape:

• Wake pays a fixed one-time cost — one luksOpen when the volume re-attaches on wake (mostly the cryptsetup process spawn, not the KDF, which is a fast keyslot). It's ~45 ms added, not a proportional tax, and the app is still up in well under a quarter-second.
• Encrypting the memory snapshot is free. With --work-base itself on dm-crypt (the full "nothing plaintext on disk" setup — WORK_ENCRYPT=1 in the bench), the wake baseline is unchanged (~114 ms either way): device-mapper decrypts pages as the snapshot pager faults them in, so there's no per-page cost. Encrypt --work-base for complete at-rest coverage without a wake penalty.
• Steady-state I/O is not cipher-bound. AES-XTS runs at ~8–9 GB/s here — far above any real disk or guest→host rate — so dm-crypt never bounds throughput. (The bench's guest-write row is dominated by the host Writeback cache, not the cipher; read it as guest→host bandwidth, not encryption cost.)
• Secrets injection is free — decrypting a bundle and merging its keys at boot is lost in the noise of VM startup.

Reproduce (needs cryptsetup, mkfs.btrfs, a pullable nginx:alpine):

sudo … scripts/bench_encryption.sh                  # per-volume encryption on a reflink work root
sudo … WORK_ENCRYPT=1 scripts/bench_encryption.sh   # also encrypt --work-base (memory snapshot at rest)

The WORK_ENCRYPT=1 run also confirms the startup advisory goes silent on an encrypted --work-base (and fires on a plaintext one). Knobs: R_WAKE, IO_MIB, SECRET_KEYS, VOL_SIZE, MEM, PORT_BASE, KEEP=1.