Changelog

App→app networking for any TCP protocol

Reach a peer's raw TCP port by name — postgres, redis, any protocol — not just HTTP.

• --internal-l4 + crucible app create db --internal-port 5432. With the daemon's --internal-l4, an app that declares an internal port gets its own stable per-app VIP, and a granted peer reaches it at <app>.internal:PORT over a blind byte splice — any protocol crosses untouched and TLS passes straight through, so a client speaks the service's native wire protocol (e.g. psql "host=db.internal sslmode=verify-full") end to end. Everything the HTTP path already had still applies: default-deny (--can-call), wake-on-connect (a scale-to-zero database wakes on the first connection), and peer isolation (the VIP is the only path). See apps.
• Only declared ports. A peer can reach a VIP only on a port the app declared — never an arbitrary host service. Ports carry a protocol: --internal-port 5432 (or /tcp) is a raw splice; --internal-port 80/http routes that port through the L7 proxy. The assigned address shows in app get as internal_vip; L4 connections and bytes are on /metrics (app_internal_l4_connections_total{outcome}, app_internal_l4_bytes_total).
• Fair and bounded. A per-app connection cap under a global ceiling keeps one grantee from starving the mesh, a per-source rate limit bounds connection churn, and the firewall opens a VIP only for the exact ports declared. scripts/smoke_internal_l4.sh proves the raw-TCP splice (a redis PING/+PONG), default-deny, undeclared-port refusal, wake-on-connect, and peer isolation on real KVM.

Hardening

Pre-1.0 hardening. Fuzzing the v0.9.3 incremental-backup parsers found two ways a crafted backup could crash the daemon on restore — both fixed.

• Crash-safe backup restore. An out-of-range block index in an imported .delta, and a corrupt manifest header, could panic the daemon (negative slice / unbounded allocation). The delta and manifest readers now bound the block size, index, image size, and hash count; the crash inputs are kept as regression seeds. Reachable only with the default-deny volume_backup op, but a crash is a crash.
• Go 1.25.12. The toolchain is bumped to clear 29 standard-library security advisories the build was exposed to — no code change, and govulncheck now reports zero vulnerabilities in called code.

Incremental backups

Back up only what changed — a base full plus a chain of small deltas.

• crucible volume backup db --incremental (or --parent <id>) records only the blocks changed since a parent backup. volume restore --from <tip> reassembles the whole chain — base + every delta — and verifies the reconstructed image block-for-block, so a corrupt or missing link is caught. See backups.
• Encryption-transparent. A delta hashes each 1 MiB block and ships only the ones that differ; on an encrypted volume those blocks are the LUKS ciphertext, so nothing decrypts. backup ls shows KIND/PARENT; a parent can't be deleted while a child depends on it; incrementals export/import off-host like fulls.

Cold app stop / start

A desired-state stop/start, and the recipe to grow a running app's volume.

• crucible app stop <name> / app start <name>. A cold stop/start — stop destroys the instance and detaches its volume (unlike sleep/wake, which snapshot and keep the single-writer guard held), retaining the app spec; start boots a fresh instance that re-attaches the volume at its current size. POST /apps/{name}/stop + /start, SDK StopApp/StartApp, MCP app_stop/app_start.
• The grow recipe. So growing a running app's volume is app stop db → volume grow pgdata --size 20G → app start db. See apps.

Grow a volume

Enlarge a volume in place, data untouched — no more backup → restore-to-a-bigger-one → redeploy.

• crucible volume grow <name> --size 20G grows a volume's backing store and its ext4 filesystem to the new total size. Grow-only (a size at or below the current one is rejected — ext4 can't shrink online). Encrypted volumes grow too: the LUKS container, its mapping, and the ext4 are all resized in one step. See volumes.
• Detached-only. The volume must be detached — a 409 otherwise, because a snapshot-slept volume's guest has its device size pinned by the snapshot until it next boots. The shared resize runs e2fsck first, so a volume detached from a hard-killed guest resizes cleanly. POST /volumes/{name}/grow, SDK GrowVolume.

Guest metrics scrape

A workload's own metrics — a database's pg_stat_* / Redis INFO, or any app's Prometheus endpoint — folded into the daemon's /metrics.

• Scrape a guest /metrics. crucible app create db --metrics-port 9187 points the daemon at a Prometheus endpoint inside the guest (a postgres_exporter, redis_exporter, or the app itself); it scrapes on --guest-scrape-interval and re-exposes the series on its own /metrics + OTLP with app/instance labels — so a postgres_exporter's pg_stat_database_blks_hit sits next to the daemon's metrics. DB-agnostic; the exporter is a guest process. See observability.
• Scale-to-zero aware. A slept app is never scraped and a scrape never wakes it (the daemon dials the guest directly, not the wake forwarder); crucible_guest_scrape_up drops to 0 while asleep — you only pay for insights while it runs.
• Bounded. Each scrape is body/series/timeout-capped, so a runaway exporter can't flood the daemon or fail the endpoint.

Encryption key management

Multiple encryption keys, and key rotation that re-encrypts no data.

• Keyring. More than one key (via CRUCIBLE_VOLUME_KEY_<ID> env or --volume-key-dir), so different volumes can use different keys; --volume-default-key picks the one for new volumes, and volume ls shows each volume's KEY. See encryption.
• Rotate without re-encrypting data. crucible volume rewrap <name> --to-key <id> (and --all --from-key <id>) re-wraps a volume's key under a different key — the key that encrypts the data never changes, so there's no cryptsetup run, no data movement, and no downtime, even on a live volume. Gated by a default-deny volume_key op.
• Reload, retire, audit. crucible volume keys reload swaps the keyring in without a restart (refusing to drop a key a volume still uses); every key operation is audit-logged with names and key ids only, never key material.

Encryption at rest

Each persistent volume can be its own encrypted container with its own key, so its data can be crypto-shredded by destroying that key.

• Per-volume encryption. With a master key (--volume-encrypt-key-file or CRUCIBLE_VOLUME_KEY), crucible volume create pgdata --encrypt makes the volume a LUKS2 container (aes-xts-plain64, AES-256-XTS); --volume-encrypt encrypts every new volume by default, --no-encrypt opts one out. The per-volume key is sealed under the master key and stored in the record — never in the clear. See encryption.
• Transparent to the guest, encrypted while asleep. The daemon opens the container to a decrypted device and stages that device node into the VM's chroot under the jailer (never the ciphertext file), so the kernel does the crypto and the snapshot/wake pager pays no per-page cost. A scale-to-zero app closes its device on sleep and re-opens it on wake — ciphertext at rest even while idle.
• Crypto-shred. crucible volume shred <name> destroys the keyslots and wrapped key so the data is permanently unrecoverable. Backups carry the wrapped key; volume restore re-wraps it under the new name. Protects a stolen/seized disk (the AWS-EBS model), not a compromised host root.

Secrets

Sensitive config out of the cleartext app spec and into an encrypted store.

• Encrypted secret bundles — a secret is a named key→value bundle (a whole .env becomes one bundle), sealed with AES-256-GCM (the bundle name as AEAD additional data) in a dedicated store. Manage it write-only: crucible secret set web-env --from-env-file .env / secret ls / secret rm, and PUT/GET/DELETE /secrets/{name} gated by a default-deny secret op — no endpoint ever returns a value. See secrets.
• envFrom injection — app create --secrets <bundle> (or --secrets-from .env) injects every key of a bundle as an environment variable at boot, so the app spec, API, backups, and logs carry only the bundle name, never the value. Fixes the plaintext---env-in-the-database leak.
• Opt-in master key — --secrets-key-file or CRUCIBLE_SECRETS_KEY (no key ⇒ secrets disabled, no silent plaintext fallback); the store rides admin backup as ciphertext, with the key excluded.

App lifecycle events

The app activity timeline — and the exact sleep/wake timing that makes usage accounting precise.

• Event stream — created / phase_changed (booted / slept / woke / crashed, carrying from/to and wake_latency_ms) / health_changed / domain_* / deleted, on GET /events?since=&app= (cursor-follow, read-gated), crucible events -f / crucible app events, and as OTLP log records. See observability.
• Bounded + non-blocking — an in-memory ring (--events-buffer, default 1024); a slow consumer never back-pressures an app (drop-on-full). Emitted via a de-dup helper so no net phase change is missed and none is emitted twice.

Egress bytes

The fifth usage dimension: how much each app sends to the outside world.

• Per-app egress accounting — app_usage_egress_bytes_total counts each app's accepted external outbound bytes via a dedicated nftables accounting chain (a forward hook past the ct established short-circuit, so it sees every packet, not just the SYN). Intra-host DNS and app→app traffic are excluded. Folded into the durable per-app usage ledger as a per-instance delta, so a redeploy neither loses nor double-counts.

Usage metrics & cert status

Durable, restart-surviving usage accounting, and per-domain certificate health.

• Persistent usage metrics — a durable, cumulative per-app ledger across compute (vCPU-seconds awake), memory (MiB-seconds awake), storage (GiB-seconds), and requests, accrued on lifecycle hooks + a --usage-interval tick and surviving a daemon restart with no back-fill. Read via crucible app usage, GET /usage + /apps/{name}/usage (read-gated), and app_usage_* Prometheus/OTLP counters; a deleted app's final usage is retained.
• Per-domain TLS cert status — active / expiring / pending / failed / manual / passthrough on app domain ls, GET /apps/{name}/domains?detail=1, and app_cert_state / app_cert_not_after_seconds metrics.

TLS termination & custom domains

Real HTTPS deploys — the proxy terminates TLS and issues the certificates itself.

• Automatic HTTPS — with the proxy's TLS listener open and --acme-email (or --cert-dir) set, the ingress proxy terminates TLS with a certificate it issues and renews over ACME (CertMagic; --acme-ca production|staging, HTTP-01 + TLS-ALPN-01), and reverse-proxies plain HTTP to the guest. :80 serves the challenge and 301-redirects to HTTPS (--no-https-redirect opts out). See tls.
• Custom domains — crucible app domain add|rm|ls (globally unique; MCP app_domain_*; SDK AddDomain/RemoveDomain/ListDomains) attach your own hostnames alongside the generated <app>.<proxy-domain> name. On-demand issuance is gated to registered terminate-mode domains so a stray SNI can't burn a cert.
• Per-app mode — --tls-mode terminate (default) or passthrough; with no ACME/cert config, :443 stays SNI-passthrough as before.

Volume backups

A point-in-time backup of a volume, restorable to a new volume, plus a clone.

• volume backup / restore / clone takes a consistent, point-in-time copy of a volume and restores it into a new volume (never overwrites), or clones a volume straight into a new, independent one. Backups reflink (O(1)) when the backup dir shares the volume filesystem, a full byte copy otherwise. See backups.
• Consistency by state. A detached volume is copied directly; a slept app's volume is copied from its already-fsync'd backing file; a live database is frozen with fsfreeze (only the volume mount, never the guest root) for the instant of the copy, then thawed, with an agent-side watchdog that auto-thaws if the daemon fails to. A live backup needs a reflink filesystem (btrfs/XFS); on ext4 it is refused.
• --backup-dir (default <volume-dir>/backups): point it at another disk or mount to keep backups off-host. New MCP tools volume_backup and volume_restore (32 total).

Instant serverless-stateful

A volume-backed database now snapshot-sleeps and wakes in place, no cold boot.

• Snapshot wake for volume apps — a scale-to-zero postgres/redis on a volume sleeps by snapshotting its instance (RAM freed, the single-writer guard held, backing file host-fsync'd) and wakes with the volume re-attached: same instance + IP, ~170 ms (reflink), no cold boot and no WAL recovery — the DB process is already running in the restored memory. A wake after a daemon restart forks a fresh instance from the durable snapshot. See benchmarks.
• Durable while asleep — the volume backing file is host-fsync'd before the VMM stops, so a host crash while a volume app is asleep can't lose committed rows.
• Wake never fails — a restore failure falls back to a stop/start cold-create, so a wake always succeeds (just not instant that once).

Wake-on-TCP: serverless for any TCP service

Scale-to-zero for any self-hosted TCP service, not just HTTP.

• Wake-on-TCP — a scale-to-zero app that publishes a port (-p HOST:GUEST --min-scale 0 --idle-timeout <dur>) is fronted by an L4 forwarder that wakes it on the first TCP connection and sleeps it on inactivity, with no proxy in the path, protocol-agnostic. Any database, cache, or broker (postgres, mysql, redis, mongo, your own daemon) becomes a self-hosted serverless service. See serverless.
• Idle-connection reaping (--connection-idle-timeout) — closes a byte-idle pooled connection so a connection-pooled client still drains to zero and sleeps; the client's pool reconnects on its next query. This is what makes serverless work for pooled databases, not just connect-per-request clients.
• Connection-scoped mode (--keep-connections) — reaping off for pub/sub, LISTEN/NOTIFY, and streaming: awake while any client is subscribed, asleep when the last one disconnects.
• Guest /dev/fd — the guest init now provides /dev/fd + /dev/std{in,out,err}, so bash process-substitution entrypoints (like the postgres image's password init, initdb --pwfile=<(…)) work.

Persistent volumes

Data that outlives the sandbox — durable, fsync-honest block storage for stateful workloads.

• Volumes — crucible run --volume NAME:/path (and sandbox/app create --volume) attaches a named block device, formatted ext4 on first use and reattached by name thereafter. Backed by a sparse file under the daemon's new --volume-dir, attached with cache_type=Writeback so a guest fsync reaches the host — committed data survives a hard kill, destroy/re-create, redeploy, sleep, and a daemon restart.
• Volume lifecycle — volume create [--size 5G] / ls / rm (refused while attached), a durable bbolt record store, REST /volumes, and MCP volume_create / list_volumes / delete_volume (30 MCP tools).
• Volume-backed apps — single-writer, so a volume app redeploys destroy-then-boot and (at first) slept via stop/start; snapshot wake for volume apps landed in v0.6.2.

Observability

A running app is now legible — open standards out, routing delegated to your collector.

• Per-app metrics — request rate, latency, and status class per app on /metrics, plus lifecycle gauges (app_replicas, app_up, app_asleep, app_last_wake_latency_ms, …), with a reference Grafana dashboard.
• OTLP export — one --otlp-endpoint flag pushes the same series over OTLP (via an OpenTelemetry Prometheus bridge, so /metrics is unchanged) and streams app logs as OTLP records, honoring the standard OTEL_* env. Point it at any collector (Grafana/Tempo/Loki, SigNoz, Datadog, Honeycomb).
• Profiling + packet capture — --pprof-listen serves Go pprof for the daemon; sandbox capture / app capture streams a host-side pcap (no in-guest tcpdump, so distroless/scratch works), gated by a default-deny capture scoped-token op.
• MCP tools 24 → 27.

Reliability & isolation hardening

Close the sharp edges scale-out and app→app surfaced.

• No orphaned VMs — a rolling app update's draining instance is always reaped, even when the app is deleted, updated again, or slept mid-drain. An asleep app now truly runs zero VMs.
• Agent-fresh image cache — converted images are keyed by the injected guest agent's digest, so a daemon upgrade re-converts instead of booting a stale agent (the cause of wake failing on a previously-cached image).
• Publish + app→app coexist — a published host port (-p 80:80 / -P) no longer clashes with the <app>.internal VIP on the same port (SO_REUSEPORT + a one-owner-per-port registry).

Scale out

An app runs multiple replicas behind the proxy, forked warm from a snapshot in milliseconds.

• Horizontal scaling — app create <app> --min-scale N runs N warm replicas, each forked from a golden snapshot of the healthy primary (clone-safe: distinct machine-id + IP). The reconciler self-heals the fleet.
• Load balancing — power-of-two-choices least-request across live instances, with a slow-start ramp and passive outlier ejection. External and <app>.internal traffic share the one balancer.
• Autoscaling — --max-scale M --target-concurrency C scales between the floor and M on request concurrency (fast up, slow down); composes with scale-to-zero.

App-to-app networking

Deploy web + backend as separate apps and let them talk.

• Reach by name — an app calls another at http://<app>.internal/, routed through the ingress proxy VIP; the call inherits wake-on-request (a scaled-to-zero callee wakes) and per-sandbox isolation holds.
• Default-deny — app create web --can-call backend declares allowed calls; ungranted calls get 403 (proxy) / NXDOMAIN (DNS). Experimental, off by default (--internal-networking).

Scale to zero

• Sleep / wake — app sleep/wake, or app create --idle-timeout <dur> --min-scale 0: an idle app snapshots to ~zero RAM and wakes in place on the next request through the proxy in under a second (same IP + identity, clock stepped to now). A slept app survives a daemon restart.

Private registries

• Registry credentials — crucible registry login <host> stores a per-registry credential on the daemon so run, app create, and an app's re-pull on restart can fetch private images (Docker Hub, GHCR, GitLab, Quay, self-hosted). Gated by a registry scoped-token op; never reads ~/.docker/config.json. Plus one-shot run --registry-auth for CI.

Zero-downtime updates + operate by name

• Rolling app update — boot the new instance, gate on readiness, flip the route, drain the old; a failed update keeps the old instance serving, so the cutover drops nothing.
• Operate by name — app exec/logs/shell (and MCP app_exec/app_logs) resolve to the app's current instance per call, so they keep working across a self-heal or redeploy.

Reach an app by name

• Ingress proxy — a daemon-owned listener routes inbound traffic to an app by name (web.<domain>) instead of a fixed host port, following the app across self-heal and redeploys. L7 host routing + optional L4 SNI passthrough.
• Plus app update (replace the spec + redeploy) and image HEALTHCHECK seeding (an app with no --health inherits the image's healthcheck).

Apps you can actually deploy

Real config and real egress for the durable apps v0.4.0 introduced — across app create, run, and sandbox create.

• App env — -e/--env KEY=VALUE delivers config to the entrypoint (image ENV < your --env).
• Real egress (A6) — --net-full-egress (reach any public host) and --net-allow-cidr 203.0.113.0/24 (reach IP literals in a public prefix), for a workload you deploy yourself. Public-hosts-only, no exceptions: metadata/link-local (169.254.169.254), RFC1918, CGNAT, and reserved ranges are always dropped — the nft guard is unit-tested to agree with the DNS-layer SSRF filter. Gated by a net_full_egress scoped-token grant so a hostname ceiling can't be bypassed.
• Exec health checks — --health-cmd '<command>' runs a command in the guest (exit 0 = healthy), joining http/tcp.
• Publish the image's ports — -P/--publish-all publishes every port the image EXPOSEs (guest N → host N).

The MCP create_app/create_sandbox/run tools gained the matching arguments.

Durable, self-healing apps

Promote a workload to a named app the daemon manages over time — the answer to "my sandbox died on a daemon restart." See apps.md.

• Durable apps — crucible app create <name> --image … -p H:G --restart always --health http:PORT[:PATH]. The daemon keeps a healthy instance of the app, restarts it on failure with exponential backoff + a crash-loop guard, health-checks it (http/tcp), and — the headline — re-creates it from persisted desired state after a daemon restart or host reboot (desired-state reconcile, not live-VM re-attach; a bbolt control-plane store + a reconcile loop).
• Full surface — app ls|get|rm|logs|exec|shell, REST /apps, the Go SDK (CreateApp/ListApps/GetApp/DeleteApp + an App handle), and four MCP tools (create_app/list_apps/get_app/delete_app), bringing the MCP surface to 19 tools.
• The ephemeral sandbox primitive is unchanged; forks stay ephemeral by design.

Fork with port publish

• crucible fork -p HOST:GUEST — publish a host port on a fork (docker run -p semantics for copies): fork a running server onto its own port. The fork API takes an optional JSON body ({count, publish}); publishing requires count 1 since host ports are exclusive. Go SDK Fork gained variadic publish mappings (source-compatible).

The SDK foundation

The typed client became a public, dependency-free Go module, and the whole REST contract now fans out from one drift-guarded OpenAPI spec.

• Public Go SDK — github.com/gnana997/crucible/sdk, versioned independently as sdk/vX.Y.Z: package crucible (client + handles), sdk/api (DTOs), sdk/wire (frame codec). Typed errors, Page[T] lists, SDK-owned Identity. The CLI/TUI/MCP now run on it.
• Interactive exec over WebSocket — GET /sandboxes/{id}/exec + upgrade: the cross-language transport (fetch-style stacks can't speak the hijacked stream). Same frame protocol either way.
• The wire protocol, specified — wire.md plus recorded conformance fixtures, so an SDK codec in any language is buildable and testable with no daemon and no KVM. Generated TypeScript + Python types from the spec, with a CI drift guard.

Drop your code in and run it

• crucible cp — push a local file or directory into a running sandbox as a tar stream (no image build, no Dockerfile), path-escape safe. Plus MCP write_files / read_file.
• TUI live logs view, one-command Linux install that provisions the guest kernel, and a mirrored vmlinux release asset.

The safe docker run + cross-platform client

• OCI image boot — crucible run <image> boots an unmodified image's entrypoint in a microVM; crucible build converts a Dockerfile (daemon stays Docker-free); publish host ports with -p.
• Interactive shell (crucible shell, no PTY), --disk sizing, top-level stop/rm, durable logs, an MCP server, and scoped/policy API-key auth.
• The CLI, TUI, and mcp serve cross-compile to macOS and Windows and drive a remote Linux daemon; a reworked one-line installer for the daemon and client-only installs.

The core runtime

Firecracker microVMs under jailer, snapshot & fork with lazy userfaultfd memory, clone-safety (per-fork RNG reseed + machine-identifier rotation), per-sandbox networking (netns / veth / nftables / DHCP / DNS proxy) with default-deny egress, cgroup v2 quotas, a durable reconciled registry, a Prometheus /metrics endpoint, and native rootfs profiles.

See CHANGELOG.md for the full per-release detail.