Observability

crucible is a telemetry source, not a pipeline. It emits open standards — Prometheus /metrics today, OTLP (metrics, logs, traces) as it lands — and delegates routing and fan-out to the ecosystem (an OpenTelemetry Collector, Vector, or Grafana Alloy). One OTLP export reaches Grafana/Tempo/Loki, SigNoz, Datadog, Honeycomb, and the rest without any vendor-specific code in the daemon.

Metrics — GET /metrics

The daemon exposes a Prometheus endpoint (default on the API listener). Point a scrape at it:

scrape_configs:
  - job_name: crucible
    static_configs:
      - targets: ["<daemon-host>:7878"]

Per-app series

Labels are kept to a fixed, low-cardinality set — app, code (HTTP status class: 2xx…5xx), never a raw path or client IP. A request for an unknown host is not counted, so an attacker can't inflate labels.

Plus the existing global series: sandboxes_active, sandboxes_created_total, snapshots_active, fork_duration_seconds, snapshot_restore_duration_seconds, app_wake_latency_seconds (aggregate histogram), app_internal_requests_total.

Disk usage

Scale-to-zero trades RAM for disk: every sleeping app owns a snapshot set. Three global gauges make that visible, so density never silently becomes disk bloat:

All three are sparse-aware (allocated blocks, not logical file size): a lazily-faulted memory file or a mostly-empty volume counts what it actually occupies. Reflink-shared blocks are counted per file, so on btrfs/XFS the gauges report logical allocation, an upper bound on unique physical usage. The volume/backup series exist only when volumes are enabled (--volume-dir).

Retention contract behind snapshot_disk_bytes: a sleeping app owns exactly one snapshot set — each sleep supersedes and deletes the previous one — plus at most one golden template per scaled-out app generation, and deleting an app releases its snapshots. Growth in this gauge tracks the number of slept apps, not their sleep history.

Persistent usage metrics

Per-app usage counters that survive a daemon restart: unlike the Prometheus series above (which reset when the daemon does), these are a durable, cumulative ledger persisted alongside the app records. Read them to see how much an app has used over any window — take two readings and subtract.

Four dimensions accrue per app:

Read them three ways, all from one durable source:

• crucible app usage [<name>] — a table (or -o json), including retained records for deleted apps (a deleted app's final usage is kept so you can still read it).

• GET /usage and GET /apps/{name}/usage — the same, as JSON. Cumulative values plus a snapshot_unix_nano, so a reader reconciles readings across restarts and scrape gaps. Gated by the read scoped-token op.

• Prometheus (and OTLP, via the bridge below):

  Metric	Type	Labels
  app_usage_compute_vcpu_seconds_total	counter	app
  app_usage_memory_mib_seconds_total	counter	app
  app_usage_storage_gib_seconds_total	counter	app
  app_usage_egress_bytes_total	counter	app
  app_usage_requests_total	counter	app, code

Accrual is checkpointed on a tick (--usage-interval, default 60s) and at each lifecycle transition, so an unclean crash loses at most one interval. A daemon restart does not back-fill the downtime — the app wasn't serving, so it accrues nothing for that gap.

TLS certificate status

For a TLS-terminating app, per-domain certificate state is both a metric and an API field — so a domain whose DNS isn't pointed at the host (issuance failing) is visible, not silent:

state is one of passthrough, pending, active, expiring, failed, manual. The same detail is on GET /apps/{name}/domains?detail=1 and crucible app domain ls — see tls.md.

Reference dashboard

Import docs/observability/grafana-dashboard.json into Grafana (Dashboards → Import → upload JSON, pick your Prometheus source). It charts RPS and 5xx ratio per app, request-latency percentiles, replicas (desired vs ready), the fraction of the fleet asleep, wake-latency p95, and disk usage (snapshots / volumes / backups).

Guest metrics — scrape an app's own /metrics

The metrics above are the daemon's host/VM view of a workload. To also surface a workload's own metrics — a database's pg_stat_* / Redis INFO, or any app's Prometheus endpoint — point the daemon at a /metrics port inside the guest:

crucible app create db --image my-postgres-with-exporter \
  -p 5432:5432 --min-scale 0 --idle-timeout 30s \
  --metrics-port 9187          # a postgres_exporter listening on :9187 in the guest

The daemon scrapes that endpoint on its interval (--guest-scrape-interval, 15s by default) and folds the series straight into this /metrics (and OTLP), so a postgres_exporter's pg_stat_database_blks_hit shows up right next to the daemon's own metrics — with app and instance labels added:

pg_stat_database_blks_hit{app="db",instance="sbx_…",datname="app"} 1.2345e6
crucible_guest_scrape_up{app="db",instance="sbx_…"} 1

The exporter is a process inside the guest (a postgres_exporter, redis_exporter, or the app itself) — the daemon is DB-agnostic, it only federates whatever Prometheus text the guest exposes. The port is not published; the daemon reaches it host-side over the guest's private link (the same reach as health checks), so nothing is exposed externally.

Two properties matter for scale-to-zero:

• A slept app is never scraped — and a scrape never wakes it. The daemon only scrapes awake, routable instances and dials the guest directly (not the wake-on-connect forwarder). crucible_guest_scrape_up goes to 0 while the app sleeps, and the scraped series drop out — so you only pay for insights while the workload is actually running.
• A guest can't flood the daemon. Each scrape is capped (--guest-scrape-max-body, --guest-scrape-max-series, --guest-scrape-timeout); over a cap the scrape is dropped, crucible_guest_scrape_up reports 0, and crucible_guest_scrape_samples shows how many series the last scrape ingested.

This is the raw signal; building dashboards or a query/wait-event analysis on top of it is a downstream job for your metrics stack.

Profiling — --pprof-listen

For profiling the daemon itself (CPU, heap, goroutines):

crucible daemon … --pprof-listen 127.0.0.1:6060
go tool pprof http://127.0.0.1:6060/debug/pprof/heap

Off by default. pprof exposes process memory, so bind loopback (or protect the port) — the daemon warns on a non-loopback bind.

OTLP metric export

The daemon can push the same /metrics series over OTLP to any OTLP backend or your own Collector/Vector/Alloy — no metric is redefined; an OpenTelemetry Prometheus bridge pulls the registry and exports it. One flag turns it on:

crucible daemon … --otlp-endpoint http://collector:4317        # gRPC (default)
crucible daemon … --otlp-endpoint http://collector:4318 --otlp-protocol http

• --otlp-protocol grpc|http, --otlp-headers k=v,k=v (auth/routing), --otlp-insecure (plaintext).
• Standard OTEL_EXPORTER_OTLP_* and OTEL_RESOURCE_ATTRIBUTES / OTEL_SERVICE_NAME env vars are honored natively (flags override them), so if you already run OpenTelemetry it just works.
• The export carries a resource of service.name (default crucible), service.version, and host.name, plus your OTEL_RESOURCE_ATTRIBUTES.
• Off by default. Setup failures are logged and skipped — /metrics keeps serving regardless.

/metrics and OTLP are two views of one registry; use either, or both.

OTLP log export

When --otlp-endpoint is set (and --log-dir is on), the daemon also streams app logs over OTLP — every durable log line becomes an OTel log record with:

• service.* resource + crucible.app.instance (the instance id),
• log.source (service | exec) and log.stream (stdout | stderr | event),
• the original timestamp; stderr maps to severity WARN, else INFO.

It taps the log store's best-effort fanout — a slow OTLP backend can never back-pressure the app (records drop rather than block). Disable with --otlp-logs=false (metrics-only). Logs remain locally readable via crucible logs / crucible app logs -f regardless.

App lifecycle events — GET /events

A stream of app lifecycle transitions — created, phase_changed (booted / slept / woke / crashed), health_changed, domain_added / domain_removed, updated, deleted. Where /metrics is a numeric snapshot, this is the timeline: what happened, in order, with exact timestamps. A control plane renders an activity feed from it and — because phase_changed carries the exact sleep/wake moments — computes precise awake-intervals for usage accounting that a 60s metrics poll can only approximate.

crucible events -f                 # tail all apps
crucible app events web -f         # tail one app

2026-07-15T20:52:01Z  web  created
2026-07-15T20:52:04Z  web  phase pending→running   (reconcile)
2026-07-15T21:07:36Z  web  phase running→asleep    (sleep)
2026-07-15T21:09:12Z  web  phase asleep→running    (wake)

• GET /events?since=<seq>&app=<name> returns a batch of events after the cursor plus the current max cursor; poll with that cursor to follow (client side, like logs). read-gated. app= filters to one app.
• OTLP: each event is also pushed as a structured OTel log record (event.type, crucible.app, phase.from/phase.to, event.seq) when OTLP is configured — so an OTel-native consumer gets them without polling.

The stream is an in-memory ring (--events-buffer, default 1024): a consumer offline longer than the ring loses old events, but usage totals stay correct (reconcile against /usage) — the ring is a best-effort activity signal, not an event-sourcing log. phase_changed de-dups: only an actual phase change emits, so a steady app is quiet.

Traces over OTLP (coming)

Trace export is the next milestone (deploy / sleep / wake / proxy spans).