Observability specification: /_spindle/metrics and the fork-case counter

[hanthor]

Child of #19, which scopes M5's observability but does not specify it. SPEC §17.2 names what to measure and §18.3 gives the targets those measurements are supposed to falsify; this is the endpoint-and-instrument-level spec for both.

Nothing exists today: no metrics crate, no #[instrument], 29 tracing:: calls total. So this is greenfield, and the ordering matters — the interesting decisions are below, and one of them is the reason this issue outranks the rest of M5's remaining scope.

---

1. The fork-case counter is the falsification instrument, not a nice-to-have

SPEC §17.2 calls the class-1/2/3 split from §9.2 "the single most important health metric — case 3 should be near zero", and §18.3 turns that into a falsifiable target: case-3 fork resolutions < 0.1% of federated events.

That is the number the entire architectural bet rests on. If case 3 is not rare in the wild, "no state resolution on the hot path" is false and we need to know from a counter rather than from a user complaint.

It is also a hard dependency of two other issues, which is why this one should land first:

• M3: prove class-D fork handling with a Synapse interop rig #16 requires "assert the selected fast/bounded/full path via metrics" and "no test can pass by silently taking a more expensive path". That test cannot be written until the path is counted.
• Performance parity gate: measure Spindle against Tuwunel and Synapse under protocol workload #42 wants Complement-under-load to emit "runtime metrics, compared against prior runs".

So the first slice is not a /metrics endpoint in the abstract — it is this counter, plus the smallest endpoint that can expose it.

2. A metric that never moves is worse than a missing one

The same argument as surface.rs and the admin API's honest-advertisement rule, and it bites harder here. An endpoint that 404s is obviously absent. A gauge exported as a flat 0 is indistinguishable on a dashboard from a healthy system, and the operator finds out it was never wired during the incident it was supposed to catch.

Therefore: every exported metric ships with a test that proves it moves — drive the operation, scrape, assert the delta. A metric without one does not get exported. This is the acceptance criterion that keeps this issue from becoming decoration.

3. Per-room detail does not go in labels

§17.2 says "per-room and per-shard metrics", and a literal reading means a room-ID label. That is the classic Prometheus cardinality failure: a server with 10,000 rooms and six per-room series becomes 60,000 series that mostly never change, and the scrape cost grows with the room count rather than with traffic.

The split this proposes:

Question	Where it is answered
"How is the server doing?"	/metrics — aggregate histograms and counters, bounded label sets
"How is this room doing?"	The admin API (#83), which already has per-room endpoints and an authorization model
"Which shard is hot?"	/metrics with a shard label — bounded by shard_count, which is config

Bounded labels only: shard, fork case, federation destination (capped, with an other bucket — a hostile or misconfigured peer must not be able to mint series), request route from the router's own template rather than the raw path.

4. /metrics is an operational surface, not a public one

Scrape output names rooms indirectly, exposes traffic volumes and reveals which peers this server talks to. It must not be reachable from the open internet by default. Options, in preference order:

1. A separate listener bound to 127.0.0.1 by default ([metrics] bind = …), which is what most operators expect and what makes a reverse-proxy config obvious.
2. On the main listener behind the AdminActor guard, for deployments that cannot run a second port.

Proposed: (1) by default, (2) available, and off entirely unless configured — consistent with builtin_oidc and [auth.delegated] being opt-in.

---

What to export

From §17.2, in dependency order. Each row is a slice; each ships with the test from §2.

Slice 1 — the counter that matters

Metric	Type	Labels	Why
spindle_fork_resolutions_total	counter	case (1/2/3)	§18.3's < 0.1% target; #16's path assertion
spindle_events_appended_total	counter	origin (local/federated)	The denominator that turns the above into a rate
spindle_build_info	gauge	version	Standard; makes a dashboard able to say what it is watching

Plus the endpoint and the config to serve it.

Slice 2 — the hot path

Metric	Type	Labels
spindle_append_duration_seconds	histogram	durability
spindle_http_requests_total	counter	route, method, status
spindle_http_request_duration_seconds	histogram	route

§18.3's local-send p50/p99 targets become checkable here.

Slice 3 — the queues and the caches

Metric	Type	Labels
spindle_federation_queue_depth	gauge	destination (capped)
spindle_federation_transactions_total	counter	outcome
spindle_state_node_cache_hits_total / _misses_total	counter	—
spindle_sync_subscribers	gauge	—
spindle_sync_watermark_lag_seconds	gauge	—

This is the set #19's exit criteria name: "durability, federation backlog, sync watermark lag and unexpected fork paths".

Slice 4 — traces

OpenTelemetry spans for client request → executor → commit → fan-out, per §17.2. Deliberately last: traces are the most invasive change and the least useful without the counters to tell you which trace to go looking for.

---

Deliberately not in scope

• A metrics library shootout. Pick one, note it in an ADR, move on. The exposition format is the contract, not the crate.
• Bundled dashboards and alert rules — worth having, but they belong with the operator runbooks in M5: build admin, moderation, policy, and observability surfaces #19's last bullet, once the metrics they reference exist and have been watched for a while.
• Per-room label cardinality, per §3.

Acceptance

• The fork-case counter exists and M3: prove class-D fork handling with a Synapse interop rig #16 can assert the path taken from it — the specific dependency this issue exists to unblock.
• Every exported metric has a test that drives the operation and asserts the value moved. No metric ships without one.
• /metrics is off unless configured, and not on the public listener by default.
• Label cardinality is bounded by config or by a cap, never by room count or by a remote peer's choice of hostname.
• docs/ gains an operator page saying what each metric means and which SPEC target it validates — a metric nobody can interpret is not observability.

[hanthor]

Slices 1 to 3 are shipped and each metric has the test that proves it moves: the fork-case counter and its denominator, the build gauge, the append and HTTP histograms, the federation queue depth with the capped destination label, the sync subscriber and lag gauges. /metrics is off unless [metrics] bind is set, docs/metrics.md is the operator page, the alert rules and dashboard followed in #328, and every server has its own registry (#367). Slice 4, OpenTelemetry traces, was deliberately last and stays with the telemetry roadmap in docs/telemetry-guidelines.md. Closing as complete.

---

Generated by Claude Code