tracking: health, readiness, and internal status system

[zancas]

Task description

Zaino lacks proper health and readiness signaling. Consumers have resorted to log-matching on launch messages or inferring readiness from unrelated endpoints. This tracker groups the work needed to deliver industry-standard liveness/readiness probes and the internal status plumbing they depend on.

External probes (observable API surface)

• We need proper health and ready endpoints #722 — We need proper health and ready endpoints
• Implement liveness and readiness gates #643 — Implement liveness and readiness gates (HTTP, à la Zebra)

Internal status system (foundation the probes depend on)

• unify Status types #889 — unify Status types (AtomicStatus vs NamedAtomicStatus)
• Improve Status System to Support Typed Error Propagation #923 — Improve Status System to Support Typed Error Propagation
• Service subscriber status may show Syncing when cache is up-to-date but mempool is not ready #649 — Service subscriber status may show Syncing when cache is up-to-date but mempool is not ready (aggregate-status correctness)
• Audit check_for_critical_errors against the NamedAtomicStatus logging pattern #980 — Audit check_for_critical_errors against the NamedAtomicStatus logging pattern (critical-state reporting policy)

Context / history

Merged groundwork:

• PR Introduce Status trait for service readiness probing #776 — Introduce Status trait for service readiness probing
• PR Add readiness polling to test utilities after block generation #780 — Add readiness polling to test utilities after block generation
• PR fix early crash failed due to incomplete readiness #962 — Fix early crash due to incomplete readiness

Resolved sibling issues:

• Flaky tests due to StateService readiness race condition #775 — Flaky tests due to StateService readiness race condition
• Test utilities should poll for service readiness after block generation #779 — Test utilities should poll for service readiness after block generation

Reference: ZcashFoundation/zebra#8830 (Zebra liveness/readiness gates).

Not in scope, but cross-referenced:

• Zaino Observability Metrics discussion #856 — Observability metrics discussion (metrics, not probes)
• ReadStateService in zaino not syncing non-finalised state on testnet. #903 — ReadStateService not syncing non-finalised state on testnet (sync correctness)

Acceptance criteria

• zainod exposes /livez and /readyz (or equivalent) HTTP probes that a load balancer can consume without matching log output
• Readiness semantics are defined precisely: which sub-component states map to ready / not-ready, including the mempool-vs-cache case raised in Service subscriber status may show Syncing when cache is up-to-date but mempool is not ready #649
• Status types are unified (unify Status types #889) and structured error propagation is in place (Improve Status System to Support Typed Error Propagation #923), so probe responses carry actionable failure categories rather than opaque booleans
• All child issues above closed

Area

zainod, Testing / CI