Bound PoS stall memory and lock signer startup

[Zk-nd3r]

BFT stalls can retain catch-up state and push validators into high-memory restart loops.

This draft currently:

• bounds Tenderlink packet, gossip, round, and recent-commit state
• prunes stale-round BftBlock payloads while retaining proposal ids, vote evidence, counts, and roster data
• keeps proposal bodies only for the current, locked, and valid rounds
• keeps only the newest committed proposal body in the Tenderlink recent-commit cache and reloads older authenticated PoS history from disk
• locks signer startup behind WAL, anchor, roster, checkpoint, and structured migration-receipt checks
• keeps current s1_dev new-network queue ownership
• removes the unsafe Unix PoS store opener

Checked on head 873b2eab7da63756b436aaf21b996eed4653e303:

• Tenderlink: 54 passed, 0 failed, 2 ignored
• Zebra liveness: 15 passed, 0 failed
• committed diff: clean

Blocking P0s:

1. Durable signer memory and restart replay

DurableSigner stores full proposal, signable, and lock/valid transition payloads for every signed round until the height changes. Startup reads the complete WAL into memory, rebuilds the full intent map, and clones every intent for replay before round pruning. The round/cache compaction therefore does not prove bounded memory during a long stall, and restart peak memory can scale with WAL size.

2. No compatibility boundary with pre-rule history

The operator A/B soak found both migration paths blocked:

• an existing genesis-era pos.chain exits at byte 0 because its pre-roster certificate is rejected as an empty consensus roster
• with pos.chain removed, fresh live-network catch-up exits after about 50 seconds with Tenderlink terminated with an I/O error: invalid data, while base f811d789 under the same state and network continues past cert 1,600

The source audit corroborates the missing rule epoch: strict non-empty roster validation is applied during store replay and Tenderlink startup, with no anchored pre-rule boundary shared by disk and wire paths. The exact wire callsite still needs the offered logs, but the A/B regression and absent boundary are already merge blockers.

Closure needs one authenticated compatibility design, not a blanket local height bypass. Either validate the legacy prefix under exact legacy rules below a network-anchored checkpoint, or consume a sealed migration snapshot that binds the complete prefix, checkpoint identity, parent commit, roster, namespace, chain id, finality, and new-rule activation. Disk replay and wire catch-up must use the same height/epoch verifier. Strict rules must remain mandatory above the boundary.

Source-trace corrections:

• BftBlock does not carry the full unfinalized header span. It carries exactly bc_confirmation_depth_sigma = 3 headers, and the proposer limits each BFT step to at most 40 PoW heights. The 8 MiB per-proposal cap is therefore not shown to be a finality-gap-scaled liveness ceiling. A prior version of this body said otherwise; that claim is retracted.
• current head constructs SignerStartup::Durable only after the complete authority and structured receipt gate passes. Missing authority remains observer-only without creating the WAL or anchor. That earlier implementation concern is closed in source and focused tests, but a real migration is still unqualified.
• the Tenderlink duplicate-history compaction is real, but Crosslink still retains one full BftBlock per committed height in internal.bft_blocks. That is not a stalled-round multiplier, but it is linear long-run history retention and must not be described as an overall constant memory bound.

Additional operator compatibility gap:

• legacy identity configuration becomes observer-only. That is fail-safe, but an upgrade needs an explicit migration command, exact validator_* key diagnostics, and a loud release note.

Do not merge or deploy this draft. Closure still requires:

• crash-safe compact signer high-water or snapshot state preserving anti-double-sign guarantees
• bounded or streaming WAL replay without whole-file and whole-intent duplication
• deterministic 220+ stalled-round tests with realistic proposal payloads and crash/restart interleavings
• legacy-store replay and empty-store live catch-up across the same anchored rule boundary
• rejection tests for an empty roster above the boundary and for any anchor, roster, hash, namespace, or chain-id mismatch
• an exact serialized-proposal boundary test, without claiming size scales with the total finality gap
• measurement and, if an overall memory bound is claimed, compaction of Crosslink's full in-memory committed BftBlock chain
• a long real-stall heap, RSS, WAL, and replay-peak plateau
• a Linux release build of the exact current head
• composite qualification with open PRs wallet persistence #29 and viz: tolerate a best-chain switch during tfl_block_sequence walk #50
• CI and independent review

Draft only. No live node change.

[br33zybail]

Context: I operate a bonded finalizer (rank 17, 91cbc5bd…) and rode through the 2026-07-26..28 stalls this PR targets. Data below is from my prod node plus a throwaway-seed heaptrack rig stood up during yesterday’s stall. Raw files available.

The failure class is real and this PR bounds the right structures.

•	07-26..27 (s2v9): zebrad hit 172GB RSS during the cert-20626 stall and broke internally (roster self-RPC failures, inbound timeouts) hours before any OOM.
•	07-28 (s2v10 base + PR29 cherry-picks): wedged mid-commit at height 268261 during the cert-20751 stall (200+ nil rounds), RSS 2GB → 66GB in 4h09m, ~15.4GB/h. I now auto-restart at 64GB, so the “high-memory restart loop” in your description is literally my crontab.
•	Propose traffic during the stall ran ~4.3MB/s at ~1005 bytes/packet; proposals arrived in 7 chunks.
•	heaptrack on the rig (10 min of catch-up and cert replay, 943MB peak heap): the top two live-heap stacks are both BftBlock::zcash_serialize_to_vec → BlockHeader::write_data under spawn_new_tfl_service, together ~235MB, 25% of peak. 

This is the profile behind the Signal exchange; the follow-up patch (id + evidence only, bodies for current/locked/valid, older bodies from the PoS store) matches exactly what this data asks for.

Question 1: the byte budget vs gap-scaled proposals. BftBlock.headers grows ~1.5KB per block of finality gap, so the one body a validator must still hold, the current round’s, reaches MAX_INFLIGHT_PROPOSAL_BYTES (32MB) by itself at a gap near ~21k blocks (~6 days at 25s spacing). When that happens, does reserve_proposal_storage reject its chunks, meaning a validator can no longer assemble and prevote the proposal that would end the stall? If the design is “evict everything else, always admit the current round,” a comment and a test pinning that would help; if not, deep-stall recovery may need a floor that always admits one current-round proposal. (Long-term, the gap scaling itself points at header-span compression the PoPoW? note on the field.)

Question 2: window semantics under nil grinds. We saw 200+ nil rounds at one height. RETAIN_PAST_ROUND_WINDOW=64 plus the budget looks right for that regime; the soak couldn’t reach it yet (the branch exits at startup or early catch-up, findings below); I’ll confirm once the branch survives on real data.

The harness behind the findings below: throwaway-seed rig built from consistent prod state snapshots, heaptrack wrapping, RSS-cap auto-stop, and 5-minute smaps telemetry on prod for A/B (yesterday’s stall is captured end to end). It stays standing; I’ll re-run each push and against the next real stall.

Soak findings (head 873b2ea, real operator state, live workshop network). Rig: throwaway-seed node from a consistent prod state snapshot (PoW state version 27.1.0, accepted fine), heaptrack-wrapped, on the live network. Control for attribution: the same rig under the PR’s own base (f811d78, no PR commits).

1.	Existing stores cannot start. On a pos.chain carried since genesis, startup exits: PoS-store certificate at byte 0 is rejected: WAL integrity failure: consensus roster is empty. The genesis-era certificate predates any roster, so every long-lived operator hits this on upgrade. Failing closed is the right mode, but the intended path needs one documented answer; bootstrap_bft_roster (its doc covers fresh nodes, not existing stores), a receipt, or a store migration.
2.	Legacy identity configs silently stop signing. legacy BFT identity configuration is observer-only and fails readiness. Correct fail-safe, but a bonded finalizer upgrading with a wallet-seed-derived identity goes observer without ceremony. Worth a loud release note plus a startup line that says exactly which validator_* keys to set.
3.	Fresh-store catch-up dies on live network data. With no pos.chain, the node runs ~50s of cert catch-up then exits: Tenderlink terminated with an I/O error: invalid data. Attribution is clean: the base commit under identical conditions (same rig, same PoW state, same fresh store, same network) survives, syncs, and decides values past the same point. Suspect the strict store validation rejecting a network-delivered early cert (plausibly the same empty-roster genesis cert as finding 1, surfacing through the catch-up path). Net effect of 1+3 as drafted: a node can neither upgrade in place nor rejoin fresh on this network.

[Zk-nd3r]

Thanks for running this against real operator state with a control build, that is exactly the harness this branch needed, and the heaptrack stacks match the paths the compaction targets (BftBlock::zcash_serialize_to_vec under spawn_new_tfl_service is the serialize-per-consumer pattern the id+evidence follow-up removes).

Your three soak findings are the same three blockers disclosed in the PR description, and they share one root: the new strict rules have no authenticated boundary with the pre-rule prefix, and the disk replay and wire catch-up paths do not share a height-aware legacy verifier. Your suspicion on finding 3 is right as far as I can tell from the callsites: the same genesis-era empty-roster certificate that kills startup at byte 0 surfaces through catch-up as invalid data on the wire path. The direction I am taking is one network-anchored rule epoch (or an equivalently sealed migration snapshot) that binds the full legacy prefix across all consensus domains, with the strict rules mandatory above activation and no invented checkpoint. That covers in-place upgrade, fresh rejoin, and keeps your finding 2 as a loud release note plus a startup hint naming the validator_* keys.

On question 1, your concern is right and the binding constraint is tighter than the 32 MiB math. This branch introduces both budgets: MAX_INFLIGHT_PROPOSAL_BYTES = 32 MiB (aggregate reassembly) and a per-proposal hard cap MAX_PROPOSAL_BYTES = 8 MiB (signer_wal.rs), and the per-proposal cap is enforced at the wire chunking layer: proposal_chunk_count returns None for anything over 8 MiB, so the chunks are refused before reserve_proposal_storage is ever consulted. At your measured ~1.5 KiB of header growth per block of gap, the current round's proposal stops being assemblable at a gap near 5,600 blocks, roughly 1.6 days at 25s spacing, well before the aggregate budget matters for a single proposal. So as drafted there is no "evict everything else, always admit the current round" floor, and adding one is not sufficient on its own; deep-stall recovery needs the floor plus either a raised cap for the current round or the header-span compression you point at. I will pin the intended semantics with a comment and a test once the boundary design lands, since the honest current answer is that a deep enough stall cannot end via this path.

On question 2, agreed that the 64-round window plus the budget is the right shape for nil grinds; your confirmation once the branch survives on real data is exactly the evidence it needs. The startup and catch-up blockers gate that soak, so they go first. The PR stays draft until the boundary work is in, and I will take you up on the re-run offer on the next push.