Investigate PoW sync/reorg behavior under sparse solo mining

[Zk-nd3r]

Summary

Field report from active featurenet operation suggests a PoW sync / fork-choice failure mode under sparse solo mining.

A node at tip mined a competing block at height 202921, accepted it locally, then continued extending that local branch for thousands of blocks instead of reorging back to the network branch after peers moved ahead. This makes solo mining at the tip operationally risky when there are very few miners online.

Motivating Report

Reported shape:

• Last matching height: 202920
• Last matching hash: 00490cfa56dc627ad92b2efed1e566a23d94ca9681c2f517754e473d63997bdd
• First divergent height: 202921
• Local self-mined hash: 01117e8991d3ea9efeb904cb747506d15181d7d478cb913543e9bde8bd1525ca
• Reference/explorer hash: 0058a975e8b669efa8c82d4275704e876313993692d3ae4f5f5d99c858526877
• Both h202921 blocks reportedly shared the same parent at h202920.
• Immediately after accepting the local block, peers were still advertising h202920.
• The node then mined consecutive local heights for roughly 3k blocks.
• Later, peers were ahead on the network branch, but the node remained on its local branch and did not reorg.

The operator recovered by restoring a known-good snapshot from before the divergence and resyncing from peers.

Why This Matters

If a miner wins a single sparse race at the tip and then keeps extending its own branch, there may be no practical reorg trigger even when peers later advertise a different/heavier branch. That undermines the practical instruction to mine up to the activation height, because the miners most likely to help advance the chain can silently self-fork.

Local operational response on one node: mining was disabled and the external cTAZ submitter was stopped until this behavior is understood. After stopping the submitter, there was a quiet window with zero new submitblock, mined-broadcast, or internal solver/template-mining log lines.

Questions

• What fork-choice rule should pull a node back when peers advertise a competing/heavier branch after the node self-mines ahead?
• Is solo mining at tip currently expected to be unsafe under sparse network participation, or is this a sync/reorg bug?
• Should block production pause/throttle when peers are behind or branch disagreement is detected?
• Can PoW sync add explicit tests for:
  • same-parent competing block at tip,
  • local branch extended for many heights,
  • peers later advertising a higher/heavier competing branch,
  • node reorging back without manual state restore?

Related

This is separate from BFT transport cleanup in PR #36 and BFT replay hardening in issue #37. Those address finality/transport failure modes; this one is PoW sync/fork-choice.

[Zk-nd3r]

Follow-up with a clean mining-off reproduction from my s1_dev node.

At 2026-07-10 12:55:30 UTC it committed:

• h207818: 01d1b10902b11c7a4c21a749dbc12c7bf27a9d62b462f724061569efd8c0baeb
• h207819: 0051ba9c6f15205a296f85b598ea9b1c623052f51af5cd57720581b58309fb21

That is the same h207819 branch shown by ctaz.cash.

Mining has been off since 00:13:30 UTC. internal_miner=false, and the guard has recorded zero submitblock, template, solver, or mined-broadcast events after the cutoff. The external stratum/RPC submitter is also stopped.

At 15:02 UTC the node was still at h207819, with no state advance for 126 minutes. Connected peers were advertising ranges ending as high as h208136. The syncer reported is_syncer_stopped=false. There are zero does not link anywhere known drops in this run.

This node is running s1_dev HEAD 18cea69, which includes e1bfcb55, the experimental change that queues blocks whose parent is not yet known. That removed the drop symptom, but the node still did not link and commit the continuation or switch branches.

This matches the separate mining-off h206287 and h207589 stalls reported in Signal. It also lines up with today's h207819 split:

• common h207818: 01d1b10902b11c7a4c21a749dbc12c7bf27a9d62b462f724061569efd8c0baeb
• my/explorer h207819: 0051ba9c6f15205a296f85b598ea9b1c623052f51af5cd57720581b58309fb21
• other reported h207819: 0230703a637cee113532d4700414fd6e0c49a519c5076f7fc7feaa75b3963dbf

I am not calling either h207819 child canonical. The useful fact is that nodes can sit on different children while seeing peers advertise later heights, and automatic recovery does not happen.

The regression test I would want before mining clearance:

1. Create two siblings at height N from the same parent.
2. Give the node branch A first.
3. Advertise a longer candidate branch B with missing ancestry.
4. Verify the node fetches the missing ancestors, links B, compares cumulative work, and reorgs when B wins.
5. Run it once with mining off and once while the node is producing blocks.
6. Bound the orphan queue so the e1bfcb55 behavior cannot be used for memory DoS.

Can you confirm whether the planned tip-sync fix covers ancestor backfill plus fork choice, rather than only retaining unknown-parent blocks? I can share the exact log window and state artifacts if useful.