Bulk synchronization: v2 protocol primitives and Block Chain Synchronization ZIP - #1346
Open
arya2 wants to merge 14 commits into
Open
Bulk synchronization: v2 protocol primitives and Block Chain Synchronization ZIP#1346arya2 wants to merge 14 commits into
arya2 wants to merge 14 commits into
Conversation
arya2
force-pushed
the
agent/checkpointed-sync-zip
branch
3 times, most recently
from
August 6, 2026 22:09
8c07e78 to
a867274
Compare
arya2
changed the title
arya2
force-pushed
the
agent/checkpointed-sync-zip
branch
from
August 17, 2026 20:12
ae76452 to
62c49e7
Compare
12 tasks
arya2
force-pushed
the
agent/checkpointed-sync-zip
branch
from
August 17, 2026 22:23
c593c65 to
0d723b9
Compare
Recommends how nodes synchronize over the version 2 P2P protocol, which defines the primitives (get-headers/get-blocks/get-hashes) and the normative synchronization rules. This ZIP recommends two concrete strategies satisfying those rules: - Headers-first synchronization (the baseline full-validation method). - Checkpointed synchronization: verify checkpoint-spaced get-hashes responses against a local trusted commitment (e.g. compiled-in hash chunk hashes), fetch stride-1 hashes as download handles, and hash-chain each downloaded range to its verified checkpoints. Plus strategy selection and peer-diversity recommendations, with graceful degradation to headers-first when get-hashes is unavailable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C36xn69yRXtupScMCYarca
Research-driven redesign removing the legacy-protocol assumptions the IBD design was built around: v2 P2P ZIP: - get-block-range (0x07): anchored, descending block streaming — every block verified on arrival against a committed anchor; resumable truncation; kills per-hash download handles and request round trips. - get-tree-roots (0x08): per-block note commitment tree roots + history-tree leaf data + auth data roots, verified against header commitments (ZIP 221/244) — eliminates the ~70%-of-commit-CPU tree recomputation with zero added trust. - get-object (0x09): ranged fetch of content-addressed sync artifacts (known-hash chunks, snapshot pieces) with SHA-256 verification. - Service flags: NODE_TREE_ROOTS, NODE_SYNC_ARTIFACTS, NODE_NETWORK_LIMITED (≥2,304 recent blocks) with advertisement rules for snapshot-synced and pruned nodes. - Bulk transfer performance guidance (BDP-sized flow control windows, paced congestion control, UDP offload, per-connection core bound). - Scope ZIP 204 download parameters to headers-first; bulk transfer is byte-budgeted per the sync ZIP. Sync ZIP: - Checkpointed synchronization rebuilt on verified known-hash chunk artifacts + byte-sized get-block-range work units. - Verified Tree Roots strategy (header-commitment verification procedure, one-block-lag history tree rebuild). - Snapshot Synchronization: manifest + pieces, chain-verifiable tree components, assumeutxo-model trust for uncommitted state, background backfill, service flag transitions. - Download Scheduling: 16-64 MiB units, EWMA per-peer budgets, 4-8 peer concurrency, byte-bounded lookahead, dual stall detection, end-game duplication, exact blame. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C36xn69yRXtupScMCYarca
- get-block-range: add a max_bytes request bound (cap 64 MiB) so work units are sized in bytes, not blocks — count alone was meaningless across eras whose block sizes differ by three orders of magnitude, and byte truncation self-chunks spans even without size metadata. Make the header-to-body binding (transaction merkle root) an explicit on-arrival check. - Sync ZIP: new Spentness Hints strategy adapted from SwiftSync: salted-aggregate, lookup-free, order-independent state construction. Transparent outputs use a 1-bit-per-output hint artifact; nullifier sets need no hint (snapshot sets serve as the hint during backfill, proving set equality and distinctness); value pools come from commutative sums. Hint-verified backfill discharges the trust in the snapshot's uncommitted components. Wrong hints fail sync; they can never cause acceptance of incorrect state. No new wire messages — hints are content-addressed artifacts over get-object. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C36xn69yRXtupScMCYarca
…tacks Adversarial review of the misbehavior rules found places where penalties prescribed for security could be turned against honest nodes. Fixes: - Scope the wrong-reference-type penalty to senders that hold the transaction (announcements, get-mempool records); wrong-typed get-tx requests get a not-found result and MUST NOT be penalized, since a requester can have derived the type from another peer's data. - get-object: penalize a content-hash failure only when one peer delivered every byte; forbid penalizing multi-peer assemblies and specify re-fetch-whole-from-one-peer to isolate the liar. Make overdelivery a FLOOD connection error. - Constrain the open-ended validation-failure penalty to provable, context-free invalidity, with explicit exemptions: expiry races, local policy, missing/moved state, content of requested objects, and transactions served under the block-reconstruction obligation. - Anchor get-tree-roots requests with a final_hash field so entries describe requester-identified blocks; divergent responders refuse with REFUSED instead of eating a 100-point penalty for honest disagreement. - Key misbehavior scores by address with persistence, so deferred penalties (span hints, tree roots) survive reconnection whitewashing. - Add a governing principle: no penalty unless the violation is provable independent of local chain view, mempool, clock, and policy. - Specify handling for nonconforming announcement records, get-block-range overdelivery, and oversized get-headers/get-addr responses; extend the pre-validation exemption to substituted header announcements and served transactions; clarify whitelist scope and ban address granularity. - Sync draft: align exact-blame and security-claim wording with the attribution rules; anchor tree-root requests at authenticated hashes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fourteen verified findings from a security review of the two draft ZIPs. Verification correctness: - Tree-root verification used hashFinalSaplingRoot as the Sapling root "between Sapling and NU5", but ZIP 221 repurposes that field from Heartwood. Split the procedure at Heartwood and state that comparing a Sapling root against that field above Heartwood would penalize every honest responder. - Roots that no header commits to (Ironwood at every height, and the highest entry of each response, which reconstruction has not yet reached) were adopted unverified; they are now never adopted. Align the pre-activation sentinel with the p2p ZIP's 32 zero bytes. - get-block-range's arrival checks do not bind v5 authorizing data, so "accepts no unverified bytes" was false from NU5. Correct the claim, bound block size, and require the auth-data check before a block is stored, served, or used to regenerate artifacts. Snapshot trust: - Hint-verified backfill did not discharge what it claimed: the transparent aggregate hashed outpoints only, and the Sprout tree and value pool balances were checked by nothing. Reconcile whole UTXO entries, recompute the Sprout tree and pool balances, and scope the discharge claim to what is actually compared. - Require the trusted commitment to reach H+1, since the chain history tree at H is committed by the following block's header. Spentness hints: - Specify the construction (keyed BLAKE2b-256, 32-byte CSPRNG salt, per-aggregate personalization, 256-bit modular accumulator). A non-PRF or a machine-word accumulator defeats the "failure, not fraud" guarantee. - Scope order-independence to the state construction: ZIP 209's per-block non-negative pool balance rule and the maturity rules are not functions of the terminal totals. Require checked arithmetic for value sums. Bounds and allocation: - Bound the Tor preamble's network string and flow-control fields, which are parsed before every existing limit; make the handshake timeout run from transport establishment so it covers them. - Bound get-object's offset and specify the evaluation order, so the natural min(offset+length, size) clamp cannot underflow into a huge read; reject a size below the bytes already received. - Bound compact-block ids_count and prefilled_count individually as well as their sum, and bound prefilled indexes by the block's own transaction count rather than by 65535. - Bound get-headers ids_count, which the record payload limit does not reach because ids is a fixed-width array. - Add a protocol-wide rule against allocating from a declared count. Attribution and amplification: - Require get-blocks and get-tx responses to match the request; give both a responder byte budget, since count limits do not bound bytes. - Restrict the span-metadata penalty to hash-determined fields: span_size is not fixed by a block's hash, so a padding attacker could make the penalty fall on the honest hint-serving peer. - Add misbehavior rows for auth-data and artifact-hash failures. - Validate the addrv2 time field, the freshness signal peer selection ranks on, which was attacker-suppliable and unclamped. Also: rate-limit get-mempool and gate it on relay; align the reorganization margin across both documents; de-duplicate two colliding "Example (non-normative)" anchors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replace restated definitions with references to their canonical homes: - ZIP 204 owns CompactSize/strings/integers, default ports, service-flag bit numbering and reserved bits, and TX_EXPIRING_SOON_THRESHOLD - BIP 152 owns the short-ID algorithm and prefilled index encoding; only the Zcash-specific substitutions remain - The v2 protocol draft owns get-block-range delivery/resumption rules, the service-flag advertisement rule (and 2,304 constant), the get-object per-request maximum, and the trusted-commitment/artifact/ anchor definitions the sync draft previously restated - The misbehavior table is now the single registry of penalty points; inline sites cross-reference it - Drop repeated intra-document statements (self-connection nonce, init-field immutability, failure-not-fraud, re-anchoring mechanics) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Pad the example nullifier-pool personalization string to the mandated 16 bytes. - 'Header field at offset 4' read as a byte offset; say 'fourth header field'. - The false-zero probability of 2^-32 describes a 32-bit accumulator, not any machine word. - The v2 protocol has no named 'resumption rule'; describe the get-block-range truncation behavior instead. - Drop get-tx from the imported terminology; nothing in this ZIP concerns transaction relay. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The compiled-in commitment is only the expected hashes of the list's chunks over fixed height ranges; the full known-hash list is obtained verifiably from the P2P network with get-hashes, with get-object artifacts, bundling, and mirrors as alternative sources of the same bytes rather than the primary path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Spentness hints become per-range hint chunks bound by the same trusted commitment as the known-hash list, verified against pinned per-range hashes; entry-chunk hashes stay stable as coverage extends, hint-chunk hashes change with the terminal height. - Confine hints to the reach of the trusted commitment, dropping the per-spent-output creation-height/coinbase-flag artifact extension that only the above-commitment mode needed. - Drop the nullifier distinctness checks: double-reveal below the final checkpoint is excluded by the commitment, and the aggregate's multiset equality transfers distinctness to the snapshot set. - Stop carrying the transparent pool balance in the snapshot (it is the sum of the UTXO set's amounts) and drop its backfill comparison, which the whole-entry UTXO check already implies; the shielded pool balances remain in the manifest, needed for ZIP 209 enforcement above H. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…unks Per-range hint hashes bought no stability: extending the commitment's coverage changes every hint hash anyway, since the bits are relative to the final checkpoint height. Bind all hints up to the final checkpoint under a single SHA-256 hash, fetched and verified whole, and keep the per-range hashes for the entry chunks alone, whose hashes never change as coverage extends. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The node needs them — the transparent pool balance included — as the starting point for the pool-balance consensus rules when it validates above H, so they stay in the manifest and in the backfill reconciliation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reflects zebra/docs/design-swiftsync-hints.md decisions: - Normative table of aggregates, item encodings (36-byte outpoint, 32-byte nullifier), and 16-byte personalization strings, replacing the examples. - Verify the hint bitmap upfront — hash, and bit count against the entry chunks' txouts metadata (added to get-hashes in the v2 draft) — before any body download; recovery keeps the verified known-hash list and header chain and re-downloads bodies without hints. - Genesis-anchored sync drops the nullifier aggregates entirely: inserts are sequential writes, and double-reveal is excluded by the commitment. - Older commitments stay valid for their shorter span. - Optional frontier artifacts at the final checkpoint, verified against header-authenticated roots (Ironwood excepted), so verified tree roots can hand off to full validation without replaying leaves. - Deployment: purely in-band distribution is possible given NODE_SYNC_ARTIFACTS seeders; bundling and mirrors become MAY. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The commitment's pinned hash, updated by each release that updates the checkpoints, is the whole verification story: the bitmap stands or falls with the commitment itself, exactly as the checkpoint hashes do, and is auditable the same way, being a deterministic function of the chain. Remove the salt, the keyed-hash aggregates, the personalization table, and every zero-check; the snapshot backfill reconciliation becomes a direct whole-entry comparison of the reconstructed and snapshot sets, and backfill proceeds without transparent hints (the bits are relative to the final checkpoint, not H). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Recommends how nodes synchronize over the version 2 P2P protocol. The v2 protocol draft (#1344) now defines the primitives (
get-headers/get-blocks/get-hashes) and the normative synchronization rules for both headers-first and checkpointed synchronization; this ZIP recommends concrete strategies satisfying those rules:get-hashesresponses against a local trusted commitment (e.g. compiled-in hash chunk hashes), fetch stride-1 hashes as download handles, and hash-chain each downloaded range to its verified checkpoints.Plus strategy selection and peer-diversity recommendations, degrading gracefully to headers-first when
get-hashesis unavailable.Stacked on #1344, which it references.
🤖 Generated with Claude Code
https://claude.ai/code/session_01C36xn69yRXtupScMCYarca