Skip to content

feat(network): run the experimental v2 (QUIC) transport in the peer set - #11277

Draft
arya2 wants to merge 7 commits into
p2p-v2-4-peer-connectionfrom
p2p-v2-5-peer-set-integration
Draft

feat(network): run the experimental v2 (QUIC) transport in the peer set#11277
arya2 wants to merge 7 commits into
p2p-v2-4-peer-connectionfrom
p2p-v2-5-peer-set-integration

Conversation

@arya2

@arya2 arya2 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Motivation

Final PR of the five-PR stack implementing the draft version 2 Zcash P2P network protocol (zcash/zips#1344). This PR wires the v2 transport into the peer set behind experimental config options, and removes the stack's #[allow(dead_code)] scaffolding.

Solution

  • Adds peer_set/initialize/v2_transport.rs: a QUIC endpoint that accepts inbound v2 connections on the UDP port of the listen address (when network.experimental_v2_listen is enabled) and maintains connections to network.initial_v2_peers, redialling them when they fail or disconnect.
  • Inbound v2 connections go through the same shared admission policy as the TCP listener (canonicalization, dual-representation ban check, shared connection counter with atomic slot reservation, and one shared per-IP limiter), so the two transports cannot drift and together stay within the configured limits.
  • Initial v2 peers are resolved and dialed from a supervised task, so slow or failing DNS cannot delay the listener; the dialer and per-connection handshake tasks are supervised from the accept loop like the legacy listener, so panics stop the node instead of silently reducing connectivity.
  • v2 peers produce ordinary peer Clients and join the peer set alongside legacy peers; connection attempt metrics are recorded for both stages of inbound v2 connections.
  • New config options (both default off / empty): network.experimental_v2_listen and network.initial_v2_peers.

Tests

  • Integration tests over real QUIC connections: the listener accepts a v2 connection and serves a request through the inbound service, and refuses connections from banned peers.
  • Full zebra-network suite passes (260 tests); cargo clippy -p zebra-network --all-targets warning-free; zebrad builds; config compatibility test updated.
  • Not yet tested: a live two-node sync session between separate zebrad instances on a public network.

Specifications & References

Follow-up Work

  • Add v2 peers to the crawler once address gossip indicates which transports a peer is reachable on.
  • Live two-node test evidence on Regtest/Testnet.
  • Structured v2 peer error enum replacing the stringly V2Protocol/V2Internal variants.

AI Disclosure

  • No AI tools were used in this PR
  • AI tools were used: Claude Code was used to implement the integration and tests, run the test/lint verification, and draft the commit messages and this description; the changes were reviewed by the author.

PR Checklist

  • The PR title follows conventional commits format: type(scope): description
  • The PR follows the contribution guidelines.
  • This change was discussed in an issue or with the team beforehand.
  • The solution is tested.
  • The documentation and changelogs are up to date.

@v12-auditor

v12-auditor Bot commented Aug 17, 2026

Copy link
Copy Markdown

Note

Complete: Audit complete. V12 found 25 issues worth reviewing.

Open the full results here.

FindingSeverityDetails
F-232246 🟠 High
Rejected peers leak uncounted QUIC connections

A v2 Client rejected by the peer set can leave its QUIC connection alive after its admission slot has been released. Duplicate-address and per-IP checks drop the service, whose generic Client::drop path aborts the v2 connection task rather than letting it execute the cleanup after its main loop. If cancellation wins the wakeup race after that task has spawned its handshake monitor and announcement writer, their detached tasks retain QUIC handles while waiting indefinitely, and the configured keepalive prevents the transport idle timeout from closing the connection. The ConnectionTracker is nevertheless destroyed with the aborted parent future, so the shared counter no longer represents this live state. A malicious v2 peer can repeatedly trigger the rejection path and acknowledge QUIC traffic to accumulate uncounted task, buffer, timer, and connection state; for configured outbound peers, the extra handle waiting on closed() can also prevent maintain_v2_peer from ever redialling.

F-232247 🔵 Low
Spoofed Initials consume shared admission state

The v2 listener commits shared admission state using incoming.remote_address() before the peer has proved return routability. It checks bans, reserves an inbound slot, and records the source in the shared RecentByIp limiter before the spawned task awaits quic::accept. Neither the endpoint setup nor this path uses Quinn Retry or checks remote_address_validated(), and the application handshake's three-second timeout starts only after QUIC establishment. An off-path sender capable of spoofing UDP sources can therefore poison a chosen IP's default 119-second quota with a tokenless Initial, preventing the real host from connecting over either TCP or QUIC. By rotating spoofed sources and leaving transport handshakes incomplete, the attacker can also hold most or all shared inbound capacity for the transport idle window and repeatedly deny new inbound peers.

F-232248 🟡 Medium
Public QUIC peers consume reserved sidecar capacity

The legacy listener reserves inbound capacity for configured zcashd-compatible sidecars by subtracting reserved slots from the public limit and tracking sidecars on a separate counter. The v2 listener receives only the shared public counter and admits public QUIC peers against the full inbound limit, not the reduced public limit. A Sybil set using distinct IPs can consequently establish and hold the full limit as valid v2 sessions. When the sidecar starts or reconnects, the strict total-capacity check rejects the reserved branch, and fallback admission also fails because the public counter already exceeds its reduced limit. The capacity intended to guarantee wallet-sidecar reconnects is therefore public whenever v2 listening is enabled.

F-232249 🟡 Medium
Shared Testnet identity admits incompatible peers

All configured non-Regtest Testnets use the same v2 Testnet ALPN even when their network magic, genesis, or consensus parameters differ from public Testnet. The v2 init record contains no network identity, and the application handshake validates only version and self-connection nonce before constructing an ordinary peer client. Thus a public-Testnet endpoint can complete either an inbound or configured-outbound v2 handshake and enter an incompatible custom-Testnet peer set. Normal block validation prevents commitment when custom genesis, checkpoints, or consensus rules differ, but only after peer slots, downloads, and verification work are consumed. If the custom network differs only by network magic, that protection does not apply because magic is a legacy framing property rather than a block-consensus field, allowing the node to follow public Testnet state despite its incompatible configuration.

F-232250 🟡 Medium
QUIC liveness pollutes TCP peer discovery

Successful QUIC handshakes are recorded as ordinary direct connections in a shared address book that has no transport discriminator. The shared success helper stores both inbound and outbound v2 socket addresses as MetaAddr::new_connected; although the inbound flag excludes inbound addresses from gossip, it does not exclude them from the disk cache or TCP candidate selection. After the recency delay, the legacy crawler can therefore attempt TcpStream::connect to a QUIC-derived UDP endpoint, and a cached entry is loaded as a legacy initial peer after restart. Outbound v2 successes are not marked inbound, so they can additionally be returned in ordinary address gossip without any indication that they require QUIC. A remote v2 peer can use many addresses or source ports to create persistent QUIC-only TCP candidates, while configured QUIC-only peers propagate wasted TCP attempts to other nodes.

F-232251 🟠 High
One peer monopolizes inbound request capacity

Each remote-opened v2 bidirectional stream is served in its own spawned task, with no per-connection semaphore or request concurrency limit. The QUIC configuration permits 128 such streams, while production exposes one process-wide inbound service wrapped by a 200-slot buffer, load shedding, and a five-second timeout. One admitted peer can therefore occupy roughly two thirds of the node's inbound request capacity, unlike the legacy connection state machine, which awaits one inbound request at a time. When the shared service returns Overloaded or Elapsed, v2 collapses the error to ServeError::Refused and resets only that stream. It neither records overload history nor applies the legacy probabilistic connection shedding, so the peer can immediately replace refused streams and sustain the starvation indefinitely.

F-232252 🟠 High
Duplicate requests amplify response work

V2 get-blocks and get-tx requests are deduplicated only for the internal lookup, not for response generation. The decoder accepts up to 128 block entries or 50,000 transaction references without requiring uniqueness, and the response loops then iterate the original vectors and serialize one response for every duplicate. Repeating one known large block 128 times causes one state lookup but 128 full or compact response constructions and writes; repeating one transaction reference causes up to 50,000 serializations from one lookup. Each connection can run 128 of these request streams concurrently, and writes can remain flow-control-blocked until the 40-second stream timeout while their request vectors and available-object maps remain live. Compact-format duplicates are especially expensive because every entry rebuilds the compact block and recomputes transaction IDs, including a redundant second pass in the short-ID table builder.

F-232253 🟡 Medium
QUIC migration leaves policy on stale IP

The server uses Quinn's migration-enabled defaults but permanently identifies an inbound connection by the address captured from Incoming before establishment. After a client moves from reachable address A to reachable address B and completes QUIC path validation, Quinn carries traffic over B while Zebra continues using A in ConnectedAddr, the peer-set key, RecentByIp, and SharedConnection::transient_addr. Scored violations sent after migration are therefore accumulated against A and can ban unrelated peers sharing A, while B remains unbanned and reusable. The same stale keys let several connections admitted under different original addresses migrate to one current IP without being counted against that IP's configured connection cap. This requires control and reachability of each migration address; blind spoofing alone cannot complete path validation.

F-232255 🟠 High
V2 DNS fanout can monopolize peer discovery

initial_v2_peers() returns the complete DNS expansion of every configured name, and the dialer creates one permanent maintenance loop per resulting address. Unlike legacy initial peers, this set is not sampled down to peerset_initial_target_size; each responsive v2 endpoint competes for the same outbound counter used by the normal crawler. A controller of one configured hostname can return at least the outbound-limit number of distinct responsive addresses and keep those sessions open. Once they occupy the limit, startup produces no fill demand and crawler demand is dropped, so independently discovered legacy peers cannot be connected. DNS can also return arbitrarily more addresses than the limit, leaving an unbounded number of sleeping permanent maintenance tasks.

F-232256 🟠 High
Non-serving peers occupy sync slots

The v2 outbound handshake accepts any version-valid init record regardless of advertised services. Unlike the legacy outbound handshake, it does not reject a peer missing NODE_NETWORK while the local node is syncing, even though such a peer cannot serve historical blocks. A controller of a configured v2 bootstrap DNS name can return enough responsive addresses to occupy shared outbound capacity, complete each handshake with empty services, and keep those ordinary clients ready by answering requests with valid not-found results. The peer set filters ready clients by protocol version rather than services, and crawler demand is dropped once the shared counter is full. On a fresh node, these maintained non-serving clients can prevent genesis and historical block acquisition until honest capacity becomes available.

F-232257 🔵 Low
Source-port rotation resets misbehavior scores

Inbound v2 misbehavior is reported and batched by full PeerSocketAddr, including the peer-controlled UDP source port. Address-book score accumulation also updates only the exact socket-address entry, but the eventual sanction is an IP-wide ban after one entry reaches the threshold. An inbound peer can therefore send a scored violation, disconnect, wait for the recent-IP window to expire, and reconnect from a new source port to receive a fresh score entry. Repeating that sequence keeps every individual entry below the ban threshold while producing an unlimited series of violations from the same IP. Canonicalization normalizes IP representation but does not remove the transient port.

F-232258 🔵 Low
Banned QUIC attempts bypass all throttles

The banned-source branch in the new QUIC accept loop refuses an Incoming and immediately continues. It bypasses both the failed-attempt sleep and the successful-attempt sleep/yield, and banned attempts are checked before the shared counter or RecentByIp limiter. Unlike TCP, a tokenless QUIC attempt is a spoofable UDP Initial rather than a completed kernel three-way handshake, and incoming.refuse() requires server-side packet processing and generates a response. A banned peer, or a sender spoofing an already-banned source, can submit fresh Initial connection IDs at packet rate with no application-level rate bound. The busy accept loop can consume CPU and egress while legitimate v2 attempts queue behind the refusal flood.

F-232259 🟡 Medium
V2 bootstrap bypasses Regtest isolation

Regtest's legacy bootstrap path filters DNS results to localhost, enforcing the documented rule that Zebra does not make remote Regtest peer connections. The new initial_v2_peers() method returns the generic resolver's complete output without the same network.is_regtest() and is_localhost filter. A configured Regtest hostname that is compromised, misconfigured, or answered publicly can therefore resolve to a remote v2 endpoint. The maintenance loop dials that address and inserts a successful Regtest-protocol client into the ordinary peer set with no later locality check. Regtest also skips proof-of-work validation, increasing the importance of its local-only networking assumption.

F-232260 🔵 Low
One-shot DNS resolution prevents v2 recovery

V2 bootstrap names are resolved only once, and resolution completes when the aggregate contains any address even if other configured names failed. Each numeric result is then captured by an infinite maintain_v2_peer loop; hostnames that failed are discarded, and successful hostnames are never re-resolved after DNS changes. For example, if name A returns an unreachable address while healthy name B temporarily times out, A makes the aggregate nonempty and B is omitted for the process lifetime. Correcting B or changing A's authoritative answer does not restore connectivity because the maintainers know only fixed PeerSocketAddr values. V2 currently has no crawler path that could discover a replacement transport endpoint.

F-232261 🟡 Medium
Peer-set shutdown detaches v2 dialers

The v2 task hierarchy is supervised for panic propagation but not for cancellation. run_v2_endpoint spawns a dialer and stores its ordinary Tokio handle only inside a local wait_for_panics future; that dialer similarly spawns permanent per-peer maintenance tasks. Peer-set shutdown owns and aborts only the top-level endpoint guard. Dropping nested Tokio JoinHandles detaches their tasks rather than aborting them, so configured maintainers can survive shutdown with endpoint and counter clones and continue their dial/handshake/send loops. Inbound handshake tasks stored in the endpoint's local task pool have the same parent-cancellation problem until their own paths finish.

F-232262 🔵 Low
Outbound-only mode still opens QUIC server

When v2 outbound dialing is enabled without experimental_v2_listen, the code describes its wildcard port-zero socket as outbound-only. It nevertheless constructs that socket with quinn::Endpoint::server, installing a full server configuration on every local interface. The non-listen branch then waits on the dialer and never calls endpoint.accept, so inbound QUIC attempts are neither screened by bans/counters/RecentByIp nor explicitly refused or ignored. Every configured remote peer can observe the ephemeral UDP source port and direct inbound attempts at the undrained server-configured endpoint. Such attempts cannot become application clients because no Incoming is awaited, but they still create unscreened endpoint-level work governed only by Quinn defaults rather than Zebra's admission policy.

F-232263 🟡 Medium
Announcements evict shared inventory updates

Each inbound v2 transaction or block announcement record sends one single-hash change into the process-wide inventory broadcast channel. The channel capacity is only the total peer limit and overflow deliberately displaces older messages; the inventory registry reports lag and discards those advertisements. Legacy inv processing batches many hashes into one InventoryChange, but v2 consumes one channel slot per record and applies no announcement rate limit. The v2 path also ignores the inbound-service result after registering an announcement, so overload does not penalize or slow the announcing connection. A sustained record stream can therefore evict availability and missing-inventory updates contributed by unrelated peers across both transports.

F-232264 🔵 Low
Compact cache pins transaction data

Compact-block responses populate a per-connection short-ID cache with an Arc<Transaction> clone for every transaction in each block. The only bound is eight block-hash entries; there is no retained-byte budget or expiry, and the entries live with SharedConnection. Both triggers are remote-controlled: the peer advertises full_ids = false and requests eight distinct large blocks in compact format. Dropping the temporary Arc<Block> does not release the transaction bodies because the short-ID tables keep owning clones. The peer can then remain idle while QUIC keepalives preserve the connection, retaining roughly several maximum-sized blocks of transaction data per admitted connection.

F-232265 🔵 Low
Maintainer redials banned bootstrap peers

The infinite maintainer for initial_v2_peers does not receive the ban watch and never checks it before reconnecting. After a configured peer accumulates enough misbehavior to enter the IP ban map and its existing connection closes, the maintainer simply waits the fixed delay and establishes another QUIC and application handshake to that same address. Only after the new client reaches the peer set can the general ready/unready ban checks discard it. The reconnect loop then repeats for the process lifetime, spending transport and handshake resources on an address policy has already excluded. This is separate from whether the banned client can serve requests, because the waste happens before peer-set rejection.

F-232267 🟡 Medium
Independent listeners double admission rate

TCP and QUIC admission run in independent tasks, and each enforces the one-second successful and ten-millisecond failed pacing intervals only with local sleeps. The shared counter and RecentByIp map constrain capacity and per-IP history but do not serialize the global admission rate. An attacker using distinct source IPs can therefore make both listeners admit and launch a handshake in the same interval, approximately doubling the intended one-connection-per-second work. It can also keep one transport on its admitted-handshake schedule while driving capacity rejections through the other's separate ten-millisecond failure path. The security comments on the intervals describe a node-wide DoS defense, but enabling v2 creates two independent instances of that defense.

F-232268 🟡 Medium
V2 bootstrap bypasses outbound pacing

The configured v2 dialer spawns every maintain_v2_peer task immediately, and each task performs its first connect before sleeping. There is no shared pacing gate or per-index staggering, so all available outbound slots can begin QUIC, TLS, and application handshakes in one scheduling window. The legacy initial-peer path explicitly treats MIN_OUTBOUND_PEER_CONNECTION_INTERVAL as a security control and delays each indexed task by a multiple of that interval. V2 peers that fail at similar times then take the same fixed reconnect delay, synchronizing later bursts. An oversized configured list or adversarial configured DNS response can repeatedly trigger these spikes even if outbound capacity itself is made atomic.

F-232269 🟡 Medium
Legacy DNS still blocks v2 startup

Working v2 peers cannot make peer-set initialization complete when all legacy seed DNS lookups fail. The initializer spawns the v2 endpoint, but then unconditionally awaits add_initial_peers before starting the crawler and returning the network service. Legacy initial_peers() awaits DNS before loading its cache, and the shared resolver loops forever when every configured legacy name produces no address. A successful v2 handshake merely queues a client on peerset_tx; it does not release the legacy initialization barrier. Consequently a legacy DNS outage or manipulation prevents startup even when operator-configured numeric v2 endpoints are connected and usable.

F-232271 🟡 Medium
Capacity misses trigger full reconnect backoff

maintain_v2_peer applies the full 119-second reconnection delay after every iteration, including when it did not attempt a connection because shared outbound capacity was full. If capacity opens just after that check, the configured v2 peer remains disconnected for the whole delay instead of competing for the slot. The independently running legacy crawler can consume the new capacity during this sleep. Repeated connection churn can therefore make legacy dials repeatedly win while v2 maintainers are in unconditional backoff. The behavior reduces configured transport diversity even though no remote v2 failure occurred.

F-232272 🟡 Medium
Documented limit silently weakens inbound admission

The public max_connections_per_ip documentation says the setting does not currently limit accepted inbound connections. Initialization now passes that value directly into the single RecentByIp limiter shared by TCP and v2, and both listeners consult it before admitting a public connection. An operator may increase the setting to support several outbound peers behind one NAT while relying on the stated inbound behavior. The same remote IP then gains the larger allowance for recent inbound admissions across both transports. This silently weakens inbound Sybil and connection-flood resistance as a consequence of a documented outbound-oriented configuration change.

F-232280 🟠 High
Concurrent dialers bypass connection limits and can abort the node

The shared outbound counter is not used as an atomic reservation boundary after v2 maintainers were introduced. Each v2 maintainer performs update_count() and track_connection() separately, so concurrent tasks can all observe the final available slot before reserving it. Legacy initial-peer setup runs concurrently with v2 setup and unconditionally calls track_connection(), allowing it to reserve peers after v2 has already consumed capacity. The peer set accepts these excess discovered peers because insertion checks duplicate addresses and per-IP counts, but not the total limit; update_metrics then panics once the limit is exceeded. Since release builds configure panic = "abort", the oversubscription can progress from excess live connections and resource use to process termination.

And one more auto-invalidated finding.

Analyzed five files, diff 73ab5f6...b5486d6.

@mergify

mergify Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 1 protections blocking · waiting on ⛓️ dependency

Protection Waiting on
🔴 ⛓️ Depends-On Requirements ⛓️ dependency

🔴 ⛓️ Depends-On Requirements

Waiting for

This rule is failing.

Requirement based on the presence of Depends-On in the body of the pull request

@arya2
arya2 force-pushed the p2p-v2-5-peer-set-integration branch from b5486d6 to c8c2e6d Compare August 17, 2026 20:51
@arya2 arya2 linked an issue Aug 17, 2026 that may be closed by this pull request
12 tasks
arya2 added 7 commits August 17, 2026 19:27
A node that only dials version 2 to its configured peers cannot grow the
version 2 network: relayed addresses carry no indication of which
transports their peer accepts, so every peer learned from the network is
dialed over the legacy protocol forever.

Learn reachability by dialing. The peer book records which transports
each address has answered on, the crawler dials version 2 wherever a peer
is known to accept it, and a small share of peers with unknown version 2
reachability are probed before their legacy dial. A probe is one QUIC
initial packet to an address that already passed the book's filters, in
place of the TCP SYN the node would have sent anyway, and every failure
falls back to the legacy transport immediately -- so a node is never
worse off than a legacy-only node.

Requests that only version 2 peers can answer are marked, so the peer set
routes them to a version 2 peer instead of spending the request on a
legacy peer that would refuse it.

Reachability is keyed by address, not by transport: a peer banned over
one transport cannot return over the other, and cannot occupy two address
book entries by being reachable two ways.

Reachability is not persisted across restarts, and is learned only from
outbound dials. book/src/dev/dual-protocol-networking.md records that and
the rest of what is left before the network upgrade.
… watch cells

Zebra's concurrency policy forbids mutexes in new code. Replace every
mutex the version 2 transport work added:

- SharedConnectionCounter now counts with atomic slots; its trackers
  hold a slot guard, and try_track_connection reserves a slot in one
  atomic update, so the transports' accept loops cannot overshoot a
  shared limit.
- HandshakeNonces stores the nonce set in a watch channel used as a
  cell: registration mutates inside send_modify, and the
  self-connection check reads a borrow snapshot.
- The v2 connection's shared per-connection caches (pushed and sent
  blocks, reconstruction and mempool mirrors, address cache, token
  bucket, and the announcement singleton set) live in watch-backed
  SharedCell values with the same atomic-update discipline.
…imiter

The two transports' accept loops checked the shared inbound counter and
then reserved a slot as separate steps, so both could pass the check at
the last free slot and overshoot the configured limit; and each kept its
own per-IP limiter, letting one IP hold max_connections_per_ip
connections on TCP plus the same again on QUIC. Reserve the slot in one
atomic counter update, and share one watch-backed per-IP limiter between
the listeners.
…v2 handshakes

Outbound v2 requests run concurrently, so a peer that stalled on one
request while several were in flight would trip the consecutive-timeout
disconnect when their timers fired together, even if it had just
answered other requests. Snapshot the response count when each request
is sent, and only count a timeout when no response at all arrived while
it was outstanding.

Also supervise the accept loop's handshake tasks like the legacy
listener, so their panics stop the node instead of silently reducing
connectivity; record the terminal connection-attempt metric for both
stages of inbound v2 connections; and yield after each accepted
connection, matching the legacy listener's starvation protection.
…ban and gossip gaps

The blocking actor thread inhibited tokio auto-advance, hanging
paused-clock tests. Sub-threshold misbehavior reports re-created banned
peers' book entries, expired bans never left the published bans watch,
gossip-bucket admission never ran on the production intake path, ban
removals skipped the recently-live refresh, and bucket evictions cleared
the per-IP most-recent cache for entries that were not cached.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tracking: v2 P2P protocol (QUIC) and known-hash sync with spentness hints

1 participant