repair: add new socket and block id repair service #707

AshwinSekar · 2026-02-02T19:40:12Z

Problem

We need a new socket and logic for the block id based repair.
As a refresher this is the repair algo outlined in the paper:

Summary of Changes

Add new socket and repair service.
Plug from NotarizeFallback (or stronger) certificates. SafeToNotar (the other case where we need to repair blocks) will happen in a follow up.

The service sends 3 types of requests:

Metadata (ParentAndFecSetCount & FecSetRoot) processing responses here in block_id_repair_service
ShredForBlockId using repair_service socket. This is sent separately so that we can reuse the sigverify and blockstore insertion logic happening in shred fetch.
Pong

It responds to 3 types of responses:

ParentAndFecSetCount -> Potentially kick off repair for the parent, request FecSetRoot for all the fec sets
FecSetRoot -> request ShredForBlockId for all shreds in this fec set
Ping -> Send Pong

Because of the expected low throughput of this service (until we turn off eager repair), I opted for a simple approach processing responses and sending requests all from one thread.

Couple of useful things:

Responder logic is in repair_handler.rs
OutstandingRequests verifies responses (check merkle proofs etc.)
When a block becomes full in blockstore, the DMR is atomically calculated and inserted with the shred batch. This is used for location lookups.

AshwinSekar · 2026-02-02T21:43:08Z

core/src/repair/block_id_repair_service.rs

+
+                let mut last_stats_report = Instant::now();
+                // throttle starts at 1024 responses => 1 second of compute
+                let mut throttle = DynamicPacketToProcessThreshold::default();


i know this is a really dumb throttle, but it's what eager repair uses 🤷
ideally we should be throttling based on sender but there was also pushback when I suggested we use quic here.

AshwinSekar · 2026-02-02T21:48:14Z

core/src/repair/serve_repair.rs


+    /// Similar to [`Self::repair_request`] but for [`BlockIdRepairType`] requests.
+    /// Uses stake-weighted peer selection rather than cluster_slots weights.
+    pub(crate) fn block_id_repair_request(


no ❌ EpochSlots

AshwinSekar · 2026-02-02T21:50:27Z

core/src/repair/block_id_repair_service.rs

+
+type OutstandingBlockIdRepairs = OutstandingRequests<BlockIdRepairType>;
+
+const MAX_REPAIR_REQUESTS_PER_ITERATION: usize = 200;


🤷 can adjust this based on tuning, repair service uses 512 per 1ms sleep

512 only really needed during restart. Seems like 200 should be plenty in most cases

AshwinSekar · 2026-02-02T21:50:54Z

core/src/repair/block_id_repair_service.rs

+
+/// We prioritize Pong first (to respond to ping challenges), then requests with
+/// lower slot #s, and then prefer metadata requests before shred requests.
+impl Ord for RepairRequest {


this ordering is because we cap the # of repair requests we send per iteration. Better to repair stuff we can actually replay first

AshwinSekar · 2026-02-02T21:56:13Z

core/src/repair/block_id_repair_service.rs

+        if !shred_socket_batch.is_empty() {
+            let total = shred_socket_batch.len();
+            let _ = batch_send(
+                repair_socket,


We send shred requests via repair socket so they eventually get back to sigverify shreds / window service /blockstore.

Bit annoying to maintain two sockets here but better than copying all that logic over here

bw-solana

Reviewed everything except core/src/repair/block_id_repair_service.rs

Submitting comments so far while I review this giant new file

core/src/tvu.rs

gossip/src/node.rs

ledger/src/blockstore.rs

votor/src/event_handler.rs

core/src/repair/block_id_repair_service/stats.rs

core/src/repair/serve_repair.rs

bw-solana · 2026-02-02T22:20:57Z

core/src/repair/serve_repair.rs

+            peer.pubkey,
+            repair_request
+        );
+        Ok((out, peer.serve_repair))


I thought we were using the new socket for both send and receive of block ID repairs

It's a bit confusing:

Metadata (ParentAndFecSetCount & FecSetRoot) & Pong:

A BlockIdRepairService -> A.block_id_socket.send(B.serve_repair, request) -> B.serve_repair.recv -> B ServeRepairService -> B.serve_repair.send(A.block_id_socket, response) -> A.block_id_socket.recv -> A BlockIdRepairService

ShredForBlockId:

A BlockIdRepairService -> A.repair_socket.send(B.serve_repair, request) -> B.serve_repair.recv -> B ServeRepairService -> B.serve_repair.send(A.repair_socket, response) -> A.repair_socket.recv -> A ShredFetch / ShredSigverify -> A Blockstore insert

Serve repair service just processes incoming requests and sends back responses to the same socket

BlockID socket => send new Metadata/Pong requests and receive responses / Pings
repair socket => send new ShredForBlockId requests and receive shreds

Serve repair socket => Receive any sort of repair request from any socket, send back repair_handler's response to that same socket (or a Ping)

~~In the ShredForBlockId flow, does B send shreds to A's TVU port? Or does it actually go to the repair_socket?~~

I think I have this right now:

block_id_socket - send all meta requests out of this socket. Receive meta responses on this socket

serve_repair - receive all requests (meta/shred) on this socket. Send all responses (meta/shred) out of this this socket

repair_socket - send shreds requests out of this socket. Receive shreds on this socket

bw-solana · 2026-02-03T00:37:57Z

core/src/repair/block_id_repair_service.rs

+
+type OutstandingBlockIdRepairs = OutstandingRequests<BlockIdRepairType>;
+
+const MAX_REPAIR_REQUESTS_PER_ITERATION: usize = 200;


512 only really needed during restart. Seems like 200 should be plenty in most cases

bw-solana · 2026-02-03T00:38:27Z

core/src/repair/block_id_repair_service.rs

+        match self {
+            // Pong is always highest priority and handled separately in Ord,
+            // so this should never be called. Return 0 as a fallback.
+            RepairRequest::Pong { .. } => 0,


maybe RepairRequest::Pong { .. } => unimplemented!("Pong requests do not have a slot"), ?

bw-solana · 2026-02-03T00:41:49Z

core/src/repair/block_id_repair_service.rs

+/// We prioritize Pong first (to respond to ping challenges), then requests with
+/// lower slot #s, and then prefer metadata requests before shred requests.
+impl Ord for RepairRequest {
+    fn cmp(&self, other: &Self) -> std::cmp::Ordering {


if we add more Request types, this is going to be a pain to maintain

bw-solana · 2026-02-03T00:45:05Z

core/src/repair/block_id_repair_service.rs

+            Arc::new(StreamerReceiveStats::new(
+                "block_id_repair_response_receiver",
+            )),
+            Some(Duration::from_millis(1)), // coalesce


have you done any perf measurements? Idk if we really need to coalesce here at all

bw-solana · 2026-02-03T00:48:43Z

core/src/repair/block_id_repair_service.rs

+                                Ok(event) => state.pending_repair_events.push(event),
+                                Err(_) => break,
+                            },
+                            default(Duration::from_secs(1)) => (),


should this be a continue?

bw-solana · 2026-02-03T00:51:55Z

core/src/repair/block_id_repair_service.rs

+                        // for repair events that are currently deferred
+                        select! {
+                            recv(completed_slots_receiver) -> result => match result {
+                                Ok(_) => (),


should we be doing something here?

bw-solana · 2026-02-03T00:59:28Z

core/src/repair/block_id_repair_service.rs

+        // pong has highest priority - must respond to ping challenges immediately
+        match (&self, &other) {
+            (Pong { .. }, Pong { .. }) => return Ordering::Equal,
+            (Pong { .. }, _) => return Ordering::Greater,


is this DoSable? Can I flood pings?

bw-solana · 2026-02-03T01:05:17Z

core/src/repair/block_id_repair_service.rs

+                    }
+                    Some(turbine_block_id) if turbine_block_id != block_id => {
+                        // Turbine has a different block
+                        info!(


should probably make this a warn at least

bw-solana · 2026-02-03T01:09:18Z

core/src/repair/block_id_repair_service.rs

+    /// For shred requests, we check if the shred has been received before retrying.
+    fn retry_timed_out_requests(blockstore: &Blockstore, state: &mut RepairState, now: u64) {
+        // TODO(ashwin): use extract_if when we upstream (rust 1.88+)
+        let mut timed_out = Vec::new();


should we just push directly to state.pending_repair_requests?

AshwinSekar force-pushed the repair branch 5 times, most recently from 78a3dde to e28c2b9 Compare February 2, 2026 21:29

AshwinSekar requested a review from bw-solana February 2, 2026 21:31

AshwinSekar force-pushed the repair branch 2 times, most recently from 6a09c1b to e2ebd5a Compare February 2, 2026 21:56

AshwinSekar commented Feb 2, 2026

View reviewed changes

repair: add new socket and block id repair service

f6fbda7

AshwinSekar force-pushed the repair branch from e2ebd5a to f6fbda7 Compare February 2, 2026 22:16

bw-solana reviewed Feb 2, 2026

View reviewed changes

AshwinSekar added 4 commits February 2, 2026 23:00

pr feedback: remove unecessary clones

fc0e218

pr feedback: handle repair event send errors

48cb69e

pr feedback: add slot accessor

8a3293f

pr feedback: clarify stats

d12ca6b

bw-solana reviewed Feb 3, 2026

View reviewed changes


		type OutstandingBlockIdRepairs = OutstandingRequests<BlockIdRepairType>;

		const MAX_REPAIR_REQUESTS_PER_ITERATION: usize = 200;

repair: add new socket and block id repair service #707

Are you sure you want to change the base?

repair: add new socket and block id repair service #707

Uh oh!

Conversation

AshwinSekar commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Summary of Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bw-solana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AshwinSekar Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AshwinSekar Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bw-solana Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bw-solana Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AshwinSekar commented Feb 2, 2026 •

edited

Loading

AshwinSekar Feb 2, 2026 •

edited

Loading

AshwinSekar Feb 2, 2026 •

edited

Loading

bw-solana Feb 3, 2026 •

edited

Loading

bw-solana Feb 3, 2026 •

edited

Loading