Skip to content

Implement aggregation in pipeline service#10202

Open
mirgee wants to merge 2 commits intobesu-eth:mainfrom
mirgee:snap-pipeline-aggregation
Open

Implement aggregation in pipeline service#10202
mirgee wants to merge 2 commits intobesu-eth:mainfrom
mirgee:snap-pipeline-aggregation

Conversation

@mirgee
Copy link
Copy Markdown
Contributor

@mirgee mirgee commented Apr 8, 2026

Currently BlockAccessListDataRequest#doPersist is called for each batch of responses corresponding to the batched BAL request GetBlockAccessListsMessage.

Instead of committing those changes for each batch, this PR adds an aggregation pipe to the pipeline, which aggregates all the batches into a single list and commits the corresponding changes to the flat database in one atomic operation. This allows us to compare state of each account in the BALs with the account trie (which by that point is already healed).

TODO:

  • It is possible to avoid unchecked casts and rely on compile-time type safety by making inSingleBatch available only on PipelineBuilders with pipeEnd being a ReadPipe<List<T>> (e,g, those created by inBatches), but it might require a more invasive change. We will do this later

@mirgee mirgee force-pushed the snap-pipeline-aggregation branch from a442b48 to ba1aa2a Compare April 8, 2026 20:04
Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>
@mirgee mirgee force-pushed the snap-pipeline-aggregation branch from ba1aa2a to fb1c122 Compare April 9, 2026 09:22
Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>
@mirgee mirgee marked this pull request as ready for review April 9, 2026 13:23
@mirgee mirgee requested a review from matkt April 9, 2026 14:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant