Improve block proposal cancellation to (best effort) avoid concurrency issues#10219
Open
fab-10 wants to merge 2 commits intobesu-eth:mainfrom
Open
Improve block proposal cancellation to (best effort) avoid concurrency issues#10219fab-10 wants to merge 2 commits intobesu-eth:mainfrom
fab-10 wants to merge 2 commits intobesu-eth:mainfrom
Conversation
fab-10
changed the title
fab-10
force-pushed
the
fix-cancelled-block-creation-exception
branch
from
f391044 to
cf708bc
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Improves block proposal cancellation behavior to reduce race conditions between transaction selection threads and post-selection world state mutations, and reduces log noise for expected cancellation-related exceptions.
Changes:
- Add a
CountDownLatchto internal tx selection and a sharedwaitForCancellationToBeProcessed(...)helper to best-effort wait for selection threads to stop. - Refine exception handling to avoid rollback on cancellation/interruption paths and to handle plugin-selection cancellation gracefully.
- Demote post-cancellation block creation exceptions in
MergeCoordinatorfrom WARN to INFO and add tests covering these scenarios.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| ethereum/blockcreation/src/main/java/org/hyperledger/besu/ethereum/blockcreation/txselection/BlockTransactionSelector.java | Adds latches + wait helper and refines cancellation/interrupt/exception handling during selection. |
| ethereum/blockcreation/src/test/java/org/hyperledger/besu/ethereum/blockcreation/AbstractBlockTransactionSelectorTest.java | Adds tests for external cancellation during plugin selection and waiting for internal selection thread completion. |
| consensus/merge/src/main/java/org/hyperledger/besu/consensus/merge/blockcreation/MergeCoordinator.java | Logs post-cancellation exceptions at INFO with guidance; keeps WARN for non-cancelled failures. |
| consensus/merge/src/test/java/org/hyperledger/besu/consensus/merge/blockcreation/MergeCoordinatorTest.java | Adds a regression test ensuring post-cancellation exceptions don’t escape the background task. |
...n/java/org/hyperledger/besu/ethereum/blockcreation/txselection/BlockTransactionSelector.java
Show resolved
Hide resolved
...n/java/org/hyperledger/besu/ethereum/blockcreation/txselection/BlockTransactionSelector.java
Show resolved
Hide resolved
...t/java/org/hyperledger/besu/ethereum/blockcreation/AbstractBlockTransactionSelectorTest.java
Outdated
Show resolved
Hide resolved
...t/java/org/hyperledger/besu/ethereum/blockcreation/AbstractBlockTransactionSelectorTest.java
Outdated
Show resolved
Hide resolved
...merge/src/main/java/org/hyperledger/besu/consensus/merge/blockcreation/MergeCoordinator.java
Outdated
Show resolved
Hide resolved
…ection When block creation is cancelled or times out, the selection thread may still be running briefly. This change adds a CountDownLatch to internal tx selection (mirroring the existing plugin selection mechanism) and extracts a shared waitForCancellationToBeProcessed method that correctly handles negative remaining-time values and logs the outcome of the wait. Exception handling in both selection phases is split by type so that rollback() is only called for ExecutionException, where the selection thread is guaranteed to have finished. CancellationException and InterruptedException no longer trigger a rollback, removing a potential race on shared world state. In MergeCoordinator, exceptions thrown after a cancellation are now logged at INFO with guidance to report if unexpected, rather than at WARN, reducing noise from the expected concurrency edge cases during block proposal cancellation. Signed-off-by: Fabio Di Fabio <[email protected]>
fab-10
force-pushed
the
fix-cancelled-block-creation-exception
branch
from
cf708bc to
93f4a62
Compare
- Add test verifying CancellationException during plugin selection is handled gracefully (no exception propagated to caller) - Add test verifying internal selection CountDownLatch causes buildTransactionListForBlock() to wait for the selection thread - Add test verifying Throwable thrown after block creation cancellation is handled gracefully (logged at INFO, not propagated) - Remove early return from timeLimitedSelection when isCancelled is true: the guard was causing validPendingTransactionIsNotIncludedIf SelectionCancelled to fail because evaluatePendingTransaction (which marks each tx as SELECTION_CANCELLED) was never reached; the check is unnecessary since evaluatePendingTransaction already handles isCancelled on every iteration without touching world state Signed-off-by: Fabio Di Fabio <[email protected]>
fab-10
force-pushed
the
fix-cancelled-block-creation-exception
branch
from
93f4a62 to
4c40b4f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Today I spent some time reviewing the exception reported below, and it ended up to be an interesting investigation, that led to an improvement of the cancellation path to avoid (as much as possible) this concurrency issue.
Usually, like in the reported case, the exception is not an issue, since there were previous proposals available to return, but in the case the cancellation happens on the very first block creation iteration, then it could result in an empty proposal returned, instead of something better.
When block creation is cancelled or times out, the selection thread may still be running briefly after
FutureTask.get()returns. Without proper synchronisation, post-selection steps (withdrawals, EL requests, rewards processing) can race against the still-running thread on the shared world state.Exactly what happened in the above exception, the race condition between the withdrawals processing and the cancelled but still running tx selection thread.
CountDownLatchto internal tx selection — mirrors the existing mechanism in plugin selection; thefinallyblock counts down after the selection loop completes, giving a reliable signal that the thread has stopped.waitForCancellationToBeProcessed— shared helper called after both plugin and internal selection phases when the latch is non-zero. Correctly handles a negative remaining-time value (logs and returns immediately instead of silently passing a negative timeout toCountDownLatch.await) and logs the outcome of the wait.rollback()is now only called forExecutionException, where the callable has already thrown and thefinallyblock is guaranteed to have run (latch at 0, selection thread stopped).CancellationExceptionandInterruptedExceptionno longer trigger a rollback, removing a potential race where the selection thread could still be mutatingselectionPendingActionsand the world state updaters.MergeCoordinator— exceptions thrown afterisBlockCreationCancelledis true are now logged at INFO with a message guiding the user to report if unexpected, rather than at WARN, reducing noise from the expected concurrency edge cases during block proposal cancellation.Test plan
BlockTransactionSelectorandMergeCoordinatorpassengine_forkchoiceUpdatedfollowed by a new payload)🤖 Generated with Claude Code