Tell workers to kill operations the scheduler no longer has executing on them - #2693
Open
amankrx wants to merge 2 commits into
Open
Tell workers to kill operations the scheduler no longer has executing on them#2693amankrx wants to merge 2 commits into
amankrx wants to merge 2 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
amankrx
requested review from
MarcusSorealheis,
corcillo and
palfrey
and removed request for
MarcusSorealheis
August 20, 2026 01:57
Member
Author
|
/build-image nativelink-worker-init |
Member
Author
|
/build-image nativelink |
Contributor
|
Image built and pushed! |
Contributor
|
Image built and pushed! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
The scheduler never sent KillOperationRequest (only tests constructed one), so when an operation was finished server-side (client timeout, cancellation, execution deadline, requeue) the worker kept running it and holding its slot until the process ended on its own. This adds a periodic pass that asks the state manager whether each running operation is still executing on its worker and tells the worker to kill it if not. The worker's later report for a killed operation is not forwarded, so the kill's error cannot burn a retry on a re-queued copy.
How was this verified?
Unit tests only. is_executing_on_worker is table-tested across queued, assigned, requeued, reassigned, completed and unknown. A scheduler-level test drives a real client timeout while a single-slot worker holds the operation and checks that exactly one kill is sent, the worker's report returns Ok, and the slot frees for the next action; without the change, no kill is sent and the report fails with "already completed". A third test drops the worker channel and checks the worker is evicted when the kill cannot be delivered. Not yet run against a real deployment.
Risk
Behaviour change: an operation whose client vanished is now killed instead of running to completion and warming the AC. Every 5s each running operation costs one state-manager lookup, which is one Redis GET on store-backed deployments. New trait method on WorkerStateManager and WorkerScheduler. Wire format unchanged, the message already existed.
This change is