Allow worker-saturation=0 to disable worker queuing#9173
Open
Allow worker-saturation=0 to disable worker queuing#9173
Conversation
Fixes dask#9116 This change allows setting `worker-saturation` to 0, which disables worker-side queuing entirely. Tasks are only sent to completely idle workers (no tasks in processing queue, excluding seceded tasks). This is useful for workloads with long-running tasks where you want to avoid head-of-line blocking - shorter tasks won't get stuck behind long-running tasks in worker queues. Changes: - Modified `_task_slots_available()` to handle saturation_factor=0 by only allowing task assignment when worker is completely idle - Updated validation to accept values >= 0 (previously required > 0) - Updated schema to allow minimum: 0 (previously exclusiveMinimum: 0) - Added documentation for saturation=0 special case - Added test case for saturation=0.0 - Added test to ensure negative values are still rejected 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Contributor
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 26 files - 1 26 suites - 1 9h 47m 11s ⏱️ - 37m 1s For more details on these failures, see this check. Results for commit c066abb. ± Comparison against base commit 5da04d0. This pull request removes 1 and adds 2 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
The previous implementation returned 0 for idle workers when saturation_factor=0, which caused _worker_full() to incorrectly mark idle workers as full (since it checks `<= 0`). This prevented any tasks from being assigned, causing test timeouts. Changed _task_slots_available() to return 1 for idle workers when saturation_factor=0, allowing idle workers to accept tasks while still preventing queuing on busy workers. Behavior with saturation_factor=0: - Idle worker: slots = 1 (can accept tasks) - Worker with 1 task: slots = 0 (marked as full) - Worker with seceded task: slots = 1 (can accept tasks) Fixes test_saturation_factor[0.0-expected_task_counts5] timeout. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
When saturation_factor=0, workers should accept tasks up to their thread count without queuing. The previous implementation returned only 1 slot for idle workers, causing multi-threaded workers to underutilize threads. Now returns ws.nthreads - active_tasks, allowing workers to fill all idle threads while preventing queuing (head-of-line blocking). Fixes test_saturation_factor[0.0] which expects (2, 1) task distribution for workers with (2, 1) threads respectively. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR allows setting
worker-saturationto0, which disables worker-side queuing entirely. Tasks are only sent to completely idle workers (no tasks in processing queue, excluding seceded tasks).Fixes #9116
Motivation
Users running long-running tasks (minutes to hours) were experiencing head-of-line blocking where shorter tasks get stuck behind very long tasks in worker queues. With all workers saturated, work-stealing doesn't help.
Previously, the minimum worker queue length was hardcoded to 1 via
max(..., 1)in_task_slots_available(), regardless of how low theworker-saturationsetting was configured.Changes
Modified
_task_slots_available()to handlesaturation_factor=0as a special casesaturation_factor == 0, returns0 - (len(ws.processing) - len(ws.long_running))secede()) don't count as occupying slotsUpdated validation logic in
SchedulerState.__init__():WORKER_SATURATION <= 0toWORKER_SATURATION < 0Updated schema (
distributed-schema.yaml):exclusiveMinimum: 0tominimum: 00: "Only send tasks to completely idle workers (no queuing)"Added test cases:
(0.0, (2, 1))totest_saturation_factorparametrizationBehavior
Before
After
Examples
0 - 0 = 00 - 1 = -10 - 0 = 0Usage
Testing
Checklist
🤖 Generated with Claude Code