Timeboxed push for simplifying work stealing #6993
Description
Work stealing is a known source of problems. It's current implementation is overly complex and has a couple of known problems, some of which are almost fixed.
Specifically, I propose to time box this to ~1-2weeks and try to wrap up a few known issues while pushing for a drastic simplification of the implementation. Once the dust settles, we can reevaluate how this feature has to evolve.
- Remove "steal from public". I strongly believe this is responsible for all the aggressiveness we're seeing, e.g. Avoid load balancing when there is plenty of load #5243. This may need an adjustment to how we define saturated workers, see also https://github.com/dask/distributed/pull/6614/files/36a60a5e358ea2a5d16597651126ac5892203b01#r952608704
- Fix dashboard Improve work stealing for scaling situations #4920 (comment)
- Finish Improve work stealing for scaling situations #4920
- Improve measurements of bandwidth Allow stealing of fast tasks in some situations #6115 (comment)
- Potentially revisit logic about cost_multiplier + level
The short to mid term target of this effort should be to reduce the number of steal requests drastically such that we can afford spending more time on "good" decisions (e.g. reusing the actual scheduler decide_worker logic or something even better)
Even if we want to get rid of work stealing entirely, there is some need for it to balance inhomogeneous workloads and allow cluster upscaling, see #6600 The most valuable component of the current implementation is the handshake mechanism move_task_request / move_task_confirm that ensures consistent transitions without recomputing a key. I believe by tearing down the infrastructure around this handshake piece by piece we can iterate towards a more stable and maintainable implementation.
Previously I approached changes to this logic very carefully due to the lack of repeatable benchmarks. Therefore, I suggest that this effort should utilize benchmarks in coiled-runtime to the best of our abilities.