Make MPITaskScheduler prioritize large tasks
#3805
Conversation
benclifford
changed the title
MPITaskScheduler to prioritize large tasksMPITaskScheduler prioritize large tasks
| task_package = {"task_id": task_id, "buffer": mock_task_buffer} | ||
| scheduler.put_task(task_package) | ||
|
|
||
| # Confirm that the tasks are sorted in decreasing order |
There was a problem hiding this comment.
this test checks that tests coming out of the queue are sorted in priority order (that is, is the priority part of PrioritizedTask implemented correctly).
It doesn't check that they are sorted in num_nodes order, I think? (for example, introduce a bug into the calculation at line 200 and I think this test will still pass, even though the code is buggy?)
There was a problem hiding this comment.
Good point, I'll update this test.
There was a problem hiding this comment.
I just pushed the updated test that tests for the backlog queue returning tasks in decreasing order of nodes requested.
yadudoc
force-pushed
the
fix_mpi_priority
branch
from
75b46bf to
a80264f
Compare
yadudoc
force-pushed
the
fix_mpi_priority
branch
from
a80264f to
55d46e1
Compare
|
what is happening with the base of this PR? this should be going to |
I think this is not the right change description. |
Description
The python
PriorityQueueprioritizes items with smaller priority values, which leads us to our currentMPITaskSchedulerprioritizing MPI tasks with fewer nodes requested since the nodes requested is used as the priority. This PR changes the ordering to prefer larger jobs.Changed Behaviour
MPI Tasks with larger node requests are prioritized by the manager. Please note that as @WardLT pointed out in the comments on #3783, the greedy scheduling that we use can lead to larger tasks getting delayed in favor of smaller tasks that can get scheduled immediately.
These are change split from #3783 to keep the PR concise.
This is split 3 of 3.
Type of change
Choose which options apply, and delete the ones which do not apply.