Prioritize tasks from near-complete jobs

[lahwaacz]

I have the following job info now (with some values redacted):

+----------------------+--------------------------------------------+
| ID                   | 1                                          |
| Name                 |                                            |
| State                | [########################################] |
|                      | FINISHED (3593)                            |
| Session              | closed                                     |
| Tasks                | 3593; Ids: 0-3592                          |
| Workers              | … (38 total)                      |
| Submission date      | 2025-03-27 18:39:12 UTC                    |
| Submission directory | /home/user/somewhere    |
| Makespan             | 8days 13h 54m                              |
+----------------------+--------------------------------------------+
+----------------------+--------------------------------------------+
| ID                   | 2                                          |
| Name                 |                                            |
| State                | [########################################] |
|                      | FINISHED (1382)                            |
|                      | WAITING (27)                               |
| Session              | closed                                     |
| Tasks                | 1409; Ids: 0-1408                          |
| Workers              | … (18 total)                      |
| Submission date      | 2025-04-04 20:49:27 UTC                    |
| Submission directory | /home/user/somewhere    |
| Makespan             | 3days 19h 16m                              |
+----------------------+--------------------------------------------+
+----------------------+--------------------------------------------+
| ID                   | 3                                          |
| Name                 |                                            |
| State                | [########################################] |
|                      | FINISHED (1390)                            |
|                      | WAITING (20)                               |
| Session              | closed                                     |
| Tasks                | 1410; Ids: 0-1409                          |
| Workers              | … (17 total)                      |
| Submission date      | 2025-04-04 21:04:02 UTC                    |
| Submission directory | /home/user/somewhere    |
| Makespan             | 3days 19h 2m                               |
+----------------------+--------------------------------------------+
+----------------------+--------------------------------------------+
| ID                   | 4                                          |
| Name                 |                                            |
| State                | [#######.................................] |
|                      | RUNNING (111)                              |
|                      | FINISHED (390)                             |
|                      | WAITING (2929)                             |
| Session              | closed                                     |
| Tasks                | 3430; Ids: 0-3429                          |
| Workers              | … (24 total)                      |
| Submission date      | 2025-04-07 05:53:59 UTC                    |
| Submission directory | /home/user/somewhere    |
| Makespan             | 1day 10h 12m                               |
+----------------------+--------------------------------------------+

It turns out that HQ keeps scheduling tasks from job 4 and jobs 2-3 will have to wait until their tasks get scheduled. In this case, ideally, I would like to have the near-complete jobs finish as soon as possible.

I could change the priority of my tasks (which are all equal right now) but there are two problems:

1. In the Python API, priority can be set for individual tasks rather than jobs. So if I wanted to raise priority of some tasks within a job, but also ensure that each job with smaller ID has higher priority than any job with higher ID, I would have to do some tricky math 😄 Instead, it would be better if the HQ scheduler considered (job-priority, task-priority) as a tuple.
2. In some cases, depending on the tasks and number of workers, the "near complete" characteristic might shift from jobs 2-3 to job 4 so maybe this factor should be considered by the scheduler dynamically. What do you think? Are there any scheduling options already?

[spirali]

This situation is unfortunate and HQ scheduler should definitely avoid this situation even without user tweaking.

We already supports dynamic change of priorities but it is now used only for resource management (e.g. when we get in the situation where few gpu workers are available, gpu tasks are prioritized). But it is implemented in the inner scheduler that is not aware of jobs and see all tasks equally.

I like both ideas, but I have to think about it more, because there are lots of nuances in priorities, but it is solvable.

[spirali]

FYI I have spent some time on this problem and I have found some other shortcomings of the current internal priority assignments. I will try to fix it (at least partially) next week.

[spirali]

Update: I did some simple tests and it seems that part of the problem is fixable while not being too disruptive. But the solution is conflicting with a big change that I am currently preparing. I hope that I will finally merge the change next week and then I will fix this.

[spirali]

#899 should fix the problem.

[lahwaacz]

@spirali Could you assign even a bit higher priority to the near-complete jobs? Right now, after the fix in #899, I have the following jobs:

The jobs were started in 2 batches: first I started the first 5 jobs and after a few days the last 4 jobs. I would like if some tasks from jobs 1 and 3 were started before the 258 tasks in job 6.

[spirali]

Thanks a lot! I hope I will have time to look at it within a week.

[lahwaacz]

@spirali Any progress? This is a very frustrating issue for me, because when all jobs are near complete, the workers don't have enough work, but when I add more jobs, they prevent the timely completion of the old jobs.

[spirali]

Sorry for delay. Last week I had some obligations. I still need to finish some non-programming stuff, but I hope to return to normal work today. This is top item in my TODO list.

[spirali]

So the problem seems to be following:

In Job 6 remains two unfinished tasks. Task 6@55 is ready, the other unfinished task depends on 6@55.
Task 6@55 is computed on a worker 1860 that is computing 8@5976. It is a task with lower priority, so this 8@5976 had to get to the worker before 6@55.
Probably 6@55 had some dependencies (so it was enabled later) or it was rescheduled there later when scheduler did load balanacing.
The problem is that 8@5976 is long running (it is running in all 3 snapshots) and 6@55 has to wait.

Even there are other workers, the scheduler did not try to schedule to other workers because:

1. Most of workers are fully utilized (and the server scheduler are happy when cluster is fully computing)
2. There was a new worker that occurred between 1. and 3. snapshot, but balancing in this case is priority-blind and focus only on resource balancing, so same mostly random tasks ends in the new worker.

This problem comes from the original architecture that is focused mainly to schedule 1M+ of very short tasks. This is mostly fixed in the new scheduler; however, the new scheduler is still unfinished.
I will try to fix it in the current scheduler by making "balancing" priority aware. It would fix the problem in these snapshots. However, it may still get in to this situation if there would be no newly arriving workers & whole cluster is in the situation where there is no balancing needs for long time (all workers have something to compute).

[lahwaacz]

I see. I'm not familiar with the scheduler architecture so can't make any suggestions for that, but making the re-scheduling for new workers priority-aware would be probably good enough for me. I have usually many allocations waiting in the PBS queue due to resource limits, so the workers are "recycled" often enough when their time limit ends and a new worker gets started.