How to submit many jobs

[goxeq]

In the project description you write that the overhead per task is below 100 us.

However, I cannot find how to submit many tasks that fast. When I use for loop in bash:
for i in {1..2000} ; do hq submit -- command ; done
it only does ~100 hq submit per second.

What is the proper way?

[spirali]

For submitting a larger number of tasks, please use Task arrays:

https://it4innovations.github.io/hyperqueue/stable/jobs/arrays/

Your example can be submitted as:

hq submit --array 1-2000 -- command

[Kobzol]

Hi, thanks for the question.

There is a distinction between jobs and tasks in HyperQueue.

Task is a unit of computation, usually it is the invocation of a single binary executable, but it can also be e.g. the execution of a Python function (in our Python API).

Job corresponds to a computational workflow that can perform many tasks, in fact it represents a task graph (although there is currently no way of expressing dependencies between tasks using the CLI). Jobs thus serve to group multiple tasks, in order to have a better CLI visualisation and also some kind of boundary between unrelated tasks. Jobs are relatively heavyweight and their overhead is much higher than for individual tasks.

If you want to run a lot of (related) tasks, you can use Task arrays. The idea is that if you run the same command (binary), but with different input parameters (this is a very common use case), you can just create a single job with e.g. 2000 tasks, and provide a different input parameters to each task. The input parameters can currently be one of the following three things:

1. Integer IDs (e.g. run tasks with IDs 1234-3234)
2. Individual lines from a file
3. Items from a JSON array

You can find more details in the provided link about task arrays.