Hyperqueue at Metacentrum (autofs, heterogenous HPC system, /home is symlink ) together with own on-prem HPC

[jose-d]

Goal of this disscusion thread:

• feed search engine with working example
• collect feedback and experience from other users of CESNET Metacentrum
• show how HQ is helping to do science at Institute of Physics, ASCR :)

Our usecase for HQ

We aim to create unifying layer on top of our shared compute resource provider, Metacentrum/CESNET and our on-prem HPC cluster.

Our cluster is firmly in our hands, and our users are provided with extensive scientific support. But we understand the benefits of letting our users expand to external providers, like Cesnet Metacentrum, IT4i, Lumi, etc, especially in cases when the application is stable and required tooling is already well understood by user, and we need only .. CPUs.

Without hq, users manually distribute their workloads between multiple HPC providers, writing both PBS and Slurm scripts .. and I think that is the right moment to bring hq into game:

Problem we observed

So lets submit one hq worker at metacentrum, and one hq worker at our on-prem system, and submit some dummy job to them (eg. hostname, or sleep 1, etc. ). So taking the quick start .... and .. nope:

(BOOKWORM)jose@tarkil:~$ ./tools/hq job info 1
+----------------------+--------------------------------------------------------+
| ID                   | 1                                                      |
| Name                 | sleep                                                  |
| State                | FAILED                                                 |
| Tasks                | 1; Ids: 0                                              |
| Workers              |                                                        |
| Resources            | cpus: 1 compact                                        |
| Priority             | 0                                                      |
| Command              | sleep                                                  |
|                      | 1                                                      |
| Stdout               | /auto/vestec1-elixir/home/jose/job-1/%{TASK_ID}.stdout |
| Stderr               | /auto/vestec1-elixir/home/jose/job-1/%{TASK_ID}.stderr |
| Environment          |                                                        |
| Working directory    | /auto/vestec1-elixir/home/jose                         |
| Task time limit      | None                                                   |
| Crash limit          | 5                                                      |
| Submission date      | 2024-07-23 09:36:54 UTC                                |
| Submission directory | /auto/vestec1-elixir/home/jose                         |
| Makespan             | 2ms                                                    |
+----------------------+--------------------------------------------------------+
+---------+--------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Task ID | Worker | Error                                                                                                                                                                                          |
+---------+--------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 0       |        | Error: Cannot create stdout directory at File { path: "/auto/vestec1-elixir/home/jose/job-1/0.stdout", on_close: None }: Os { code: 13, kind: PermissionDenied, message: "Permission denied" } |
+---------+--------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 tasks failed.
(BOOKWORM)jose@tarkil:~$

My understanding of this problem

• Metacentrum is federation of multiple clusters. When job is submitted to the scheduler (PBS), by default, job can be executed at any cluster
• Different clusters have /home implemented as symlink to different directories
• also $HOME env var of our metacentrum worker is different than $HOME of our on-prem worker

eg. cluster skirit:

$ ll /home
lrwxrwxrwx 1 root root 20 čec 21  2020 /home -> /storage/praha1/home/

eg. cluster tarkil:

$ ls -lah /home
lrwxrwxrwx 1 root root 19 15. zář  2020 /home -> /storage/brno2/home

and we need to go deeper, because at least for praha1, that's another symlink, but that's already real directory, nice :)

$ ll /storage/praha1/home
lrwxrwxrwx 1 root root 25 úno  3  2021 /storage/praha1/home -> /auto/vestec1-elixir/home/

We can also see, that variable ${HOME} differs across multiple clusters in federation, so if job gets alocation at kirke4.meta.zcu.cz, $HOME is /storage/plzen1/home/jose where are indeed different files than in /storage/praha1.

That means, that we can not reliably use ${HOME} at all. So...

Workaround / solution:

in hq worker launchers, export REALHOME pointing to system-specific directory where our app.sh is located:

Metacentrum PBS worker batch script:

#!/bin/bash
#PBS -N testing_HQ
#PBS -l select=1:ncpus=4:mem=1gb
#PBS -l walltime=0:30:00

hq_dir='/storage/praha1/home/jose/tools/'
credsdir="/storage/praha1/home/jose/projects/2024_07_22__distributed_hyperqueue/wd"

export REALHOME=/storage/praha1/home/jose

${hq_dir}/hq worker --server-dir=${credsdir} start --manager pbs

Koios (one of our on-prem systems) Slurm batch script:

#!/bin/bash
  
#SBATCH --job-name=hq_worker
#SBATCH --time=00:33:33
#SBATCH --partition=cpu
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --threads-per-core=1

hq_dir='/home/jose/tools/hq'
credsdir="/home/jose/projects/2024_07_18__hyperqueue"

export REALHOME=/home/jose

${hq_dir}/hq worker --server-dir=${credsdir} start --manager slurm

submit app.sh job:

./tools/hq submit --array=1-30 --log=aa.log --cwd=/tmp bash -c '$REALHOME/app.sh'

Note that we use aa.log to stream the logs over hq and set cwd to /tmp.

Without setting cwd to /tmp the job fails due to /home and $HOME concept at Metacentrum.

[Kobzol]

Hi, this is indeed an interesting use-case. As you have probably noticed, HyperQueue by default assumes a shared filesystem. It's not that it cannot work without it, but it makes certain decisions that are designed to make the life of users on clusters with a shared filesystem much easier (but this also conversely makes the life of users on clusters without a shared filesystem harder) :)

Regarding the home directory, I don't think there's that much that HQ can do for you. If you are sharing the connection information through the filesystem automatically, you will need to remap the server directory paths somehow, as you have shown. Note that there is also support for running HQ on clusters without any filesystem sharing at all (https://it4innovations.github.io/hyperqueue/stable/deployment/cloud/), although I'm not sure if it will be that useful for your use-case.

What is more annoying is the setting of task-level paths (working directory, stdout and stderr). In theory, it would be possible to just use relative paths for these (e.g. --cwd=foo), and these could then be resolved relatively to the working directory of the worker, which would mostly resolve the issues caused by the filesystem not being shared. However, HQ currently expands these paths eagerly relative to the submission directory, so if you specify --cwd=foo, it will not be set to <worker-cwd>/foo, but to <submit-dir>/foo and that won't help your use-case. It was designed like this explicitly, it's usually much nicer if the data appears in the directory from which you submit, rather than in some random directory where the worker is started (often it is started automatically by the automatic allocator, who knows where :) ). However, for your use-case it's a bit unfortunate.

Currently, you're getting around this by using a directory that is accessible everywhere (/tmp) as a working directory, and by using I/O streaming for avoiding having to deal with stdout and stderr, but that is of course not a great solution.

I think that we could resolve this by adding a new placeholder called e.g. %{WORKER_CWD}, and then you could set --cwd=%{WORKER_CWD}/foo, which would mostly resolve issues related to paths on workers. Let me know what do you think about this, would it help your use-case? (I think that in any case, this would be a useful addition, @spirali what do you think?).

By the way, HQ was mostly designed to run inside a single cluster (with a shared filesystem), as you have probably already noticed. While using it across clusters does partially work (and I know that some people are using it in that way), I wonder if perhaps a different solution for that might be better, e.g. something like Lexis (https://portal.beta.lexis.tech/login, https://www.e-infra.cz/file/76b003888693abdd9a06846a20cd5d00/1222/2_LEXIS_eInfraCZ_2024.pdf) or HEAppE (https://heappe.eu).

In any case, let me know what do you think, or if you have any other suggestions/requests.

[jose-d]

there is also support for running HQ on clusters without any filesystem sharing at all (https://it4innovations.github.io/hyperqueue/stable/deployment/cloud/), although I'm not sure if it will be that useful for your use-case.

without this feature, our scenario would be impossible! It is very helpful, that we can take access.json literally anywhere and connect to any hq server.

so if you specify --cwd=foo, it will not be set to <worker-cwd>/foo, but to <submit-dir>/foo

exactly! and submit-dir is the /{metacentrum-specific}/home/{username}, it is non-existent at our facilities, where we have real /home. We even have users with different usernames at different systems, so then the paths are broken in even more obvious way.

That IMO affects dealing with the stdout and stderr logfiles too - because logfiles directory is derived from <submit-dir> too, right?

I think that we could resolve this by adding a new placeholder called e.g. %{WORKER_CWD}, and then you could set --cwd=%{WORKER_CWD}/foo

yes. This would fix the path to user app in home, not sure if that would fix the stdout and stderr dirs too?

[Kobzol]

without this feature, our scenario would be impossible! It is very helpful, that we can take access.json literally anywhere and connect to any hq server.

I see! Good to know :)

That IMO affects dealing with the stdout and stderr logfiles too - because logfiles directory is derived from too, right?

If you don't use output streaming (so in the default case), then yes, this also affects the location of the stdout/stderr files in the same way. With output streaming, the server writes stdout/stderr into a single file (usually stored on the host node), so there it should be fine.

Note that stdout/stderr paths are already relative to the working directory by default, so just changing --cwd to e.g. %{WORKER_DIR}/%{TASK_ID} should also fix stdout/stderr file paths.

yes. This would fix the path to user app in home, not sure if that would fix the stdout and stderr dirs too?

Hmm, actually it should be the opposite 😆 With %{WORKER_CMD}, you could set the cwd/stdout/stderr paths of a job to be relative to the worker directory, thus resolving the issues with different filesystems on different workers.

This wouldn't help the configuration of the server directory though, because that does not use any placeholders like %{WORKER_CMD}. Users usually have control over starting the worker, so they can configure the right server directory at that time. This is different from submitting jobs, where you don't know beforehand on which worker will a givne task execute.

By the way, if you have a specific workflow/use-case for which you're using sucessfully HyperQueue, and you'd be OK with sharing it with me, I would love to hear about it! You can contact me at [email protected] if you'd like to share some more details. I'm currently trying to gather some real-world usages of HQ so that I can describe them in my PhD thesis :)

[jose-d]

I'm currently trying to gather some real-world usages of HQ so that I can describe them in my PhD thesis :)

sure, my colleague I am supporting ( I'm HPC guy in our team ) does research in area of gravitational waves. The problem he is exploring can be decomposed into many independent sub-problems.

I asked him to take look at this thread and possibly write few sentences about the scientific essence of this code. From ethical reasons I can not speak about scientific concepts myself, especially before publication is out.

But I can tell what we expect we'll gain when using hyperqueue / why we decided to use it:

• unified access to multiple clusters. We have two on-prem clusters + Metacentrum. We're thinking about IT4i. Splitting problem between multiple systems manually is ineffective, indeed in various times the systems have various amount of available nodes, jobs are subject of QoS rules, so by using hq we get something like loadbalancing..
• resiliency - sometimes job array element hangs and is killed. Hq can reschedule it.
• increase the amount of job array elements. Our Slurm can deal with quite large (N00 000) job arrays, PBS@Metacentrum support only small ones.

Well, and all that is possibly answer why we went with hqand not with Lexis or HEAppE.. hq seems to be quite transparent, the concepts are straightforward. After few initial sessions and P-O-C I am sure my colleague will be able to use it himself, tweak the jobscripts, and use it for his future research projects. With Lexis or HEAppE.. it looks we'd need to dedicate some FTE just for that! :)

[lahwaacz]

How do you make hq server really persistent on Metacentrum? Because of Kerberos, each user session is limited to a few hours by default and Kerberos tickets can be renewed only up to 7d without re-entering the password. Even if you run e.g. tmux on a Metacentrum login node, without a valid Kerberos ticket it loses access to PBS and even NFS servers.

So the common advice does not apply here. Should we run hq server on a compute node through PBS with e.g. 720:00:0 walltime? Would networking between hq components even work in that case? There is also some oven node but I don't even know how to use it or if it is suitable for this...

[Kobzol]

If you cannot use login node, one alternative could be to start the server at a cloud partition(https://it4innovations.github.io/hyperqueue/stable/deployment/cloud/), if there is such a partition available, of course.

Running the server on a long running PBS job on a compute node is also a possibility, although it might be a bit tricky to automate. Whether networking in this use-case will work depends on the cluster, on some of them it's possible to connect between different compute jobs, on others it's not.

[jose-d]

TL/DR:

How do you make hq server really persistent

impossible. 1month is the maximum.

Longer version:

@lahwaacz hi, yup, we ( me and colleagues at FZU) use oven node.
Over time, we used two ways:

(1) naive way - that's how we started:

• to log in - using plain good old ssh - to oven, start screen session there, set environment variables, and in screen do the hq server xxxxxxx stuff - this is nice for testing but annoying to do more than twice.

(2) better way ™

• run hq server as PBS job in queue "oven" - jobs there live longer, and their Kerberos tickets somehow too.
  The job could be similar to this snipped I found in my home directory, I don'T remember if it is 100% correct so do not take it too litarally..:

#!/bin/bash
#PBS -N HQ_server_test
#PBS -l select=1:ncpus=1:mem=1gb
#PBS -l walltime=720:00:00
#PBS -q oven

hq_dir='/storage/praha1/home/jose/tools/'
credsdir="/storage/praha1/home/jose/projects/2024_07_22__distributed_hyperqueue/wd"

export REALHOME=/storage/praha1/home/${USER}
mkdir -p ${REALHOME}/logs

${hq_dir}/hq server start --access-file=${credsdir}/myaccess.json &> ${REALHOME}/logs/hq.log
(BOOKWORM)jose@tarkil:~/projects/2024_07_22__distributed_hyperqueue$

I guess from this script it is obvious that we were trying to workaround the issue with heterogenous architecture of Metacentrum, having unpredictable $HOME. Our job was CPU-heavy and doing almost zero IO, so we were not considering usage of local scratch, using geographically more suitable storages, etc...

Experience gathered:

• 720 hours is so long, that "my" users often forgot it, and were surprised after 720 hours / 1month what happened :))
• oven is not behind firewall, so if you use high-enough port, you can connect from other clusters to Oven server.

[lahwaacz]

@jose-d Thanks for your reply, it is very useful! In the end, I've ended up doing something different:

1. Start a screen session on my workstation (i.e. not in metacentrum). From there, I run kinit -r 7d username@META (this prompts for a password) to initialize Kerberos and then ssh to a metacentrum login node (in as many shells as needed).
2. The kerberos ticket lasts just a few hours, but can be renewed with kinit -R -r 7d (without prompting for password until 7d since the last prompt). I've automated this with the following script:

   #!/bin/bash
   
   set -eu
   
   function renew_all()
   {
       for krb_file in /tmp/krb5cc_*; do
           # The -O file option tests if file exists and is owned by the effective user ID.
           # The -s file option tests if file exists and has a size greater than zero.
           if [[ -O "$krb_file" ]] && [[ -s "$krb_file" ]]; then
               echo "==> Attempting to renew kerberos ticket in file $krb_file"
               KRB5CCNAME=FILE:"$krb_file" kinit -R -r 7d || true
               echo "==> List cached kerberos tickets:"
               KRB5CCNAME=FILE:"$krb_file" klist
           fi
       done
   }
   
   while true; do
       renew_all
       timeout=3600
       echo
       echo "$(date): sleeping for $timeout seconds ..."
       sleep "$timeout"
   done

3. Kerberos is used also for authentication between metacentrum components, e.g. for connections to the PBS server. Hence, tickets need to be renewed on the workstation as well as on the login node, so I run the script twice. I think that the ticket on the workstation must be valid in order to successfully renew tickets on the login node.
4. When the 7d period is coming to an end, I just manually run kinit -r 7d username@META again on the workstation, enter the password and get another week 😄

So this is not really persistent and if the network went down, the session would just crash, but it's reliable enough for me. There is also some manual intervention needed, but I guess it is better than forgetting what I did 720h ago 😅

As for HQ, the server keeps running on the login node as a background process in one of the ssh shells running in the screen session. Our application keeps waiting until all tasks are computed, so I'm not likely to accidentally close the session while something is in progress.