From ec8ea6f73cbcbe5b3a56cb9e42d3d105b56f2814 Mon Sep 17 00:00:00 2001 From: torri Date: Thu, 20 Feb 2025 12:49:33 +0100 Subject: [PATCH 01/10] first info for the cvmfsexec way --- docs/getting_access/non_standard.md | 130 ++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 docs/getting_access/non_standard.md diff --git a/docs/getting_access/non_standard.md b/docs/getting_access/non_standard.md new file mode 100644 index 0000000000..eeff61ed32 --- /dev/null +++ b/docs/getting_access/non_standard.md @@ -0,0 +1,130 @@ +# Non-standard ways to use EESSI + +In case that an EESSI installation is not possible or desirable, there are a couple of "non-standard" ways to access EESSI. + +## Using cvmfsexec + +This option relies on the [cvmfsexec package](https://github.com/cvmfs/cvmfsexec) provided by CernVM-FS team. It will allow you to mount cvmfs as an unprivileged user, so no need to have a native cvmfs installation. + +### `cvmfsexec_eessi.sh` wrapper script + + +This wrapper script, `cvmfsexec_eessi.sh`, can be used to run a command in a subshell in which the EESSI CernVM-FS repository (`software.eessi.io`) is mounted via cvmfsexec`. + +```bash +#!/bin/bash +if [ -d /cvmfs/software.eessi.io ]; then + # run command directly, EESSI CernVM-FS repository is already mounted + "$@" +else + # run command via in subshell where EESSI CernVM-FS repository is mounted, + # via cvmfsexec which is set up in a unique temporary directory + orig_workdir=$(pwd) + mkdir -p /tmp/$USER + tmpdir=$(mktemp -p /tmp/$USER -d) + cd $tmpdir + git clone https://github.com/cvmfs/cvmfsexec.git > $tmpdir/git_clone.out 2>&1 + cd cvmfsexec + ./makedist default > $tmpdir/cvmfsexec_makedist.out 2>&1 + cd $orig_workdir + $tmpdir/cvmfsexec/cvmfsexec software.eessi.io -- "$@" + # cleanup + rm -rf $tmpdir +fi + +``` + +Do make sure that this script is executable: + +```bash + chmod u+x ~/bin/cvmfsexec_eessi.sh +``` + +A simple way to test this script is to use it to inspect the contents of the EESSI repository: + +```bash + ~/bin/cvmfsexec_eessi.sh ls /cvmfs/software.eessi.io +``` + +or to start an interactive shell in which the EESSI repository is mounted: + +```bash + # Expected output from a machine without a cvmfs installation + $ ls /cvmfs + ls: cannot access '/cvmfs': No such file or directory + + # Starting interactive shell + $ ~/bin/cvmfsexec_eessi.sh /bin/bash -l + CernVM-FS: loading Fuse module... done + CernVM-FS: mounted cvmfs on /tmp/hvela/tmp.iH6e993Adw/cvmfsexec/dist/cvmfs/cvmfs-config.cern.ch + CernVM-FS: loading Fuse module... done + CernVM-FS: mounted cvmfs on /tmp/hvela/tmp.iH6e993Adw/cvmfsexec/dist/cvmfs/software.eessi.io + + # Now it's mounted! + $ ls /cvmfs/ + cvmfs-config.cern.ch software.eessi.io + +``` + +Notice how now that EESSI is mounted, you will need to [set up the environment](https://www.eessi.io/docs/using_eessi/setting_up_environment/) in order to access the software itself. + +### `orted` wrapper script + +In order to get multi-node runs of software working without having EESSI available system-wide, we also had to create a small wrapper script for the `orted` command that is used by Open MPI to start processes on remote nodes. This is necessary because `mpirun` launches `orted`, which must be run in an environment in which the EESSI repository is mounted. If not, MPI startup will fail with an error like + +```bash +"error: execve(): orted: No such file or directory". +``` + +This wrapper script must be named `orted`, and must be located in a path that is listed in `$PATH`. + +We placed it in `~/bin/orted`, and you can add `export PATH=$HOME/bin:$PATH` to your `~/.bashrc` login script. + +Contents of ~/bin/orted: + +```bash +#!/bin/bash + +# first remove path to this orted wrapper from $PATH, to avoid infinite loop +orted_wrapper_dir=$(dirname $0) +export PATH=$(echo $PATH | tr ':' '\n' | grep -v $orted_wrapper_dir | tr '\n' ':') + +~/bin/cvmfsexec_eessi.sh orted "$@" +``` + +Again, do make sure that also this `orted` wrapper script is executable: + +```bash +chmod u+x ~/bin/orted +``` + +If not, you will likely run into an error that starts with: + + +```bash +An ORTE daemon has unexpectedly failed after launch ... +``` + +### Slurm job script + +The `cvmfsexec_eessi.sh` can be used insided a Slurm job script inside your HPC system to initialize the EESSI environment in a subshell which the EESSI CernVM-FS repository is mounted. Thus, you will be able to load any module from EESSI you might need. + +Example job script: + +```bash +#!/bin/bash +#SBATCH --ntasks=4 +#SBATCH --ntasks-per-node=2 +#SBATCH --cpus-per-task=1 +#SBATCH --time=5:0:0 +#SBATCH --export=None +#SBATCH --mem=30000M +~/bin/cvmfsexec_eessi.sh << EOF +source /cvmfs/software.eessi.io/versions/2023.06/init/bash +module load TensorFlow/2.13.0-foss-2023a +export SLURM_EXPORT_ENV=HOME,PATH,LD_LIBRARY_PATH,PYTHONPATH +mpirun -np 4 python test.py +EOF + +``` +You can see the original blog post on how they used this solution in Deucalion [here](https://www.eessi.io/docs/blog/2024/06/28/espresso-portable-test-run-eurohpc/#running-espresso-on-deucalion-via-eessi-cvmfsexec). From 895b7998f4a4a3771cadfa8fc5fc165e2d10112f Mon Sep 17 00:00:00 2001 From: torri Date: Thu, 20 Feb 2025 12:53:19 +0100 Subject: [PATCH 02/10] now the index --- mkdocs.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/mkdocs.yml b/mkdocs.yml index 9a8c47e21c..a3aad3a2b4 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -33,6 +33,7 @@ nav: - Is EESSI already installed?: getting_access/is_eessi_accessible.md - Native: getting_access/native_installation.md - Container: getting_access/eessi_container.md + - Non-standard ways: getting_access/non_standard.md - Windows and macOS: - Windows with WSL: getting_access/eessi_wsl.md - macOS with Lima: getting_access/eessi_limactl.md From f969e85cfff66d8447e66339c81ef0ee848bf671 Mon Sep 17 00:00:00 2001 From: torri Date: Thu, 20 Feb 2025 13:05:27 +0100 Subject: [PATCH 03/10] typo fix --- docs/getting_access/non_standard.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting_access/non_standard.md b/docs/getting_access/non_standard.md index eeff61ed32..d724916218 100644 --- a/docs/getting_access/non_standard.md +++ b/docs/getting_access/non_standard.md @@ -107,7 +107,7 @@ An ORTE daemon has unexpectedly failed after launch ... ### Slurm job script -The `cvmfsexec_eessi.sh` can be used insided a Slurm job script inside your HPC system to initialize the EESSI environment in a subshell which the EESSI CernVM-FS repository is mounted. Thus, you will be able to load any module from EESSI you might need. +The `cvmfsexec_eessi.sh` can be used inside a Slurm job script inside your HPC system to initialize the EESSI environment in a subshell which the EESSI CernVM-FS repository is mounted. Thus, you will be able to load any module from EESSI you might need. Example job script: From 425ccc3017c99d99a31ee9f389b65ba4bea0bdd6 Mon Sep 17 00:00:00 2001 From: torri Date: Thu, 20 Feb 2025 16:55:08 +0100 Subject: [PATCH 04/10] WIP on the squashfs part --- docs/getting_access/non_standard.md | 105 ++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/docs/getting_access/non_standard.md b/docs/getting_access/non_standard.md index d724916218..fbe94f22c7 100644 --- a/docs/getting_access/non_standard.md +++ b/docs/getting_access/non_standard.md @@ -128,3 +128,108 @@ EOF ``` You can see the original blog post on how they used this solution in Deucalion [here](https://www.eessi.io/docs/blog/2024/06/28/espresso-portable-test-run-eurohpc/#running-espresso-on-deucalion-via-eessi-cvmfsexec). + +## Via `squashfs` + cvmfs's `shrinkwrap` utility + +CernVM-FS provides the Shrinkwrap utility, allowing users to create a portable snapshot of a CVMFS repository. This can be exported and distributed without the need of a CVMFS client or network access. + +To create an export of EESSI in user space, you will first need to create the config file `software.eessi.io.config`: + +```bash +CVMFS_REPOSITORIES=software.eessi.io +CVMFS_REPOSITORY_NAME=software.eessi.io +CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch +CVMFS_SERVER_URL='http://aws-eu-west-s1-sync.eessi.science/cvmfs/software.eessi.io' +CVMFS_HTTP_PROXY=DIRECT # Avoid filling up any local squid's cache +CVMFS_CACHE_BASE=/tmp/shrinkwrap +CVMFS_KEYS_DIR=/etc/cvmfs/keys/eessi.io # Need to be provided for shrinkwrap +CVMFS_SHARED_CACHE=no # Important as libcvmfs does not support shared caches +CVMFS_USER=cvmfs +CVMFS_UID_MAP=uid.map +CVMFS_GID_MAP=gid.map + +``` +You will need to create the files `uid.map` and `gid.map` with the respective value you will use preceded by a `*`. P.e., assuming UID 1000, set the two following files: + +```bash + $ cat uid.map + * 1000 + + $ cat gid.map + * 1000 +``` +In addition, you need to create a spec file `software.eessi.io.spec` with the files you want to include and/or exclude in the shrinkwrap. Contents are: + +```bash +/versions/2023.06/compat/linux/x86_64/* +/versions/2023.06/init/* +/versions/2023.06/scripts/* +/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/.lmod/* +/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/* +/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/* +# Exclude the Gentoo ebuild repo and cache files +!/versions/2023.06/compat/linux/x86_64/var/db/repos/gentoo +!/versions/2023.06/compat/linux/x86_64/var/cache + +``` + +Then, execute `cvmfs_shrinkwrap`to create the export: + +```bash +$ cvmfs_shrinkwrap -r software.eessi.io -f software.eessi.io.config -t software.eessi.io.spec --dest-base /tmp/cvmfs -j 4 + +LibCvmfs version 2.12, revision 31 +(libcvmfs) (manager 'standard') switching proxy from (none) to DIRECT. Reason: set random start proxy from the first proxy group [Current host: http://aws-eu-west-s1-sync.eessi.science/cvmfs/software.eessi.io] +(libcvmfs) (manager 'external') switching proxy from (none) to DIRECT. Reason: cloned [Current host: http://aws-eu-west-s1-sync.eessi.science/cvmfs/software.eessi.io] +(libcvmfs) Starting 4 workers +(libcvmfs) cntByte|0|Byte count of projected repository +cntDir|1|Number of directories from source repository +cntFile|0|Number of files from source repository +cntSymlink|0|Number of symlinks from source repository +dataBytes|0|Bytes transferred from source to destination +dataBytesDeduped|0|Number of bytes not copied due to deduplication +dataFiles|0|Number of data files transferred from source to destination +dataFilesDeduped|0|Number of files not copied due to deduplication +entriesDest|1|Number of file system entries processed in the destination +entriesSrc|1|Number of file system entries processed in the source + +.... + +# This takes a long time +``` + +Once completed, the contents will be available in /tmp/cvmfs. You can create an squashfs image from it: + +```bash + sudo mksquashfs /tmp/cvmfs software.eessi.io.sqsh + +``` + +This squashfs image can be mounted in any container: + +```bash + apptainer shell -B software.eessi.io.sqsh:/cvmfs:image-src=/ docker://ubuntu + +``` + +Right now its a manual process but there is work in progress towards an automated solution. + + + + + + + + + + + + + + + + + + + + From fb4bcaa3ba672c9b476bec6ef9051bf3dcdb8b16 Mon Sep 17 00:00:00 2001 From: Helena Vela Beltran <47674829+hvelab@users.noreply.github.com> Date: Mon, 24 Feb 2025 10:44:23 +0100 Subject: [PATCH 05/10] Update non_standard.md clarirfication for cvmfsexec --- docs/getting_access/non_standard.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting_access/non_standard.md b/docs/getting_access/non_standard.md index fbe94f22c7..32832e4904 100644 --- a/docs/getting_access/non_standard.md +++ b/docs/getting_access/non_standard.md @@ -4,7 +4,7 @@ In case that an EESSI installation is not possible or desirable, there are a cou ## Using cvmfsexec -This option relies on the [cvmfsexec package](https://github.com/cvmfs/cvmfsexec) provided by CernVM-FS team. It will allow you to mount cvmfs as an unprivileged user, so no need to have a native cvmfs installation. +This option only works for RHEL-like systems and relies on the [cvmfsexec package](https://github.com/cvmfs/cvmfsexec) provided by CernVM-FS team. It will allow you to mount cvmfs as an unprivileged user, so no need to have a native cvmfs installation. ### `cvmfsexec_eessi.sh` wrapper script From f836832448c4b6053807c24f48b034cfbe9f0df2 Mon Sep 17 00:00:00 2001 From: Helena Vela Beltran <47674829+hvelab@users.noreply.github.com> Date: Thu, 20 Mar 2025 10:01:39 +0100 Subject: [PATCH 06/10] Update docs/getting_access/non_standard.md Co-authored-by: ocaisa --- docs/getting_access/non_standard.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting_access/non_standard.md b/docs/getting_access/non_standard.md index 32832e4904..5de144fe6e 100644 --- a/docs/getting_access/non_standard.md +++ b/docs/getting_access/non_standard.md @@ -70,7 +70,7 @@ Notice how now that EESSI is mounted, you will need to [set up the environment]( ### `orted` wrapper script -In order to get multi-node runs of software working without having EESSI available system-wide, we also had to create a small wrapper script for the `orted` command that is used by Open MPI to start processes on remote nodes. This is necessary because `mpirun` launches `orted`, which must be run in an environment in which the EESSI repository is mounted. If not, MPI startup will fail with an error like +In order to get multi-node runs of software working without having EESSI available system-wide, you also need to create a small wrapper script for the `orted` command that is used by Open MPI to start processes on remote nodes. This is necessary because `mpirun` launches `orted`, which must be run in an environment in which the EESSI repository is mounted. If not, MPI startup may fail with an error like ```bash "error: execve(): orted: No such file or directory". From 5736bbc7221601da42f856c6720718b0255c9077 Mon Sep 17 00:00:00 2001 From: Helena Vela Beltran <47674829+hvelab@users.noreply.github.com> Date: Thu, 20 Mar 2025 10:02:01 +0100 Subject: [PATCH 07/10] Update docs/getting_access/non_standard.md Co-authored-by: ocaisa --- docs/getting_access/non_standard.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting_access/non_standard.md b/docs/getting_access/non_standard.md index 5de144fe6e..31f920930d 100644 --- a/docs/getting_access/non_standard.md +++ b/docs/getting_access/non_standard.md @@ -78,7 +78,7 @@ In order to get multi-node runs of software working without having EESSI availab This wrapper script must be named `orted`, and must be located in a path that is listed in `$PATH`. -We placed it in `~/bin/orted`, and you can add `export PATH=$HOME/bin:$PATH` to your `~/.bashrc` login script. +For example, it can be placed in `~/bin/orted`, and you can add `export PATH=$HOME/bin:$PATH` to your `~/.bashrc` login script. Contents of ~/bin/orted: From b3ba9a55828dd6a8121f71abd404dc2603132f03 Mon Sep 17 00:00:00 2001 From: Helena Vela Beltran <47674829+hvelab@users.noreply.github.com> Date: Thu, 20 Mar 2025 10:02:25 +0100 Subject: [PATCH 08/10] Update docs/getting_access/non_standard.md Co-authored-by: ocaisa --- docs/getting_access/non_standard.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting_access/non_standard.md b/docs/getting_access/non_standard.md index 31f920930d..0c957b9487 100644 --- a/docs/getting_access/non_standard.md +++ b/docs/getting_access/non_standard.md @@ -212,7 +212,7 @@ This squashfs image can be mounted in any container: ``` -Right now its a manual process but there is work in progress towards an automated solution. +Right now, this may seem like quite a manual process, but there is work in progress towards creating a script to automate this process. From e5600c9b0187be2b95d0c36906a5936b8e28c357 Mon Sep 17 00:00:00 2001 From: Helena Vela Beltran <47674829+hvelab@users.noreply.github.com> Date: Thu, 20 Mar 2025 10:02:56 +0100 Subject: [PATCH 09/10] Update docs/getting_access/non_standard.md Co-authored-by: ocaisa --- docs/getting_access/non_standard.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting_access/non_standard.md b/docs/getting_access/non_standard.md index 0c957b9487..867a06a0f0 100644 --- a/docs/getting_access/non_standard.md +++ b/docs/getting_access/non_standard.md @@ -149,7 +149,7 @@ CVMFS_UID_MAP=uid.map CVMFS_GID_MAP=gid.map ``` -You will need to create the files `uid.map` and `gid.map` with the respective value you will use preceded by a `*`. P.e., assuming UID 1000, set the two following files: +You will need to create the files `uid.map` and `gid.map` with the respective value you will use preceded by a `*`. For example, assuming UID 1000, set the two following files: ```bash $ cat uid.map From d77b762e42d41e124792b65a4d7abf34604a525e Mon Sep 17 00:00:00 2001 From: torri Date: Thu, 20 Mar 2025 15:31:34 +0100 Subject: [PATCH 10/10] first batch of corrections --- docs/getting_access/non_standard.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/getting_access/non_standard.md b/docs/getting_access/non_standard.md index 867a06a0f0..8e32e00a12 100644 --- a/docs/getting_access/non_standard.md +++ b/docs/getting_access/non_standard.md @@ -131,7 +131,7 @@ You can see the original blog post on how they used this solution in Deucalion [ ## Via `squashfs` + cvmfs's `shrinkwrap` utility -CernVM-FS provides the Shrinkwrap utility, allowing users to create a portable snapshot of a CVMFS repository. This can be exported and distributed without the need of a CVMFS client or network access. +CernVM-FS provides the [Shrinkwrap utility](https://cvmfs.readthedocs.io/en/stable/cpt-shrinkwrap.html), allowing users to create a portable snapshot of a CVMFS repository. This can be exported and distributed without the need of a CVMFS client or network access. To create an export of EESSI in user space, you will first need to create the config file `software.eessi.io.config`: @@ -149,14 +149,16 @@ CVMFS_UID_MAP=uid.map CVMFS_GID_MAP=gid.map ``` -You will need to create the files `uid.map` and `gid.map` with the respective value you will use preceded by a `*`. For example, assuming UID 1000, set the two following files: +You will need to create the files `uid.map` and `gid.map` with the respective value you will use preceded by a `*`. You will probably want to use the UID and GID of the current user, so, to set the two files: ```bash + $ echo '*' $(id -u $USER) > uid.map $ cat uid.map - * 1000 + * 1001 + $ echo '*' $(id -g $USER) > gid.map $ cat gid.map - * 1000 + * 1001 ``` In addition, you need to create a spec file `software.eessi.io.spec` with the files you want to include and/or exclude in the shrinkwrap. Contents are: @@ -195,13 +197,13 @@ entriesSrc|1|Number of file system entries processed in the source .... -# This takes a long time +# This takes a long time and space depending on how much you want to export ``` Once completed, the contents will be available in /tmp/cvmfs. You can create an squashfs image from it: ```bash - sudo mksquashfs /tmp/cvmfs software.eessi.io.sqsh + mksquashfs /tmp/cvmfs software.eessi.io.sqsh ```