Merge branch 'easybuilders:develop' into toolchains

Author: branfosj

docs/changelog-docs.md

@@ -7,6 +7,12 @@ search:
 
 (for EasyBuild release notes, see [EasyBuild release notes][release_notes])
 
+- **release 20230911.01** (*9 Sept 2023*): update
+    release notes for EasyBuild v4.8.1 (see
+    [EasyBuild v4.8.1 (11 Sept 2023)][release_notes_eb481])
+- **release 20230707.01** (*7 July 2023*): update
+    release notes for EasyBuild v4.8.0 (see
+    [EasyBuild v4.8.0 (7 July 2023)][release_notes_eb480])
 - **release 20230527.01** (*27 May 2023*): update
     release notes for EasyBuild v4.7.2 (see
     [EasyBuild v4.7.2 (27 May 2023)][release_notes_eb472])

docs/code-style.md

@@ -19,7 +19,7 @@ The only (major) exception to PEP8 is our preference for longer line lengths: li
 ## Notes
 
 Code style in easyconfig files can be **automatically checked** using `--check-contrib`, 
-for example: `eb --check-contrib sympy-1.3-intel-2018a-Python-2.7.14.eb` 
+for example: `eb --check-contrib HPL-2.3-foss-2022b.eb`
 (see [Code style review][contributing_review_process_code_style] for more details).
 
 Style guides that go a step beyond PEP8:

docs/hooks.md

@@ -43,22 +43,28 @@ eb ...
 
 ## Available hooks
 
-Currently (since EasyBuild v3.7.0), three types of hooks are supported:
+Since EasyBuild v4.8.1, six different types of hooks are supported:
 
-* `start_hook` and `end_hook`, which are triggered *once* before starting software installations,
-  and *once* right after completing all installations, respectively
+* `start_hook`, `pre_build_and_install_loop_hook`, `post_build_and_install_loop_hook`, and `end_hook` which are triggered *once* right after
+  EasyBuild starts, *once* before looping over the easyconfigs to be built, *once* after completing the loop over the eayconfigs to be installed,
+  and *once* shortly before EasyBuild completes, respectively.
 * `parse_hook`, which is triggered when an easyconfig file is being parsed
 * `module_write_hook`, which is triggered right before a module file is written.
   This includes the temporary module file used when installing extensions and during the sanity check,
   as well as the devel module.
 * "*step*" hooks that are triggered before and after every step of each installation procedure that is performed,
-  also aptly named '`pre`'- and '`post`'-hooks
+  also aptly named '`pre`'- and '`post`' hooks.
+* `cancel_hook` and `fail_hook` which are triggered when a `KeyboardInterrupt` or `EasyBuildError` is raised,
+  respectively.
+* `pre_run_shell_cmd_hook` and `post_run_shell_cmd_hook` which are triggered right before and after running
+  a shell command, respectively.
 
 The list of currently available hooks in order of execution,
 which can also be consulted using `eb --avail-hooks`, is:
 
 * `start_hook` *(only called once in an EasyBuild session)*
 * `parse_hook` *(available since EasyBuild v3.7.0)*
+* `pre_build_and_install_loop` *(available since EasyBuild v4.8.1)*
 * `pre_fetch_hook`, `post_fetch_hook`
 * `pre_ready_hook`, `post_ready_hook`
 * `pre_source_hook`, `post_source_hook`
@@ -68,16 +74,27 @@ which can also be consulted using `eb --avail-hooks`, is:
 * `pre_build_hook`, `post_build_hook`
 * `pre_test_hook`, `post_test_hook`
 * `pre_install_hook`, `post_install_hook`
-* `pre_extensions_hook`, `post_extensions_hook`
+* `pre_extensions_hook`
+* `pre_single_extension_hook`, `post_single_extension_hook` *(available since EasyBuild v4.7.1)*
+* `post_extensions_hook`
+* `pre_postiter_hook`, `post_postiter_hook` *(available since EasyBuild v4.8.1)*
 * `pre_postproc_hook`, `post_postproc_hook`
 * `pre_sanitycheck_hook`, `post_sanitycheck_hook`
 * `pre_cleanup_hook`, `post_cleanup_hook`
-* `pre_module_hook`, `post_module_hook`
+* `pre_module_hook`
+* `module_write_hook` *(called multiple times per installation, available since EasyBuild v4.4.1)*
+* `post_module_hook`
 * `pre_permissions_hook`, `post_permissions_hook`
 * `pre_package_hook`, `post_package_hook`
 * `pre_testcases_hook`, `post_testcases_hook`
+* `post_build_and_install_loop` *(available since EasyBuild v4.8.1)*
 * `end_hook` *(only called once in an EasyBuild session)*
-* `module_write_hook` *(called multiple times per installation, available since EasyBuild v4.4.1)*
+* `cancel_hook` *(available since EasyBuild v4.8.1)*
+* `fail_hook` *(available since EasyBuild v4.8.1)*
+
+`pre_run_shell_cmd_hook` and `post_run_shell_cmd_hook` *(available since EasyBuild v4.8.1)* are not included in
+the list above because they can not be put in a particular order relative to other hooks, since these hooks
+are triggered several times throughout an EasyBuild session.
 
 All functions implemented in the provided Python module for which the name ends with `_hook` are considered.
 
@@ -102,20 +119,46 @@ Run 'eb --avail-hooks' to get an overview of known hooks
 To implement hooks, simply define one or more functions in a Python module (`*.py`),
 each named after an available hook.
 
+In hooks you have access to the full functionality provided by the EasyBuild framework,
+so do `import` from `easybuild.tools.*` (or other `easybuild.*` namespaces) to leverage
+those functions.
+
 Do take into account the following:
 
-* for `start_hook` and `end_hook`, no arguments are provided
-* for `parse_hook`, one argument is provided: the `EasyConfig` instance
-  that corresponds to the easyconfig file being parsed (usually referred to as `ec`)
-* for `module_write_hook`, 3 arguments are provided:
-   * the `EasyBlock` instance used to perform the installation (usually referred to as `self`)
-   * the filepath of the module that will be written
-   * the module text as a string
-  The return value of this hook, when set, will replace the original text that is then written to the module file.
-* for the step hooks, one argument is provided:
-  the `EasyBlock` instance used to perform the installation (usually referred to as `self`)
-* the parsed easyconfig file can be accessed in the step hooks via the `EasyBlock` instance,
-  i.e., via `self.cfg`
+* [Hook arguments][hooks-arguments]
+* [Return value of hooks][hooks-return-value]
+* [Parse hook specifics][parse-hook-specifics]
+
+
+### Hook arguments {: #hooks-arguments }
+
+* For both `start_hook` and `end_hook` no arguments are provided.
+* For `cancel_hook` and `fail_hook` the `KeyboardInterrupt` or `EasyBuildError` exception that was raised
+  is provided as an argument.
+* For `parse_hook` the `EasyConfig` instance that corresponds to the easyconfig file being parsed
+  (usually referred to as `ec`) is passed as an argument.
+* For `pre_build_and_install_loop_hook` a list of easyconfigs is provided as an argument.
+* For `post_build_and_install_loop_hook` a list of easyconfigs with build results is provided as an argument.
+* For `pre_run_shell_cmd_hook`, multiple arguments are passed:
+    * An unnamed argument (often called `cmd`) that corresponds to the shell command that will be run,
+      which could be provided either as a string value (like `"echo hello"`) or a list value (like `['echo', 'hello']`).
+    * A named argument `work_dir` that specifies the path of the working directory in which the command will be run.
+    * For interactive commands (which are run via the `run_cmd_qa` function), an additional named argument
+      `interactive` is set to `True`.
+* For `post_run_shell_cmd_hook`, multiple arguments are passed:
+    * An unnamed argument (often called `cmd`) that corresponds to the shell command that was run,
+      which could be provided either as a string value (like `"echo hello"`) or a list value (like `['echo', 'hello']`).
+    * A named argument `work_dir` that specifies the working directory in which the shell command was run.
+    * A named argument `exit_code` that specifies the exit code of the shell command that was run.
+    * A named argument `output` that specifies the output of the shell command that was run.
+    * For interactive commands (which are run via the `run_cmd_qa` function), an additional named argument
+      `interactive` is set to `True`.
+* For `module_write_hook`, 3 arguments are provided:
+    * The `EasyBlock` instance used to perform the installation (usually referred to as `self`).
+    * The filepath of the module that will be written.
+    * The module text as a string.
+* For the step hooks, the `EasyBlock` instance used to perform the installation (usually referred to as `self`).
+  The parsed easyconfig file can be accessed in the step hooks via the `EasyBlock` instance, i.e., via `self.cfg`.
 
 It is recommended to anticipate possible changes in the provided (named) arguments,
 using the `*args` and `**kwargs` mechanism commonly used in Python. This
@@ -127,11 +170,22 @@ def pre_configure_hook(self, *args, **kwargs):
     ...
 ```
 
-In hooks you have access to the full functionality provided by the EasyBuild framework,
-so do `import` from `easybuild.tools.*` (or other `easybuild.*` namespaces) to leverage
-those functions.
 
-### Parse hook specifics
+### Return value of hooks {: #hooks-return-value }
+
+The return value of a hook is usually ignored by EasyBuild, except in particular cases:
+
+* If the `module_write_hook` returns a (string) value, it **replaces the original text that was going to be
+  written to the module file**. This way the `module_write_hook` can extend, change, or entirely replace the
+  module text that was provided as an argument.
+
+* If the `pre_run_shell_cmd_hook` returns a value, it **replaces the shell command that was going to be run**.
+  Hence, this hook can change or entirely replace particular shell commands right before they are executed.
+  Note that the value type of the return value of `pre_run_shell_cmd_hook` *must* match with the type of the
+  first (unnamed) argument that provides the shell command that would have been run originally.
+
+
+### Parse hook specifics {: #parse-hook-specifics }
 
 `parse_hook` is triggered right *after* reading the easyconfig file,
 before further parsing of some easyconfig parameters (like `*dependencies`) into
@@ -220,7 +274,7 @@ To achieve the intended effect, you can either:
     ```
 
 A better approach for manipulating easyconfig parameters is to use the `parse_hook` that
-was introduced in EasyBuild v3.7.0 (see [Parse hook specifics](#parse-hook-specifics)),
+was introduced in EasyBuild v3.7.0 (see [Parse hook specifics][parse-hook-specifics]),
 where these kind of surprises will not occur (because templating is automatically disabled
 before `parse_hook` is called and restored immediately afterwards).
 See also [Example hook to inject a custom patch file](#inject-a-custom-patch-file).
@@ -276,3 +330,30 @@ def module_write_hook(self, filepath, module_txt, *args, **kwargs):
         replace = 'prepend_path("EBPYTHONPREFIXES", root)'
         return re.sub(search, replace, module_txt)
 ```
+
+### Log running of shell commands + prepend `make install` with `sudo`
+
+```py
+shell_cmds_log = '/tmp/eb_shell_cmds.log'
+
+def pre_run_shell_cmd_hook(cmd, work_dir=None, interactive=None):
+    """
+    Log shell commands before they are run,
+    and replace 'make install' with 'sudo make install'.
+    """
+    with open(shell_cmds_log, 'a') as fp:
+        cmd_type = 'interactive' if interactive else 'non-interactive'
+        fp.write("%s command '%s' will be run in %s ...\n" % (cmd_type, cmd, work_dir))
+
+    if cmd == "make install":
+        return "sudo make install"
+
+
+def post_run_shell_cmd_hook(cmd, work_dir=None, interactive=None, exit_code=None, output=None):
+    """
+    Log shell commands that were run.
+    """
+    with open(shell_cmds_log, 'a') as fp:
+        cmd_type = 'interactive' if interactive else 'non-interactive'
+        fp.write("%s command '%s' in %s exited with %s - output: %s\n" % (cmd_type, cmd, work_dir, exit_code, output))
+```

docs/installation.md

@@ -62,10 +62,10 @@ $ module load EasyBuild
 $ module list
 
 Currently Loaded Modules:
-  1) EasyBuild/4.7.2
+  1) EasyBuild/4.8.0
 
 $ eb --version
-This is EasyBuild 4.7.2 (framework: 4.7.2, easyblocks: 4.7.2) on host example.local
+This is EasyBuild 4.8.0 (framework: 4.8.0, easyblocks: 4.8.0) on host example.local
 ```
 
 !!! tip
@@ -228,7 +228,7 @@ $ EB_VERBOSE=1 eb --version
 >> 'python3' version: 3.6.8, which matches Python 3 version requirement (>= 3.5)
 >> Selected Python command: python3 (/usr/bin/python3.6)
 >> python3.6 -m easybuild.main --version
-This is EasyBuild 4.7.2 (framework: 4.7.2, easyblocks: 4.7.2) on host example
+This is EasyBuild 4.8.0 (framework: 4.8.0, easyblocks: 4.8.0) on host example
 ```
 
 

docs/locks.md

@@ -25,7 +25,7 @@ Locks are created in the [locks directory][locks_dir].
 The lock created by EasyBuild is an empty directory (rather than a file),
 because that can be created more atomically on modern filesystems.
 
-For example, if `OpenFOAM-7-foss-2019b.eb` is being installed to `/apps/easybuild/software`,
+For example, if `OpenFOAM-v2206-foss-2022a.eb` is being installed to `/apps/easybuild/software`,
 an empty directory that serves as a lock for this installation will be created at
 `/apps/easybuild/software/.locks/_apps_easybuild_software_OpenFOAM_7_foss_2019b.lock`
 (assuming the default [locks directory][locks_dir] is used).
@@ -60,20 +60,29 @@ performing those installations!**
 
 #### Waiting for locks to be removed {: #locks_wait }
 
-*(`--wait-on-lock`)*
+*(`--wait-on-lock-interval`)*
 
-Using the `--wait-on-lock` configuration option, you can change how EasyBuild deals with existing locks,
-by specifying how frequently EasyBuild should check whether an existing lock was removed. By specifying a non-zero value `S`,
-you can indicate how many seconds EasyBuild should wait before checking again whether the lock is still in place.
+Wait interval (in seconds) to use when waiting for existing lock to be removed
 
-!!! note
-    EasyBuild will wait indefinitely for an existing lock to be removed if `--wait-on-lock` is set to a non-zero value...
+*(`--wait-on-lock-limit`)*
+
+Maximum amount of time (in seconds) to wait until lock is released (0 means no waiting at all, exit with error;
+-1 means no waiting limit, keep waiting).
+
+*(`--wait-on-lock <secs>`)* (DEPRECATED)
 
+Using the `--wait-on-lock` configuration option, you can change how EasyBuild deals with existing locks, by specifying
+how frequently EasyBuild should check whether an existing lock was removed. By specifying a non-zero value `secs`, you
+can indicate how many seconds EasyBuild should wait before checking again whether the lock is still in place.
+
+!!! warning
+    The `--wait-on-lock` configuration option is deprecated, please use `--wait-on-lock-interval` and `--wait-on-lock-limit` instead!
+
+!!! note
     If the lock is never removed, the EasyBuild session will never terminate; it will keep checking every `S` seconds whether the lock is still in place.
 
 By default, EasyBuild will *abort* the installation with an error like "`Lock ... already exists, aborting!`"
-if a corresponding lock already exists, which is equivalent to setting `--wait-on-lock` to zero (`0`),
-implying that no waiting should be done at all.
+if a corresponding lock already exists.
 
 
 ### Locks directory {: #locks_dir }

docs/log-files.md

@@ -17,24 +17,24 @@ file is copied to the install directory for future reference.
 
 By default, the application log file is copied to a subdirectory of the
 installation prefix named `easybuild`, and has a filename like
-`easybuild-HPL-2.0-20141103.104412.log` for example, which corresponds
+`easybuild-WRF-4.4.1-20230426.143346.log` for example, which corresponds
 to the filename template
 `easybuild-%(name)s-%(version)s-%(date)s.%(time)s.log`. This aspect can
 be tweaked via the `--logfile-format` configuration option.
 
 Example:
 
 ``` console
-$ eb HPL-2.0-goolf-1.4.10.eb
-== temporary log file in case of crash /tmp/easybuild-rHHgBu/easybuild-XD0Ae_.log
+$ eb WRF-4.4.1-foss-2022b-dmpar.eb
+== Temporary log file in case of crash /tmp/eb-1zbqix6e/easybuild-1ixy86r2.log
 [...]
-== building and installing HPL/2.0-goolf-1.4.10...
+== building and installing WRF/4.4.1-foss-2022b-dmpar...
 [...]
 == COMPLETED: Installation ended successfully
-== Results of the build can be found in the log file /home/example/.local/easybuild/software/HPL/2.0-goolf-1.4.10/easybuild/easybuild-HPL-2.0-20141103.104412.log
+== Results of the build can be found in the log file(s) /home/example/.local/easybuild/software/WRF/4.4.1-foss-2022b-dmpar/easybuild/easybuild-WRF-4.4.1-20230531.181321.log
 == Build succeeded for 1 out of 1
-== temporary log file /tmp/easybuild-rHHgBu/easybuild-XD0Ae_.log has been removed.
-== temporary directory /tmp/easybuild-rHHgBu has been removed.
+== temporary log file /tmp/eb-1zbqix6e/easybuild-1ixy86r2.log* has been removed.
+== temporary directory /tmp/eb-1zbqix6e has been removed.
 ```
 
 !!! note
@@ -63,16 +63,16 @@ Each log message as emitted by EasyBuild follows a well-defined format.
 Example:
 
 ``` console
-== 2014-11-03 13:34:31,906 main.EB_HPL INFO This is EasyBuild 1.15.2 (framework: 1.15.2, easyblocks: 1.15.2) on host example.
+== 2023-05-31 16:11:21,044 easyblock.py:313 INFO This is EasyBuild 4.7.3 (framework: 4.7.3, easyblocks: 4.7.3) on host example.
 ```
 
 Each log line consists of the following parts:
 
 - a prefix label `==`, which is useful to discriminate between
     EasyBuild log messages and the output of executed shell commands;
-- date and time information (e.g., `2014-11-03 13:34:31,906`);
+- date and time information (e.g., `2023-05-31 16:11:21,044`);
 - the Python module/class/function that is responsible for the log
-    message (e.g., `main.EB_HPL`);
+    message (e.g., `easyblock.py:313`);
 - the log level (e.g., `INFO`);
 - and a string with the actual log message at the end
 
@@ -88,10 +88,10 @@ For each step performed in the build and installation process,
 corresponding log messages is emitted. For example:
 
 ``` console
-== 2014-11-03 13:34:48,816 main.EB_HPL INFO configuring...
-== 2014-11-03 13:34:48,817 main.EB_HPL INFO Starting configure step
+== 2023-05-31 16:11:44,977 build_log.py:267 INFO configuring...
+== 2023-05-31 16:11:44,981 easyblock.py:3926 INFO Starting configure step
 [...]
-== 2014-11-03 13:34:48,823 main.EB_HPL INFO Running method configure_step part of step configure
+== 2023-05-31 16:11:44,982 easyblock.py:3934 INFO Running method configure_step part of step configure
 ```
 
 This allows you to navigate a log file step by step, for example using
@@ -104,9 +104,10 @@ command line, the location where the command was executed and the
 command's output and exit code. For example:
 
 ``` console
-== 2014-11-03 13:34:48,823 main.run DEBUG run_cmd: running cmd /bin/bash make_generic (in /tmp/user/easybuild_build/HPL/2.0/goolf-1.4.10/hpl-2.0/setup)
-== 2014-11-03 13:34:48,823 main.run DEBUG run_cmd: Command output will be logged to /tmp/easybuild-W85p4r/easybuild-run_cmd-XoJwMY.log
-== 2014-11-03 13:34:48,849 main.run INFO cmd "/bin/bash make_generic" exited with exitcode 0 and output:
+== 2023-05-31 18:59:24,222 run.py:176 DEBUG run_cmd: Output of "/home/example/.local/easybuild/software/WRF/4.4.1-foss-2022b-dmpar/WRF-4.4.1/compile -j 8 wrf" will be logged to /tmp/eb-1zbqix6e/easybuild-run_cmd-ueqo5bn0.log
+== 2023-05-31 18:59:24,225 run.py:217 DEBUG run_cmd: running cmd /home/example/.local/easybuild/software//WRF/4.4.1-foss-2022b-dmpar/WRF-4.4.1/compile -j 8 wrf (in /home/example/.local/easybuild/software//WRF/4.4.1-foss-2022b-dmpar/WRF-4.4.1)
+[...]
+== 2023-05-31 20:06:35,979 run.py:650 DEBUG cmd "/home/example/.local/easybuild/software/WRF/4.4.1-foss-2022b-dmpar/WRF-4.4.1/compile -j 8 wrf" exited with exit code 0 and output:
 ```
 
 If you are primarily interested in the different commands as they were