Change spack workflow for end-user to simplify their experience

[atteggiani]

From previous discussions related to access-hive.org.au/#777.

Description

Currently, the spack workflow for an end-user, on Gadi, requires setting up spack.
For this, they are required to:

• Clone three repositories (spack, spack-config, spack-packages)
• Create symlinks of the spack configurations

In addition, before they can build an executable, they will have to set configuration parameters (env variables and paths) by sourcing the spack-config/spack-enable.bash (this step is required for any new session).

All these steps add complexity, especially for a beginner user that might only want to re-build a model with a change in the source code of one component (e.g.: they developed a new convection scheme for the UM, they want to use a custom version of CABLE (embedded in UM), they have changed some physics in MOM, etc.).

Ideal workflow

The optimal spack build workflow on Gadi, for an end-user to produce a custom-build of an ACCESS-NRI-released model is listed below.
Note: These are the minimum steps required to build a model, but any user could customize their build in a more "advanced" way if they want to (for example create custom environments, custom spack-packages, etc.).
Also, each step is representative of what happens from the end-user point of view, and might not exactly reflect all that happens under the hood.

1. Load a module to access the right spack executable

   module load access-spack
   

2. (ONLY IF NEEDED) Run a command that configures spack with custom configuration (one-off step)

   spack-config [--dir <path-to-releases-parent-dir] [--any-other-important-flag]
   

3. (ONLY IF NEEDED) Load a pre-configured environment for ACCESS-NRI-released models, depending on the model they want to build.

   spack env load access-esm1.5
   

4. Build the executables (in any session)
   Proceed with the actual build of the executables.

   spack install ... 
   

I think overall this change would highly enhance the user experience and increase spack usability for ACCESS model builds, especially among the beginner users.

[harshula]

Apparently Gadi's version of modules may be too old to achieve the desired outcome.

[atteggiani]

To better try and address this issue, we might need more details on on why an old(er) version of modules would prevent the realisation of the desired workflow.

@aidanheerdegen would you be able to elaborate more on it?

[aidanheerdegen]

@aidanheerdegen would you be able to elaborate more on it?

There is a topic on the forum with many replies that contains information about attempts to do this

https://forum.access-hive.org.au/t/shared-access-spack-configuration/227

[atteggiani]

Thank you for the useful forum post link @aidan, and for already trying to set up something on the lines of what I proposed in this issue.

Reading the forum post I could capture the following issues that need to be addressed for the proposed workflow to be realised:

• Gadi's version of modules does not allow sourcing a file within the modulefiles and still being able to revert the shell to it’s previous state when unloading the module.
• The system Python version has a bug with the lustre file system, so the SPACK_PYTHON environment variable needs to be set to a known good python3 executable.
• The SPACK_INSTALL_TREE environment variable (that defines where packages will be installed by Spack) needs to be unique for every user ad potentially "customisable" by a user, to decide where they want to build their "custom" packages. It is still preferred this variable to be set when the spack module is loaded.

Is there anything I missed or do you think there are any other issues that need to be addressed?

[harshula]

Upstream PR: spack/spack#11871

[atteggiani]

That PR is useful, but too long and most of those things are not really needed for the workflow mentioned above.
The solution to not being able to modify system environments would be "cloning" a system environment and modify that.

Also, it has been sitting there for very long (> 5 years) and I don't see it being merged anytime soon, so I would suggest finding our best approach to our needs.

Reading the forum post I could capture the following issues that need to be addressed for the proposed workflow to be realised:

• Gadi's version of modules does not allow sourcing a file within the modulefiles and still being able to revert the shell to it’s previous state when unloading the module.
• The system Python version has a bug with the lustre file system, so the SPACK_PYTHON environment variable needs to be set to a known good python3 executable.
• The SPACK_INSTALL_TREE environment variable (that defines where packages will be installed by Spack) needs to be unique for every user ad potentially "customisable" by a user, to decide where they want to build their "custom" packages. It is still preferred this variable to be set when the spack module is loaded.

I have solutions for all these points above (need to be tested but they should work).
Are there any other requirements we need to consider?

[harshula]

[Updated: 23/04/2026]

Spack Architecture for CI and Gadi with Spack v1.1 (2026)

Objective

• A user on Gadi shall NOT need to run git clone to use Spack.
• The CI Spack architecture/configuration shall be based on the Gadi architecture/configuration to ensure additional testing coverage.

Users

• Gadi service user

  • Responsible for Prerelease and Release

• Gadi regular user

  • Modify and build Model Deployment Repository

• Gadi advanced user

  • Modify Spack Package Recipes

• Build-ci

  • Upstream Docker image
  • Runner Docker image

Solution

• Using Spack v1.1's include.yaml mechanism allows defining tiered configuration directories with a common directory for both Gadi and CI.

• Using Spack v1.1's include.yaml mechanism choose a spack-config directory based on the hostname of the system.

• Define an upstream for the Gadi user and CI runner to speed up builds.

• Use Spack scopes to differentiate between the configuration of an upstream and a downstream.

Example (Gadi)

╭╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╮
┊ Upstream / Admin User     ┊
┊ ---------------------     ┊
┊                           ┊
┊ $spack/..                 ┊
┊ ├── access-spack-packages ┊
┊ ├── bootstrap             ┊
┊ ├── cache                 ┊
┊ ├── environments          ┊
┊ ├── intel                 ┊
┊ ├── release               ┊
┊ ├── release/modules       ┊
┊ ├── spack                 ┊
┊ ├── spack-config          ┊
┊ ├── spack-packages        ┊
┊ ├── spack-admin-cache     ┊
┊ └── spack-admin-config    ┊
╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯
              |
              V
╭╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╮
┊ Downstream / Regular User        ┊
┊ -------------------------        ┊
┊                                  ┊
┊ /g/data/$PROJECT/$USER/spack/1.1 ┊
┊ ├── bootstrap                    ┊
┊ ├── cache                        ┊
┊ ├── environments                 ┊
┊ ├── release                      ┊
┊ ├── release/modules              ┊
┊ ├── sourcecache                  ┊
┊ ├── spack-user-cache             ┊
┊ └── spack-user-config            ┊
╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯

Problem 1

• When using scopes we need a way to temporarily disable a higher-precedence scope, that is used by the downstream, in order for an action to taken as an upstream.

Workaround

CI Admin user

• export ACCESS_SPACK_ADMIN=1

Gadi Admin user

• export ACCESS_SPACK_ADMIN=1

Regular user

• Running printenv | grep ACCESS_SPACK_ADMIN should not return any string.

Problem 2

spack-enable.bash allowed us to set $SPACK_USER_CACHE_PATH, thereby having a separate user cache for each Spack instance. By default Spack sets $SPACK_USER_CACHE_PATH to ~/.spack resulting in all Spack instances using the same directory.

The following locations can not be overridden via spack-config:

reports_path = os.path.join(user_cache_path, "reports")
user_repos_cache_path = os.path.join(user_cache_path, "git_repos")
package_repos_path = os.path.join(user_cache_path, "package_repos")

lib/spack/spack/paths.py:

# Below paths are where Spack can write information for the user.
# Some are caches, some are not exactly caches.
#
# The options that start with `default_` below are overridable in
# `config.yaml`, but they default to use `user_cache_path/<location>`.
#
# You can override the top-level directory (the user cache path) by
# setting `SPACK_USER_CACHE_PATH`. Otherwise it defaults to ~/.spack.
#
def _get_user_cache_path():
    return os.path.expanduser(os.getenv("SPACK_USER_CACHE_PATH") or "~%s.spack" % os.sep)


user_cache_path = str(PurePath(_get_user_cache_path()))

#: junit, cdash, etc. reports about builds
reports_path = os.path.join(user_cache_path, "reports")

#: installation test (spack test) output
default_test_path = os.path.join(user_cache_path, "test")

#: spack monitor analysis directories
default_monitor_path = os.path.join(reports_path, "monitor")

#: git repositories fetched to compare commits to versions
user_repos_cache_path = os.path.join(user_cache_path, "git_repos")

#: default location where remote package repositories are cloned
package_repos_path = os.path.join(user_cache_path, "package_repos")

#: bootstrap store for bootstrapping clingo and other tools
default_user_bootstrap_path = os.path.join(user_cache_path, "bootstrap")

#: transient caches for Spack data (virtual cache, patch sha256 lookup, etc.)
default_misc_cache_path = os.path.join(user_cache_path, spack_instance_id, "cache")

Workaround 2.1

export SPACK_USER_CACHE_PATH="$spack/../spack-user-cache"

(Partial) Workaround 2.2

Explicitly override options default_user_bootstrap_path and default_misc_cache_path:

# spack config blame | egrep 'root:|source_cache|environments_root|misc_cache|tcl:|lmod:|destination:|install_tree:'
spack-config/v1.1/repos.yaml:6                            destination: $spack/../access-spack-packages
spack-config/v1.1/repos.yaml:11                           destination: $spack/../spack-packages
spack-config/v1.1/modules.yaml:7                            tcl: $spack/../release/modules
spack-config/v1.1/modules.yaml:8                            lmod: $spack/../release/lmod
spack-config/v1.1/modules.yaml:9                          tcl:
spack/etc/spack/defaults/base/modules.yaml:51             lmod:
spack-config/v1.1/config.yaml:2                         install_tree:
spack-config/v1.1/config.yaml:4                           root: $spack/../release
spack-config/v1.1/config.yaml:6                         source_cache: $spack/../sourcecache
spack-config/v1.1/config.yaml:8                         environments_root: $spack/../environments
spack-config/v1.1/config.yaml:9                         misc_cache: $spack/../cache
spack-config/v1.1/bootstrap.yaml:4                      root: $spack/../bootstrap

Implementation

• Spack Architecture for CI and Gadi with Spack v1.1 (2026) #110
• include.yaml: redirect to ACCESS-NRI's spack-config spack#30
• Replace user scope with access.nri.admin, access.nri.ci.user and access.nri.gadi.user #114
• Added spack module build-cd#367

Spack set up Instructions

Gadi Admin user

1. Git clone ACCESS-NRI/spack and ACCESS-NRI/spack-config
2. . spack/share/spack/setup-env.sh
3. export SPACK_USER_CACHE_PATH="$SPACK_ROOT/../spack-admin-cache"
4. export ACCESS_SPACK_ADMIN=1
5. spack bootstrap now

Gadi Regular user (default install_tree: /g/data/$PROJECT/$USER/spack/1.1/release)

0. Move any existing Spack directories out of the way:

mv ~/.spack ~/.spack-$(date +%s)
mv /g/data/$PROJECT/$USER/spack/1.1/ /g/data/$PROJECT/$USER/spack/1.1-$(date +%s)

module use /g/data/vk83/prerelease/modules
module load spack/1.1

2. spack bootstrap now
3. List user installed Spack packages, Release Spack packages and Prerelease packages (Run spack config get upstreams to see the names of available upstreams):

spack find --install-tree local
spack find --install-tree release
spack find --install-tree prerelease 

Gadi Advanced user

4. (Optional) Add an editable Spack repository (Run spack repo list to see enabled repositories in order of highest precedence to lowest precedence):

cd /g/data/$PROJECT/$USER/spack/1.1
git clone https://github.com/ACCESS-NRI/access-spack-packages
spack repo add --scope=user access-spack-packages/spack_repo/access/nri

5. (Optional) Disable upstreams: run spack config --scope=access.nri.gadi.user edit upstreams and insert upstreams:: {}.

CI Admin user

• https://github.com/ACCESS-NRI/build-ci/blob/dockerfile-fixes/containers/Dockerfile
• docker build -f Dockerfile -t spack-upstream-rocky:v1.1.1-20260330 --target upstream --build-arg SPACK_REPO_VERSION=test-v1.1 --build-arg SPACK_CONFIG_REPO_VERSION=isolate-cache-config --no-cache --progress=plain .

CI Regular user

• https://github.com/ACCESS-NRI/build-ci/blob/dockerfile-fixes/containers/Dockerfile
• docker build -f Dockerfile -t spack-runner-rocky:v1.1.1-20260330 --target runner --build-arg SPACK_REPO_VERSION=test-v1.1 --build-arg SPACK_CONFIG_REPO_VERSION=isolate-cache-config --no-cache --progress=plain .

[atteggiani]

Related to this, I created a test spack module using the testing instance in /g/data/vk83/testing/apps/spack/.

The test module is in /scratch/public/spack_modules and can be loaded with

module use /scratch/public/spack_modules
module load spack/1.1

It seems to work fine, but I have not tried using it to go over the Test Spack Hive Docs example yet.

I've given read and execute permissions to everyone in tm70, so feel free to try it and share feedback on it.

[aidanheerdegen]

Nice.

It seems to work fine

Yes it does load correctly, but it is unload that has always been the issue.

$ module use /scratch/public/spack_modules
$ module load spack/1.1
Spack environment set up using /g/data/vk83/testing/apps/spack/1.1/spack/share/spack/setup-env.sh
$ echo $PATH
/g/data/vk83/testing/apps/spack/1.1/spack/bin:/home/502/aph502/bin:/opt/pbs/default/bin:/opt/nci/bin:/opt/bin:/opt/Modules/v4.3.0/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/default/bin:.
$ module list
Currently Loaded Modulefiles:
 1) pbs   2) dot   3) spack/1.1(default)  
$ module unload spack
==> Error: SPACK_ROOT must point to spack's prefix when using bash
Run this with the correct prefix before sourcing setup-env.sh:
 export SPACK_ROOT=</path/to/spack>
$ echo $PATH
/g/data/vk83/testing/apps/spack/1.1/spack/bin:/home/502/aph502/bin:/opt/pbs/default/bin:/opt/nci/bin:/opt/bin:/opt/Modules/v4.3.0/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/default/bin:.

To do this properly the environment script that is sourced needs to be translated into equivalent module (TCL) commands.

https://modules.readthedocs.io/en/v5.1.0/cookbook/source-script-in-modulefile.html

This is what the current version of the modules on gadi cannot do (and I don't think they've upgraded it).

I think I did try installing a new version of modules and do the conversion but it didn't seem to work with the module version on gadi, but it is probably worth trying again. Given it doesn't change much perhaps if there are small problems perhaps we could edit the resulting TCL so that it does work.

Otherwise we could offer this as an option with the express proviso that it is not reversible, and users will need to login in again, or first start a new shell before doing this so they can exit and have a clean session.

Note there will be a new machine next year, and this may no longer be a problem.

[atteggiani]

I see.

Yes, unfortunately the ability to source files in Environment Modules was added in version 4.6, and the one on gadi is 4.3.

I had another solution before that was not as simple (for the modulefile itself), but would work also for unloads.
I placed it in /scratch/public/spack_modules/spack/1.1_test

The problem about this is that uses set-function to set functions, but that only sets them in the users environment (not in the process environment).
This means that spack (the function set by setup-env.sh, not the executable in the PATH) would not be available in subprocesses.
Apparently there is no specific command to export functions to the process environment with Environment Modules (or at least I couldn't find a solution).

A final solution could be to use a similar approach to the 1.1_test modulefile, but setting functions by running export -f <func_name>=<body> (which would be straightforward with the current implementation) on module load, instead of set-function ....
Then, on module unload / purge, we could run something like unset -f <func_name>.

I will have a test later today.

[aidanheerdegen]

I wouldn't spend too much time on a solution that won't be required in a year when we can pretty much get the functionality we want by sourcing a bash file. Not quite as nice, but almost as effective.