Branch infra-update-v8 (Spack 1.1) fails to build

[penguian]

In trying to diagnose problems with https://github.com/ACCESS-NRI/access-spack-packages/releases/tag/2026.02.004 I tried to spack install the infra-update-v8 branch of https://github.com/ACCESS-NRI/access-ram3 with the following result.

[pcl851@cylc ACCESS-rAM3]$ cat config/versions.json
{
  "$schema": "https://raw.githubusercontent.com/ACCESS-NRI/schema/main/au.org.access-nri/model/deployment/config/versions/4-0-0.json",
  "spack": "1.1",
  "access-spack-packages": "2026.02.004",
  "custom-scopes": [
    "ukmo-restricted-scope"
  ]
}
[pcl851@cylc ACCESS-rAM3]$ spack clean --downloads
==> Removing cached downloads
[pcl851@cylc ACCESS-rAM3]$ spack install --no-cache
[+] /g/data/tm70/pcl851/spack/1.1/release/linux-x86_64/compiler-wrapper-1.0-spro2dgnjjawulhw7fxxrqskm73cf4dd
[+] /apps/cmake/3.31.6 (external cmake-3.31.6-c6epq3ufiplry3hihbrr5qtcqd2xezuo)
[+] /usr (external glibc-2.28-vuczjrbyzfif5nzgt5gqbrdrzaioihy6)
==> intel-oneapi-compilers-classic@2021.10.0 : has external module in ['intel-compiler/2021.10.0']
[+] /apps/intel-ct/wrapper (external intel-oneapi-compilers-classic-2021.10.0-rjzvcxkrnsiza6vps4v2kkgfn2swyhfs)
==> python@3.11.7 : has external module in ['python3/3.11.7']
[+] /apps/python3/3.11.7 (external python-3.11.7-7loeqvkv4rl3e3bgp7oqlrr4vyabqeaz)
[+] /g/data/tm70/pcl851/spack/1.1/release/linux-x86_64/gmake-4.4.1-d5nd3lks3aornpidrxz2hbteka3uuk3r
[+] /g/data/tm70/pcl851/spack/1.1/release/linux-x86_64/fcm-2021.05.0-t2zjdq6prcofscqzamgzazexwskb7do3
[+] /g/data/tm70/pcl851/spack/1.1/release/linux-x86_64/zlib-ng-2.3.2-ilvuvnwmuk6d6ciz56tz5snlpvpmlqp4
[+] /g/data/tm70/pcl851/spack/1.1/release/linux-x86_64/pkgconf-2.5.1-r4kdoungwiova7mfdjjjxl4eqblpjnw5
==> Installing gcom-8.3-tuibhrr4semgie65pejevosbxvh27n2s [10/15]
error: /g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-14.1.0/gh-2.43.1-qg3yh3qyl2nox4fhoixo6puzqvqel5lm/bin/gh auth git-credential get died of signal 4
error: /g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-14.1.0/gh-2.43.1-qg3yh3qyl2nox4fhoixo6puzqvqel5lm/bin/gh auth git-credential store died of signal 4
==> No patches needed for gcom
==> gcom: Executing phase: 'install'
==> Error: ProcessError: Command exited with status 255:
    '/g/data/tm70/pcl851/spack/1.1/release/linux-x86_64/fcm-2021.05.0-t2zjdq6prcofscqzamgzazexwskb7do3/bin/fcm' 'make' '-f' 'fcm-make/gcom.cfg'
See build log for details:
  /scratch/tm70/pcl851/tmp/spack-stage/spack-stage-gcom-8.3-tuibhrr4semgie65pejevosbxvh27n2s/spack-build-out.txt

==> Warning: Skipping build of um-13.5-3tivnsdhl7pbnb6m4pfcotreymjhdga5 since gcom-8.3-tuibhrr4semgie65pejevosbxvh27n2s failed
==> Warning: Skipping build of access-ram3-latest-yundwrs6fvg2ztwtvhjqgdb54626r5jl since um-13.5-3tivnsdhl7pbnb6m4pfcotreymjhdga5 failed
[+] /g/data/tm70/pcl851/spack/1.1/release/linux-x86_64/hdf5-1.14.6-knhjqfj4qrh735eqjr6suhld7wb3ggjt
[+] /g/data/tm70/pcl851/spack/1.1/release/linux-x86_64/netcdf-c-4.9.2-dgnxg5v2avq6rqhh5ghfa2rqw7hwynls
==> Installing eccodes-2.34.0-o7vqax2mguufjwytdaynljqxaz7k5urx [13/15]
==> Fetching https://mirror.spack.io/_source-cache/archive/3c/3cd208c8ddad132789662cf8f67a9405514bfefcacac403c0d8c84507f303aba.tar.gz
    [100%]   12.19 MB @  146.7 MB/s
==> No patches needed for eccodes
==> eccodes: Executing phase: 'cmake'
==> eccodes: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    '/g/data/tm70/pcl851/spack/1.1/release/linux-x86_64/gmake-4.4.1-d5nd3lks3aornpidrxz2hbteka3uuk3r/bin/make' '-j16'

713 errors found in build log:
     2895    cd /scratch/tm70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7vqax2mguufjwytdaynljqxaz7k5urx/spack-build-o7vqax2/src && /g/data/tm70/pcl851/spack/1.1/release/linux-x86_
             64/compiler-wrapper-1.0-spro2dgnjjawulhw7fxxrqskm73cf4dd/libexec/spack/intel/icpc -DHAVE_ECCODES_CONFIG_H -I/scratch/tm70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7v
             qax2mguufjwytdaynljqxaz7k5urx/spack-build-o7vqax2 -I/scratch/tm70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7vqax2mguufjwytdaynljqxaz7k5urx/spack-src/src -I/scratch/t
             m70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7vqax2mguufjwytdaynljqxaz7k5urx/spack-build-o7vqax2/src -O3 -DNDEBUG -std=gnu++17 -fPIC -MD -MT src/CMakeFiles/eccodes-s
             tatic.dir/grib_accessor_class_transient_darray.cc.o -MF CMakeFiles/eccodes-static.dir/grib_accessor_class_transient_darray.cc.o.d -o CMakeFiles/eccodes-static.dir/grib_accesso
             r_class_transient_darray.cc.o -c /scratch/tm70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7vqax2mguufjwytdaynljqxaz7k5urx/spack-src/src/grib_accessor_class_transient_d
             array.cc
     2896    [ 80%] Building CXX object src/CMakeFiles/eccodes-static.dir/grib_accessor_class_values.cc.o
     2897    cd /scratch/tm70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7vqax2mguufjwytdaynljqxaz7k5urx/spack-build-o7vqax2/src && /g/data/tm70/pcl851/spack/1.1/release/linux-x86_
             64/compiler-wrapper-1.0-spro2dgnjjawulhw7fxxrqskm73cf4dd/libexec/spack/intel/icpc -DHAVE_ECCODES_CONFIG_H -I/scratch/tm70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7v
             qax2mguufjwytdaynljqxaz7k5urx/spack-build-o7vqax2 -I/scratch/tm70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7vqax2mguufjwytdaynljqxaz7k5urx/spack-src/src -I/scratch/t
             m70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7vqax2mguufjwytdaynljqxaz7k5urx/spack-build-o7vqax2/src -O3 -DNDEBUG -std=gnu++17 -fPIC -MD -MT src/CMakeFiles/eccodes-s
             tatic.dir/grib_accessor_class_values.cc.o -MF CMakeFiles/eccodes-static.dir/grib_accessor_class_values.cc.o.d -o CMakeFiles/eccodes-static.dir/grib_accessor_class_values.cc.o 
             -c /scratch/tm70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7vqax2mguufjwytdaynljqxaz7k5urx/spack-src/src/grib_accessor_class_values.cc
     2898    [ 80%] Building CXX object src/CMakeFiles/eccodes-static.dir/grib_accessor_class_simple_packing_error.cc.o
     2899    cd /scratch/tm70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7vqax2mguufjwytdaynljqxaz7k5urx/spack-build-o7vqax2/src && /g/data/tm70/pcl851/spack/1.1/release/linux-x86_
             64/compiler-wrapper-1.0-spro2dgnjjawulhw7fxxrqskm73cf4dd/libexec/spack/intel/icpc -DHAVE_ECCODES_CONFIG_H -I/scratch/tm70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7v
             qax2mguufjwytdaynljqxaz7k5urx/spack-build-o7vqax2 -I/scratch/tm70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7vqax2mguufjwytdaynljqxaz7k5urx/spack-src/src -I/scratch/t
             m70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7vqax2mguufjwytdaynljqxaz7k5urx/spack-build-o7vqax2/src -O3 -DNDEBUG -std=gnu++17 -fPIC -MD -MT src/CMakeFiles/eccodes-s
             tatic.dir/grib_accessor_class_simple_packing_error.cc.o -MF CMakeFiles/eccodes-static.dir/grib_accessor_class_simple_packing_error.cc.o.d -o CMakeFiles/eccodes-static.dir/grib
             _accessor_class_simple_packing_error.cc.o -c /scratch/tm70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7vqax2mguufjwytdaynljqxaz7k5urx/spack-src/src/grib_accessor_class
             _simple_packing_error.cc
     2900    /g/data/tm70/pcl851/spack/1.1/release/linux-x86_64/netcdf-c-4.9.2-dgnxg5v2avq6rqhh5ghfa2rqw7hwynls/lib/libnetcdf.a(libnchdf5_la-hdf5file.o): In function `nc4_enddef_netcdf4_fi
             le':
  >> 2901    hdf5file.c:(.text+0x31e): undefined reference to `H5Fflush'
...
     4048    /bin/ranlib ../lib/libeccodes.a
     4049    make[2]: Leaving directory '/scratch/tm70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7vqax2mguufjwytdaynljqxaz7k5urx/spack-build-o7vqax2'
     4050    [100%] Built target eccodes-static
     4051    make[1]: Leaving directory '/scratch/tm70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7vqax2mguufjwytdaynljqxaz7k5urx/spack-build-o7vqax2'
  >> 4052    make: *** [Makefile:169: all] Error 2

See build log for details:
  /scratch/tm70/pcl851/tmp/spack-stage/spack-stage-eccodes-2.34.0-o7vqax2mguufjwytdaynljqxaz7k5urx/spack-build-out.txt

==> Error: access-ram3-latest-yundwrs6fvg2ztwtvhjqgdb54626r5jl: Package was not installed
==> Error: Installation request failed.  Refer to reported errors for failing package(s).
[pcl851@cylc ACCESS-rAM3]$ git status -uno
On branch infra-update-v8
Your branch is up to date with 'origin/infra-update-v8'.

nothing to commit (use -u to show untracked files)
[pcl851@cylc ACCESS-rAM3]$ cat  /scratch/tm70/pcl851/tmp/spack-stage/spack-stage-gcom-8.3-tuibhrr4semgie65pejevosbxvh27n2s/spack-build-out.txt
==> gcom: Executing phase: 'install'
==> [2026-03-04-00:32:45.871611] '/g/data/tm70/pcl851/spack/1.1/release/linux-x86_64/fcm-2021.05.0-t2zjdq6prcofscqzamgzazexwskb7do3/bin/fcm' 'make' '-f' 'fcm-make/gcom.cfg'
[init] make                # 2026-03-03T13:32:46Z
[info] FCM 2021.05.0 (/g/data/tm70/pcl851/spack/1.1/release/linux-x86_64/fcm-2021.05.0-t2zjdq6prcofscqzamgzazexwskb7do3)
[init] make config-parse   # 2026-03-03T13:32:46Z
[info] config-file=/scratch/tm70/pcl851/tmp/spack-stage/spack-stage-gcom-8.3-tuibhrr4semgie65pejevosbxvh27n2s/spack-src/fcm-make/gcom.cfg
[info] config-file= - /scratch/tm70/pcl851/tmp/spack-stage/spack-stage-gcom-8.3-tuibhrr4semgie65pejevosbxvh27n2s/spack-src/fcm-make/machines/nci_gadi_ifort_serial.cfg
[done] make config-parse   # 0.7s
[init] make dest-init      # 2026-03-03T13:32:47Z
[info] dest=pcl851@cylc.pcl851.tm70.ps.gadi.nci.org.au:/scratch/tm70/pcl851/tmp/spack-stage/spack-stage-gcom-8.3-tuibhrr4semgie65pejevosbxvh27n2s/spack-src
[info] mode=new
[done] make dest-init      # 0.0s
[init] make extract        # 2026-03-03T13:32:47Z
[info] location  gcom: 0: /scratch/tm70/pcl851/tmp/spack-stage/spack-stage-gcom-8.3-tuibhrr4semgie65pejevosbxvh27n2s/spack-src
[info]   dest:  191 [A added]
[info] source:  191 [U from base]
[done] make extract        # 1.1s
[init] make preprocess     # 2026-03-03T13:32:48Z
[info] sources: total=191, analysed=191, elapsed-time=0.3s, total-time=0.2s
[FAIL] mpicc -E -I./include /scratch/tm70/pcl851/tmp/spack-stage/spack-stage-gcom-8.3-tuibhrr4semgie65pejevosbxvh27n2s/spack-src/extract/gcom/gc/gc__abort.c: command not found
[FAIL] process    0.0 ! gcom/gc/gc__abort.c  <- gcom/gc/gc__abort.c
[info] target-tree-analysis: elapsed-time=0.0s
[FAIL] ! gcom/gc/gc__abort.c : update task failed

[info] install   targets: modified=7, unchanged=0, failed=0, total-time=0.0s
[info] process   targets: modified=183, unchanged=0, failed=1, total-time=12.2s
[info] TOTAL     targets: modified=190, unchanged=0, failed=1, elapsed-time=12.6s
[FAIL] make preprocess     # 13.0s
[FAIL] make                # 14.8s

[penguian]

The diagnosis of this issue is subtle. I asked Claude Opus 4.6 (Thinking) for a diagnosis and fix, and it came up with:

Root Cause Analysis

Analysis based on access-spack-packages tag 2026.02.004 (commit 7646b52).

There are two independent build failures in this issue. Both can be fixed with changes to spack.yaml alone — no changes to access-spack-packages are required.

Failure 1: gcom — mpicc: command not found

Symptom. The gcom build log shows:

[info] config-file= - .../fcm-make/machines/nci_gadi_ifort_serial.cfg
...
[FAIL] mpicc -E ... command not found

gcom selected the serial FCM machine config (no MPI) and then tried to invoke mpicc, which is not on its PATH because openmpi was never included in the build environment.

Root cause: the spack v1.1 concretizer chose gcom@8.3~mpi.

Even though the gcom recipe declares variant("mpi", default=True), the concretizer is free to override that default. According to the Spack FAQ on concretizer precedence, variant defaults in package.py are the lowest priority optimization criterion:

requirements + constraints > strong preferences > reuse > preferences > defaults

The default variant("mpi", default=True) sits at the "defaults" level. Without an explicit require or a root-spec constraint forcing +mpi, the concretizer is free to choose ~mpi if that yields a better overall solution. In this case it does, because:

1. The root spec in spack.yaml is just access-ram3 — it places no constraint on gcom's mpi variant.
2. The packages: gcom: require: section says '@8.3' but does not include '+mpi'.
3. The gcom recipe has a conditional dependency depends_on("openmpi@4.1.3:", when="+mpi^[virtuals=mpi] openmpi"). With +mpi, the concretizer must pull in openmpi and satisfy the @4.1.3: constraint. Choosing ~mpi avoids this entire dependency chain — a "cheaper" solution.

In short, +mpi was never required anywhere, so the concretizer optimized it away.

Note on the "+mpi" in spec recipe check. The gcom recipe at 2026.02.004 uses:

if "+mpi" in spec:
    mach_m = "mpp"
else:
    mach_m = "serial"

During debugging, this was changed to spec.satisfies("+mpi") to follow spack v1 best practices, but that change is not the fix and is not required. The "in spec" operator is used in 100+ places in the official spack/spack-packages repository and still works correctly in spack v1.x. The problem is that the concretizer chose ~mpi in the first place — both "+mpi" in spec and spec.satisfies("+mpi") correctly return False for a ~mpi spec.

Failure 2: eccodes — 713 undefined HDF5 symbol references

Symptom. The eccodes link step fails with hundreds of undefined references like:

libnetcdf.a(libnchdf5_la-hdf5file.o): undefined reference to `H5Fflush'

Root cause: static linking of netcdf-c without HDF5 on the link line.

The concretizer chose netcdf-c built as a static library (~shared). When eccodes links against libnetcdf.a, the linker needs the transitive dependency libhdf5.a to be explicitly on the link line (static linking does not resolve transitive dependencies automatically). The eccodes CMake build does not add HDF5 to its link flags because it relies on netcdf-c's pkg-config/cmake config, which in turn assumes shared libraries.

The fix is to require shared libraries for netcdf-c and hdf5.

---

Fix — spack.yaml changes only

All changes are to spack.yaml on the infra-update-v8 branch. No changes to access-spack-packages recipes are required.

Change 1: Require +mpi on gcom

Add '+mpi' to the existing gcom require list:

    gcom:
      require:
      - '@8.3'
+     - '+mpi'

Per the Spack FAQ on concretizer precedence, a require directive is a hard constraint at the same precedence level as command-line constraints and root-spec dependencies — it overrides defaults, preferences, and reuse. The existing require: ['@8.3'] only constrained the version; adding '+mpi' forces the concretizer to select the MPI variant.

Alternatively, this can be expressed in the root spec instead:

  specs:
-  - access-ram3
+  - access-ram3 ^gcom@8.3+mpi

Both approaches are equivalent. The require form is preferred here because it keeps the root spec clean and groups all gcom constraints together.

Change 2: Require shared libraries for netcdf-c and hdf5

Add +shared to the existing netcdf-c requirements, and add HDF5 requirements if not already present:

    netcdf-c:
      require:
      - '@4.9.2'
+     - '+shared'
+   hdf5:
+     require:
+     - '+shared'

This ensures that eccodes can link against libnetcdf.so without needing to explicitly list HDF5 symbols.

Change 3: Disable OpenGL on jasper

+   jasper:
+     require:
+     - '~opengl'

This prevents a counter-intuitive transitive dependency chain. Without ~opengl, the concretizer pulls in jasper with OpenGL support (its default), which triggers the following cascade:

jasper +opengl → mesa (OpenGL implementation) → llvm (mesa's shader compiler) → z3, python, ...

This has two effects:

1. z3 fails to build with icpc (Intel classic C++ compiler), so the entire build fails.
2. The package count explodes from ~24 to ~40, including building Python 3.14 from source (the external Python 3.11 on Gadi isn't sufficient for llvm's requirements).

Disabling OpenGL on jasper is safe because jasper is only used for JPEG2000 support in eccodes — OpenGL is irrelevant.

Change 4: Pin Python to the Gadi external

+   python:
+     require:
+     - '@3.11.7'

Without this, a fresh spack concretize --force can choose Python 3.14 and attempt to build it from source (along with all its dependencies), even though Python 3.11.7 is available as an external on Gadi. This is another instance of the same concretizer behaviour: the external Python is only a "preference", not a hard constraint. Adding a require makes it a hard constraint.