MODULEPATH broken across different shell invocations, due to poor package delivery for env. modules

[fgeorgatos]

Dear GCP,

Here is an example of the bug manifestation.

Would you agree that the duplications and arbitrary ordering across different shell variants is a bug?

mkdir -p /tmp/hello; module use /tmp/hello ; for i in sh csh tcsh zsh ksh bash "bash -l"; do echo == SHELL is $i; echo 'echo $MODULEPATH'| $i ; done
== SHELL is sh
/tmp/hello:/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles:/opt/apps/modulefiles
== SHELL is csh
/etc/scl/modulefiles:/tmp/hello:/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles:/opt/apps/modulefiles
== SHELL is tcsh
/etc/scl/modulefiles:/tmp/hello:/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles:/opt/apps/modulefiles
== SHELL is zsh
/tmp/hello:/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles:/opt/apps/modulefiles
== SHELL is ksh
/tmp/hello:/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles:/opt/apps/modulefiles
== SHELL is bash
/tmp/hello:/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles:/opt/apps/modulefiles
== SHELL is bash -l
/etc/scl/modulefiles:/tmp/hello:/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles:/opt/apps/modulefiles

[fgeorgatos]

btw. for the record: this is routinely broken in some HPC sites - alas, it is still a bug

[shubpal07]

RCA: Slurm Lmod MODULEPATH Corruption

We conducted a technical investigation and live cluster validation of the MODULEPATH corruption issue on Rocky Linux login nodes. Below is a summary of the root cause, a temporary user-space workaround, and the permanent system-level fix.

---

1. Core Diagnosis

On Enterprise Linux distributions (RHEL / Rocky Linux 8 & 9), the installation of legacy packages like environment-modules and Software Collections (scl-utils) deploys default system startup scripts under /etc/profile.d/ (specifically modules.sh, modules.csh, scl-init.sh, and scl.sh).

When a user logs into the cluster:

1. Both the legacy Tcl-based startup scripts and Lmod's modern z00_lmod.sh script execute in sequence.
2. The legacy scripts overwrite and distort the correct MODULEPATH defined by Lmod and Slurm.
3. This clashing initialization prepends /etc/scl/modulefiles or duplicates paths, pushing custom application modulepaths to lower priority (e.g. prepended paths via module use are pushed back from Index 0).
4. Consequently, custom cluster application modules become completely invisible or fail to load inside interactive login shells (bash -l).

Note: Ubuntu Slurm login nodes are entirely immune to this bug, as Ubuntu does not ship those legacy RHEL packages/scripts by default.

---

2. Immediate Workaround (User-Space)

If users need an immediate, safe fix on existing clusters without rebuilding their VM images, they can append a reversed-deduplication pipeline to their shell profile (~/.bashrc).

This dynamically corrects the MODULEPATH priority, pushing custom paths back to the front (Index 0) and removing duplicates upon logging in:

# Temporary workaround for SchedMD Lmod MODULEPATH bug
export MODULEPATH=$(echo -n $MODULEPATH | tr ":" "\n" | tac | awk '!x[$0]++' | tac | paste -sd:)

The robust, permanent solution is to cleanly disable the conflicting legacy shell hooks on the VM image. Our team is working on disabling them. GoogleCloudPlatform/slurm-gcp#344

[fgquantori]

hi,

Let's ask, after merging the PR 344, what would the output be of the following command under rhel8 and rhel9 ? (to check both)
mkdir -p /tmp/hello; module use /tmp/hello ; for i in sh csh tcsh zsh ksh bash "bash -l"; do echo == SHELL is $i; echo 'echo $MODULEPATH'| $i ; done

More to it, we may wish to deploy environment-modules or lmod via our own rpm or git mechanism.
If we do as such, there may still be a conflict because there are certain packaged files being deployed by default.
i.e. we shall need the lmod equivalent of "install_ompi": false, and for env.modules itself (which is to my undersanding why this case remains open)

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[fgeorgatos]

hi @shubpal07,

what is the intention with this issue? any ETA known?

[fgquantori]

FYI. this issue has also been reported here: GoogleCloudPlatform/slurm-gcp#346
and is expected to receive a fix via this path: GoogleCloudPlatform/slurm-gcp#350

Kindly do not close the case until we have independent validations of the fix!