Search for module taking loaded modules into account

[Flamefire]

In short: We need a module search command between module spider <foo> and module available <foo>.

The former lists all modules even when you need to load other modules, while the latter lists only modules that don't need other modules to be loaded.

Now suppose you have 2 installations of GCC, each expanding the modulepath such that more modules become visible. Same with e.g. OpenMPI.

So in order to load Foo/1 you have to load GCC/1 OpenMPI/1 and for Foo/2 you need GCC/2 OpenMPI/2.

Now I would want this (new?) module search command to show exactly one of the 2 Foo modules when I have any of the 2 GCC modules loaded. module av Foo would be empty because OpenMPI isn't yet loaded and module spider Foo would show both and you need to run module spider on each of them to find which one you can use given the currently loaded modules.

In other words I want the module spider list filtered by requirements matching the currently loaded modules.

This is useful for hierarchical module systems (as described here: GCC enables OpenMP which enables Foo) as well as (how we call it) "module environments" where we have special modenv/<arch> modules that enable modules for a given architecture. But as those modules are also found by module spider users see modules for other architectures then the one they are using which makes it hard to use.

[mboisson]

For your <arch> example, you actually don't want it to be displayed by spider at all, ever. We do that by excluding it from spider:

$ cat /cvmfs/soft.computecanada.ca/custom/modules/arch/avx512.lua
if (mode() ~= "spider") then
	prepend_path("MODULEPATH","/cvmfs/soft.computecanada.ca/custom/modules-avx512")
	setenv("EBVERSIONARCH","avx512")
end

I guess you could do some more complex logic in that condition, to make it included in spider only based on loaded modules.

[mboisson]

Basically, you want to see whether there is already a GCC loaded which is different from the current module. If so, then don't prepend_path... something like

if ( not isloaded("GCC") or mode() ~= "spider") then
    prepend_path("MODULEPATH", "....")
end

[Flamefire]

@mboisson But then how does LMod (and the spider cache) know about modules inside /cvmfs/soft.computecanada.ca/custom/modules-avx512? Without the spider cache Lmod is just to slow and I guess once that path ends up in the spider cache somehow the modules will be listed by module spider, won't they?

So

you actually don't want it to be displayed by spider at all, ever.

doesn't actually applies: We want those modules to be displayed by spider if that module is loaded.

It might make sense to have some "global spider" which does show them without the module being loaded, i.e. the current behavior. Not sure how useful that is hence the idea of a more restricted spider as an additional command/flag.

[mboisson]

@mboisson But then how does LMod (and the spider cache) know about modules inside /cvmfs/soft.computecanada.ca/custom/modules-avx512? Without the spider cache Lmod is just to slow and I guess once that path ends up in the spider cache somehow the modules will be listed by module spider, won't they?

Ah, for caching, we actually have separate caches for separate architectures. When swapping architecture, it swaps the cache

So

you actually don't want it to be displayed by spider at all, ever.

doesn't actually applies: We want those modules to be displayed by spider if that module is loaded.

it would still get displayed if that module is loaded, because then the MODULEPATH is already set.

[Flamefire]

Ah, for caching, we actually have separate caches for separate architectures. When swapping architecture, it swaps the cache

Yes I was thinking about that too, it is just that the idea proposed here would also solve that use case.

Recent example where this came up: User has some module he needs, hence a fixed compiler, and needs some TensorFlow module. Now we have multiple TensorFlow modules with different versions, different toolchains/compilers (and even different archs) and finding the 1 or 2 modules that could be loaded with the current compiler is quite cumbersome.

[mboisson]

Yes, I can see how it could be useful, but I suspect it may be more complicated than expected. Introducing a new command also requires user training (or they won't use it) unless the default spider mode is changed (and then you get the surprise of not finding all modules...)

[mboisson]

I wonder if it could be implemented through a spider hook.