Completion of the symlink feature

[muffato]

Hi

Follow-up of #502 with these changes:

1. Added support for shorter symlinks (no module in the name) for Lmod when default_version is False. Looks like in Adding testing of a symlink tree install #502 (comment)
2. A few bug-fixes
3. General tidy-up

Tested with Lmod and TCL

[muffato]

:/ Let me check why the CI is failing

[vsoch]

My guess is that we are trying to write a .version file and the directory above isn't created yet?

[muffato]

See the error:

FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/work/singularity-hpc/singularity-hpc/modules/tmp-modules/python/.version'

and notice modules/tmp-modules: write_version_file was concatenating $module_base with the path it got (here the symlink path tmp-modules/python). I didn't spot this before because I had always used an absolute path for $symlink_base.

[vsoch]

@muffato one thing that just came up in a conversation with someone is this idea of development environments. Arguably a symlink base (as we have it here) allows for generating an entire tree at some single named symlink base, and then loading from it. But is it so crazy to extend the idea to say "you can have as many little unique trees as you like"? The issue (as I see it) is that we don't have a good way to manage that space - e.g., let's say you have a module symlinked in three places, and we decide to uninstall a module, are we 1. happy with uninstalling in all three places, and 2. can we even find all the places?

Just something to chew on - I think shpc (for now) is primarily a "run production stuff" sort of tool, but even for that if we have use cases like we were talking about with @georgiastuart and @marcodelapierre of wanting to have particular groups of things kept together (e.g., for different MPI or cuda or whatever) allowing for multiple symlink trees (and maybe calling them environments or module groups) isn't such a crazy idea.

Let me know what y'all think! (side note, these discussions are very fun to have :) )

Update: also see marcodelapierre/toy-shpc-nf#2 !

[muffato]

lol I just noticed that you had already dubbed this feature view in #502 (comment) . All credits go back to you, then !

[vsoch]

lol I just noticed that you had already dubbed this feature view in #502 (comment) . All credits go back to you, then !

No worries @muffato ! I was also thinking of a database view when I used that term :) If we both thought it was more an appropriate term, it probably would be a good choice.

If they're held centrally, say in the container.yaml or nearby, then shpc can correctly install and uninstall everything.

I think there are two ideas here:

1. where to put the actual view
2. how to define the view

For 1., I think I still like the first option - the user can ask to create a named view:

$ shpc view create myview

And it gets generated either in the default of the user home, OR a custom view path (views_base).

$ ~/.singularity-hpc/views

The reason we don't want to just throw it into the symlink base is because I expect people would want to do:

$ module use $symlink_base

and if they were under there you'd just get a mess of conflicts, too many things trying to load, etc. For 2., how to define the view, it looks like you are suggesting we have "named views" provided by container.yaml, e.g.,:

view:
   - module_a
   - module_b

But I don't think that's the right approach because it means that everyone is going to have to write their cluster-specific modules into the container.yaml - what if they have different needs? Different names? it breaks very quickly. On the other hand, if the views are flexible to "whatever" you could just do:

$ shpc view create mpi
$ shpc view add mpi-my-custom-1 
$ shpc view add mpi-my-custom-2 

OR you might have some equivalent requirements file that mimics the above:

view:
  name: mpi
  modules:
    - mpi-my-custom-1
    - mpi-my-custom-2

And then

$ shpc view create view.yaml
>> Creating view mpi from view.yaml...

And arguably if you create it manually first, you could just export it.

$ shpc view export mpi
>> Exporting view mpi to view.yaml

And share that view file somewhere else. This creates the needed separation (I think) between a container module and how you want to group it / use it in the context with others. It's flexible to different names or needs for groups of modules, and still able to save / share.

The reason I paused us from merging here is because I think having views would make the symlink_base defunct. The default view would generate a symlink tree. It would be confusing to merge support for a symlink_base and then provide views, which are a slight variant on that? The view is actually the same thing, just with a more elegant design (imho).

In terms of the cleaning up bits, as long as we are storing views in a consistent place it shouldn't be any different than it is now - it would just take more checks to ensure that a module isn't symlinked across any views.

And the question is who maintains those views. If they're made for the users upon their request, that could substantially increase the load on the people who are maintaining the shpc installation.

This could be an issue, I agree. But arguably if you want separation, couldn't the user just shpc install from their own clone? The use case this meets is what @marcodelapierre and @georgiastuart were talking about with respect to groups of modules to be installed together / alongside. Which makes me realize, we would want to allow the view.yaml to have separate sections for shpc and "other modules"

view:
  name: mpi
  modules:
    - mpi-my-custom-1
    - mpi-my-custom-2
  system_modules:
    - mpi
    - system_module

And then perhaps you'd "install" an shpc module to a view, and "add" a system module:

$ shpc view install clingo    # shpc symlink
$ shpc view add mpi           # system module

Actually I'm just realizing - if we are adding a system module, we'd need to change the modulefile to require the module. The symlink won't work here. But is there a way to ask the module software to, given we are loading children of some path, to write some file at a higher level that requires the modules? Then we could still do the symlinks (the module files are unchanged) but "add" would rewrite the file that specifies that in the root of the view. If that doesn't work, then symlinks won't work for this use case, and we don't actually get any added functionality with view, because you could just do shpc install with a different module base. But perhaps as you said, given you have the same shared container directory, given that we are more conservative and don't delete containers (or have setting that says not to) it would be fairly trivial for a user to do:

$ shpc config set module_base:/my/custom/base
$ shpc config set container_base:/shared/location

The issue though is permissions - the user would need write permission to write new containers, and thus they could delete old ones. If all users don't set some config command like:

$ shpc config set container_delete:false

It could get ugly really quickly! There would need to be some kind of support for a set of "admin commands" that create a setting that absolutely doesn't allow delete. Still, someone could do it manually 🤷

Okay so after thinking about this - I think:

• views could work using symlinks, but only if we can write the requirements files for the system modules separately
• an install to a view would still require install of the module to the root and then symlink, but if we don't allow that, it could install to the view directly.

Okay so now this other approach:

If we go this way, I suggest we add a views section in the container.yaml files with the name of those environments.

I think "where we define them" is a separate thing, and I don't think they should be hardcoded alongside the container.yaml, for the reasons I said earlier.

Symlinks would still always be created under $symlink_base, and also, optionally, under $symlink_base/.views/$view_name should they be added to views. I haven't checked TCL, but the hidden .views means that a Lmod that points at $symlink_base wouldn't see the views, only the main tree.

Oh interesting. But this still assumes a shared set of admin managed views... which maybe is what is desired anyway? E.g., providing groups of modules? At least at Stanford the module system had a way to provide those groups, although I don't know how it was implemented.

Then, users can decide to instead use views should they want that. I think that could be implemented relatively easily in the current code-base.

I agree the implementation won't be too bad - but we do need to figure out how this design will look, and if it relates to the symlink tree (or not) and if not, what makes it functionally different? I'm not an admin of an HPC system so I have the least experience here, but I've done my best to write out some thoughts. I'll think more on it and post again if I get any insights.

Oh, and before I forget, one use case I want to bring up (that isn't obvious here) is being able to generate quick "throw away" install environments for running something like Snakemake or Nextflow. A view would be perfect for that.

[muffato]

Right, I get your main point. Making container.yaml define the views would mean editing the registry and making it specific to the organisation/group/etc. So the views have to be defined somewhere else.

Overall it sounds like the only possibility to still allow clean uninstalls is that the user who installs/maintains shpc containers+modules would also control the views. Just need to define a config file for storing the views, etc.

I would request an option to have a view that automatically contains all software. Not only it's what I want (being selfish), I think it's what @marcodelapierre had in mind as well in #456 (comment) . That's what I thought Jacqui meant in #456 (because that's how the modules are organised on her screenshot) but paying attention to her first comment #456 (comment) she asked for groupname/software/version which could be nicely implemented with views.

---

The use case this meets is what @marcodelapierre and @georgiastuart were talking about with respect to groups of modules to be installed together / alongside

I agree that views can be used to group software that have different major dependencies, like open-mpi vs mpich. In #527 (comment) I was thinking about distinct shpc installations, but views could definitely help.

---

an install to a view would still require install of the module to the root and then symlink, but if we don't allow that, it could install to the view directly.

As a ground rule, I would keep views as strict views, i.e. one can only make an already-installed software visible in a view, but not install new software.

[vsoch]

As a ground rule, I would keep views as strict views, i.e. one can only make an already-installed software visible in a view, but not install new software.

I like that too.

[vsoch]

Okay so here is my proposal then:

We first create a new view.

$ shpc view create myview

My original thinking is that all views would be the shortened version of the symlink first. But the need to "install all modules" gives me pause for that. @muffato would you prefer an option to create a view that is configured for "full paths" (and then be able to install all) or shortened paths (default) but then possibly have some conflicts that we'd either need to automatically manage or let the last one win?

Either way, creating a new view actually means generating a new directory in some $view_base (to replace symlink base) and akin to modules and containers, we will provide a "views" directory in the root (to default to).

views/
  myview

Only software that is installed by shpc can be added to the view. E.g., this would work:

$ shpc install ghcr.io/autamus/clingo
$ shpc view install ghcr.io/autamus/clingo

And then since it's shortened symlinks by default, you'd have:

views/
  myview/
      clingo/
         module.lua

of course then you would want to cd there and do use from the myview root to avoid the extra path:

cd vieww
module use ./myview

and maybe there could be an shpc command to help that:

$ shpc view myview
$ module load $(shpc view myview)

and that we would need to figure out tricks to only load some specific level, e.g., for that to work you'd need a relative path so it couldn't be run from anywhere. Or I wonder if we could figure out how to do:

shpc load myview

Do any other tools load modules on behalf of the user and it works? Thoughts?

Then you could export, view, etc.

$ shpc view export myview view.yaml
>> myview exported to view.yaml

$ shpc view import anotherview view.yaml
>> view.yaml imported into anotherview

or probably more meaningful messages than that, or even:

$ shpc view import anotherview view.yaml
>> my view requires shpc install ghcr.io/autamus/clingo, do that first

And then to show all views, in case you forget

$ shpc views

[muffato]

Hi @vsoch .

To answer your first question, I think the case for shortened paths stands regardless of how we do views, cf #456.
Yes, there could be an option for a view to use full paths or short paths. That would work.

Something that's been bothering me about this is who would control the views, where they would be stored, and the risk of having "rogue" views that shpc uninstall could not reach.
The best I can come up with is that the admin would initiate the the directory + config files, and set up a UNIX group and the setgid bit, or an ACL, so that other users can administer views and still those views would be registered in shpc.

module load $(shpc view myview) is a good one. I was asked recently about that: can we avoid having to load X modules to run a pipeline. Alternatively, shpc could write up a single module file for an entire set of software, an entire view, that would combine all $PATH etc. Or shpc view could physically create a bin/ for the entire view and put copies of all the wrapper scripts there, so that a single directory is added to $PATH.

[marcodelapierre]

Hi @vsoch, @muffato, it's some quite overly busy weeks here, I haven't got the chance yet to review your proposed features for environments and views...sorry!

Overall, they indeed sound like a potentially very powerful new addition to SHPC.
Some of thoughts that I have, without having read the above:

1. one thing that I would like to see is the possibility to keep ONE single copy of the SIF files themselves (the bulky, disk hungry ones), and instead provide multiple copies of the lightweights moduletree, each one for each environment/view and its specific needs
2. I have noticed you're suggesting syntax to manipulate envs/views, e.g. add, delete, load, unload....amazing!! :-)
3. on the purpose of the SHPC environments/views: because by nature the containers are already sandboxed, I assume the key purpose is that to differentiate the various envs/views by the contents of the settings.yaml...? In my mind, this is indeed the key underlying differentiating factor for the multiple module trees I was talking about in point 1. above. This would also be a simple but very powerful way of providing more flexibility for container module deployments. ....Or are you considering other facets to this differentiation (vs just the settings one I am mentioning)?
4. one user case that came to mind to me (a bit adventurous to be honest): we're toying with the idea of deploying the same set of containers at different infrastructures (different sites, even in different cities), using CernVM-fs. In this context, I am suggesting we add SHPC on top of that. Now, the envs/views would be tremendous here, because they would enable one single admin to manage multiple views, one per site, each one with the site-specific configurations.

TLDR: right now, if I think of SHPC envs/views, I pretty much see it as the simple, but super-duper powerful equation one environment == one settings.yml .
I will find some time to read all of the above, and digest all of the additional aspects I am missing at the moment.

[marcodelapierre]

Are you folks thinking of decoupling the release of the symlink features from that of the new envs/views? Or are you thinking of doing all of it before releasing?

[vsoch]

I think my realization is that the symlink feature is a kind of view, so I wanted to figure views out and refactor to that, and not for example have a few releases of shpc that then change how it works entirely.

[marcodelapierre]

Makes total sense, thanks for sharing this!
I shall read the whole story above asap :-)

[marcodelapierre]

Ok, did some more reading above :)

Short story: fantastic discussion and fantastic ideas, I loved reading it!

I like all of what you guys said, in general: functionalities, UI commands, ideas on the specifications of a view. I am saying "in general" just because I am at end of day and was unable to read all of the details :-)

I have some thoughts I will try and share, focusing for now on the specifications and functionalities of the views.
I will start from a concept above of an hypothetical YAML for a view; if I miss something from your ideas is not because I disagree, just because I forgot; in reading, just focus on the additions I am proposing.

So, for this view YAML:

# your starting point
view:
  name: mpi
  modules:
    - mpi-my-custom-1
    - mpi-my-custom-2
  system_modules:
    - mpi
    - system_module

# some more thoughts

# a form of setting to specify whether the view has long/short naming convention or whatsoever
tree_naming:    # values: long (registry/repo/tool/version) or short (tool/version) or something like that

# here some seed of a concept I have in mind, two types of views, not sure about how to call them
# type1: symlink, the  one we've talked about so far
# type2: full copy of the modules PLUS its own independent setting.yaml, to allow further customisation of the module tree...I think we have quite some few user cases already
view_type:   # value to be discussed

# more ...?

[marcodelapierre]

So, to summarise from my YAML lines above, a couple of thoughts:

1. having a setting in the view YAML to decide whether view naming convention for paths is long or short
2. having two types of view, the 2nd type being one with additional customisation possible, achieved by adding its own settings.yml and hence copying, not symlinking the modules

Why point number 2.? Some cases:

• same cluster, multiple GPU vendors -> need a different GPU setting
• same cluster, multiple MPIs -> needed or not, depending on final implementation of this feature
• same site/filesystem, multiple clusters with different required settings
• (my point 4. in a previous comment, probably a bit too creative) federation of sites with shared containers deployments via frameworks like CernVM-fs

So, my implied idea for point 2. is to try and have one single SHPC installation that tames them all (paraphrasing Vanessa here ;) ).

[marcodelapierre]

How would management of this views with their own settings work?
Mmm not entirely sure right now, but was thinking something.

I agree with your comments above, suggesting that to avoid going crazy we should probably make it so that a view can only be managed by its creator, and stop.
If this is the case, then the view YAMLs and settings could be stored in the user home in a dot dir.
This is already the location where we store the optional, user-specific settings file.

One possible limitation that might be implied by equating view == only creator can admin is that we need to accept SIF files can only be managed by that same creator. In other words, if there's a system wide installation provided by a centre, users of the centre may create their own views and installations, but this will imply duplicating SIFs for them. Or maybe I am wrong, this is not a required constraint. This constraint might not be a big deal any way, not sure right now.

....and here more thoughts. I like the idea some of you stated of starting with a default view with all modules. So, almost wondering....could we reinterpret the default starting context of SHPC as a kind of "base" view? and a base view which can only have "all modules", and possibly only "long" module tree naming? We could even add the system_module functionality to the main settings file..
Not sure whether this thought is going though.

It would be nice if all of this could keep working with the site vs user settings.yml, which I think is the case right now. What I like about the options for the user settings is that it enables one single SHPC installation to be used by multiple users, that can customise EVERYTHING, including, crucially the SIF file storage path.

[marcodelapierre]

One more thought on the location of the view module tree.

Scenario 1.
Let's start from module_base in the settings.yml. If we think of the default, non view, installation as the base view, then its modules might go in a subdirectory base of the module_base.
Then, each named view will go in a subdirectory of module_base with same name as the view.

Scenario 2.
If we want more flexibility, we could instead have a view_base, or module_base, setting in the view YAML, that provides full customisation of its path.

[marcodelapierre]

And last one for today...

About my idea above of two kinds of view, 1. typical symlinks , 2. new module tree with its own settings.

In this case, a hierarchy of which settings are used first is needed, eg

1. view settings if they exist, or
2. user settings if they exist, or
3. site settings otherwise

With this idea of the view setting, we would also need to decide whether to allow the customisation of the container_base. Maybe best to force it to be the same as the underlying user/site setting, but not sure.

[marcodelapierre]

I hope this can contribute to your great conversation and work, feel free to pick/edit/ignore, and to ping me! :-)

[vsoch]

Ready to go! I think next I'll start from this branch and put in some early ideas for a PR. I don't want to merge into main (and thus release) because I expect the view work will tweak some of this a bit, and better to put one new feature in at once instead of a new feature that immediately gets deprecated for another one.

As always, great work @muffato !

[vsoch]

@marcodelapierre don't worry I've kept a link to the discussion here - will try to write it all up in one place to stare at before working on views, should be next!

[marcodelapierre]

No worries @vsoch , thank you!

Thinking back at what I shared yesterday, I think a chunk of it is probably pushing it too much, in terms of adding complexity without big benefit. I am thinking of the view-with-specific-settings idea. Glad to hear what you guys think, but otherwise feel free to skip.

The ideas which I most invite you and @muffato to think about are the couple of extra specifications for the views:

• naming convention ("long" vs "short", as per original symlink tree)
• view location (where and how to specify it - this can be useful for the cases you were mentioning, i.e. @georgiastuart 's MPI, and Nextflow as well)

One more thing I thought about this morning is on the GPU case. IF we want to handle the case of a AMD+Nvidia container, this may complicate things: the symlinks would not do, because we need to provide the right Singularity flag inside the aliases/wrappers definitions. Or...?

And one more morning thought on the UI, it would be good to have a view command that just produces the root path of the view, eg

shpc view get-root     # don't like `get-root` but just to give the idea

which could eventually be used by module use:

module use $( shpc view get-root )

Finally, I think it might help the conversation to re-state what we're trying to achieve with the views, in terms of functionalities and user cases. It might help to decide what features to include, and some aspects of their mechanics.

I hope this can help your very interesting brainstorming on the views feature.
As usual, feel free to pick/edit/disregard, and ping me, as you think it can best contribute to the project.

Thank you!

[vsoch]

Thanks @marcodelapierre your feedback is really helpful! This is a holiday (3 day) weekend so I anticipate getting in our first PR for views soon - I know how I'm going to restructure shpc to prepare and it's exciting :)

[vsoch]

Discussion here moved / continued in #545.