modular sqfs #224

simonpintarelli · 2025-06-05T16:56:10Z

remove compiler bootstrap (use a base uenv with a compiler or system gcc12)
base images (for example /user-environment/gcc/12.3.0) are regular stackinator sqfs images located in a subdirectory of the root image.
new sandbox
1. squashfs-mount-wrapper.sh
- mount base sqfs in $BUILDROOT/tmp/mounts/ via squashfs-mount
1. bwrap-mutable-root.sh
- make a mutable root (as squashfs-run does), this is required for building base images where mount-points don't exist (for example /user-environment/gcc)
- bind mount base images from $BUILDROOT/tmp/mounts/ to their final destination.
update stack-debug.sh to mount the base uenvs
a json/schema, example file is used to describe a modular uenv (used by uenv2) and during the build process
uses spack@develop
compilers.toolchain has been removed from environments.yaml. The json description can be used to generate the same constraints in spack.yaml
base uenvs expose their rootspecs via env/default/packages.yaml. The dependent uenvs then include these packages.yaml in spack.include. The python script used to generate packages.yaml from spack.lock is in gen_packages_yaml.py
packages from environments.yaml is ignored (it was in Makefile.compilers). Why not use packages (incl variants like in regular spack.yaml? And then use it as override for auto generated package section.
unit tests need to be updated (removed compilers.yaml)

The corresponding cluster-config is in https://github.com/eth-cscs/alps-cluster-config/tree/feat/modular-uenv. It contains the repo changes for spack 1.0 and gcc12 configured in packages.yaml for daint/eiger (overlaps/conflicts with eth-cscs/alps-cluster-config#35)

Prototype for uenv start/run which can mount modular sqfs from the json description: eth-cscs/uenv2#86

Example recipe for cuda/gcc and osu-micro-benchmarks: https://github.com/simonpintarelli/daint-modular-uenv

remove bootstrap from schema remove compilers.yaml add squashfs-mount for base-uenvs cleanup cleanup change order typo typo typo skip compiler target convert to string fix path skip base uenvs in post-install loop.last tab look for squashfs in global tree always install squashfs add dependency do not generate {{env}}/compilers.yaml skip {{env}}/compilers.yaml in envvars.py skip -C /user-environment/config do not find compilers cleanup deps remove compilers.yaml install squashfs to bind mounted /tmp do not use filesystem locks remove dependency remove -d install squashfs in an env fix fix cleanup install -D the mount point doesn't need to exist base-uenv.json is optional wip fix squashfs-mount-wrapper allow to use a non-existent directory for STORE comment add padded_length for cache add missing file undo use padded_lenght in env do not use self.mount for mirror path use same sandbox in stack-debug.sh Revert "do not use self.mount for mirror path" This reverts commit f187450. ignore store/mountpath for build cache fix stack-debug.sh fix bwrap-store Revert "ignore store/mountpath for build cache" This reverts commit a13fc2b. remove padded_length remove stdout add generation of packages.yaml for externals fix chmod wip avoid spack --color fix structure set buildable false fix gen packages.yaml update do not use -all-root remove unused compiler target v2.0 repo format path must contain spack_repo repo -> spack_repo repo path must include namespace repo -> spack_repo update base-uenv.json schema compilers.yaml -> packages.yaml fix sandbox fix path, version doesn't belong here add missing version to packages.yaml set extra_attributes for compilers fix fix split debug Revert "debug" This reverts commit 4172a0a. fix find command cleanup update the schema remove compilers.yaml and the bits related to it from recipe.py remove more occurences of compilers.yaml avoid empty packages.all in environments.spack.yaml update for new base_uenv.json schema typo in schema base_uenv schema, default null for optional entries fix schema wip schema fix schema typo fix environments.spack.yaml typo fix environment.spack.yaml include exports from the gpu sub uenv fix schema change title remove compiler from env schema remove compiler from _internal_utils

msimberg

A few cleanup comments.

I did not look at the modular squashfs parts, only briefly at the changes to remove compilers.yaml and compiler bootstrapping, and that looks fine to me.

msimberg · 2025-06-17T14:26:02Z

stackinator/recipe.py

+        # for name, config in environments.items():
+        #     compilers = config["compiler"]
+        #     if len(compilers) == 1:
+        #         config["toolchain_constraints"] = []
+        #         continue
+        #     requires = [f"%{compilers[0]['spec']}"]
+        #     for spec in config["specs"]:
+        #         if "%" in spec:
+        #             requires.append(spec)
+
+        #     config["toolchain_constraints"] = requires


This I forgot to mention yesterday, the constraints for using nvhpc and gcc are missing at the moment. They used to be generated from the compiler section in environments.yaml

Is that the packages:all:compiler:gcc constraints (modulo me misremembering which exact config section it is; i.e. making sure environments are concretized only using one compiler)? If yes, I think it's anyway currently ignored on spack develop and we can maybe add that back separately from this PR? I think it's #237.

I was referring to this:
https://github.com/eth-cscs/stackinator/blob/master/stackinator/templates/environments.spack.yaml#L24
In packages.all.require we used one_of: [%gcc, pkgA%nvhpc, pkgB%nvhpc] to ensure that packages were built using gcc, unless explicitly requiring nvhpc.

Ah, I see. I was referring to

stackinator/stackinator/templates/environments.spack.yaml

Line 20 in e440d50

compiler: [{% for c in config.compiler %}{{ separator() }}'{{ c.spec }}'{% endfor %}]

, so close but not quite the same. I see you removed that anyway which is good.

To keep things simple for this PR, can we live without the constraint for now? Or how complicated do you think it would be to add back the constraints?

I think I've made a mess there. I can't recall why I commented the lines. environments.compiler perhaps should be kept as it was. Implicitly generating the compiler preference gcc/nvhpc based on which compilers are present in the modular uenv isn't a good idea, better to make this explicit.

stackinator/templates/Makefile

stackinator/templates/Makefile.environments

stackinator/templates/Makefile.generate-config

msimberg · 2025-06-17T14:35:09Z

Summarizing the quick chat we had with @simonpintarelli, @albestro, and myself today:

Assuming that this PR keeps non-modular uenvs/recipes working exactly as before we try to merge this PR first for support for spack 1.0. The modular uenv stuff can be considered "experimental" and tested further on main. After merging this:

Merge any remaining required changes from adapt for compiler as dependencies in spack #215
Merge Add spack:packages to recipe schema to allow specifying a fixed commit of the builtin packages #236
Merge Add Openmpi using new libfabric to stackinator options #210

stackinator/templates/compilers.gcc.spack.yaml

albestro · 2025-06-20T09:37:37Z

stackinator/etc/squashfs-mount-wrapper.sh

+	  mount_point=${elem#*:}
+    sqfs=${elem%%:*}
+    tmp_mount_point="${BUILD_ROOT}/tmp/mounts/${mount_point}"
+	  mkdir -p ${tmp_mount_point}
+    build_root_mounts="${build_root_mounts} ${sqfs}:${tmp_mount_point}"


minor: formatting to be fixed?

albestro · 2025-06-20T09:42:45Z

stackinator/schema/compilers.json

    "type": "object",
    "additionalProperties": false,
-    "required": ["bootstrap", "gcc"],
+    "required": ["gcc"],


does it mean that we can remove defs:properties:bootstrap as well?

albestro · 2025-06-20T10:09:39Z

stackinator/templates/Makefile.generate-config


-# These will be the prefixes of the GCCs, LLVMs and NVHPCs in the respective environments.
-ALL_COMPILER_PREFIXES ={% for compiler in all_compilers %} $$($(SPACK_HELPER) -e ../compilers/{{ compiler }} find --format='{prefix}' gcc llvm nvhpc){% endfor %}
+# COMPILER_PREFIXES ={% for compiler in release_compilers %} $$($(SPACK_HELPER) -e ../compilers/{{ compiler }} find --format='{prefix}' gcc llvm nvhpc){% endfor %}


Suggested change

# COMPILER_PREFIXES ={% for compiler in release_compilers %} $$($(SPACK_HELPER) -e ../compilers/{{ compiler }} find --format='{prefix}' gcc llvm nvhpc){% endfor %}

?

albestro · 2025-06-20T12:30:06Z

stackinator/templates/Make.user

+SANDBOX := $(SQFS_MOUNT) -- $(BUILD_ROOT)/bwrap-mutable-root.sh $\
+	--tmpfs ~ $\
+	--bind $(BUILD_ROOT)/tmp /tmp $\
+  --bind $(BUILD_ROOT)/store $(STORE) $\


Suggested change

--bind $(BUILD_ROOT)/store $(STORE) $\

--bind $(BUILD_ROOT)/store $(STORE) $\

I'm missing why this differs from SANDBOX_NO_BASE for what concerns store. Can we use bwrap-store here too?

albestro · 2025-06-20T12:30:41Z

stackinator/templates/Make.user

 	--tmpfs ~ $\
 	--bind $(BUILD_ROOT)/tmp /tmp $\
-	--bind $(BUILD_ROOT)/store $(STORE)
+  -- $(BUILD_ROOT)/bwrap-store.sh


Suggested change

-- $(BUILD_ROOT)/bwrap-store.sh

-- $(BUILD_ROOT)/bwrap-store.sh

albestro · 2025-06-20T12:45:04Z

stackinator/schema/base-uenv.json

+                },
+                "image": {
+                    "$ref": "#/defs/image",
+                    "description": "TOOD"


mispelled TODO that might escape the search for TODOs

simonpintarelli added 2 commits February 24, 2025 22:34

typo

97716ec

remove duplicate target

eff1127

simonpintarelli force-pushed the feat/modular-sqfs branch from 8a44d89 to b99664b Compare June 5, 2025 17:09

msimberg moved this to In Progress in stackinator recipe v2 work Jun 5, 2025

msimberg added this to stackinator recipe v2 work Jun 5, 2025

simonpintarelli marked this pull request as draft June 5, 2025 17:22

remove unused Makefiler.compilers

a51de61

msimberg mentioned this pull request Jun 10, 2025

prgenv-gnu/25.6 eth-cscs/alps-uenv#224

Merged

simonpintarelli force-pushed the feat/modular-sqfs branch from 8490715 to edbd6d1 Compare June 17, 2025 14:01

simonpintarelli added 2 commits June 17, 2025 16:02

Merge remote-tracking branch 'origin/main' into feat/modular-sqfs

edbd6d1

format

0f932c7

msimberg requested review from albestro, bcumming and msimberg June 17, 2025 14:19

msimberg requested changes Jun 17, 2025

View reviewed changes

simonpintarelli added 2 commits June 18, 2025 10:14

cleanup

bc8bf63

cleanup

9f9c85f

bcumming reviewed Jun 18, 2025

View reviewed changes

stackinator/templates/compilers.gcc.spack.yaml Outdated Show resolved Hide resolved

delete unused templates

146b453

albestro reviewed Jun 20, 2025

View reviewed changes

msimberg removed this from stackinator recipe v2 work Jun 24, 2025

	--bind $(BUILD_ROOT)/store $(STORE) $\
	--bind $(BUILD_ROOT)/store $(STORE) $\

	-- $(BUILD_ROOT)/bwrap-store.sh
	-- $(BUILD_ROOT)/bwrap-store.sh

modular sqfs #224

Are you sure you want to change the base?

modular sqfs #224

Uh oh!

Conversation

simonpintarelli commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

msimberg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

msimberg commented Jun 17, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

simonpintarelli commented Jun 5, 2025 •

edited

Loading