Several updates to config and environment settings (#403)

* several updates to config and environment settings
* temporary fix of moving registry entiry to end (so no comments are eaten) and printing warning
* ensuring that multiple podman images are deleted if no tag is provided
* fix for ruamel deleting comments - we need to preserve its type for the list
* fixing typo for shpc uninstall
* try doing uninstall after test, and looking at docker/podman images before and after

Signed-off-by: vsoch <[email protected]>

Author: vsoch

.github/workflows/test.yml

@@ -101,6 +101,7 @@ jobs:
           grep --quiet 'Python 3.9.5' test_output
           rm test_output
           shpc uninstall --force python:3.9.5-alpine 
+
         
       - name: Run python module tests (tcsh)
         shell: tcsh -e {0}

CHANGELOG.md

@@ -14,6 +14,10 @@ and **Merged pull requests**. Critical items to know are:
 The versions coincide with releases on pip. Only major versions will be released as tags on Github.
 
 ## [0.0.x](https://github.scom/singularityhub/singularity-hpc/tree/master) (0.0.x)
+ - Allow environment variables in settings (0.0.29)
+   - User settings file creation and use with shpc config inituser
+   - registry is now a list to support multiple registry locations
+   - config supports add/remove to append/delete from list
  - Add test for docker and podman (0.0.28)
    - namespace as format string for command named renamed to repository
    - shpc test/uninstall should be run for all tests

README.md

@@ -14,6 +14,7 @@ And container technologies:
 
  - [Singularity](https://github.com/sylabs/singularity)
  - [Podman](https://podman.io)
+ - [Docker](https://docker.io)
 
 Coming soon:
 

docs/getting_started/user-guide.rst

@@ -13,16 +13,17 @@ you should do that first.
 Why shpc?
 =========
 
-While the library is currently focused on Singularity containers (hence
-the name) it's created to be modular, meaning that if another container technology
-is wanted, it can be added. The module name would still be appropriate, as
-singularity does imply a single entity that is "one library to rule them all!"
+Singularity Registry HPC is created to be modular, meaning that we support a distinct
+set of container technologies and module systems. The name of the library "Singularity
+Registry HPC" does not refer specifically to the container technology "Singularity,"
+but more generally implies the same spirit -- a single entity that is "one library to rule them all!"
+
 
 What is a registry?
 ===================
 
 A registry consists of a database of local containers configuration files, ``container.yaml``
-files organized in the root of the shpc install in the ``registry`` folder. The namespace
+files organized in the root of the shpc install in one of the ``registry`` folders. The namespace
 is organized by Docker unique resources identifiers. When you install an identifier
 as we saw above, the container binaries and customized module files are added to 
 the ``module_dir`` defined in your settings, which defaults to ``modules`` in the
@@ -99,15 +100,27 @@ Setup
 =====
 
 Setup includes, after installation, editing any configuration values to
-customize your install. The defaults are likely suitable for most.
-For any configuration value that you might set, the following variables
-are available to you:
+customize your install. The configuration file will default to ``shpc/settings.yml``
+in the installed module, however you can create your own user settings file to
+take preference over this one as follows:
+
+.. code-block:: console
+
+    $ shpc config userinit
+
+
+The defaults in either file are likely suitable for most. For any configuration value 
+that you might set, the following variables are available to you:
 
  - ``$install_dir``: the shpc folder
  - ``$root_dir``: the parent directory of shpc (where this README.md is located)
 
 
-A summary table of variables is included below, and then further discussed in detail.
+Additionally, the variables ``module_base``, ``container_base``, and ``registry``
+can be set with environment variables that will be expanded at runtime. You cannot
+use the protected set of substitution variables (``$install_dir`` and ``$install_root``)
+as environment variables, as they will be subbed in by shpc before environment
+variable replacement. A summary table of variables is included below, and then further discussed in detail.
 
 
 .. list-table:: Title
@@ -121,8 +134,8 @@ A summary table of variables is included below, and then further discussed in de
      - Set a default module system. Currently lmod and tcl are supported
      - [lmod, tcl]
    * - registry
-     - The full path to the registry folder (with subfolders with container.yaml recipes)
-     - $root_dir/registry
+     - A list of full paths to one or more registry folders (with subfolders with container.yaml recipes)
+     - [$root_dir/registry]
    * - module_base
      - The install directory for modules. Defaults to the install directory/modules
      - $root_dir/modules
@@ -271,8 +284,10 @@ directory.
 Registry
 --------
 
-The registry folder in the root of the repository, but you can change it to
-be a custom one with the config variable ``registry``
+The registry parameter is a list of one or more registry locations (filesystem
+directories) where shpc will search for ``container.yaml`` files. The default
+registry shipped with shpc is the folder in the root of the repository, but 
+you can add or remove entries via the config variable ``registry``
 
 
 .. code-block:: console
@@ -409,6 +424,8 @@ file directly, or you can use ``shpc config``, which will accept:
 
  - set to set a parameter and value
  - get to get a parameter by name
+ - add to add a value to a parameter that is a list (e.g., registry)
+ - remove to remove a value from a parameter that is a list
 
 The following example shows changing the default module_base path from the install directory modules folder.
 
@@ -428,6 +445,14 @@ And then to get values:
     $ shpc config get module_base
 
 
+And to add and remove a value to a list:
+
+.. code-block::console
+
+    $ shpc config add registry:/tmp/registry
+    $ shpc config remove registry:/tmp/registry
+
+
 You can also open the config in the editor defined in settings at ``config_editor``
 
 .. code-block:: console

docs/index.rst

@@ -20,11 +20,11 @@ And container technologies:
 
  - `Singularity <https://github.com/sylabs/singularity>`_
  - `Podman <https://podman.io>`_
+ - `Docker <https://www.docker.com/>`_
 
 And coming soon:
 
  - `Shifter <https://github.com/NERSC/shifter>`_
- - `Docker <https://www.docker.com/>`_
  - `Sarus <https://github.com/eth-cscs/sarus>`_
 
 

shpc/client/__init__.py

@@ -43,7 +43,7 @@ def get_parser():
     parser.add_argument(
         "--version",
         dest="version",
-        help="suppress additional output.",
+        help="show software version.",
         default=False,
         action="store_true",
     )
@@ -142,10 +142,26 @@ def get_parser():
         help="update configuration settings. Use set or get to see or set information.",
         formatter_class=argparse.RawTextHelpFormatter,
     )
+
+    config.add_argument(
+        "--central",
+        "-c",
+        dest="central",
+        help="make edits to the central config file.",
+        default=False,
+        action="store_true",
+    )
+
     config.add_argument(
         "params",
         nargs="*",
-        help="Set or get a config value, or edit the config.\nshpc config set key:value\nshpc config get key\nshpc edit",
+        help="""Set or get a config value, edit the config, add or remove a list variable, or create a user-specific config.
+shpc config set key:value
+shpc config get key
+shpc edit
+shpc config inituser
+shpc config add registry:/tmp/registry
+shpc config remove registry:/tmp/registry""",
         type=str,
     )
     # Generate markdown docs for a container registry entry

shpc/client/config.py

@@ -2,6 +2,7 @@
 __copyright__ = "Copyright 2021, Vanessa Sochat"
 __license__ = "MPL 2.0"
 
+import shpc.defaults as defaults
 from shpc.logger import logger
 import sys
 
@@ -18,15 +19,21 @@ def main(args, parser, extra, subparser):
     # The first "param" is either set of get
     command = args.params.pop(0)
 
+    # If the user wants the central config file
+    if args.central:
+        args.settings_file = defaults.default_settings_file
+
     validate = True if not command == "edit" else False
     cli = get_client(
         quiet=args.quiet, settings_file=args.settings_file, validate=validate
     )
 
     # For each new setting, update and save!
+    if command == "inituser":
+        return cli.settings.inituser()
     if command == "edit":
         return cli.settings.edit()
-    elif command == "set":
+    elif command in ["set", "add", "remove"]:
         for param in args.params:
             if ":" not in param:
                 logger.warning(
@@ -35,8 +42,15 @@ def main(args, parser, extra, subparser):
                 )
                 continue
             key, value = param.split(":", 1)
-            cli.settings.set(key, value)
-            logger.info("Updated %s to be %s" % (key, value))
+            if command == "set":
+                cli.settings.set(key, value)
+                logger.info("Updated %s to be %s" % (key, value))
+            elif command == "add":
+                cli.settings.add(key, value)
+                logger.info("Added %s to %s" % (key, value))
+            elif command == "remove":
+                cli.settings.remove(key, value)
+                logger.info("Removed %s from %s" % (key, value))
 
         # Save settings
         cli.settings.save()

shpc/defaults.py

@@ -12,6 +12,14 @@
 # The default settings file in the install root
 default_settings_file = os.path.join(reps["$install_dir"], "settings.yml")
 
+# The user settings file can be created to over-ride default
+user_settings_file = os.path.join(
+    os.path.expanduser("~/.singularity-hpc"), "settings.yml"
+)
+
+# variables in settings that allow environment variable expansion
+allowed_envars = ["container_base", "module_base", "registry"]
+
 # The GitHub repository with recipes
 github_url = "https://github.com/singularityhub/singularity-hpc"
 

shpc/main/__init__.py

@@ -7,6 +7,7 @@
 
 import shpc.utils
 from shpc.logger import logger
+import shpc.defaults
 
 
 def get_client(quiet=False, **kwargs):

shpc/main/client.py

@@ -86,6 +86,18 @@ def add_namespace(self, name):
             name = "%s/%s" % (self.settings.namespace.strip("/"), name)
         return name
 
+    def load_registry_config(self, name):
+        """
+        Given an identifier, find the first match in the registry.
+        """
+        for registry, fullpath in self.container.iter_registry():
+            package_dir = os.path.join(registry, name)
+            package_file = os.path.join(package_dir, "container.yaml")
+            if package_file == fullpath:
+                return container.ContainerConfig(package_file)
+
+        logger.exit("%s is not a known recipe in any registry." % name)
+
     def _load_container(self, name, tag=None):
         """
         Given a name and an optional tag to default to, load a package
@@ -94,12 +106,8 @@ def _load_container(self, name, tag=None):
         if ":" in name:
             name, tag = name.split(":", 1)
 
-        # The recipe folder must exist in the registry
-        package_dir = os.path.join(self.settings.registry, name)
-        package_file = os.path.join(package_dir, "container.yaml")
-        config = container.ContainerConfig(package_file)
-
         # If the user provides a tag, set it
+        config = self.load_registry_config(name)
         config.set_tag(tag)
         return config
 
@@ -169,6 +177,7 @@ def cleanup(tmpdir):
         if stage:
             logger.info(tmpdir)
         else:
+            self.uninstall(module_name, force=True)
             cleanup(tmpdir)
 
     def check(self, module_name):
@@ -205,12 +214,10 @@ def show(self, name, names_only=False, out=None, filter_string=None):
             out = out or sys.stdout
 
             # List the known registry modules
-            for fullpath in utils.recursive_find(self.settings.registry):
+            for registry, fullpath in self.container.iter_registry():
                 if fullpath.endswith("container.yaml"):
                     module_name = (
-                        os.path.dirname(fullpath)
-                        .replace(self.settings.registry, "")
-                        .strip(os.sep)
+                        os.path.dirname(fullpath).replace(registry, "").strip(os.sep)
                     )
 
                     # If the user has provided a filter, honor it

shpc/main/container/base.py

@@ -90,24 +90,36 @@ def container_dir(self, name):
             return os.path.join(self.settings.module_base, name)
         return os.path.join(self.settings.container_base, name)
 
+    def iter_registry(self):
+        """
+        Iterate over known registries defined in settings.
+        """
+        for registry in self.settings.registry:
+            for filename in shpc.utils.recursive_find(registry):
+                yield registry, filename
+
     def guess_tag(self, module_name, allow_fail=False):
         """
         If a user asks for a name without a tag, try to figure it out.
         """
         if ":" in module_name:
             return module_name
-        tags = os.listdir(os.path.join(self.settings.module_base, module_name))
+        tags = self.installed_tags(module_name)
         if not tags and allow_fail:
             logger.exit("%s does not have any tags installed." % module_name)
-        elif (tags or len(tags) > 1) and allow_fail:
+        elif tags and len(tags) == 1:
+            return "%s:%s" % (module_name, tags[0])
+        elif tags and len(tags) > 1 and allow_fail:
             return
-        elif len(tags) > 1:
-            logger.exit(
-                "Multiple tags found for %s: %s." % (module_name, ", ".join(tags))
-            )
-        else:
-            module_name = "%s:%s" % (module_name, tags[0])
-        return module_name
+
+        # Length of tags is > 1
+        logger.exit("Multiple tags found for %s: %s." % (module_name, ", ".join(tags)))
+
+    def installed_tags(self, module_name):
+        """
+        Get a list of installed tags.
+        """
+        return os.listdir(os.path.join(self.settings.module_base, module_name))
 
     def get_environment_file(self, module_name):
         """