diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 9df786c44..24bb424ed 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -101,6 +101,7 @@ jobs: grep --quiet 'Python 3.9.5' test_output rm test_output shpc uninstall --force python:3.9.5-alpine + - name: Run python module tests (tcsh) shell: tcsh -e {0} diff --git a/CHANGELOG.md b/CHANGELOG.md index c7a33d934..960a1ac68 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,10 @@ and **Merged pull requests**. Critical items to know are: The versions coincide with releases on pip. Only major versions will be released as tags on Github. ## [0.0.x](https://github.scom/singularityhub/singularity-hpc/tree/master) (0.0.x) + - Allow environment variables in settings (0.0.29) + - User settings file creation and use with shpc config inituser + - registry is now a list to support multiple registry locations + - config supports add/remove to append/delete from list - Add test for docker and podman (0.0.28) - namespace as format string for command named renamed to repository - shpc test/uninstall should be run for all tests diff --git a/README.md b/README.md index 0e55d88f2..4f489bcf9 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,7 @@ And container technologies: - [Singularity](https://github.com/sylabs/singularity) - [Podman](https://podman.io) + - [Docker](https://docker.io) Coming soon: diff --git a/docs/getting_started/user-guide.rst b/docs/getting_started/user-guide.rst index 48398a7f3..664342d74 100644 --- a/docs/getting_started/user-guide.rst +++ b/docs/getting_started/user-guide.rst @@ -13,16 +13,17 @@ you should do that first. Why shpc? ========= -While the library is currently focused on Singularity containers (hence -the name) it's created to be modular, meaning that if another container technology -is wanted, it can be added. The module name would still be appropriate, as -singularity does imply a single entity that is "one library to rule them all!" +Singularity Registry HPC is created to be modular, meaning that we support a distinct +set of container technologies and module systems. The name of the library "Singularity +Registry HPC" does not refer specifically to the container technology "Singularity," +but more generally implies the same spirit -- a single entity that is "one library to rule them all!" + What is a registry? =================== A registry consists of a database of local containers configuration files, ``container.yaml`` -files organized in the root of the shpc install in the ``registry`` folder. The namespace +files organized in the root of the shpc install in one of the ``registry`` folders. The namespace is organized by Docker unique resources identifiers. When you install an identifier as we saw above, the container binaries and customized module files are added to the ``module_dir`` defined in your settings, which defaults to ``modules`` in the @@ -99,15 +100,27 @@ Setup ===== Setup includes, after installation, editing any configuration values to -customize your install. The defaults are likely suitable for most. -For any configuration value that you might set, the following variables -are available to you: +customize your install. The configuration file will default to ``shpc/settings.yml`` +in the installed module, however you can create your own user settings file to +take preference over this one as follows: + +.. code-block:: console + + $ shpc config userinit + + +The defaults in either file are likely suitable for most. For any configuration value +that you might set, the following variables are available to you: - ``$install_dir``: the shpc folder - ``$root_dir``: the parent directory of shpc (where this README.md is located) -A summary table of variables is included below, and then further discussed in detail. +Additionally, the variables ``module_base``, ``container_base``, and ``registry`` +can be set with environment variables that will be expanded at runtime. You cannot +use the protected set of substitution variables (``$install_dir`` and ``$install_root``) +as environment variables, as they will be subbed in by shpc before environment +variable replacement. A summary table of variables is included below, and then further discussed in detail. .. list-table:: Title @@ -121,8 +134,8 @@ A summary table of variables is included below, and then further discussed in de - Set a default module system. Currently lmod and tcl are supported - [lmod, tcl] * - registry - - The full path to the registry folder (with subfolders with container.yaml recipes) - - $root_dir/registry + - A list of full paths to one or more registry folders (with subfolders with container.yaml recipes) + - [$root_dir/registry] * - module_base - The install directory for modules. Defaults to the install directory/modules - $root_dir/modules @@ -271,8 +284,10 @@ directory. Registry -------- -The registry folder in the root of the repository, but you can change it to -be a custom one with the config variable ``registry`` +The registry parameter is a list of one or more registry locations (filesystem +directories) where shpc will search for ``container.yaml`` files. The default +registry shipped with shpc is the folder in the root of the repository, but +you can add or remove entries via the config variable ``registry`` .. code-block:: console @@ -409,6 +424,8 @@ file directly, or you can use ``shpc config``, which will accept: - set to set a parameter and value - get to get a parameter by name + - add to add a value to a parameter that is a list (e.g., registry) + - remove to remove a value from a parameter that is a list The following example shows changing the default module_base path from the install directory modules folder. @@ -428,6 +445,14 @@ And then to get values: $ shpc config get module_base +And to add and remove a value to a list: + +.. code-block::console + + $ shpc config add registry:/tmp/registry + $ shpc config remove registry:/tmp/registry + + You can also open the config in the editor defined in settings at ``config_editor`` .. code-block:: console diff --git a/docs/index.rst b/docs/index.rst index f73aa07fb..221ba7814 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -20,11 +20,11 @@ And container technologies: - `Singularity `_ - `Podman `_ + - `Docker `_ And coming soon: - `Shifter `_ - - `Docker `_ - `Sarus `_ diff --git a/shpc/client/__init__.py b/shpc/client/__init__.py index af9bec7d1..8f265fd25 100644 --- a/shpc/client/__init__.py +++ b/shpc/client/__init__.py @@ -43,7 +43,7 @@ def get_parser(): parser.add_argument( "--version", dest="version", - help="suppress additional output.", + help="show software version.", default=False, action="store_true", ) @@ -142,10 +142,26 @@ def get_parser(): help="update configuration settings. Use set or get to see or set information.", formatter_class=argparse.RawTextHelpFormatter, ) + + config.add_argument( + "--central", + "-c", + dest="central", + help="make edits to the central config file.", + default=False, + action="store_true", + ) + config.add_argument( "params", nargs="*", - help="Set or get a config value, or edit the config.\nshpc config set key:value\nshpc config get key\nshpc edit", + help="""Set or get a config value, edit the config, add or remove a list variable, or create a user-specific config. +shpc config set key:value +shpc config get key +shpc edit +shpc config inituser +shpc config add registry:/tmp/registry +shpc config remove registry:/tmp/registry""", type=str, ) # Generate markdown docs for a container registry entry diff --git a/shpc/client/config.py b/shpc/client/config.py index 164ee2295..bef91fe3b 100644 --- a/shpc/client/config.py +++ b/shpc/client/config.py @@ -2,6 +2,7 @@ __copyright__ = "Copyright 2021, Vanessa Sochat" __license__ = "MPL 2.0" +import shpc.defaults as defaults from shpc.logger import logger import sys @@ -18,15 +19,21 @@ def main(args, parser, extra, subparser): # The first "param" is either set of get command = args.params.pop(0) + # If the user wants the central config file + if args.central: + args.settings_file = defaults.default_settings_file + validate = True if not command == "edit" else False cli = get_client( quiet=args.quiet, settings_file=args.settings_file, validate=validate ) # For each new setting, update and save! + if command == "inituser": + return cli.settings.inituser() if command == "edit": return cli.settings.edit() - elif command == "set": + elif command in ["set", "add", "remove"]: for param in args.params: if ":" not in param: logger.warning( @@ -35,8 +42,15 @@ def main(args, parser, extra, subparser): ) continue key, value = param.split(":", 1) - cli.settings.set(key, value) - logger.info("Updated %s to be %s" % (key, value)) + if command == "set": + cli.settings.set(key, value) + logger.info("Updated %s to be %s" % (key, value)) + elif command == "add": + cli.settings.add(key, value) + logger.info("Added %s to %s" % (key, value)) + elif command == "remove": + cli.settings.remove(key, value) + logger.info("Removed %s from %s" % (key, value)) # Save settings cli.settings.save() diff --git a/shpc/defaults.py b/shpc/defaults.py index 28ea5da5e..a7fada0d5 100644 --- a/shpc/defaults.py +++ b/shpc/defaults.py @@ -12,6 +12,14 @@ # The default settings file in the install root default_settings_file = os.path.join(reps["$install_dir"], "settings.yml") +# The user settings file can be created to over-ride default +user_settings_file = os.path.join( + os.path.expanduser("~/.singularity-hpc"), "settings.yml" +) + +# variables in settings that allow environment variable expansion +allowed_envars = ["container_base", "module_base", "registry"] + # The GitHub repository with recipes github_url = "https://github.com/singularityhub/singularity-hpc" diff --git a/shpc/main/__init__.py b/shpc/main/__init__.py index 12c60c861..7315defc4 100644 --- a/shpc/main/__init__.py +++ b/shpc/main/__init__.py @@ -7,6 +7,7 @@ import shpc.utils from shpc.logger import logger +import shpc.defaults def get_client(quiet=False, **kwargs): diff --git a/shpc/main/client.py b/shpc/main/client.py index 86f4972f1..0cf5a125f 100644 --- a/shpc/main/client.py +++ b/shpc/main/client.py @@ -86,6 +86,18 @@ def add_namespace(self, name): name = "%s/%s" % (self.settings.namespace.strip("/"), name) return name + def load_registry_config(self, name): + """ + Given an identifier, find the first match in the registry. + """ + for registry, fullpath in self.container.iter_registry(): + package_dir = os.path.join(registry, name) + package_file = os.path.join(package_dir, "container.yaml") + if package_file == fullpath: + return container.ContainerConfig(package_file) + + logger.exit("%s is not a known recipe in any registry." % name) + def _load_container(self, name, tag=None): """ Given a name and an optional tag to default to, load a package @@ -94,12 +106,8 @@ def _load_container(self, name, tag=None): if ":" in name: name, tag = name.split(":", 1) - # The recipe folder must exist in the registry - package_dir = os.path.join(self.settings.registry, name) - package_file = os.path.join(package_dir, "container.yaml") - config = container.ContainerConfig(package_file) - # If the user provides a tag, set it + config = self.load_registry_config(name) config.set_tag(tag) return config @@ -169,6 +177,7 @@ def cleanup(tmpdir): if stage: logger.info(tmpdir) else: + self.uninstall(module_name, force=True) cleanup(tmpdir) def check(self, module_name): @@ -205,12 +214,10 @@ def show(self, name, names_only=False, out=None, filter_string=None): out = out or sys.stdout # List the known registry modules - for fullpath in utils.recursive_find(self.settings.registry): + for registry, fullpath in self.container.iter_registry(): if fullpath.endswith("container.yaml"): module_name = ( - os.path.dirname(fullpath) - .replace(self.settings.registry, "") - .strip(os.sep) + os.path.dirname(fullpath).replace(registry, "").strip(os.sep) ) # If the user has provided a filter, honor it diff --git a/shpc/main/container/base.py b/shpc/main/container/base.py index 734ca4bbc..79ff963cd 100644 --- a/shpc/main/container/base.py +++ b/shpc/main/container/base.py @@ -90,24 +90,36 @@ def container_dir(self, name): return os.path.join(self.settings.module_base, name) return os.path.join(self.settings.container_base, name) + def iter_registry(self): + """ + Iterate over known registries defined in settings. + """ + for registry in self.settings.registry: + for filename in shpc.utils.recursive_find(registry): + yield registry, filename + def guess_tag(self, module_name, allow_fail=False): """ If a user asks for a name without a tag, try to figure it out. """ if ":" in module_name: return module_name - tags = os.listdir(os.path.join(self.settings.module_base, module_name)) + tags = self.installed_tags(module_name) if not tags and allow_fail: logger.exit("%s does not have any tags installed." % module_name) - elif (tags or len(tags) > 1) and allow_fail: + elif tags and len(tags) == 1: + return "%s:%s" % (module_name, tags[0]) + elif tags and len(tags) > 1 and allow_fail: return - elif len(tags) > 1: - logger.exit( - "Multiple tags found for %s: %s." % (module_name, ", ".join(tags)) - ) - else: - module_name = "%s:%s" % (module_name, tags[0]) - return module_name + + # Length of tags is > 1 + logger.exit("Multiple tags found for %s: %s." % (module_name, ", ".join(tags))) + + def installed_tags(self, module_name): + """ + Get a list of installed tags. + """ + return os.listdir(os.path.join(self.settings.module_base, module_name)) def get_environment_file(self, module_name): """ diff --git a/shpc/main/container/docker.py b/shpc/main/container/docker.py index b65da498c..5d8040fc1 100644 --- a/shpc/main/container/docker.py +++ b/shpc/main/container/docker.py @@ -60,9 +60,11 @@ def registry_pull(self, module_dir, container_dir, config, tag): if pull_type != "docker": logger.exit("%s only supports Docker (oci registry) pulls." % self.command) - # Podman doesn't keep a record of digest->tag, so we use tag - uri = "%s:%s" % (self.add_registry(config.docker), tag.name) - return self.pull(uri) + tag_uri = "%s:%s" % (self.add_registry(config.docker), tag.name) + tag_digest = "%s@%s" % (self.add_registry(config.docker), tag.digest) + self.pull(tag_digest) + # Podman doesn't keep a record of digest->tag, so we tag after + return self.tag(tag_digest, tag_uri) def pull(self, uri): """ @@ -73,6 +75,15 @@ def pull(self, uri): logger.exit("There was an issue pulling %s" % uri) return uri + def tag(self, image, tag_as): + """ + Given a container URI, tag as something else. + """ + res = shpc.utils.run_command([self.command, "tag", image, tag_as]) + if res["return_code"] != 0: + logger.exit("There was an issue tagging %s as %s" % (image, tag_as)) + return tag_as + def inspect(self, image): """ Inspect an image @@ -116,9 +127,19 @@ def delete(self, image): """ Delete a container when a module is deleted. """ - image = self.get(image) - if self.exists(image): - shpc.utils.run_command([self.command, "rmi", image]) + container = self.get(image) + + # If we can't get a specific image, the user wants to delete all tags + # and we have more than one tag! + if not container: + tags = self.installed_tags(image) + containers = ["%s:%s" % (image, tag) for tag in tags] + else: + containers = [container] + + for container in containers: + if self.exists(container): + shpc.utils.run_command([self.command, "rmi", "--force", container]) def check(self, module_name, config): """ diff --git a/shpc/main/container/singularity.py b/shpc/main/container/singularity.py index e69c632d6..d374623d0 100644 --- a/shpc/main/container/singularity.py +++ b/shpc/main/container/singularity.py @@ -71,21 +71,20 @@ def add(self, sif, module_name, modulefile, template, **kwargs): """ Manually add a registry container. """ - registry_dir = self.settings.registry - # Ensure the container exists sif = os.path.abspath(sif) if not os.path.exists(sif): logger.exit("%s does not exist." % sif) # First ensure that we aren't using a known namespace - for subfolder in module_name.split("/"): - registry_dir = os.path.join(registry_dir, subfolder) - if os.path.exists(registry_dir): - logger.exit( - "%s is a known registry namespace, choose another for a custom addition." - % subfolder - ) + for registry_dir, _ in self.iter_registry(): + for subfolder in module_name.split("/"): + registry_dir = os.path.join(registry_dir, subfolder) + if os.path.exists(registry_dir): + logger.exit( + "%s is a known registry namespace, choose another for a custom addition." + % subfolder + ) # The user can have a different container directory defined container_dir = self.container_dir(module_name) diff --git a/shpc/main/modules/templates/docker.lua b/shpc/main/modules/templates/docker.lua index db6551417..5280ef17a 100644 --- a/shpc/main/modules/templates/docker.lua +++ b/shpc/main/modules/templates/docker.lua @@ -16,11 +16,11 @@ Container: Commands include: - {|module_name|}-run: - {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} -v ${PWD} -w ${PWD} {% endif %} "$@" + {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}-v ${PWD} -w ${PWD} "$@" - {|module_name|}-shell: - {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} --entrypoint {{ shell }} -v ${PWD} -w ${PWD} {% endif %} + {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}--entrypoint {{ shell }} -v ${PWD} -w ${PWD} - {|module_name|}-exec: - {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint "" {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v ${PWD} -w ${PWD} "$@" + {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint "" {% if envfile %}--env-file {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v ${PWD} -w ${PWD} "$@" - {|module_name|}-inspect: {{ command }} inspect diff --git a/shpc/main/modules/templates/docker.tcl b/shpc/main/modules/templates/docker.tcl index a950cfe5e..ae8f44622 100644 --- a/shpc/main/modules/templates/docker.tcl +++ b/shpc/main/modules/templates/docker.tcl @@ -15,15 +15,15 @@ proc ModulesHelp { } { puts stderr " - {{ image }}" puts stderr "Commands include:" puts stderr " - {|module_name|}-run:" - puts stderr " {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v . -w . " + puts stderr " {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v . -w . " puts stderr " - {|module_name|}-shell:" - puts stderr " {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint {{ shell }}{% if envfile %} --env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v . -w . " + puts stderr " {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint {{ shell }} {% if envfile %} --env-file {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v . -w . " puts stderr " - {|module_name|}-exec:" - puts stderr " {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint \"\" {% if envfile %} --env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v . -w . $*" + puts stderr " {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint \"\" {% if envfile %} --env-file {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v . -w . $*" puts stderr " - {|module_name|}-inspect:" puts stderr " {{ command }} inspect " {% if aliases %}{% for alias in aliases %} puts stderr " - {{ alias.name }}:" - puts stderr " {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --entrypoint {{ alias.entrypoint }} {% if envfile %}--envfile {{ module_dir }}/{{ envfile }} {% endif %}{% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if alias.options %}{{ alias.options }} {% endif %} -v . -w . {{ alias.args }}" + puts stderr " {{ command }} run -i{% if tty %}t{% endif %} --rm -u `id -u`:`id -g` --entrypoint {{ alias.entrypoint }} {% if envfile %}--envfile {{ module_dir }}/{{ envfile }} {% endif %}{% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if alias.options %}{{ alias.options }} {% endif %} -v . -w . {{ alias.args }}" {% endfor %}{% endif %} puts stderr "For each of the above, you can export:" @@ -59,7 +59,7 @@ conflict {{ name }} set shellCmd "{{ command }} \${PODMAN_OPTS} run \${PODMAN_COMMAND_OPTS} -u `id -u`:`id -g` --rm -i{% if tty %}t{% endif %} --entrypoint {{ shell }} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v $workdir -w $workdir ${containerPath}" # execCmd needs entrypoint to be the executor -set execCmd "{{ command }} \${PODMAN_OPTS} run -i{% if tty %}t{% endif %} \${PODMAN_COMMAND_OPTS} -u `id -u`:`id -g` --rm {% if envfile %} --env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} -v $workdir -w $workdir {% endif %} " +set execCmd "{{ command }} \${PODMAN_OPTS} run -i{% if tty %}t{% endif %} \${PODMAN_COMMAND_OPTS} -u `id -u`:`id -g` --rm {% if envfile %} --env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }}{% endif %} -v $workdir -w $workdir" set runCmd "{{ command }} \${PODMAN_OPTS} run -i{% if tty %}t{% endif %} \${PODMAN_COMMAND_OPTS} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v $workdir -w $workdir ${containerPath}" set inspectCmd "{{ command }} \${PODMAN_OPTS} inspect ${containerPath}" @@ -68,7 +68,7 @@ set-alias {|module_name|}-shell "${shellCmd}" # exec functions to provide "alias" to module commands {% if aliases %}{% for alias in aliases %} -set-alias {{ alias.name }} "${execCmd} {% if alias.options %} {{ alias.options }} {% endif %} --entrypoint {{ alias.entrypoint }} ${containerPath} {{ alias.command }}" +set-alias {{ alias.name }} "${execCmd} {% if alias.options %} {{ alias.options }} {% endif %} --entrypoint {{ alias.entrypoint }} ${containerPath} {{ alias.args }}" {% endfor %}{% endif %} # A customizable exec function diff --git a/shpc/main/schemas.py b/shpc/main/schemas.py index 8f0cd3e72..aa7b33b50 100644 --- a/shpc/main/schemas.py +++ b/shpc/main/schemas.py @@ -107,9 +107,8 @@ shells = ["/bin/bash", "/bin/sh", "/bin/csh"] # Currently all of these are required - settingsProperties = { - "registry": {"type": "string"}, + "registry": {"type": "array", "items": {"type": "string"}}, "module_base": {"type": "string"}, "container_base": {"type": ["string", "null"]}, "namespace": {"type": ["string", "null"]}, diff --git a/shpc/main/settings.py b/shpc/main/settings.py index cf5fa73af..5a5dda5e5 100644 --- a/shpc/main/settings.py +++ b/shpc/main/settings.py @@ -7,17 +7,26 @@ import shpc.defaults as defaults import shpc.main.schemas import shpc.utils +import shutil try: from ruamel_yaml import YAML + from ruamel_yaml.comments import CommentedSeq except: from ruamel.yaml import YAML + from ruamel.yaml.comments import CommentedSeq from datetime import datetime import jsonschema import os +def OrderedList(*l): + ret = CommentedSeq(l) + ret.fa.set_flow_style() + return ret + + class SettingsBase: def __init__(self): """ @@ -39,6 +48,21 @@ def validate(self): """ jsonschema.validate(instance=self._settings, schema=shpc.main.schemas.settings) + def inituser(self): + """ + Create a user specific config in user's home. + """ + user_home = os.path.dirname(defaults.user_settings_file) + if not os.path.exists(user_home): + os.makedirs(user_home) + if os.path.exists(defaults.user_settings_file): + logger.exit( + "%s already exists! Remove first before re-creating." + % defaults.user_settings_file + ) + shutil.copyfile(self.settings_file, defaults.user_settings_file) + logger.info("Created user settings file %s" % defaults.user_settings_file) + def edit(self): """ Interactively edit a config file. @@ -47,11 +71,24 @@ def edit(self): logger.exit("Settings file not found.") shpc.utils.run_command([self.config_editor, self.settings_file], stream=True) + def get_settings_file(self, settings_file=None): + """ + Get the preferred used settings file. + """ + # Only consider user settings if the file exists! + user_settings = None + if os.path.exists(defaults.user_settings_file): + user_settings = defaults.user_settings_file + + # First preference to command line, then user settings, then default + return settings_file or user_settings or defaults.default_settings_file + def load(self, settings_file=None): """ Load the settings file into the settings object """ - self.settings_file = settings_file or defaults.default_settings_file + # Get the preferred settings flie + self.settings_file = self.get_settings_file(settings_file) # Exit quickly if the settings file does not exist if not os.path.exists(self.settings_file): @@ -59,14 +96,25 @@ def load(self, settings_file=None): # Default to round trip so we can save comments yaml = YAML() + yaml.preserve_quotes = True # Store the original settings for update as we go with open(self.settings_file, "r") as fd: self._settings = yaml.load(fd.read()) def get(self, key, default=None): + """ + Get a settings value, doing appropriate substitution and expansion. + """ value = self._settings.get(key, default) - return self._substitutions(value) + value = self._substitutions(value) + # If we allow environment substitution, do it + if key in defaults.allowed_envars and value: + if isinstance(value, list): + value = [os.path.expandvars(v) for v in value] + else: + value = os.path.expandvars(value) + return value def __getattr__(self, key): """ @@ -74,6 +122,41 @@ def __getattr__(self, key): """ return self.get(key) + def add(self, key, value): + """ + Add a value to a list parameter + """ + # We can only add to lists + current = self._settings.get(key) + if current and not isinstance(current, list): + logger.exit("You cannot only add to a list variable.") + + if value not in current: + # Add to the beginning of the list + current = [value] + current + self._settings[key] = OrderedList() + [self._settings[key].append(x) for x in current] + self.change_validate(key, value) + logger.warning( + "Warning: Check with shpc config edit - ordering of list can change." + ) + + def remove(self, key, value): + """ + Remove a value from a list parameter + """ + current = self._settings.get(key) + if current and not isinstance(current, list): + logger.exit("You cannot only remove from a list variable.") + if not current or value not in current: + logger.exit("%s is not in %s" % (value, key)) + current.pop(current.index(value)) + self._settings[key] = current + self.change_validate(key, current) + logger.warning( + "Warning: Check with shpc config edit - ordering of list can change." + ) + def set(self, key, value): """ Set a setting based on key and value. If the key has :, it's nested @@ -81,6 +164,11 @@ def set(self, key, value): value = True if value == "true" else value value = False if value == "false" else value + # List values not allowed for set + current = self._settings.get(key) + if current and isinstance(current, list): + logger.exit("You cannot use 'set' for a list. Use add/remove instead.") + # This is a reference to a dictionary (object) setting if ":" in key: key, subkey = key.split(":") @@ -88,6 +176,13 @@ def set(self, key, value): else: self._settings[key] = value + # Validate and catch error message cleanly + self.change_validate(key, value) + + def change_validate(self, key, value): + """ + A courtesy function to validate a new config addition. + """ # Don't allow the user to add a setting not known try: self.validate() @@ -120,10 +215,14 @@ def delete(self, key): del self._settings[key] def save(self, filename=None): + """ + Save settings, but do not change order of anything. + """ filename = filename or self.settings_file if not filename: logger.exit("A filename is required to save to.") yaml = YAML() + with open(filename, "w") as fd: yaml.dump(self._settings, fd) diff --git a/shpc/settings.yml b/shpc/settings.yml index 82279a848..da6b15030 100644 --- a/shpc/settings.yml +++ b/shpc/settings.yml @@ -13,8 +13,9 @@ config_editor: vim # set a default container technology (singularity or podman) container_tech: singularity -# Registry Recipes (currently we support just one path) -registry: $root_dir/registry +# Registry Recipes (order of paths here is honored for search path)) +# Please preserve the flat list format for the yaml loader +registry: [$root_dir/registry] # Lmod or Environment Modules settings # The install directory for modules. Defaults to the install directory/modules diff --git a/shpc/tests/test_settings.py b/shpc/tests/test_settings.py new file mode 100644 index 000000000..614430ac9 --- /dev/null +++ b/shpc/tests/test_settings.py @@ -0,0 +1,50 @@ +#!/usr/bin/python + +# Copyright (C) 2021 Vanessa Sochat. + +# This Source Code Form is subject to the terms of the +# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed +# with this file, You can obtain one at http://mozilla.org/MPL/2.0/. + +import pytest +import shutil +import os + +from shpc.main.settings import Settings + +here = os.path.dirname(os.path.abspath(__file__)) +root = os.path.dirname(here) + + +def test_environment_substitution(tmp_path): + """Test variable substitution""" + settings_file = os.path.join(root, "settings.yml") + settings = Settings(settings_file) + + assert "/tmp" not in settings.module_base + os.environ["SOME_PATH"] = "/tmp" + os.putenv("SOME_PATH", "/tmp") + settings.set("module_base", "$SOME_PATH/modules") + assert settings.module_base == "/tmp/modules" + + +def test_set_get(tmp_path): + """Test variable set/get""" + settings_file = os.path.join(root, "settings.yml") + settings = Settings(settings_file) + assert not settings.container_base + settings.set("container_base", "/tmp/containers") + assert settings.container_base == "/tmp/containers" + + +def test_add_remove(tmp_path): + """Test variable add/remove""" + settings_file = os.path.join(root, "settings.yml") + settings = Settings(settings_file) + assert len(settings.registry) == 1 + settings.add("registry", "/tmp/containers") + assert len(settings.registry) == 2 + settings.remove("registry", "/tmp/containers") + assert len(settings.registry) == 1 + with pytest.raises(SystemExit): + settings.remove("registry", "/does/not/exist") diff --git a/shpc/version.py b/shpc/version.py index ebb951a6b..73dfb0a43 100644 --- a/shpc/version.py +++ b/shpc/version.py @@ -2,7 +2,7 @@ __copyright__ = "Copyright 2021, Vanessa Sochat" __license__ = "MPL 2.0" -__version__ = "0.0.28" +__version__ = "0.0.29" AUTHOR = "Vanessa Sochat" NAME = "singularity-hpc" PACKAGE_URL = "https://github.com/singularityhub/singularity-hpc"