Skip to content

Set ENV_LAUNCHER_SCRIPT_PATH and only expose payu commands#19

Closed
jo-basevi wants to merge 2 commits intomainfrom
17-set-launcher-script-env
Closed

Set ENV_LAUNCHER_SCRIPT_PATH and only expose payu commands#19
jo-basevi wants to merge 2 commits intomainfrom
17-set-launcher-script-env

Conversation

@jo-basevi
Copy link
Copy Markdown
Collaborator

This PR:

  • Add ENV_LAUNCHER_SCRIPT_PATH to launcher_conf.sh: This means for scripts that are run as part of launcher.sh and within the singularity container, it is possible to find out how to launch the container again. This is useful for payu, so it can pass the launcher script along to the PBS jobs. This was tested with a payu branch that adds the launcher script environment variable to the PBS commands, so it looks like qsub <qsub-flags> -- /path/to/launcher.sh /path/to/squashfs-payu-env/bin/python /path/to/squashfs-payu-env/bin/payu-run. This removes the hack of modifying the payu python headers to point to the python launcher script.
  • In a continuation of another PR which has an issue of the payu environment clashing with other conda environments, I had a go at only exposing the payu launcher script commands, now that payu didn't need a python launcher script.

Pros of restricting commands to only payu and removing any conda environment variables (so environment is not activated):

  • It works with the conda/analysis3 environments without a pre-existing payu (version <= conda/analysis3-25.04), provided both modules are loaded before running any scripts/commands. This still means it wouldn't be possible to run the conda/analysis3 environment executables in userscripts in payu as the squahfs environment overlays are needed when the container in launched on the PBS node. But it'll be possible load conda/analysis3 or another conda environment on login nodes and submit payu jobs via the command line.

Cons:

  • Creating virtual environments no longer work as I removed the python launcher script and activating the conda environment.. Virtual environments are used for repro testing and are built off the payu environment to reduce the size. A solution could be to have two options for modules, e.g. one that exposes all the launcher scripts for the environment and activates the conda environment, and one that just has the payu commands?
  • Any entry-points will need to manually be added.

This means once launcher.sh is run or within container, it is possible to find the path for the launcher script.
This is useful for payu, so it can pass the launcher script along to PBS jobs
- Add launcher_commands array to environments config.sh. If empty, all executables in the conda bin directory will be added as launcher scripts during build
- Add new modulefile that only adds the launcher scripts to the PATH and removes all conda environment values.
@jo-basevi
Copy link
Copy Markdown
Collaborator Author

Just realised I was only testing pip installed versions of payu, which has the python shebang header modified to point to the environment python (in the build_inner.sh script). I ran a deployment with a versioned payu from conda (with no modifications) and it defaults to using the system python within the container, so I would need to activate the conda environment when payu commands are run.

@jo-basevi
Copy link
Copy Markdown
Collaborator Author

Closing this PR as it's superseded by #32

@jo-basevi jo-basevi closed this Aug 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant