Skip to content

Pull UKMO components from Github #385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Whyborn wants to merge 17 commits into from
Closed

Pull UKMO components from Github #385

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
7eabff3
Change UKCA source
Whyborn Feb 10, 2026
26e2087
Fix missing bracket
Whyborn Feb 10, 2026
743214a
Missed a _ref on a name
Whyborn Feb 10, 2026
381aaf3
Add shumlib, socrates, casim to github refs
Whyborn Feb 16, 2026
ba20703
Set GCOM source to Github
Whyborn Feb 16, 2026
c0f985e
Fix incorrect syntax
Whyborn Feb 16, 2026
a68a97e
Change UKCA source
Whyborn Feb 10, 2026
34a9f2d
Fix missing bracket
Whyborn Feb 10, 2026
0e6d9e9
Missed a _ref on a name
Whyborn Feb 10, 2026
25cb004
Add shumlib, socrates, casim to github refs
Whyborn Feb 16, 2026
381e0ef
Set GCOM source to Github
Whyborn Feb 16, 2026
67bbaaf
Fix incorrect syntax
Whyborn Feb 16, 2026
374891d
Rebase onto api-v2
Whyborn Feb 17, 2026
89c8977
Remove _rev_variants from um package
Whyborn Feb 17, 2026
2302f86
ci: Remove um and gcom from excluded packages list
CodeGat Feb 19, 2026
43e0d97
Remove all references to _rev_variants
Whyborn Feb 23, 2026
5f4de99
Merge branch 'api-v2' into 381-pull-UKMO-components-from-github
Whyborn Feb 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,6 @@ jobs:
- package: {template_value: "access-test"}
- package: {template_value: "coastri-roms"}
- package: {template_value: "gcom4"}
- package: {template_value: "um"}
uses: access-nri/build-ci/.github/workflows/ci.yml@v3
with:
spack-manifest-path: ${{ matrix.package.filepath }}
Expand Down
18 changes: 9 additions & 9 deletions spack_repo/access/nri/packages/gcom/package.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,18 @@ class Gcom(Package):
GCOM is a wrapper around multiprocessing libraries such as MPI
"""

homepage = "https://github.com/ACCESS-NRI/gcom"
homepage = "https://code.metoffice.gov.uk/trac/gcom"
# svn = "file:///g/data/ki32/mosrs/gcom/main/trunk"
Copy link
Copy Markdown
Collaborator

@penguian penguian Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want to remove a line, just remove it, otherwise explain in the comment why the line is commented out.

git = "https://github.com/ACCESS-NRI/gcom"

maintainers("scottwales", "paulleopardi")
Copy link
Copy Markdown
Collaborator

@penguian penguian Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should you add yourself to maintainers?


version("7.8", tag="vn7.8")
version("7.9", tag="vn7.9")
version("8.0", tag="vn8.0")
version("8.1", tag="vn8.1")
version("8.2", tag="vn8.2")
version("8.3", tag="vn8.3")
version("8.4", tag="vn8.4")
version("7.8", tag="vn7.8", commit="65c857cc3201833360ff62b285e49082378dae42")
version("7.9", tag="vn7.9", commit="6319ae016dcadc192842a06178f3dbd21a8af64f")
version("8.0", tag="vn8.0", commit="5a122e9e2147c9a7486a00b9588356eb65324af9")
version("8.1", tag="vn8.1", commit="e061e2787bd643d9a65679f153fb8e3ffd9d3186")
version("8.2", tag="vn8.2", commit="fd143bb38e21fe03c7150c0754852c80e15df3d4")
version("8.3", tag="vn8.3", commit="b7b890a181d8e31e4e80b731b9f8ad9a6e1a8bed")
version("8.4", tag="vn8.4", commit="f4fa92eb4af4f1e4cf9d608b441e3c96f77b6a6d")

variant("mpi", default=True, description="Build with MPI")

Expand Down
84 changes: 34 additions & 50 deletions spack_repo/access/nri/packages/um/package.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,17 +64,13 @@ class Um(Package):
# String variants have their default values set to "none" here.
# The real default is set by the model.

# Revision variants.
_rev_variants = (
"casim_rev",
"jules_rev",
"shumlib_rev",
"socrates_rev",
"ukca_rev")

Copy link
Copy Markdown
Collaborator

@penguian penguian Feb 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does removing the "_rev" variants force all builds to use GitHub only? Will MOSRS-based builds still succeed?

Copy link
Copy Markdown
Collaborator Author

@Whyborn Whyborn Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm genuinely unsure. The logic in the spack package is too hard to follow. The existing AM3 spack.yaml did not specify revisions or variants for these components- did that mean it took the revision from the specified UM revision?

Copy link
Copy Markdown
Collaborator

@penguian penguian Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will do some testing today.

Copy link
Copy Markdown
Collaborator

@penguian penguian Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI is failing with

Run . /opt/spack/share/spack/setup-env.sh
==> Error: cannot load package 'um' from the 'access.nri' repository: name '_rev_variants' is not defined
==> Created environment default in: /opt/environments/default
==> Activate with: spack env activate default
==> Activated default environment in /opt/environments/default
==> Error: cannot load package 'um' from the 'access.nri' repository: name '_rev_variants' is not defined
Error: Process completed with exit code 1.

It is probably best to revert your removal of the _rev variants so that the remainder of the CI check can proceed.

Copy link
Copy Markdown
Collaborator

@penguian penguian Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it is may be better to migrate from MOSRS to GitHub gradually. I will need to think of how to do this.

Copy link
Copy Markdown
Collaborator

@penguian penguian Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the UM Spack build, the main configuration elements are the model/*/rose-app.conf files. The Python code for the UM Spack package recipe starts from the appropriate rose-app.conf file corresponding to the version, modifies this config based on the variants provided by the Spack spec, exports the result to corresponding environment variables, and then runs fcm make using fcm-make.cfg. The fcm make run depends only on the environment variables and the contents of fcm-make.cfg. In particular, the config_root_path variable determines where

include = $config_root_path/fcm-make/$platform_config_dir/um-$config_type-$optimisation_level.cfg$config_revision

points to. If (e.g.) config_root_path is left as fcm:um.xm_tr, then fcm make will use the UM trunk from the MOSRS mirror. See #362

Copy link
Copy Markdown
Collaborator

@penguian penguian Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At this point, I would recommend waiting until I have made the changes needed for #362 to be merged.

# Git reference variants.
_ref_variants = (
"casim_ref",
"jules_ref",
"shumlib_ref",
"socrates_ref",
"ukca_ref",
"um_ref")

# Other string variants.
Expand Down Expand Up @@ -115,7 +111,7 @@ class Um(Package):
"um_sources"
)

_str_variants = _rev_variants + _ref_variants + _other_variants
_str_variants = _ref_variants + _other_variants

for var in _str_variants:
variant(var, default="none", description=var, values="*", multi=False)
Expand Down Expand Up @@ -174,15 +170,32 @@ class Um(Package):

# Optional Github sources to be used in build (i.e. AM3)
_resource_cfg = {
"casim_ref": {
"sources_var": "casim_sources",
"git_url": "https://github.com/ACCESS-NRI/casim.git",
"subdir": "casim"},
"jules_ref": {
"sources_var": "jules_sources",
"git_url": "https://github.com/ACCESS-NRI/JULES.git",
"subdir": "jules"},
"shumlib_ref": {
"sources_var": "shumlib_sources",
"git_url": "https://github.com/ACCESS-NRI/shumlib.git",
"subdir": "shumlib"},
"socrates_ref": {
"sources_var": "socrates_sources",
"git_url": "https://github.com/ACCESS-NRI/socrates.git",
"subdir": "socrates"},
"um_ref": {
"sources_var": "um_sources",
"git_url": "https://github.com/ACCESS-NRI/UM.git",
"subdir": "um"}}

"subdir": "um"},
"ukca_ref": {
"sources_var": "ukca_sources",
"git_url": "https://github.com/ACCESS-NRI/ukca.git",
"subdir": "ukca"
}
}

def _config_file_path(self, model):
"""
Expand Down Expand Up @@ -320,17 +333,6 @@ def check_model_vs_sources_vs_ref(
check_model_vs_spec(model, config_env, var, spec_str_value)
config_env[var] = spec_str_value

# Override those environment variables where a revision variant is specified.
# If the variant is left unspecified, and the model does not specify a revision,
# then use a component revision based on the spec UM version.
for var in self._rev_variants:
spec_value = spec.variants[var].value
if spec_value != "none":
check_model_vs_spec(model, config_env, var, spec_value)
config_env[var] = spec_value
elif var not in config_env or config_env[var] == "":
config_env[var] = f"um{spec.version}"

# Override those environment variables where any other string variant is specified.
for var in self._other_variants:
spec_value = spec.variants[var].value
Expand Down Expand Up @@ -359,34 +361,16 @@ def check_model_vs_sources_vs_ref(
linker_args = self._get_linker_args(spec, var)
config_env[f"ldflags_{fcm_name}_on"] = linker_args

# The _resource_cfg is relevant only for models that use Github URLs.
# Only one model so far, but this may change in future.
if model == "vn13p1-am":
# Get the root to the resources
resources_root = join_path(self.stage.source_path, "resources")
# Add sources to the environment if requested
for ref_var in self._resource_cfg:
ref_value = spec.variants[ref_var].value
if ref_value != "none":
sources_var = self._resource_cfg[ref_var]["sources_var"]
subdir = self._resource_cfg[ref_var]["subdir"]
resource_path = join_path(resources_root, subdir)
# Output appropriate warning messages.
check_model_vs_sources_vs_ref(
model,
config_env,
sources_var,
ref_var,
resource_path)
config_env[sources_var] = resource_path
else:
# The model does not use Github URLs and ignores the ref variants.
for ref_var in self._resource_cfg:
ref_value = spec.variants[ref_var].value
if ref_value != "none":
tty.warn(
f"The {model} model ignores the variant "
f"{ref_var}={ref_value}.")
# Get the root to the resources
resources_root = join_path(self.stage.source_path, "resources")
# Add sources to the environment if requested
for ref_var in self._resource_cfg:
ref_value = spec.variants[ref_var].value
if ref_value != "none":
sources_var = self._resource_cfg[ref_var]["sources_var"]
subdir = self._resource_cfg[ref_var]["subdir"]
resource_path = join_path(resources_root, subdir)
config_env[sources_var] = resource_path

# Set environment variables based on config_env.
for key in config_env:
Expand Down
Loading