Post-Build Testing

.github/actions/setup-instance/README.md

@@ -0,0 +1,51 @@
+# Setup Container Instance
+
+Action that prepares a build-ci runner container instance for CI by exporting key environment variables, updating repositories to requested refs, and configuring Spack mirrors/upstreams where required.
+
+## Inputs
+
+| Name | Type | Description | Required | Default | Example |
+| ---- | ---- | ----------- | -------- | ------- | ------- |
+| `spack-ref` | `string` (git branch, tag or sha) | The git ref to checkout for the `spack` repository. | `true` | N/A | `"develop"` or `"v1.0.0"` or `"5a1cdc4e"` or `""` |
+| `spack-config-ref` | `string` (git branch, tag or sha) | The git ref to checkout for the `spack-config` repository. | `true` | N/A | `"main"` or `"v0.5.2"` or `"3f14e99"` or `""` |
+| `builtin-spack-packages-ref` | `string` (git branch, tag or sha) | The git ref to checkout for the builtin spack packages repository. | `true` | N/A | `"develop"` or `"release-1.2"` or `"a8b7c6d"` or `""` |
+| `access-spack-packages-ref` | `string` (git branch, tag or sha) | The git ref to checkout for the `access-spack-packages` repository. | `true` | N/A | `"main"` or `"v2026.04"` or `"9e72b1c"` or `""` |
+| `spack-oci-buildcache-url` | `string` (URL) | The OCI registry URL to add as a Spack buildcache mirror. Pass an empty string (`''`) to skip adding an OCI mirror. | `true` | N/A | `"ghcr.io/access-nri/spack-buildcache"` or `""` |
+| `run-self-hosted` | `string` (boolean) | Whether this instance is running on a self-hosted runner. Controls upstream disabling and runner-set buildcache setup. | `true` | N/A | `"true"` or `"false"` |
+
+## Outputs
+
+| Name | Type | Description | Example |
+| ---- | ---- | ----------- | ------- |
+| `SPACK_ROOT` | `string` (path) | The Spack instance root path exported from the container environment. | `"/opt/spack"` |
+| `INITIAL_SPACK_REPO_VERSION` | `string` | The initial Spack repository version before any updates. | `"releases/v1.0"` |
+| `spack-env-dir` | `string` (path) | Absolute path to the runner environments directory used for artifact collection. | `"/opt/runner/environments"` |
+| `spack-config-sha` | `string` (sha) | The git SHA checked out for the `spack-config` repository. | `"5a1cdc4e4617fcd6ba1cccf1cd0432b5631983be"` |
+| `spack-sha` | `string` (sha) | The git SHA checked out for the `spack` repository. | `"d4e2fb7636e9ef3a8ebcf0f36f7f076f605ed87c"` |
+| `builtin-spack-packages-sha` | `string` (sha) | The git SHA checked out for the builtin spack packages repository. | `"4319878c9a9714f938df8f9a58d4f79aece39b7b"` |
+| `access-spack-packages-sha` | `string` (sha) | The git SHA checked out for the `access-spack-packages` repository. | `"77f9026d7e18bd4a0f9524f4b8a1e6f3f2345c9a"` |
+
+## Examples
+
+### Simple
+
+```yaml
+# ...
+jobs:
+  setup:
+    runs-on: ubuntu-latest
+    container:
+      image: access-nri/build-ci-runner:rocky
+    steps:
+    - id: instance
+      uses: access-nri/build-ci/.github/actions/setup-instance@v3
+      with:
+        spack-ref: releases/v1.1
+        spack-config-ref: main
+        builtin-spack-packages-ref: develop
+        access-spack-packages-ref: api-v2
+        spack-oci-buildcache-url: oci://ghcr.io/ACCESS-NRI/build-ci-buildcache
+        run-self-hosted: false
+
+    - run: echo "Spack root is ${{ steps.instance.outputs.SPACK_ROOT }}"
+```

.github/actions/setup-instance/action.yml

@@ -0,0 +1,172 @@
+name: Setup Container Instance
+description: Common action for getting a build-ci-runner containers repositories up to date and ready to run
+inputs:
+  spack-ref:
+    description: The git ref to checkout for the spack repository. Can be a branch, tag, or commit. If not provided, spack will not be updated and the version in the image will be used.
+    required: true
+  spack-config-ref:
+    description: The git ref to checkout for the spack-config repository. Can be a branch, tag, or commit. If not provided, spack-config will not be updated and the version in the image will be used.
+    required: true
+  builtin-spack-packages-ref:
+    description: The git ref to checkout for the builtin spack packages repository. Can be a branch, tag, or commit. If not provided, the ref specified in spack-config's repos.yaml will be used, or "develop" if not specified there either.
+    required: true
+  access-spack-packages-ref:
+    description: The git ref to checkout for the access spack packages repository. Can be a branch, tag, or commit. If not provided, access-spack-packages will not be updated and the version in the image will be used.
+    required: true
+  spack-oci-buildcache-url:
+    description: The URL of an OCI registry to use as a Spack buildcache. If not provided, no OCI registry will be added as a Spack mirror.
+    required: true
+  run-self-hosted:
+    description: Whether this instance is running on a self-hosted runner. Used to determine whether to disable upstreams and whether to enable the runner set buildcache.
+    required: true
+outputs:
+  # From env step
+  SPACK_ROOT:
+    value: ${{ steps.env.outputs.SPACK_ROOT }}
+    description: The SPACK_ROOT of the spack instance. This is used by future steps to source the spack environment and for path construction purposes.
+  INITIAL_SPACK_REPO_VERSION:
+    value: ${{ steps.env.outputs.INITIAL_SPACK_REPO_VERSION }}
+    description: The initial spack repository version before any updates. This is used to determine if the spack version in the image is compatible with build-ci.
+  spack-env-dir:
+    value: ${{ steps.env.outputs.spack-env-dir }}
+    description: The directory where spack environments are stored. This is used for artifact collection purposes in future steps.
+  # From *-update steps
+  spack-config-sha:
+    value: ${{ steps.spack-config-update.outputs.sha }}
+    description: The git sha that was checked out for the spack-config repository.
+  spack-sha:
+    value: ${{ steps.spack-update.outputs.sha }}
+    description: The git sha that was checked out for the spack repository.
+  builtin-spack-packages-sha:
+    value: ${{ steps.builtin-spack-packages-update.outputs.sha }}
+    description: The git sha that was checked out for the builtin spack packages repository.
+  access-spack-packages-sha:
+    value: ${{ steps.access-spack-packages-update.outputs.sha }}
+    description: The git sha that was checked out for the access spack packages repository.
+runs:
+  using: composite
+  steps:
+  - name: Init - Export environment variables into GitHub Actions format
+    shell: bash
+    # Environment variables inside containers are not accessible in the `env` context,
+    # a context which would be used in future conditional steps.
+    # This step exports important container environment variables as outputs.
+    # And yes, this loop echoes name-of-var=value-of-var!
+    id: env
+    run: |
+      for var in SPACK_ROOT INITIAL_SPACK_REPO_VERSION; do
+        echo "$var=${!var}" >> $GITHUB_OUTPUT
+      done
+
+      # The upload step later requires a non-relative path, hence the realpath
+      echo "spack-env-dir=$(realpath $SPACK_ROOT/../runner/environments)" >> $GITHUB_OUTPUT
+
+      # Initialize artifact content directory
+      mkdir -p ${{ env.artifact-content-dir }}
+
+  - name: Init - Disable Self-Hosted Upstreams
+    shell: bash
+    if: ${{ ! inputs.run-self-hosted }}
+    # For GitHub-hosted versions of build-ci, we don't have an upstream, so we disable in our user scope
+    # NOTE: In future may be able to be replaced by spack config --scope=user add upstreams::{}
+    run: |
+      . ${{ steps.env.outputs.SPACK_ROOT }}/share/spack/setup-env.sh
+
+      upstream_location=$(spack config --scope=user edit upstreams --print-file)
+      mkdir -p $(dirname $upstream_location)
+      echo "upstreams:: {}" > $upstream_location
+      spack config blame upstreams
+
+  - name: Init - Enable Runner Set Buildcache
+    if: inputs.run-self-hosted
+    shell: bash
+    run: |
+      . ${{ steps.env.outputs.SPACK_ROOT }}/share/spack/setup-env.sh
+
+      spack mirror add --scope=user --autopush --unsigned runner_set_buildcache /opt/runner_set_buildcache
+      spack mirror list
+
+  - name: Update - spack-config version
+    id: spack-config-update
+    uses: access-nri/build-ci/.github/actions/git-checkout-updated-ref@v3
+    with:
+      repository-path: ${{ steps.env.outputs.SPACK_ROOT }}/../spack-config
+      ref: ${{ inputs.spack-config-ref }}
+
+  - name: Update - spack version
+    id: spack-update
+    uses: access-nri/build-ci/.github/actions/git-checkout-updated-ref@v3
+    with:
+      repository-path: ${{ steps.env.outputs.SPACK_ROOT }}
+      ref: ${{ inputs.spack-ref }}
+
+  - name: Update - Validate non-updated spack
+    if: steps.spack-update.outputs.updated == 'false' && steps.spack-config-update.outputs.updated == 'false'
+    # We don't support spack < v1.0 in build-ci@v3, so we error out if the spack version is not updated and the version in the image is < v1.0
+    shell: bash
+    run: |
+      if [[ "${{ steps.env.outputs.INITIAL_SPACK_REPO_VERSION }}" == "releases/v0"* ]]; then
+        echo "::error::spack branch ${{ steps.env.outputs.INITIAL_SPACK_REPO_VERSION }} is not a v1.x branch, and build-ci@v3 only supports spack >= v1.0. Use build-ci@v2 for spack < v1.0"
+        exit 2
+      fi
+
+  - name: Spack - Initial Enable
+    shell: bash
+    run: |
+      . ${{ steps.env.outputs.SPACK_ROOT }}/share/spack/setup-env.sh
+      echo "----- Compilers -----"
+      spack compiler list
+      echo "----- Packages -----"
+      spack find
+
+  - name: Spack - Get builtin ref from spack-config
+    if: inputs.builtin-spack-packages-ref == ''
+    id: get-default-builtin
+    shell: bash
+    # If inputs.builtin-spack-packages-ref is not provided, get the builtin-spack-packages-ref
+    # from spack-config's repos.yaml or use "develop" (the default branch) if not specified there either.
+    run: |
+      . ${{ steps.env.outputs.SPACK_ROOT }}/share/spack/setup-env.sh
+      # Accept both the overridden .builtin: and regular .builtin key. See ACCESS-NRI/spack-config#105 for more info.
+      ref=$(spack config get repos | yq '.repos | .builtin // ."builtin:" | .branch // .tag // .commit // "develop"')
+      echo "Default builtin-spack-packages-ref is $ref"
+      echo "ref=$ref" >> $GITHUB_OUTPUT
+
+  - name: Spack - Get spack-packages repo locations
+    id: spack-packages-locations
+    shell: bash
+    run: |
+      . ${{ steps.env.outputs.SPACK_ROOT }}/share/spack/setup-env.sh
+
+      echo "builtin=$(spack location --repo builtin)" >> $GITHUB_OUTPUT
+      echo "access-spack-packages=$(spack location --repo access.nri)" >> $GITHUB_OUTPUT
+
+  - name: Update - builtin spack-package version
+    id: builtin-spack-packages-update
+    uses: access-nri/build-ci/.github/actions/spack-checkout-updated-ref@v3
+    with:
+      spack-packages-repository-name: builtin
+      spack-packages-repository-path: ${{ steps.spack-packages-locations.outputs.builtin }}
+      ref: ${{ inputs.builtin-spack-packages-ref || steps.get-default-builtin.outputs.ref }}
+      spack-instance-root-path: ${{ steps.env.outputs.SPACK_ROOT }}
+
+  - name: Update - access-spack-package version
+    id: access-spack-packages-update
+    uses: access-nri/build-ci/.github/actions/spack-checkout-updated-ref@v3
+    with:
+      spack-packages-repository-name: access_spack_packages
+      spack-packages-repository-path: ${{ steps.spack-packages-locations.outputs.access-spack-packages }}
+      ref: ${{ inputs.access-spack-packages-ref }}
+      spack-instance-root-path: ${{ steps.env.outputs.SPACK_ROOT }}
+
+  - name: Spack - OCI Buildcache Init
+    if: inputs.spack-oci-buildcache-url != ''
+    shell: bash
+    run: |
+      . ${{ steps.env.outputs.SPACK_ROOT }}/share/spack/setup-env.sh
+      spack mirror add --scope=user \
+        --unsigned \
+        --oci-username-variable OCI_USERNAME \
+        --oci-password-variable OCI_PASSWORD \
+        oci_buildcache ${{ inputs.spack-oci-buildcache-url }}
+      spack mirror list

.github/workflows/README.md

@@ -28,13 +28,8 @@ This workflow handles building and running short CI tests on a given spack manif
 | `container-image-version` | `string` (Docker version ref) | The version of the container image to use for the runner. Can be either a `:TAG` or a `@sha256:SHA`. | `false` | `":rocky"` | `':8.9'` (tag), `'@sha256:1234...'` (SHA) |
 | `spack-oci-buildcache-url` | `string` (OCI URL) | The URL to an oci-backed buildcache, available in spack >= v1.0. OCI-backed buildcaches are the only option for GitHub-hosted CI, and can be used as a backup for self-hosted CI's runner buildcache | `false` | N/A | `"oci://ghcr.io/ACCESS-NRI/build-ci-buildcache"`, `"oci://ghcr.io/ORG/IMAGE"` |
 | `run-self-hosted` | `boolean` | Whether to run the job on a self-hosted runner. For security, workflow runs will hang if this is `true` but it is not using a valid `v*` branch or tag for the workflow ref | `false` | `true` | `true`, `false` |
-
-##### Future Inputs
-
-| Name | Type | Description | Required | Default | Example |
-| ---- | ---- | ----------- | -------- | ------- | ------- |
-| `pytest-test-directory-path` | `string` (Directory path relative to component repository root) | Directory path in the caller model component repository that contains pytests to run against the built manifests | `false` | N/A | `".github/build/tests/"` |
-| `pytest-test-markers` | `string` (Pytest-style markers) | A string of pytest markers to use to filter tests in the caller model component repository | `false` | `""` (runs all tests) | `"not slow and not mpi"` |
+| `enable-pytest` | `boolean` | Whether to enable post-build testing of the manifest | `false` | `false` | `true`, `false` |
+| `pytest-search-path` | `string` | The path to search for pytest tests, relative to the caller repository root. Note that hidden directories are not searched in this default. | `false` | `.` | `.github/build/tests`, `tests/` |
 
 #### Secrets
 
@@ -60,12 +55,6 @@ This workflow handles building and running short CI tests on a given spack manif
 | `artifact-pattern` | `string` (glob) | Wildcard pattern to match all artifacts across a matrix job | `'output-*'` |
 | `artifact-url` | `string` (URL) | The URL of the artifact | `"https://github.com/ACCESS-NRI/MOM5/actions/runs/15890554355/artifacts/3406449135"` |
 
-##### Future Outputs
-
-| Name | Type | Description | Example |
-| ---- | ---- | ----------- | ------- |
-| `test-artifact-url` | `string` (URL) | The URL of the pytest result artifact | `"https://github.com/ACCESS-NRI/MOM5/actions/runs/15890554355/artifacts/3406449136"` |
-
 #### Examples
 
 ##### Minimal
@@ -263,6 +252,19 @@ jobs:
         done
 ```
 
+##### Post-Build Testing via Pytest
+
+Using `inputs.enable-pytest` and having tests discoverable by `pytest` under `inputs.pytest-search-path` in the MCR, one can run post-build testing of the given manifest. We use `pytest` as a sort of testing meta-framework - so we have a central interface for testing many different model components with many different testing frameworks.
+
+In order to add args to the `pytest` invocation, one can add a Reserved Definition (a section used by spack for reusable package definitions, etc, but prepended with a `_` to note that it is unused in the manifest), like so:
+
+```yaml
+spack:
+  definitions:
+  - _pytest-args: ['-m marker1 and not marker2', '--verbosity=1']
+  # ...
+```
+
 ## Other Workflows
 
 ### `containers-ci.yml` - Build and Push `build-ci-[upstream|runner]` Images