Post-Build Testing

[CodeGat]

Closes #193

Background

Right now, we install the component manifests but don't do any testing outside of the installation status of spack. We need a framework for post-build testing, and right now we're going with pytest due to the strength of test specification via --markers, it's use as a testing meta-framework in other repositories such as ACCESS-NRI/model-config-tests, and it's integration with GitHub Actions.

Essentially we will have a second job (Test) after the first (Install), that will reinstall the lockfile via a buildcache, then discover all tests in the MCR, and run them. Users can specify args to pytest at the manifest level via a reserved definition _pytest-args, like so:

spack:
  definitions:
  - _pytest-args: ['-m marker1 and not marker2', '--verbosity=1']
  # ... and the rest

Note

The exact machinations of pytest discovery will be based on the resolution of #298

The PR

• Add new test-build dependent job in the ci.yml workflow that install the previously installed-and-pushed packages from the lockfile, then runs pytest
• Optional args to pytest are given at the manifest level via a _pytest-args reserved definition in the manifest.
• Consolidated a large amount of setup code for both jobs under the new .github/actions/setup-instance action.

Testing

Proof of Concept

Tested in ACCESS-NRI/access-test-component#19, specifically https://github.com/ACCESS-NRI/access-test-component/actions/runs/24760630710?pr=19. Note the tests under .github/build-ci/tests, discovered by inputs.pytest-search-path (as hidden directories aren't found traditionally.

Also note the reading of reserved definitions, for example written in https://github.com/ACCESS-NRI/access-test-component/pull/19/changes#diff-b5e432583957cd6026f987a7ddf4b2df3a0daab1e7afd40293562f16c0d6b4b1R3 and added successfully in https://github.com/ACCESS-NRI/access-test-component/actions/runs/24760630710/job/72443635335#step:11:8.

Installation in the test job is much quicker when using a lockfile and spack install --use-cache - possibly because we skip concretization and go to querying the buildcache immediately - from 2m with perfect reuse in the initial install job to 20s with perfect reuse in the later test job.

[CodeGat]

Hmm...currently we activate the environment in the CI before we run pytest, meaning we don't have to have pre-test environment activation. But maybe that's a good thing that we're explicit, so tests outside of CI (but using spack) attempt to activate the environment. In any case, the fixture would look like:

@pytest.fixture(autouse=True)
def spack_env(pytestconfig):
    spack_env_path: str | None = pytestconfig.getoption("spack_env_path", default=None)

    if not spack_env_path:
        # Don't use this fixture if the spack_env_path is not set
        return

    # Reference the default environment the packages were installed in, before each test
    env: spack.environment.Environment = spack.environment.Environment(
        spack_env_path
    )

    # Activate the spack environment, and collect the users environment variable modifications
    env_activate_mods = spack.environment.shell.activate(env)
    # Apply the environment variable modifications to the user
    env_activate_mods.apply_modifications()

    if not env.active:
        raise RuntimeError(
            f"Couldn't activate the default environment at {spack_env_path}"
        )

    # Return control to the test
    yield

    # Once the above completes, deactivate the environment and collect the now-reversed user environment modifications
    env_deactivate_mods = spack.environment.shell.deactivate()
    # Apply those modifications so we're now back where we started!
    env_deactivate_mods.apply_modifications()