Add ExaChem CI integration for GHA and ADO pipelines + reorganize build scripts

[Loris Ercole (lorisercole)]

Adds ExaChem (external CCSD/TAMM MPI application) build/cache/test CI integration to both the GitHub Actions and Azure DevOps pipelines (Linux x86_64 only), reusing qdk-chemistry's already-built C++ dependencies wherever possible instead of letting TAMM's CMSB superbuild rebuild redundant copies. See code comments in the changed files for the detailed rationale behind each fix.

Scope / sequencing (read this first)

This PR only adds the build/cache/install side. The Python-level consumer (qdk_chemistry.plugins.exachem, tests/test_exachem_ccsd_integration.py) lives on the separate, still-open PR #611 and is not part of this branch. Until #611 merges, the only thing exercising the built ExaChem binary is install-exachem.sh's own internal smoke test (link check + minimal mpirun run), not a qdk-chemistry-level pytest.

Pipeline changes

GitHub Actions (build-and-test.yaml)

• Each Linux coverage matrix leg builds/caches GlobalArrays+TAMM+ExaChem (via install-exachem.sh) alongside the existing C++ deps cache, keyed on the same manifest/install-script hashes.
• ExaChem's PATH/LD_LIBRARY_PATH are scoped only to the "Run Python tests" step, not the whole job.
• test-with-qdk-branch no longer rebuilds C++/Python from scratch: it now depends on coverage and downloads that job's already-built install directory as an artifact, then just overlays the qdk branch package and re-runs pytest.
• Removed the "Free up disk space" cleanup step (targeted GitHub-hosted-runner bloat that doesn't apply to our self-hosted pools).
• Linux and Windows legs cap CMAKE_BUILD_PARALLEL_LEVEL by available RAM (~8 GB/job) instead of raw core count, to avoid OOM/heap-exhaustion kills on shared runners (issue emerged after Bumping LibInt2 version in CI/CD and CMake #692).

Azure DevOps (python-wheels.yaml)

• New Build_linux_x86_64_exachem job (gated on linux_x86_64 being in the matrix) builds the same GlobalArrays+TAMM+ExaChem stack against BLIS+LibFLAME inside the wheel-build container, caches it (Cache@2, keyed including dockerImage), and publishes it as a pipeline artifact.
• The linux_x86_64 wheel-test job downloads that artifact and exposes it to test-pip-wheels.sh the same way GHA does; a command -v ExaChem check confirms the binary is actually on PATH in the test environment.
• dockerImage is now also an explicit parameter where python-wheels.yaml calls the ExaChem build template.
• Removed the same disk-space cleanup step from build-pip-wheels.yml for consistency.

Shared install scripts (.pipelines/install-scripts/)

• Centralized install-blaspp.sh/install-lapackpp.sh/install-cpp-deps.sh, used by both the devcontainer and the pipelines instead of duplicated inline logic; both forward BUILD_TYPE to blaspp/lapackpp, and install-cpp-deps.sh's job count now defers to the pipeline's own memory-aware CMAKE_BUILD_PARALLEL_LEVEL before falling back to raw core count.
• BLIS/LibFLAME versions are now hardcoded (no caller ever overrode them); HDF5's version/URL/hash are sourced from cgmanifest.json (converted to an other-type entry recording the actual downloaded tarball, matching Libint's pattern) instead of a separate pipeline parameter.
• Checksum verification prefers sha256sum/sha1sum over macOS-only shasum for portability.
• .pipelines/install-scripts/exachem/patches/: upstream-portable patches for CMSB dependency reuse, GlobalArrays' LAPACK vendor selection, and ExaChem/GauXC/LibInt2 compatibility fixes — each documents its own root cause and pinned-commit verification. Bumping Libint to 2.13.1 (via a main merge) retired the old DEPRECATED-macro patch and required a new DATADIR→LIBINT_DATADIR cherry-pick patch, since upstream's own fix for this also raises the minimum GCC version past what our runners have.
• cpp/manifest/qdk-chemistry/cgmanifest.json: added GlobalArrays/TAMM/ExaChem/CMSB as development dependencies, for Component Governance visibility (they execute in CI even though never shipped in the wheel). The pins in install-exachem.sh remain authoritative for the build; keep both in sync if bumped.

Validation

Green ADO and GHA runs building/caching/smoke-testing ExaChem end-to-end on Linux (both OpenBLAS and BLIS+LibFLAME legs), plus green Linux/macOS/Windows wheel builds. No qdk-chemistry-level Python test yet exercises ExaChem (see "Scope / sequencing" above).

[Copilot]

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

.pipelines/install-scripts/install-cpp-deps.sh:62

• JOBS is always set to nproc/sysctl output here, so callers cannot cap dependency build parallelism (e.g., to avoid OOM on constrained runners). This script already supports overriding other knobs via env vars, so JOBS should follow the same pattern (default only when unset).

if command -v nproc >/dev/null 2>&1; then
    JOBS=$(nproc) # Linux
else
    JOBS=$(sysctl -n hw.logicalcpu) # macOS
fi

.pipelines/install-scripts/common.sh:68

• The get_hash() doc comment says Boost doesn’t declare a hash, but cgmanifest.json now includes a Boost hash. This is misleading for future callers/maintainers; update the comment to be accurate/generic.

# Resolve the declared SHA-1 (`hash`) for an "other" (non-git) type component from cgmanifest.json, by
# component name. Not every "other" entry declares one (e.g. Boost doesn't); callers must check for empty.
get_hash() {

[Copilot]

Pull request overview

Copilot reviewed 25 out of 26 changed files in this pull request and generated 1 comment.

Suppressed comments (5)

Previously missed (3) — in code that hasn't changed since the last review.

.github/workflows/build-and-test.yaml:213

• The dependency cache key doesn’t include .pipelines/install-scripts/common.sh, but install-cpp-deps.sh and exachem/install-exachem.sh both source it. If common.sh changes, the cache can restore stale installs built with different helper logic. Add common.sh to the hashFiles(...) list (and keep the save key in sync).

This issue also appears on line 257 of the same file.

        key: >-
          cpp-deps-${{ runner.os }}-${{ matrix.uarch }}-
          ${{ hashFiles('cpp/manifest/qdk-chemistry/cgmanifest.json', 'external/macis/manifest/cgmanifest.json', '.pipelines/install-scripts/install-cpp-deps.sh', '.pipelines/install-scripts/install-blaspp.sh', '.pipelines/install-scripts/install-lapackpp.sh', '.pipelines/install-scripts/exachem/**') }}

.pipelines/templates/build-exachem-linux.yml:33

• The ExaChem cache key doesn’t include .pipelines/install-scripts/common.sh, but multiple scripts in this key list source it (e.g., install-cpp-deps.sh and exachem/install-exachem.sh). If common.sh changes, the cache can restore stale builds even though the build logic changed. Include common.sh in the Cache@2 key inputs so the cache is invalidated correctly.

      "linux-exachem-${{ parameters.march }}-v1" | "${{ parameters.dockerImage }}" |
      cpp/manifest/qdk-chemistry/cgmanifest.json |
      external/macis/manifest/cgmanifest.json |
      .pipelines/install-scripts/build-exachem-linux.sh |
      .pipelines/install-scripts/install-blis.sh |

.pipelines/templates/build-exachem-linux.yml:35

• The Cache@2 path points to $(System.DefaultWorkingDirectory)/exachem_install, but the docker build writes ./exachem_install under /workspace/qdk-chemistry (i.e., under $(Agent.BuildDirectory)/qdk-chemistry/exachem_install). With the current path, the cache will likely miss and/or save an empty directory. Point the cache path at the actual produced install directory.

This issue also appears on line 49 of the same file.

      .pipelines/install-scripts/exachem/**/*
    path: $(System.DefaultWorkingDirectory)/exachem_install
    cacheHitVar: EXACHEM_INSTALL_CACHE_RESTORED

.pipelines/templates/build-exachem-linux.yml:51

• This ownership-fix step uses the same $(System.DefaultWorkingDirectory)/exachem_install path as the cache, but the docker build output lives under the checkout directory (.../qdk-chemistry/exachem_install). If the path is wrong, the step won’t fix permissions and later cache/save/artifact steps can fail. Use the same install directory path that the build produces.

- script: |
    sudo chown -R $(id -u):$(id -g) $(System.DefaultWorkingDirectory)/exachem_install
  displayName: Fix ExaChem install directory ownership

.github/workflows/build-and-test.yaml:259

• The cache save key should match the restore key. Since .pipelines/install-scripts/common.sh is sourced by install scripts, it should be part of the hashFiles(...) inputs here as well; otherwise a change to common.sh won’t invalidate the saved cache entry.

        key: >-
          cpp-deps-${{ runner.os }}-${{ matrix.uarch }}-
          ${{ hashFiles('cpp/manifest/qdk-chemistry/cgmanifest.json', 'external/macis/manifest/cgmanifest.json', '.pipelines/install-scripts/install-cpp-deps.sh', '.pipelines/install-scripts/install-blaspp.sh', '.pipelines/install-scripts/install-lapackpp.sh', '.pipelines/install-scripts/exachem/**') }}

[Copilot]

Pull request overview

Copilot reviewed 25 out of 26 changed files in this pull request and generated no new comments.

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

.github/workflows/build-and-test.yaml:213

• The cache key hashFiles list omits .pipelines/install-scripts/common.sh, but both install-cpp-deps.sh and exachem/install-exachem.sh source it. Changes to common.sh (e.g., download verification behavior) would not invalidate this cache, risking stale or mismatched dependency installs being restored.

This issue also appears on line 260 of the same file.

          ${{ hashFiles('cpp/manifest/qdk-chemistry/cgmanifest.json', 'external/macis/manifest/cgmanifest.json', '.pipelines/install-scripts/install-cpp-deps.sh', '.pipelines/install-scripts/install-blaspp.sh', '.pipelines/install-scripts/install-lapackpp.sh', '.pipelines/install-scripts/exachem/**') }}

.pipelines/templates/build-exachem-linux.yml:33

• The Cache@2 key inputs omit .pipelines/install-scripts/common.sh, but the keyed scripts (install-cpp-deps.sh and exachem/install-exachem.sh) source it. Without including common.sh, changes to shared helpers won’t invalidate the ExaChem install cache and can restore stale builds.

      .pipelines/install-scripts/install-cpp-deps.sh |
      .pipelines/install-scripts/install-blaspp.sh |
      .pipelines/install-scripts/install-lapackpp.sh |
      .pipelines/install-scripts/exachem/**/*

.github/workflows/build-and-test.yaml:260

• Same cache-key issue as above: the save-step hashFiles list omits .pipelines/install-scripts/common.sh, so updates to shared helpers won’t invalidate the saved cache entry.

          ${{ hashFiles('cpp/manifest/qdk-chemistry/cgmanifest.json', 'external/macis/manifest/cgmanifest.json', '.pipelines/install-scripts/install-cpp-deps.sh', '.pipelines/install-scripts/install-blaspp.sh', '.pipelines/install-scripts/install-lapackpp.sh', '.pipelines/install-scripts/exachem/**') }}

[Copilot]

🔵 Needs a closer look

The current workflow/scripts have a couple of reliability issues (cache directory existence across matrix legs and non-idempotent shallow checkouts) that should be fixed to avoid CI flakes.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

.github/workflows/build-and-test.yaml:96

• actions/cache is configured to restore/save ${{ env.EXACHEM_INSTALL_PREFIX }} for every matrix leg, but ExaChem is only built on Linux. On macOS runs, this directory is never created, which can cause the cache step to warn and skip saving/restoring. Pre-create the directory early in the job so cache restore/save has a real path regardless of OS.

        echo "PATH=${{ github.workspace }}/install/bin:${{ github.workspace }}/install/lib:$PATH" >> $GITHUB_ENV
        echo "PYTHONPATH=${{ github.workspace }}/install/lib/python${{ matrix.python-version }}/site-packages:${{ github.workspace }}/install/lib:$PYTHONPATH" >> $GITHUB_ENV
        # ${LD_LIBRARY_PATH:+...} avoids a trailing ":" when unset -- the dynamic loader treats an empty
        # path component as the current working directory, an untrusted-search-path risk.
        echo "LD_LIBRARY_PATH=${{ github.workspace }}/install/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" >> $GITHUB_ENV

.pipelines/install-scripts/install-cpp-deps.sh:162

• shallow_checkout does git remote add origin ... and will fail if the destination directory already exists from a previous interrupted run (remote already defined). Since CI workspaces can be retried/reused, remove the target dir before calling shallow_checkout (and consider applying the same pattern to the other dependency checkouts below).

echo "=== Installing nlohmann_json ==="
shallow_checkout https://github.com/nlohmann/json.git "$NJSON_COMMIT" nlohmann_json
cd nlohmann_json

• Files reviewed: 25/26 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

There are concrete maintainability/documentation issues in the new dependency installer (Libint extraction block and GauXC/Catch2 comment) that should be corrected to prevent confusion and reduce the risk of incorrect builds.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

.pipelines/install-scripts/install-cpp-deps.sh:238

• The Libint extraction comments are outdated (they reference the old 2.9.0 tarball) and the directory auto-detection can pick up a stale libint-* directory if a previous run left one behind. This can lead to configuring/building the wrong extracted tree. Consider clearing any existing libint-* directories before extracting, and make the comment version-agnostic.
  .pipelines/install-scripts/install-cpp-deps.sh:286
• This comment says GauXC testing requires Catch2 v2, but the manifests pin Catch2 v3.3.2. If the rationale is simply that GauXC tests are not needed in this CI dependency build, update the comment to avoid a misleading version-specific claim.

• Files reviewed: 25/26 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The ADO ExaChem cache/ownership steps currently target a directory path inconsistent with where the Docker build writes the install, and cache invalidation in GHA misses the shared common.sh helper.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

.pipelines/templates/build-exachem-linux.yml:50

• This ownership-fix step uses $(System.DefaultWorkingDirectory)/exachem_install, but the build writes exachem_install under the repo checkout directory (see the Docker -w /workspace/qdk-chemistry above, and the artifact upload path in .pipelines/python-wheels.yaml). If these paths differ, chown will fail or silently miss the actual install dir.

- script: |
    sudo chown -R $(id -u):$(id -g) $(System.DefaultWorkingDirectory)/exachem_install

.github/workflows/build-and-test.yaml:248

• Same as the restore key above: the save key should also include .pipelines/install-scripts/common.sh so cache invalidation stays correct when shared helper logic changes.

        key: >-
          cpp-deps-${{ runner.os }}-${{ matrix.uarch }}-
          ${{ hashFiles('cpp/manifest/qdk-chemistry/cgmanifest.json', 'external/macis/manifest/cgmanifest.json', '.pipelines/install-scripts/install-cpp-deps.sh', '.pipelines/install-scripts/install-blaspp.sh', '.pipelines/install-scripts/install-lapackpp.sh', '.pipelines/install-scripts/exachem/**') }}

• Files reviewed: 24/25 changed files
• Comments generated: 3
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

It introduces wide-ranging CI/pipeline behavior changes (new caches, artifacts, external MPI application build/install flow) that are difficult to fully validate via diff-only review.

Review details

• Files reviewed: 24/25 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The GHA cache-save step can silently skip saving the C++ deps cache on non-Linux legs because EXACHEM_INSTALL_PREFIX may not exist, causing path validation to skip caching entirely.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.pipelines/install-scripts/install-cpp-deps.sh:237

• This comment still references the old libint-2.9.0 tarball name, but the manifest pins Libint 2.13.1. Updating it will avoid confusion when debugging extraction/layout issues.

• Files reviewed: 24/25 changed files
• Comments generated: 1
• Review effort level: Lite