CI: the root QA job never finishes — self-hosted runner dies ~50min in, and the declared lts QA job is never created

[ChrisRackauckas-Claude]

The root QA test group has not completed a run in CI recently. It is not a test failure — the job is killed part-way through.

Evidence

commit	job	outcome	annotation
2c31d86323 (master, #4880 merge)	92750873732	failure	"The self-hosted runner lost communication with the server."
82dee851d6 (master)	92939171244	failure	same
453c45c08e (master, current head)	92967338349	failure	"The operation was canceled."
b2692a1bc6 (#4883)	92988983176	failure	"The self-hosted runner lost communication with the server."

The b2692a1bc6 job ran 20:26:03Z → 21:15:06Z — 49 minutes — on self-hosted-4vcpu-16gb-dpg5l-runner-cmq2q before dying.

Likely cause

run_qa runs JET.report_package(ModelingToolkit; target_defined_modules = true), which is by far the heaviest thing in the group. Locally the whole test/qa/qa.jl takes 30m11s wall on Julia 1.12.6, with JET dominating and the Aqua Persistent tasks check adding another 30s; on Julia 1.10 the same file takes 3m42s, so the cost is very lopsided toward the newer compiler.

That is the same profile test/test_groups.toml already documents for two other groups:

# Memory-heavy (OOM-killed on the 4vcpu/8GB self-hosted runners, esp. on `pre`).
# Route to a GitHub-hosted runner (~16GB). `ubuntu-24.04` forces GitHub-hosted
# since the self-hosted pool only carries the `ubuntu-latest` label.
runner = "ubuntu-24.04"

[Initialization] and [FMI] carry that; [QA] does not:

[QA]
versions = ["lts", "1"]

The obvious thing to try is giving [QA] the same runner = "ubuntu-24.04". I have not verified that it fixes it — it is a CI-infrastructure change I cannot reproduce locally, so I am filing it rather than sending a speculative PR.

Secondary: the declared lts QA job is never created

[QA] declares versions = ["lts", "1"], but only a julia 1 job is ever created. Checked on 2c31d86323, 82dee851d6, 453c45c08e and b2692a1bc6 — in every workflow run the only QA job is tests / QA (julia 1, ubuntu-latest), and gh api .../actions/runs/<id>/jobs confirms there is no lts counterpart, while every other group gets all of its declared versions.

So whatever compute_affected_sublibraries.jl --root-matrix does with the [QA] entry, the lts half of the declared matrix is being dropped. That matters because ExplicitImports behaves differently before Julia 1.12 (see #4903) — those findings are invisible to CI today.

Impact

The QA lane cannot go green on any PR right now, so #4883 and #4881 have to be reviewed on locally-run evidence rather than on a CI result.

[ChrisRackauckas-Claude]

Reproduced the QA group locally on clean master at 731eff66b596fbc6bc36aff45b21bb2e81e1bddb. Three findings, one of which contradicts the proposed fix in this issue.

1. The runner change proposed here would not have fixed it — the latest failing job was already GitHub-hosted

The most recent master QA failure is https://github.com/SciML/ModelingToolkit.jl/actions/runs/31247825950/job/93089931661. Its log header says:

Current runner version: '2.336.0'
##[group]Runner Image Provisioner
Hosted Compute Agent
...
Azure Region: eastus
##[group]Runner Image
Image: ubuntu-24.04
Version: 20260720.247.2
...
  self-hosted: false

So ubuntu-latest already landed on a GitHub-hosted ubuntu-24.04 image (4 vCPU / 16 GB) for that run — exactly what runner = "ubuntu-24.04" in test_groups.toml would force — and it still died the same way:

2026-08-08T11:01:15.1810780Z ┌ Warning: `target_defined_modules::Bool` configuration is deprecated.
2026-08-08T11:15:19.5704413Z [2573] signal 15: Terminated
2026-08-08T11:15:19.5744237Z ##[error]The runner has received a shutdown signal. ...
2026-08-08T11:15:21.6505139Z ##[error]The operation was canceled.

i.e. the runner VM dies ~14 minutes after JET starts. Same shape in the previous master run (job 93078765474: JET warning 08:59:37/09:00:35, shutdown 09:15:08). It is not a job timeout-minutes — the effective timeout is the 120-minute test_groups.toml default and the two jobs died at 67 and 69 minutes.

2. Measured cost: the group peaks at ~32 GiB RSS

GROUP=QA julia +1.12 --project=. -e 'using Pkg; Pkg.test()' at 731eff66, Julia 1.12.6, JULIA_NUM_THREADS=1, under /usr/bin/time -v:

	Command being timed: "julia +1.12 --project=. -e using Pkg; Pkg.test()"
	User time (seconds): 13658.22
	System time (seconds): 994.76
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:44:07
	Maximum resident set size (kbytes): 33837408
	Exit status: 1

33,837,408 kB ≈ 32.3 GiB peak RSS. No 16 GB runner — GitHub-hosted or the self-hosted-4vcpu-16gb-* pool named in the issue body — can host that, which is why the VM (not just the Julia process) goes away.

Sampling /proc/<pid>/status every 5 s across the process tree puts the single-process peak at 33,738,772 kB, i.e. essentially all of it is one Julia process (the JET step), not the precompile fan-out.

I also tried the obvious cheap lever, JULIA_HEAP_SIZE_HINT=8G on an otherwise identical run: it cuts the peak a lot but does not get under 16 GB — the same sampler shows that process past 18,462,116 kB (17.6 GiB) and still climbing while in the JET step, and the run is materially slower (past 1h38m at the point the baseline was finishing at 1h44m). So a heap-size hint is a real lever but 8G is not a fix on its own.

3. Even with unlimited memory, QA is red at 731eff66 — with findings CI has never reached

The local run got all the way through and failed with real assertions. JET itself passes; the failures are downstream of it, which is precisely the part CI has never executed:

Test Summary:                                  | Pass  Fail  Error  Total      Time
QA                                             |   21     2      2     25  31m34.0s
  Extensions loaded                            |    4                   4      0.0s
  Quality Assurance                            |   17     2      2     21  30m29.9s
    Method ambiguity                           |    1                   1     14.6s
    Unbound type parameters                    |    1                   1      0.1s
    Undefined exports                          |    1                   1      0.0s
    Compare Project.toml and test/Project.toml |    1                   1      0.0s
    Stale dependencies                         |    1                   1     14.0s
    Compat bounds                              |    4                   4      0.5s
    Piracy                                     |    1                   1      0.3s
    Persistent tasks                           |    1                   1     26.4s
    ExplicitImports                            |    4            2      6     50.9s
      no_implicit_imports                      |                 1      1     35.9s
      no_stale_explicit_imports                |                 1      1      5.1s
      all_explicit_imports_via_owners          |    1                   1      1.7s
      all_qualified_accesses_via_owners        |    1                   1      4.8s
      all_qualified_accesses_are_public        |    1                   1      1.3s
      all_explicit_imports_are_public          |    1                   1      2.0s
    Public API documentation                   |    1     1             2      2.8s
      public API has docstrings                |          1             1      2.5s
      public API is rendered in docs           |    1                   1      0.0s
    No unapproved public reexports             |    1                   1      0.0s
ERROR: LoadError: Some tests did not pass: 21 passed, 2 failed, 2 errored, 0 broken.

The two errors are the same exception:

no_implicit_imports: Error During Test at SciMLTesting.jl:925
  Test threw exception
  Expression: check(pkg; _explicit_imports_kwargs(pkg, name, ei_kwargs)...) === nothing
  UnanalyzableModuleException
  Module `ModelingToolkit` was found to be unanalyzable. Include this module in the
  `allow_unanalyzable` keyword argument to allow it to be unanalyzable.
   [1] check_no_implicit_imports(mod::Module, file::String; ...)
     @ ExplicitImports ~/.julia/packages/ExplicitImports/sATsO/src/checks.jl:257

(identically for no_stale_explicit_imports), and the failure is one undocumented public name:

public API has docstrings: Test Failed at SciMLTesting.jl:1231
  Expression: isempty(undocumented)
   Evaluated: isempty([:StructuralTransformations])

Tool versions match the CI job exactly: Aqua v0.8.16, ExplicitImports v1.15.0, JET v0.11.6 (SciMLTesting v2.6.3 locally vs v2.6.2 in that CI run).

Consequence for the fix

Two independent problems are stacked here, and fixing the runner alone will just move the red from "runner died" to "4 QA assertions fail":

1. Memory. ~32 GiB peak is not survivable on any 16 GB runner, and JULIA_HEAP_SIZE_HINT=8G does not get it under the line either. Options are a larger runner, a much tighter heap hint (which grouped-tests.yml/tests.yml do not currently expose as an input, so it would be a SciML/.github change), or narrowing what JET analyses — today it is JET.test_package(ModelingToolkit; target_defined_modules = true) over the whole package, and the same run also logs that target_defined_modules::Bool is deprecated in favour of target_modules.
2. Assertions. no_implicit_imports / no_stale_explicit_imports currently throw UnanalyzableModuleException on ModelingToolkit, and :StructuralTransformations is public without a docstring.

I have not pushed anything for either; the memory lever lives in SciML/.github, and (2) wants a decision on whether ModelingToolkit goes on allow_unanalyzable or the module is made analyzable.

[ChrisRackauckas-Claude]

Correction / update on the JULIA_HEAP_SIZE_HINT=8G data point above — I posted it while that run was still in its JET step, and "cuts the peak a lot" was too generous. Continuing to sample the same process:

elapsed in JET step	single-process peak RSS
(as posted above)	18,462,116 kB
+9 min	20,525,040 kB
+18 min	23,456,976 kB
+27 min	26,992,056 kB

against a 33,738,772 kB peak for the unhinted baseline. So the hint delays the growth rather than bounding it — JET's live set genuinely reaches tens of GB and a heap-size hint is not the lever. Scratch that option; the real choices are a bigger runner or narrowing/removing the whole-package JET.test_package(ModelingToolkit; target_defined_modules = true) analysis.

The two ExplicitImports errors and the :StructuralTransformations docstring failure in the comment above are unaffected by any of this — they are what QA fails on once it is given enough memory to get past JET.