Flaky CI: segfault in HiGHS (Highs_run) during test suite on Julia 1.12

[vtjeng]

The Julia 1.12 / ubuntu-latest / x64 CI job crashed with a segfault (signal 11) inside the HiGHS C library while running the package test suite.

• Job: https://github.com/vtjeng/MIPVerify.jl/actions/runs/29267847943/job/86877738025
• Commit: d4b9084 (PR Instrument verification runs with formulation and bound-tightening statistics #214)
• Stack: Highs_run (libhighs.jl:344) → _Highs_run_workaround_issue_316 (HiGHS.jl MOI_wrapper.jl:2253) → optimize!
• Versions: Julia 1.12.6, HiGHS.jl v1.24.1, HiGHS_jll v1.15.1+0
• The job runs with coverage enabled and --check-bounds=yes.

Evidence that this is intermittent rather than deterministic:

• The same dependency versions (HiGHS.jl v1.24.1) passed the full CI suite on master the same day (run 29233189449).
• The same test suite passed on Julia 1.12 macOS and Windows within the failing run.

If it recurs, check the HiGHS.jl issue tracker for matching reports before digging locally; a reproducible case would need the exact model that was being solved when the crash happened.

Occurrences

Date	Branch/PR	Job
2026-07-13	#214 (d4b9084)	Julia 1.12 - ubuntu-latest - x64
2026-07-14	master (ed9194c, merge of #214)	Julia 1.12 - windows-latest - x64
2026-07-16	#236 (681c44c)	Julia 1 - macos-latest - x64

Append new occurrences to this table.

Exit criteria for the temporary HiGHS pin

PR #220 pins the repository workspace to HiGHS_jll 1.14.x while the HiGHS 1.15.x failures are investigated. Keep this issue open until the pin has been removed and the unpinned configuration has completed the observation period below.

• A released HiGHS version includes ERGO-Code/HiGHS#3141, and compatible HiGHS.jl and HiGHS_jll releases are available.
• Run the Flaky CI: blur-(5,5) norm-1 test returns objective 9.32 instead of 6.96 on macOS #219 reproducer for 10 process invocations with the candidate release. Require no native crashes and require every full solve to finish OPTIMAL at 6.959316.
• Remove the direct HiGHS_jll dependency, the ~1.14 compatibility bound, and the temporary pin comment. Keep the SolveStatus == OPTIMAL assertion.
• Run the full local test suite and all current CI jobs.
• Run three paired full-suite timings with the pinned and candidate versions. Report each run, aggregate totals, the candidate/baseline ratio, and the one-machine limitation.
• Merge the unpin change, then leave this issue open for at least 14 days and until at least 30 package-test CI jobs have completed with the unpinned versions across Ubuntu, macOS, and Windows.
• During the observation period, record any native HiGHS crash or recurrence of the Flaky CI: blur-(5,5) norm-1 test returns objective 9.32 instead of 6.96 on macOS #219 objective mismatch, including SolveStatus. A matching failure resets the observation period and keeps this issue open.
• After the observation period, summarize the tested versions, date range, job count, and outcome here. Update Flaky CI: blur-(5,5) norm-1 test returns objective 9.32 instead of 6.96 on macOS #219 separately, then close this issue if the evidence supports it.

[vtjeng]

First data point after filing: the next CI run on the same PR (007eda8, same HiGHS.jl v1.24.1 / Julia 1.12.6) passed the Julia 1.12 / ubuntu job, so the segfault did not reproduce on the immediately following run.

[vtjeng]

Root cause found upstream: a use-after-free in HiGHS's process-global parallel scheduler, not anything specific to this test suite.

The mechanism, diagnosed in ERGO-Code/HiGHS#3136 after the same intermittent segfaults hit jump-dev/MultiObjectiveAlgorithms.jl#211:

• The C API's Highs_destroy called Highs::resetGlobalScheduler(true), which tears down a static scheduler shared by every Highs instance in the process.
• HiGHS.jl marks Highs_run as GC-safe (jl_gc_safe_enter in MOI_wrapper.jl), so Julia's GC can run while a solve is in progress. When it collects a dead HiGHS model, the finalizer calls Highs_destroy, which resets the global scheduler out from under the running solve.
• Per the upstream discussion, Julia test suites that spawn many HiGHS instances are the prime candidate. Ours creates a fresh optimizer per find_adversarial_example call plus per tightening solve, so dead instances are usually pending finalization while another solve runs.

Both occurrences so far match this signature: the crash is inside HighsMipSolver::run in code that reads the shared state — HighsProfiling::isSubMip on ubuntu (2026-07-13), HighsSplitDeque::WorkerBunk::publishWork on windows (2026-07-14, added to the table above). Both crashed in test/integration/sequential/generated_weights/conv+fc+softmax.jl, but that is likely just where the heaviest MIP solves are, since GC timing decides whether the race fires. I retitled the issue since it is not ubuntu-specific.

Fix: ERGO-Code/HiGHS#3141 ("Remove resetGlobalScheduler calls from C API"), merged 2026-07-14 — after the latest HiGHS release (v1.15.1, 2026-07-02), so no released HiGHS_jll contains it yet. HiGHS is a test-only dependency here with compat HiGHS = "1.4", so CI picks the fix up automatically once the next HiGHS release is tagged and HiGHS_jll / HiGHS.jl ship it. This issue can be closed once CI has run on a HiGHS_jll built from a release containing #3141.

Observed rate is low (2 crashes across roughly 120 test-suite jobs, 2026-07-13 to 07-16), so waiting seems fine. If it starts recurring frequently, an interim mitigation to try is "threads" => 1 in the HiGHS solver options in test/TestHelpers.jl, which should keep the MIP solve off the parallel scheduler.

[vtjeng]

Root cause identified: the parallel MIP solver machinery introduced in HiGHS 1.15 (jll 1.15.1+0). Full investigation written up in #219 (comment); #220 pins the test environment to HiGHS_jll 1.14.

Specific to this issue's crashes: the reproducible case the issue asks for exists. Locally (Linux x64, Julia 1.12.6, same jll artifact as CI), repeatedly re-solving the conv+fc+softmax blur MIP from the test suite crashed HiGHS within minutes in two separate processes:

• bus error in HighsSplitDeque::WorkerBunk::publishWork, reached from HighsMipSolverData::evaluateRootNode
• segfault in HighsPrimalHeuristics::RINS → solveSubMip → HighsProfiling::running, with threads=1, parallel=off set

This run's ubuntu stack is a third distinct site (HighsSeparation → HighsLpRelaxation → HighsProfiling::isSubMip). The named frames all involve symbols (HighsMipWorker, HighsProfiling) that do not exist in jll 1.14.0's libhighs.so. Related upstream reports: ERGO-Code/HiGHS#3150, ERGO-Code/HiGHS#2885.

[vtjeng]

Third occurrence, appended to the table: Julia 1 - macos-latest - x64 on PR #236 (681c44c), 2026-07-16 — first sighting on macOS, so all three CI OSes have now exhibited the crash.

Stack matches the third distinct site from the investigation comment above exactly: HighsProfiling::isSubMip ← HighsLpRelaxation::run ← resolveLp ← HighsSeparation::separationRound, reached from HighsMipSolver::run in libhighs.1.15.dylib, while solving a MIP from test/integration/sequential/generated_weights/conv+fc+softmax.jl (the same blur family as the local repro). #220's HiGHS_jll <1.15 pin remains unmerged, so 1.15.1 was in play. Rerunning the failed job once the run completes.