fix(test): mlxcel-core CUDA test binary crashes at the default thread count

[inureyes]

cargo test -p mlxcel-core --release --features cuda --lib crashes the test binary on a CUDA host when run with its default thread count. The same suite passes under --test-threads=1.

Evidence

Tesla V100-PCIE-32GB (sm_70), CUDA 12.9.41, 16 cores, MLX_CUDA_ARCHITECTURES=70.

Run	Mode	Result
main at 41844b5b	--test-threads=1	1678 passed, 1 failed (#1563), no crash
main at 41844b5b	default parallel	SIGFPE after 746 tests
main at 41844b5b	default parallel	SIGFPE after 791 tests
main at 41844b5b, epic #1536 test modules excluded	default parallel	SIGABRT after 732 tests
092d3dd0, before epic #1536	default parallel	SIGFPE after 734 tests

The signal and the crash point both vary between runs, and no individual test is implicated: the binary dies mid-suite rather than a test failing.

It is pre-existing

Verified by building and running 092d3dd0, the commit main sat on before epic #1536 began, in an isolated target directory. It crashes the same way. The epic's six merged units (#1537, #1538, #1539, #1541, #1544, #1545) did not introduce this, and excluding all five test modules they added does not avoid it either, which additionally rules out the new tests as the trigger.

Why it has not been noticed

The CUDA test suite has never run in CI. The CUDA runners carry CUDA 13, which removed Volta support, and the cuda-sm70-compile gate added in #1556 is compile-only and skips on those runners (see the CUDA architecture selection section of docs/installation.md). The other CUDA jobs do not run this suite. Sub-agent workflows in this repository are also directed to narrow test scope because a full cargo test --lib exceeds the 600 second stream-idle watchdog, so the default invocation is rarely exercised by automation either.

Scope

1. Identify what is racing. SIGFPE points at an integer division, SIGABRT at a C++ abort or uncaught exception; both arriving from the same workload suggests shared mutable state reached from several test threads rather than two distinct bugs. MLX's CUDA allocator, the JIT module cache, and device/context initialization are the candidates worth checking first.
2. Establish whether this is Volta-specific or general. Nothing here suggests an architecture dependency, but no sm_80-or-later host has been tried.
3. Decide the fix: make the shared state thread-safe, or serialize the CUDA-touching tests explicitly rather than relying on whoever runs the suite to pass --test-threads=1.

Acceptance criteria

• The racing state is identified, with the mechanism for at least one of the two signals.
• cargo test -p mlxcel-core --release --features cuda --lib completes without crashing at the default thread count, repeatedly.
• A statement of whether sm_80 and later are affected, or an explicit note that it could not be tested and why.
• If the resolution is to serialize rather than to fix the race, that constraint is enforced in the test harness rather than left to the caller.

Related

• fix(inference): fused_sample_probs differs by 1 ULP at temperature 1.0 on sm_70 #1563, a genuine 1 ULP assertion failure in the same suite, is the only test-level failure and is unrelated to this crash.
• fix(cuda/quant): quantized prefill is not bitwise reproducible on sm_70, and its determinism test never runs #1558, also found by widening test scope, is the third defect in this class.