build: add MLXCEL_CXX_MARCH override and pin the x86_64 CUDA release asset to x86-64-v3 (#208)

build.rs reads MLXCEL_CXX_MARCH in release builds: unset keeps -march=native (default unchanged), `none` omits the flag, and any other value is passed as -march=<value> via flag_if_supported, so a value the host compiler rejects is dropped rather than failing the build.

The existing build-linux-x86_64-cuda release job now sets MLXCEL_CXX_MARCH=x86-64-v3 and RUSTFLAGS=-C target-cpu=x86-64-v3, pinning the C++ bridge and rustc to a portable AVX2 (~2013+) baseline. Without this the redistributable asset inherited the build host's -march=native ISA (possibly AVX-512) and could SIGILL on older customer CPUs. Local and per-machine GB10/GH200 builds keep native.

Docs: MLXCEL_CXX_MARCH reference in environment-variables.md and installation.md, plus troubleshooting for the qmm Error 137 OOM and the liblapacke-dev (lapacke.h) build requirement.

Co-authored-by: Joongi Kim <joongi@lablup.com>

Author: achimnol

.github/workflows/release.yml

@@ -655,6 +655,14 @@ jobs:
       - name: Build release binaries
         run: cargo build --release --features cuda --locked
         # MLX_CUDA_ARCHITECTURES is inherited from the job env above.
+        env:
+          # Redistributable asset: pin the C++ bridge and Rust code to the
+          # portable x86-64-v3 baseline (AVX2, ~2013+) instead of inheriting
+          # this build host's native ISA. build.rs defaults the bridge to
+          # -march=native, which on a newer runner could bake in AVX-512 and
+          # SIGILL on older customer CPUs. Set MLXCEL_CXX_MARCH=none to omit.
+          MLXCEL_CXX_MARCH: x86-64-v3
+          RUSTFLAGS: "-C target-cpu=x86-64-v3"
 
       - name: Smoke test (10-token generate on GPU)
         run: |

docs/environment-variables.md

@@ -65,6 +65,7 @@ These are read by the `mlxcel-core` build script.
 |----------|--------|---------|-------|
 | `MLXCEL_BUILD_METAL` | `1/0`, `on/off`, `true/false`, `yes/no` | `on` on macOS | Overrides the CMake `MLX_BUILD_METAL` setting for local builds. Invalid values fail the build. |
 | `MLXCEL_BUILD_ACCELERATE` | `1/0`, `on/off`, `true/false`, `yes/no` | `on` on macOS | Overrides the CMake `MLX_BUILD_ACCELERATE` setting for local builds. Invalid values fail the build. |
+| `MLXCEL_CXX_MARCH` | a `-march=` value, or `none` | `native` | ISA baseline for the C++ bridge in release builds. Set a portable baseline (e.g. `x86-64-v3`) for binaries that run on machines other than the build host; `none` omits the flag. See [Installation](installation.md#c-isa-baseline-mlxcel_cxx_march). |
 
 CUDA builds also use non-`MLXCEL_*` variables such as `CUDA_HOME` and
 `MLX_CUDA_ARCHITECTURES`; see [Installation](installation.md#linux-with-cuda).

docs/installation.md

@@ -145,12 +145,30 @@ system temp dir), so only the first run of each kernel variant pays the NVRTC
 cost. Point `MLX_PTX_CACHE_DIR` at a persistent path to keep the cache across
 sessions.
 
+### C++ ISA baseline (`MLXCEL_CXX_MARCH`)
+
+In release builds the C++ bridge defaults to `-march=native`, which tunes for
+(and only runs on) the build host's CPU. That is correct for builds that run
+where they are built (developer machines, the per-machine GB10/GH200 release
+assets). For a binary that must run on other machines, set `MLXCEL_CXX_MARCH`
+to a portable baseline; the release workflow's x86-64 assets use `x86-64-v3`
+(AVX2):
+
+```bash
+# Portable x86-64 build (any AVX2-capable CPU, ~2013+).
+MLXCEL_CXX_MARCH=x86-64-v3 cargo build --release --features cuda
+
+# Omit -march entirely (compiler default baseline).
+MLXCEL_CXX_MARCH=none cargo build --release --features cuda
+```
+
 ## Runtime environment variables
 
 | Variable | Description | Default |
 |----------|-------------|---------|
 | `CUDA_HOME` | CUDA toolkit root, build-time and for runtime NVRTC headers | `/usr/local/cuda` when present |
 | `MLX_CUDA_ARCHITECTURES` | CUDA SM target list, build-time | auto-detect via `nvidia-smi`, then `90a` fallback |
+| `MLXCEL_CXX_MARCH` | C++ bridge `-march` value, build-time; `none` omits the flag | `native` |
 | `MLXCEL_CCCL_DIR` | Override for the bundled CCCL (libcu++) header dir used by the CUDA NVRTC JIT | bundled `<exe-dir>/../include/cccl`, then build-time fallback |
 | `MLX_PTX_CACHE_DIR` | On-disk cache for JIT-compiled CUDA kernels | system temp dir |
 | `MLXCEL_QUIET_JIT` | Suppress the one-time "compiling CUDA kernels" notice on a cold first run | unset (notice shown) |
@@ -190,3 +208,13 @@ explicitly.
 MLX version are installed and discoverable by the linker. The root build script
 links CUDA runtime/math libraries directly and relies on the system driver for
 `libcuda`.
+
+**`gmake: *** Error 137` (SIGKILL) while compiling `qmm_*.cu`** — the build ran
+out of memory. The CUTLASS-heavy quantized-matmul kernels peak at ~4-5 GB of
+compiler memory per parallel job, so a default `-j$(nproc)` build needs roughly
+`5 GB × cores`. Cap the parallelism with `cargo build -j N ...` (cargo forwards
+`N` to the CMake subbuild); pick `N ≈ available_RAM_GB / 5`.
+
+**CMake error: `LAPACK_INCLUDE_DIRS ... NOTFOUND`** — install `liblapacke-dev`
+(MLX needs `lapacke.h`, which `liblapack-dev` alone does not provide) and
+`libopenblas-dev`.

src/lib/mlxcel-core/build.rs

@@ -72,8 +72,24 @@ fn main() {
         bridge
             .flag_if_supported("-O3")
             .flag_if_supported("-DNDEBUG")
-            .flag_if_supported("-ffast-math")
-            .flag_if_supported("-march=native");
+            .flag_if_supported("-ffast-math");
+        // ISA baseline for the bridge C++. Defaults to the build host's ISA
+        // (-march=native), which is correct for builds that run where they
+        // are built (developer machines, the per-machine gb10/gh200 release
+        // assets). Redistributable release builds must override this with a
+        // portable baseline via MLXCEL_CXX_MARCH (e.g. "x86-64-v3" for the
+        // generic Linux x86-64 asset), otherwise the binary inherits the
+        // build runner's ISA (possibly AVX-512) and SIGILLs on older CPUs.
+        // Set MLXCEL_CXX_MARCH=none to omit the flag entirely.
+        match env::var("MLXCEL_CXX_MARCH").as_deref() {
+            Err(_) => {
+                bridge.flag_if_supported("-march=native");
+            }
+            Ok("none") => {}
+            Ok(march) => {
+                bridge.flag_if_supported(&format!("-march={march}"));
+            }
+        }
         // On macOS, Clang produces LLVM bitcode with -flto, which is compatible
         // with Rust's LLVM LTO. On Linux with GCC, -flto produces GIMPLE IR
         // objects that are incompatible, causing undefined-reference linker errors.
@@ -155,6 +171,7 @@ fn main() {
     println!("cargo:rerun-if-env-changed=MLX_CUDA_ARCHITECTURES");
     println!("cargo:rerun-if-env-changed=MLXCEL_BUILD_METAL");
     println!("cargo:rerun-if-env-changed=MLXCEL_BUILD_ACCELERATE");
+    println!("cargo:rerun-if-env-changed=MLXCEL_CXX_MARCH");
 }
 
 /// Expected MLX git commit — must match GIT_TAG in mlx-cpp/CMakeLists.txt.