fix(server): stamp the speculative burst's first token at its prefill… #2726

Workflow file for this run

	# PR / push CI for lightweight quality gates. Runs cargo-deny (license /
	# advisory) and cargo-fmt on every touched-Rust change so formatting drift and
	# license issues are caught at PR time.
	#
	# The general unit suite is not gated at PR time, and clippy is gated only in
	# the narrow shape described below. Both used to run on the self-hosted Apple
	# Silicon runner, first here and then briefly in release.yml, and in both cases
	# consumed ~30 min per run, blocking either PRs or releases on a shared
	# resource for failures that `make verify` reliably catches on the developer's
	# machine in a fraction of the time (#21, #23).
	#
	# The `clippy` job below is deliberately not a return to that arrangement. It
	# runs `cargo clippy -p mlxcel --lib --tests -- -D warnings` at default
	# features on the self-hosted GB10 runner with its own persistent target
	# directory, so it neither touches the Apple Silicon runner nor pays a cold
	# build after its first run. It exists because #916 merged an `err_expect`
	# that reddened `make verify` for every contributor on every platform and sat
	# on `main` until the nightly backstop caught it a day later (#1283): the
	# per-crate lint that catches it is cheap, and only its absence at PR time was
	# expensive.
	#
	# They are not absent from every workflow, though, so do not read the above as
	# "nothing anywhere runs them":
	#
	# - pipeline-parallel-ci.yml runs `cargo clippy -p mlxcel --lib --tests
	# -- -D warnings`, and reaches `cargo test` through scripts/ci/run-pp-*.sh
	# with `distributed::`-prefixed selectors. It is debug profile on
	# ubuntu-latest and path-filtered to src/distributed/pipeline/** and its
	# siblings, so it never runs on a model-port PR and never selects a model
	# or CLI test module.
	# - nightly-verify.yml runs the full `make verify` (fmt + clippy + test) once
	# a day on the self-hosted Apple Silicon runner and files an issue when it
	# fails. It is a backstop against a red `main`, not a PR gate; see #939 for
	# the two deterministic failures that sat on `main` before it existed, and
	# for why it is nightly rather than per-PR.
	#
	# Quality gate still lives locally at PR time. Pre-push checklist:
	# make verify # fmt + clippy(workspace, metal,accelerate, -D warnings)
	# # + test(workspace, test-fast)
	# make verify-clean # same, after `cargo clean` — use when clippy's
	# # per-crate cache may be hiding a regression
	#
	# Note on CUDA gating: running the CUDA test suite stays exclusive to
	# release.yml because it requires a Linux self-hosted runner (currently only the
	# GB10 node used for release builds). Adding a PR-level CUDA test gate would
	# double runner cost for limited additional safety on PRs that don't touch
	# CUDA-specific code paths. Compiling CUDA is a different question and three
	# jobs below do it: `xla-compile` and `xla-link` at the shipped sm_121, and
	# `cuda-sm70-compile` at sm_70, each on a narrow path filter.

	name: CI

	on:
	push:
	branches: [main]
	pull_request:
	branches: [main]

	permissions:
	contents: read

	jobs:
	changes:
	name: Detect changes
	runs-on: ubuntu-latest
	permissions:
	contents: read
	pull-requests: read
	outputs:
	rust: ${{ steps.filter.outputs.rust }}
	mlx_pin: ${{ steps.filter.outputs.mlx_pin }}
	xla_link: ${{ steps.filter.outputs.xla_link }}
	cuda_arch: ${{ steps.filter.outputs.cuda_arch }}
	steps:
	- uses: actions/checkout@v7
	with:
	persist-credentials: false
	- uses: dorny/paths-filter@v4
	id: filter
	with:
	filters: \|
	rust:
	- '*/.rs'
	- '**/Cargo.toml'
	- 'Cargo.lock'
	- 'deny.toml'
	- 'build.rs'
	- 'scripts/iree/**'
	- '.github/workflows/ci.yml'
	mlx_pin:
	- 'src/lib/mlx-cpp/CMakeLists.txt'
	- 'src/lib/mlxcel-core/build_support/mlx_pin.rs'
	- 'src/lib/mlxcel-mlx-pin/**'
	- 'scripts/ci/mlx_pinned_commit.sh'
	- 'scripts/ci/mlx_pinned_commit_test.sh'
	- '.github/workflows/ci.yml'
	# Narrower than `rust` above: only the paths that can move the IREE
	# link recipe itself, so the release link job below does not run on
	# every Rust PR. See the `xla-link` job comment for the rationale.
	xla_link:
	- 'build.rs'
	- 'src/lib/mlxcel-xla/build.rs'
	- 'src/lib/mlxcel-xla/csrc/**'
	- 'scripts/iree/**'
	- 'rust-toolchain.toml'
	- '.github/workflows/ci.yml'
	# Everything that can compile differently per CUDA architecture: the
	# MLX overlays and turbo kernels, the pinned MLX commit, and the
	# build scripts that hand CMake the architecture list. Rust sources
	# outside these paths cannot break an arch-conditional compile, so
	# the sm_70 gate below does not run on every Rust PR. See the
	# `cuda-sm70-compile` job comment for why sm_70 specifically.
	cuda_arch:
	- 'src/lib/mlx-cpp/**'
	- 'src/lib/mlxcel-core/build.rs'
	- 'src/lib/mlxcel-core/build_support/**'
	- 'build.rs'
	- '.github/workflows/ci.yml'

	deny:
	name: cargo-deny
	needs: changes
	if: needs.changes.outputs.rust == 'true'
	runs-on: ubuntu-latest
	permissions:
	contents: read
	steps:
	- uses: actions/checkout@v7
	with:
	persist-credentials: false
	- uses: EmbarkStudios/cargo-deny-action@v2
	with:
	command: check

	fmt:
	name: cargo-fmt
	needs: changes
	if: needs.changes.outputs.rust == 'true'
	runs-on: ubuntu-latest
	permissions:
	contents: read
	steps:
	- uses: actions/checkout@v7
	with:
	persist-credentials: false
	# The tag on this action IS the toolchain version it installs, so it must
	# track `rust-toolchain.toml`'s `channel` rather than be bumped on its own.
	# Dependabot reads the tag as an ordinary version and will try to raise it;
	# `.github/dependabot.yml` ignores this action for that reason.
	- uses: dtolnay/rust-toolchain@1.97.1
	with:
	components: rustfmt
	- run: cargo fmt --all -- --check

	clippy:
	name: cargo-clippy
	needs: changes
	# On the repository guard, because it is easy to read more into it than it
	# does. It keeps this job from queueing forever in a fork of this
	# repository, which has no GB10 runner of its own. It does not stop a pull
	# request opened from a fork against lablup/mlxcel: on that event
	# `github.repository` is the base repository, so the guard is true and the
	# job runs here, executing the PR's `build.rs` and `scripts/iree/**` as the
	# runner's own user. What actually gates that case is the repository's
	# Actions fork-PR approval policy, currently `first_time_contributors`.
	# The same reading applies to `xla-compile` and `xla-link` below.
	if: github.repository == 'lablup/mlxcel' && needs.changes.outputs.rust == 'true'
	runs-on: GB10
	permissions:
	contents: read
	timeout-minutes: 60
	steps:
	- uses: actions/checkout@v7
	with:
	persist-credentials: false

	- name: Use a persistent target directory
	run: \|
	# Clippy emits different artifacts than a build, so it gets its own
	# directory rather than sharing (and repeatedly invalidating) the
	# xla-compile or release caches. Cold on first run, warm after.
	CARGO_TARGET="$HOME/.cargo-target/mlxcel-clippy-ci"
	mkdir -p "$CARGO_TARGET"
	echo "CARGO_TARGET_DIR=$CARGO_TARGET" >> $GITHUB_ENV

	# Default features (`surgery`, no accelerator). The lint that motivated
	# this job (#1283) reproduces on every feature set, so the cheapest one
	# that compiles the crate is enough, and it is the same command
	# pipeline-parallel-ci.yml already runs on its own path filter.
	#
	# On the runner choice, measured rather than assumed, so the question does
	# not get re-litigated from scratch. This job is here and not on
	# `ubuntu-latest` because `src/lib/mlxcel-core/build.rs` builds MLX through
	# cmake unconditionally: the accelerator features pick a backend, they do
	# not decide whether the C++ builds. A GitHub-hosted runner would pay that
	# cold on every run, since the one `ubuntu-latest` job this repository
	# already has (`pipeline-parallel-ci.yml`) caches the cargo registry and
	# not `target/`, and it takes about 28 minutes. Here the same command is
	# 2m44s cold and 27 to 40 seconds warm off the persistent target directory.
	#
	# The cost of staying on GB10 is that this runner also serves
	# `xla-compile` and the release build, so a release in flight queues this
	# job behind it and delays a merge. That happened once on 2026-08-22
	# (#1301). It is a delay and not a failure: a queued job consumes nothing,
	# and the release was not slowed by it. Trading a rare queue wait for
	# roughly 20 minutes on every PR would re-create the cost objection that
	# removed clippy from PR-time CI in the first place (#21, #23).
	- name: Clippy
	run: \|
	cargo clippy \
	-p mlxcel \
	--lib \
	--tests \
	-- -D warnings

	crate-versions:
	name: crate versions
	# Deliberately not behind the `changes` filter. It needs no toolchain and
	# runs in seconds, and a version-consistency gate that can be skipped is
	# exactly the gate that gets skipped on the one PR that breaks it.
	runs-on: ubuntu-latest
	permissions:
	contents: read
	steps:
	- uses: actions/checkout@v7
	with:
	persist-credentials: false
	# Gating, not advisory. From v0.3.3 onward the root `mlxcel` and every
	# member crate that ships with it carry the same `[package] version`, and
	# a release has to bump several manifests at once. The list used to live
	# in prose (CLAUDE.md, the release skill) and went stale the moment a
	# member was added: `mlxcel-mlx-pin` tracked the root version from the day
	# it was created without either document saying so. The script inverts the
	# rule, so a new member fails this job until it is either bumped with the
	# root or declared independent with a reason. No toolchain needed.
	- name: Check every version-tracking crate carries the root version
	run: python3 scripts/ci/check_crate_versions.py

	kernel-dtype-keys:
	name: kernel dtype keys
	# Deliberately not behind the `changes` filter, for the same reason as
	# `crate-versions`: it needs no toolchain, runs in seconds, and the defect it
	# guards against is silent. MLX's CUDA JIT memoises a compiled kernel under a
	# name that does not carry the input dtypes, so a launch whose template args
	# are all ints serves the first dtype's compiled module to every later dtype
	# at the same geometry, reading buffers through the wrong pointer type.
	# Issues #1053 and #1054 are two symptoms of exactly that. macOS never sees
	# it, because Metal's key does carry the dtypes, so this job is the only
	# place the omission is caught without CUDA hardware.
	runs-on: ubuntu-latest
	permissions:
	contents: read
	steps:
	- uses: actions/checkout@v7
	with:
	persist-credentials: false
	- name: Check every CUDA JIT launch keys its cache on the input dtypes
	run: python3 scripts/ci/check_kernel_dtype_keys.py

	llama-compat-manifest:
	name: llama-compat manifest
	# Deliberately not behind the `changes` filter, like `crate-versions`: it
	# needs no toolchain and runs in seconds. Validates the checked-in
	# llama-server b10621 compatibility manifest (compat/llama-server/b10621/,
	# issue #1443, epic #1431): pinned counts (249 help entries, 323 long
	# spellings, 134 LLAMA_* env vars), one policy state per entry (five
	# states since #1499: supported / aliased / not_applicable / deferred /
	# by_design), issue links on deferred entries, test ids on non-supported
	# entries, the entry key allowlist, the rule that a non-empty
	# `divergence` forbids `supported`, the by_design obligations (non-empty
	# divergence, notes, a resolving test pointer, and a well-formed
	# `rationale` whose policy kind must state its revisit condition, with
	# `rationale` null on every other state), and canonical serialization.
	# --check-issues-open additionally asserts every issue a deferred entry
	# links is still open, so closing a chain issue without flipping its
	# manifest entries fails here; by_design entries never feed that check,
	# because a permanent, tested divergence has no issue left to keep open. The second step
	# is the validator's own negative coverage: it mutates a throwaway copy of
	# the manifest and asserts the gate rejects it. The binary-facing half of the
	# gate (option spellings, env bindings, defaults against both server
	# binaries, hidden arguments included, plus mounted routes and native
	# request fields) runs inside the workspace test suite as
	# tests/llama_compat_manifest.rs and src/server/llama_compat_tests.rs.
	runs-on: ubuntu-latest
	permissions:
	contents: read
	issues: read
	env:
	GH_TOKEN: ${{ github.token }}
	steps:
	- uses: actions/checkout@v7
	with:
	persist-credentials: false
	- name: Validate the b10621 compatibility manifest
	run: python3 scripts/ci/check_llama_compat_manifest.py --check-issues-open
	- name: Validator negative coverage
	run: bash scripts/ci/check_llama_compat_manifest_test.sh

	mlx-pin:
	name: MLX pin extraction
	needs: changes
	if: needs.changes.outputs.mlx_pin == 'true'
	runs-on: ubuntu-latest
	permissions:
	contents: read
	steps:
	- uses: actions/checkout@v7
	with:
	persist-credentials: false
	# The pinned MLX commit has one home, GIT_TAG in
	# src/lib/mlx-cpp/CMakeLists.txt, and two independent parsers read it:
	# mlxcel-core's build.rs in Rust (build_support/mlx_pin.rs, unit-tested by
	# mlxcel-mlx-pin) and this script in awk, for release.yml's "Validate MLX
	# build cache" steps. The shell half accepts strictly less than the Rust
	# half: it needs GIT_TAG as the first token of its own line, after the
	# GIT_REPOSITORY line, with no closing parenthesis in between. A CMake
	# reformat that leaves every local build and `make verify` green can
	# therefore break every release build, and release.yml is the only other
	# place this script runs, so without this job the breakage would first
	# appear when a release is cut. Issue #1047.
	#
	# `mlxcel-mlx-pin` compiles in under a second (a leaf crate with no
	# production role, only `std` plus a `tempfile` dev-dependency), so it
	# runs first here rather than only in nightly-verify.yml's `make verify`.
	# Its `tests/cross_parser.rs` shells out to mlx_pinned_commit.sh itself
	# and asserts that whenever the shell parser accepts an input, the Rust
	# parser resolves to the identical commit for that same input: the two
	# unit-test files below each cover one parser in isolation and cannot
	# catch a same-input value mismatch on their own.
	- uses: dtolnay/rust-toolchain@1.97.1
	- name: Run the Rust pin parser's unit and cross-parser tests
	run: cargo test -p mlxcel-mlx-pin

	- name: Extract the pinned MLX commit
	run: scripts/ci/mlx_pinned_commit.sh

	# The other half of the contract: the shell parser must also reject
	# everything the Rust parser rejects. A value it accepts and the Rust
	# parser refuses is not harmless, because release.yml purges every MLX
	# build cache that does not match what this script prints and the build
	# then fails on the same file the purge was justified by.
	- name: Check the pinned MLX commit parser rejects malformed input
	run: scripts/ci/mlx_pinned_commit_test.sh

	cross-repo-refs:
	name: cross-repo refs
	if: github.event_name == 'pull_request'
	runs-on: ubuntu-latest
	permissions:
	contents: read
	steps:
	- uses: actions/checkout@v7
	with:
	persist-credentials: false
	fetch-depth: 0
	# Advisory: lists bare 3+-digit '#NNN' added on this PR so unqualified
	# upstream refs (-> org/repo#NNN) and any leaked private-repo numbers are
	# caught in review. Does not fail the build (a genuine lablup/mlxcel #N is
	# fine); run locally with STRICT=1 to gate. See CONTRIBUTING.md.
	- name: Flag unqualified cross-repository references
	# Pass the base ref via env (not interpolated into the shell) so a branch
	# name can never reach shell parsing; the script reads BASE_REF as input.
	env:
	BASE_REF: origin/${{ github.base_ref }}
	GITHUB_TOKEN: ${{ github.event.pull_request.head.repo.full_name == github.repository && github.token \|\| '' }}
	run: \|
	bash scripts/ci/check_cross_repo_refs_test.sh
	python3 scripts/ci/check_cross_repo_refs.py

	# ============================================================================
	# OpenXLA feature compile (self-hosted GB10)
	# ============================================================================
	# Nothing else in CI compiles any XLA feature. `deny`, `fmt` and the other
	# jobs here never build the crate, `pipeline-parallel-ci.yml` runs clippy on
	# default features, and `nightly-verify.yml` uses `metal,accelerate`. Two
	# defects reached `main` through that gap: a non-exhaustive `match` in the
	# OpenXLA serve worker after a new `ModelRequest` variant landed, and a dead
	# re-export that `-D warnings` would have rejected. Both were found by hand
	# while rebasing an unrelated branch.
	#
	# Why this runs on the self-hosted GB10 runner rather than a hosted one:
	# `xla-iree` links a C shim against a prebuilt IREE runtime, which
	# `scripts/iree/setup-cuda.sh` provisions by building the runtime from source
	# against a pinned revision. That is far too slow to do per PR from scratch,
	# but the GB10 runner already holds the build under
	# `~/.cache/mlxcel/iree-cuda-<version>` and the script is idempotent, so a
	# warm runner reuses it and only a fresh one pays the one-time cost.
	# `xla-diagnostics` additionally implies `cuda`, which points at the same
	# runner.
	#
	# Deliberately NOT covered, so the gap is recorded rather than assumed shut:
	# - Linking. `cargo check` never invokes the linker, so a regression in the
	# IREE link recipe in `build.rs` (see #1274/#1275) is invisible here. The
	# `xla-link` job immediately below closes that gap for the CUDA/IREE
	# recipe; see its comment for what it covers and what it still does not.
	# - `cargo test`. This is a compile gate; running the XLA suites needs a
	# GPU that is not contended with development work on the same host.
	# - Clippy-only lints. This job runs `cargo check`, and `RUSTFLAGS: "-D
	# warnings"` denies rustc lints only, so `dead_code` and `unused_imports`
	# are now gated but `collapsible_if`, `needless_match`,
	# `too_many_arguments`, `while_let_loop` and the rest of clippy's own
	# lints are not gated by anything in CI under these feature sets. The
	# `clippy` job above covers default features only. That half of the
	# backlog cleared by #1304 can therefore regrow silently; closing it means
	# running clippy here, which was left out of #1304 deliberately.
	# - The path dependencies' own test targets. There is no `default-members`
	# (see the note at the top of `Cargo.toml`), so `cargo check` run here
	# resolves to `-p mlxcel` and `--all-targets` expands within that package
	# only. `mlxcel-core`, `mlxcel-surgery` and `mlxcel-xla` are compiled as
	# plain library dependencies, so their `#[cfg(test)]` modules are never
	# built. That leaves `mlxcel-xla`'s unit tests ungated by every job,
	# because the `clippy` job above builds default features and the crate is
	# default-off there. `cargo clippy -p mlxcel-xla --all-targets` covers
	# them, but only by hand.
	# - macOS and `IREE_DIST` builds of the same features. Neither has a runner
	# with the required distribution.
	xla-compile:
	name: OpenXLA feature compile
	needs: changes
	if: github.repository == 'lablup/mlxcel' && needs.changes.outputs.rust == 'true'
	runs-on: GB10
	permissions:
	contents: read
	timeout-minutes: 120
	env:
	# The value shipped for this runner's architecture. Auto-detection yields
	# `121a`, which is numerically identical here but is not what releases
	# build, so the gate compiles what ships.
	MLX_CUDA_ARCHITECTURES: "121"
	# Every rustc lint, not just the `unused_imports` this job started with.
	# The narrower value existed because the XLA feature combination carried a
	# dead-code backlog that would have made the job red on arrival; #1304
	# cleared it. See the note above for what `-D warnings` on `cargo check`
	# still does not cover.
	#
	# Blast radius, deliberately: `RUSTFLAGS` reaches every unit cargo builds
	# from a path, so the `mlxcel-core`, `mlxcel-surgery` and `mlxcel-xla`
	# library units and the build scripts are gated here too, and a warning in
	# any of them reds a job named for OpenXLA. Registry and git dependencies
	# cannot, because cargo compiles those with `--cap-lints allow`. The
	# toolchain is pinned exactly in `rust-toolchain.toml`, so a new rustc
	# release cannot turn this red on its own; bumping that pin is what has to
	# re-clear these two feature sets.
	RUSTFLAGS: "-D warnings"
	steps:
	- uses: actions/checkout@v7
	with:
	persist-credentials: false

	- name: Use a persistent target directory
	run: \|
	# Separate from the release job's directory so a CI check cannot
	# invalidate a release build's cache, or be slowed by it.
	CARGO_TARGET="$HOME/.cargo-target/mlxcel-xla-ci"
	mkdir -p "$CARGO_TARGET"
	echo "CARGO_TARGET_DIR=$CARGO_TARGET" >> $GITHUB_ENV

	- name: Provision the IREE runtime
	run: \|
	# Idempotent: reuses `~/.cache/mlxcel/iree-cuda-<version>` when the
	# runtime is already built, and builds it once when it is not.
	bash scripts/iree/setup-cuda.sh
	# The script emits shell `export VAR=value` lines; GITHUB_ENV wants
	# bare `VAR=value`, and without stripping the prefix the variables
	# would be named "export IREE_CUDA_HOME" and build.rs would abort
	# claiming no IREE distribution is configured.
	bash scripts/iree/setup-cuda.sh --env \| sed 's/^export //' >> "$GITHUB_ENV"

	- name: Compile the production XLA feature set
	run: cargo check --features cuda,xla-iree --all-targets

	- name: Compile the diagnostics XLA feature set
	run: cargo check --no-default-features --features xla-diagnostics --all-targets

	# ============================================================================
	# OpenXLA feature link (self-hosted GB10)
	# ============================================================================
	# `xla-compile` above never invokes the linker, so it cannot catch a
	# regression in the IREE link recipe in `build.rs`. That is not hypothetical:
	# #1274 was exactly a link failure, and `cargo check --features
	# cuda,xla-iree --all-targets` passed cleanly on the same tree that could not
	# link a single integration test. It reached `main` and was found by hand
	# while validating an unrelated PR. The fix, #1275, appends a second `-lc`
	# after the IREE archives in `build.rs`, because rustc emits its own `-lc`
	# before anything `cargo:rustc-link-arg` can append. This job is a direct
	# reproducer of that failure class.
	#
	# What it links, and why: `cargo test --release --features cuda,xla-iree
	# --test xla_prepared_prefill --no-run`. `build.rs` emits the IREE recipe
	# through `cargo:rustc-link-arg`, which applies to every linked artifact of
	# the crate, so any linked target exercises the same recipe `mlxcel-server`
	# does, and an integration test is the direct reproducer because #1274 was an
	# integration-test link failure. Cargo also builds the package's `[[bin]]`
	# targets whenever an integration test is selected, so `mlxcel-server` and
	# the other three binaries are linked here as well; the costs measured below
	# already include them, which makes `cargo build --release --features
	# cuda,xla-iree --bin mlxcel-server` a strict subset of this command rather
	# than a cheaper alternative to it. `--no-run` links without executing, so
	# this job never touches the GPU and never contends with development work on
	# the shared GB10 host.
	#
	# Profile: `--release` is mandatory, not a choice. On this host the debug
	# profile cannot link these targets at all: it fails with hundreds of
	# `relocation truncated to fit: R_AARCH64_CALL26` errors against ordinary
	# `libstd`/`compiler_builtins` symbols, because the unoptimized binary
	# exceeds the AArch64 direct-branch range.
	#
	# Trigger: path-filtered via the `changes` job's `xla_link` output, not every
	# Rust PR and not scheduled. `build.rs` holds the recipe, `scripts/iree/**`
	# pins the IREE distribution whose archive set the recipe names, and
	# `src/lib/mlxcel-xla/build.rs` with its `csrc/**` sources builds the shim
	# object whose undefined symbols those archives resolve, so those are the
	# causal surface. `rust-toolchain.toml` is on the same causal path because
	# the #1274 failure was entirely about where rustc places its own `-lc`
	# relative to appended `rustc-link-arg` entries, which a toolchain bump can
	# move. `.github/workflows/ci.yml` is included so edits to
	# this job itself are testable. A schedule was rejected because it decouples
	# the failure from the PR that caused it, which is the exact failure mode of
	# #1274 (found by hand later, not by CI). Running on every Rust PR was
	# rejected because the release link is minutes of shared-runner time and the
	# overwhelming majority of Rust PRs cannot affect the link line.
	#
	# Measured cost of this job's link command on the GB10 runner: 15m40s from a
	# purged `CARGO_TARGET_DIR`, most of which is MLX's CUDA sources compiling
	# from scratch rather than the link itself, and 6m to 7m30s warm. Warm is the
	# figure that matters, because the trigger below fires on `build.rs`, which
	# invalidates the `mlxcel` crate but not `mlxcel-core`'s MLX build. For
	# comparison, `xla-compile`'s `cargo check` over the same feature set is
	# about a minute warm, which is why the two are separate jobs on separate
	# triggers.
	#
	# `RUSTFLAGS` is deliberately unset here, unlike `xla-compile`. This job
	# gates the link, not lints; `xla-compile` owns the lint policy. Leaving it
	# unset means a red run here is unambiguously a link failure, not a lint
	# failure wearing a link job's name.
	#
	# What the gate was demonstrated on: dropping `-l:libflatcc_parsing.a` from
	# the `IREE_CUDA_HOME` recipe in `build.rs`. On that tree `cargo check
	# --features cuda,xla-iree --all-targets` still passed in 59 seconds while
	# this job's link command failed with undefined references to
	# `flatcc_verify_*`, which is the same shape as #1274 and the exact gap this
	# job exists to close.
	#
	# #1274's own defect, removing the second `-lc`, was tried as the control
	# first and did not fail the link on the tree this job was validated against.
	# Why it did not is unresolved, so do not read that run as evidence the entry
	# is dead: against the pinned runtime, `nm` still reports `__stack_chk_guard`
	# undefined in 176 objects of `libiree_runtime_unified.a`, including the
	# `call.c.o` named in #1274; the symbol is still undefined in `libc.so.6`;
	# and it is still defined only in `ld-linux-aarch64.so.1`. Every precondition
	# the `build.rs` comment records holds today, so leave that entry alone until
	# a control that actually reproduces says otherwise.
	#
	# Deliberately NOT covered, so the gap is recorded rather than assumed shut:
	# - The `IREE_DIST` recipe (`build.rs:180-220`) and the macOS
	# `IREE_MACOS_HOME` recipe (`build.rs:38-90`). Both remain unverified by
	# any link, on any machine, because no runner in this repository holds
	# either distribution. Only the `IREE_CUDA_HOME` recipe this job exercises
	# is covered.
	# - Anything outside the path filter above. Because the trigger is
	# path-filtered rather than universal, a link regression arriving through
	# a path the filter does not name (a new dependency pulling in a
	# conflicting native library, for instance) is still not caught at PR
	# time.
	# - GPU execution. `--no-run` links only; the XLA test suites still have no
	# CI coverage, per the `xla-compile` comment above.
	# - A run where cargo finds nothing to redo. The target directory persists
	# across PRs, so a green run does not by itself prove a link happened:
	# this job's own first run on the PR that added it finished in 12 seconds
	# with `Finished release profile in 0.14s`. That is correct, because
	# cargo relinks exactly when the fingerprint moves, and `build.rs`
	# declares `rerun-if-env-changed` for all three IREE variables so a
	# version bump does move it. The narrow hole is a `scripts/iree/**` edit
	# that changes how the runtime is built without moving `IREE_VERSION`:
	# the script is idempotent, so nothing rebuilds and nothing relinks.
	xla-link:
	name: OpenXLA feature link
	needs: changes
	if: github.repository == 'lablup/mlxcel' && needs.changes.outputs.xla_link == 'true'
	runs-on: GB10
	permissions:
	contents: read
	# The other jobs on this runner are seconds warm; this one is minutes, so
	# it is the first here that can hold GB10 long enough to matter. Without a
	# group, three pushes to a `build.rs` PR queue three link jobs of up to
	# 120 minutes each on the one runner that also serves clippy for every Rust
	# PR and the release build. Keyed per PR, so PRs never cancel each other.
	concurrency:
	group: xla-link-${{ github.event.pull_request.number \|\| github.ref }}
	cancel-in-progress: true
	timeout-minutes: 120
	env:
	# The value shipped for this runner's architecture. Auto-detection yields
	# `121a`, which is numerically identical here but is not what releases
	# build, so the gate links what ships.
	MLX_CUDA_ARCHITECTURES: "121"
	steps:
	- uses: actions/checkout@v7
	with:
	persist-credentials: false

	- name: Use a persistent target directory
	run: \|
	# Its own directory, not shared with `xla-compile` or the release
	# job: a link job emits release artifacts and build-script output
	# that a check-only cache does not, and sharing would have the two
	# jobs repeatedly invalidate each other.
	CARGO_TARGET="$HOME/.cargo-target/mlxcel-xla-link-ci"
	mkdir -p "$CARGO_TARGET"
	echo "CARGO_TARGET_DIR=$CARGO_TARGET" >> $GITHUB_ENV

	- name: Provision the IREE runtime
	run: \|
	# Idempotent: reuses `~/.cache/mlxcel/iree-cuda-<version>` when the
	# runtime is already built, and builds it once when it is not.
	bash scripts/iree/setup-cuda.sh
	# The script emits shell `export VAR=value` lines; GITHUB_ENV wants
	# bare `VAR=value`, and without stripping the prefix the variables
	# would be named "export IREE_CUDA_HOME" and build.rs would abort
	# claiming no IREE distribution is configured.
	bash scripts/iree/setup-cuda.sh --env \| sed 's/^export //' >> "$GITHUB_ENV"

	- name: Link the OpenXLA integration test
	run: cargo test --release --features cuda,xla-iree --test xla_prepared_prefill --no-run

	# ============================================================================
	# CUDA sm_70 compile (self-hosted GB10)
	# ============================================================================
	# Every other CUDA job in this repository compiles for sm_121, and
	# release.yml ships `80;86;89;90a;100;120` on x86_64 and `90a;100;121` on
	# aarch64. Nothing below sm_80 is compiled anywhere, so an arch-conditional
	# break on Volta reaches `main` unnoticed and is found by whoever next builds
	# from source on one. That is not hypothetical for this repository: epic
	# #1536 adds `cc < 8` fallbacks to the quantized-matmul overlays precisely
	# because the sm_80+ paths are wrong there, and every one of those branches
	# is compiled only when some job asks nvcc for a pre-Ampere target.
	#
	# Compile-only, and deliberately so. The realistic failure mode is a
	# compile-time one: a `__CUDA_ARCH__` guard that does not cover cc 7, a
	# CUTLASS type that has no pre-Ampere instantiation, a `bf16` intrinsic with
	# no sm_70 implementation. Catching those needs nvcc pointed at sm_70, not a
	# Volta card, and no GPU runner is involved here. The runtime half stays
	# uncovered, and the decision to leave it uncovered is recorded in
	# `docs/benchmark_results/volta-sm70-baseline-2026-08-31.md` (issue #1538)
	# together with what a GPU-backed Volta job would and would not add: there is
	# no Volta runner, no release artifact targets sm_70, and #1537 already turns
	# an architecture mismatch into a named startup error instead of an opaque
	# CUDA load failure.
	#
	# Path-filtered on `cuda_arch`, which is narrower than `rust`: a cold MLX
	# build for a new architecture is measured in tens of minutes, and only the
	# MLX overlays, the pinned commit and the build scripts can move what nvcc
	# compiles. Its own persistent target directory, because the architecture
	# list is part of the build-script fingerprint and sharing a directory with
	# an sm_121 job would make the two rebuild MLX against each other on every
	# run.
	cuda-sm70-compile:
	name: CUDA sm_70 compile
	needs: changes
	if: github.repository == 'lablup/mlxcel' && needs.changes.outputs.cuda_arch == 'true'
	runs-on: GB10
	permissions:
	contents: read
	# Same reasoning as `xla-link`: minutes warm, tens of minutes cold, on the
	# one runner that also serves clippy for every Rust PR and the release
	# build. Keyed per PR, so PRs never cancel each other.
	concurrency:
	group: cuda-sm70-${{ github.event.pull_request.number \|\| github.ref }}
	cancel-in-progress: true
	timeout-minutes: 120
	env:
	# Volta. Plain `70`, not `70a`: architecture-specific suffixes only exist
	# from sm_90 on, and build.rs forwards this string to CMake verbatim.
	MLX_CUDA_ARCHITECTURES: "70"
	steps:
	- uses: actions/checkout@v7
	with:
	persist-credentials: false

	- name: Use a persistent target directory
	run: \|
	# Not shared with any sm_121 job: the architecture list reaches CMake
	# through the build script, so two jobs at different architectures
	# sharing one directory would invalidate each other's MLX build every
	# run and pay a cold nvcc pass each time.
	CARGO_TARGET="$HOME/.cargo-target/mlxcel-cuda-sm70-ci"
	mkdir -p "$CARGO_TARGET"
	echo "CARGO_TARGET_DIR=$CARGO_TARGET" >> $GITHUB_ENV

	- name: Check whether the toolkit can target sm_70
	id: toolkit
	run: \|
	# CUDA 13 removed Volta: nvcc there fails with
	# `nvcc fatal : Unsupported gpu architecture 'compute_70'` before it
	# compiles anything. The CUDA runners currently carry 13.x, so this
	# gate cannot run on them, and a job that simply failed would red-light
	# every PR touching these paths for a toolkit limitation rather than a
	# code defect. Probe instead, and skip loudly. The moment a CUDA 12.x
	# runner joins the fleet this job starts gating for real with no edit.
	set -euo pipefail
	NVCC=$(command -v nvcc \|\| echo /usr/local/cuda/bin/nvcc)
	if [ ! -x "$NVCC" ]; then
	echo "nvcc not found" >&2
	exit 1
	fi
	VERSION=$("$NVCC" --version \| sed -n 's/.release $[0-9.]$.*/\1/p' \| head -1)
	if "$NVCC" --list-gpu-arch 2>/dev/null \| grep -qx compute_70; then
	echo "supported=true" >> "$GITHUB_OUTPUT"
	echo "CUDA $VERSION targets sm_70; gating for real."
	else
	echo "supported=false" >> "$GITHUB_OUTPUT"
	echo "CUDA $VERSION cannot target sm_70; skipping the sm_70 build." \| tee -a "$GITHUB_STEP_SUMMARY"
	{
	echo ""
	echo "sm_70 gate skipped. This runner's CUDA toolkit is $VERSION, and CUDA 13 removed Volta (sm_70) support, so nvcc rejects \`compute_70\` outright. Nothing was compiled for Volta and nothing was verified. This is a toolkit limitation, not a passing gate. Building mlxcel for Volta requires a CUDA 12.x toolchain; see the CUDA architecture selection section of docs/installation.md."
	} >> "$GITHUB_STEP_SUMMARY"
	fi

	- name: Compile the CUDA feature set for sm_70
	if: steps.toolkit.outputs.supported == 'true'
	run: cargo check --features cuda --all-targets

	- name: Verify the emitted cubins are sm_70
	if: steps.toolkit.outputs.supported == 'true'
	run: \|
	# `cargo check` still runs the build script, so MLX is compiled and
	# archived even though nothing links. Asserting on the archive turns
	# "the environment variable was set" into "nvcc actually emitted
	# pre-Ampere code", which is what the job claims to gate. It also
	# catches the silent trap described in #1538: build.rs auto-detects
	# the host and falls back to `90a` when it cannot, so a job whose
	# architecture list failed to reach CMake would otherwise pass here
	# having compiled nothing for Volta.
	set -euo pipefail
	# cuobjdump ships next to nvcc, which the build above already
	# required, so resolving it from PATH and then from nvcc's directory
	# covers both a CUDA toolkit on PATH and one reached only through an
	# absolute nvcc.
	CUOBJDUMP=$(command -v cuobjdump \|\| true)
	if [ -z "$CUOBJDUMP" ]; then
	NVCC=$(command -v nvcc \|\| true)
	[ -n "$NVCC" ] && CUOBJDUMP="$(dirname "$NVCC")/cuobjdump"
	fi
	if [ ! -x "${CUOBJDUMP:-}" ]; then
	echo "cuobjdump not found; it ships with the CUDA toolkit next to nvcc" >&2
	exit 1
	fi
	LIB=$(find "$CARGO_TARGET_DIR/debug/build" -path '*/out/build/lib/libmlx.a' \| head -1)
	if [ -z "$LIB" ]; then
	echo "libmlx.a not found under $CARGO_TARGET_DIR/debug/build" >&2
	exit 1
	fi
	echo "archive: $LIB"
	ARCHES=$("$CUOBJDUMP" --list-elf "$LIB" \| grep -oE 'sm_[0-9]+a?' \| sort -u)
	echo "architectures present: $(echo "$ARCHES" \| tr '\n' ' ')"
	if [ "$ARCHES" != "sm_70" ]; then
	echo "expected sm_70 only, found: $ARCHES" >&2
	exit 1
	fi

Navigation Menu

Uh oh!

fix(server): stamp the speculative burst's first token at its prefill… #2726

Workflow file

fix(server): stamp the speculative burst's first token at its prefill… #2726

Uh oh!

Workflow file for this run