@@ -138,74 +138,98 @@ Linux/CUDA builds use the `cuda` feature and require the CUDA toolkit plus the s
138138## Performance
139139
140140mlxcel targets near-` mlx-lm ` / ` mlx-vlm ` decode throughput for MLX-format
141- checkpoints while keeping a native Rust runtime. In the mlxcel 0.1.0 M5 Max
142- 128GB benchmark set , the headline result has two parts: faster short-prompt
143- text prefill and near-reference decode throughput.
141+ checkpoints while keeping a native Rust runtime. In the M5 Max 128GB benchmark
142+ campaign , the headline result has two parts: faster short-prompt text prefill
143+ and near-reference decode throughput.
144144
145145### Prefill: prompt ingestion before the first generated token
146146
147- Short-prompt text prefill is the standout result. mlxcel measured ** 2.70x **
148- the ` mlx-lm ` median on M5 Max across 66 comparable text pairs, and ** 1.76x **
149- on M1 Ultra across 73 comparable text pairs. VLM prefill is listed separately
147+ Short-prompt text prefill is the standout result. mlxcel measured ** 2.78x **
148+ the ` mlx-lm ` median on M5 Max across 67 comparable text pairs, and ** 1.79x **
149+ on M1 Ultra across 74 comparable text pairs. VLM prefill is listed separately
150150because image preprocessing, vision encoder, and projector work can be included
151151in the prefill path.
152152
153153| Mode | Baseline | M5 Max pairs | M5 Max median vs baseline | M1 Ultra pairs | M1 Ultra median vs baseline |
154154| ------| ----------| -------------:| --------------------------:| ---------------:| ----------------------------:|
155- | Text | ` mlx-lm ` | 66 | ** 2.70x ** | 73 | ** 1.76x ** |
156- | VLM | ` mlx-vlm ` | 20 | 0.94x | 17 | ** 1.33x ** |
155+ | Text | ` mlx-lm ` | 67 | ** 2.78x ** | 74 | ** 1.79x ** |
156+ | VLM | ` mlx-vlm ` | 25 | ** 1.01x ** | 20 | ** 1.05x ** |
157157
158158### Decode: steady-state token generation
159159
160160Decode stays close to the Python MLX references on the same host. For M5 Max,
161- text decode averaged ** 99%** of ` mlx-lm ` with a ** 99 %** median, while VLM decode
162- averaged ** 102 %** of ` mlx-vlm ` with a ** 101 %** median.
161+ text decode averaged ** 99%** of ` mlx-lm ` with a ** 100 %** median, while VLM decode
162+ averaged ** 98 %** of ` mlx-vlm ` with a ** 98 %** median.
163163
164164| Mode | Baseline | Comparable pairs | Average vs baseline | Median vs baseline | >=90% parity | >= baseline | Range |
165165| ------| ----------| -----------------:| --------------------:| -------------------:| -------------:| ------------:| ------:|
166- | Text | ` mlx-lm ` | 66 | 99% | ** 99%** | 62 / 66 (94%) | 27 / 66 (41%) | 72%-127% |
167- | VLM | ` mlx-vlm ` | 22 | 102% | ** 101%** | 18 / 22 (82%) | 11 / 22 (50%) | 74%-123% |
168-
169- Representative decode throughput is shown below in tokens per second. M5 Max
170- reference columns are same-host ` mlx-lm ` or ` mlx-vlm ` runs; M1 Ultra values are
171- included as mlxcel-only capacity references. Absolute results depend on model
172- family, quantization, prompt shape, decode length, and hardware. See
166+ | Text | ` mlx-lm ` | 67 | 99% | ** 100%** | 62 / 67 (93%) | 31 / 67 (46%) | 45%-129% |
167+ | VLM | ` mlx-vlm ` | 24 | 98% | ** 98%** | 18 / 24 (75%) | 10 / 24 (42%) | 59%-121% |
168+
169+ Representative decode throughput is shown below in tokens per second. The
170+ mlxcel columns are the 2026-06-15 sweep on each host (v0.3.0, including the fix
171+ to a quantized-decode regression on bf16-scale checkpoints that mostly affected
172+ M1 Ultra). The M5 Max ` mlx-lm ` / ` mlx-vlm ` reference columns are retained from
173+ the earlier same-host campaign, so each ratio is mlxcel (2026-06-15) over that
174+ retained reference; a fresh same-host mlx-lm / mlx-vlm run validated that the
175+ reference is stable. M1 Ultra values are mlxcel-only capacity references.
176+ Absolute results depend on model family, quantization, prompt shape, decode
177+ length, and hardware. See
173178[ Benchmark results] ( docs/benchmark_results/benchmark-report.md ) and
174179[ Benchmarks] ( docs/benchmarks.md ) for methodology and caveats.
175180
176181| Text model | M1 Ultra mlxcel | M5 Max mlxcel | M5 Max mlx-lm | mlxcel / mlx-lm |
177182| ------------| ----------------:| --------------:| --------------:| ----------------:|
178- | SmolLM-135M 4bit | 384 tok/s | 905 tok/s | 712 tok/s | 127 % |
179- | Llama 3.1 8B 4bit | 109 tok/s | 117 tok/s | 117 tok/s | 99 % |
183+ | SmolLM-135M 4bit | 375 tok/s | 917 tok/s | 712 tok/s | 129 % |
184+ | Llama 3.1 8B 4bit | 108 tok/s | 117 tok/s | 117 tok/s | 100 % |
180185| Qwen2.5 7B 4bit | 113 tok/s | 126 tok/s | 124 tok/s | 102% |
181- | Gemma 2B 4bit | 195 tok/s | 217 tok/s | 223 tok/s | 97 % |
182- | Gemma 3 4B 4bit | 118 tok/s | 182 tok/s | 182 tok/s | 100 % |
183- | Gemma 2 2B 4bit | 170 tok/s | 242 tok/s | 242 tok/s | 100% |
184- | Phi-3.5-mini 4bit | 167 tok/s | 205 tok/s | 208 tok/s | 98% |
185- | Jamba v0.1 4bit (hybrid SSM) | 124 tok/s | 216 tok/s | 219 tok/s | 98 % |
186- | Gemma 4 26B-A4B 4bit | 72 tok/s | 137 tok/s | 141 tok/s | 97 % |
187- | Qwen3 MoE 30B 4bit | 71 tok/s | 156 tok/s | 147 tok/s | 106 % |
188- | GLM-4 Flash 4bit | 48 tok/s | 104 tok/s | 104 tok/s | 100% |
189- | Nemotron-H 30B 4bit | 92 tok/s | 177 tok/s | 179 tok/s | 99 % |
190- | Mixtral 8x7B 4bit | 55 tok/s | 65 tok/s | 66 tok/s | 99 % |
191- | StarCoder2 3B 4bit | 173 tok/s | 216 tok/s | 215 tok/s | 101 % |
192- | Qwen3.5 0.8B 4bit | 244 tok/s | 517 tok/s | 545 tok/s | 95 % |
193- | Qwen3-VL 30B-A3B 4bit, text path | 71 tok/s | 151 tok/s | 147 tok/s | 103% |
194- | Qwen3-VL 32B 4bit, text path | 21 tok/s | 28 tok/s | 29 tok/s | 96 % |
195- | GPT-OSS 120B 4bit | 61 tok/s | 114 tok/s | 110 tok/s | 103 % |
196- | Solar Open 100B 4bit | 36 tok/s | 65 tok/s | 66 tok/s | 99 % |
186+ | Gemma 2B 4bit | 196 tok/s | 215 tok/s | 223 tok/s | 96 % |
187+ | Gemma 3 4B 4bit | 117 tok/s | 183 tok/s | 182 tok/s | 101 % |
188+ | Gemma 2 2B 4bit | 166 tok/s | 241 tok/s | 242 tok/s | 100% |
189+ | Phi-3.5-mini 4bit | 164 tok/s | 203 tok/s | 208 tok/s | 98% |
190+ | Jamba v0.1 4bit (hybrid SSM) | 122 tok/s | 216 tok/s | 219 tok/s | 99 % |
191+ | Gemma 4 26B-A4B 4bit | 80 tok/s | 151 tok/s | 141 tok/s | 107 % |
192+ | Qwen3 MoE 30B 4bit | 84 tok/s | 176 tok/s | 147 tok/s | 120 % |
193+ | GLM-4 Flash 4bit | 46 tok/s | 104 tok/s | 104 tok/s | 100% |
194+ | Nemotron-H 30B 4bit | 92 tok/s | 176 tok/s | 179 tok/s | 98 % |
195+ | Mixtral 8x7B 4bit | 54 tok/s | 65 tok/s | 66 tok/s | 98 % |
196+ | StarCoder2 3B 4bit | 166 tok/s | 216 tok/s | 215 tok/s | 100 % |
197+ | Qwen3.5 0.8B 4bit | 230 tok/s | 504 tok/s | 545 tok/s | 92 % |
198+ | Qwen3-VL 30B-A3B 4bit, text path | 69 tok/s | 151 tok/s | 147 tok/s | 103% |
199+ | Qwen3-VL 32B 4bit, text path | 21 tok/s | 27 tok/s | 29 tok/s | 93 % |
200+ | GPT-OSS 120B 4bit | 58 tok/s | 114 tok/s | 110 tok/s | 104 % |
201+ | Solar Open 100B 4bit | 33 tok/s | 65 tok/s | 66 tok/s | 98 % |
197202
198203| VLM model | M1 Ultra mlxcel | M5 Max mlxcel | M5 Max mlx-vlm | mlxcel / mlx-vlm |
199204| -----------| ----------------:| --------------:| ---------------:| -----------------:|
200- | LLaVA Interleave Qwen 0.5B bf16 | 266 tok/s | 344 tok/s | 345 tok/s | 100 % |
201- | Qwen3.5 0.8B 4bit | 234 tok/s | 506 tok/s | 411 tok/s | 123 % |
202- | Qwen3.5 35B-A3B 4bit | 70 tok/s | 151 tok/s | 129 tok/s | 117 % |
203- | Gemma 4 E2B 4bit | 107 tok/s | 217 tok/s | 202 tok/s | 108 % |
205+ | LLaVA Interleave Qwen 0.5B bf16 | 265 tok/s | 341 tok/s | 345 tok/s | 99 % |
206+ | Qwen3.5 0.8B 4bit | 232 tok/s | 454 tok/s | 411 tok/s | 110 % |
207+ | Qwen3.5 35B-A3B 4bit | 75 tok/s | 149 tok/s | 129 tok/s | 116 % |
208+ | Gemma 4 E2B 4bit | 106 tok/s | 220 tok/s | 202 tok/s | 109 % |
204209| Gemma 3n E2B 4bit | 73 tok/s | 151 tok/s | 125 tok/s | 121% |
205- | InternVL3 1B | 229 tok/s | 601 tok/s | 529 tok/s | 114 % |
206- | Gemma 4 26B-A4B 4bit | 66 tok/s | 134 tok/s | 137 tok/s | 98 % |
210+ | InternVL3 1B | 238 tok/s | 575 tok/s | 529 tok/s | 109 % |
211+ | Gemma 4 26B-A4B 4bit | 70 tok/s | 144 tok/s | 137 tok/s | 105 % |
207212| Molmo2 4B | 60 tok/s | 64 tok/s | 67 tok/s | 96% |
208- | Phi 3.5 Vision 4bit | 123 tok/s | 169 tok/s | 160 tok/s | 106% |
213+ | Phi 3.5 Vision 4bit | 122 tok/s | 168 tok/s | 160 tok/s | 105% |
214+
215+ ### DiffusionGemma (block diffusion)
216+
217+ DiffusionGemma generates a canvas block at a time through iterative denoising
218+ rather than left-to-right autoregression. The decode harness above measures
219+ inter-token timing, which does not apply to diffusion's burst output, so the
220+ automated sweep records this checkpoint as a benchmark failure. The numbers
221+ below are a manual same-host comparison (192-token generation, chat template,
222+ seed 42, ` max_denoising_steps=48 ` , median of 3 runs):
223+
224+ | Diffusion model | M1 Ultra mlxcel | M1 Ultra mlx-vlm | mlxcel / mlx-vlm |
225+ | -----------------| ----------------:| -----------------:| -----------------:|
226+ | DiffusionGemma 26B-A4B 4bit | 32 tok/s | 29 tok/s | 110% |
227+
228+ Released ` mlx-vlm ` (0.4.4) does not include ` diffusion_gemma ` , so the reference
229+ column is ` mlx-vlm ` upstream ` main ` . The reported tok/s amortizes the per-block
230+ denoising passes and is not directly comparable to the autoregressive decode
231+ rows above. No M5 Max figure is listed because that comparison was not run on
232+ the same-host campaign.
209233
210234The M5 Max sweep covers 98 text model directories and a matching 98-entry VLM
211235mode pass. Ratio summaries include only rows where both mlxcel and the Python
0 commit comments