Skip to content

Commit 6631a33

Browse files
authored
docs(readme): refresh performance tables for v0.3.0 (#300)
Update the decode throughput tables (M1 Ultra + M5 Max, text and VLM) to the 2026-06-15 sweep and recompute the prefill/decode aggregate summaries against the retained mlx-lm / mlx-vlm campaign reference, which a fresh same-host run confirmed is stable. The #289 bf16-scale decode fix shows up here (e.g. Qwen3 MoE 30B M5 156->176, Gemma 4 26B-A4B M5 137->151). Correct the M1 Ultra VLM prefill figure: the prior 1.33x was a non-reproducible 2026-05-19 sweep anomaly (confirmed by rebuilding v0.0.28 / v0.1.0; see #299) and is actually ~parity with mlx-vlm. Add a DiffusionGemma 26B-A4B block-diffusion mini-table (M1 Ultra: mlxcel 32 tok/s vs mlx-vlm 29 tok/s, 110%).
1 parent 7cb6ede commit 6631a33

1 file changed

Lines changed: 66 additions & 42 deletions

File tree

README.md

Lines changed: 66 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -138,74 +138,98 @@ Linux/CUDA builds use the `cuda` feature and require the CUDA toolkit plus the s
138138
## Performance
139139

140140
mlxcel targets near-`mlx-lm` / `mlx-vlm` decode throughput for MLX-format
141-
checkpoints while keeping a native Rust runtime. In the mlxcel 0.1.0 M5 Max
142-
128GB benchmark set, the headline result has two parts: faster short-prompt
143-
text prefill and near-reference decode throughput.
141+
checkpoints while keeping a native Rust runtime. In the M5 Max 128GB benchmark
142+
campaign, the headline result has two parts: faster short-prompt text prefill
143+
and near-reference decode throughput.
144144

145145
### Prefill: prompt ingestion before the first generated token
146146

147-
Short-prompt text prefill is the standout result. mlxcel measured **2.70x**
148-
the `mlx-lm` median on M5 Max across 66 comparable text pairs, and **1.76x**
149-
on M1 Ultra across 73 comparable text pairs. VLM prefill is listed separately
147+
Short-prompt text prefill is the standout result. mlxcel measured **2.78x**
148+
the `mlx-lm` median on M5 Max across 67 comparable text pairs, and **1.79x**
149+
on M1 Ultra across 74 comparable text pairs. VLM prefill is listed separately
150150
because image preprocessing, vision encoder, and projector work can be included
151151
in the prefill path.
152152

153153
| Mode | Baseline | M5 Max pairs | M5 Max median vs baseline | M1 Ultra pairs | M1 Ultra median vs baseline |
154154
|------|----------|-------------:|--------------------------:|---------------:|----------------------------:|
155-
| Text | `mlx-lm` | 66 | **2.70x** | 73 | **1.76x** |
156-
| VLM | `mlx-vlm` | 20 | 0.94x | 17 | **1.33x** |
155+
| Text | `mlx-lm` | 67 | **2.78x** | 74 | **1.79x** |
156+
| VLM | `mlx-vlm` | 25 | **1.01x** | 20 | **1.05x** |
157157

158158
### Decode: steady-state token generation
159159

160160
Decode stays close to the Python MLX references on the same host. For M5 Max,
161-
text decode averaged **99%** of `mlx-lm` with a **99%** median, while VLM decode
162-
averaged **102%** of `mlx-vlm` with a **101%** median.
161+
text decode averaged **99%** of `mlx-lm` with a **100%** median, while VLM decode
162+
averaged **98%** of `mlx-vlm` with a **98%** median.
163163

164164
| Mode | Baseline | Comparable pairs | Average vs baseline | Median vs baseline | >=90% parity | >= baseline | Range |
165165
|------|----------|-----------------:|--------------------:|-------------------:|-------------:|------------:|------:|
166-
| Text | `mlx-lm` | 66 | 99% | **99%** | 62 / 66 (94%) | 27 / 66 (41%) | 72%-127% |
167-
| VLM | `mlx-vlm` | 22 | 102% | **101%** | 18 / 22 (82%) | 11 / 22 (50%) | 74%-123% |
168-
169-
Representative decode throughput is shown below in tokens per second. M5 Max
170-
reference columns are same-host `mlx-lm` or `mlx-vlm` runs; M1 Ultra values are
171-
included as mlxcel-only capacity references. Absolute results depend on model
172-
family, quantization, prompt shape, decode length, and hardware. See
166+
| Text | `mlx-lm` | 67 | 99% | **100%** | 62 / 67 (93%) | 31 / 67 (46%) | 45%-129% |
167+
| VLM | `mlx-vlm` | 24 | 98% | **98%** | 18 / 24 (75%) | 10 / 24 (42%) | 59%-121% |
168+
169+
Representative decode throughput is shown below in tokens per second. The
170+
mlxcel columns are the 2026-06-15 sweep on each host (v0.3.0, including the fix
171+
to a quantized-decode regression on bf16-scale checkpoints that mostly affected
172+
M1 Ultra). The M5 Max `mlx-lm` / `mlx-vlm` reference columns are retained from
173+
the earlier same-host campaign, so each ratio is mlxcel (2026-06-15) over that
174+
retained reference; a fresh same-host mlx-lm / mlx-vlm run validated that the
175+
reference is stable. M1 Ultra values are mlxcel-only capacity references.
176+
Absolute results depend on model family, quantization, prompt shape, decode
177+
length, and hardware. See
173178
[Benchmark results](docs/benchmark_results/benchmark-report.md) and
174179
[Benchmarks](docs/benchmarks.md) for methodology and caveats.
175180

176181
| Text model | M1 Ultra mlxcel | M5 Max mlxcel | M5 Max mlx-lm | mlxcel / mlx-lm |
177182
|------------|----------------:|--------------:|--------------:|----------------:|
178-
| SmolLM-135M 4bit | 384 tok/s | 905 tok/s | 712 tok/s | 127% |
179-
| Llama 3.1 8B 4bit | 109 tok/s | 117 tok/s | 117 tok/s | 99% |
183+
| SmolLM-135M 4bit | 375 tok/s | 917 tok/s | 712 tok/s | 129% |
184+
| Llama 3.1 8B 4bit | 108 tok/s | 117 tok/s | 117 tok/s | 100% |
180185
| Qwen2.5 7B 4bit | 113 tok/s | 126 tok/s | 124 tok/s | 102% |
181-
| Gemma 2B 4bit | 195 tok/s | 217 tok/s | 223 tok/s | 97% |
182-
| Gemma 3 4B 4bit | 118 tok/s | 182 tok/s | 182 tok/s | 100% |
183-
| Gemma 2 2B 4bit | 170 tok/s | 242 tok/s | 242 tok/s | 100% |
184-
| Phi-3.5-mini 4bit | 167 tok/s | 205 tok/s | 208 tok/s | 98% |
185-
| Jamba v0.1 4bit (hybrid SSM) | 124 tok/s | 216 tok/s | 219 tok/s | 98% |
186-
| Gemma 4 26B-A4B 4bit | 72 tok/s | 137 tok/s | 141 tok/s | 97% |
187-
| Qwen3 MoE 30B 4bit | 71 tok/s | 156 tok/s | 147 tok/s | 106% |
188-
| GLM-4 Flash 4bit | 48 tok/s | 104 tok/s | 104 tok/s | 100% |
189-
| Nemotron-H 30B 4bit | 92 tok/s | 177 tok/s | 179 tok/s | 99% |
190-
| Mixtral 8x7B 4bit | 55 tok/s | 65 tok/s | 66 tok/s | 99% |
191-
| StarCoder2 3B 4bit | 173 tok/s | 216 tok/s | 215 tok/s | 101% |
192-
| Qwen3.5 0.8B 4bit | 244 tok/s | 517 tok/s | 545 tok/s | 95% |
193-
| Qwen3-VL 30B-A3B 4bit, text path | 71 tok/s | 151 tok/s | 147 tok/s | 103% |
194-
| Qwen3-VL 32B 4bit, text path | 21 tok/s | 28 tok/s | 29 tok/s | 96% |
195-
| GPT-OSS 120B 4bit | 61 tok/s | 114 tok/s | 110 tok/s | 103% |
196-
| Solar Open 100B 4bit | 36 tok/s | 65 tok/s | 66 tok/s | 99% |
186+
| Gemma 2B 4bit | 196 tok/s | 215 tok/s | 223 tok/s | 96% |
187+
| Gemma 3 4B 4bit | 117 tok/s | 183 tok/s | 182 tok/s | 101% |
188+
| Gemma 2 2B 4bit | 166 tok/s | 241 tok/s | 242 tok/s | 100% |
189+
| Phi-3.5-mini 4bit | 164 tok/s | 203 tok/s | 208 tok/s | 98% |
190+
| Jamba v0.1 4bit (hybrid SSM) | 122 tok/s | 216 tok/s | 219 tok/s | 99% |
191+
| Gemma 4 26B-A4B 4bit | 80 tok/s | 151 tok/s | 141 tok/s | 107% |
192+
| Qwen3 MoE 30B 4bit | 84 tok/s | 176 tok/s | 147 tok/s | 120% |
193+
| GLM-4 Flash 4bit | 46 tok/s | 104 tok/s | 104 tok/s | 100% |
194+
| Nemotron-H 30B 4bit | 92 tok/s | 176 tok/s | 179 tok/s | 98% |
195+
| Mixtral 8x7B 4bit | 54 tok/s | 65 tok/s | 66 tok/s | 98% |
196+
| StarCoder2 3B 4bit | 166 tok/s | 216 tok/s | 215 tok/s | 100% |
197+
| Qwen3.5 0.8B 4bit | 230 tok/s | 504 tok/s | 545 tok/s | 92% |
198+
| Qwen3-VL 30B-A3B 4bit, text path | 69 tok/s | 151 tok/s | 147 tok/s | 103% |
199+
| Qwen3-VL 32B 4bit, text path | 21 tok/s | 27 tok/s | 29 tok/s | 93% |
200+
| GPT-OSS 120B 4bit | 58 tok/s | 114 tok/s | 110 tok/s | 104% |
201+
| Solar Open 100B 4bit | 33 tok/s | 65 tok/s | 66 tok/s | 98% |
197202

198203
| VLM model | M1 Ultra mlxcel | M5 Max mlxcel | M5 Max mlx-vlm | mlxcel / mlx-vlm |
199204
|-----------|----------------:|--------------:|---------------:|-----------------:|
200-
| LLaVA Interleave Qwen 0.5B bf16 | 266 tok/s | 344 tok/s | 345 tok/s | 100% |
201-
| Qwen3.5 0.8B 4bit | 234 tok/s | 506 tok/s | 411 tok/s | 123% |
202-
| Qwen3.5 35B-A3B 4bit | 70 tok/s | 151 tok/s | 129 tok/s | 117% |
203-
| Gemma 4 E2B 4bit | 107 tok/s | 217 tok/s | 202 tok/s | 108% |
205+
| LLaVA Interleave Qwen 0.5B bf16 | 265 tok/s | 341 tok/s | 345 tok/s | 99% |
206+
| Qwen3.5 0.8B 4bit | 232 tok/s | 454 tok/s | 411 tok/s | 110% |
207+
| Qwen3.5 35B-A3B 4bit | 75 tok/s | 149 tok/s | 129 tok/s | 116% |
208+
| Gemma 4 E2B 4bit | 106 tok/s | 220 tok/s | 202 tok/s | 109% |
204209
| Gemma 3n E2B 4bit | 73 tok/s | 151 tok/s | 125 tok/s | 121% |
205-
| InternVL3 1B | 229 tok/s | 601 tok/s | 529 tok/s | 114% |
206-
| Gemma 4 26B-A4B 4bit | 66 tok/s | 134 tok/s | 137 tok/s | 98% |
210+
| InternVL3 1B | 238 tok/s | 575 tok/s | 529 tok/s | 109% |
211+
| Gemma 4 26B-A4B 4bit | 70 tok/s | 144 tok/s | 137 tok/s | 105% |
207212
| Molmo2 4B | 60 tok/s | 64 tok/s | 67 tok/s | 96% |
208-
| Phi 3.5 Vision 4bit | 123 tok/s | 169 tok/s | 160 tok/s | 106% |
213+
| Phi 3.5 Vision 4bit | 122 tok/s | 168 tok/s | 160 tok/s | 105% |
214+
215+
### DiffusionGemma (block diffusion)
216+
217+
DiffusionGemma generates a canvas block at a time through iterative denoising
218+
rather than left-to-right autoregression. The decode harness above measures
219+
inter-token timing, which does not apply to diffusion's burst output, so the
220+
automated sweep records this checkpoint as a benchmark failure. The numbers
221+
below are a manual same-host comparison (192-token generation, chat template,
222+
seed 42, `max_denoising_steps=48`, median of 3 runs):
223+
224+
| Diffusion model | M1 Ultra mlxcel | M1 Ultra mlx-vlm | mlxcel / mlx-vlm |
225+
|-----------------|----------------:|-----------------:|-----------------:|
226+
| DiffusionGemma 26B-A4B 4bit | 32 tok/s | 29 tok/s | 110% |
227+
228+
Released `mlx-vlm` (0.4.4) does not include `diffusion_gemma`, so the reference
229+
column is `mlx-vlm` upstream `main`. The reported tok/s amortizes the per-block
230+
denoising passes and is not directly comparable to the autoregressive decode
231+
rows above. No M5 Max figure is listed because that comparison was not run on
232+
the same-host campaign.
209233

210234
The M5 Max sweep covers 98 text model directories and a matching 98-entry VLM
211235
mode pass. Ratio summaries include only rows where both mlxcel and the Python

0 commit comments

Comments
 (0)