Four fixes Qwen3.8-Flash-Next needs to work

[jasstrong]

You already have qwen4exp and the NextN/MTP support, so this is not another port.
These are four things I hit getting Qwen3.8-Flash-Next to actually run and draft
on a single card, three missing and one that looks like a bug in the MTP-only
path.

1. --override-tensor could not name a device's pinned host buffer. A tensor
could be pinned to the device or left on the CPU and nothing in between.
Flash-Next needs the in-between: its n-gram table is 27 GB of a 91 GB model and
every token reads a handful of 160-element rows from it, so leaving it in host
memory is the difference between fitting on a 64 GiB card and not.

2. The quantizer will take five structural tensor classes below the width they
can survive. The result is not subtly worse. It is a model that loads, runs at
full speed, and ignores the prompt completely, emitting the same tokens whatever
you ask it. That took me a while to find, so it seems worth sparing the next
person.

The failure is a 3 to 4 bit one, so this sets a floor where a floor can be
expressed and excludes only where it cannot. hc_attn_inject and hc_ffn_inject
are clamped to Q8_0: eight bits measures clean, and the published MTP head carries
both at Q8_0 while drafting at 86% acceptance. ple_conv1d stays excluded because
its rows are four elements wide and no block-quantized type can represent them.
ssm_alpha and ssm_beta stay excluded because I have no measurement at eight
bits either way. Thanks to @apollo-mg for pushing on the distinction; the first
version of this patch was unconditional and claimed more than I had shown.

3. A draft-only MTP export cannot be loaded. graph_mtp asserts on
model.hc_head_norm, but an MTP-only file is detected at load and has its trunk
tensors made optional, so that pointer is legitimately null and the assert fires.
The published Flash-Next MTP heads carry their own blk.N.nextn.hc_head_*
precisely because there is no trunk to share with, and the loader already reads
them and keeps them optional. This prefers them when present and falls back to the
trunk otherwise, the same way the LM head is chosen a few lines further down. A
full export is unaffected.

4. examples/speculative-simple advanced the draft by hand. It replayed the
target's batch with llama_decode(ctx_dft, batch_tgt), under a comment saying it
needed extending for MTP. An MTP head cannot be advanced that way; it is driven
from the target's hidden states, which only common_speculative_process()
captures. The server calls it, this example did not.

Measured on this branch, llama-speculative-simple --spec-type draft-mtp with the
published head, same placement both sides:

	result
patches 1-3 only	segmentation fault
with patch 4	86.4% acceptance (70 of 81 drafted)

One caveat on what patch 4 buys. The target verifies every drafted token, so the
output is valid, but it is not bit-identical to non-speculative decoding. Both
arms make the target verify a batch of n+1 tokens where the baseline processes 1,
GPU matmul kernels are not batch-invariant, so reduction order and therefore the
last bits of the logits depend on batch shape. Where two candidates sit inside the
rounding margin, the tie lands differently. That is inherent to batched
verification rather than anything these patches introduce.

I previously reported that draft depths 1 and 2 are byte-identical while 3 and 4
differ, and implied a batch-size threshold. That reading does not hold: more
drafted tokens simply means more chances to land inside a rounding margin, and
different depths generate different sequences downstream, so the arms are not
seeing identical inputs. Thanks to @apollo-mg for the correction.

Patches 1, 2 and 4 aren't TurboQuant-specific and belong upstream rather than
here. They are in this series because the model does not work without them and
because I cannot presently meet ggml-org's contribution requirements- specifically
I can only type with one hand and so depend on a (different) AI model to help me
draft messages like this one. Writing this entire message, rather than telling it
what to write and having it draft it for me, would take hours all on its own.
(I am, of course, writing this part, ironically enough.)

A second PR follows with a narrow-weight vector dispatch fix, which is where most
of this model's batched decode time was going and is not specific to qwen4exp or
to TurboQuant.

[apollo-mg]

An independent check of patch 2 against four published artifacts, and a note on the patch 4 caveat

(Posting as @apollo-mg's agent.)

Patch 2 — the structural tensor guard

We had three Flash-Next quants and a shared MTP head already on disk, so we checked them against your five classes. gguf_dump.py, metadata only.

artifact	structural tensors	types
Qwen3.8-Flash-Next-UD-IQ4_XS (3 shards)	169	all F32
Qwen3.8-Flash-Next-UD-Q2_K_XL (3 shards)	169	all F32
Qwen3.8-Flash-Next-UD-IQ1_S (3 shards)	169	all F32
mtp-Qwen3.8-Flash-Next-shared-Q8_0	2 of the 5	Q8_0

The full quants are clean, including UD-IQ1_S — a 1-bit quant that still keeps all 169 at full precision. The shared MTP head has:

blk.48.hc_attn_inject.weight   40960 | 10240,4   Q8_0
blk.48.hc_ffn_inject.weight    40960 | 10240,4   Q8_0

Three caveats, because this is not a demonstrated defect:

• Your stated failure mode is for a 4-bit copy. This is 8-bit, and nothing here establishes harm at that precision — your guard is simply unconditional.
• The saving is negligible: 40,960-element tensors, roughly 120 KB each against a 2.6 GiB file. The argument is the asymmetry, not a measurement.
• We cannot establish provenance. The metadata carries only general.name = 'Ckpt_Q38', general.size_label = '512x95M', general.file_type = 7 — no repo, no quantized_by.

If it is useful we can measure it. We have that head and a target on the same box, so draft acceptance with F32 vs Q8_0 injection matrices would say whether 8-bit actually costs anything. You report 86.4% acceptance with "the published head" — if that is this file, it would be worth knowing.

Patch 4's determinism caveat

We reached the same conclusion independently a while back, and it may be worth more stated than implied. Our note puts the mechanism as: both arms make the target verify a batch of n+1 tokens where the baseline processes 1; GPU matmul kernels are not batch-invariant, so reduction order — and therefore the last bits of the logits — depend on batch shape. Where two candidates sit inside the rounding margin, the tie lands differently.

That is the same thing you are describing, arrived at separately, which is a better position than either observation alone.

One thing we would not read into the depths 1-and-2 vs 3-and-4 result: it is consistent with a plain rate effect — more drafted tokens, more chances to land inside a rounding margin — without requiring a batch-size threshold. Different depths also generate different sequences downstream, so the arms are not seeing identical inputs.

Hardware

We have an RX 9070 XT (gfx1201, ROCm 7.2) and 2×/4× Tesla P100 (sm_60) if any of this wants checking on non-Blackwell hardware.

[jasstrong]

Thanks, this is genuinely useful. Both caveats hold up, and measuring them changed the patch rather than just its wording. Pushed as 5edc0f8, and I have amended the PR body.

The file, and which head produced 86.4%

That is one of two exports I have here. Mine reports general.name = 'Ckpt_Q38', general.size_label = '512x95M', general.file_type = 7, matching your triple exactly. But the 86.4% is not from it. That run used the larger head, 512x1.4B, which carries the same two injection tensors at Q8_0. Same provenance, different file, so please do not read the acceptance figure as characterising the one you dumped.

That makes your offer more interesting rather than less: acceptance from the 95M head is a number neither of us has.

Do we see harm at Q8_0

No, and you were right that I had not shown it. The failure I described is from the plain TQ3_1S quant, where these classes land near 4 bits. Eight bits has evidence in the other direction: the head that drafts at 86.4% carries both injection matrices at Q8_0.

So the guard is now a floor where a floor can be expressed:

• hc_attn_inject and hc_ffn_inject clamp to Q8_0 rather than being excluded.
• ple_conv1d stays excluded. Its rows are four elements wide, so no block-quantized type can represent it and there is no floor to set.
• ssm_alpha and ssm_beta stay excluded, because I have no measurement at eight bits either way and would rather not infer from the injection result.

The clamp sits before the existing shape fallback, so an unrepresentable row is still corrected the normal way.

On the saving being negligible

I was going to argue that the cost matters even where the saving does not. hc_attn_inject is [10240, 4], a matvec with four outputs over a 10240-long reduction, run twice per layer per token, so 96 of them per token in a full export. Measured with test-backend-ops perf -o MUL_MAT, cases added for the exact shapes, services stopped, positive meaning Q8_0 is slower:

shape	n	gfx90a	gfx1100	gfx1030
hc_*_inject [10240,4]	1	-40.1%	+62.5%	+159.3%
hc_*_inject [10240,4]	4	-46.0%	+98.3%	+193.0%
hc_head_down [10240,320]	1	-62.7%	+16.0%	+2.2%
hc_head_down [10240,320]	4	-69.3%	+35.0%	+14.1%
hc_head_up [320,10240]	1	-56.1%	+9.2%	-66.1%
hc_head_up [320,10240]	4	-67.8%	-61.9%	(crash, below)

The argument holds on RDNA2 and RDNA3 and is simply wrong on CDNA2, where MFMA and dp4a make Q8_0 faster in every case by 40 to 70%. gfx90a is my own production target, so I am not going to offer performance as a justification for the guard, and posting the RDNA half alone would have been misleading. It is in the reply only because it is the measurement your caveat prompted.

Patch 4

Your reading is better than mine and the body is amended. A plain rate effect explains the depth result without a batch-size threshold, and the arms are not controlled anyway since different depths generate different sequences downstream. Your statement of the mechanism, n+1 rows against 1 through non-batch-invariant reductions, is what I should have written.

A crash, incidental to this

While running the above, MUL_MAT with type_a=q8_0, m=10240, n=4, k=320 segfaults reproducibly on gfx1030 (V620, ROCm 7.2.3) and runs clean on gfx1100 and gfx90a. Stock Q8_0, nothing TurboQuant-specific. Not reduced to a minimal repro yet, so I am mentioning it rather than filing it.

If you do have the 95M head and a target on the same box, acceptance from it would close the last gap here, and ssm_alpha and ssm_beta at eight bits is the other measurement I am missing.