GatherBlockQuantized default zero_point differs between CPU and CUDA
Describe the issue
com.microsoft::GatherBlockQuantized produces different values on CPU and CUDA when the optional zero_points input is omitted.
For GGUF Q4_0 data the correct dequantization convention is (q - 8) * scale. ORT CPU appears to use an implicit zero point of 8, while ORT CUDA appears to use an implicit zero point of 0. This silently corrupts quantized embedding rows on CUDA whenever a converter relies on the default.
A downstream mobius conversion bug exposed this on large Q4_0 LLMs: every token embedding was offset by 8 * scale on CUDA. Adding an explicit packed zero point initializer of 0x88 fixed CUDA output without changing anything else.
Minimal reproducer shape
A single-node graph is enough:
- op:
com.microsoft::GatherBlockQuantized
- inputs:
qweight, indices, scales
- omit optional
zero_points
- attrs:
bits=4, block_size=32, gather_axis=0, quantize_axis=1
qweight: rows filled with nibble value 10 (0xAA bytes)scales: e.g. [[0.5], [0.25]]indices: [0, 1]
Expected under Q4_0-style dequantization with default zp 8:
- row 0:
(10 - 8) * 0.5 = 1.0
- row 1:
(10 - 8) * 0.25 = 0.5
Observed behavior from the downstream investigation:
- CPU matches
(q - 8) * scale
- CUDA matches
q * scale
Why this matters
The CPU/CUDA default mismatch makes model correctness depend on execution provider. In the downstream model, ORT CPU generated correct text, while ORT CUDA and native CUDA generated deterministic garbage until explicit zero points were added.
MatMulNBits was tested separately with the same model weights and CPU/CUDA agreed to fp16 rounding, so this report is specifically about GatherBlockQuantized's omitted zero_points default.
Downstream references
Request
Please make the GatherBlockQuantized default zero point consistent across CPU and CUDA, or document that converters must always provide zero_points because the default is not portable.
GatherBlockQuantized default zero_point differs between CPU and CUDA
Describe the issue
com.microsoft::GatherBlockQuantizedproduces different values on CPU and CUDA when the optionalzero_pointsinput is omitted.For GGUF Q4_0 data the correct dequantization convention is
(q - 8) * scale. ORT CPU appears to use an implicit zero point of8, while ORT CUDA appears to use an implicit zero point of0. This silently corrupts quantized embedding rows on CUDA whenever a converter relies on the default.A downstream mobius conversion bug exposed this on large Q4_0 LLMs: every token embedding was offset by
8 * scaleon CUDA. Adding an explicit packed zero point initializer of0x88fixed CUDA output without changing anything else.Minimal reproducer shape
A single-node graph is enough:
com.microsoft::GatherBlockQuantizedqweight,indices,scaleszero_pointsbits=4,block_size=32,gather_axis=0,quantize_axis=1qweight: rows filled with nibble value10(0xAAbytes)scales: e.g.[[0.5], [0.25]]indices:[0, 1]Expected under Q4_0-style dequantization with default zp 8:
(10 - 8) * 0.5 = 1.0(10 - 8) * 0.25 = 0.5Observed behavior from the downstream investigation:
(q - 8) * scaleq * scaleWhy this matters
The CPU/CUDA default mismatch makes model correctness depend on execution provider. In the downstream model, ORT CPU generated correct text, while ORT CUDA and native CUDA generated deterministic garbage until explicit zero points were added.
MatMulNBitswas tested separately with the same model weights and CPU/CUDA agreed to fp16 rounding, so this report is specifically aboutGatherBlockQuantized's omittedzero_pointsdefault.Downstream references
Request
Please make the
GatherBlockQuantizeddefault zero point consistent across CPU and CUDA, or document that converters must always providezero_pointsbecause the default is not portable.