Qwen3.5-VL vision returns hallucinated output for every image (mm_features + load order + prefill shape)

[yaanfpv]

Summary

Vision requests against Qwen3.5-VL models return the same hallucinated description (often "a grid of Thai tone marks") for every input image, regardless of content. The bug spans three sequential issues across the Python plugin and the Swift bridge.

Fix submitted as #15.

Affected versions

• vllm-swift v0.6.0 (Homebrew bottle, arm64_tahoe)
• vllm 0.19.1
• mlx-swift / mlx-swift-lm pinned in v0.6.0
• macOS 26 (Tahoe), Apple Silicon

Reproduction

Model: any Qwen3.5-VL checkpoint converted to MLX (verified on Qwen3.5-9B-VL, yaanfpv/Vayaan_9B_OmniClaw, MLX 4-bit).

vllm-swift serve <Qwen3.5-9B-VL MLX dir> \
  --served-model-name qwen35-9b-vl \
  --max-model-len 32768 \
  --no-enable-prefix-caching \
  --max-num-seqs 1 \
  --host 127.0.0.1 --port 8080 \
  --trust-remote-code

Then post a chat completion with an image_url content part:

curl -s -X POST http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen35-9b-vl",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Describe this image."},
        {"type": "image_url", "image_url": {"url": "<any image URL>"}}
      ]
    }],
    "max_tokens": 200
  }'

Observed: model produces a hallucinated description that is identical (or near-identical) regardless of the input image. A synthetic solid-red PNG and a Himalayan mountain photograph both produce a description about "Thai tone marks in a grid".

Expected: model produces a description grounded in the actual image content. LM Studio with the same MLX checkpoint produces correct descriptions, so the model itself is fine.

Root cause (three layered bugs)

1. vllm_swift/worker.py — mm_features API drift. vLLM v0.19+ delivers mm_features on NewRequestData as list[MultiModalFeatureSpec], but the existing check hasattr(mm_features, "pixel_values") always returns False on a list. Pixel data is silently dropped before it reaches the FFI bridge.

2. Bridge.swift — load order on dual-registered model types. MLXLLM.LLMModelFactory accepts qwen3_5 (which is registered in both MLXLLM and MLXVLM) and successfully loads a VL checkpoint as a text-only model. The vision tower weights load but never execute. Image-placeholder tokens reach the LM as raw text embeddings.

3. Bridge.swift — prefill_vlm token shape. vsm_engine_prefill_vlm constructs tokenArray as flat 1D. Downstream, MLXVLM/Qwen35.prepare calls embedTokens which on a 1D input produces 2D [S, H], then routes that into the language model where the gated-delta linear-attention reshape wants 3D [B, S, H] and crashes:

   [reshape] Cannot reshape array of size 372736 into shape (91,4096,32,128)
   

   (91, 4096, 32, 128) corresponds to (inputs.dim(0), inputs.dim(1), linearNumValueHeads, linearValueHeadDim) for Qwen3.5-9B (i.e., the linear-attention layer treated S=91 as B and H=4096 as S).

Why bug 1 alone produces the visible symptom

Bug 1 is upstream of bugs 2 and 3 in the request flow. With pixels silently dropped, the Swift engine never sees image data; image-placeholder token ids land as untrained / random-ish embeddings in the LM, and the model confidently hallucinates the same generic content for any input. Fixing bug 1 alone exposes bug 2 (visible as zero-effect vision because LLMModelFactory still claims the model). Fixing bugs 1 and 2 then exposes bug 3 (visible as a Swift fatal during prefill).

Fix

PR #15 ports all three fixes:

• fix(worker): handle mm_features as list[MultiModalFeatureSpec]
• fix(bridge): load via VLMModelFactory when config has vision_config
• fix(bridge): reshape prefill_vlm tokenArray to [1, N] for mlx-vlm

After the PR, vision works end-to-end on Qwen3.5-9B-VL with both real photographs and a synthetic solid-red PNG. Plain text inference is unchanged. Plain LLM loading still goes through LLMModelFactory first to preserve the BatchedHybridLLM fast path.

Out of scope

A separate issue thread covers the head_dim=256 Metal kernel coverage gap (TurboQuant turbo_dequant_rotated_*_256_* not present in the v0.6.0 metallib bottle, and likely flash-attention coverage too). That is independent of this bug.

Reported by @yaanfpv.