Skip to content

fix(sdk): use resolved device for VLM encoder - #1437

Merged
RemiliaForever (RemiliaForever) merged 1 commit into
mainfrom
fix/vlm-device-selection
Sep 8, 2026
Merged

fix(sdk): use resolved device for VLM encoder#1437
RemiliaForever (RemiliaForever) merged 1 commit into
mainfrom
fix/vlm-device-selection

Conversation

@RemiliaForever

@RemiliaForever RemiliaForever (RemiliaForever) commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Use the resolved llama.cpp device selection when initializing the VLM vision encoder.
  • Disable mtmd GPU selection by default so CPU and hybrid paths remain on CPU.
  • Preserve CPU fallback when the selected accelerator cannot initialize.

Test plan

  • clang-format-18 on sdk/plugins/llama_cpp/src/vlm.cpp
  • QCS Windows ARM64: cpu → CPU CLIP; gpu → OpenCL CLIP; npu → HTP0 CLIP; hybrid → CPU CLIP; all VLM generations passed.
  • QCS9075M Linux: cross-compiled SDK successfully; cpu and hybrid -ngl 0 VLM cells passed, with logs confirming CPU CLIP.
  • QCS9075M Linux gpu: resolved GPUOpenCL and CLIP selected OpenCL, but the platform Adreno driver aborts with back-end instruction validation failed during kernel compilation.
  • QCS9075M Linux npu: CLIP selected HTP0, but only work with small model.
  • git diff --check

Tracks qcom-ai-hub/geniex#1562

Signed-off-by: RemiliaForever <remilia@koumakan.cc>
@RemiliaForever
RemiliaForever (RemiliaForever) merged commit 404a0c0 into main Sep 8, 2026
36 of 38 checks passed
@RemiliaForever
RemiliaForever (RemiliaForever) deleted the fix/vlm-device-selection branch September 8, 2026 07:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants