Skip to content

Vulkan: Fix Deepseek V2 inference by making ggml_vk_op_supports_incontiguous(GGML_OP_RMS_NORM) return true #12960

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

stduhpf
Copy link
Contributor

@stduhpf stduhpf commented Apr 15, 2025

Fixes #12956

It seems to work perfectly fine so far, but i don't really know what I'm doing, so maybe it's actually not supported and this is breaking something else.

(Also maybe the same should be done for GGML_OP_RMS_NORM_BACK?)

@github-actions github-actions bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Apr 15, 2025
@jeffbolznv
Copy link
Collaborator

I don't think the shader actually supports noncontiguous tensors because it doesn't take into account the strides the way the cuda kernel does. If this is supported, we should also be able to return true from ggml_backend_vk_device_supports_op, but when I do that it fails the backend tests.

@stduhpf
Copy link
Contributor Author

stduhpf commented Apr 16, 2025

I see I was kinda expecting that kind of problem. Still, doing this workaround seems fixes deepseek v2 inference on my machine.

@stduhpf stduhpf marked this pull request as draft April 16, 2025 09:18
@0cc4m
Copy link
Collaborator

0cc4m commented Apr 16, 2025

We would have to modify the shader to allow noncontiguous tensors, then.

@jeffbolznv
Copy link
Collaborator

@stduhpf do you want to try to make the shader changes yourself? If not, I can probably do it soon.

@stduhpf
Copy link
Contributor Author

stduhpf commented Apr 16, 2025

I don't think I have the skills to do it myself, I can give it a try, but I will probably not be able to figure it out.

@stduhpf stduhpf closed this Apr 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ggml changes relating to the ggml tensor library for machine learning Vulkan Issues specific to the Vulkan backend
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Eval bug: Deepseek V2 Lite no longer working with Vulkan (assert fail during tg)
3 participants