Skip to content

Support for KVCacheFusion Optimization #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 154 commits into
base: dev_backend_openvino
Choose a base branch
from
Draft

Support for KVCacheFusion Optimization #9

wants to merge 154 commits into from

Conversation

cavusmustafa
Copy link
Collaborator

  • Translation updates to support KVCacheFusion in GPU plugin
  • 4D inputs for SDPA which will enable further optimizations for GPU
  • The changes are only effective for stateful model for now.

YangleiZouIntel and others added 30 commits September 26, 2025 15:56
@YangleiZouIntel @wine99
Add ggml-openvino base files
e57aeda
@zhanmyz @wine99
add openvino as optional backend for Llama.cpp ggml
e169d0c
@zhanmyz @wine99
* Configure the device(default CPU) that uses OpenVINO to compile th…
5db7996 
…e model

 * Add OpenVINO ADD operator to Llama.cpp. The output is somewhat abnormal and needs further debugging.
@zhanmyz @wine99
Solve the issue of abnormal model output caused by using OpenVINO ADD…
d632239 
… operator
@zhanmyz @wine99
Add OpenVINO MUL operator to GGML of Llama.cpp.
f78c52a
@zhanmyz @wine99
Add compile options
6d4d318
@zhanmyz @wine99
add OpenVINO frontend convert process steps
a0495c5
@zhanmyz @wine99
add get openvino available ops function
0d6cf3a
@yumengbo @wine99
Add PoC of integration of openvino frontend. Main changes: ggml-ov-fr…
d191a77 
…ontend-utils, GraphIterator, Decoder
@yumengbo @wine99
Implement GgmlOvDecoder. Add dump functions.
451e201
@yumengbo @wine99
Convert subgraph with add, sub, mul, div op to ov model and do infer …
42641b6 
…on openvino device
@yumengbo @wine99
Add GGML_OV_FRONTEND option. Add readme.
a26a28b
@yumengbo @wine99
Change output for infer request to set output tensor. Support scale, …
59495bf 
…view op.
@zhanmyz @wine99
add GET_ROWS operator of OpenVINO to GGML of llama.cpp
e17fcbf
@zhanmyz @wine99
Update build.md and add operation mapping(GGML to OpenVINO)
d844353
@zhanmyz @wine99
add the rms_norm operator implemented using OpenVINO to the GGML back…
13a1af6 
…end of llama.cpp
@yumengbo @wine99
Fix issue for output memory copy of infer request
58cd50c
@yumengbo @wine99
Change to implementation following pytorch frontend
108ac50
@yumengbo @wine99
Add support for UNARY SILU op . Fix pytorch impl bugs.
8a12ffa
@yumengbo @wine99
Support Softmax op
18d53db
@yumengbo @wine99
Support Softmax op
f4856e3
@yumengbo @wine99
Support ROPE op.
87ea61d
@zhanmyz @wine99
Add support for RMS_NORM OP
d200348
@zhanmyz @wine99
Add MUL_MAT,CPY,CONT as operators implemented in OpenVINO for GGML ba…
a582b3e 
…ckend
@zhanmyz @wine99
Move CPY from GGML OV Backend to OV Frontend
e32ac44
@zhanmyz @wine99
add implementation of MUL_MAT, CPY, CONT of GGML ops using OV ops
d1983ec
@zhanmyz @wine99
add implementation of CPY when the output tensor is non-contiguous
dadaa6a
@zhanmyz @wine99
add tmp source code files
83fe0fa
@zhanmyz @wine99
Execute singel CONT operator is OK
8f43a86
@zhanmyz @wine99
Execute CONT & VIEW operators in OV Frontend is OK
3d292bc
wine99 and others added 24 commits September 26, 2025 15:58
@wine99
Add custom quant type: q8_1_c, q4_0_128
7dfcb6b
@wine99
Set m_is_static=false as default in decoder
b3bb37c
@wine99
Simpilfy translation of get_rows
7b526f2
@wine99
Fix after rebasing
ad06c0b
@wine99
Improve debug util; Eliminate nop ReshapeReshape
eb30874
@wine99
STYLE: make get_types_to_requant a function
169d067
@wine99
Support BF16 model
d6db0ae
@wine99
Fix NPU compile
121113f
@wine99
WA for npu 1st token acc issue
382668f
@wine99
Apply EliminateZP only for npu
ef88e6f
@wine99
Add GeGLU
2774507
@wine99
Fix Hunyuan
3c0c0c7
@wine99
Support iSWA
2e8b899
@wine99
Fix NPU accuracy
158129f
@wine99
Fix ROPE accuracy when freq_scale != 1
d00aa4c
@wine99
Minor: not add attention_size_swa for non-swa model
e855e09
@wine99
Minor refactor
8c95d78
@wine99
Add Q5_K to support phi-3-q4_k_m
6db0773
@wine99
Requantize Q6_K (gs16) to gs32 on GPU
a436bc7
@wine99
Fix after rebasing
76ab76e
@cavusmustafa
initial kvcachefusion support
c7ac35d
@cavusmustafa
openvino backend 4d sdpa
767f192
@cavusmustafa
openvino backend fix for values shape issue
5d8f984
@cavusmustafa
disable sdpa optimization
a5fbb53
@github-actions github-actions bot added documentation Improvements or additions to documentation testing devops ggml labels Sep 30, 2025
@wine99 wine99 force-pushed the dev_backend_openvino branch from e727c65 to 66e503b Compare October 11, 2025 05:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

devops documentation Improvements or additions to documentation ggml testing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@cavusmustafa @YangleiZouIntel @zhanmyz @yumengbo @wine99 @virajwad @ravi9