Skip to content

Support for KVCacheFusion Optimization #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 154 commits into from
Closed

Support for KVCacheFusion Optimization #9

wants to merge 154 commits into from

Conversation

@cavusmustafa
Copy link
Collaborator

@cavusmustafa cavusmustafa commented Sep 30, 2025

  • Translation updates to support KVCacheFusion in GPU plugin
  • 4D inputs for SDPA which will enable further optimizations for GPU
  • The changes are only effective for stateful model for now.

YangleiZouIntel and others added 30 commits September 26, 2025 15:56
@YangleiZouIntel @wine99
Add ggml-openvino base files
e57aeda
@zhanmyz @wine99
add openvino as optional backend for Llama.cpp ggml
e169d0c
@zhanmyz @wine99
* Configure the device(default CPU) that uses OpenVINO to compile th…
5db7996 
…e model

 * Add OpenVINO ADD operator to Llama.cpp. The output is somewhat abnormal and needs further debugging.
@zhanmyz @wine99
Solve the issue of abnormal model output caused by using OpenVINO ADD…
d632239 
… operator
@zhanmyz @wine99
Add OpenVINO MUL operator to GGML of Llama.cpp.
f78c52a
@zhanmyz @wine99
Add compile options
6d4d318
@zhanmyz @wine99
add OpenVINO frontend convert process steps
a0495c5
@zhanmyz @wine99
add get openvino available ops function
0d6cf3a
@yumengbo @wine99
Add PoC of integration of openvino frontend. Main changes: ggml-ov-fr…
d191a77 
…ontend-utils, GraphIterator, Decoder
@yumengbo @wine99
Implement GgmlOvDecoder. Add dump functions.
451e201
@yumengbo @wine99
Convert subgraph with add, sub, mul, div op to ov model and do infer …
42641b6 
…on openvino device
@yumengbo @wine99
Add GGML_OV_FRONTEND option. Add readme.
a26a28b
@yumengbo @wine99
Change output for infer request to set output tensor. Support scale, …
59495bf 
…view op.
@zhanmyz @wine99
add GET_ROWS operator of OpenVINO to GGML of llama.cpp
e17fcbf
@zhanmyz @wine99
Update build.md and add operation mapping(GGML to OpenVINO)
d844353
@zhanmyz @wine99
add the rms_norm operator implemented using OpenVINO to the GGML back…
13a1af6 
…end of llama.cpp
@yumengbo @wine99
Fix issue for output memory copy of infer request
58cd50c
@yumengbo @wine99
Change to implementation following pytorch frontend
108ac50
@yumengbo @wine99
Add support for UNARY SILU op . Fix pytorch impl bugs.
8a12ffa
@yumengbo @wine99
Support Softmax op
18d53db
@yumengbo @wine99
Support Softmax op
f4856e3
@yumengbo @wine99
Support ROPE op.
87ea61d
@zhanmyz @wine99
Add support for RMS_NORM OP
d200348
@zhanmyz @wine99
Add MUL_MAT,CPY,CONT as operators implemented in OpenVINO for GGML ba…
a582b3e 
…ckend
@zhanmyz @wine99
Move CPY from GGML OV Backend to OV Frontend
e32ac44
@zhanmyz @wine99
add implementation of MUL_MAT, CPY, CONT of GGML ops using OV ops
d1983ec
@zhanmyz @wine99
add implementation of CPY when the output tensor is non-contiguous
dadaa6a
@zhanmyz @wine99
add tmp source code files
83fe0fa
@zhanmyz @wine99
Execute singel CONT operator is OK
8f43a86
@zhanmyz @wine99
Execute CONT & VIEW operators in OV Frontend is OK
3d292bc
wine99 and others added 21 commits September 26, 2025 15:58
@github-actions github-actions bot added documentation Improvements or additions to documentation testing devops ggml labels Sep 30, 2025
@wine99 wine99 force-pushed the dev_backend_openvino branch 3 times, most recently from ade4a2d to f89292d Compare October 15, 2025 08:06
@ravi9 ravi9 closed this Oct 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

devops documentation Improvements or additions to documentation ggml testing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@cavusmustafa @ravi9 @YangleiZouIntel @zhanmyz @yumengbo @wine99 @virajwad