Skip to content

Commit 90cb787

Browse files
committed
fix(embedding): set kv_unified=True when embedding=True to enable batch processing
1 parent 4a1a8ec commit 90cb787

2 files changed

Lines changed: 3 additions & 1 deletion

File tree

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,3 +179,5 @@ cython_debug/
179179

180180
# downloaded model .bin files
181181
docker/open_llama/*.bin
182+
test_model.gguf
183+
.claude/

‎llama_cpp/llama.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -402,7 +402,7 @@ def __init__(
402402
self.n_batch,
403403
llama_cpp.llama_max_parallel_sequences(),
404404
)
405-
405+
self.context_params.kv_unified = True
406406
self._ctx = self._stack.enter_context(
407407
contextlib.closing(
408408
internals.LlamaContext(

0 commit comments

Comments
 (0)