Repository navigation
snap: bump version to llama.cpp b10549 - #313
Merged
Merged
Conversation
Collaborator
|
🚫 Moon Skywalker — PR Blacklisted After 7 retry attempts, CI is still failing on this PR. I've created issue #314 to track this for manual review. This PR will no longer be automatically monitored. — Moon Skywalker |
talhaHavadar
enabled auto-merge (rebase)
August 24, 2026 06:32
talhaHavadar
approved these changes
Aug 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains the following updates:
b10486→b10549Release Notes
ggml-org/llama.cpp (ggml-org/llama.cpp)
vb10549Compare Source
Details
TP: enable tensor split for LFM2/LFM2MOE (#26993)
Assisted-by: deepseek-v4-flash
Website:
Attestations:
macOS/iOS:
Linux:
Android:
Windows:
openEuler:
UI:
vb10545Compare Source
Details
metal : clamp K extent in tensor API mat-mat kernel for K not a multiple of 32 (#27450)
The Tensor API mat-mat path of kernel_mul_mm (GGML_METAL_HAS_TENSOR) fed a
static K=32 tile to the matmul2d op on every iteration. On the last, partial
K tile (ne00 % 32 != 0) the src1 slice extends past the K extent of the
tensor, and the op reads those out-of-bounds elements (undefined behavior per
the MSL specification, section 2.22.2). Depending on stale memory contents,
this corrupted the result or produced NaN.
Make the matmul2d op use dynamic_extent for K, and clamp the K extent of both
operand tensor views to the remaining valid K range (min(32, K - loop_k)) per
iteration, so the op reads exactly the valid K range on every iteration
(mirroring the tail handling of the MPP matmul2d examples). On K-aligned
inputs the clamp degenerates to the full 32-wide tile: the only difference
from the static-K op is that the dynamic-K op derives K from the operand
extents and edge-checks the tile against the tensor extents (a handful of
integer ops per iteration).
Add test-backend-ops MUL_MAT cases with K not a multiple of 32 to exercise
the unaligned K path.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
Website:
Attestations:
macOS/iOS:
Linux:
Android:
Windows:
openEuler:
UI:
vb10541Compare Source
Details
mtmd: add --mmproj-device argument (#23255)
feat: add --mmproj-device arg & backwards compatible MTMD_BACKEND_DEVICE env var
feat: load mmproj device backend immediately, add -mmdev shortflag
fix: its a pointer now get the name
clean up
gen docs
nits
Co-authored-by: Xuan Son Nguyen son@huggingface.co
Website:
Attestations:
macOS/iOS:
Linux:
Android:
Windows:
openEuler:
UI:
vb10539Compare Source
Details
vulkan: FA MMQ should use fp32 for Q quantization calculations (#27413)
Codex found that qd could be a denorm and 1/qd would overflow.
Website:
Attestations:
macOS/iOS:
Linux:
Android:
Windows:
openEuler:
UI:
vb10538Compare Source
Details
metal : dequant kv cache only for large batches (#27438)
Website:
Attestations:
macOS/iOS:
Linux:
Android:
Windows:
openEuler:
UI:
vb10537Compare Source
Details
CI: Use LLVM's OpenMP over MSVC_DEBUG_non_redist on Windows (#26678)
Currently, we ship the non-redist debug version of microsoft's libomp.
This PR changes this to official LLVM's release, also packaging
the license as needed.
Remove LLVM SHA from job name to increase legibility
Add temp validations to CI
Revert "Add temp validations to CI"
This reverts commit
eef97c8.Build OpenMP in CI
Make OpenMP fetch self-contained in cmake and cache in CI
Robustify Licens-packaging
Remove stale reference in docs/build.md
No longer package base license in release
This was scope-creep
Add explanatory comment to OpenMP license
Remove arm64 smoke
Forgot this during conflict resolution during rebase of
c54c0e9Remove GGML_OPENMP_FETCH_CACHE_DIR as requested by @CISC
whitespace changes
Website:
Attestations:
macOS/iOS:
Linux:
Android:
Windows:
openEuler:
UI:
vb10536Compare Source
Details
server: (router) lazy-load startup_models after main setup (#27424)
server: (router) lazy-load startup_models after main setup
only allow is_first_load to populate it
nits
nits 2
Website:
Attestations:
macOS/iOS:
Linux:
Android:
Windows:
openEuler:
UI:
vb10534Compare Source
Details
CUDA: adding switch points per HW and quant type to tune the mvq->MMQ decode crossover (#26079)
Add a runtime override of the mul_mat_vec_q -> MMQ batch crossover
(default MMVQ_MAX_BATCH_SIZE). Lowering it routes batches above the
threshold from the CUDA-core vector kernel to the int8 MMQ tensor-core
path, which is faster once quantized decode becomes compute-bound at
B>1 (measured +23-41% at B=8 on RTX 5090 for Q4_K dense, no low-batch loss).
The value is parsed once and clamped to [1, MMVQ_MAX_BATCH_SIZE], since
mul_mat_vec_q asserts ncols_dst <= that; invalid input warns and falls
back to the default. The override is applied consistently in both the
mul_mat_vec_q and MUL_MAT_ID dispatch paths. Default behavior unchanged.
Added Blackwell specific switch point, to reduce dependence on runtime env var.
Add per-HW switch point values for DGX Spark and removing runtime env var
Adding switch points for Ada, tested on RTX 4090
Modifying DGX Spark numbers based on latest run and adding some comments and small functional changes relating to MoE
Reverting an unnecessary conditional
Update ggml/src/ggml-cuda/mmvq.cu
Co-authored-by: praneshgo 227579474+praneshgo@users.noreply.github.com
Co-authored-by: Oliver Simons osimons@nvidia.com
Website:
Attestations:
macOS/iOS:
Linux:
Android:
Windows:
openEuler:
UI:
vb10533Compare Source
Details
common : gracefully fallback on unsupported regex patterns in JSON schema (#26939)
Website:
Attestations:
macOS/iOS:
Linux:
Android:
Windows:
openEuler:
UI:
vb10532Compare Source
Details
metal : dequantize quantized KV to F16 before flash attention (#27390)
Add a preprocessing pass for GGML_OP_FLASH_ATTN_EXT on the Metal backend:
when the KV cache is quantized (Q8_0 for now), dequantize K and V into a
contiguous F16 scratch buffer and run the existing F16 flash attention
kernels on it, instead of the in-kernel dequantization path.
one thread per quant block (K then V), stride-aware so permuted KV is
supported; instantiated for Q8_0 (extending to Q4_0/Q4_1/Q5_0/Q5_1 is
one instantiation + one gate case)
regardless of head sizes, GQA ratio or n_kv; the attention kernels
themselves are untouched
(ggml_metal_op_flash_attn_ext_extra_dequant_f16); the KV pad kernel
reads the dequantized buffers when the path is active
f16 kernels and contiguous strides
Verification (M2 Ultra):
new q8_0 eval cases (decode/prompt, permuted, sinks+ALiBi+softcap,
kv=113 pad path, kv=16384)
f16 KV reference (PPL 1.0008 vs 1.0008)
Assisted-by: pi:llama.cpp/Qwen3.8-27B
Simplify kernel_flash_attn_ext_dequant_to_f16: it now dequantizes a single
tensor (its own ne/nb and dst) with no is_v branching, and the op dispatches
it twice with the same pipeline - once for K and once for V. The kargs
struct shrinks to a single ne/nb set plus nblocks.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
The dequant pass now covers all quantized KV types supported by the Metal
flash attention kernels. The dequant kernel, kargs, scratch allocation and
dispatch are type-generic, so each type is one kernel instantiation plus one
gate case.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
In MLA-based models, the V of the FA op is a view of K (the first ne20
elements of each K row); the dequantized V is then a view of the dequantized
K, so skip the second dequant dispatch, do not reserve the V scratch region,
and let the pad and attention kernels read V from the K F16 buffer with K's
strides. The detection follows the CUDA backend:
V->view_src && (V->view_src == K || (V->view_src == K->view_src && V->view_offs == K->view_offs))
Also fix the FA pipeline getters: ns10/ns20 are function constants baked into
the kernels and must be the actual K/V row widths as seen by the kernel. The
dispatch now passes them explicitly (nb11_attn/nb10_attn, nb21_attn/nb20_attn)
instead of the getters assuming contiguous F16 KV (ns20 = dv), which was wrong
when V is read from K with K's row pitch (e.g. 576 vs 512).
New test cases: 576/512 q8_0 (MLA shape, V is a view of K) at kv=113 (KV pad),
nb=1 (vec) and nb=64 (non-vec).
Assisted-by: pi:llama.cpp/Qwen3.8-27B
Assisted-by: pi:llama.cpp/Qwen3.8-27B
Assisted-by: pi:llama.cpp/Qwen3.8-27B
Assisted-by: pi:llama.cpp/Qwen3.8-27B
cont : clean-up
cont : remove TODO
Website:
Attestations:
macOS/iOS:
Linux:
Android:
Windows:
openEuler:
UI:
vb10531Compare Source
Details
Revert "tensor-split meta backend fixes (#26502)" (#27433)
This reverts commit
d59d455.Website:
Attestations:
macOS/iOS:
Linux:
Android:
Windows:
Configuration
📅 Schedule: (UTC)
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR was generated by Mend Renovate. View the repository job log.