Check KV4 compatibility with attention backends and add KV4 support to the attention_backend doc #14467
+105
−24
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… FP4 note - Introduce FP4 KV cache support in the backend matrix. - Add note on FA4 + KV4 scenario. Signed-off-by: Ho-Ren (Jack) Chuang <[email protected]>
- Add _handle_kv4_compatibility() to validate backend choices for KV4 scenarios. - Warns on potential edge-case incompatibilities. - Asserts correct decode_attention_backend for FA4 + MLA/MHA and non-FA4 + MLA/MHA setups. - Raises error if KV4 is used on non-CUDA platforms. Signed-off-by: Ho-Ren (Jack) Chuang <[email protected]>
JackChuang
requested review from
Fridge003,
Ying1123,
hnyls2002,
ispobock and
merrymercy
as code owners
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
github-actions
bot
added
the
documentation
Fridge003
reviewed
Dec 5, 2025
| port=self.dist_port, | ||
| ) | ||
|
|
||
| def _handle_kv4_compatibility(self): |
Collaborator
There was a problem hiding this comment.
Can we move this function to server_args.py, somewhere after _handle_attention_backend_compatibility
Contributor
Author
There was a problem hiding this comment.
No problem. I will fix this as soon as I can. Thank you for your feedback!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Prevent users from using KV4 with incompatible attention backends by clearly documenting supported backends and enforcing runtime checks.
Improves reliability and reduces runtime errors by ensuring users cannot accidentally run KV4 with unsupported attention backends.
Modifications
Ensure code executes after default settings are set by placing it in model_runner.py instead of server_args.py.
Description / Changes:
1. Backend documentation updates
• Added FP4 KV cache column to the MLA/MHA backend table.
• Clarifies which backend combinations are supported with FP4 KV caches.
2. ModelRunner updates
• Added _handle_kv4_compatibility() to check KV4 compatibility with attention backends at runtime.
• Logs warnings for potential edge-case incompatibilities.
• Adds assertions to enforce correct decode_attention_backend for FA4 + MLA/MHA and non-FA4 + MLA/MHA setups.
• Raises an error if KV4 is used on non-CUDA platforms.
Testing
The compatibility results are tested on B200 (sm100), using Qwen3-235B-A22B as MHA and DeepSeek-R1-0528-FP4 as MLA to test.
Next (WIP)
Test kv4 with fa3 and flashmla backend on sm90 to complete the table. I will send another PR for this.
Checklist