feat: allow user to adjust kv_transfer_config #2517
Conversation
alec-flowers
requested review from
nnshah1,
tanmayv25,
piotrm-nvidia,
ptarasiewiczNV,
ryanolson,
grahamking,
paulhendricks,
biswapanda,
tmonty12,
GuanLuo,
rmccorm4,
tedzhouhk,
kkranen,
jthomson04,
ishandhanani,
PeaBrane,
hhzhang16 and
a team
as code owners
alec-flowers
force-pushed
the
aflowers/kv-transfer-config-updates
branch
from
c6eda51 to
c380e13
Compare
alec-flowers
force-pushed
the
aflowers/kv-transfer-config-updates
branch
from
c380e13 to
f203490
Compare
|
@ZichengMa What do you think of this? The user has to do more to set up lmcache correctly (using the flags), however it gives them also more control and flexibility. |
|
Caution Review failedFailed to post review comments. Configuration used: .coderabbit.yaml 💡 Knowledge Base configuration:
You can enable these sources in your CodeRabbit configuration. 📒 Files selected for processing (5)
🧰 Additional context used🧬 Code Graph Analysis (2)components/backends/vllm/src/dynamo/vllm/main.py (1)
components/backends/vllm/src/dynamo/vllm/args.py (1)
🔇 Additional comments (5)
WalkthroughAdds a connector-based KV-transfer configuration to vLLM: new CLI flag --connector, validation, and dynamic side-channel port allocation only when needed (e.g., NIXL). Updates launcher scripts to pass --connector values (none/lmcache). Adjusts logging levels in main.py. Removes side_channel_port from Config and introduces connector_list. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant User
participant Launcher as Shell launcher (agg*.sh)
participant CLI as dynamo.vllm (CLI)
participant Args as Args parser
participant Ports as Port allocator
participant Engine as vLLM Engine
User->>Launcher: run agg*.sh (--connector X)
Launcher->>CLI: invoke dynamo.vllm with flags
CLI->>Args: parse args (--connector, --kv-transfer-config, ...)
Args->>Args: validate connectors (none/lmcache/nixl/kvbm)
alt user provided --kv-transfer-config
Args-->>Engine: use provided KVTransferConfig
else connector-based config
Args->>Args: create_kv_transfer_config()
alt connectors include "nixl"
Args->>Ports: allocate side-channel ports (per DP rank)
Ports-->>Args: base_side_channel_port
Args->>Args: set_side_channel_host_and_port(port)
end
Args-->>Engine: KVTransferConfig (LMCache/Nixl/KVBM/Multi)
end
Engine-->>User: start model service
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
Tip 🔌 Remote MCP (Model Context Protocol) integration is now available!Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
Status, Documentation and Community
|
|
I think this makes sense to me. But we may need to specify this in our documents. |
Signed-off-by: Hannah Zhang <[email protected]>
Overview:
Previously, we had under the hood handled overriding kv_transfer_config in vLLM. The main reason is it is a json argument and is quite verbose and at the time I had imagined we wanted to always enable NIXL.
With lmcache and kvbm being added in we are seeing more connectors. There are also multiple people who don't necessarily have nixl installed and want a way to turn off the nixl connector.
This refactor addresses this by allowing the user more control over --kv-transfer-config.
Details:
User can specify --kv-transfer-config and this will override everything else. This gives the most flexibility.
As a helper flag the user can also specify --connector and we will set up the connectors for them. The args we support art --connector [nixl, kvbm, lmcache] with the default being the nixl connector. This is to keep the old behavior impact. If you want to have no connector you can specify --connector none or --connector null or --connector and just leave it blank.
Where should the reviewer start?
Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Summary by CodeRabbit