Skip to content

Auto Model Detection, Simplified Args, and Overflow Safeguards - #203

Merged
guoqingbao merged 1 commit into
EricLBuehler:masterfrom
guoqingbao:develop
Jul 26, 2025
Merged

Auto Model Detection, Simplified Args, and Overflow Safeguards#203
guoqingbao merged 1 commit into
EricLBuehler:masterfrom
guoqingbao:develop

Conversation

@guoqingbao

Copy link
Copy Markdown
Collaborator

This PR includes three major improvements:


1) Automatic model type detection

The model type is now automatically inferred based on the input path or file, significantly simplifying usage.

Examples:

Safetensors model:

cargo run --release --features cuda,nccl -- --w /home/phi3-3.8b

GGUF model:

cargo run --release --features cuda,nccl -- --f /home/DeepSeek-R1-Distill-Llama-8B-Q2_K.gguf

Multi-rank inference:
Automatically enables multi-process mode (via daemon processes):

cargo run --release --features cuda,nccl -- --w /home/QwQ-32B/ --d 4,5,6,7

ISQ inference:

cargo run --release --features cuda,nccl -- --w /home/phi3-3.8b --isq q4k

2) Simplified CLI parameters for improved usability

To make the command-line interface more concise and user-friendly, key arguments have been shortened:

  • Safetensors model path: --weight-path--w
  • GGUF model path: --weight-file--f
  • Model ID: --model-id--m
  • Device IDs: --device-ids--d
  • Server port: --port--p

Also:

  • The --multi-process flag has been removed. Multi-process mode is now automatically enabled for multi-rank inference.
  • To force multithreaded execution (e.g., for easier debugging), use the --multithread flag.

3) Improved safeguards against overflow

This update introduces a mechanism to prevent generation overflow.
It checks the available KV cache and model context length before accepting new requests, ensuring that inputs exceeding the model’s capacity are rejected early and safely.

@guoqingbao
guoqingbao merged commit 4158aad into EricLBuehler:master Jul 26, 2025
11 checks passed
ephemeral2eternity pushed a commit to ephemeral2eternity/candle-vllm that referenced this pull request Nov 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant