Skip to content

Auto Model Detection, Simplified Args, and Overflow Safeguards - #201

Merged
guoqingbao merged 5 commits into
EricLBuehler:masterfrom
guoqingbao:develop
Jul 26, 2025
Merged

Auto Model Detection, Simplified Args, and Overflow Safeguards#201
guoqingbao merged 5 commits into
EricLBuehler:masterfrom
guoqingbao:develop

Conversation

@guoqingbao

Copy link
Copy Markdown
Collaborator

This PR includes three major improvements:


1) Automatic model type detection

The model type is now automatically inferred based on the input path or file. This simplifies usage significantly.

Examples:

Safetensors model:

cargo run --release --features cuda,nccl -- --w /home/phi3-3.8b

GGUF model:

cargo run --release --features cuda,nccl -- --f /home/DeepSeek-R1-Distill-Llama-8B-Q2_K.gguf

Multi-rank inference:
This automatically enables multi-process mode using daemon processes:

cargo run --release --features cuda,nccl -- --w /home/QwQ-32B/ --d 4,5,6,7

ISQ inference:

cargo run --release --features cuda,nccl -- --w /home/phi3-3.8b --isq q4k

2) Simplified CLI parameters for improved usability

To make the CLI more concise and user-friendly, key arguments have been shortened:

  • Safetensors model path: --weight-path--w
  • GGUF model path: --weight-file--f
  • Model ID: --model-id--m
  • Device IDs: --device-ids--d
  • Server port: --port--p

Additionally:

  • The --multi-process flag has been removed. Multi-process mode is now enabled automatically for multi-rank inference.
  • You can disable multi-process mode and force multithreaded execution by passing the --multithread flag (useful for easier debugging).

3) Improved safeguards against overflow

This update introduces a mechanism to prevent generation overflow.
It checks the available KV cache and model context length before accepting new requests, ensuring that inputs exceeding the model’s capacity are rejected early and safely.

@guoqingbao guoqingbao mentioned this pull request Jul 26, 2025
@guoqingbao
guoqingbao merged commit 0e663d9 into EricLBuehler:master Jul 26, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant