Add vLLM Semantic Router (vllm-sr) integration

[carlory]

Summary

This PR adds support for the vLLM Semantic Router (vllm-sr) to RouterArena, along with significant infrastructure improvements for batch evaluation and parallel processing:

• New Router: Implements VLLMSR router that classifies queries by category and routes to appropriate models based on semantic intent
• Router Integration: Updates prediction file generation to support dynamic router instantiation via router_name and router_cls_name
• Documentation: Updates README links from GitHub to Code references

Key Changes

1. vLLM-SR Router Implementation (router_inference/router/vllm_sr.py)

   • Calls vllm-project/semantic-router classification API
   • Maps query categories to optimal models
   • Configurable category-to-model mapping

2. Results & Configuration

   • Added vllm-sr configuration (router_inference/config/vllm-sr.json)
   • Included prediction files for vllm-sr and vllm-sr-robustness

Close vllm-project/semantic-router#1005

🤖 Generated with Claude Code

[yl231]

@carlory Thanks for the PR. There is a small error in the workflow, and I am looking into that.

[yl231]

@carlory Hi, thank you for the submission again. The workflow failed because of two reasons.

1. the router inference has only 809 entries whereas the complete benchmark has 8400 entries.
   You could run this command to acquire the complete prediction file
   uv run python ./router_inference/check_config_prediction_files.py your-router full

2. Another issue is that the llm inference is not performed, so the generated_answer field is empty. If you want to use the LLMs that pipeline doesn't support, you could send me the model list and I could add these llm inference entrypoints to the pipeline.

Thank you!
Yifan

[carlory]

@yl231 What's the relationship between cost.json and model_cost.json files in the model_costs directory? They are both related to tracking the costs associated with different models. vllm-sr uses gpt-4o-mini in our model pool, but the cost information for gpt-4o-mini is not stored in model_cost.json. I can find the cost information for gpt-4o-mini in cost.json. Should I update the model_cost.json to include the cost information for gpt-4o-mini?

[carlory]

If you want to use the LLMs that pipeline doesn't support, you could send me the model list and I could add these llm inference entrypoints to the pipeline.

Thank you. Let me test it with all the existing models in the pipeline first. The uploaded file is outdated. I will regenerate it.

[yl231]

@yl231 What's the relationship between cost.json and model_cost.json files in the model_costs directory? They are both related to tracking the costs associated with different models. vllm-sr uses gpt-4o-mini in our model pool, but the cost information for gpt-4o-mini is not stored in model_cost.json. I can find the cost information for gpt-4o-mini in cost.json. Should I update the model_cost.json to include the cost information for gpt-4o-mini?

cost.json is a legacy file, and it should be replaced with model_cost.json. I will update the repo to reflect this. Sorry about the confusion!

[carlory]

You could run this command to acquire the complete prediction file
uv run python ./router_inference/check_config_prediction_files.py your-router full

cached results:

• claude-3-haiku-20240307: 809 items
• gemini-2.0-flash-001: 809 items
• gpt-4o-mini: 8400 items

@yl231 Cloud you provide full cached results for other models if possible? I don't have access to them right now.

[yl231]

You could run this command to acquire the complete prediction file
uv run python ./router_inference/check_config_prediction_files.py your-router full

cached results:

• claude-3-haiku-20240307: 809 items
• gemini-2.0-flash-001: 809 items
• gpt-4o-mini: 8400 items

@yl231 Cloud you provide full cached results for other models if possible? I don't have access to them right now.

Yes, I could do so! Could you give me a list of what you want?

[carlory]

Could you give me a list of what you want?

model list: claude-3-haiku-20240307 and gemini-2.0-flash-001.

[carlory]

Yes, I could do so! Could you give me a list of what you want?

@yl231 I regenerate the router_inference/predictions/vllm-sr.json with full dataset.

[github-actions]

Router Evaluation Results

Router: vllm-sr
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.6723
Accuracy	66.53%
Total Cost	$0.537853
Avg Cost per Query	$0.000064
Avg Cost per 1K Queries	$0.0640
Number of Queries	8400
Robustness Score	0.9095

Optimality Metrics

Metric	Value
Opt.Sel (Optimal Selection)	0.9410
Opt.Cost (Cost Efficiency)	0.9012
Opt.Acc (Accuracy vs Optimal)	1.0000

---

Evaluation completed by RouterArena automated workflow

[yl231]

@carlory I have run the inference and evaluation for vLLM-SR, and you rank in first place now! Congratulations!

[yl231]

Thank you for your submission!

[yl231]

@carlory I have run the inference and evaluation for vLLM-SR, and you rank in first place now! Congratulations!

Do you want this result be posted on the leaderboard? If so, I will update the README.md to reflect it.

Thanks again!

[carlory]

Cool. Thank you! Please update the README file. @yl231

[Xunzhuo]

thanks @carlory nice work! i think we can investigate more signal using in the arena

[Xunzhuo]

btw we recently re-trained long context multilingual bert, just got into the main, which may largely improve our accuracy as well. we can continuously update the scores in follow up

[carlory]

Selected Models (used in this PR):

drwxr-xr-x  14 kiki  staff   448B Jan 28 12:04 mmbert-intent-classifier-lora
drwxr-xr-x  13 kiki  staff   416B Jan 28 11:47 mmbert-intent-classifier-merged
drwxr-xr-x  14 kiki  staff   448B Jan 23 11:04 mom-domain-classifier
drwxr-xr-x  20 kiki  staff   640B Jan 23 11:04 mom-embedding-flash
drwxr-xr-x  21 kiki  staff   672B Jan 23 11:04 mom-embedding-light
drwxr-xr-x  15 kiki  staff   480B Jan 23 11:04 mom-embedding-pro
drwxr-xr-x  11 kiki  staff   352B Jan 23 11:04 mom-feedback-detector
drwxr-xr-x  10 kiki  staff   320B Jan 23 11:04 mom-halugate-detector
drwxr-xr-x  10 kiki  staff   320B Jan 23 11:04 mom-halugate-explainer
drwxr-xr-x  13 kiki  staff   416B Jan 23 11:04 mom-halugate-sentinel
drwxr-xr-x  14 kiki  staff   448B Jan 23 11:04 mom-jailbreak-classifier
drwxr-xr-x  14 kiki  staff   448B Jan 23 11:04 mom-pii-classifier

I removed some models because they need more memory than my laptop can provide.

btw we recently re-trained long context multilingual bert, just got into the main, which may largely improve our accuracy as well. w

Ok, I can do it in a follow-up PR.

[Xunzhuo]

FYI vllm-project/semantic-router#1227

[Xunzhuo]

Thanks