Support Phi-4 Multi-Modal (text + vision only)

[lifuhuang]

Motivation

Support Phi4-MM model with text + vision.

Modifications

This change introduced the basic text + image support.

It's worth noting that the current MMMU run (without LoRA) is lower than advertised because Phi4MM relies on LoRA for full image understanding capabilities. However, LoRA support requires refactoring / generalizing the existing SGL LoRA handling, which will hopefully be addressed in this separate PR: #6585

Example: degraded image understanding without LoRA (MMMU is only 38). As comparison in our local branch (#6585) with LoRA, MMMU is boosted to ~50:

TODO in this PR:

• add unit tests
• clean-up styling issues

TODO in follow-up PR (ordered by priority):

1. Precomputed feature support.
2. LoRA support (required for multi-image understanding)
3. SGLang LoRA compatibility with CUDA Graph and Radix Attention
4. Refactor SGL MM processor logic support for support the original token variable image token (e.g., <image_1>)
5. perf optimization
6. audio support
7. pipeline parallelism support

Tracked in #6544

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[zhaochenyang20]

@mickqian @yizhang2077

[mickqian]

Better to be merged after #4969 , due to some change to the omni model processing and testing

[lifuhuang]

Better to be merged after #4969 , due to some change to the omni model processing and testing

Hi @mickqian, thank you so much for reviewing my PR :)

Can you share more details about the concern you have such that I can test them locally? JFYI, I was able to merge your branch mickqian:qwen2.5-omni locally without conflict and get a green TestOpenAIVisionServer run for phi4mm.

[mickqian]

Can you share more details

It's mostly that, for omni models, there's a new TestOpenaiOmniServer. And yes, you can cherry-pick it.

I just noticed audio input is not supported this time.