[Feature] Support file:// URL format for multimodal inputs

[ppraneth]

Motivation

Currently, the OpenAI server URL only supports standard path formats and does not accept URLs prefixed with file://. vLLM already supports this input format. This PR adds support for file:// URL formats in image and video loading functions, enabling sglang to handle local file URLs properly.

Modifications

• Updated load_image, get_image_bytes, and load_video in sglang/srt/utils/common.py to explicitly handle the file:// prefix using urllib.parse.urlparse.
• Added logic to extract the file path from the file:// URL and pass it to the underlying loading functions (Image.open, open, VideoReader).

Resolves #14479

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.
• Work with maintainers to merge your PR. See the PR Merge Process

[gemini-code-assist]

Summary of Changes

Hello @ppraneth, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances sglang's multimodal input capabilities by adding support for the file:// URL format. Previously, the system did not recognize this prefix for local files, limiting its ability to process images and videos directly from the local filesystem. By integrating file:// parsing, sglang can now seamlessly load and utilize local media files, improving flexibility and aligning with input formats supported by other systems like vLLM.

Highlights

• file:// URL Support: Introduced support for the file:// URL format in multimodal input functions, allowing sglang to properly handle local file paths for images and videos.
• URL Parsing Logic: Implemented urllib.parse.urlparse and unquote to correctly extract and decode local file paths from file:// URLs before processing them.
• Function Updates: Modified load_image, get_image_bytes, and load_video functions in sglang/srt/utils/common.py to incorporate the new file:// handling logic.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds support for file:// URL formats for multimodal inputs, which is a valuable feature. The changes in load_image and get_image_bytes are well-implemented. However, in load_video, the logic for handling file paths is a bit redundant and inconsistent. I've provided a suggestion to refactor this part to improve clarity and maintainability.

[gemini-code-assist]

The logic for handling local file paths and file:// URIs can be simplified. The two elif blocks are redundant and can be combined into a single, more robust check. This also resolves the inconsistency where one branch modifies video_file and the other doesn't, and it makes it clear that a plain file path is always passed to VideoReader.

The comment on line 956 also appears to be incorrect with the new logic, as the path is extracted from file:// URIs before being passed to VideoReader.

Suggested change

	elif video_file.startswith("file://"):
	video_file = unquote(urlparse(video_file).path)
	vr = VideoReader(video_file, ctx=ctx)
	# `urlparse` supports file:// paths, and so does VideoReader
	elif os.path.isfile(urlparse(video_file).path):
	elif os.path.isfile(unquote(urlparse(video_file).path)):
	vr = VideoReader(video_file, ctx=ctx)
	# Handle local file paths, including file:// URIs
	elif os.path.isfile(parsed_path := unquote(urlparse(video_file).path)):
	vr = VideoReader(parsed_path, ctx=ctx)