Add LiteLLM proxy provider support #161
Conversation
qdrddr
commented
Nov 17, 2025
qdrddr
commented
- Add new LiteLLMProvider class using pydantic-ai's native LiteLLMProvider
- Register litellm_proxy provider in PROVIDER_REGISTRY
- Support OpenAI-compatible API endpoints via LiteLLM proxy
- Enable use of multiple LLM providers through a unified proxy interface
Summary of ChangesHello @qdrddr, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request integrates LiteLLM as a new model provider, allowing the system to connect to a wider array of large language models through a standardized, OpenAI-compatible interface. This enhancement simplifies the process of switching between or utilizing multiple LLM services, improving the system's flexibility and extensibility. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
qdrddr
force-pushed
the
add-litellm-proxy-provider
branch
from
54f1597 to
5096773
Compare
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for LiteLLM as a provider, allowing the use of various LLM backends through a unified proxy interface. The implementation includes a new LiteLLMProvider class and dynamically registers it. The changes are well-structured. My main feedback is to improve the robustness of the new provider by adding a health check to validate that the LiteLLM proxy is running, similar to how the OllamaProvider is handled. This will provide better error messages to users.
- Add new LiteLLMProvider class using pydantic-ai's native LiteLLMProvider - Register litellm_proxy provider in PROVIDER_REGISTRY - Support OpenAI-compatible API endpoints via LiteLLM proxy - Enable use of multiple LLM providers through a unified proxy interface - Upgrade pydantic-ai-slim to 1.0.1 (required for LiteLLM provider support) - Add check_litellm_proxy_running helper for robust proxy validation - Validate proxy is running before creating models (consistent with OllamaProvider)
qdrddr
force-pushed
the
add-litellm-proxy-provider
branch
from
3078f96 to
c069209
Compare
- Upgrade pydantic-ai-slim to 1.18.0 which includes LiteLLM provider support - Fix circular import in providers/base.py by moving LiteLLM import after registry definition - Fix circular import in providers/litellm.py by using local import for check_litellm_proxy_running - Add debug logging when LiteLLM provider is not available
vitali87
left a comment
vitali87
left a comment
There was a problem hiding this comment.
LGTM! One minor issue:
The .env.example adds "Example 5" but one already exists at line 39 ("Example 5: Mixed - OpenAI orchestrator + Google cypher"). Should be "Example 6".
|
@qdrddr please address the minor issue I mentioned and resolve the merge conflict to get this into master. Thanks |
|
@vitali87 .env.example is fixed |
vitali87
left a comment
vitali87
left a comment
There was a problem hiding this comment.
Thanks for the update!
Same situation as PR #162 - I've recently refactored to centralize strings and enforce stricter coding standards. This PR needs adjustments to align with CONTRIBUTING.md.
Question: The codebase now uses pydantic-ai>=1.27.0. Does the LiteLLM provider API still work the same way? Please verify compatibility after rebasing.
Option 1: You address:
- Rebase on latest main
- Verify LiteLLM API compatibility with pydantic-ai 1.27.0
- Remove docstrings (module, functions)
- Move log message to
logs.py - Install pre-commit hooks:
pre-commit install
Option 2: I can make these adjustments myself and merge, with attribution to you.
Let me know which option you'd prefer!
2 participants
