Skip to content

fix: resolve schema conversion errors for Gemini in evaluator optimizer #215

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

monotykamary
Copy link
Contributor

This pull request addresses errors encountered with the evaluator optimizer pattern when using Google Gemini models. The primary issue was related to the conversion and handling of Pydantic model schemas for the Google Generative AI API.

Changes Made:

  1. src/mcp_agent/llm/providers/augmented_llm_google_native.py:

    • Modified the logic for determining response_schema.
    • When a Pydantic schema is provided, it's now explicitly converted to a Google Schema object using self._converter.json_schema_to_google_schema. This ensures that the schema structure is compatible with what the Google API expects for structured output.
    • If no explicit schema is provided, it falls back to inferring the schema using _get_schema_type(model).
    • Added comments to clarify the schema handling logic.
  2. src/mcp_agent/llm/providers/google_converter.py:

    • Enhanced json_schema_to_google_schema: This method was significantly improved to robustly convert Pydantic-generated JSON schemas to google.genai.types.Schema.
      • Implemented proper handling of local $refs within the JSON schema, resolving them against the root_schema.
      • Improved type inference for JSON schema types (e.g., type: null, type: ["string", "null"]) to map them correctly to google.genai.types.Type and the nullable attribute.
      • Ensured recursive conversion for nested schemas in properties (for objects) and items (for arrays).
      • Added logic to use a schema's title as a fallback for description if the latter is missing.
      • Refined handling of enum to ensure only string enums are passed as Google schema enums.
    • Refined _clean_schema_for_google: Removed some potentially overly aggressive cleaning of schema keywords to preserve necessary information like title.
    • Docstrings and Comments: Added comprehensive docstrings to all public and private methods, explaining their purpose, arguments, and return values. Removed redundant inline comments and clarified existing ones.
    • Corrected minor inconsistencies in how tool results and content parts were being constructed to ensure they align with Google API expectations, particularly around FunctionResponse and media parts.

Impact:

These changes ensure that Pydantic models used in tool calls and for structured output with Gemini models are correctly converted to the Google API's schema format. This resolves previous errors related to schema validation and should lead to more reliable behavior of the evaluator optimizer pattern and other features relying on structured data exchange with Gemini.

@monotykamary monotykamary changed the title Fix: Resolve schema conversion errors for Gemini in evaluator optimizer fix: resolve schema conversion errors for Gemini in evaluator optimizer Jun 5, 2025
@evalstate
Copy link
Owner

Hi -- is there a minimal MCP Server that I can add to the e2e test suite to demonstrate the failure/fix?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants