Feature: MCP Tools

[d42me]

Run via:

uv run vf-eval mcp-tool \
  -m gpt-4o-mini \
  -n 1 \
  --verbose \
  -a '{
    "mcp_launch_commands": [
      {
        "name": "test-calc",
        "script": "/workspace/test_server.py"
      }
    ],
    "num_examples": 1
  }

Test server example:

"""
Simple FastMCP test server for the mcp-tool environment.
Provides basic calculator functions for testing.
"""

from fastmcp import FastMCP

mcp = FastMCP("test-calc")

@mcp.tool
def add(a: int, b: int) -> int:
    """Add two numbers together."""
    return a + b

@mcp.tool  
def multiply(a: int, b: int) -> int:
    """Multiply two numbers together."""
    return a * b

@mcp.tool
def subtract(a: int, b: int) -> int:
    """Subtract b from a."""
    return a - b

@mcp.tool
def divide(a: float, b: float) -> float:
    """Divide a by b."""
    if b == 0:
        raise ValueError("Cannot divide by zero")
    return a / b

if __name__ == "__main__":
    mcp.run()

Description

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Test improvement

Testing

• All existing tests pass
• New tests have been added to cover the changes
• Tests have been run locally with python -m pytest tests/

Test Coverage

• Current coverage: ___%
• Coverage after changes: ___%

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• Any dependent changes have been merged and published

Additional Notes

[CLAassistant]

All committers have signed the CLA.

[rasdani]

looks mostly good. couldn't run yet.