[BUG] Cannot run evals with Custom Provider only - requires API key in main LLM Providers

[ziondamore]

Description

I cannot run evals when using only a Custom Provider. The system shows the error:
"Please add your custom API key in Settings before running evals"

Steps to Reproduce

1. Go to Settings → Custom Providers
2. Add a custom provider with:
   • Provider Name: my-proxy
   • Protocol: OpenAI Compatible
   • Base URL: https://my-proxy-url.com
   • API Key: (valid key)
   • Model Names: gpt-5.2
3. Go to Test Cases
4. Select the custom model from "YOUR PROVIDERS → Custom"
5. Click "Run"

Expected Behavior

Evals should run using the configured Custom Provider since:

• Custom Provider is properly configured with API key
• Model is selected and appears in "Active Models"
• The model works in Playground/Inspector

Actual Behavior

Error message appears: "Please add your custom API key in Settings before running evals"

The system requires an API key in the main LLM Providers (OpenAI, Anthropic, etc.) even though I'm using a Custom Provider.

Environment

• MCPJam Inspector version: v2.0.3
• Browser: Chrome/Safari
• OS: macOS

Additional Context

My organization uses a custom AI proxy that is OpenAI-compatible. I cannot add this key to the main OpenAI provider because it
uses a different base URL. The Custom Provider feature exists exactly for this use case, but evals don't support it.

Suggested Solution

Allow evals to run when a Custom Provider is configured and selected, without requiring API keys in the main LLM Providers
section.

[dosubot]

Hi @ziondamore! I'm Dosu and I’m helping the inspector team.

This is a confirmed missing feature rather than a bug. The evals system doesn't support Custom Providers at all, even though they work in Chat/Playground.

Root cause: The validation logic in use-eval-handlers.ts assumes all non-MCPJam models are standard providers (OpenAI, Anthropic, etc.) and checks them against ProviderTokens. Custom providers with their own embedded API keys aren't accounted for in this flow.

Additionally:

• The evals API endpoint only accepts modelApiKeys: Record<string, string> — unlike the chat endpoint, there's no customProviders parameter to pass base URLs and full configurations
• The eval model selector UI only displays OpenRouter models, not custom providers

Unfortunately there's no workaround — this would require implementing custom provider support across the validation logic, API schema, and UI for evals. Your suggested solution is the right approach.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

  

[xXMrNidaXx]

Root Cause

This is likely a validation issue where the eval runner checks for API keys in the main provider config before checking custom providers.

Likely Fix Location

Looking at the described behavior, the issue is probably in the eval runner's pre-flight validation:

// Pseudocode of the likely bug
function validateBeforeEval(selectedModel: string) {
  // ❌ Bug: Only checks main providers, ignores custom
  const provider = getMainProvider(selectedModel);
  if (!provider?.apiKey) {
    throw new Error("Please add your custom API key in Settings...");
  }
}

Suggested Fix

function validateBeforeEval(selectedModel: string, config: Config) {
  // First check if it's a custom provider model
  const customProvider = config.customProviders.find(
    p => p.models.includes(selectedModel)
  );
  
  if (customProvider) {
    // Custom provider has its own API key configured
    if (!customProvider.apiKey) {
      throw new Error(`Custom provider "${customProvider.name}" missing API key`);
    }
    return; // ✅ Valid custom provider config
  }
  
  // Fall back to main provider validation
  const mainProvider = getMainProvider(selectedModel);
  if (!mainProvider?.apiKey) {
    throw new Error("Please add your API key in Settings...");
  }
}

Quick Workaround (Until Fixed)

If you need this working immediately, you can:

1. Add a dummy API key to the OpenAI provider in Settings (even "sk-dummy" may pass validation)
2. Select your custom provider model
3. Run evals

The actual API calls will use your custom provider's URL and key. This works because the validation is likely checking for any main provider key, not specifically the one being used.

Additional Context

This is a common pattern bug in multi-provider systems — the custom provider path often gets missed in validation logic that was written before custom providers existed.