Merge pull request #14 from aprilgittens/main

Support Foundry OpenAI v1 and Responses API endpoints in live evaluations

Author: leestott

.env.example

@@ -1,21 +1,21 @@
 # Copy this file to .env before running a live evaluation.
 # .env is ignored by Git and must never be committed.
 
-# Azure Model Router endpoint
-AZURE_MODEL_ROUTER_ENDPOINT=https://your-resource.services.ai.azure.com/models
+# Foundry OpenAI v1 base URL (do not include /chat/completions)
+AZURE_MODEL_ROUTER_ENDPOINT=https://your-resource.services.ai.azure.com/openai/v1
 AZURE_MODEL_ROUTER_KEY=your-model-router-api-key
 AZURE_MODEL_ROUTER_DEPLOYMENT=model-router
 
-# Azure OpenAI endpoint (for baseline model)
-AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
+# Foundry OpenAI v1 base URL for the baseline (do not include /responses)
+AZURE_OPENAI_ENDPOINT=https://your-resource.services.ai.azure.com/openai/v1
 AZURE_OPENAI_KEY=your-azure-openai-api-key
 # Must match a key in the pricing section of the config file you run (e.g. gpt-4o, gpt-5).
 # If you use a custom deployment name, add a matching entry under pricing: in that same config
 # using your exact deployment name. Otherwise baseline costs show as $0.00.
 AZURE_BASELINE_DEPLOYMENT=gpt-5
 
-# Judge model (defaults to same endpoint as baseline; override to use a different model)
-AZURE_JUDGE_ENDPOINT=https://your-resource.openai.azure.com
+# Judge model (defaults to the same Foundry OpenAI v1 base URL as the baseline)
+AZURE_JUDGE_ENDPOINT=https://your-resource.services.ai.azure.com/openai/v1
 AZURE_JUDGE_KEY=your-azure-openai-api-key
 AZURE_JUDGE_DEPLOYMENT=gpt-5
 

README.md

@@ -164,13 +164,13 @@ cp .env.example .env
 
 | Variable | Description |
 |----------|-------------|
-| `AZURE_MODEL_ROUTER_ENDPOINT` | Model Router endpoint URL |
+| `AZURE_MODEL_ROUTER_ENDPOINT` | Foundry OpenAI v1 base URL ending in `/openai/v1` |
 | `AZURE_MODEL_ROUTER_KEY` | Model Router API key |
 | `AZURE_MODEL_ROUTER_DEPLOYMENT` | Model Router deployment name (e.g. `model-router`) |
-| `AZURE_OPENAI_ENDPOINT` | Azure OpenAI endpoint URL (baseline) |
+| `AZURE_OPENAI_ENDPOINT` | Baseline OpenAI v1 base URL ending in `/openai/v1` |
 | `AZURE_OPENAI_KEY` | Azure OpenAI API key (baseline) |
 | `AZURE_BASELINE_DEPLOYMENT` | Baseline model deployment name (e.g. `gpt-5`) |
-| `AZURE_JUDGE_ENDPOINT` | Judge model endpoint URL (can be same as baseline) |
+| `AZURE_JUDGE_ENDPOINT` | Judge OpenAI v1 base URL (can be same as baseline) |
 | `AZURE_JUDGE_KEY` | Judge model API key |
 | `AZURE_JUDGE_DEPLOYMENT` | Judge model deployment name (e.g. `gpt-5`) |
 | `AZURE_AI_PROJECT_ENDPOINT` | Microsoft Foundry project endpoint (optional, for cloud eval) |

configs/README.md

@@ -5,13 +5,18 @@
 | [default.yaml](default.yaml) | All | Enabled | Standard evaluation |
 | [quick_test.yaml](quick_test.yaml) | 5 | Disabled | Fast smoke test |
 | [large_scale.yaml](large_scale.yaml) | All | Enabled | 1000+ prompts (higher concurrency, longer timeouts) |
+| [live_demo.yaml](live_demo.yaml) | All | Enabled | Foundry OpenAI v1 live demo |
 | [foundry.yaml](foundry.yaml) | — | — | Foundry cloud eval settings (graders, thresholds) |
 
 Edit `default.yaml` to set your endpoint URLs, baseline model, and pricing.
 Environment variables (`${VAR}`) are resolved from `.env`. Supported Foundry
 model prices are refreshed from the Azure Retail Prices API and cached under
 `.cache/`; YAML prices are retained as offline and ambiguity fallbacks.
 
+Foundry OpenAI v1 endpoints use `type: openai_compatible` and a base URL
+ending in `/openai/v1`. Set `api_mode` to `chat_completions` or `responses`;
+do not append the operation name to `endpoint_url`.
+
 ## Prompt Templates
 
 - [`judge_prompts/`](judge_prompts/) — Local LLM-as-judge prompt templates (absolute and pairwise)

configs/default.yaml

@@ -9,21 +9,21 @@ evaluation:
 
 endpoints:
   model_router:
-    type: "azure_openai"
+    type: "openai_compatible"
+    api_mode: "chat_completions"
     endpoint_url: "${AZURE_MODEL_ROUTER_ENDPOINT}"
     api_key: "${AZURE_MODEL_ROUTER_KEY}"
     deployment_name: "${AZURE_MODEL_ROUTER_DEPLOYMENT}"
     parameters:
-      temperature: 0.7
       max_tokens: 1024
 
   baseline:
-    type: "azure_openai"
+    type: "openai_compatible"
+    api_mode: "responses"
     endpoint_url: "${AZURE_OPENAI_ENDPOINT}"
     api_key: "${AZURE_OPENAI_KEY}"
     deployment_name: "${AZURE_BASELINE_DEPLOYMENT}"
     parameters:
-      temperature: 0.7
       max_tokens: 1024
 
 pricing:                     # USD per 1M tokens
@@ -172,7 +172,8 @@ output:
 judge:
   enabled: true
   endpoint:
-    type: "azure_openai"
+    type: "openai_compatible"
+    api_mode: "responses"
     endpoint_url: "${AZURE_JUDGE_ENDPOINT}"
     api_key: "${AZURE_JUDGE_KEY}"
     deployment_name: "${AZURE_JUDGE_DEPLOYMENT}"

configs/large_scale.yaml

@@ -10,15 +10,17 @@ evaluation:
 
 endpoints:
   model_router:
-    type: "azure_openai"
+    type: "openai_compatible"
+    api_mode: "chat_completions"
     endpoint_url: "${AZURE_MODEL_ROUTER_ENDPOINT}"
     api_key: "${AZURE_MODEL_ROUTER_KEY}"
     deployment_name: "${AZURE_MODEL_ROUTER_DEPLOYMENT}"
     parameters:
       max_tokens: 1024
 
   baseline:
-    type: "azure_openai"
+    type: "openai_compatible"
+    api_mode: "responses"
     endpoint_url: "${AZURE_OPENAI_ENDPOINT}"
     api_key: "${AZURE_OPENAI_KEY}"
     deployment_name: "${AZURE_BASELINE_DEPLOYMENT}"
@@ -106,7 +108,8 @@ output:
 judge:
   enabled: true
   endpoint:
-    type: "azure_openai"
+    type: "openai_compatible"
+    api_mode: "responses"
     endpoint_url: "${AZURE_JUDGE_ENDPOINT}"
     api_key: "${AZURE_JUDGE_KEY}"
     deployment_name: "${AZURE_JUDGE_DEPLOYMENT}"

configs/live_demo.yaml

@@ -8,15 +8,17 @@ evaluation:
 
 endpoints:
   model_router:
-    type: "azure_openai"
+    type: "openai_compatible"
+    api_mode: "chat_completions"
     endpoint_url: "${AZURE_MODEL_ROUTER_ENDPOINT}"
     api_key: "${AZURE_MODEL_ROUTER_KEY}"
     deployment_name: "${AZURE_MODEL_ROUTER_DEPLOYMENT}"
     parameters:
       max_tokens: 4096
 
   baseline:
-    type: "azure_openai"
+    type: "openai_compatible"
+    api_mode: "responses"
     endpoint_url: "${AZURE_OPENAI_ENDPOINT}"
     api_key: "${AZURE_OPENAI_KEY}"
     deployment_name: "${AZURE_BASELINE_DEPLOYMENT}"
@@ -80,7 +82,8 @@ output:
 judge:
   enabled: true
   endpoint:
-    type: "azure_openai"
+    type: "openai_compatible"
+    api_mode: "responses"
     endpoint_url: "${AZURE_JUDGE_ENDPOINT}"
     api_key: "${AZURE_JUDGE_KEY}"
     deployment_name: "${AZURE_JUDGE_DEPLOYMENT}"

docs/faq.md

@@ -37,8 +37,22 @@ cp .env.example .env
 
 | Endpoint | URL format |
 |----------|-----------|
-| Model Router | `https://<resource>.services.ai.azure.com/models` |
-| Azure OpenAI (baseline/judge) | `https://<resource>.openai.azure.com` |
+| Model Router | `https://<resource>.services.ai.azure.com/openai/v1` |
+| Baseline/judge | `https://<resource>.services.ai.azure.com/openai/v1` |
+
+Use `type: openai_compatible` for these Foundry v1 URLs. Set
+`api_mode: chat_completions` for Model Router and `api_mode: responses` for
+models whose portal target URI ends in `/responses`. Do not include
+`/chat/completions` or `/responses` in the environment variable itself.
+
+### `404 Resource not found`
+
+- Confirm the resource hostname and deployment name exactly match the portal.
+- Use the `/openai/v1` base URL, without an operation suffix.
+- Ensure `type` is `openai_compatible` and `api_mode` matches the portal target URI.
+
+An invalid hostname usually produces `APIConnectionError`, while a reachable
+host with the wrong route or deployment commonly produces `404`.
 
 ---
 

docs/how-to-run-live-eval.md

@@ -51,13 +51,13 @@ copy .env.example .env     # Windows
 Open `.env` and fill in your real values:
 
 ```
-AZURE_MODEL_ROUTER_ENDPOINT=https://your-resource.services.ai.azure.com/models
+AZURE_MODEL_ROUTER_ENDPOINT=https://your-resource.services.ai.azure.com/openai/v1
 AZURE_MODEL_ROUTER_KEY=your-model-router-key
 AZURE_MODEL_ROUTER_DEPLOYMENT=model-router
-AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
+AZURE_OPENAI_ENDPOINT=https://your-resource.services.ai.azure.com/openai/v1
 AZURE_OPENAI_KEY=your-azure-openai-key
 AZURE_BASELINE_DEPLOYMENT=your-baseline-deployment
-AZURE_JUDGE_ENDPOINT=https://your-resource.openai.azure.com
+AZURE_JUDGE_ENDPOINT=https://your-resource.services.ai.azure.com/openai/v1
 AZURE_JUDGE_KEY=your-azure-openai-key
 AZURE_JUDGE_DEPLOYMENT=your-judge-deployment
 AZURE_PRICING_REGION=eastus
@@ -71,6 +71,11 @@ AZURE_PRICING_REGION=eastus
 - *Azure Portal → your Foundry resource → Keys and Endpoint*
 - *Azure Portal → your Azure OpenAI resource → Keys and Endpoint*
 
+Use the resource name exactly as shown by the portal. If the portal gives a
+target URI ending in `/chat/completions` or `/responses`, remove that final
+operation segment and keep `/openai/v1`. The live presets select the operation
+with `api_mode`.
+
 ## Step 3: Configure the evaluation (optional)
 
 The default config (`configs/default.yaml`) works out of the box. The settings most people change first:
@@ -80,6 +85,7 @@ The default config (`configs/default.yaml`) works out of the box. The settings m
 | Baseline model | `endpoints.baseline.deployment_name` | `gpt-5` | Which model the router is compared against |
 | Number of prompts | `evaluation.sample_size` | `null` (all) | How many dataset prompts to use |
 | Judge enabled | `judge.enabled` | `true` | Whether to run quality scoring (costs extra API calls) |
+| Endpoint API | `api_mode` | Per endpoint | `chat_completions` or `responses` |
 | Concurrency | `concurrency.max_parallel_requests` | `5` | How many prompts run in parallel |
 
 To change the baseline model, edit `configs/default.yaml`:

pyproject.toml

@@ -10,7 +10,7 @@ readme = "README.md"
 license = {text = "MIT"}
 requires-python = ">=3.9"
 dependencies = [
-    "openai>=1.0",
+    "openai>=2.0",
     "pyyaml>=6.0",
     "matplotlib>=3.7",
     "numpy>=1.24",

requirements.txt

@@ -6,7 +6,7 @@
 # in pyproject.toml: `pip install -e .` (core), `.[foundry]`, `.[dev]`, `.[db]`.
 
 # --- Core (required by scripts/run_eval.py and src/) ---
-openai>=1.0
+openai>=2.0
 pyyaml>=6.0
 matplotlib>=3.7
 numpy>=1.24