feat(evaluation) 5/15: generate the two promptfoo configs a run executes

[Ahmath-Gadji]

Part 5 of 15 of the split of #811. Targets eval/04-promptfoo-result-metrics (#815). Completes core/evaluation/ — everything after this is wiring.

What

build_retrieval_config and build_answer_config. Pure: they return plain dicts. Serialisation and execution live in the worker (part 9).

Notable

• Two configs, not one. The two questions need different endpoints: retrieval hits GET /search/partition/{p}, whose documents carry the chunk content that context-relevance grades and the metadata.file_id that feeds the ranking metrics; answers hit POST /v1/chat/completions, whose message content is what factuality and llm-rubric grade. Splitting them means every assertion in a config applies to that config's single provider — no assertion can run against an output shape it cannot read.
• transformResponse must be a single JavaScript expression. A statement or an IIFE makes promptfoo error every row before grading. This was found by running the feature end to end, not by reading the docs. The chat transform therefore extracts the message content and nothing more; retrieved sources come from the retrieval pass instead.
• The grader is OpenRAG's own configured LLM (llm.base_url / llm.model), so model-graded assertions need no third-party credentials. An apiKey is always set because the OpenAI client refuses to send without one, even though vLLM ignores it.
• Questions go through Nunjucks' urlencode, so a question containing & or ? cannot corrupt the search query string.
• Assertions are copied per test. A shared list serialises as a YAML anchor plus aliases, which is valid but unreadable in a config an operator may need to inspect.
• expected_file_ids is deliberately absent from vars — no assertion reads it. The ranking metrics are computed from the retrieved ids in summarize (part 4), not by promptfoo.

Testing

10 unit tests asserting the generated structure: one provider per config, the endpoints and methods, bearer headers, per-test assertion copies, urlencode templating, top_k propagation, and the grader provider. The configs are asserted structurally rather than executed — promptfoo is not available in CI.

[coderabbitai]

Warning

Review limit reached

@EnjoyBacon7, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 59 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 04666563-bdab-4b2c-ac2e-cf55184d9ec5

📥 Commits

Reviewing files that changed from the base of the PR and between 204206e and 35e2f62.

📒 Files selected for processing (3)

• openrag/core/evaluation/__init__.py
• openrag/core/evaluation/promptfoo_config.py
• tests/unit/core/evaluation/test_promptfoo_config.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch eval/05-promptfoo-config

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Ahmath-Gadji]

@coderabbitai full review

[coderabbitai]

✅ Action performed

Full review finished.

---

Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 57 minutes.

[hedhoud]

I found one blocking issue in the default Helm deployment path. The rest of the configuration and focused tests look good.

[hedhoud]

[P1] Normalize the grader base URL

Could you please remove a trailing slash from this value? The bundled Helm configuration provides the grader URL ending in /v1/, and promptfoo appends /chat/completions directly. That makes grading requests use /v1//chat/completions, so both retrieval and answer evaluations fail with the default deployment settings. A regression test using a grader URL ending in / would cover this case.