docs(evaluation) 15/15: feature guide and a sample dataset

.gitignore

@@ -65,6 +65,8 @@ volumes/*
 !/services/.gitkeep # Keep the placeholder
 
 *.csv
+# The committed evaluation test set is source, not scratch data.
+!tests/evaluation/*.csv
 *.pkl
 
 #helm

CLAUDE.md

@@ -242,6 +242,59 @@ Optional web search augmentation via the Staan API, allowing the LLM to combine
 - `openrag/services/orchestrators/query_service.py` — `_prepare_for_web_only()`, web search logic in `_prepare_for_chat_completion()`
 - `openrag/api/routers/user/chat.py` — `__prepare_sources()` merges document and web sources
 
+### Evaluation (admin System page → Evaluation tab)
+
+On-demand benchmarking of indexing speed, retrieval quality and answer quality.
+
+**Flow** (`EvalRunner` Ray actor, `openrag/services/workers/eval_runner.py`): create the
+throwaway partition `__eval_<run_id>` → upload and time each corpus file over the real HTTP
+API → shell out to `promptfoo eval` twice → fold the outputs into metrics → drop the partition.
+
+- **Layering**: `EvaluationService` never touches Ray. It dispatches through the
+  `EvaluationRunner` port (`openrag/core/evaluation/runner.py`), implemented by
+  `RayEvaluationRunner` (`openrag/services/workers/eval_dispatcher.py`) and injected by the
+  container — the same port/adapter shape as `IndexingDispatcher`. The adapter resolves its
+  detached actor on first use, so building the service does not spawn a worker.
+- **Datasets** are admin-uploaded: a corpus plus a CSV test set
+  (`question,expected_answer,expected_file_ids`; the last column is optional and
+  `;`-separated). Files live under `<data_dir>/eval/<dataset_id>/`. The corpus is
+  streamed to disk rather than buffered, so its cap is a disk cost, not a RAM one.
+- **`file_id` sanitisation** (`sanitize_file_id`, `openrag/core/evaluation/identity.py`):
+  the indexing API accepts only `[A-Za-z0-9._:-]` in a `file_id`, so a corpus file cannot be
+  uploaded under a raw human filename. The runner uploads a sanitised id, and
+  `metrics.summarize` sanitises **both** sides of the ground-truth comparison, so a test set
+  naming `A B.pdf` still matches the stored `A_B.pdf`. Note `metadata.source` is the
+  server's storage path, not the original name — the `file_id` is what a human sees.
+- **Two promptfoo configs, not one** (`openrag/core/evaluation/promptfoo_config.py`):
+  retrieval hits `GET /search/partition/{partition}` (documents carry the chunk under
+  `content`, plus `metadata.file_id`), answers hit `POST /v1/chat/completions`. Each config
+  has one provider, so no assertion runs against an output shape it cannot read.
+  `transformResponse` must be a single JavaScript **expression** — an IIFE or any statement
+  makes promptfoo error every row before grading.
+- **Metrics** (`openrag/core/evaluation/metrics.py`): throughput from wall-clock, plus
+  hit rate / MRR / recall using the definitions in
+  `tests/load/automatic-evaluation-pipeline/README.md`. Rows without `expected_file_ids`
+  are reported as `skipped_cases`, never as misses. Percentiles are nearest-rank
+  (`ceil`) — `round` would break ties to even and report the wrong observation.
+- **Auth**: runs authenticate as the non-admin service user `__openrag_eval__`, whose token
+  is regenerated at the start of every run, so no usable plaintext token is stored at rest.
+- **Concurrency**: one run at a time. Enforced by the partial unique index
+  `ux_eval_runs_single_active`, not by a read-then-insert — the run row is created *before*
+  the token is regenerated, so two racing starts cannot revoke each other's credentials.
+  `POST /evaluation/runs` returns 409 on the loser, and 503 if the runner cannot be pinged.
+  A run orphaned by an actor restart is reaped by cancelling it, which writes the terminal
+  status directly; a failed provision releases the row the same way.
+- **Config** (`openrag/core/config/evaluation.py`, `evaluation:` in `conf/config.yaml`):
+  limits and timeouts are env-overridable (`EVAL_*`, `PROMPTFOO_BIN`). The API base URL the
+  runner calls back on is `server.internal_url` (env: `OPENRAG_INTERNAL_URL`) — it is a
+  server property, not an eval one. The reserved partition prefix, CSV column names and the
+  `file_id` alphabet are deliberately *not* config — they are contracts with stored datasets.
+- Requires **Node 22** (from NodeSource; distro packages predate promptfoo's floor) + a
+  pinned promptfoo in **both** `infra/docker/api.Dockerfile` (compose runs Ray inside the
+  API container) and `infra/docker/ray.Dockerfile` (separate Ray cluster). Dataset files are
+  read from disk by the runner, so a **separate** Ray cluster needs `<data_dir>` on shared
+  storage.
+
 ### File Quota System
 
 Per-user file quota enforcement tracked via the `file_count` and `file_quota` columns on `users`, and `created_by` on `files`.

tests/evaluation/README.md

@@ -0,0 +1,49 @@
+# Sample evaluation dataset
+
+`rag_dataset_sample.csv` is a ready-made test set for the admin **System →
+Evaluation** tab: 11 questions over 6 documents, every answer checked against
+the source PDF rather than generated.
+
+The corpus itself is not committed — the documents are third-party PDFs
+totalling ~36 MB. `corpus.txt` lists the 24 filenames, drawn from the internal
+`rag_dataset` collection (French public-sector, agricultural, medical and AI
+documents).
+
+## Why 24 documents for 11 questions
+
+Only 6 documents are the subject of a question. The other 18 are deliberate
+distractors, each topically adjacent to a question's source — other AI-policy
+papers, other gut/neuro medical papers, other agricultural press releases. A
+corpus where every document is on a different subject makes retrieval look
+better than it is: any half-working retriever scores a perfect hit rate when
+there is only one candidate per topic.
+
+## Assembling it
+
+```bash
+mkdir -p /tmp/eval-corpus
+while IFS= read -r f; do cp "<path-to>/rag_dataset/$f" /tmp/eval-corpus/; done \
+  < tests/evaluation/corpus.txt
+```
+
+Then upload it, either through the Evaluation tab or the API:
+
+```bash
+args=(); for f in /tmp/eval-corpus/*; do args+=(-F "corpus=@${f}"); done
+curl -X POST "$OPENRAG_URL/evaluation/datasets" \
+  -H "Authorization: Bearer $AUTH_TOKEN" \
+  -F "name=rag_dataset sample" \
+  -F "testset=@tests/evaluation/rag_dataset_sample.csv" \
+  "${args[@]}"
+```
+
+## Test set format
+
+`question,expected_answer,expected_file_ids` — `expected_file_ids` is optional
+and semicolon-separated. Name the files as they appear on disk; the indexer
+sanitises the id it stores (spaces are not valid in a `file_id`), and the
+ranking metrics match against the original filename in the chunk metadata.
+
+Rows without `expected_file_ids` still count toward answer quality, but are
+reported as `skipped_cases` in hit rate / MRR / recall rather than scored as
+misses.

tests/evaluation/corpus.txt

@@ -0,0 +1,24 @@
+2017-10-30-DP-sauvons-le-colza-francais.pdf
+2018-01-18-CP-red-2-energie-renouvelable-la-contribution-agricole-largement-reconnue.pdf
+2019-12-03-DP-la-semence-certifiee-de-soja.pdf
+2024-04-17-upcycling.pdf
+202407_charte-de-deontologie-AFCL.pdf
+20240930_Note_LINAGORA_IA_OpenSource_Universelle.pdf
+20241121_CP_ OSPX24_LUCIE-V2.pdf
+579_Urban-immunization-toolkit_final-1563547313.pdf
+9789240049130-eng.pdf
+Antiinflammatoire SCFA acetate propionate.pdf
+Competition_in_cloud_sector.pdf
+GAVI_use_case_Sample_draft.pdf
+IA Ethique.pdf
+Intelligence-artificielle-01-2024.pdf
+Kynurenine pathway ALS.pdf
+Leaky gut in systemic infammation.pdf
+Lignes-directrices-nouvelle-version-2024-10.pdf
+Make_France_AI_Powerhouse.pdf
+Manifeste-CannabiSante-PrincipesActifs.pdf
+Note Syndrome de Prader-Willi V2 Jan 22.pdf
+Note de positionnement Pensons Patients.pdf
+Note_aux_Operateurs_432_08_12_23.pdf
+Nouveau_Bigster_Dacia_en_grand.pdf
+ReST meets ReAct.pdf

tests/evaluation/rag_dataset_sample.csv

@@ -0,0 +1,12 @@
+question,expected_answer,expected_file_ids
+Quand la communauté OpenLLM France a-t-elle été créée et à l'initiative de quelle entreprise ?,En juin 2023 à l'impulsion de LINAGORA.,20241121_CP_ OSPX24_LUCIE-V2.pdf
+Quand le pré-entraînement du modèle LUCIE a-t-il démarré ?,Dès décembre 2023.,20241121_CP_ OSPX24_LUCIE-V2.pdf
+À quelle date se tient le Paris Open Source AI Summit ?,Le 22 janvier 2025.,20241121_CP_ OSPX24_LUCIE-V2.pdf
+Quelle est la fréquence à la naissance du syndrome de Prader-Willi ?,Environ une naissance sur 21 000.,Note Syndrome de Prader-Willi V2 Jan 22.pdf
+Sur quel chromosome se situe l'anomalie génétique à l'origine du syndrome de Prader-Willi ?,Le chromosome 15.,Note Syndrome de Prader-Willi V2 Jan 22.pdf
+Qui a décrit le syndrome de Prader-Willi et en quelle année ?,"En 1956, par trois médecins suisses : Andrea Prader, Alexis Labhart et Heinrich Willi.",Note Syndrome de Prader-Willi V2 Jan 22.pdf
+Quel pourcentage de biocarburants dans les transports le secteur agricole européen a-t-il permis d'atteindre selon le communiqué sur la directive RED II ?,"7,7 % de biocarburants dans les transports.",2018-01-18-CP-red-2-energie-renouvelable-la-contribution-agricole-largement-reconnue.pdf
+Quelle obligation d'énergie renouvelable dans les transports le Parlement européen a-t-il retenue dans la directive RED II ?,Une obligation de 12 % d'énergie renouvelable dans les transports.,2018-01-18-CP-red-2-energie-renouvelable-la-contribution-agricole-largement-reconnue.pdf
+Combien de morts par an les maladies cardioneurovasculaires causent-elles en France ?,Environ 140 000 morts par an.,Note de positionnement Pensons Patients.pdf
+À partir de quelle taille de population le président d'un établissement public de coopération intercommunale est-il visé par les incompatibilités de la charte de déontologie du conseil en lobbying ?,Plus de 100 000 habitants.,202407_charte-de-deontologie-AFCL.pdf
+Who is Tony in the GAVI use case scenario and what is his role?,"Tony is a 35-year-old community health worker in a remote rural village in Africa, managing a local health facility that is the first point of contact for village residents.",GAVI_use_case_Sample_draft.pdf