Skip to content

Commit fb179e3

Browse files
authored
ci(eval): gate the Gemma-4-E4B consolidation on agent evals (amd#2283)
Consolidating every agent onto `Gemma-4-E4B-it-GGUF` is an LLM-affecting change, and CLAUDE.md requires `gaia eval agent` runs against the committed baselines for exactly that class of change — but no CI job ran them, because they need Lemonade on real AMD hardware. Today the consolidation could merge on unit tests alone, which cover code paths and say nothing about model behavior. This adds the missing job on the self-hosted `lemonade-eval` pool: it runs `rag_quality`, `context_retention` and `tool_selection` serially, compares each against its committed Gemma baseline, and uploads the scorecards and traces. It **reports** rather than blocks by default. Code-generation and data-analysis scenarios are expected to regress on a 35B MoE → ~4B swap and that tradeoff is accepted; wiring `--compare`'s exit 2 straight to the step result would make the job permanently red, and a job that is always red gates nothing. Set `enforce: true` (or put `[eval-enforce]` in the commit message) to make regressions blocking. One gate is **not** softened by report mode. `gaia eval agent --compare` only scores scenarios present on both sides, so scenarios that vanish from a run land in an `only_in_baseline` bucket outside its exit-2 verdict — I confirmed that dropping 6 of 7 `rag_quality` scenarios still exits **0**. A broken harness would otherwise sail through as "no regression". The integrity gate therefore fails the job unconditionally when a baseline scenario is missing or came back unmeasured, whatever `enforce` says. ### Verified against a live Lemonade 10.10.0 Behavior this workflow depends on, checked on real fixtures rather than assumed: - `--compare` exits **0** clean, **2** on a real regression (`PASS → FAIL`), **1** on a missing file — the workflow treats only 2 as a verdict and any other non-zero as a harness error. - `gaia eval agent --category X` exits **0** regardless of scenario scores, so the `&&` chain does not short-circuit on the expected regressions. - Scenarios present in CURRENT but absent from BASELINE are neutral — relevant because `tool_selection` now has 5 scenarios on disk vs 4 in the baseline (`data_vs_recall_disambiguation`, added by amd#1844, is currently ungated until a baseline refresh). - The integrity gate itself: passes a healthy run, fails the 6-of-7-vanished case, fails an all-`INFRA_ERROR` category. - Backend `/api/health` returns 200; the `Output: <run-dir>` parsing handles trailing slashes and multiple matches. ### Test plan - [ ] First push to a branch touching a `paths:` entry fires the job — this is the first real runner execution and the debugging pass. - [ ] Confirm the `lemonade-eval` pool is bash and has a persistent Lemonade; the preflight fails loudly with an actionable message if not. - [ ] Confirm `pip install -e ".[dev,eval,ui,api]"` resolves on the runner. - [ ] Check the wall time against the 150-min budget and tighten if there is slack. - [ ] Re-dispatch with `enforce: true` once the accepted regressions are folded into refreshed baselines. ### Notes for the reviewer - Timeout is **150**, not the siblings' 90. The three baselines sum to ~55 min of pure scenario time before install, backend boot and judge latency; a timeout cancels the job and can take the artifact with it. Happy to drop it back once a few runs establish the real number. - Time regressions are **not** comparable to this baseline — it was captured against a remote Lemonade over ngrok on different hardware (see its `meta.json`). `--compare` folds a >2x wall-clock delta into the same exit 2 as a quality regression. The summary block says so; regenerating the baselines on the runner would fix it properly. - `gemma-4-e4b-d71cd914` is the only baseline covering all three categories. `gemma-4-e4b-95e4b372` is newer but `rag_quality`-only.
1 parent 6b2aa2d commit fb179e3

1 file changed

Lines changed: 455 additions & 0 deletions

File tree

0 commit comments

Comments
 (0)