Commit fb179e3
authored
ci(eval): gate the Gemma-4-E4B consolidation on agent evals (amd#2283)
Consolidating every agent onto `Gemma-4-E4B-it-GGUF` is an LLM-affecting
change, and CLAUDE.md requires `gaia eval agent` runs against the
committed baselines for exactly that class of change — but no CI job ran
them, because they need Lemonade on real AMD hardware. Today the
consolidation could merge on unit tests alone, which cover code paths
and say nothing about model behavior. This adds the missing job on the
self-hosted `lemonade-eval` pool: it runs `rag_quality`,
`context_retention` and `tool_selection` serially, compares each against
its committed Gemma baseline, and uploads the scorecards and traces.
It **reports** rather than blocks by default. Code-generation and
data-analysis scenarios are expected to regress on a 35B MoE → ~4B swap
and that tradeoff is accepted; wiring `--compare`'s exit 2 straight to
the step result would make the job permanently red, and a job that is
always red gates nothing. Set `enforce: true` (or put `[eval-enforce]`
in the commit message) to make regressions blocking.
One gate is **not** softened by report mode. `gaia eval agent --compare`
only scores scenarios present on both sides, so scenarios that vanish
from a run land in an `only_in_baseline` bucket outside its exit-2
verdict — I confirmed that dropping 6 of 7 `rag_quality` scenarios still
exits **0**. A broken harness would otherwise sail through as "no
regression". The integrity gate therefore fails the job unconditionally
when a baseline scenario is missing or came back unmeasured, whatever
`enforce` says.
### Verified against a live Lemonade 10.10.0
Behavior this workflow depends on, checked on real fixtures rather than
assumed:
- `--compare` exits **0** clean, **2** on a real regression (`PASS →
FAIL`), **1** on a missing file — the workflow treats only 2 as a
verdict and any other non-zero as a harness error.
- `gaia eval agent --category X` exits **0** regardless of scenario
scores, so the `&&` chain does not short-circuit on the expected
regressions.
- Scenarios present in CURRENT but absent from BASELINE are neutral —
relevant because `tool_selection` now has 5 scenarios on disk vs 4 in
the baseline (`data_vs_recall_disambiguation`, added by amd#1844, is
currently ungated until a baseline refresh).
- The integrity gate itself: passes a healthy run, fails the
6-of-7-vanished case, fails an all-`INFRA_ERROR` category.
- Backend `/api/health` returns 200; the `Output: <run-dir>` parsing
handles trailing slashes and multiple matches.
### Test plan
- [ ] First push to a branch touching a `paths:` entry fires the job —
this is the first real runner execution and the debugging pass.
- [ ] Confirm the `lemonade-eval` pool is bash and has a persistent
Lemonade; the preflight fails loudly with an actionable message if not.
- [ ] Confirm `pip install -e ".[dev,eval,ui,api]"` resolves on the
runner.
- [ ] Check the wall time against the 150-min budget and tighten if
there is slack.
- [ ] Re-dispatch with `enforce: true` once the accepted regressions are
folded into refreshed baselines.
### Notes for the reviewer
- Timeout is **150**, not the siblings' 90. The three baselines sum to
~55 min of pure scenario time before install, backend boot and judge
latency; a timeout cancels the job and can take the artifact with it.
Happy to drop it back once a few runs establish the real number.
- Time regressions are **not** comparable to this baseline — it was
captured against a remote Lemonade over ngrok on different hardware (see
its `meta.json`). `--compare` folds a >2x wall-clock delta into the same
exit 2 as a quality regression. The summary block says so; regenerating
the baselines on the runner would fix it properly.
- `gemma-4-e4b-d71cd914` is the only baseline covering all three
categories. `gemma-4-e4b-95e4b372` is newer but `rag_quality`-only.1 parent 6b2aa2d commit fb179e3
1 file changed
Lines changed: 455 additions & 0 deletions
0 commit comments