Skip to content
Agent Eval — Gemma-4-E4B consolidation

fix(email): scoped 'anything suspicious?' query no longer dumps the full triage report #159

Sign in to view logs

fix(email): scoped 'anything suspicious?' query no longer dumps the full triage report

fix(email): scoped 'anything suspicious?' query no longer dumps the full triage report #159

Workflow file for this run

.github/workflows/test_eval_agent_gemma_consolidation.yml at 6638d3b
# Copyright(C) 2025-2026 Advanced Micro Devices, Inc. All rights reserved.
# SPDX-License-Identifier: MIT
# Agent eval gate for the Gemma-4-E4B consolidation.
#
# Every agent is moving onto `Gemma-4-E4B-it-GGUF`; agents that previously left
# `model_id` unset fell back to `Qwen3.5-35B-A3B-GGUF`. That is an LLM-affecting
# change, and CLAUDE.md requires `gaia eval agent` runs against the committed
# baselines before such a change lands. Unit tests cover code paths, not model
# behavior — this job is the only thing that catches a behavioral regression.
#
# Runner-only by construction. `gaia eval agent` drives the real GAIA UI backend
# against a real Lemonade Server on AMD hardware; a GitHub-hosted runner has
# neither. Hence the self-hosted pool, and the same shared `lemonade-eval`
# concurrency group as email_scorecard_refresh.yml / test_email_agent_eval.yml so
# this queues BEHIND them instead of race-evicting their loaded model (CLAUDE.md:
# at most one `gaia eval agent` process at a time, period).
#
# THIS POOL IS WINDOWS. Every inline `run:` step below is PowerShell, and Python
# setup goes through ./.github/actions/setup-venv (uv, Windows-aware) — the
# lab-runner convention shared by test_email_agent_eval.yml,
# email_scorecard_refresh.yml and test_agent_behavior_e2e.yml. Do NOT reintroduce
# bash here: this workflow's first-ever execution died instantly on
# `source .venv/bin/activate` because GitHub ran it through powershell.EXE, and
# every other bash step in the job hit the same parser. Windows also breaks the
# text-processing assumptions bash invites — the eval prints an `Output:` line
# containing a backslash path, which bash test/`[ -f ]` cannot consume.
#
# WHY THIS REPORTS BY DEFAULT INSTEAD OF HARD-FAILING (`enforce`, default false)
# ---------------------------------------------------------------------------
# `gaia eval agent --compare BASELINE CURRENT` already exits 2 on any status /
# score / time regression and 0 otherwise (src/gaia/cli.py). Wiring that exit
# code straight to the step result would make this job PERMANENTLY RED, because
# some regressions here are EXPECTED and accepted: code-generation and
# data-analysis scenarios are dropping from a 35B MoE to a ~4B dense model. A
# gate that is always red gates nothing — people stop reading it.
#
# So the default posture is REPORT: every category is compared, the full diff is
# printed, the scorecards are uploaded as an artifact, and a regression surfaces
# as a `::warning::` annotation. Make it blocking either by dispatching with
# `enforce: true`, or — since workflow_dispatch is only offered for workflows
# already on the default branch — by putting `[eval-enforce]` in the PR TITLE,
# which works on a branch before this file has merged.
#
# This mirrors how intentional regressions are handled elsewhere: the
# `--allow-regression` flag in src/gaia/eval/scorecard_gate.py prints a
# `::warning::` and returns 0 rather than pretending the regression did not
# happen, and test_email_agent_eval.yml ships its gates in report mode until the
# bars are trustworthy. Same idea, inverted default: report unless asked to
# enforce.
#
# REPORT MODE SOFTENS "WORSE", NEVER "DIDN'T MEASURE". `--compare` scores only
# scenarios present on BOTH sides: scenarios that vanish from the current run land
# in an `only_in_baseline` bucket that is NOT part of its exit-2 verdict
# (runner.py::compare_scorecards). Verified: dropping 6 of 7 rag_quality scenarios
# still exits 0. So a broken harness would otherwise sail through as "no
# regression". The integrity gate below therefore runs UNCONDITIONALLY — ignoring
# `enforce` entirely — and fails the job if a baseline scenario is missing or any
# scenario failed to produce a measurement (infra_error / errored / timeout /
# blocked / budget_exceeded). Report mode is for "we measured and it got worse",
# never for "we did not measure".
#
# TIME REGRESSIONS ARE NOT COMPARABLE TO THIS BASELINE. `--compare` folds a >2x
# per-scenario wall-clock delta into the same exit 2 as a quality regression, but
# the committed baseline was captured against a REMOTE Lemonade over ngrok
# (see meta.json) on different hardware and an older Lemonade. Treat a time-only
# regression here as a hardware artifact, not a defect, until the baselines are
# regenerated on the runner itself.
#
# SEQUENCING. `--compare` only DIFFS two scorecards — it does NOT run an eval.
# So each category is evaluated first, the ABSOLUTE run directory is parsed from
# the eval's own `Output: <run-dir>` line, and `<run-dir>/scorecard.json` is then
# compared against the committed Gemma baseline. Never glob by mtime — a fresh
# clone stamps every file with the checkout time.
#
# PREREQUISITES ON THE RUNNER. Lemonade is NOT assumed to be already running:
# install-lemonade pins the runner to the expected LEMONADE_VERSION and
# ensure-lemonade-running.ps1 starts it as a persistent Scheduled Task and warms
# the model — exactly what test_email_agent_eval.yml and
# email_scorecard_refresh.yml do. (An earlier revision of this file claimed those
# two workflows assume a persistent server and start nothing; they do not, and
# that claim is what left this job with no way to bring Lemonade up.) The GAIA UI
# backend is separate and NOT persistent, so this workflow starts it itself — in
# the SAME shell session as the evals, because a backgrounded child does not
# survive a GitHub Actions step boundary (see the single-session note in
# test_agent_behavior_e2e.yml).
#
# Lemonade must serve TWO models, not one. ensure-lemonade-running.ps1 warms
# only the chat model, but rag_quality and context_retention answer out of RAG
# chunks, so the embedder is just as load-bearing — and when it cannot start,
# those categories collapse to ~2/10 non-answers that read as a model
# regression. The preflight therefore re-registers, re-pulls and then makes a
# real /embeddings call through tests/ci_lemonade_check.py — repairing the one
# cause CI can repair (a stale registration missing the embeddings label) and
# failing in seconds on the ones it cannot, instead of spending three and a half
# hours measuring nothing.
#
# The third prerequisite is the Claude Code CLI, which is easy to miss because
# ANTHROPIC_API_KEY looks like the whole story: `gaia eval agent` DRIVES each
# scenario by shelling out to `claude -p` with an MCP config, and its preflight
# refuses to start without it on PATH. Installed below via npm, as
# docs/guides/eval-ci.mdx prescribes.
#
# WHAT TRIGGERS IT. A `pull_request` path filter, so the gate fires on the PR's
# own base..head diff and only when that diff touches an LLM-affecting path. It
# used to be a `push` filter, which matches the files in the PUSH — so merging
# main into a branch replayed all of main's churn through the filter and fired
# this multi-hour gate on PRs containing no Python at all (#2807 and #2809 are
# C++-only and both wore its red X). Scoping is the only thing that changed: the
# path list is untouched, and a PR that does touch one still runs and still
# fails loudly.
#
# Two consequences of leaving `push` behind, both intended. A branch with no PR
# no longer triggers the gate — use `workflow_dispatch --ref <branch>`. And the
# checkout is now `refs/pull/N/merge`, so the eval measures main+PR rather than
# the branch head, which is the thing about to land.
#
# FORKS AND DRAFTS. Both are skipped by the job-level `if` below, not silently.
# A fork PR could not pass anyway — ANTHROPIC_API_KEY, the eval judge, is not
# exposed to fork PRs — and must not put unreviewed code on the lab pool. For
# the PRs that do run the key is present; if it ever is not, the preflight fails
# with instructions instead of running a judge-less eval that would produce a
# meaningless scorecard.
name: Agent Eval — Gemma-4-E4B consolidation
on:
workflow_dispatch:
inputs:
enforce:
description: 'Fail the build on a regression (false = report as ::warning:: only)'
required: false
type: boolean
default: false
eval_model:
description: 'Driver + judge model for the eval harness (must match the baseline)'
required: false
default: 'claude-sonnet-4-6'
# `pull_request`, NOT `push`. A `push` path filter matches the files in the
# PUSH, so merging main into a branch replays all of main's churn through the
# filter — which is how #2807 and #2809, both pure C++ PRs with no Python in
# their diff at all, ended up wearing a red X from an LLM-quality gate and
# burning hours of the scarcest runner pool. `pull_request` filters on the
# PR's own base..head diff, so the gate fires only when the PR itself edits a
# path below. `branches: [main]` and the `types:` list follow
# test_email_agent_eval.yml; `ready_for_review` is what runs the gate on a PR
# that was opened as a draft (see the draft clause in the job `if:`).
#
# THE PATH LIST BELOW IS UNCHANGED ON PURPOSE. This change is mechanism only:
# it makes the filter mean what it already says. Note the list now includes
# `hub/agents/*/python/**`, so 18 hub packages newly qualify for a
# single-slot pool — scoping to the PR diff is what keeps that affordable.
pull_request:
branches: [ main ]
types: [opened, synchronize, reopened, ready_for_review]
paths:
# The consolidation itself.
- 'src/gaia/agents/base/agent.py'
- 'src/gaia/agents/registry.py'
- 'src/gaia/llm/lemonade_client.py'
- 'src/gaia/llm/factory.py'
- 'hub/agents/*/python/**'
- 'tests/fixtures/eval_baselines/**'
- '.github/workflows/test_eval_agent_gemma_consolidation.yml'
# In-core agents that carry their own model_id, plus the prompt/tool
# surfaces CLAUDE.md lists as eval-requiring — a tool docstring edit
# changes the JSON tool schema and is exactly what tool_selection scores.
# (ChatAgent is not listed: it now lives under hub/agents/chat/python/,
# already covered above.)
- 'src/gaia/agents/builder/**'
- 'src/gaia/agents/tools/**'
- 'src/gaia/chat/**'
concurrency:
# Share the single Lemonade backend slot with the other self-hosted evals so two
# runs never race-evict each other's model (CLAUDE.md: evals run serially).
group: lemonade-eval
cancel-in-progress: false
# `queue: single` (the default) keeps only ONE pending run and cancels the
# previously-queued one — a phantom red X on the PR. `max` queues them FIFO.
queue: max
permissions:
contents: read
env:
BASELINE_DIR: tests/fixtures/eval_baselines/gemma-4-e4b-d71cd914
BACKEND_URL: http://127.0.0.1:4200
# A dispatch input always wins. Otherwise `[eval-enforce]` in the PR TITLE opts
# in — workflow_dispatch is not offered until this file is on the default
# branch, so without this there is no way to enforce pre-merge. Reads the PR
# title, not `head_commit.message`: that context exists only on `push`, so it
# is permanently null now that this triggers on `pull_request`.
#
# `edited` is deliberately NOT in the `types:` above — a title tweak must not
# be able to launch a multi-hour run on the shared slot. So retitling alone
# does not re-run the gate: add the tag, then push (or close/reopen).
ENFORCE: ${{ github.event.inputs.enforce || (contains(github.event.pull_request.title, '[eval-enforce]') && 'true') || 'false' }}
# PINNED TO THE BASELINE'S JUDGE, and that is not a stale value to bump.
# `--model` is not a judge-only knob: runner.py passes it to `claude -p`, which
# DRIVES every scenario as well as scoring it. The committed baselines record
# `eval_model: claude-sonnet-4-6` (meta.json), so driving with anything else
# compares two different agents-under-test and folds the new driver's wall
# clock into the same >2x time verdict. #2859 moved every Claude workflow to
# Opus 5; this one value is exempt until the baselines are regenerated with
# that driver. Override per-run via the dispatch input above.
EVAL_MODEL: ${{ github.event.inputs.eval_model || 'claude-sonnet-4-6' }}
# Loopback services — bypass any runner proxy. This covers the PYTHON side
# (httpx/requests in the eval, the agents and the backend); PowerShell's
# Invoke-RestMethod ignores NO_PROXY and uses the system proxy instead.
NO_PROXY: 'localhost,127.0.0.1'
# Bare host (no /api/v1): LemonadeClient normalizes the suffix itself. Mirrors
# test_email_agent_eval.yml so the preflight probe and the agents agree.
LEMONADE_BASE_URL: 'http://127.0.0.1:13305'
# The eval prints box-drawing characters and em dashes; the runner's default
# console codepage is cp1252 and Python dies on the first one without this.
PYTHONIOENCODING: 'utf-8'
# Connector resolution blocks on the OS keyring in a headless runner, which
# hangs agent construction. Same guard as test_email_agent_eval.yml.
PYTHON_KEYRING_BACKEND: keyring.backends.null.Keyring
# Persistent runner — keep long-term memory out of the loop so state cannot
# bleed between the three chained categories or across runs. Established
# convention (test_chat_agent.yml, test_email_agent_eval.yml).
GAIA_MEMORY_DISABLED: '1'
jobs:
agent-eval:
name: rag_quality + context_retention + tool_selection vs Gemma baselines
# Fork PRs and drafts never reach the lab pool. A fork PR would schedule
# untrusted code onto a self-hosted runner and could not pass anyway
# (ANTHROPIC_API_KEY is not exposed to fork PRs, so the judge preflight
# fails); a draft would spend a multi-hour single-slot run on work still in
# progress. A job-level `if` resolves before a runner is assigned, so a
# skipped job never reaches the pool at all. `ready_for_ci` is the repo-wide
# escape hatch for a draft that does want CI. Mirrors the guard in
# test_email_agent_eval.yml.
#
# Kept flat-indented on purpose: a YAML folded scalar preserves newlines on
# more-indented lines, and an `if:` expression must fold to a single line.
if: >-
github.event_name != 'pull_request' ||
(github.event.pull_request.head.repo.full_name == github.repository &&
(github.event.pull_request.draft == false ||
contains(github.event.pull_request.labels.*.name, 'ready_for_ci')))
# `Windows` is load-bearing, not decoration: self-hosted pool labels here also
# match a Linux box (xsj-aimlab-stxp-04), and every step below is PowerShell +
# LemonadeServer.exe. Landing on Linux dies with 'powershell: command not
# found'. Same guard as test_email_agent_eval.yml.
#
# `lemonade-eval` — as opposed to the `stx` label the sibling evals use — is
# currently carried by exactly one runner (sjlab-stx-halo-18, itself an stx
# box). Serialisation comes from the concurrency group above, NOT from this
# label; the label only narrows WHICH box. Widen to `stx` if this job should
# be schedulable across the whole pool.
runs-on: [self-hosted, Windows, lemonade-eval]
defaults:
run:
# Pinned to Windows PowerShell 5.1, not left to the runner default (pwsh
# when installed, powershell otherwise). Everything below is written to
# 5.1's rules — the Tee-Object/`2>&1` and $LASTEXITCODE notes are
# 5.1-specific — and this file's first run died because nothing declared
# a shell at all and the steps were bash.
shell: powershell
# BUDGET DERIVATION — measured on this runner, not guessed.
#
# The first clean run (30869194618) was CANCELLED by a hand-picked
# `timeout-minutes: 150` after 148 min, having finished exactly ONE of three
# categories. It produced the only real numbers this pool has:
#
# setup (checkout -> preflight, inclusive) 0:01:18
# rag_quality 8 scenarios 2:04:07 measured
# context_retention 1 of 4 scenarios before the cancel 0:23:06 partial
# tool_selection never reached —
#
# rag_quality alone ran 124 min against a 22.4-min committed baseline — 5.5x,
# because that baseline was captured on different hardware over a remote
# Lemonade (see meta.json). The old 150 was sized off the ~55-min baseline sum
# and never had a chance. Per-category CEILINGS from the measured rates:
#
# rag_quality measured 124 -> 150
# context_retention 4 x the 23.1-min scenario measured above = 92 -> 110
# tool_selection 4 scenarios, same shape as context_retention -> 110
# eval step ceiling 150+110+110 = 370 -> 400
# setup (the Lemonade start step already carries its own 15-min cap) -> 20
# collect + integrity + compare + upload -> 15
# job: 400+20+15 = 435 -> 450
#
# These are CEILINGS, not expectations — expected wall clock is ~5h. Sizing
# generously is close to free here: the pool is self-hosted (no per-minute
# billing, and the 6h job cap is a GitHub-HOSTED limit; self-hosted is 5 days),
# and the shared `lemonade-eval` concurrency group already serialises the pool,
# so a high ceiling cannot let this job trample another eval. The asymmetry is
# what matters: too low costs hours of runner time AND the evidence, too high
# costs nothing. email_scorecard_refresh.yml sits at 840 on this same pool for
# exactly this reason (#2094: a hand-typed 90 against a measured 2h workload
# burned 100 consecutive dispatches without one completing).
#
# The eval step carries its OWN timeout-minutes below so an overrun fails at a
# NAMED step instead of a silent job-level `cancelled` — a job-level timeout
# skips the `!cancelled()` collect/integrity/compare steps, which is precisely
# how run 30869194618 threw away the 124 min of rag_quality data it had already
# produced. Re-tighten both once a few complete runs establish the true wall
# time; the per-category [TIMING] lines in the eval step exist to supply it.
timeout-minutes: 450
steps:
- name: Checkout
uses: actions/checkout@v7
- name: Setup Python Environment
uses: ./.github/actions/setup-venv
with:
python-version: '3.12'
# [dev] + [eval] for the eval harness and the Anthropic judge deps;
# [ui] for fastapi/uvicorn + the RAG deps gaia.ui.server imports at boot
# ([dev] alone fails at import), [api] for the connectors layer.
# ChatAgent ships as the standalone gaia-agent-chat wheel (#1102) and is
# NOT pulled by those extras — gaia.ui.server imports gaia_agent_chat to
# build the backend-default agent these scenarios run against, so install
# the LOCAL editable hub package or every scenario dies at session
# creation. Core gaia stays editable so eval fixtures resolve in-repo.
install-package: '-e .[dev,eval,ui,api] -e hub/agents/chat/python'
# `gaia eval agent` does not merely call the Anthropic API - it shells out
# to `claude -p` with an MCP config to DRIVE each scenario (runner.py:937),
# and its own preflight hard-fails with "'claude' CLI not found on PATH"
# before running anything. ANTHROPIC_API_KEY is necessary but not
# sufficient; the CLI has to be installed too. Same prerequisite the
# eval-CI recipe documents (docs/guides/eval-ci.mdx).
- name: Set up Node
uses: actions/setup-node@v7
with:
node-version: '22'
- name: Install Claude Code CLI (the eval's scenario driver)
run: |
npm install -g @anthropic-ai/claude-code
if ($LASTEXITCODE -ne 0) { throw "Claude Code CLI install failed" }
# Fail here, not after the backend has booted.
claude --version
if ($LASTEXITCODE -ne 0) { throw "claude installed but not runnable on PATH" }
- name: Install Lemonade Server
uses: ./.github/actions/install-lemonade
- name: Start Lemonade Server (version-matched, warmed)
timeout-minutes: 15
env:
HUGGINGFACE_ACCESS_TOKEN: ${{ secrets.HUGGINGFACE_ACCESS_TOKEN }}
HF_TOKEN: ${{ secrets.HUGGINGFACE_ACCESS_TOKEN }}
run: |
# Launches the version-matched server as a persistent Scheduled Task
# (survives across CI jobs), reuses it if healthy, and warms the model.
# Default -WarmModel is already Gemma-4-E4B-it-GGUF, the model under test.
powershell -ExecutionPolicy Bypass -File installer\scripts\ensure-lemonade-running.ps1
if ($LASTEXITCODE -ne 0) { throw "Lemonade server not ready" }
- name: Preflight — judge key, baselines, Lemonade
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
$ErrorActionPreference = "Stop"
$ProgressPreference = "SilentlyContinue"
if (-not $env:ANTHROPIC_API_KEY) {
Write-Host "::error::ANTHROPIC_API_KEY is not set. The eval judge (src/gaia/eval/claude.py) reads it from the environment; without it every scenario scores as an infra error and the scorecard is meaningless. Add the ANTHROPIC_API_KEY repository secret, or run the eval locally on AMD hardware."
exit 1
}
foreach ($category in @("rag_quality", "context_retention", "tool_selection")) {
$baseline = Join-Path $env:BASELINE_DIR "scorecard_$category.json"
if (-not (Test-Path $baseline)) {
Write-Host "::error::Missing baseline $baseline. Pick the baseline matching the model under test; do not substitute another directory."
exit 1
}
Write-Host "baseline ok: $baseline"
}
# Resolve the URL and the model under test through the client's own
# constants rather than hardcoding them here: the default port already
# moved once (8000 -> 13305 in Lemonade v10.1.0) and a stale literal
# would fail a healthy runner.
# The embedder's checkpoint/recipe come from the same MODELS entry
# gaia.rag.sdk registers it from, so the probe below cannot drift from
# what RAG will actually ask Lemonade for.
$global:LASTEXITCODE = 0
$probe = @(python -c "from gaia.llm.lemonade_client import DEFAULT_EMBEDDING_MODEL, DEFAULT_MODEL_NAME, MODELS, _get_lemonade_config; m = next(x for x in MODELS.values() if x.model_id == DEFAULT_EMBEDDING_MODEL); print(_get_lemonade_config()[2]); print(DEFAULT_MODEL_NAME); print(DEFAULT_EMBEDDING_MODEL); print(m.checkpoint); print(m.recipe)")
if ($LASTEXITCODE -ne 0 -or $probe.Count -lt 5) {
Write-Host "::error::Could not resolve the Lemonade URL / models under test - the venv install is broken."
exit 1
}
$lemonadeUrl = $probe[-5]
$modelUnderTest = $probe[-4]
$embeddingModel = $probe[-3]
$embeddingCheckpoint = $probe[-2]
$embeddingRecipe = $probe[-1]
try {
Invoke-RestMethod -Uri "$lemonadeUrl/health" -TimeoutSec 10 | Out-Null
} catch {
Write-Host "::error::Lemonade Server not reachable at $lemonadeUrl. The previous step should have started it; check its log, start it on the runner (installer\scripts\ensure-lemonade-running.ps1), or set LEMONADE_BASE_URL to a reachable instance."
exit 1
}
Write-Host "Lemonade reachable at $lemonadeUrl"
# ensure-lemonade-running.ps1 SWALLOWS a failed warm pull ("WARN: warm
# pull failed") and still exits 0, so a healthy server proves nothing
# about the model this job exists to measure. `/models` (no show_all)
# lists DOWNLOADED models only, so this is the assertion that the model
# is actually on disk. Without it a run can score every scenario against
# whatever was resident and compare it to the Gemma baseline.
try {
$downloaded = @((Invoke-RestMethod -Uri "$lemonadeUrl/models" -TimeoutSec 60).data | ForEach-Object { $_.id })
} catch {
Write-Host "::error::Could not list downloaded models at $lemonadeUrl/models. Lemonade answered /health but not /models - check the server log on the runner."
exit 1
}
if ($downloaded -notcontains $modelUnderTest) {
Write-Host "::error::$modelUnderTest is not downloaded on this runner (Lemonade has: $($downloaded -join ', ')). The warm-pull step failed silently. Pull it on the runner (POST $lemonadeUrl/pull with model_name=$modelUnderTest) and re-run; do NOT let this eval score a different model against the Gemma baseline."
exit 1
}
Write-Host "Model under test present: $modelUnderTest"
# THE RAG CATEGORIES RUN ON A SECOND MODEL. rag_quality and
# context_retention answer entirely out of chunks the EMBEDDER
# produced, so an embedder that cannot start turns every doc-grounded
# scenario into a ~2/10 non-answer that reads exactly like a model
# regression. ensure-lemonade-running.ps1 warms only the chat model
# and nothing asserted the embedder, so run 31051933420 spent 3h31m
# proving it: llama-server returned model_load_error on every index,
# RAG got 0 chunks, and rag_quality scored 3.1 against a 9.4 baseline
# captured with the SAME chat model. Probe it here, in seconds.
#
# Probed AFTER the chat model is warm on purpose: the embedder is
# co-resident with it (#1544), so probing an idle server would not
# reproduce the condition the eval actually runs under.
#
# Presence in /models is not enough - it proves the GGUF is on disk,
# not that llama-server can serve it, and "on disk but unservable" is
# the exact failure above. tests/ci_lemonade_check.py is the shared CI
# helper for precisely this (test_embeddings.yml, test_rag.yml,
# test_api.yml all use it, and it is green on the sibling stx runners),
# so this reuses it rather than hand-rolling a weaker probe:
#
# --register-embedding DELETES any prior registration and re-registers
# with the embeddings label before pulling. That is the one failure
# in this class CI can actually REPAIR: a stale registration left by
# an earlier job (wrong recipe, missing label - #1745) makes every
# subsequent load fail forever, and ensure_model_downloaded alone
# short-circuits on "already downloaded" and never re-applies it.
# --llamacpp-args loads with the SAME flag gaia.rag.sdk passes
# (_load_embedder: --ubatch-size 2048). Without it the probe proves
# the embedder can be served, not that RAG's load of it succeeds -
# which is the difference between a green preflight and a green run.
# the client's load retry absorbs the known transient AMD-Vulkan
# "llama-server failed to start" fault instead of tripping on it.
#
# Continue, not Stop: under Stop the helper's first stderr line becomes
# a terminating NativeCommandError and the step dies before the
# ::error:: below. Same reasoning as the eval step's own note.
$ErrorActionPreference = "Continue"
$global:LASTEXITCODE = 0
$embedOut = & python tests/ci_lemonade_check.py `
--model $embeddingModel `
--checkpoint $embeddingCheckpoint `
--recipe $embeddingRecipe `
--register-embedding `
--embeddings `
--llamacpp-args "--ubatch-size 2048" 2>&1
$embedCode = $LASTEXITCODE
$ErrorActionPreference = "Stop"
$embedOut | ForEach-Object { Write-Host "$_" }
# Assert the EMBEDDINGS line, not the helper's terminal `[ci] OK` - that
# one prints whether or not --embeddings was passed, so a dropped flag
# would take this preflight green having verified nothing. `dim=` only
# prints after a real vector came back. Exit 0 alone is not enough
# either: a command that fails to LAUNCH leaves $LASTEXITCODE at the
# reset 0 and would read as a pass.
if ($embedCode -ne 0 -or -not ($embedOut -match '\[ci\] embeddings OK \(dim=')) {
# Why llama-server refused to start is in Lemonade's own log, which
# the probe cannot see. Print it so the fix does not need an RDP
# session onto the box. The server runs as the SYSTEM scheduled task
# (ensure-lemonade-running.ps1), so the log is in the SYSTEM
# profile's ~/.cache/lemonade - server.log per lemonade_client.py,
# but glob both that dir and a logs/ subdir since the layout has
# moved before.
Write-Host "=== Lemonade server log (tail) ==="
$logGlobs = @(
"$env:SystemRoot\System32\config\systemprofile\.cache\lemonade\*.log",
"$env:SystemRoot\System32\config\systemprofile\.cache\lemonade\logs\*.log",
"$env:USERPROFILE\.cache\lemonade\*.log",
"$env:USERPROFILE\.cache\lemonade\logs\*.log"
)
$logged = $false
Get-ChildItem $logGlobs -ErrorAction SilentlyContinue |
Sort-Object LastWriteTime -Descending | Select-Object -First 1 |
ForEach-Object {
Write-Host "--- $($_.FullName) ---"
Get-Content $_.FullName -Tail 120 -ErrorAction SilentlyContinue
$logged = $true
}
$logHint = if ($logged) { "the llama-server error is in the Lemonade log tail above" } else { "no Lemonade log was found on this runner (looked in: $($logGlobs -join ', ')) - read llama-server's failure from the Lemonade console instead" }
if (-not $logged) { Write-Host "(none found)" }
Write-Host "::error::The RAG embedder $embeddingModel cannot serve embeddings on this runner (probe exit $embedCode; its output is above). rag_quality and context_retention would score ~2/10 as non-answers against a ~9/10 baseline and look like a model regression. The probe already re-registered and re-pulled the model, so a load that still fails is the RUNNER, not the registration: $logHint, then on sjlab-stx-halo-18 run ``powershell -File installer\scripts\ensure-lemonade-running.ps1 -ForceRestart`` and re-run this probe by hand (``python tests/ci_lemonade_check.py --model $embeddingModel --checkpoint $embeddingCheckpoint --recipe $embeddingRecipe --register-embedding --embeddings --llamacpp-args '--ubatch-size 2048'``). Do NOT re-run the eval until it passes - it costs 3.5h and measures nothing."
exit 1
}
- name: Run the three eval categories serially
# Own timeout so an overrun fails HERE, by name, with the collect /
# integrity / compare / upload steps still running on the partial results.
# A job-level timeout instead CANCELS, which skips every `!cancelled()`
# step and discards the categories that did finish. See the budget
# derivation on the job above. Same convention as email_scorecard_refresh.yml.
timeout-minutes: 400
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
# Deliberately NOT "Stop": the eval's stderr is piped through `2>&1`
# below, and under Stop the first stderr line from a native command
# becomes a terminating NativeCommandError. Every failure here is
# checked explicitly instead.
$ErrorActionPreference = "Continue"
$ProgressPreference = "SilentlyContinue"
# PYTHONIOENCODING makes the eval emit UTF-8; without this PowerShell
# decodes it as the OEM codepage and the logs come back as mojibake.
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8
New-Item -ItemType Directory -Force -Path eval-out | Out-Null
function Test-Backend {
try { Invoke-RestMethod -Uri "$env:BACKEND_URL/api/health" -TimeoutSec 5 | Out-Null; return $true }
catch { return $false }
}
# A leftover backend from a cancelled run still holds :4200 on this
# persistent runner, and the cleanup below does not fire on job
# cancellation. Without this check the wait loop can get 200 from that
# STALE server and evaluate the previous checkout while reporting green.
if (Test-Backend) {
Write-Host "::error::$env:BACKEND_URL is already serving - a stale GAIA backend is running on this runner. Kill it (gaia kill) and re-run; otherwise this eval measures the previous checkout, not this one."
exit 1
}
# The GAIA UI backend must live in the SAME shell session as the evals:
# a backgrounded child does not survive a step boundary, which is why
# test_agent_behavior_e2e.yml starts its server and runs its harness in
# one session. Hence backend start + all three categories in one step.
# Start-Process resolves relative paths against the .NET process CWD,
# which is not guaranteed to track the PowerShell location. Pin the
# child's working directory AND give it absolute redirect targets, so
# the backend and its logs both land in the workspace.
$outLog = Join-Path $PWD "eval-out\backend.out.log"
$errLog = Join-Path $PWD "eval-out\backend.err.log"
$backend = Start-Process -FilePath "python" `
-ArgumentList "-m","gaia.ui.server","--port","4200","--host","127.0.0.1" `
-WorkingDirectory $PWD `
-RedirectStandardOutput $outLog `
-RedirectStandardError $errLog `
-NoNewWindow -PassThru
if (-not $backend) {
Write-Host "::error::Could not start gaia.ui.server - the venv install is broken or python is not on PATH."
exit 1
}
try {
# 120 x 5s sleeps = ~10 min of waiting, more when a probe itself
# times out. The job timeout is the real backstop.
Write-Host "Waiting ~10 min for the GAIA UI backend on $env:BACKEND_URL (pid $($backend.Id))..."
$ready = $false
foreach ($attempt in 1..120) {
if ($backend.HasExited) {
Write-Host "::error::gaia.ui.server exited during startup (exit $($backend.ExitCode)). Backend log follows."
Get-Content $outLog, $errLog -ErrorAction SilentlyContinue
exit 1
}
if (Test-Backend) { $ready = $true; break }
Start-Sleep -Seconds 5
}
if (-not $ready) {
Write-Host "::error::GAIA UI backend did not become healthy at $env:BACKEND_URL/api/health in time. Backend log follows."
Get-Content $outLog, $errLog -ErrorAction SilentlyContinue
exit 1
}
Write-Host "Backend healthy."
# Per-category wall clock. The job timeout was guessed once and cost a
# 148-min run (30869194618) that produced one category out of three;
# these lines are what replaces the guess with a measurement. Each is
# written the moment its category ends - appended to the step summary,
# not buffered to the end - so a run that overruns or crashes still
# leaves behind the timings of whatever DID finish.
#
# Not `Out-File -Encoding utf8`: on Windows PowerShell 5.1 that
# prepends a BOM, and a BOM ahead of the leading `##` stops GitHub
# rendering it as a heading. AppendAllText with UTF8Encoding($false)
# has an empty preamble, so no BOM on create or append.
$summaryEnc = New-Object System.Text.UTF8Encoding($false)
function Add-Summary($line) {
if ($env:GITHUB_STEP_SUMMARY) {
[System.IO.File]::AppendAllText($env:GITHUB_STEP_SUMMARY, "$line`r`n", $summaryEnc)
}
}
$swTotal = [Diagnostics.Stopwatch]::StartNew()
Add-Summary "## Eval wall clock (sjlab-stx-halo-18)"
Add-Summary ""
Add-Summary "| category | minutes |"
Add-Summary "| --- | ---: |"
# Serial by construction: a plain foreach in one session, never
# backgrounded. Two concurrent `gaia eval agent` runs race-evict each
# other's model out of the single Lemonade slot (CLAUDE.md).
#
# `gaia eval agent --category X` exits 0 whatever the scenarios score -
# it writes the scorecard and returns. So a non-zero exit here is a real
# harness crash, never a regression; the regression verdict comes from
# the --compare step below.
#
# No --agent-type: the committed baselines were produced against the
# backend default (their `config` records only backend_url/model/budget),
# so passing one here would compare against a different agent.
foreach ($category in @("rag_quality", "context_retention", "tool_selection")) {
Write-Host "=================================================================="
Write-Host " gaia eval agent --category $category"
Write-Host "=================================================================="
$swCat = [Diagnostics.Stopwatch]::StartNew()
# Reset first: a command that fails to LAUNCH leaves $LASTEXITCODE
# at its previous value, so an unset/stale 0 would read as success.
$global:LASTEXITCODE = 0
# Not Tee-Object: on Windows PowerShell 5.1 it has no -Encoding and
# writes UTF-16, which the `Output:` parse in the next step then has
# to guess at. Echo + Out-File keeps the log UTF-8 and greppable.
gaia eval agent --category $category --backend "$env:BACKEND_URL" --model "$env:EVAL_MODEL" 2>&1 |
ForEach-Object { Write-Host "$_"; "$_" } |
Out-File -FilePath "eval-out\$category.log" -Encoding utf8
# Capture BEFORE anything else can clobber it. .NET calls and
# Write-Host leave $LASTEXITCODE alone, but reading it first keeps
# that from being a thing a later edit has to remember.
$code = $LASTEXITCODE
$swCat.Stop()
$mins = [math]::Round($swCat.Elapsed.TotalMinutes, 1)
Write-Host "[TIMING] ${category}: $mins min (running total $([math]::Round($swTotal.Elapsed.TotalMinutes, 1)) min of the 400-min step budget)"
Add-Summary "| ``$category`` | $mins |"
if ($code -ne 0) {
# Never `exit $LASTEXITCODE` - on a launch failure it can be $null,
# which PowerShell turns into exit 0 and the step passes GREEN
# having measured nothing.
Write-Host "::error::gaia eval agent --category $category crashed with exit $code (a harness failure, not a regression verdict)."
exit 1
}
}
$swTotal.Stop()
$totalMins = [math]::Round($swTotal.Elapsed.TotalMinutes, 1)
Write-Host "[TIMING] all three categories: $totalMins min"
Add-Summary "| **total** | **$totalMins** |"
}
finally {
if ($backend -and -not $backend.HasExited) {
Write-Host "Stopping GAIA UI backend (pid $($backend.Id))..."
Stop-Process -Id $backend.Id -Force -ErrorAction SilentlyContinue
}
}
- name: Collect scorecards from the printed run directories
# Not always(): on a concurrency cancel or a job-level timeout this would
# pile misleading "missing scorecard" errors on top of the cancellation.
# An eval STEP timeout is deliberately not in that bucket - it fails
# rather than cancels, so this runs and names exactly which categories
# went unmeasured, which is the diagnostic you want.
if: ${{ !cancelled() }}
run: |
$ErrorActionPreference = "Stop"
$missing = $false
foreach ($category in @("rag_quality", "context_retention", "tool_selection")) {
$log = "eval-out\$category.log"
if (-not (Test-Path $log)) {
Write-Host "::error::No log for $category - the chain stopped before it ran."
$missing = $true
continue
}
# The runner prints an ABSOLUTE `Output: <run-dir>` line
# (src/gaia/eval/runner.py::_print_summary) - a Windows path with
# backslashes. Take the last match so a re-run inside one invocation
# wins. Never glob by mtime.
$hit = Select-String -Path $log -Encoding UTF8 -Pattern '^Output:\s*(.+?)\s*$' | Select-Object -Last 1
$runDir = if ($hit) { $hit.Matches[0].Groups[1].Value } else { $null }
$scorecard = if ($runDir) { Join-Path $runDir "scorecard.json" } else { $null }
if (-not $runDir -or -not (Test-Path $scorecard)) {
$shown = if ($runDir) { $runDir } else { "<none>" }
Write-Host "::error::Could not resolve a scorecard for $category (parsed run dir: '$shown'). The eval did not print an 'Output:' line, or it did not write scorecard.json - see eval-out\$category.log."
$missing = $true
continue
}
New-Item -ItemType Directory -Force -Path "eval-out\$category" | Out-Null
Copy-Item $scorecard "eval-out\$category\scorecard.json"
# Traces are what you actually read to triage a regression - a bare
# scorecard tells you a score moved, not why. Best-effort though: a
# deep trace tree can blow MAX_PATH, and losing the triage material
# must not red a job whose scorecard was already collected.
$traces = Join-Path $runDir "traces"
if (Test-Path $traces) {
try {
Copy-Item $traces "eval-out\$category\traces" -Recurse
} catch {
Write-Host "::warning::${category}: could not copy traces from $traces ($($_.Exception.Message)). The scorecard was collected; only the triage detail is missing."
}
}
Write-Host "${category}: $runDir"
}
if ($missing) { exit 1 }
- name: Integrity gate — did we actually measure? (ignores `enforce`)
if: ${{ !cancelled() }}
run: |
$ErrorActionPreference = "Stop"
# UNCONDITIONAL. `--compare` only scores scenarios present on both sides,
# so a run whose scenarios vanished exits 0 and would pass as "no
# regression" (verified: 6 of 7 dropped -> exit 0). `enforce` decides
# whether a WORSE result blocks; it must never decide whether a
# NON-EXISTENT result blocks.
#
# Statuses that mean "no measurement was produced", as distinct from
# "measured and failed" (FAIL is a legitimate, comparable outcome).
#
# `skipped` (SKIPPED_NO_DOCUMENT, corpus file absent from disk) is the
# subtlest of these and the reason this list is not just the obvious
# errors: those scenarios KEEP their ids, so the missing-scenario check
# below stays clean, and compare_scorecards files them under
# `corpus_changed` = "not a quality signal" -> exit 0. A runner whose
# corpus never materialised would otherwise go green having measured
# nothing. All three committed baselines have skipped=0.
$noMeasurement = @("infra_error", "errored", "timeout", "blocked", "budget_exceeded", "skipped")
$problems = @()
foreach ($category in @("rag_quality", "context_retention", "tool_selection")) {
$current = "eval-out\$category\scorecard.json"
if (-not (Test-Path $current)) {
$problems += "${category}: no scorecard produced"
continue
}
$card = Get-Content $current -Raw -Encoding UTF8 | ConvertFrom-Json
$baseline = Get-Content (Join-Path $env:BASELINE_DIR "scorecard_$category.json") -Raw -Encoding UTF8 | ConvertFrom-Json
$got = @(@($card.scenarios) | ForEach-Object { $_.scenario_id })
$want = @(@($baseline.scenarios) | ForEach-Object { $_.scenario_id })
$absent = @($want | Where-Object { $got -notcontains $_ })
if ($absent.Count -gt 0) {
$problems += "${category}: $($absent.Count) baseline scenario(s) missing from the run: $(($absent | Sort-Object) -join ', ')"
}
$unmeasured = @()
foreach ($status in $noMeasurement) {
$count = $card.summary.$status
if ($count) { $unmeasured += "${status}=$count" }
}
if ($unmeasured.Count -gt 0) {
$problems += "${category}: scenarios without a measurement: $($unmeasured -join ', ')"
}
$shown = if ($unmeasured.Count -gt 0) { $unmeasured -join ', ' } else { "none" }
Write-Host "${category}: $($got.Count) scenario(s) run, $($want.Count) in baseline, unmeasured=$shown"
}
if ($problems.Count -gt 0) {
foreach ($p in $problems) { Write-Host "::error::Integrity: $p" }
Write-Host ""
Write-Host "The eval did not produce a complete set of measurements, so the baseline comparison below is not trustworthy. This fails regardless of ``enforce`` - report mode softens a WORSE score, never a MISSING one. Check the backend/Lemonade logs in the artifact, then re-run."
exit 1
}
Write-Host ""
Write-Host "Integrity OK - every baseline scenario ran and produced a measurement."
- name: Compare each category against its committed Gemma baseline
if: ${{ !cancelled() }}
run: |
$regressed = @()
$failedCompare = $false
foreach ($category in @("rag_quality", "context_retention", "tool_selection")) {
$current = "eval-out\$category\scorecard.json"
$baseline = Join-Path $env:BASELINE_DIR "scorecard_$category.json"
Write-Host ""
Write-Host "=================================================================="
Write-Host " ${category}: $baseline -> $current"
Write-Host "=================================================================="
if (-not (Test-Path $current)) {
Write-Host "::warning::${category}: no scorecard produced - nothing to compare."
$failedCompare = $true
continue
}
# `--compare` exits 2 on status/score/time regressions, 0 otherwise
# (src/gaia/cli.py). It only DIFFS - the eval above already ran.
# Reset first: a launch failure would otherwise inherit the PREVIOUS
# category's 2 and be misreported as this category regressing.
$global:LASTEXITCODE = 0
gaia eval agent --compare "$baseline" "$current"
switch ($LASTEXITCODE) {
0 { Write-Host "${category}: no regression vs baseline." }
2 {
Write-Host "::warning::${category}: regression detected vs $baseline (see the diff above)."
$regressed += $category
}
default {
Write-Host "::error::${category}: compare failed with exit $LASTEXITCODE (not a regression verdict - a harness/IO error)."
$failedCompare = $true
}
}
}
Write-Host ""
Write-Host "=================================================================="
Write-Host " SUMMARY - enforce=$env:ENFORCE"
Write-Host "=================================================================="
if ($regressed.Count -gt 0) {
Write-Host "Regressed categories: $($regressed -join ' ')"
} else {
Write-Host "Regressed categories: none"
}
Write-Host "Baselines: $env:BASELINE_DIR"
Write-Host "Scorecards uploaded as the 'agent-eval-gemma-consolidation' artifact."
Write-Host ""
Write-Host "Code-generation and data-analysis scenarios are EXPECTED to regress on"
Write-Host "this consolidation (35B MoE -> ~4B dense) and that tradeoff is accepted."
Write-Host "Read the per-scenario diff above before treating a regression as a bug."
Write-Host ""
Write-Host "TIME regressions (>2x per-scenario wall clock) count toward the same"
Write-Host "verdict but are NOT comparable to this baseline: it was captured against"
Write-Host "a remote Lemonade over ngrok on different hardware. Treat a time-only"
Write-Host "regression as a hardware artifact until the baselines are regenerated"
Write-Host "on this runner."
Write-Host "=================================================================="
# A harness/IO error is never tolerated - it means we did not actually measure.
if ($failedCompare) {
Write-Host "::error::One or more comparisons could not be performed. Failing regardless of enforce."
exit 1
}
if ($regressed.Count -gt 0) {
if ($env:ENFORCE -eq "true") {
Write-Host "::error::Regressions in: $($regressed -join ' '). enforce=true - failing the build. Fix the prompt/tooling and re-run, or (if the regression is intentional) regenerate the baseline with --save-baseline and call it out in the PR description."
exit 1
}
Write-Host "Report mode (enforce=false): regressions logged as warnings, build stays green."
Write-Host "Re-dispatch this workflow with enforce=true to make them blocking."
}
# THIS LINE IS THE REPORT MODE. GitHub appends
# `if (Test-Path variable:\LASTEXITCODE) { exit $LASTEXITCODE }` to every
# PowerShell step, so falling off the end here would re-export the LAST
# `--compare` exit code: a report-mode regression in the final category
# would fail the build anyway, and whether it did would depend on which
# category regressed. Same guard as install-lemonade/action.yml.
$global:LASTEXITCODE = 0
- name: Upload scorecards + eval logs
if: always()
uses: actions/upload-artifact@v7
with:
name: agent-eval-gemma-consolidation
path: eval-out/
if-no-files-found: warn