Commit 4a24d66
ci(email): run the triage eval on PRs that touch the email agent (#2849)
Changes to the email agent could merge with no evidence they hadn't
broken its behavior. The eval that would catch that only ran weekly, on
demand, and at release time — never on a PR — so pytest went green and
the change landed. #2695 is the proof: it injected three Agent Skills
into the agent's system prompt by default and re-sized the bulk-triage
envelope budget, having already blown the 16,384-token context window
once at 16,602 tokens during development. It merged at 16,106 tokens —
1.7% headroom, measured by one manual run and pinned by nothing. Now a
PR touching the email agent runs the real
triage/drafting/action-item/briefing eval on AMD hardware and reports
every gate verdict in the job summary.
It stays affordable on the single serial Lemonade slot: narrow path
filters, a per-PR concurrency group that cancels superseded runs, a
20-email slice instead of 50, and no fork or draft runs.
**Advisory, not blocking — and that is not a special case for PRs.**
Every gate manifest under `tests/fixtures/email/` ships `enforce: false`
today, so no gate blocks anything on any trigger. A "report only on PRs"
override was written and deliberately removed: with `enforce:false`
everywhere, the only exits it could soften are the fail-loudly ones
(missing judge credential, unjudged case, harness crash) — the inverse
of report mode. To make a gate block, flip `"enforce": false` → `true`
in that gate's manifest; it then blocks the weekly, the release and PRs
together. What still fails a PR run loudly: a missing judge key, an
unreachable Lemonade, or a benchmark that dies — which is what a
#2695-style context blowout looks like.
**Scorecard staleness: deferred, not dropped.**
`hub/agents/email/npm/SCORECARD.md` can only be regenerated by a ~10h
full-profile run, so a per-PR "you didn't regenerate it" check would
fire on every email PR and be ignored within a week. The honest place
for it is the release gate, which already parses the card. Filed as
#2850 rather than added here; the job summary states plainly that the
card is not refreshed by this run.
## Test plan
- [ ] Workflow YAML parses and the job `if:` folds to a single-line
expression (verified locally; `python -c "import yaml;
yaml.safe_load(...)"`)
- [ ] Path globs match real files — all 8 verified against `git
ls-files`, and the #2695 commit hits 22 of them (so it would have
triggered)
- [ ] Serial group name is byte-identical to the three siblings: `git
grep -n "group: lemonade-eval" .github/workflows/` returns
`email_scorecard_refresh.yml`,
`test_eval_agent_gemma_consolidation.yml`, `test_eval_rag.yml` and this
file's job
- [ ] `python -m pytest
hub/agents/email/python/tests/test_eval_summary.py` — 14 pass. The two
skip-path tests fail if `find_gates` drops the `skipped` key, confirmed
by reverting the predicate
- [ ] `python util/lint.py --all` clean; `black`/`isort` clean on both
new files
- [ ] Every PowerShell `run:` block parses, including when decoded as
ANSI (the runner writes the script without a BOM; a UTF-8 em dash would
otherwise mojibake into a smart quote and terminate a string)
- [ ] After merge: open a no-op PR touching `hub/agents/email/python/**`
and confirm the job queues on `lemonade-eval`, and that a second push
cancels the first run
- [ ] After merge: confirm the "Gate verdicts" job summary renders and
annotates
Co-authored-by: Ovtcharov <kovtchar@amd.com>1 parent c3d9ec1 commit 4a24d66
4 files changed
Lines changed: 729 additions & 60 deletions
File tree
- .github/workflows
- hub/agents/email/python
- packaging
- tests
0 commit comments