Skip to content

Commit 4a24d66

Browse files
kovtcharov-amdOvtcharov
andauthored
ci(email): run the triage eval on PRs that touch the email agent (#2849)
Changes to the email agent could merge with no evidence they hadn't broken its behavior. The eval that would catch that only ran weekly, on demand, and at release time — never on a PR — so pytest went green and the change landed. #2695 is the proof: it injected three Agent Skills into the agent's system prompt by default and re-sized the bulk-triage envelope budget, having already blown the 16,384-token context window once at 16,602 tokens during development. It merged at 16,106 tokens — 1.7% headroom, measured by one manual run and pinned by nothing. Now a PR touching the email agent runs the real triage/drafting/action-item/briefing eval on AMD hardware and reports every gate verdict in the job summary. It stays affordable on the single serial Lemonade slot: narrow path filters, a per-PR concurrency group that cancels superseded runs, a 20-email slice instead of 50, and no fork or draft runs. **Advisory, not blocking — and that is not a special case for PRs.** Every gate manifest under `tests/fixtures/email/` ships `enforce: false` today, so no gate blocks anything on any trigger. A "report only on PRs" override was written and deliberately removed: with `enforce:false` everywhere, the only exits it could soften are the fail-loudly ones (missing judge credential, unjudged case, harness crash) — the inverse of report mode. To make a gate block, flip `"enforce": false` → `true` in that gate's manifest; it then blocks the weekly, the release and PRs together. What still fails a PR run loudly: a missing judge key, an unreachable Lemonade, or a benchmark that dies — which is what a #2695-style context blowout looks like. **Scorecard staleness: deferred, not dropped.** `hub/agents/email/npm/SCORECARD.md` can only be regenerated by a ~10h full-profile run, so a per-PR "you didn't regenerate it" check would fire on every email PR and be ignored within a week. The honest place for it is the release gate, which already parses the card. Filed as #2850 rather than added here; the job summary states plainly that the card is not refreshed by this run. ## Test plan - [ ] Workflow YAML parses and the job `if:` folds to a single-line expression (verified locally; `python -c "import yaml; yaml.safe_load(...)"`) - [ ] Path globs match real files — all 8 verified against `git ls-files`, and the #2695 commit hits 22 of them (so it would have triggered) - [ ] Serial group name is byte-identical to the three siblings: `git grep -n "group: lemonade-eval" .github/workflows/` returns `email_scorecard_refresh.yml`, `test_eval_agent_gemma_consolidation.yml`, `test_eval_rag.yml` and this file's job - [ ] `python -m pytest hub/agents/email/python/tests/test_eval_summary.py` — 14 pass. The two skip-path tests fail if `find_gates` drops the `skipped` key, confirmed by reverting the predicate - [ ] `python util/lint.py --all` clean; `black`/`isort` clean on both new files - [ ] Every PowerShell `run:` block parses, including when decoded as ANSI (the runner writes the script without a BOM; a UTF-8 em dash would otherwise mojibake into a smart quote and terminate a string) - [ ] After merge: open a no-op PR touching `hub/agents/email/python/**` and confirm the job queues on `lemonade-eval`, and that a second push cancels the first run - [ ] After merge: confirm the "Gate verdicts" job summary renders and annotates Co-authored-by: Ovtcharov <kovtchar@amd.com>
1 parent c3d9ec1 commit 4a24d66

4 files changed

Lines changed: 729 additions & 60 deletions

File tree

0 commit comments

Comments
 (0)