You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
## Why this matters
Five E2E findings on the Email Triage agent, batched because they share
the same package and (for the LLM-affecting ones) a serial eval.
Per-issue user-observable impact:
- **#2116** — Before: a fresh Google Cloud project without the
Gmail/Calendar API enabled dumped Google's raw 403 JSON into the tool
result. After: GAIA's three-part actionable error naming the enable URL
Google returns in `extendedHelp` + the 1-2 minute propagation note
(shared `google_errors` helper, both backends).
- **#2114** — Before: "archive the Netflix promo" searched the literal
phrase → zero hits → "couldn't find it" even though the message was
present. After: docstring + prompt steer toward `from:`/`subject:`
operators, and a bare-phrase zero-result query retries once as an
operator query. `triage_inbox` gets timeout headroom (600s) over the
180s default for full-inbox scans on slow hardware. _(Maintainer
re-verified the phishing-timeout half no longer reproduces on main;
narrowed to the search-operator half per that comment.)_
- **#2115** — Before: final answers could be a bare render fence with no
prose (CLI/integrators saw empty replies) and carried LaTeX artifacts
(`$\rightarrow$`); the settings-path label drifted between
"Connectors"/"Connections". After: prompt requires prose alongside any
render payload and forbids TeX; LaTeX normalized at the output boundary;
every settings-path string says "Connectors" (Outlook backends +
connectors-demo + webui comment) with a grep-level consistency guard.
- **#2113** — Before: a Gmail promotions/social/updates label
short-circuited the heuristic to a confident classification before any
body read, so a real deadline / attendance requirement / "budget
exceeded" consequence was confidently archived or filed informational.
After: a body-signal veto forces LLM escalation when a
deadline/commitment/consequence signal is present; ordinary marketing
urgency ("sale ends", "limited time") still archives confidently. Adds
synthetic commitment cases to the eval corpus seed.
- **#2110** — Before: "daily briefing" and "extract action items" had no
agent-loop tool and silently degraded to a raw `pre_scan_inbox` fence
(dishonest by omission). After: `get_briefing`, `list_tasks`, and
`extract_action_items` are registered agent-loop tools; the prompt binds
each NL ask to its dedicated tool. `extract_action_items` drives a fresh
scan so a cold "what do I need to do?" works. Capability matrix +
`tools_count` (52→55) + guide synced.
## Status — ready for review
All five issues are implemented, rebased on current `main`, and the full
deterministic CI is green (unit + integration email suites,
connectors-demo, code-quality/lint). No stubs or half-finished paths.
**Verified in CI:**
- #2113 veto mechanism: promotions/social/updates + commitment signal →
`confident=False` escalation; ordinary promos unchanged; corpus cases
present with needs-attention ground truth.
- #2110 tool registration + behavior: all three tools register;
`get_briefing` cold-generates and returns persisted;
`extract_action_items` drives a scan and populates the task store from
cold; `list_tasks` reads back; drift guards (capability matrix,
`tools_count` across 3 sources, and the `test_email_agent` tool-registry
allowlist) green.
- #2114 operator-retry logic + #2116 403 mapping + #2115 LaTeX/naming:
fully unit-tested.
**LLM behavioral eval — self-hosted lane, not a PR gate:** the email
triage benchmark / scorecard (`Email Agent Eval` workflow) runs on the
self-hosted Windows/strix-halo Lemonade pool via weekly cron +
`workflow_dispatch`; it is not part of PR-gating CI. Its committed
baselines are AMD-hardware-calibrated, so it must be refreshed on that
pool — running it on non-AMD dev hardware would produce non-comparable
numbers. Recommended before/after merge:
- [ ] `workflow_dispatch` the `Email Agent Eval` workflow on the
strix-halo pool; confirm #2113 commitment cases hit the recall bar
without regressing urgent/needs-response, and regenerate the AMD
baselines if the batch shifts them.
## Test plan
- [x] Rebased on current `origin/main` (clean merge — no conflicts).
- [x] `python util/lint.py --all` — green.
- [x] Email Agent Unit Tests (py3.10 + py3.12 CI matrix) — green (2603
passed / 6 skipped locally in a clean-install venv).
- [x] Test Email Agent (integration + repo-side email unit suite) —
green after adding the #2110 tools to the `test_email_agent` registry
allowlist.
- [x] Test Connectors Demo Agent — green (naming change).
Part of #2014Closes#2116Closes#2114Closes#2115Closes#2113Closes#2110
---------
Co-authored-by: Ovtcharov <kovtchar@amd.com>
Co-authored-by: Kalin Ovtcharov <kalin@extropolis.ai>
0 commit comments