Skip to content

Commit c921fa7

Browse files
kovtcharov-amdOvtcharovkovtcharov
authored
feat(email): quality + robustness batch (#2110/#2113/#2114/#2115/#2116) (#2192)
## Why this matters Five E2E findings on the Email Triage agent, batched because they share the same package and (for the LLM-affecting ones) a serial eval. Per-issue user-observable impact: - **#2116** — Before: a fresh Google Cloud project without the Gmail/Calendar API enabled dumped Google's raw 403 JSON into the tool result. After: GAIA's three-part actionable error naming the enable URL Google returns in `extendedHelp` + the 1-2 minute propagation note (shared `google_errors` helper, both backends). - **#2114** — Before: "archive the Netflix promo" searched the literal phrase → zero hits → "couldn't find it" even though the message was present. After: docstring + prompt steer toward `from:`/`subject:` operators, and a bare-phrase zero-result query retries once as an operator query. `triage_inbox` gets timeout headroom (600s) over the 180s default for full-inbox scans on slow hardware. _(Maintainer re-verified the phishing-timeout half no longer reproduces on main; narrowed to the search-operator half per that comment.)_ - **#2115** — Before: final answers could be a bare render fence with no prose (CLI/integrators saw empty replies) and carried LaTeX artifacts (`$\rightarrow$`); the settings-path label drifted between "Connectors"/"Connections". After: prompt requires prose alongside any render payload and forbids TeX; LaTeX normalized at the output boundary; every settings-path string says "Connectors" (Outlook backends + connectors-demo + webui comment) with a grep-level consistency guard. - **#2113** — Before: a Gmail promotions/social/updates label short-circuited the heuristic to a confident classification before any body read, so a real deadline / attendance requirement / "budget exceeded" consequence was confidently archived or filed informational. After: a body-signal veto forces LLM escalation when a deadline/commitment/consequence signal is present; ordinary marketing urgency ("sale ends", "limited time") still archives confidently. Adds synthetic commitment cases to the eval corpus seed. - **#2110** — Before: "daily briefing" and "extract action items" had no agent-loop tool and silently degraded to a raw `pre_scan_inbox` fence (dishonest by omission). After: `get_briefing`, `list_tasks`, and `extract_action_items` are registered agent-loop tools; the prompt binds each NL ask to its dedicated tool. `extract_action_items` drives a fresh scan so a cold "what do I need to do?" works. Capability matrix + `tools_count` (52→55) + guide synced. ## Status — ready for review All five issues are implemented, rebased on current `main`, and the full deterministic CI is green (unit + integration email suites, connectors-demo, code-quality/lint). No stubs or half-finished paths. **Verified in CI:** - #2113 veto mechanism: promotions/social/updates + commitment signal → `confident=False` escalation; ordinary promos unchanged; corpus cases present with needs-attention ground truth. - #2110 tool registration + behavior: all three tools register; `get_briefing` cold-generates and returns persisted; `extract_action_items` drives a scan and populates the task store from cold; `list_tasks` reads back; drift guards (capability matrix, `tools_count` across 3 sources, and the `test_email_agent` tool-registry allowlist) green. - #2114 operator-retry logic + #2116 403 mapping + #2115 LaTeX/naming: fully unit-tested. **LLM behavioral eval — self-hosted lane, not a PR gate:** the email triage benchmark / scorecard (`Email Agent Eval` workflow) runs on the self-hosted Windows/strix-halo Lemonade pool via weekly cron + `workflow_dispatch`; it is not part of PR-gating CI. Its committed baselines are AMD-hardware-calibrated, so it must be refreshed on that pool — running it on non-AMD dev hardware would produce non-comparable numbers. Recommended before/after merge: - [ ] `workflow_dispatch` the `Email Agent Eval` workflow on the strix-halo pool; confirm #2113 commitment cases hit the recall bar without regressing urgent/needs-response, and regenerate the AMD baselines if the batch shifts them. ## Test plan - [x] Rebased on current `origin/main` (clean merge — no conflicts). - [x] `python util/lint.py --all` — green. - [x] Email Agent Unit Tests (py3.10 + py3.12 CI matrix) — green (2603 passed / 6 skipped locally in a clean-install venv). - [x] Test Email Agent (integration + repo-side email unit suite) — green after adding the #2110 tools to the `test_email_agent` registry allowlist. - [x] Test Connectors Demo Agent — green (naming change). Part of #2014 Closes #2116 Closes #2114 Closes #2115 Closes #2113 Closes #2110 --------- Co-authored-by: Ovtcharov <kovtchar@amd.com> Co-authored-by: Kalin Ovtcharov <kalin@extropolis.ai>
1 parent b9391ec commit c921fa7

24 files changed

Lines changed: 1302 additions & 71 deletions

docs/guides/email.mdx

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -233,6 +233,14 @@ are planned on the autonomy engine (#555).
233233

234234
`list_inbox`, `get_message`, `get_thread`, `search_messages`, `list_labels`, `triage_inbox`, `pre_scan_inbox`, `check_followups`
235235

236+
### Briefing & tasks
237+
238+
`get_briefing`, `list_tasks`, `extract_action_items` — natural-language asks like
239+
"give me a daily briefing", "what do I need to do from my inbox", and "show my
240+
tasks" bind to these dedicated tools. `extract_action_items` drives a fresh scan
241+
of your recent mail (it does not require a prior triage run), and `get_briefing`
242+
returns the latest scheduled briefing or generates one on demand.
243+
236244
### Classification preferences (persist across restarts)
237245

238246
`set_priority_sender`, `set_low_priority_sender`, `set_category_default`, `clear_session_preferences`

hub/agents/python/connectors-demo/gaia_agent_connectors_demo/agent.py

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
2. ``get_credential_sync(connector_id, agent_id, required_scopes)``
1515
— the central entrypoint that fires the grant-ledger check before
1616
returning a usable credential.
17-
3. The Settings → Connections per-agent grants UI — the user must be
17+
3. The Settings → Connectors per-agent grants UI — the user must be
1818
able to grant scopes from inside the AgentUI.
1919
2020
This agent ships four tools that fan out across two connector kinds:
@@ -92,9 +92,9 @@
9292
- Call exactly the tool that matches the question. Don't speculate;
9393
if the user asks "what's in my inbox?" call gmail_recent_subjects.
9494
- If a tool returns an error mentioning "AGENT_NOT_GRANTED", tell the
95-
user which scope they need to grant in Settings → Connections.
95+
user which scope they need to grant in Settings → Connectors.
9696
- If a tool returns an error mentioning "NOT_CONNECTED", tell them to
97-
connect that service in Settings → Connections first.
97+
connect that service in Settings → Connectors first.
9898
- Summarize tool output in 1–3 sentences. Don't recite raw JSON.
9999
- Do NOT make up data. If a tool fails, say so.
100100
"""
@@ -146,7 +146,7 @@ def _github_pat() -> str:
146146
if not token:
147147
raise ConnectorsError(
148148
"GitHub MCP credential resolved but GITHUB_TOKEN was empty. "
149-
"Re-run Settings → Connections → GitHub → Configure to set the "
149+
"Re-run Settings → Connectors → GitHub → Configure to set the "
150150
"Personal Access Token."
151151
)
152152
return token

hub/agents/python/email/CAPABILITY_MATRIX.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,8 @@ python hub/agents/python/email/packaging/capability_matrix.py
4646

4747
## Surface totals
4848

49-
- Internal `@tool` agent-loop functions: **52**
49+
- Internal `@tool` agent-loop functions: **55**
50+
- `briefing_tools`: 3
5051
- `calendar_tools`: 6
5152
- `delete_tools`: 3
5253
- `followup_tools`: 1

hub/agents/python/email/gaia-agent.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ icon: mail
1919
# tests/test_capability_matrix.py (AST-derived, reconciled against
2020
# __init__.py) and tests/test_email_agent.py (live registry, #1232);
2121
# a mismatch fails CI.
22-
tools_count: 52
22+
tools_count: 55
2323

2424
language: python
2525
min_gaia_version: "0.22.0"

hub/agents/python/email/gaia_agent_email/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -120,5 +120,5 @@ def email_factory(**kwargs):
120120
category="productivity",
121121
tags=["email", "gmail", "calendar", "triage"],
122122
icon="mail",
123-
tools_count=52, # guarded by tests/test_email_agent.py (#1232)
123+
tools_count=55, # guarded by tests/test_email_agent.py (#1232)
124124
)

hub/agents/python/email/gaia_agent_email/agent.py

Lines changed: 96 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ class never passes ``use_claude=True`` / ``use_chatgpt=True`` to
3737
from pathlib import Path
3838
from typing import Any, ClassVar, Dict, List, Optional
3939

40-
from gaia_agent_email import action_store, schedule_store
40+
from gaia_agent_email import action_store, schedule_store, task_store
4141
from gaia_agent_email.config import ConfigurationError, EmailAgentConfig
4242
from gaia_agent_email.model_select import (
4343
NPU_EMAIL_MODEL_ID,
@@ -53,6 +53,7 @@ class never passes ``use_claude=True`` / ``use_chatgpt=True`` to
5353
ALL_SCOPES,
5454
)
5555
from gaia_agent_email.supervision import is_daemon_supervised
56+
from gaia_agent_email.tools.briefing_tools import BriefingToolsMixin
5657
from gaia_agent_email.tools.calendar_tools import CalendarToolsMixin
5758
from gaia_agent_email.tools.delete_tools import DeleteToolsMixin
5859
from gaia_agent_email.tools.followup_tools import FollowupToolsMixin
@@ -174,10 +175,11 @@ def _detect_targeted_mailboxes(query: str) -> set:
174175
175176
ACTIONS:
176177
- Read tools (list_inbox, get_message, get_thread, search_messages,
177-
list_labels, triage_inbox, pre_scan_inbox, check_followups) — never
178-
require confirmation. check_followups flags sent mail still awaiting a
179-
reply; it only reports — never draft or send a follow-up nudge unless the
180-
user explicitly asks, and any send remains confirmation-gated.
178+
list_labels, triage_inbox, pre_scan_inbox, check_followups, get_briefing,
179+
list_tasks, extract_action_items) — never require confirmation.
180+
check_followups flags sent mail still awaiting a reply; it only reports —
181+
never draft or send a follow-up nudge unless the user explicitly asks, and
182+
any send remains confirmation-gated.
181183
- Organize tools (archive_message, mark_read, mark_unread, add_star,
182184
remove_star, label_message, move_to_label) — reversible via the undo
183185
log; do not require per-action confirmation, but bulk operations
@@ -220,6 +222,24 @@ def _detect_targeted_mailboxes(query: str) -> set:
220222
do not re-state its contents in prose. For follow-up questions about
221223
specific items, refer to the message_id values from the card.
222224
225+
ALWAYS write at least one sentence of plain prose in your final answer. A
226+
render payload (a ```email_pre_scan fence or any raw JSON) must NEVER stand
227+
alone as your entire reply — render-less consumers (CLI, integrators) see
228+
only your text, so a bare fence reads as an empty answer to them. If you
229+
have nothing to add beyond the card, still write the one framing sentence.
230+
231+
BRIEFING & TASKS:
232+
- For a daily briefing / morning brief / "summarize my inbox for today",
233+
call ``get_briefing`` — NOT ``pre_scan_inbox``. The briefing is the
234+
dedicated tool for that ask; do not fall back to a raw pre-scan.
235+
- For "extract action items" / "what do I need to do from my inbox", call
236+
``extract_action_items`` — it scans your recent mail and captures the
237+
to-dos even if you have not triaged yet.
238+
- For "show my tasks" / "what's on my task list", call ``list_tasks``
239+
(add status 'open' or 'done' to filter).
240+
Never answer any of these three asks with a bare ``pre_scan_inbox`` fence —
241+
each has its own tool.
242+
223243
MAILBOX TARGETING:
224244
Read/triage tools scan only CONNECTED mailboxes, and every result item is
225245
tagged with its source mailbox (google or microsoft). If the user asks
@@ -228,13 +248,74 @@ def _detect_targeted_mailboxes(query: str) -> set:
228248
NEVER present one mailbox's data as if it came from the provider the user
229249
asked for.
230250
251+
SEARCH:
252+
When searching, translate the user's words into Gmail operators — never pass
253+
the raw phrase to search_messages. "archive the Netflix promo email" →
254+
search_messages("from:netflix"), NOT search_messages("Netflix promotional
255+
email"). Map a sender/brand to ``from:``, expected subject words to
256+
``subject:``, and status/recency to ``is:unread`` / ``newer_than:7d`` /
257+
``label:promotions``. A literal-phrase search that returns zero results has
258+
almost certainly mis-formed the query — retry with ``from:``/``subject:``
259+
operators before telling the user the message can't be found.
260+
231261
OUTPUT:
232262
Tool results come back as JSON envelopes ``{"ok": true, "data": ...}``
233263
or ``{"ok": false, "error": "..."}``. Summarize tool output briefly for
234-
the user — do not recite raw JSON.
264+
the user — do not recite raw JSON. Write plain text only: use Unicode
265+
symbols directly (→, ≤, ×), never LaTeX/TeX markup like $\\rightarrow$.
235266
"""
236267

237268

269+
# ---------------------------------------------------------------------------
270+
# Output normalization
271+
# ---------------------------------------------------------------------------
272+
273+
# LaTeX/TeX commands that models sometimes emit inside plain-text answers
274+
# (e.g. ``$\rightarrow$`` instead of ``→``). Map them to the Unicode symbol.
275+
_LATEX_SYMBOLS = {
276+
r"\rightarrow": "→",
277+
r"\Rightarrow": "⇒",
278+
r"\leftarrow": "←",
279+
r"\Leftarrow": "⇐",
280+
r"\leftrightarrow": "↔",
281+
r"\to": "→",
282+
r"\times": "×",
283+
r"\div": "÷",
284+
r"\leq": "≤",
285+
r"\geq": "≥",
286+
r"\neq": "≠",
287+
r"\approx": "≈",
288+
r"\pm": "±",
289+
r"\cdot": "·",
290+
r"\ldots": "…",
291+
r"\bullet": "•",
292+
r"\deg": "°",
293+
}
294+
295+
# Match an optional ``$``/``\(`` math wrapper around a single known command,
296+
# so ``$\rightarrow$`` and a bare ``\rightarrow`` both normalize.
297+
_LATEX_CMD_RE = re.compile(
298+
r"\$?\\(" + "|".join(cmd[1:] for cmd in _LATEX_SYMBOLS) + r")\b\$?"
299+
)
300+
301+
302+
def _normalize_plain_text_answer(text: str) -> str:
303+
"""Strip LaTeX artifacts from a plain-text answer (#2115).
304+
305+
Models occasionally emit TeX markup (``$\\rightarrow$``) in prose meant
306+
to be plain text. Rewrite the known commands to their Unicode symbol so
307+
CLI / integrator consumers see ``→`` rather than raw TeX. Leaves text
308+
without any such artifact untouched.
309+
"""
310+
if not text or "\\" not in text:
311+
return text
312+
313+
def _sub(m: "re.Match[str]") -> str:
314+
return _LATEX_SYMBOLS["\\" + m.group(1)]
315+
316+
return _LATEX_CMD_RE.sub(_sub, text)
317+
318+
238319
# ---------------------------------------------------------------------------
239320
# Agent
240321
# ---------------------------------------------------------------------------
@@ -245,6 +326,7 @@ class EmailTriageAgent(
245326
MemoryMixin,
246327
DatabaseMixin,
247328
ReadToolsMixin,
329+
BriefingToolsMixin,
248330
FollowupToolsMixin,
249331
OrganizeToolsMixin,
250332
ReplyToolsMixin,
@@ -403,6 +485,7 @@ def __init__(self, config: Optional[EmailAgentConfig] = None):
403485
self.init_db(db_path)
404486
action_store.init_schema(self)
405487
schedule_store.init_schema(self)
488+
task_store.init_schema(self)
406489

407490
# LLM connection. Default to Lemonade — the config's base_url
408491
# allowlist guarantees the host is local. Resolved BEFORE init_memory()
@@ -654,7 +737,12 @@ def process_query(self, user_input: str, *args, **kwargs):
654737
guard = self._mailbox_target_guard(user_input)
655738
if guard is not None:
656739
return guard
657-
return super().process_query(user_input, *args, **kwargs)
740+
result = super().process_query(user_input, *args, **kwargs)
741+
# Normalize LaTeX artifacts at the output boundary so render-less
742+
# consumers never see raw TeX in the final answer (#2115).
743+
if isinstance(result, dict) and isinstance(result.get("result"), str):
744+
result["result"] = _normalize_plain_text_answer(result["result"])
745+
return result
658746

659747
def _mailbox_target_guard(self, user_input: str) -> Optional[Dict[str, Any]]:
660748
"""Reject a request that explicitly targets an unavailable mailbox (#2164).
@@ -715,6 +803,7 @@ def _register_tools(self) -> None:
715803
_TOOL_REGISTRY.clear()
716804
self._reset_organize_counter()
717805
self._register_read_tools()
806+
self._register_briefing_tools()
718807
self._register_followup_tools()
719808
self._register_organize_tools()
720809
self._register_reply_tools()

hub/agents/python/email/gaia_agent_email/api_routes.py

Lines changed: 48 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -412,6 +412,53 @@ def _split_sentences(text: str) -> List[str]:
412412
return [s.strip() for s in _SENTENCE_SPLIT_RE.split(text) if s.strip()]
413413

414414

415+
def extract_action_items_from_body(body: str) -> List[ActionItem]:
416+
"""Extract action items from a message body (cue-based, deterministic).
417+
418+
Module-level so both the REST triage path
419+
(``EmailTriageService._extract_action_items``) and the agent-loop
420+
``extract_action_items`` tool (#2110) share ONE extractor — the two
421+
surfaces can't drift.
422+
"""
423+
items: List[ActionItem] = []
424+
seen: set[str] = set()
425+
for sentence in _split_sentences(body):
426+
low = sentence.lower()
427+
if not any(cue in low for cue in _ACTION_CUES):
428+
continue
429+
normalized = sentence.strip()
430+
key = normalized.lower()
431+
if key in seen:
432+
continue
433+
seen.add(key)
434+
due_match = _DUE_HINT_RE.search(sentence)
435+
due_hint = due_match.group(1) if due_match else None
436+
url_match = _URL_RE.search(sentence)
437+
if url_match:
438+
# Trim trailing sentence punctuation the greedy match grabs
439+
# ("...report." → "...report") so the link is well-formed.
440+
# Strip char-by-char, but keep a ")" that closes a "(" inside
441+
# the URL itself (e.g. .../Python_(programming_language)) so we
442+
# don't silently truncate Wikipedia/Confluence-style links.
443+
url = url_match.group(0)
444+
_trailing = ".,;:!?)]}\"'"
445+
while url and url[-1] in _trailing:
446+
if url[-1] == ")" and "(" in url:
447+
break
448+
url = url[:-1]
449+
items.append(
450+
ActionItem(
451+
description=normalized,
452+
due_hint=due_hint,
453+
type="link",
454+
url=url,
455+
)
456+
)
457+
else:
458+
items.append(ActionItem(description=normalized, due_hint=due_hint))
459+
return items
460+
461+
415462
def _aggregate_usage(call_stats: List[dict]) -> Optional[TriageUsage]:
416463
"""Sum the per-call usage/stats entries across the classify + summarize
417464
LLM calls into a single :class:`TriageUsage`.
@@ -901,43 +948,7 @@ def _summarize(self, subject: str, body: str) -> str:
901948
return summary
902949

903950
def _extract_action_items(self, body: str) -> List[ActionItem]:
904-
items: List[ActionItem] = []
905-
seen: set[str] = set()
906-
for sentence in _split_sentences(body):
907-
low = sentence.lower()
908-
if not any(cue in low for cue in _ACTION_CUES):
909-
continue
910-
normalized = sentence.strip()
911-
key = normalized.lower()
912-
if key in seen:
913-
continue
914-
seen.add(key)
915-
due_match = _DUE_HINT_RE.search(sentence)
916-
due_hint = due_match.group(1) if due_match else None
917-
url_match = _URL_RE.search(sentence)
918-
if url_match:
919-
# Trim trailing sentence punctuation the greedy match grabs
920-
# ("...report." → "...report") so the link is well-formed.
921-
# Strip char-by-char, but keep a ")" that closes a "(" inside
922-
# the URL itself (e.g. .../Python_(programming_language)) so we
923-
# don't silently truncate Wikipedia/Confluence-style links.
924-
url = url_match.group(0)
925-
_trailing = ".,;:!?)]}\"'"
926-
while url and url[-1] in _trailing:
927-
if url[-1] == ")" and "(" in url:
928-
break
929-
url = url[:-1]
930-
items.append(
931-
ActionItem(
932-
description=normalized,
933-
due_hint=due_hint,
934-
type="link",
935-
url=url,
936-
)
937-
)
938-
else:
939-
items.append(ActionItem(description=normalized, due_hint=due_hint))
940-
return items
951+
return extract_action_items_from_body(body)
941952

942953
def _build_draft(
943954
self,

hub/agents/python/email/gaia_agent_email/calendar_backend.py

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@
2424
import httpx
2525

2626
from gaia_agent_email.scopes import AGENT_NAMESPACED_ID, CALENDAR_SCOPES
27+
from gaia_agent_email.google_errors import (
28+
access_not_configured_message,
29+
access_not_configured_url,
30+
)
2731

2832
from gaia.connectors.api import get_access_token_sync
2933
from gaia.connectors.errors import ConnectorsError
@@ -111,6 +115,11 @@ def _raise_http(self, response: httpx.Response, where: str) -> None:
111115
"scopes were revoked. Reconnect Google in Settings → "
112116
f"Connectors. (where: {where})"
113117
)
118+
enable_url = access_not_configured_url(response)
119+
if enable_url:
120+
raise ConnectorsError(
121+
access_not_configured_message("Calendar API", enable_url)
122+
)
114123
raise ConnectorsError(
115124
f"Calendar API {where} returned {response.status_code}: "
116125
f"{response.text[:300]}"

hub/agents/python/email/gaia_agent_email/gmail_backend.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,11 @@
5353
GMAIL_SCOPES,
5454
)
5555

56+
from gaia_agent_email.google_errors import (
57+
access_not_configured_message,
58+
access_not_configured_url,
59+
)
60+
5661
from gaia.connectors.api import get_access_token_sync
5762
from gaia.connectors.errors import ConnectorsError
5863
from gaia.logger import get_logger
@@ -434,6 +439,11 @@ def _raise_http(self, response: httpx.Response, where: str) -> None:
434439
"scopes were revoked. Reconnect Google in Settings → "
435440
"Connectors. (where: " + where + ")"
436441
)
442+
enable_url = access_not_configured_url(response)
443+
if enable_url:
444+
raise ConnectorsError(
445+
access_not_configured_message("Gmail API", enable_url)
446+
)
437447
raise ConnectorsError(
438448
f"Gmail API {where} returned {response.status_code}: "
439449
f"{response.text[:300]}"

0 commit comments

Comments
 (0)