fix(email-agent): search_messages defaults to metadata-only, fixing context overflow on counting questions

[itomek]

Asking the email agent a counting question about a long-bodied sender ("how many emails from Every in the last two weeks?") returned no answer at all — the search succeeded, then the model blew its context window re-reading the full 4000-char-per-message result and gave up with a canned "I had to trim the conversation..." apology, 8 times out of 8 across two machines. search_messages now defaults to metadata-only (subject/from/date/snippet, no body) — a counting or listing question never needs message content, and metadata cuts the result envelope by roughly an order of magnitude. When a turn still can't fit, the fallback message now names the actual constraint and a next step instead of a generic apology that looked identical regardless of cause.

Closes #2763.

A docstring instruction was tried first and measured to fail live

The first design kept include_bodies=True as the default and told the model, via the tool's docstring, to pass include_bodies=False for a counting question. On the very probe this issue is about, the model did not do it — the request reproduced the original overflow almost exactly (n_prompt_tokens: 72,783 vs. 72,438 pre-fix, real Lemonade-reported numbers, same live GPU/Gmail run). Flipping the default instead — metadata-only unless the model explicitly opts into include_bodies=True — makes the fix correct regardless of whether the model reads or follows the instruction, and went 3-for-3 on live re-runs. This is a direct, measured demonstration that a tool-docstring instruction alone isn't reliable for changing model behavior on the failure path that matters — the tool should compute the safe default, not ask the model to.

A small, concrete illustration of the same point turned up in this PR's own tooling: the review bot caught that scripts/email_context_budget_measurement_2763.py crashed on its own default flip — it called search_messages_impl without include_bodies, silently started getting metadata-only results, and then read a full-body-only field. Fixed by passing include_bodies=True explicitly at both call sites. That's the argument for the flip, not against it: a default that changes behavior silently breaks every caller that assumed the old one — including code shipped in the same diff — so the fix has to be a real default change, not a hope that every caller (model or script) notices and adapts.

Root cause — measured, not assumed, before the fix (per the issue's own requirement)

Two comments on #2763 walk through this in full, including a public correction where an initial estimate (using this repo's own char-based token estimator) overstated the effect by ~3x and was retracted in favor of real-tokenizer and real-Lemonade-error numbers. Short version: the confirmed primary cause is prior conversation-turn history — a live 2-turn session showed turn 2's own search_messages call cost ~13.8K tokens, yet the actual request was 72,438 tokens against a 65,536 window. This fix leaves more headroom by making a new turn's own tool result cheap; it does not stop cross-turn accumulation, which is filed separately as #2781 (with a related shrink-and-retry gap as #2780 — the one-shot overflow recovery recovered 81 of the 6,902 tokens needed). Do not read this PR as closing the root cause.

Test plan

• python -m pytest hub/agents/email/python/tests/ tests/unit/agents/ -q — 1697 + 1372 passed (run locally, output below)
• python util/lint.py --all — clean (run locally)
• 3 consecutive live runs on GPU + Gmail (Gemma-4-E4B-it-GGUF, device gpu, ctx 65536 — verified before every capture), 2-turn sessions (a neutral first turn, then this issue's exact probe as the second — a single isolated turn is now known not to reproduce the failure): all 3 state 15 (ground truth confirmed via the same backend primitive /v1/email/search uses), zero occurrences of "I had to trim the conversation"
• Before capture on main, same session structure: fails 100% of the time with the canned apology (n_prompt_tokens: 72438 / 72357 after the shrink-and-retry)

CI note: this PR is a draft, and this repo's test workflows gate on draft == false (see #2755/#2767) — close/reopen alone did not bypass that (draft status is unaffected by reopening; confirmed via gh pr checks showing every gated job skipping). Adding the ready_for_ci label and then closing/reopening did work — the gated suites (Email Agent Unit Tests, Test Email Agent, Code Quality, Security Tests, etc.) are running as CI, not inferred from a green page. The checkboxes above were also independently verified locally first, with real output, before CI ran at all.

Merged origin/main after #2779 landed (touches the same file, _build_needs_you_view/is_meeting_request routing inside pre_scan_inbox). Auto-merged with no conflicts — disjoint regions from this PR's _format_messages_within_budget/search_messages_impl/the search_messages wrapper. Re-ran the full suite post-merge (hub/agents/email/python/tests/ 1697 passed, tests/unit/agents/ 1376 passed/25 skipped). Did not re-run the live 3-run evidence against the merged commit: #2779 changes what pre_scan_inbox classifies as needs_you, not search_messages's formatting, budget, or include_bodies default — the mechanism this fix relies on is untouched by the merge, and the box used for the original evidence run is now held by a sibling worker (#2765) for their own evidence capture.

Evidence

Before (main) — 2-turn session, fails

Turn 1 ("What's new in my inbox today?") answers correctly — 44.6s · 2 steps · 1 tools.

Turn 2 (the probe) fails:

[!] I had to trim the conversation to fit my memory but I'm still not making progress.
Could you re-ask in a fresh chat with just the essentials?

Sidecar log, Lemonade's own error:

request (72438 tokens) exceeds the available context size (65536 tokens)
n_ctx: 65536, n_prompt_tokens: 72438

After the one-shot shrink-and-retry: n_prompt_tokens: 72357 — recovered 81 of the 6,902 tokens needed (see #2780).

After (this branch) — 3 consecutive runs, same 2-turn structure

Model/device/ctx verified via Lemonade's /api/v1/health before every capture: Gemma-4-E4B-it-GGUF, gpu, ctx_size: 65536.

Run 1: "There are 15 emails from Every that were found in your inbox within the last two weeks." — 17.6s · ttft 0.0s · ~22 tokens · 1.3 tok/s · 2 steps · 1 tools

Run 2: "There are 15 emails from Every in your inbox over the last two weeks. They all appear to be informational updates or articles about AI development and tools." — 16.5s · ttft 0.0s · ~40 tokens · 2.4 tok/s · 2 steps · 1 tools

Run 3: "There are 15 emails from Every in your inbox over the last two weeks. They appear to be mostly informational updates regarding AI tools and industry trends." — 19.5s · ttft 0.0s · ~40 tokens · 2.1 tok/s · 2 steps · 1 tools

search_messages latency dropped from 2,719ms (full-body, pre-fix) to 273-313ms (metadata-only) — consistent with the size reduction; no context_length_exceeded errors in any of the 3 runs.

Ground truth, direct call to the same backend primitive POST /v1/email/search uses: 15, nextPageToken: None.

Limitation: the exact tool_args (max_results) the model chose per run could not be extracted — the sidecar's plain-text log formatter does not render log_tool_call's structured extra dict (which carries tool_args/result_summary). Noted as an observability gap in #2781 rather than a fabricated number here.

Timing not meaningful: this Mac's Lemonade is a persistent system service with other models resident (co-tenant by design) — the latency figures above are directional (order-of-magnitude drop, consistent across 3 runs) but not a clean benchmark.

Unit-test envelope-size assertions (registered-tool layer, not just "call returned")

hub/agents/email/python/tests/test_search_messages_metadata_only_2763.py — 15 long-body messages, same shape as the failing probe:

• Metadata-only envelope carries no body/body_truncated/body_chars_dropped/attachments field on any message
• Envelope size drops by ≥10x vs. the same query with include_bodies=True (measured ~13x)
• Envelope size asserted against the actual computed envelope_budget_tokens() for both GPU and NPU profiles, not a hardcoded literal

Two pre-existing #2514 budget tests (test_read_tools_list_inbox_budget_2514.py) that implicitly relied on the old include_bodies=True default now pass it explicitly — they specifically exercise the full-body shrink contract, so this keeps that coverage intact rather than loosening it.

[github-actions]

🟡 The diagnostic script added in this PR crashes immediately when run — and its own header says to re-run it after future changes to the tool registry or context_budget.py.

main() calls search_messages_impl without include_bodies, so it gets metadata-only results (the new default). It then tries to read m["body_chars_dropped"] from those results — a field that only exists on full-body messages. That's a KeyError on the very first message, before any output is printed.

Fix: pass include_bodies=True to the three search_messages_impl calls in main() (and the --dump-dir block below) so the shrink-accounting measurements exercise the full-body path they were written for.

🔍 Technical details

scripts/email_context_budget_measurement_2763.py line 922–933:

result = search_messages_impl(
    gmail,
    query="from:every",
    max_results=max_results,
    debug=False,
    operator_retry=False,
    budget_tokens=None,
)
messages = result["messages"]
...
dropped = sorted({m["body_chars_dropped"] for m in messages})  # KeyError — metadata rows have no such field

The include_bodies parameter added in this same diff defaults to False; without it, search_messages_impl returns _format_message_metadata_for_llm rows which carry id/subject/from/to/date/label_ids/snippet only — no body_chars_dropped. The fix is straightforward: add include_bodies=True to each search_messages_impl call inside main(). Same applies to the --dump-dir invocation at line 944, which also omits the flag (it won't KeyError there, but it will silently dump metadata rather than the full-body shrink payload the script header describes).

[itomek]

Fixed — passed include_bodies=True explicitly at both search_messages_impl call sites in main() (the max_results loop and the --dump-dir block), matching the same explicit-argument fix already applied to the two pre-existing #2514 tests. Ran the script for real after the fix (both plain and --dump-dir paths); output is in the latest commit message and the PR description.

[github-actions]

Verdict: Approve with suggestions

This fixes the #2763 P0 where a counting/listing question against a long-bodied sender ("how many emails from X in the last two weeks") blew the context window and returned only the canned "I had to trim the conversation" apology. search_messages now defaults to metadata-only (no body decode), so a counting question costs a fraction of the context; full bodies are still available with include_bodies=True. The base-agent overflow message is also rewritten to name the constraint and a next step. The core fix is well-reasoned, well-tested, and backed by real evidence.

One thing to fix before merge: the diagnostic script this PR adds (scripts/email_context_budget_measurement_2763.py) crashes when run. It calls search_messages_impl without opting into bodies, then reads a body-only field that no longer exists on metadata rows — so the script dies with a KeyError before printing the very measurement it exists to produce. The script is documented as "re-run after any change," so it should work on its own defaults. One-line fix in two spots (pass include_bodies=True where it's measuring the full-body envelope). This is a dev-only script, not a user path, so it's non-blocking for the fix itself — but worth correcting since the PR ships it for reuse.

Real-world evidence

evidence-bundle.md is present and substantive (non-inference runner). It relays, verbatim:

• The PR's new test file — 10 passed (metadata carries no body content, ≥10× envelope reduction measured at the registered-@tool layer, fits GPU/NPU budgets, default-is-False).
• The sibling budget suite test_read_tools_list_inbox_budget_2514.py — 9 passed (full-body shrink/fail-loud path unaffected).
• test_parse_error_recovery.py — 27 passed, including the new streaming overflow case; plus the live constant printed from a real import.
• Spot regression: gaia email --spec renders (93KB HTML), UI server boots and answers /api/health 200.
• The script crash above was observed, not fabricated — the traceback is in the bundle. My verdict accounts for it.

Screenshot deferred "pending strix-halo lane" — appropriate: neither changed surface (a tool, a base-agent string) has a dedicated UI screen, and reaching either needs a live LLM turn this runner can't do. Adequate for merge-time CI per the rubric.

🔍 Technical details

Issues

🟡 Diagnostic script crashes on its own default (scripts/email_context_budget_measurement_2763.py:220, and again at :243)

The "search_messages_impl on 15 long-body messages" section measures the full-body envelope (it checks shrink_fired and body_chars_dropped), but no longer passes include_bodies=True. With the new default the returned rows are metadata-only and carry no body_chars_dropped, so line 231 raises KeyError: 'body_chars_dropped' and the loop never prints. The --dump-dir branch (:243) has the same omission — it dumps tool_result.json for a real-tokenizer cross-check of the full payload, so it also needs bodies.

        result = search_messages_impl(
            gmail,
            query="from:every",
            max_results=max_results,
            debug=False,
            operator_retry=False,
            budget_tokens=None,  # production default: active_profile_ctx_size()
            include_bodies=True,  # this section measures the FULL-body envelope
        )

And for the --dump-dir branch at :243:

        result = search_messages_impl(
            gmail,
            query="from:every",
            max_results=100,
            debug=False,
            operator_retry=False,
            budget_tokens=None,
            include_bodies=True,  # dump the full-body payload for cross-check
        )

Strengths

• Default-safe by design, with the reasoning captured. Making metadata-only the default (rather than a docstring-only opt-in) removes the dependency on a 4B-class model reliably choosing a new parameter on the exact failure path — the asymmetry (a content question that forgets to opt in gets a recoverable "no body" instead of a context-ending overflow) is the right call and is documented at read_tools.py:882.
• Tests assert the contract, not just invocation. Envelope reduction is measured at the registered @tool layer (catches a wrapper re-adding bulk), asserted against the actual computed envelope_budget_tokens rather than a hardcoded literal, and the impl-level test proves stub ordering survives the id-keyed _fetch_messages dict lookup.
• Shared-constant dedup with new coverage. Folding the two copies of the overflow fallback into _CONTEXT_STILL_OVERFLOWING_MESSAGE and adding the previously-missing streaming exhausted-retry test (test_context_overflow_streaming_after_retry_gives_actionable_fallback) closes a real coverage gap.
• Metadata branch reuses the existing _fetch_messages(format="metadata") primitive and re-walks stubs to preserve backend ordering — mirrors triage_inbox_impl's phase-1 pattern rather than reinventing it.

[itomek]

This finding is stale — the crash was already fixed in 7322a47 (include_bodies=True at both search_messages_impl call sites, script run to confirm real output), pushed before this review was generated. See the earlier reply on the first bot review thread for details.

[itomek]

Re-verified on a build combining all five PRs from this run (this one plus #2774, #2783, #2784, #2788) — the fix eliminates the old silent-truncation bug, but under the extra context load from #2784's now-merged on-open attention view, it isn't always enough on its own.

How it was tested — three-condition comparison

Live GAIA TUI against a real Gmail mailbox (account redacted). Ground truth confirmed independently (direct Gmail API query, bypassing the agent entirely): 15 messages from the test newsletter sender in the last 14 days. Same multi-turn structure throughout — one warm-up turn, then "how many emails from <sender> in the last two weeks?" as a follow-up — 3 repeats each:

Condition	Result
origin/main (already includes #2784's attention view)	0/3 — every run hit the exact old bug: "I had to trim the conversation to fit my memory but I'm still not making progress..."
This branch alone (original PR evidence)	3/3 — captured before #2784 landed, a lighter conversation
All five merged together	2/3 — 2 correct (exact match to ground truth); 1 different failure: a genuine context overflow (73002 vs 65536 tokens) where the shrink-and-retry fired but recovered zero tokens, surfaced through this PR's own fail-loud message rather than the old bug

Reads as: the fix is real and large — it removes the silent-truncation failure entirely — but the on-open attention view now consumes enough context headroom that the one-shot shrink isn't always sufficient. Worth a follow-up on the shrink-and-retry path recovering nothing on the failing run (separate finding, not blocking this PR).

Combined-tree suite: 10636 passed, 8 pre-existing failures (parity-confirmed against clean origin/main), 146 skipped. Lint clean.

Integrated tree: 7804d3ae · this branch's merged commit: 7322a47c