Skip to content

fix(email): don't cancel a retrying agent for a recoverable tool error - #2572

Merged
itomek-amd merged 9 commits into
mainfrom
tmi/issue-2515-m59
Jul 28, 2026
Merged

fix(email): don't cancel a retrying agent for a recoverable tool error#2572
itomek-amd merged 9 commits into
mainfrom
tmi/issue-2515-m59

Conversation

@itomek

@itomek itomek commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

When the model calls a batch tool with a bad argument (e.g. archive_message_batch with a stray mailbox kwarg), the agent loop correctly rejects it and starts retrying — but the SSE layer couldn't tell that per-tool retry apart from a genuinely fatal failure, so it ended the response and cancelled the still-retrying agent, dead-ending the turn with no answer and no stats line. Reproduced 4/4 on-hardware before this fix. Now a recoverable error surfaces as a non-terminal status line (the user still sees the failure) and the retry can reach completion.

Closes #2515

Test plan

  • python -m pytest tests/unit -k "sse or agent_error" -q — 399 passed
  • python -m pytest hub/agents/email/python/tests -k "sse or translation or query_routes" -q — 64 passed
  • New regression test drives the real /v1/email/query route + real SSEOutputHandler + real translation layer with a fake agent that emits a recoverable per-tool error then retries: asserts run.cancel_event and handler.cancelled stay unset mid-run and the stream reaches final with both the failed and retried tool calls streamed (test_recoverable_tool_error_does_not_terminate_stream_or_cancel_run)
  • python util/lint.py --all — all blocking checks pass

itomek added 2 commits July 28, 2026 07:50
The batch-tool retry path (agent.py's STATE_ERROR_RECOVERY) was being killed
by the SSE layer: a per-tool error the loop had already started retrying was
indistinguishable from a fatal top-level failure once it reached the wire,
so the email sidecar's translator mapped it to a terminal error, the query
route ended the stream, and its cleanup then cancelled the still-retrying
agent mid-recovery.

print_error() now takes a recoverable flag so the SSE event carries it
through; the email sidecar's canonical translator folds a recoverable
agent_error to a non-terminal status line (same pattern already used for
tool_confirm_denied) instead of a terminal error, so the run stays alive and
the retry can reach completion. The user still sees the failure — it's
surfaced as a status line, not silently swallowed.
Black wanted print_error's line wrapped in src/gaia/api/sse_handler.py.
@github-actions github-actions Bot added tests Test changes agents agent::email Email agent changes labels Jul 28, 2026
@itomek
itomek marked this pull request as ready for review July 28, 2026 12:26
@itomek
itomek requested a review from kovtcharov-amd as a code owner July 28, 2026 12:26
itomek-amd
itomek-amd previously approved these changes Jul 28, 2026

@itomek-amd itomek-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified. Clean threading of a recoverable flag from print_error through every SSE handler to the translator, where a recoverable agent_error folds to a non-terminal status line instead of a terminal error — so a per-tool retry (STATE_ERROR_RECOVERY) no longer gets the still-retrying agent cancelled out from under it. The flag is only set when True, keeping the wire shape unchanged for every existing fatal caller; recoverable: false and absence both stay terminal. The end-to-end route test proving both tool_calls stream and the run reaches final is the strong evidence here.

itomek-amd
itomek-amd previously approved these changes Jul 28, 2026
itomek-amd
itomek-amd previously approved these changes Jul 28, 2026
itomek-amd
itomek-amd previously approved these changes Jul 28, 2026
itomek-amd
itomek-amd previously approved these changes Jul 28, 2026
itomek-amd
itomek-amd previously approved these changes Jul 28, 2026
itomek-amd
itomek-amd previously approved these changes Jul 28, 2026
@itomek-amd
itomek-amd merged commit d01b9d4 into main Jul 28, 2026
55 checks passed
@itomek-amd
itomek-amd deleted the tmi/issue-2515-m59 branch July 28, 2026 17:52
@itomek itomek mentioned this pull request Aug 10, 2026
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent::email Email agent changes agents tests Test changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(email-agent): batch-tool retry is killed by the SSE layer treating a recoverable tool error as terminal

2 participants