Progress notifications for long calls, report synthesis by default, fix content_length always 0 - #30
Open
MrSampson wants to merge 9 commits into
Open
Conversation
added 3 commits
July 18, 2026 14:56
format_sources_for_response() read source.get("content", ""), but
GPTResearcher.get_research_sources() returns dicts populated from
gpt_researcher's scraper output (gpt_researcher/scraper/scraper.py),
which keys the scraped text as "raw_content". The "content" key never
existed, so content_length was 0 for every source regardless of how
much text was actually scraped -- indistinguishable from a source that
genuinely failed to scrape.
…ault deep_research and quick_search can legitimately run past a calling MCP client's idle timeout (Claude Code's default is 300s) with zero signal that work is ongoing, so a client has no way to distinguish "still working" from "hung" and may abort. GPTResearcher already accepts a log_handler and fires a dozen-plus internal step events via _log_event() throughout conduct_research() (gpt_researcher/agent.py), but nothing here was ever wired to it. Add ProgressLogHandler, constructed with the tool call's Context, and pass it as log_handler so each step forwards to an MCP progress notification -- resetting the client's idle timer for clients that implement the standard MCP behavior of resetting request timeouts on progress notifications tied to the request's progressToken. Separately, deep_research previously returned only raw context/sources -- getting a synthesized report required knowing to call the separate write_report tool with the returned research_id, which is easy to miss and leaves most callers with a pile of source excerpts to read themselves rather than a usable draft. deep_research now also calls write_report() and includes it as "report" by default. Pass synthesize_report=False to skip synthesis and keep the old gather-only/no-extra-LLM-call behavior (write_report can still be called separately with the returned research_id). Tests: tests/test_deep_research_fixes.py drives both tools through an in-memory fastmcp.Client with a patched GPTResearcher, asserting progress events fire, report is present by default and omittable, and content_length reflects actual scraped content.
… all gpt_researcher/actions/query_processing.py has an import-ordering bug (a typing import placed after its first use) that makes the entire package fail with NameError at module load time on 0.16.0, the currently-published release. Confirmed 0.14.8/0.15.0/0.15.1 all import fine. Fix submitted upstream as assafelovic/gpt-researcher#1943 -- once merged and released, this upper bound can be dropped. Without this pin, gptr-mcp's own unbounded `gpt-researcher>=0.14.0` would resolve to the broken release on any fresh install and the container would never start.
Author
|
Update: added one more commit pinning |
gpt-researcher accumulated enough separately-tracked bug fixes (issues assafelovic#17, assafelovic#20, assafelovic#24 at spark-ba88) that maintaining several interacting monkeypatches -- with real ordering dependencies between them -- became more complex than owning the fixes as native code. See MrSampson/gpt-researcher's spark-production branch and services/research/README.md's "Forked gpt-researcher" section (in the spark-ba88 repo) for the full rationale and the individual upstream PRs each fix was also submitted as (#1943, #1944, #1951, #1952). Supersedes the previous commit's <0.16.0 upper bound -- that pin is no longer needed since the fork already carries the NameError fix, but harmless to have been temporarily redundant with it.
Picks up the PDF-retry revalidation and default report-prompt grounding fixes from gpt-researcher's spark-production branch. Also adds missing tests/__init__.py to enable test discovery.
chore: re-pin gpt-researcher to pick up spark-ba88#23 fixes
…ering (#2) Picks up the fork's fix for sub-1KB stub content and CDN error-page titles being ingested as if they were real source content (UI stub pages, Cloudflare error pages, truncated pages all clearing the old 100-char floor). Also drops a stale internal issue-tracker reference from this file's comment while touching it. Co-authored-by: oliver <oliver.sampson@gitterdan.ai>
Picks up the topical-relevance exclusion guard added to the default report prompt.
chore: re-pin gpt-researcher to pick up a report-prompt fix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three related fixes to
deep_research/quick_search, found while running this server against long, real research queries in production:content_lengthalways 0 for every source.format_sources_for_response()readsource.get("content", ""), butGPTResearcher.get_research_sources()returns dicts keyed"raw_content"(seegpt_researcher/scraper/scraper.py). The"content"key never existed, so every source reportedcontent_length: 0regardless of how much text was actually scraped -- indistinguishable from a source that genuinely failed.No progress signal during long calls.
GPTResearcheralready accepts alog_handlerand fires a dozen-plus internal step events via_log_event()throughoutconduct_research(), but neitherdeep_researchnorquick_searchever wired one up. Adeep_researchcall against a broad query can legitimately run past a calling MCP client's idle timeout (Claude Code's default is 300s) with zero signal that work is ongoing -- the client can't distinguish "still working" from "hung," and may abort, permanently stranding the SSE session (a separate issue, tracked upstream in the MCP SDK). AddedProgressLogHandler, constructed with the tool call'sContext, forwarding each step to an MCP progress notification.No synthesized report from
deep_research. Previously, getting an actual written report required knowing to call the separatewrite_reporttool with theresearch_iddeep_researchreturned -- easy to miss, and leaves most callers with a pile of source excerpts to read themselves.deep_researchnow also callswrite_report()and includes it as"report"by default. Passsynthesize_report=Falseto keep the old gather-only behavior (no extra LLM call;write_reportcan still be called separately).Test plan
tests/test_deep_research_fixes.py: drives both tools through an in-memoryfastmcp.Clientwith a patchedGPTResearcher, asserting progress events fire for both tools,reportis present by default and omittable viasynthesize_report=False, andcontent_lengthreflects actual scraped content (including the case whereraw_contentis absent, which must not crash).tests/test_mcp_server.pylocally against a live server -- unaffected by these changes (same tool names/response shape,reportandsynthesize_reportare additive).Happy to split into separate PRs if that's easier to review -- these three were found and fixed together against the same production workload, but they're independent changes.