Skip to content

Commit 3dc75bd

Browse files
regnullclaude
andcommitted
Load MCP tools on demand; cap derived summary trigger at 100k
Every model call re-sent every bound tool definition. All four bots carry Linear's 76 MCP tools (~30k tokens) plus Serena and filesystem (~11k more), on every call, while Linear was called 19 times in ~3,200 tool calls over a week. - MCP tools stay bound but are left out of model requests until the bot loads them with the new load_tools tool (exact names or keyword query, max 15 per call); the system prompt lists them by name. The loaded set lives in agent state for the rest of the run; loaded tools go last in the tool list so a load keeps the cached prefix. Calling an unloaded tool is refused but loads it. - defer_mcp_tools setting (default on) turns it off. - MAX_DERIVED_SUMMARY_TRIGGER 200k -> 100k: on glm-5.3-flash's 1M window nothing was cleared until ~133k of messages and calls passed 170k tokens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 72c79fc commit 3dc75bd

11 files changed

Lines changed: 402 additions & 6 deletions

File tree

‎README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -466,6 +466,12 @@ Tools appear in the registry as `server__tool` (`Linear__create_issue`) with sou
466466
In the bot editor, MCP tools are grouped by server: grant a whole server with one checkbox, or expand
467467
it and pick individual tools. `approval_tools` and `TOOL_OUTPUT_CAP` apply to them like any other tool.
468468

469+
MCP tools load on demand (`defer_mcp_tools`, on by default). A server can bring dozens of tools with
470+
long schemas (Linear's 76 come to ~30k tokens), and every model call re-sends every tool definition it
471+
is offered. So a bot's system prompt lists its MCP tools by name only, and the bot calls `load_tools`
472+
(by exact name, or by keyword `query`) to add the ones a task needs; they stay loaded for the rest of the
473+
run. A call to a tool that was never loaded does not run, but loads it so the retry has its schema.
474+
469475
Whatever you authorize is shared by every bot given that tool: a bot calling Linear acts as you.
470476
Use `approval_tools` for anything that writes.
471477

@@ -634,6 +640,7 @@ name, if set, is the default the page shows and the value a reset returns to.
634640
| Embeddings | `embedding_model`, `embedding_dims` | `provider:model` for semantic memory search (`openrouter:openai/text-embedding-3-small`/1536 using the OpenRouter key, `openai:text-embedding-3-small`/1536, `ollama:nomic-embed-text`/768). Empty turns semantic search off. Applies immediately: the memory store is reopened. |
635641
| Run limits | `max_model_calls_per_run` (60), `max_repeated_tool_failures` (3), `max_bot_hops` (20) | Model turns per run before the agent stops with a notice (a bot can lower it in `model_settings.max_model_calls`; the seeded Chief of Staff uses 6); how many times one tool call may fail with the same arguments and error before the run stops with that error as the reply (0 = off); bot-to-bot mention chain limit per thread. `model_settings` also accepts `reasoning_effort` (set from the Effort select for models that declare levels), `temperature`, `max_tokens`. |
636642
| Context | `tool_output_cap` (8000), `shell_output_cap` (4000) | Longest single tool result the model sees; shorter cap for `run_shell` so dumping files through the shell loses to `read_file` ranges. |
643+
| Context | `defer_mcp_tools` (true) | Leave MCP tool definitions out of model calls until the bot loads them with `load_tools`; the system prompt lists their names. |
637644
| Context | `context_trigger_tokens` (40000), `context_clear_at_least` (10000) | Once a run's messages pass the trigger, tool results from turns before the last two, and their call arguments, become a placeholder, oldest first; each clearing reclaims at least the second value so clearings are rare and the prompt cache stays warm. What the last two model turns fetched is never cleared. |
638645
| Context | `summary_trigger_tokens` (60000), `summary_keep_messages` (24) | Older history folds into one structured summary by the bot's own model once the run's messages pass the trigger. Measured on the messages alone: the system prompt and tool schemas are not counted. |
639646
| Context | `history_token_budget` (24000), `history_max_messages` (80) | Thread history included in a run's prompt. |

‎backend/openbot/config.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@ class Settings(BaseSettings):
7474
prompt_caching: bool = True # add Anthropic cache breakpoints to every model call
7575
include_llm_call_details: bool = Field(default=True, validation_alias="OPENBOT_INCLUDE_LLM_CALL_DETAILS") # expose per-call token details in activity events; launchers opt out explicitly
7676
direct_anthropic: bool = True # send OpenRouter `anthropic/...` models to Anthropic directly when a key exists
77+
defer_mcp_tools: bool = True # send MCP tool definitions only after a bot loads them (runtime/tool_loading.py)
7778
tool_output_cap: int = 8000 # max chars of any single tool result the model sees (head + tail kept)
7879
shell_output_cap: int = 4000 # tighter cap for run_shell, so dumping a file through cat/git show loses to read_file ranges
7980
# Context triggers: None means "derive from the model's context window" (OpenCode's compaction

‎backend/openbot/runtime/app_settings.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,8 @@ class Tunable:
7878
"Longest single tool result the model sees; longer results are shortened and say so.", 500),
7979
"shell_output_cap": Tunable("Context", "Shell output cap (chars)",
8080
"Tighter cap for run_shell, so dumping a file through the shell loses to read_file with a line range.", 500),
81+
"defer_mcp_tools": Tunable("Context", "Load MCP tools on demand",
82+
"Leave MCP tool definitions out of each model call until the bot loads them with load_tools; the system prompt lists their names. Saves the definitions' tokens (tens of thousands for large servers) on every call."),
8183
"context_trigger_tokens": Tunable("Context", "Clear old tool results above (tokens)",
8284
"Once a run's messages pass this, tool results from turns before the last two, and their call arguments, are replaced by a placeholder. The system prompt and tool schemas are not counted. Leave empty to derive it from the model's context window (falls back to 40000 when unknown).", 1000),
8385
"context_clear_at_least": Tunable("Context", "Reclaim at least (tokens)",

‎backend/openbot/runtime/context_limits.py‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,8 +52,11 @@
5252
OUTPUT_TOKEN_MAX = 32_000
5353
# Ceiling on a derived summary trigger. Every call re-sends the whole context, so a 1M window
5454
# summarizing at ~700k made each call slow and costly; past this a summary is cheaper than
55-
# carrying the history. An explicit `summary_trigger_tokens` override may still go higher.
56-
MAX_DERIVED_SUMMARY_TRIGGER = 200_000
55+
# carrying the history. At 200k (2026-10-17) a 1M-window bot cleared nothing until ~133k of
56+
# messages and its calls passed 170k tokens; 100k keeps clearing near 67k, above the ~47k that
57+
# made bots re-read what had just been cleared. An explicit `summary_trigger_tokens` override
58+
# may still go higher.
59+
MAX_DERIVED_SUMMARY_TRIGGER = 100_000
5760

5861
# Built-in fallbacks (the old fixed defaults), used when the window is unknown or a trigger
5962
# setting is None without a window to derive from.

‎backend/openbot/runtime/prompt.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
from langchain_core.messages import AIMessage, BaseMessage, HumanMessage
77

88
from openbot.db.models import Actor, Message
9+
from openbot.runtime.tool_loading import deferred_tools_section
910
from openbot.tools.builtin.scratchpad import scratchpad_section
1011
from openbot.tools.builtin.todo import todo_list_section
1112

@@ -166,6 +167,7 @@ def build_system_prompt(
166167
scoped: bool = False,
167168
todos: list[dict] | None = None,
168169
scratchpad: str | None = None,
170+
deferred_tools: list[str] | None = None,
169171
) -> str:
170172
stable, dynamic = build_system_prompt_parts(
171173
bot=bot,
@@ -179,6 +181,7 @@ def build_system_prompt(
179181
scoped=scoped,
180182
todos=todos,
181183
scratchpad=scratchpad,
184+
deferred_tools=deferred_tools,
182185
)
183186
return stable + dynamic
184187

@@ -196,6 +199,7 @@ def build_system_prompt_parts(
196199
scoped: bool = False,
197200
todos: list[dict] | None = None,
198201
scratchpad: str | None = None,
202+
deferred_tools: list[str] | None = None,
199203
) -> tuple[str, str]:
200204
roster = (
201205
"\n".join(
@@ -248,6 +252,8 @@ def build_system_prompt_parts(
248252
}
249253
stable = _STABLE_SYSTEM_PROMPT_TEMPLATE.substitute(**values)
250254
dynamic = _DYNAMIC_SYSTEM_PROMPT_TEMPLATE.substitute(**values)
255+
if deferred_tools:
256+
dynamic += f"\n\n{deferred_tools_section(deferred_tools)}"
251257
if scratchpad is not None:
252258
dynamic += f"\n\n{scratchpad_section(scratchpad)}"
253259
if todos:

‎backend/openbot/runtime/runner.py‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@
5555
from openbot.runtime.providers import builtin_tools, effective_bot_profile
5656
from openbot.runtime.retry import ModelRetryMiddleware
5757
from openbot.runtime.tool_failures import ToolFailureMiddleware
58+
from openbot.runtime.tool_loading import LOAD_TOOLS, DeferredToolsMiddleware
5859
from openbot.tools.builtin.core import CORE_TOOLS
5960
from openbot.tools.builtin.scheduling import SCHEDULING_TOOLS
6061
from openbot.tools.builtin.scratchpad import load_scratchpad
@@ -674,6 +675,7 @@ async def _prepare(self, bot: Actor, thread: Thread, run: Run) -> tuple[str, dic
674675
else []
675676
)
676677
workspace_root = thread_workspace_root(st.workspace_root, thread.working_directory)
678+
deferred = set(self.deferred_tool_names(bot))
677679
stable_prompt, dynamic_prompt = build_system_prompt_parts(
678680
bot=bot,
679681
all_bots=list(all_actors),
@@ -683,11 +685,16 @@ async def _prepare(self, bot: Actor, thread: Thread, run: Run) -> tuple[str, dic
683685
older_count=older,
684686
# Only tools that will actually be bound: an MCP server that is down
685687
# must not be advertised, or the model calls a tool it does not have.
686-
tool_names=[n for n in bot.bot.tool_names if self.s.registry.has(n)],
688+
# Tools loaded on demand are listed in their own section instead (runtime/tool_loading.py).
689+
tool_names=[
690+
*(n for n in bot.bot.tool_names if self.s.registry.has(n) and n not in deferred),
691+
*([LOAD_TOOLS] if deferred else []),
692+
],
687693
default_bot_handle=default_bot_handle,
688694
scoped=scoped,
689695
todos=todos,
690696
scratchpad=scratchpad,
697+
deferred_tools=[n for n in bot.bot.tool_names if n in deferred],
691698
)
692699
prompt = SystemMessage(
693700
content=[
@@ -783,8 +790,23 @@ def build_middleware(self, bot: Actor, model, triggers: tuple[int, int] | None =
783790
description_prefix="Tool execution requires approval",
784791
)
785792
)
793+
deferred_names = self.deferred_tool_names(bot)
794+
if deferred_names:
795+
middleware.append(DeferredToolsMiddleware(self.s.registry.resolve(deferred_names)))
786796
return middleware
787797

798+
def deferred_tool_names(self, bot: Actor) -> list[str]:
799+
"""The bot's MCP tools, which are bound but sent to the model only once it loads them
800+
(runtime/tool_loading.py). None when deferral is off, or for a CLI-agent bot, which never
801+
goes through create_agent."""
802+
if (
803+
self.s.registry is None
804+
or not getattr(self.s.settings, "defer_mcp_tools", True)
805+
or adapter_for(bot.bot.provider) is not None
806+
):
807+
return []
808+
return [n for n in bot.bot.tool_names if (self.s.registry.source(n) or "").startswith("mcp:")]
809+
788810
def _build_agent(self, bot: Actor, system_prompt: str, triggers: tuple[int, int] | None = None):
789811
p = bot.bot
790812
tools = [

0 commit comments

Comments
 (0)