Skip to content

Commit caeab94

Browse files
SonAIengineclaude
andcommitted
feat(agent_loop): add system-prompt guidance for relative time + multi-source asks
v0.18-α1-2 KRRA Conv diagnostic identified two prompt-shaped Qwen3.5-27B failure patterns on conversational Korean queries: 1. "올해 예산은?" / "내년도 일정은?" — agent injected literal year ("2024"/"2025") into deep_search, throwing away corpus docs from other years. Now told to search the topic first and only narrow by year if the topic search returns too many candidates AND there's evidence the user wants a specific year. 2. "X 관련 자료 있어?" / "내용 보여줘" — agent terminated after 1 deep_search returned 1-2 docs and answered "yes, here is one source". Now told that "자료/내용/정보" asks for multiple sources, do at least one paraphrase pass before concluding. Both lines are general-purpose, not KRRA-specific, so they apply to any Korean / English corpus. The KRRA Conv -23pp regression vs v0.13 GPT-4o-mini (47% vs 70%) is documented as a known issue: it stems from a recall ceiling on broad topical queries (예산 / 인권 / 자료) where 5 ground-truth docs share a vague topic word and deep_search top-K is dominated by phrase nodes. Verified by direct corpus probe — 0/150 GT IDs reachable as direct node ids; matching works only via properties.doc_id and depends on the right docs landing in deep_search top-K. Real fix requires higher recall or reranker-on-by-default; deferred to v0.19+ track. 45/45 agent + calibration tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent fa82a66 commit caeab94

2 files changed

Lines changed: 31 additions & 1 deletion

File tree

CHANGELOG.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,26 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
66

77
## [Unreleased]
88

9+
### Changed — `agent_loop` system prompt: relative-time + multi-source guidance
10+
11+
Two new tip lines in the agent prompt, learned from the v0.18-α1-2 KRRA
12+
Conv diagnostic:
13+
14+
- **Relative time references** ("올해" / "내년도" / "this year"): the agent
15+
should NOT inject a literal year number. The corpus may span multiple
16+
years and a hard `2024` filter throws away matches. Search the topic
17+
first, narrow by year only after evidence the user wants one.
18+
- **"X 관련 자료/내용/정보" type questions** ask for *multiple* sources.
19+
The agent should not stop after the first ``deep_search`` returns 1-2
20+
docs — at least one paraphrase pass before concluding.
21+
22+
These guidance lines are general-purpose and apply to any corpus, not
23+
just KRRA Conv. The KRRA Conv −23pp regression itself is documented as
24+
known issue: it stems from a recall ceiling on broad topical queries
25+
where 5 GT docs share a vague topic word (예산 / 인권), not from agent
26+
reasoning. Real fix would require either a higher deep_search top-K cap
27+
or reranker-on-by-default for broad queries — both v0.19+ items.
28+
929
## [0.17.2] - 2026-04-19
1030

1131
Patch release bundling the license switch and the

src/synaptic/agent_loop.py

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,17 @@
101101
often mix scripts (e.g. "치즈" → 0, "cheese" → many).
102102
- For "top N" / "most" / "least" questions, prefer ``aggregate_nodes``
103103
with ``metric=count`` and ``order=desc``.
104-
- Prefer concrete IDs over titles when chaining tools."""
104+
- Prefer concrete IDs over titles when chaining tools.
105+
- **Relative time references** ("올해" / "내년도" / "this year" / "next year"):
106+
do NOT inject a specific year number into the search query — the corpus
107+
may span multiple years, and a literal "2024" filter throws away all
108+
other matches. Search the topic without the year first; only narrow by
109+
year if the topic search returns too many candidates AND you have evidence
110+
the user wants a specific year.
111+
- **"X 관련 자료 / 내용 / 정보" type questions** ask for *multiple* sources.
112+
After the first ``deep_search`` returns a few hits, do at least one more
113+
``search`` with paraphrased keywords before concluding. A single document
114+
is rarely the complete answer to such a request."""
105115

106116

107117
AGENT_TOOLS = [

0 commit comments

Comments
 (0)