[TODO] Split large responses into subtasks: Limit Cypher Queries, some questuions lead to too large context that LLM can't handle

[qdrddr]

📋 Split large responses into

When we perform Cypher querying against a DB, we should always apply a LIMIT X clause to avoid exceeding the LLM context limit. In cases when too many results are returned by the GraphDB, we could either limit the total amount of barches OR split the queries into multiple batches and process each of them with LLM separately (using the original user request), and then merge them together when returning a reply.

🎯 Acceptance Criteria

• Implement result truncation at the query tool level (e.g., limit to first 100 results)
• Add warning messages when results are truncated
• Update Cypher prompts to automatically add LIMIT clauses for broad queries
• ADD split logic with reduced result sets when hitting limits

🔗 Related Files/Components

• File: codebase_rag/services/llm.py
• File: codebase_rag/prompts.py
• File: codebase_rag/config.py

📝 Additional Context

Any additional information, screenshots, or context

🚦 Priority

• High
• [ X ] Medium
• Low

📊 Estimated Effort

• Small (< 2 hours)
• [ X ] Medium (2-8 hours)
• Large (> 8 hours)

[vitali87]

One thought on the limiting approach.
Row-based limits (like LIMIT 100) dont map well to the actual problem.

Consider:

• 100 file paths = ~1000 tokens (fine)
• 100 function bodies = ~50,000 tokens (context blown)

Same row count, completely different context impact. The LLM context window is token-based, so the constraint should be too.

Suggested approach:

1. Execute query with a high row safety cap (e.g. 500)
2. Serialize results and count/estimate tokens
3. Truncate at a token threshold (e.g. 8000 tokens)
4. Warn: "Results truncated at ~8000 tokens (showing 47 of 312 matches)"

This way lightweight results (file lists) come through fully, while heavy results (source code) get truncated sensibly. The LLM always gets usable context regardless of what the query returns.

Thoughts?