feat: Stricter DevRel classification & GitHubToolkit fixes #128
Conversation
WalkthroughAdds bot name configuration and uses it to detect explicit bot mentions in classification; tightens fallback triage to require mentions; adjusts default needs_devrel to False; enriches classification payload with bot_mentioned; updates GitHub toolkit to return structured dicts for unimplemented paths. Changes
Sequence Diagram(s)sequenceDiagram
participant U as User
participant CR as ClassificationRouter
participant LLM as LLM
U->>CR: Message
CR->>CR: Detect @bot mention (from Settings.bot_name)
alt Can call LLM triage
CR->>LLM: DEVREL_TRIAGE_PROMPT (+mention note if mentioned)
LLM-->>CR: JSON (needs_devrel?, priority, reasoning)
CR->>CR: Default needs_devrel=False if missing
else Fallback
CR->>CR: _fallback_triage(mention_flag)
end
CR-->>U: {needs_devrel, priority, reasoning, bot_mentioned}
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~15–20 minutes Assessment against linked issues
Assessment against linked issues: Out-of-scope changes
Poem
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
backend/app/classification/classification_router.py (1)
59-59: Missing mention_flag parameter in error fallbackThe error handler at line 59 calls
_fallback_triage(message)without themention_flagparameter. While Python will use the default value ofFalse, it would be cleaner to explicitly handle the mention detection even in error cases.Consider preserving the mention flag even in error cases:
except Exception as e: logger.error(f"Triage error: {str(e)}") - return self._fallback_triage(message) + # Try to detect mention even in error case + mention_flag = self.bot_name in message.lower() if message else False + return self._fallback_triage(message, mention_flag)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
backend/.env.example(1 hunks)backend/app/agents/devrel/github/github_toolkit.py(1 hunks)backend/app/classification/classification_router.py(2 hunks)backend/app/core/config/settings.py(1 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
backend/app/classification/classification_router.py (1)
backend/app/services/embedding_service/service.py (2)
llm(55-68)model(41-52)
🪛 dotenv-linter (3.3.0)
backend/.env.example
[warning] 26-26: [UnorderedKey] The LANGSMITH_PROJECT key should go before the LANGSMITH_TRACING key
[warning] 29-29: [EndingBlankLine] No blank line at the end of the file
[warning] 29-29: [SpaceCharacter] The line has spaces around equal sign
[warning] 29-29: [ValueWithoutQuotes] This value needs to be surrounded in quotes
🔇 Additional comments (8)
backend/app/core/config/settings.py (1)
18-18: No action needed—bot_nameis already configurable via.env
Pydantic’sBaseSettings(frompydantic_settings) combined with the top-of-fileload_dotenv()call ensures that an environment variable namedBOT_NAMEwill override the default"DevRelBot". The existingConfig.env_file = ".env"(plusextra = "ignore") covers loading and ignores unrelated variables, so you can set in your.env:BOT_NAME=MyCustomBotand the
bot_namefield will pick it up.backend/app/agents/devrel/github/github_toolkit.py (2)
107-107: Good fix for dictionary consistencyThe change from returning a plain string to a dictionary ensures consistent return types across all branches, preventing the "'str' object does not support item assignment" error that would occur at lines 120-121.
113-113: Good fix for dictionary consistencyThe change ensures a consistent dictionary return type, preventing runtime errors during result augmentation.
backend/app/classification/classification_router.py (5)
11-11: Documentation accurately reflects the stricter classification approachThe updated docstring clearly indicates the shift from simple to strict DevRel triage, aligning with the PR objectives.
14-14: Dynamic bot name support implemented correctlyGood implementation of reading the bot name from settings, enabling flexibility across different deployments without code changes.
24-30: Effective bot mention detection with proper context enrichmentThe implementation correctly detects bot mentions and enriches the context for the LLM, which aligns with the PR objective of requiring explicit mentions for DevRel activation.
48-48: Stricter default reduces false positivesChanging the default
needs_devrelfromTruetoFalseis a crucial improvement that aligns with the PR objective of reducing unnecessary DevRel agent invocations.
61-69: Well-designed fallback logic aligns with PR objectivesThe fallback triage effectively implements the stricter classification by only triggering DevRel when the bot is mentioned. The priority and reasoning adjustments based on the mention flag provide clear context for downstream processing.
|
|
||
| #Discord Bot Name Configuration | ||
| BOT_NAME = "DevRelBot" #Add your own Discord Bot name |
There was a problem hiding this comment.
Fix environment variable syntax and formatting issues
The new BOT_NAME environment variable has incorrect syntax with spaces around the equal sign and missing quotes around the value. Additionally, there are formatting issues that should be addressed.
Apply this diff to fix the syntax and formatting:
LANGSMITH_PROJECT=
-#Discord Bot Name Configuration
-BOT_NAME = "DevRelBot" #Add your own Discord Bot name
+# Discord Bot Name Configuration
+BOT_NAME="DevRelBot" # Add your own Discord Bot name
+This fixes:
- Removes spaces around the equal sign (environment variable syntax)
- Adds a space after
#in the comment header for consistency - Adds a space after
#in the inline comment - Adds a blank line at the end of the file
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| LANGSMITH_PROJECT= | |
| #Discord Bot Name Configuration | |
| BOT_NAME = "DevRelBot" #Add your own Discord Bot name | |
| LANGSMITH_PROJECT= | |
| # Discord Bot Name Configuration | |
| BOT_NAME="DevRelBot" # Add your own Discord Bot name | |
🧰 Tools
🪛 dotenv-linter (3.3.0)
[warning] 26-26: [UnorderedKey] The LANGSMITH_PROJECT key should go before the LANGSMITH_TRACING key
[warning] 29-29: [EndingBlankLine] No blank line at the end of the file
[warning] 29-29: [SpaceCharacter] The line has spaces around equal sign
[warning] 29-29: [ValueWithoutQuotes] This value needs to be surrounded in quotes
🤖 Prompt for AI Agents
In backend/.env.example around lines 26 to 29, the BOT_NAME entry and nearby
comments have incorrect environment variable syntax and formatting; remove
spaces around the equal sign, set the value as BOT_NAME="DevRelBot" (no
surrounding spaces), add a space after the '#' in the comment header and the
inline comment for consistency, and ensure the file ends with a single blank
line.
| result = "Not implemented" | ||
| # result = await handle_issue_creation(query) |
There was a problem hiding this comment.
Inconsistent return type for issue_creation branch
While you fixed the repo_support and documentation_generation branches to return dictionaries, the issue_creation branch still returns a plain string "Not implemented", which will cause the same error when trying to augment it with intent_analysis and type at lines 120-121.
Apply this diff to maintain consistency:
elif classification == "issue_creation":
- result = "Not implemented"
+ result = {"status": "not_implemented", "message": "Issue creation not implemented yet"}
# result = await handle_issue_creation(query)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| result = "Not implemented" | |
| # result = await handle_issue_creation(query) | |
| elif classification == "issue_creation": | |
| - result = "Not implemented" | |
| + result = {"status": "not_implemented", "message": "Issue creation not implemented yet"} | |
| # result = await handle_issue_creation(query) |
🤖 Prompt for AI Agents
In backend/app/agents/devrel/github/github_toolkit.py around lines 110-111, the
issue_creation branch returns a plain string "Not implemented" while other
branches return dictionaries; change that branch to return a dictionary (e.g.,
{'result': 'Not implemented'}) so the code downstream can augment it with
intent_analysis and type without errors, and ensure the dict shape matches the
repo_support/documentation_generation branches.
Closes #93
This PR improves the DevRel agent’s classification logic and fixes issues in the GitHubToolkit, ensuring only relevant messages trigger the agent and reducing false positives.
Key Changes:
ClassificationRouter Enhancements:
GitHubToolkit Fixes:
Effect:
https://drive.google.com/file/d/1s9lX5iPkRdKDFV8wWVVTX3pxGeBct1DC/view?usp=drive_link
(check this drive to see the output)
Summary by CodeRabbit
New Features
Documentation