chore: add bug-triage skill for agent-driven issue triage#1725
chore: add bug-triage skill for agent-driven issue triage#1725i-trytoohard wants to merge 7 commits into
Conversation
Add .skills/bug-triage/ with SKILL.md and push_fix_to_github.py script. The skill provides a complete triage workflow: - Gather bug context from chat/issues/live observation - Search for duplicate GitHub issues - File well-structured issues with root cause analysis - Push fix PRs via GitHub API (no local checkout needed) - Git archaeology (git log -S) for regression tracking - NPM package regression diffing - Remote code inspection without local clone Reference the skill in AGENTS.md so any agent working on this repo can discover and follow the triage workflow automatically. Tested across 100+ real bug triages on ComposioHQ/agent-orchestrator.
Test Coverage ReportNo TypeScript source files changed in this PR. |
Greptile SummaryThis PR checks in a
Confidence Score: 5/5Safe to merge — all three files are documentation and a standalone helper script with no runtime integration into the main codebase. The changes are additive only: a new skill directory, a self-contained Python helper, and a one-section addition to AGENTS.md. Every defect flagged in the previous review round has been addressed — the SHA race in the push script, the macOS No files require special attention.
|
| Filename | Overview |
|---|---|
| AGENTS.md | Adds a "Skills" section pointing agents to the bug-triage skill at the correct path skills/bug-triage/SKILL.md (no leading dot). |
| skills/bug-triage/SKILL.md | Comprehensive triage playbook; screenshot upload now uses portable `base64 < file |
| skills/bug-triage/scripts/push_fix_to_github.py | Script creates the branch before fetching the file (eliminating the SHA race), checks NEW_STRING is None rather than falsy (allowing empty-string deletions), supports BASE_BRANCH env var, and prints a warning when OLD_STRING matches multiple times — all previously reported defects addressed. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Bug reported\nchat / issue / observation] --> B[Step 0: Gather context\nreporter, screenshots, logs]
B --> C[Step 1: Understand issue\ntrace code path, git archaeology\nupstream dep research]
C --> D{Root cause\nclear?}
D -- No --> E[Stop-and-ask triggers\nask reporter for OS/shell/version]
E --> C
D -- Yes --> F[Step 2: Search duplicates\ngh issue list --state all\nby symptom + component + error]
F --> G{Duplicate\nfound?}
G -- Yes --> H[Step 3: Comment\non existing issue]
G -- No --> I[Step 4.1b: Pre-submission checklist\nreporter, hash, version, repro steps]
I --> J[Step 4.1c: Upload screenshots\nbranch issue-assets-SLUG\nbase64 portable upload]
J --> K[Step 4.2: Create issue\ngh issue create]
K --> L[Step 4.3-4.4: Labels + confidence\npriority label, High/Medium/Low]
L --> M[Step 4.5: Cross-link related issues]
M --> N{Fix\ntrivial?}
N -- Yes --> O[Step 4.6: push_fix_to_github.py\ncreate branch → fetch file → apply replace → push → open PR]
N -- Complex --> P[Note fix in issue\nsuggest spawning agent]
N -- Unclear --> Q[Document findings\nflag for investigation]
O --> R[Step 4.7: Post confirmation\nissue URL, PR URL, labels, root cause]
P --> R
Q --> R
Reviews (7): Last reviewed commit: "feat: add 6 triage improvements from rea..." | Re-trigger Greptile
Any agent following this skill must include clickable URLs when mentioning issues or PRs. Bare '#123' without links is not allowed.
- Move .skills/ → skills/ (repo already has a skills/ directory) - Fix label name: 'priority:medium' → 'priority: medium' (with space) - Fix issue-assets branch naming: use slug instead of issue number (issue doesn't exist yet at upload time) - Fix base64 command: portable across Linux and macOS (tr -d '\n') - push_fix_to_github.py: allow empty NEW_STRING for deletion edits - push_fix_to_github.py: warn on multiple OLD_STRING matches - push_fix_to_github.py: create branch before fetching file (SHA race) - push_fix_to_github.py: configurable BASE_BRANCH (not hardcoded main) - Update all path references in AGENTS.md and SKILL.md
| # 2. Create branch from base SHA (before fetching file to avoid SHA race) | ||
| print(f"Creating branch {branch} from {base_branch} ({base_sha[:8]})...") | ||
| run_gh([ | ||
| "-X", "POST", f"repos/{repo}/git/refs", | ||
| "-f", f"ref=refs/heads/{branch}", | ||
| "-f", f"sha={base_sha}" | ||
| ], check=False) |
There was a problem hiding this comment.
When branch creation is silently swallowed (
check=False) and the branch already exists, the script reads the file from a branch that may have had OLD_STRING already replaced by a prior run. The next step then fails with the opaque "ERROR: OLD_STRING not found in file!" message with no indication that the branch pre-existed. This makes script retries very confusing to debug. The return value from run_gh contains the GitHub error JSON ({"message": "Reference already exists"}), which is easy to inspect.
| # 2. Create branch from base SHA (before fetching file to avoid SHA race) | |
| print(f"Creating branch {branch} from {base_branch} ({base_sha[:8]})...") | |
| run_gh([ | |
| "-X", "POST", f"repos/{repo}/git/refs", | |
| "-f", f"ref=refs/heads/{branch}", | |
| "-f", f"sha={base_sha}" | |
| ], check=False) | |
| # 2. Create branch from base SHA (before fetching file to avoid SHA race) | |
| print(f"Creating branch {branch} from {base_branch} ({base_sha[:8]})...") | |
| branch_result = run_gh([ | |
| "-X", "POST", f"repos/{repo}/git/refs", | |
| "-f", f"ref=refs/heads/{branch}", | |
| "-f", f"sha={base_sha}" | |
| ], check=False) | |
| if "ref" not in branch_result: | |
| msg = branch_result.get("message", "unknown error") | |
| if "already exists" in msg: | |
| print(f"ERROR: Branch '{branch}' already exists. Delete it or use a different branch name.", file=sys.stderr) | |
| else: | |
| print(f"ERROR: Branch creation failed: {msg}", file=sys.stderr) | |
| sys.exit(1) |
Prompt To Fix With AI
This is a comment left during a code review.
Path: skills/bug-triage/scripts/push_fix_to_github.py
Line: 70-76
Comment:
When branch creation is silently swallowed (`check=False`) and the branch already exists, the script reads the file from a branch that may have had `OLD_STRING` already replaced by a prior run. The next step then fails with the opaque "ERROR: OLD_STRING not found in file!" message with no indication that the branch pre-existed. This makes script retries very confusing to debug. The return value from `run_gh` contains the GitHub error JSON (`{"message": "Reference already exists"}`), which is easy to inspect.
```suggestion
# 2. Create branch from base SHA (before fetching file to avoid SHA race)
print(f"Creating branch {branch} from {base_branch} ({base_sha[:8]})...")
branch_result = run_gh([
"-X", "POST", f"repos/{repo}/git/refs",
"-f", f"ref=refs/heads/{branch}",
"-f", f"sha={base_sha}"
], check=False)
if "ref" not in branch_result:
msg = branch_result.get("message", "unknown error")
if "already exists" in msg:
print(f"ERROR: Branch '{branch}' already exists. Delete it or use a different branch name.", file=sys.stderr)
else:
print(f"ERROR: Branch creation failed: {msg}", file=sys.stderr)
sys.exit(1)
```
How can I resolve this? If you propose a fix, please make it concise.Add Step 1b covering: - When to ask for OS/shell/runtime/reproducibility - Common Windows-specific bug patterns (paths, shell syntax, ConPTY, named pipes, NTFS case-insensitivity, localhost IPv6 stalls) - Key cross-platform files (platform.ts, CROSS_PLATFORM.md, etc.) - Tagging OS-specific issues with 'to-reproduce' Based on the actual Windows support implementation in #1025 (platform.ts, runtime-process, CROSS_PLATFORM.md).
1. Environment Info Collection (Step 1):
- Standard template: OS, shell, runtime, AO version, Node version, install method
- Prevents wasted time tracing wrong code versions
2. Duplicate Search Strategy (Step 2):
- Search by symptom, component, AND error message
- Always search --state all (open + closed — bugs regress)
- Check PRs too (fixes sometimes land without issues)
3. Stop-and-Ask Triggers (Step 1c):
- Explicit criteria: 3 failed hypotheses, can't reproduce, upstream bug,
UI-only bug without screenshot, unknown environment
- Includes template for asking the reporter
4. Pre-Submission Checklist (Step 4.1b):
- Reporter attribution, commit hash, AO version, confidence score,
cross-links, concrete reproduction steps, screenshots ready
- Verify all before creating the issue
5. Confidence Scoring (Step 4.4):
- High/Medium/Low with clear criteria
- Maps to labels: bug only / to-explore / to-reproduce
- Example from PR #1608 where high confidence was wrong
6. Cross-Linking Related Issues (Step 4.5):
- Search by subsystem after filing
- Include Related section with one-line descriptions
- Helps maintainers see patterns across issues
7. Subsystem-Specific Triage Quick Reference:
- Table mapping subsystems to required info and key files
- Common misrouting patterns (terminal, stuck session, config)
Summary
Adds a
skills/bug-triage/directory with the bug triage skill I've been using internally, adapted for any agent working on this repo.What's included
skills/bug-triage/SKILL.md— Full triage workflow:git log -Sarchaeologyscripts/push_fix_to_github.py)skills/bug-triage/scripts/push_fix_to_github.py— Push a single-file fix and create a PR entirely via GitHub API. No local checkout needed. UsesOLD_STRING/NEW_STRINGenv vars for the replacement.AGENTS.md— Added a "Skills" section referencing the bug-triage skill so any agent (Claude Code, Codex, OpenCode, etc.) discovers it automatically.Why
Currently the bug triage workflow lives only in my internal skill store. Any other agent working on this repo has no idea how to triage bugs properly — no dedup, no root cause depth, no consistent issue formatting, no fix PR workflow.
Checking it into the repo means:
Changes from internal version
ao_sessions,ao_observability, etc.) → replaced withghCLI equivalentsTest plan
push_fix_to_github.pyis executable and well-documented