Skip to content

chore: add bug-triage skill for agent-driven issue triage#1725

Open
i-trytoohard wants to merge 7 commits into
mainfrom
chore/add-bug-triage-skill
Open

chore: add bug-triage skill for agent-driven issue triage#1725
i-trytoohard wants to merge 7 commits into
mainfrom
chore/add-bug-triage-skill

Conversation

@i-trytoohard
Copy link
Copy Markdown
Collaborator

@i-trytoohard i-trytoohard commented May 7, 2026

Summary

Adds a skills/bug-triage/ directory with the bug triage skill I've been using internally, adapted for any agent working on this repo.

What's included

skills/bug-triage/SKILL.md — Full triage workflow:

  1. Pre-flight (pull latest main, identify target repo)
  2. Gather report context (chat, issues, live observation)
  3. Root cause analysis with git log -S archaeology
  4. Upstream dependency research
  5. Duplicate search → comment on existing or file new issue
  6. Screenshot upload (never use placeholders)
  7. Label and priority assignment (only approved labels)
  8. Push fix PRs via GitHub API (scripts/push_fix_to_github.py)
  9. NPM package regression diffing
  10. Remote code inspection without local clone

skills/bug-triage/scripts/push_fix_to_github.py — Push a single-file fix and create a PR entirely via GitHub API. No local checkout needed. Uses OLD_STRING/NEW_STRING env vars for the replacement.

AGENTS.md — Added a "Skills" section referencing the bug-triage skill so any agent (Claude Code, Codex, OpenCode, etc.) discovers it automatically.

Why

Currently the bug triage workflow lives only in my internal skill store. Any other agent working on this repo has no idea how to triage bugs properly — no dedup, no root cause depth, no consistent issue formatting, no fix PR workflow.

Checking it into the repo means:

  • Every agent gets the same triage playbook
  • The skill evolves with the codebase (PR-based updates)
  • No external dependency on a specific agent's skill store

Changes from internal version

  • Removed Hermes-specific tool references (ao_sessions, ao_observability, etc.) → replaced with gh CLI equivalents
  • Removed Discord-specific fetch script (kept the generic push script)
  • Removed AO server filesystem specifics (root-owned files, sudo pnpm)
  • Made all paths relative to the repo checkout
  • Removed internal memory/session references

Test plan

  • Verify SKILL.md renders correctly on GitHub
  • Verify push_fix_to_github.py is executable and well-documented
  • Verify AGENTS.md references the skill correctly
  • An agent reading AGENTS.md can discover and follow the skill workflow

Add .skills/bug-triage/ with SKILL.md and push_fix_to_github.py script.

The skill provides a complete triage workflow:
- Gather bug context from chat/issues/live observation
- Search for duplicate GitHub issues
- File well-structured issues with root cause analysis
- Push fix PRs via GitHub API (no local checkout needed)
- Git archaeology (git log -S) for regression tracking
- NPM package regression diffing
- Remote code inspection without local clone

Reference the skill in AGENTS.md so any agent working on this repo
can discover and follow the triage workflow automatically.

Tested across 100+ real bug triages on ComposioHQ/agent-orchestrator.
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 7, 2026

Test Coverage Report

No TypeScript source files changed in this PR.

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 7, 2026

Greptile Summary

This PR checks in a skills/bug-triage/ directory containing a full agent-facing bug triage playbook (SKILL.md) and a GitHub API helper script (push_fix_to_github.py), and registers the skill in AGENTS.md so any agent working on the repo can discover and follow the same workflow.

  • SKILL.md covers the end-to-end triage loop: context gathering, git archaeology, duplicate search, screenshot upload (now using the portable base64 < file | tr -d '\n' form), issue creation with slug-based asset branches, root-cause confidence scoring, and fix PR creation.
  • push_fix_to_github.py pushes a single-file fix entirely via the GitHub API; it creates the branch first then reads the file from that branch (eliminating the SHA race), validates NEW_STRING with an is None check (allowing empty-string deletions), supports BASE_BRANCH env override, and warns when OLD_STRING has multiple matches.
  • AGENTS.md references the skill at the correct path (skills/bug-triage/SKILL.md, no leading dot).

Confidence Score: 5/5

Safe to merge — all three files are documentation and a standalone helper script with no runtime integration into the main codebase.

The changes are additive only: a new skill directory, a self-contained Python helper, and a one-section addition to AGENTS.md. Every defect flagged in the previous review round has been addressed — the SHA race in the push script, the macOS base64 -w 0 flag, the NEW_STRING falsy check, the hardcoded base branch, the duplicate-match silent failure, the mismatched label name, and the wrong path in AGENTS.md. No existing code paths are modified.

No files require special attention.

Important Files Changed

Filename Overview
AGENTS.md Adds a "Skills" section pointing agents to the bug-triage skill at the correct path skills/bug-triage/SKILL.md (no leading dot).
skills/bug-triage/SKILL.md Comprehensive triage playbook; screenshot upload now uses portable `base64 < file
skills/bug-triage/scripts/push_fix_to_github.py Script creates the branch before fetching the file (eliminating the SHA race), checks NEW_STRING is None rather than falsy (allowing empty-string deletions), supports BASE_BRANCH env var, and prints a warning when OLD_STRING matches multiple times — all previously reported defects addressed.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Bug reported\nchat / issue / observation] --> B[Step 0: Gather context\nreporter, screenshots, logs]
    B --> C[Step 1: Understand issue\ntrace code path, git archaeology\nupstream dep research]
    C --> D{Root cause\nclear?}
    D -- No --> E[Stop-and-ask triggers\nask reporter for OS/shell/version]
    E --> C
    D -- Yes --> F[Step 2: Search duplicates\ngh issue list --state all\nby symptom + component + error]
    F --> G{Duplicate\nfound?}
    G -- Yes --> H[Step 3: Comment\non existing issue]
    G -- No --> I[Step 4.1b: Pre-submission checklist\nreporter, hash, version, repro steps]
    I --> J[Step 4.1c: Upload screenshots\nbranch issue-assets-SLUG\nbase64 portable upload]
    J --> K[Step 4.2: Create issue\ngh issue create]
    K --> L[Step 4.3-4.4: Labels + confidence\npriority label, High/Medium/Low]
    L --> M[Step 4.5: Cross-link related issues]
    M --> N{Fix\ntrivial?}
    N -- Yes --> O[Step 4.6: push_fix_to_github.py\ncreate branch → fetch file → apply replace → push → open PR]
    N -- Complex --> P[Note fix in issue\nsuggest spawning agent]
    N -- Unclear --> Q[Document findings\nflag for investigation]
    O --> R[Step 4.7: Post confirmation\nissue URL, PR URL, labels, root cause]
    P --> R
    Q --> R
Loading

Reviews (7): Last reviewed commit: "feat: add 6 triage improvements from rea..." | Re-trigger Greptile

Comment thread .skills/bug-triage/scripts/push_fix_to_github.py Outdated
Comment thread .skills/bug-triage/SKILL.md Outdated
Comment thread .skills/bug-triage/scripts/push_fix_to_github.py Outdated
Comment thread .skills/bug-triage/scripts/push_fix_to_github.py Outdated
Add concrete examples with links to actual issues and PRs:
- #1129: deep diagnosis vs surface-level triage
- #1151: placeholder URL RCA
- #1391: CSS regression via git log -S archaeology
- PR #1523: optional TypeScript interface fields
- PR #1608: npm package regression diffing
Comment thread .skills/bug-triage/SKILL.md Outdated
Comment thread skills/bug-triage/scripts/push_fix_to_github.py
Any agent following this skill must include clickable URLs when
mentioning issues or PRs. Bare '#123' without links is not allowed.
Comment thread .skills/bug-triage/SKILL.md Outdated
- Move .skills/ → skills/ (repo already has a skills/ directory)
- Fix label name: 'priority:medium' → 'priority: medium' (with space)
- Fix issue-assets branch naming: use slug instead of issue number
  (issue doesn't exist yet at upload time)
- Fix base64 command: portable across Linux and macOS (tr -d '\n')
- push_fix_to_github.py: allow empty NEW_STRING for deletion edits
- push_fix_to_github.py: warn on multiple OLD_STRING matches
- push_fix_to_github.py: create branch before fetching file (SHA race)
- push_fix_to_github.py: configurable BASE_BRANCH (not hardcoded main)
- Update all path references in AGENTS.md and SKILL.md
Comment thread AGENTS.md Outdated
@Priyanchew Priyanchew self-requested a review May 8, 2026 17:39
Comment on lines +70 to +76
# 2. Create branch from base SHA (before fetching file to avoid SHA race)
print(f"Creating branch {branch} from {base_branch} ({base_sha[:8]})...")
run_gh([
"-X", "POST", f"repos/{repo}/git/refs",
"-f", f"ref=refs/heads/{branch}",
"-f", f"sha={base_sha}"
], check=False)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 When branch creation is silently swallowed (check=False) and the branch already exists, the script reads the file from a branch that may have had OLD_STRING already replaced by a prior run. The next step then fails with the opaque "ERROR: OLD_STRING not found in file!" message with no indication that the branch pre-existed. This makes script retries very confusing to debug. The return value from run_gh contains the GitHub error JSON ({"message": "Reference already exists"}), which is easy to inspect.

Suggested change
# 2. Create branch from base SHA (before fetching file to avoid SHA race)
print(f"Creating branch {branch} from {base_branch} ({base_sha[:8]})...")
run_gh([
"-X", "POST", f"repos/{repo}/git/refs",
"-f", f"ref=refs/heads/{branch}",
"-f", f"sha={base_sha}"
], check=False)
# 2. Create branch from base SHA (before fetching file to avoid SHA race)
print(f"Creating branch {branch} from {base_branch} ({base_sha[:8]})...")
branch_result = run_gh([
"-X", "POST", f"repos/{repo}/git/refs",
"-f", f"ref=refs/heads/{branch}",
"-f", f"sha={base_sha}"
], check=False)
if "ref" not in branch_result:
msg = branch_result.get("message", "unknown error")
if "already exists" in msg:
print(f"ERROR: Branch '{branch}' already exists. Delete it or use a different branch name.", file=sys.stderr)
else:
print(f"ERROR: Branch creation failed: {msg}", file=sys.stderr)
sys.exit(1)
Prompt To Fix With AI
This is a comment left during a code review.
Path: skills/bug-triage/scripts/push_fix_to_github.py
Line: 70-76

Comment:
When branch creation is silently swallowed (`check=False`) and the branch already exists, the script reads the file from a branch that may have had `OLD_STRING` already replaced by a prior run. The next step then fails with the opaque "ERROR: OLD_STRING not found in file!" message with no indication that the branch pre-existed. This makes script retries very confusing to debug. The return value from `run_gh` contains the GitHub error JSON (`{"message": "Reference already exists"}`), which is easy to inspect.

```suggestion
    # 2. Create branch from base SHA (before fetching file to avoid SHA race)
    print(f"Creating branch {branch} from {base_branch} ({base_sha[:8]})...")
    branch_result = run_gh([
        "-X", "POST", f"repos/{repo}/git/refs",
        "-f", f"ref=refs/heads/{branch}",
        "-f", f"sha={base_sha}"
    ], check=False)
    if "ref" not in branch_result:
        msg = branch_result.get("message", "unknown error")
        if "already exists" in msg:
            print(f"ERROR: Branch '{branch}' already exists. Delete it or use a different branch name.", file=sys.stderr)
        else:
            print(f"ERROR: Branch creation failed: {msg}", file=sys.stderr)
        sys.exit(1)
```

How can I resolve this? If you propose a fix, please make it concise.

AO Bot added 2 commits May 9, 2026 05:34
Add Step 1b covering:
- When to ask for OS/shell/runtime/reproducibility
- Common Windows-specific bug patterns (paths, shell syntax, ConPTY,
  named pipes, NTFS case-insensitivity, localhost IPv6 stalls)
- Key cross-platform files (platform.ts, CROSS_PLATFORM.md, etc.)
- Tagging OS-specific issues with 'to-reproduce'

Based on the actual Windows support implementation in #1025
(platform.ts, runtime-process, CROSS_PLATFORM.md).
1. Environment Info Collection (Step 1):
   - Standard template: OS, shell, runtime, AO version, Node version, install method
   - Prevents wasted time tracing wrong code versions

2. Duplicate Search Strategy (Step 2):
   - Search by symptom, component, AND error message
   - Always search --state all (open + closed — bugs regress)
   - Check PRs too (fixes sometimes land without issues)

3. Stop-and-Ask Triggers (Step 1c):
   - Explicit criteria: 3 failed hypotheses, can't reproduce, upstream bug,
     UI-only bug without screenshot, unknown environment
   - Includes template for asking the reporter

4. Pre-Submission Checklist (Step 4.1b):
   - Reporter attribution, commit hash, AO version, confidence score,
     cross-links, concrete reproduction steps, screenshots ready
   - Verify all before creating the issue

5. Confidence Scoring (Step 4.4):
   - High/Medium/Low with clear criteria
   - Maps to labels: bug only / to-explore / to-reproduce
   - Example from PR #1608 where high confidence was wrong

6. Cross-Linking Related Issues (Step 4.5):
   - Search by subsystem after filing
   - Include Related section with one-line descriptions
   - Helps maintainers see patterns across issues

7. Subsystem-Specific Triage Quick Reference:
   - Table mapping subsystems to required info and key files
   - Common misrouting patterns (terminal, stuck session, config)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant