[Quality Improvement] CLI User Experience & Help Text Quality - 2025-11-13

[github-actions[bot]]

🎯 Repository Quality Improvement Report - CLI User Experience & Help Text Quality

Analysis Date: 2025-11-13
Focus Area: CLI User Experience & Help Text Quality
Strategy Type: Standard Category
Custom Area: No

Executive Summary

The gh-aw CLI demonstrates strong command organization with 17 distinct commands and comprehensive help text. Analysis revealed 606 error message instances across 186 Go files, with good adoption of the console formatting system (297 Info messages, 179 Warnings, 76 Success messages). However, there are opportunities to improve user experience through enhanced command examples, better error recovery guidance, and improved discoverability of advanced features. The documentation structure is solid with 46 pages, but command-specific tutorials and troubleshooting guides could be expanded to reduce user friction during onboarding and complex workflows.

Key metrics show strong CLI foundation: 72-line README, 402-line developer guide, and consistent help text patterns. The --help output provides good structure but could benefit from more contextual examples and cross-references between related commands.

Full Analysis Report

Focus Area: CLI User Experience & Help Text Quality

Current State Assessment

The gh-aw CLI provides a comprehensive command-line interface for managing GitHub Agentic Workflows. The tool offers 17 main commands covering workflow lifecycle from initialization (init, new) through compilation (compile), execution (run), and analysis (logs, audit, status).

Metrics Collected:

Metric	Value	Status
Total CLI commands	17	✅
Go source files (CLI/cmd)	186	✅
Error message instances	606	⚠️
Console formatting adoption	650+ uses	✅
Documentation pages	46	✅
README length	72 lines	⚠️
Developer guide length	402 lines	✅
Code examples in docs	Multiple	✅
Active workflows	77	✅

Findings

Strengths

1. Consistent Console Formatting: Strong adoption of the console formatting system with 297 Info messages, 179 Warnings, and 76 Success messages, providing visual consistency across commands.

2. Comprehensive Command Coverage: 17 well-organized commands covering the full workflow lifecycle from init to audit.

3. Rich Help Text: Commands like compile, logs, and audit provide detailed usage examples with multiple scenarios and flag combinations.

4. Good Documentation Structure: 46 documentation pages organized by examples, setup guides, and reference material.

5. Clear Command Naming: Intuitive command names like compile, logs, audit, mcp inspect that clearly communicate their purpose.

6. Status Command: Provides comprehensive table view of all workflows with compilation status and engine information.

Areas for Improvement

1. Error Message Actionability (High Priority): With 606 error messages across the codebase, many could be enhanced with specific recovery actions or suggestions for next steps.

2. Command Discovery (Medium Priority): The main help text lists "compile" twice, suggesting potential organizational issues in command registration.

3. Interactive Examples (Medium Priority): While help text provides examples, there's limited guidance on common workflow patterns or "getting started" scenarios.

4. Troubleshooting Documentation (High Priority): Only 2 mentions of friction-related terms in documentation suggests limited troubleshooting guidance for common pain points.

5. Command Cross-References (Low Priority): Help text could better guide users between related commands (e.g., compile → run → logs → audit workflow).

Detailed Analysis

Error Message Quality

Current State:

• 606 total error messages across CLI and command packages
• Good use of console.FormatErrorMessage for consistent styling
• Examples show structured error output with context

Improvement Opportunities:

• Add "Next Steps" or "Did you mean?" suggestions to common errors
• Include references to relevant documentation for complex errors
• Provide example corrections for syntax/configuration errors

Example Current Error:

✗ workflow 'nonexistent' not found in local .github/workflows or components

Enhanced Error Could Be:

✗ workflow 'nonexistent' not found in local .github/workflows or components

Suggestions:
  - Run 'gh aw status' to see all available workflows
  - Create a new workflow with 'gh aw new nonexistent'
  - Check for typos in the workflow name

Command Discoverability

Current State:

• Main --help lists all 17 commands
• Duplicate "compile" entry suggests registration issue
• Commands organized alphabetically

Improvement Opportunities:

• Group commands by workflow phase (Setup, Development, Execution, Analysis)
• Add "Common Tasks" section in main help
• Provide quick-start examples in root help

Suggested Help Structure:

Setup Commands:
  init        Initialize repository for agentic workflows
  new         Create a new workflow file

Development Commands:
  compile     Compile Markdown to YAML workflows
  mcp         Manage MCP servers
  
Execution Commands:
  run         Run workflows on GitHub Actions
  enable      Enable agentic workflows
  disable     Disable workflows and cancel runs

Analysis Commands:
  logs        Download and analyze workflow logs
  audit       Investigate a single workflow run
  status      Show status of all workflows

Help Text Examples

Current State:

• Commands like logs and audit provide 8-10 examples
• Good coverage of flag combinations
• Clear syntax demonstrations

Improvement Opportunities:

• Add workflow-oriented examples beyond just flag combinations
• Include expected output samples for key commands
• Cross-reference related commands in examples

Example Enhancement for logs --help:

Current examples focus on flags:

gh aw logs --start-date -1w
gh aw logs --engine claude

Could add workflow-oriented examples:

# Common Workflow: Debugging a failed run
gh aw logs my-workflow -c 5       # Get recent runs
gh aw audit (run-id)              # Deep dive into failure

# Common Workflow: Cost analysis
gh aw logs --json > costs.json    # Export for analysis

Documentation Gaps

Current State:

• 46 documentation pages covering examples, setup, and reference
• Limited troubleshooting content (2 mentions of friction terms)
• Good example coverage for different trigger types

Improvement Opportunities:

• Add troubleshooting guide for common errors
• Create "Debugging Workflows" tutorial
• Add FAQ section for frequent user questions
• Expand quick-start with more scenarios

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for Claude to process.

Improvement Tasks

The following code regions and tasks should be processed by the Copilot agent. Each section is marked for easy identification by the planner agent.

Task 1: Enhance Error Messages with Actionable Suggestions

Priority: High
Estimated Effort: Medium
Focus Area: Error Message Quality

Description:
Improve error messages across the CLI to include specific recovery actions, suggestions for next steps, and references to relevant documentation. Focus on the most common error scenarios: workflow not found, compilation failures, and authentication issues.

Acceptance Criteria:

• "Workflow not found" errors include suggestions to run status or new commands
• Compilation errors reference specific frontmatter fields and provide example fixes
• Authentication errors explain token requirements and setup steps
• All enhanced errors maintain consistent formatting with console package
• Error messages include "Suggestions:" or "Next Steps:" sections where appropriate

Code Region: pkg/cli/*.go, pkg/workflow/compiler.go

# Copilot Task: Enhanced Error Messages

Update error handling in CLI commands to provide actionable suggestions and recovery guidance.

Focus on these error patterns:
1. **Workflow Not Found** (`pkg/cli/compile_command.go`, `pkg/cli/run_command.go`):
   - Add suggestions to check available workflows with `status`
   - Suggest creating workflow with `new` command
   - Check for similar workflow names (fuzzy matching hints)

2. **Compilation Failures** (`pkg/workflow/compiler.go`, `pkg/workflow/validation.go`):
   - Reference specific frontmatter field that's invalid
   - Provide example of correct syntax
   - Link to relevant documentation section

3. **Authentication Errors** (`pkg/cli/logs.go`, `pkg/cli/audit.go`):
   - Explain which token is missing (GITHUB_TOKEN vs COPILOT_GITHUB_TOKEN)
   - Provide setup instructions
   - Reference authentication documentation

Pattern to follow:
```go
// Before
return fmt.Errorf("workflow '%s' not found", workflowID)

// After
suggestions := []string{
    "Run 'gh aw status' to see all available workflows",
    "Create a new workflow with 'gh aw new " + workflowID + "'",
    "Check for typos in the workflow name",
}
return console.FormatErrorWithSuggestions(
    fmt.Sprintf("workflow '%s' not found in local .github/workflows or components", workflowID),
    suggestions,
)

Maintain consistency with existing console formatting patterns.

---

#### Task 2: Fix Duplicate Command Registration

**Priority**: High  
**Estimated Effort**: Small  
**Focus Area**: CLI Structure

**Description:**
Investigate and fix the duplicate "compile" command entry that appears in the main `--help` output. Ensure clean command registration in the CLI.

**Acceptance Criteria:**
- [ ] Main `--help` output shows each command exactly once
- [ ] No duplicate command registrations in cmd/gh-aw/main.go or related files
- [ ] Command list remains alphabetically sorted
- [ ] All compile command functionality preserved

**Code Region:** `cmd/gh-aw/main.go`, `pkg/cli/compile*.go`

```markdown
# Copilot Task: Fix Duplicate Command Registration

Investigate why "compile" appears twice in the main help output and fix the root cause.

Steps:
1. Check `cmd/gh-aw/main.go` for duplicate `AddCommand()` calls
2. Review `pkg/cli/compile_command.go` for multiple command constructors
3. Search for duplicate command initialization in CLI setup code
4. Verify no other commands have similar duplication issues

Look for patterns like:
```go
// Potential issue
rootCmd.AddCommand(compileCmd)
rootCmd.AddCommand(compileCmd) // Duplicate?

Or:

// Multiple compile command definitions
var compileCmd = &cobra.Command{...}
var compileCommand = &cobra.Command{...} // Duplicate?

Fix by ensuring single, clean command registration.

---

#### Task 3: Add Command Grouping to Main Help Text

**Priority**: Medium  
**Estimated Effort**: Medium  
**Focus Area**: Command Discoverability

**Description:**
Organize the main `--help` output to group commands by workflow phase (Setup, Development, Execution, Analysis) rather than alphabetically. This will improve discoverability for new users by showing the logical workflow progression.

**Acceptance Criteria:**
- [ ] Commands grouped into: Setup, Development, Execution, Analysis
- [ ] Group headers clearly visible in help output
- [ ] Commands within groups remain alphabetically sorted
- [ ] "Common Tasks" quick reference added at top
- [ ] Maintains compatibility with existing `--help` format

**Code Region:** `cmd/gh-aw/main.go`

```markdown
# Copilot Task: Command Grouping in Help Text

Update the main command help text to organize commands by workflow phase using Cobra's command groups feature.

Implementation approach:

```go
// In cmd/gh-aw/main.go or root command setup

rootCmd.AddGroup(&cobra.Group{
    ID:    "setup",
    Title: "Setup Commands:",
})
rootCmd.AddGroup(&cobra.Group{
    ID:    "development",
    Title: "Development Commands:",
})
rootCmd.AddGroup(&cobra.Group{
    ID:    "execution",
    Title: "Execution Commands:",
})
rootCmd.AddGroup(&cobra.Group{
    ID:    "analysis",
    Title: "Analysis Commands:",
})

// Then assign commands to groups
initCmd.GroupID = "setup"
newCmd.GroupID = "setup"
compileCmd.GroupID = "development"
mcpCmd.GroupID = "development"
runCmd.GroupID = "execution"
enableCmd.GroupID = "execution"
disableCmd.GroupID = "execution"
logsCmd.GroupID = "analysis"
auditCmd.GroupID = "analysis"
statusCmd.GroupID = "analysis"

Add "Common Tasks" section to root command's Long description:

rootCmd.Long = `GitHub Agentic Workflows from GitHub Next

Common Tasks:
  gh aw init                  # Set up a new repository
  gh aw new my-workflow       # Create your first workflow
  gh aw compile               # Compile all workflows
  gh aw run my-workflow       # Execute a workflow

...rest of description...`

Ensure this works with Cobra version in go.mod.

---

#### Task 4: Add Troubleshooting Documentation Section

**Priority**: Medium  
**Estimated Effort**: Large  
**Focus Area**: Documentation

**Description:**
Create a comprehensive troubleshooting guide in the documentation that addresses common error scenarios, their causes, and step-by-step solutions. This should reduce support burden and improve user self-service.

**Acceptance Criteria:**
- [ ] New troubleshooting.md file created in docs/src/content/docs/
- [ ] Covers at least 10 common error scenarios
- [ ] Each scenario includes: symptoms, cause, solution, prevention
- [ ] Includes debugging workflow compilation, execution, and MCP issues
- [ ] Cross-references related documentation pages
- [ ] Follows Diátaxis framework (How-To Guide format)

**Code Region:** `docs/src/content/docs/troubleshooting.md` (new file)

```markdown
# Copilot Task: Create Troubleshooting Documentation

Create a new troubleshooting guide at `docs/src/content/docs/troubleshooting.md` following the Diátaxis How-To Guide pattern.

Structure:
```markdown
---
title: Troubleshooting Guide
description: Solutions to common issues when working with GitHub Agentic Workflows
---

## Workflow Compilation Issues

### "Workflow not found" Error
**Symptoms**: `gh aw compile my-workflow` returns error
**Cause**: Workflow file doesn't exist or incorrect path
**Solution**:
1. Run `gh aw status` to see all available workflows
2. Check for typos in workflow name
3. Verify file exists at `.github/workflows/my-workflow.md`
**Prevention**: Use `gh aw new` to create workflows

[Continue with 9 more scenarios...]

## MCP Server Issues

### MCP Server Connection Failures
[Details...]

## Authentication Problems

### Missing GitHub Token
[Details...]

## Performance Issues

### Slow Compilation Times
[Details...]

Cover these topics:

• Workflow not found errors
• Frontmatter validation failures
• MCP server connection issues
• Authentication/token problems
• Compilation performance
• Workflow execution failures
• Log download issues
• Network/firewall configuration
• Safe output processing errors
• Dependency installation failures

Each section should be actionable with specific commands to run.

---

#### Task 5: Add Output Examples to Key Commands

**Priority**: Low  
**Estimated Effort**: Medium  
**Focus Area**: Help Text Quality

**Description:**
Enhance help text for the most commonly used commands (`status`, `logs`, `audit`) by adding sample output snippets. This helps users understand what to expect before running commands.

**Acceptance Criteria:**
- [ ] `status --help` includes sample table output
- [ ] `logs --help` shows example metrics summary
- [ ] `audit --help` demonstrates report format
- [ ] Examples are properly formatted in help text
- [ ] Output examples are representative of actual command results

**Code Region:** `pkg/cli/status_command.go`, `pkg/cli/logs.go`, `pkg/cli/audit.go`

```markdown
# Copilot Task: Add Output Examples to Command Help

Update the help text for `status`, `logs`, and `audit` commands to include sample output examples.

For each command, add an "Example Output:" section after the Examples section:

**Status Command** (`pkg/cli/status_command.go`):
```go
var statusCmd = &cobra.Command{
    Use:   "status",
    Short: "Show status of agentic workflows",
    Long: `Show status of agentic workflows...
    
Examples:
  gh aw status
  gh aw status -v
  
Example Output:
  Workflow            | Engine  | Compiled | Status | Time Remaining
  ------------------- | ------- | -------- | ------ | --------------
  daily-news          | copilot | Yes      | Active | 2h 15m
  issue-classifier    | claude  | Yes      | Idle   | N/A
  weekly-research     | copilot | No       | Error  | N/A
`,
    // ...rest
}

Logs Command (pkg/cli/logs.go):

// Add to Long description after examples:

Example Output:
  📊 Workflow Run Summary
  
  Workflow: daily-news
  Total Runs: 5
  Date Range: 2024-01-01 to 2024-01-05
  
  | Run ID | Status    | Duration | Tokens | Cost    |
  |--------|-----------|----------|--------|---------|
  | 123456 | Succeeded | 2m 34s   | 25.3k  | $0.052  |
  | 123457 | Failed    | 1m 12s   | 12.1k  | $0.024  |

Audit Command (pkg/cli/audit.go):

// Add to Long description after examples:

Example Output:
  # Workflow Run Audit Report
  
  **Run ID**: 1234567890
  **Workflow**: issue-classifier
  **Status**: Failed
  **Duration**: 2m 45s
  
  ## Error Summary
  - MCP server connection timeout (line 234)
  - Missing GITHUB_TOKEN environment variable
  
  ## Recommendations
  1. Configure GitHub token in repository secrets
  2. Increase MCP server timeout in workflow config

Format examples to fit within typical terminal width (80-100 characters).

---

## 📊 Historical Context

<details>
<summary><b>Previous Focus Areas</b></summary>

| Date | Focus Area | Type | Custom | Key Outcomes |
|------|------------|------|--------|--------------|
| 2025-11-13 | CLI User Experience | Standard | N | 5 tasks created focusing on error messages, help text, and documentation |

This is the first quality improvement run. Future runs will build on this foundation and explore different aspects of the repository.

</details>

---

## 🎯 Recommendations

### Immediate Actions (This Week)
1. **Fix duplicate command registration** - High priority, small effort (Task 2)
2. **Enhance critical error messages** - High priority, focus on "workflow not found" and compilation failures (Task 1)

### Short-term Actions (This Month)
1. **Implement command grouping in help text** - Improves discoverability for new users (Task 3)
2. **Create troubleshooting documentation** - Reduces support burden and improves self-service (Task 4)

### Long-term Actions (This Quarter)
1. **Add output examples to all commands** - Improves help text quality across the board (Task 5)
2. **Conduct user testing** - Validate improvements with actual users
3. **Create interactive tutorial** - Guide new users through common workflows

---

## 📈 Success Metrics

Track these metrics to measure improvement in the **CLI User Experience & Help Text Quality**:

- **Error Message Actionability**: 0% → 60%+ of errors include suggestions (count errors with "Suggestions:" or "Next Steps:")
- **Help Text Example Coverage**: 3 commands → 10+ commands with output examples
- **Troubleshooting Coverage**: 2 pain points documented → 10+ common issues with solutions
- **User Onboarding Time**: Establish baseline for "time to first successful workflow" and track improvement
- **Command Discovery**: Track which commands are discovered via help vs. documentation

---

## Next Steps

1. Review and prioritize the tasks above
2. Assign tasks to Copilot agent via planner agent
3. Track progress on improvement items
4. Re-evaluate this focus area in 30 days to measure impact
5. **Next analysis: 2025-11-14** - Focus area will be selected based on diversity algorithm (targeting 60% custom areas)

---

*Generated by Repository Quality Improvement Agent*  
*Focus Area Selection: Standard Category (87/100 random draw)*
*Strategy: Emphasizing custom repository-specific areas in future runs (60% target)*


> AI generated by [Repository Quality Improvement Agent](https://github.com/githubnext/gh-aw/actions/runs/19341450578)

[pelikhan]

/plan generate issue for Enhanced Error Messages

[github-actions[bot]]

✅ Agentic Plan Command completed successfully.

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.