🎯 Repository Quality Improvement Report - Workflow Compilation Observability

[github-actions[bot]]

🎯 Repository Quality Improvement Report - Workflow Compilation Observability

Analysis Date: 2025-11-15
Focus Area: Workflow Compilation Observability & Error Recovery
Strategy Type: Custom
Custom Area: Yes - This focus area is specifically tailored to gh-aw's unique nature as a workflow compiler that transforms markdown to GitHub Actions YAML.

Executive Summary

This analysis focuses on the observability and recoverability of workflow compilation failures - a critical user experience area unique to gh-aw as a compiler tool. Given that users write workflows in natural language markdown that get compiled to YAML, the quality of error messages, validation feedback, and recovery guidance directly impacts developer productivity and satisfaction.

Key Findings: The repository shows strong foundations in validation architecture (8,636 lines of validation code across 10+ domain-specific files), excellent error message examples (147 messages include concrete examples), and good troubleshooting documentation. However, there are significant opportunities to improve error recovery guidance, contextual help, and progressive enhancement of error messages.

Impact: Better compilation observability will reduce time-to-resolution for workflow errors, lower the barrier to entry for new users, and decrease support burden on maintainers.

Full Analysis Report

Focus Area: Workflow Compilation Observability & Error Recovery

Current State Assessment

Rationale for Custom Focus Area: Unlike typical code quality or documentation reviews, this area addresses gh-aw's unique challenge: users express complex GitHub Actions workflows in natural language, which must be compiled and validated. The quality of compilation feedback directly determines whether users can successfully author and debug workflows. This is repository-specific because few projects combine natural language authoring, schema validation, and GitHub Actions compilation.

Metrics Collected:

Metric	Value	Status
Total validation code	8,636 lines	✅
Files with error handling	385 files	✅
Error creation statements	840 instances	✅
Formatted error messages (console)	25 instances	⚠️
Error messages with examples	147 instances	✅
Error messages with guidance	7 instances	❌
Troubleshooting doc pages	20 pages	✅
Invalid YAML file generation	4 instances	✅
Workflows using safe-outputs	73 of 116 (63%)	✅

Findings

Strengths

• Comprehensive validation architecture: Well-organized domain-specific validation files (10+ validation modules)
• Schema-driven errors: JSON schema validation provides detailed error locations with line/column numbers
• Example-rich errors: 147 error messages include concrete YAML examples showing correct usage
• Good troubleshooting docs: 20+ documentation pages cover common issues
• Invalid YAML preservation: Failed compilations write .invalid.yml files for inspection

Areas for Improvement

1. Error Recovery Guidance (High Impact)

• Gap: Only 7 error messages (0.8%) include references to documentation or recovery steps
• Impact: Users see what's wrong but not how to fix it beyond the immediate example
• Example: Schema validation errors show field requirements but don't link to comprehensive frontmatter documentation

2. Contextual Help Links (High Impact)

• Gap: Error messages lack hyperlinks to relevant documentation sections
• Impact: Users must manually search documentation to understand complex configuration requirements
• Example: MCP server configuration errors don't reference the MCP integration guide

3. Progressive Error Enhancement (Medium Impact)

• Gap: Error messages are flat - no differentiation between beginner vs. advanced guidance
• Impact: New users get overwhelmed with technical details; experienced users want concise messages
• Example: Engine validation errors could offer --help engine for detailed explanation

4. Compilation Summary Statistics (Medium Impact)

• Gap: Summary shows counts but not actionable insights (e.g., most common error types)
• Impact: Users compiling multiple workflows can't quickly identify systematic issues
• Example: Compilation summary could categorize errors: "3 schema errors, 2 validation errors, 1 network error"

5. Interactive Error Resolution (Low Impact)

• Gap: No suggestions for auto-fixing common errors
• Impact: Manual fixes required even for trivial issues like typos
• Example: "Did you mean 'copilot'?" for engine: copilott

Detailed Analysis

Error Message Quality Analysis

Excellent Examples (Following error message style guide):

// From engine_validation.go
return fmt.Errorf("invalid engine: %s. Valid engines are: copilot, claude, codex, custom. Example: engine: copilot", engineID)

Pattern: [What's wrong] + [What's expected] + [Example]

Opportunities for Enhancement:

// Current (good but could be better)
return fmt.Errorf("tool '%s' mcp configuration missing required property '%s'...", toolName, propertyName)

// Enhanced (adds recovery path)
return fmt.Errorf("tool '%s' mcp configuration missing required property '%s'. See MCP configuration guide: (redacted) Example:...", toolName, propertyName)

Compilation Feedback Flow

Current Flow:

1. User runs gh aw compile
2. Errors printed to stderr (some with console formatting, most without)
3. Summary shows total errors/warnings
4. User manually investigates each error

Enhanced Flow (Proposed):

1. User runs gh aw compile
2. Errors printed with:
   • Severity indicator (🔴 Error, ⚠️ Warning)
   • Category tag ([schema], [validation], [network])
   • Documentation link (if applicable)
   • Suggested fix (for common errors)
3. Summary shows:
   • Error breakdown by category
   • Most common error type
   • Suggested next action ("Run 'gh aw compile --help schema' for frontmatter reference")
4. User follows contextual guidance

Documentation Coverage Analysis

Well-Covered Topics:

• Workflow won't compile (syntax, missing fields, invalid values)
• Lock file issues (not generated, orphaned files)
• Import/include issues (file not found, path errors)

Under-Covered Topics:

• MCP server configuration errors
• Engine-specific limitations
• Network permission errors
• Safe output configuration issues
• Validation timing (when does what get validated?)

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for Claude to process.

Improvement Tasks

The following code regions and tasks should be processed by the Copilot agent. Each section is marked for easy identification by the planner agent.

Task 1: Add Documentation Links to Error Messages

Priority: High
Estimated Effort: Medium
Focus Area: Error Recovery Guidance

Description:
Enhance error messages throughout the validation system to include documentation links for complex configuration issues. Focus on errors related to MCP servers, engines, network permissions, and safe outputs where users need detailed guidance beyond a simple example.

Acceptance Criteria:

• MCP configuration errors link to /guides/mcps/ documentation
• Engine errors link to /reference/engines/ documentation
• Network permission errors link to /reference/network/ documentation
• Safe output errors link to /reference/safe-outputs/ documentation
• Links use format: See: (redacted)}
• Documentation links added to at least 20 high-impact error messages

Code Region: pkg/workflow/*_validation.go, pkg/workflow/mcp_config*.go, pkg/workflow/engine*.go

Review error messages in validation files and add contextual documentation links to help users recover from compilation failures. Focus on complex configuration areas where users most frequently need additional guidance beyond the inline example.

Prioritize:
1. MCP server configuration errors (pkg/workflow/mcp_config_validation.go)
2. Engine validation errors (pkg/workflow/engine_validation.go)
3. Network permission errors (pkg/workflow/engine_network.go)
4. Safe output configuration errors (pkg/workflow/safe_*.go)

Follow the existing error message format:
- [What's wrong] + [What's expected] + [Documentation link] + [Example]

Ensure all documentation paths are valid and point to existing pages in docs/src/content/docs/.

---

Task 2: Enhance Compilation Summary with Error Categorization

Priority: High
Estimated Effort: Medium
Focus Area: Compilation Observability

Description:
Improve the compilation summary output to categorize errors and provide actionable insights. Instead of just showing counts, analyze error types and suggest next actions based on the most common failure patterns.

Acceptance Criteria:

• Summary categorizes errors into: Schema, Validation, Network, MCP, Safe-Output
• Shows most common error type with count
• Suggests relevant --help command or documentation based on error patterns
• Uses console formatting for better readability
• Maintains backward compatibility with JSON output mode

Code Region: pkg/cli/compile_command.go (CompilationStats, printCompilationSummary)

Enhance the printCompilationSummary function to provide categorized error reporting:

1. Add error categorization:
   - Track error types during compilation
   - Categorize into: [schema], [validation], [network], [mcp], [safe-output], [other]

2. Improve summary output format:
   - Show breakdown: "✗ 5 errors: 3 schema, 2 validation"
   - Highlight most common error type
   - Suggest action: "💡 Tip: Run 'gh aw compile --help frontmatter' for schema reference"

3. Add to CompilationStats structure:
   - ErrorCategories map[string]int
   - MostCommonCategory string

4. Maintain JSON output compatibility for programmatic usage

Reference console package for formatting: pkg/console/formatting.go

---

Task 3: Implement Fuzzy Suggestion for Common Typos

Priority: Medium
Estimated Effort: Small
Focus Area: Interactive Error Resolution

Description:
Add "Did you mean?" suggestions for common typos in engine names, permission names, and tool names. Use simple Levenshtein distance or string similarity to suggest corrections.

Acceptance Criteria:

• Engine typos suggest closest valid option (e.g., "copilott" → "Did you mean 'copilot'?")
• Permission typos suggest valid permission names
• Tool name typos suggest registered tools
• Suggestions only shown if similarity score > 0.7
• Works for both frontmatter and MCP configuration

Code Region: pkg/workflow/engine_validation.go, pkg/workflow/permissions.go, pkg/workflow/mcp_config_validation.go

Add fuzzy matching suggestions to validation errors:

1. Create helper function in pkg/workflow/suggestions.go:
   - func suggestClosestMatch(input string, validOptions []string) string
   - Use simple string similarity (Levenshtein distance or similar)
   - Return suggestion if similarity > 0.7

2. Integrate with existing validation:
   - Engine validation: Suggest closest engine name
   - Permission validation: Suggest valid permission names
   - MCP tool validation: Suggest registered MCP tools

3. Update error messages to include suggestions:
   - "invalid engine: copilott. Did you mean 'copilot'? Valid engines are: ..."

4. Keep existing example format intact, just prepend the suggestion

Consider using: github.com/agnivade/levenshtein for simple implementation

---

Task 4: Create Troubleshooting Guide for MCP Configuration

Priority: Medium
Estimated Effort: Medium
Focus Area: Documentation Enhancement

Description:
Create comprehensive troubleshooting documentation specifically for MCP server configuration errors, which are currently under-documented but frequently encountered by users based on error message analysis.

Acceptance Criteria:

• New file: docs/src/content/docs/troubleshooting/mcp-configuration.md
• Covers common MCP errors: missing command, invalid type, HTTP vs stdio confusion
• Includes working examples for each common error scenario
• Cross-referenced from existing MCP guides
• Follows Diátaxis framework (how-to guide format)

Code Region: docs/src/content/docs/troubleshooting/ (new file)

Create a comprehensive MCP configuration troubleshooting guide:

1. Document structure (following Diátaxis how-to format):
   - Title: "Troubleshooting MCP Server Configuration"
   - Sections:
     * Common MCP Configuration Errors
     * HTTP vs Stdio Transport Confusion
     * Missing Required Properties
     * Container vs Command Issues
     * Engine Compatibility Problems

2. For each error type:
   - Symptom: What error message users see
   - Cause: Why this error occurs
   - Solution: Step-by-step fix with code examples
   - Related: Links to relevant reference docs

3. Include working examples:
   - Correct stdio MCP configuration
   - Correct HTTP MCP configuration
   - Correct container-based MCP
   - Engine-specific MCP limitations

4. Cross-reference from:
   - docs/src/content/docs/guides/mcps.md
   - docs/src/content/docs/troubleshooting/common-issues.md

Use existing troubleshooting docs as template for style and format.

---

Task 5: Add Verbose Error Context Flag

Priority: Low
Estimated Effort: Large
Focus Area: Progressive Error Enhancement

Description:
Implement a --error-context flag that provides beginner-friendly explanations for validation errors, while keeping default output concise for experienced users. This enables progressive disclosure of error information.

Acceptance Criteria:

• New flag: --error-context or -c in compile command
• When enabled, shows additional context for each error
• Context includes: Why this matters, common causes, related concepts
• Preserves existing error format as default
• Works with both single-file and batch compilation

Code Region: pkg/cli/compile_command.go, pkg/workflow/compiler.go

Add progressive error context feature:

1. Add CLI flag to compile command:
   - Name: --error-context
   - Short: -c
   - Description: "Show additional context and explanations for errors"

2. Update Compiler to support context mode:
   - Add ErrorContext bool to CompileConfig
   - Pass through to validation functions
   - Store context metadata with ValidationResult

3. Enhance error output when flag is enabled:
   - Default: "invalid engine: copilott. Valid engines are: copilot, claude, codex, custom"
   - With context: "invalid engine: copilott. Valid engines are: copilot, claude, codex, custom
     
     Why this matters: The engine determines which AI service processes your workflow.
     Common causes: Typos in engine name, using unsupported engine version.
     Learn more: (redacted)"

4. Maintain backward compatibility:
   - Default behavior unchanged
   - JSON output includes context in separate field

This is a larger effort - consider phased implementation starting with high-impact validation errors.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2025-11-15	Workflow Compilation Observability	Custom	Y	First analysis run - established baseline metrics for error observability

---

🎯 Recommendations

Immediate Actions (This Week)

1. Add documentation links to top 20 error messages - Priority: High - Quick wins with immediate user value
2. Enhance compilation summary with error categorization - Priority: High - Better batch compilation feedback

Short-term Actions (This Month)

1. Implement fuzzy typo suggestions - Priority: Medium - Improves user experience for common mistakes
2. Create MCP troubleshooting guide - Priority: Medium - Fills documentation gap in frequently problematic area

Long-term Actions (This Quarter)

1. Add error context mode - Priority: Low - Progressive enhancement for different user skill levels
2. Build error message taxonomy - Track which errors are most common to prioritize improvements
3. Consider error analytics - Instrument compilation to understand real-world failure patterns

---

📈 Success Metrics

Track these metrics to measure improvement in Workflow Compilation Observability:

• Error Recovery Rate: Percentage of errors that include recovery guidance (documentation links or suggestions) - Current: 0.8% → Target: 80%
• Mean Time to Resolution: Average time users spend debugging compilation errors - Establish baseline → Target: -30%
• Documentation Link Coverage: Percentage of complex errors with relevant doc links - Current: 0.8% → Target: 100% for high-impact errors
• Compilation Summary Usefulness: User feedback on actionability of error summaries - Establish feedback loop
• Common Error Reduction: Reduction in typo-related errors after fuzzy suggestion - Measure after implementation

---

Next Steps

1. Review and prioritize the tasks above
2. Assign tasks to Copilot agent via planner agent
3. Track progress on improvement items
4. Re-evaluate this focus area in 3 months (or when significant improvements are complete)
5. Next analysis: 2025-11-16 - Focus area will be selected based on diversity algorithm (60% custom, 30% standard, 10% reuse)

---

Generated by Repository Quality Improvement Agent
Focus Area Selection Strategy: Custom (60% probability) - Tailored to gh-aw's unique workflow compilation challenges

AI generated by Repository Quality Improvement Agent

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.