[go-fan] Go Module Review: github.com/sourcegraph/conc

[github-actions[bot]]

🐹 Go Fan Report: github.com/sourcegraph/conc

Today I reviewed Sourcegraph's conc library - a structured concurrency toolkit that makes parallel programming in Go safer and more ergonomic. This module provides panic-safe goroutine pools with automatic error aggregation.

Module Overview

The conc library provides better primitives for concurrent programming compared to raw goroutines and sync.WaitGroup. Key benefits include automatic panic recovery, bounded goroutine pools, type-safe result collection with generics, and multi-error aggregation.

Version in use: v0.3.0 (pre-1.0, stable but may have breaking changes before 1.0)

Current Usage in gh-aw

The project uses conc in a single, focused location:

• File: pkg/cli/logs.go:894
• Function: downloadRunArtifactsConcurrent
• Pool Type: ResultPool[DownloadResult]
• Purpose: Parallel downloading and processing of GitHub Actions workflow artifacts
• Concurrency: Limited to 10 simultaneous downloads via WithMaxGoroutines(MaxConcurrentDownloads)

Usage Pattern

p := pool.NewWithResults[DownloadResult]().WithMaxGoroutines(MaxConcurrentDownloads)

for _, run := range actualRuns {
    run := run
    p.Go(func() DownloadResult {
        // Download artifacts, extract metrics, analyze logs
        // Network-bound operations taking 100ms to several seconds
        return result
    })
}

results := p.Wait()

What the Pool Does

Each goroutine in the pool:

1. Checks cache for previously processed run summaries
2. Downloads artifacts from GitHub API if needed
3. Extracts metrics from log files
4. Analyzes access logs, firewall logs, and redacted domains
5. Collects missing tools and MCP failures
6. Updates progress spinners atomically
7. Returns structured DownloadResult with all findings

Task duration: 100ms to several seconds per download (network-bound)

Research Findings

Repository Insights

• GitHub: sourcegraph/conc
• Stars: ~9,000
• License: MIT
• Maturity: Pre-1.0 (v0.3.0 from February 2023)
• Status: Stable API, but targeting 1.0 release with potential breaking changes

Key Features

1. Six Pool Variants: Pool (basic), ErrorPool, ResultPool, ContextPool, ResultErrorPool, ResultContextPool
2. Generics Support: Type-safe result collection with ResultPool[T]
3. Builder Pattern: Fluent configuration (WithMaxGoroutines, WithErrors, WithContext, WithCancelOnError, etc.)
4. Automatic Panic Recovery: All pools catch panics from tasks and propagate via Wait()
5. Error Aggregation: Uses errors.Join (Go 1.20+) or go.uber.org/multierr (Go 1.19)
6. Scoped Concurrency: Ensures all goroutines complete before pool exits

Recent Updates (v0.3.0)

• Configuration panics if called after first Go() (prevents race conditions)
• ContextPool always cancels context when Wait() returns
• panics.Recovered no longer implements error directly (use .AsError())
• Added panics.Try() for single-function panic recovery

Performance Characteristics

From the documentation:

"Pool is efficient, but not zero cost. Startup and teardown come with an overhead of around 1µs, and each task has an overhead of around 300ns."

Analysis: The project's tasks (100ms-seconds) are well above this overhead threshold, making pool usage highly appropriate.

Improvement Opportunities

🏃 Quick Wins

1. Add Context Support for Timeout/Cancellation

   • Current: No way to cancel in-progress downloads
   • Improvement: Use WithContext(ctx) for graceful shutdown
   • Code:

     ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
     defer cancel()
     p := pool.NewWithResults[DownloadResult]().
         WithContext(ctx).
         WithMaxGoroutines(MaxConcurrentDownloads)

   • Benefit: Better resource management, user-initiated cancellation support

2. Extract Pool Configuration to Helper Function

   • Current: Configuration inline at single call site
   • Improvement: Create newDownloadPool() helper for future reusability
   • Benefit: Easier to adjust concurrency settings, add new options

3. Consider WithCancelOnError for Fail-Fast Behavior

   • Current: All downloads proceed even if many fail
   • Improvement: Add WithCancelOnError() to stop on first critical error
   • Note: Current "complete all downloads" behavior may be intentional
   • Benefit: Faster failure feedback, reduced API quota consumption

✨ Feature Opportunities

4. Leverage conc/iter for Result Processing

   • Current: Sequential loop to count successes (lines 1081-1087, 1091-1097)
   • Improvement: Use iter.ForEach for parallel result analysis
   • Code:

     import "github.com/sourcegraph/conc/iter"
     
     successCount := atomic.Int64{}
     iter.ForEach(results, func(r *DownloadResult) {
         if r.Error == nil && !r.Skipped {
             successCount.Add(1)
         }
     })

   • Benefit: Faster aggregation for large result sets

5. Add Panic Recovery Logging

   • Current: Panics propagate but aren't logged
   • Improvement: Wrap with panics.Catcher to log before re-panic
   • Benefit: Better debugging of concurrent failures

6. Make Concurrency Limit Configurable

   • Current: Hard-coded to 10
   • Improvement: Read from environment variable or flag
   • Code:

     const DefaultConcurrentDownloads = 10
     MaxConcurrentDownloads = getEnvInt("GH_AW_DOWNLOAD_CONCURRENCY", DefaultConcurrentDownloads)

   • Benefit: Users can tune for their network/API limits

📐 Best Practice Alignment

7. Loop Variable Capture ✅ CORRECT

   • Uses run := run pattern properly
   • Note: Go 1.22+ makes this unnecessary (per-iteration loop vars)

8. Task Duration ✅ EXCELLENT

   • Tasks are 100ms-seconds (network-bound)
   • Well above ~1µs pool overhead threshold
   • Perfect use case for structured concurrency

9. Configuration Timing ✅ CORRECT

   • All With* calls before first Go()
   • Complies with v0.3.0 panic-on-late-config behavior

10. Error Handling Pattern ⚠️ ACCEPTABLE

    • Errors stored in result struct (no early termination)
    • Design choice: complete all downloads vs fail-fast
    • Current approach valid for batch operations

🔧 General Improvements

11. Add Pool Performance Monitoring

    • Track time from first Go() to Wait() completion
    • Log goroutine utilization metrics
    • Helps determine if concurrency limit is optimal

12. Test Panic Recovery

    • Add test cases that trigger panics in download goroutines
    • Verify panics propagate correctly through Wait()
    • Ensure no goroutine leaks on panic

13. Document Concurrency Choice

    • Add comment explaining why MaxConcurrentDownloads = 10
    • Note: GitHub API rate limits, network bandwidth, etc.

Recommendations

Priority 1: Robustness Improvements

1. ✅ Add context support - Enable timeout and cancellation
2. ✅ Extract pool helper - Improve maintainability

Priority 2: Consider Based on Requirements

3. 🤔 Evaluate fail-fast - Would early termination improve UX?
4. 🤔 Result processing parallelism - Is aggregation a bottleneck?

Priority 3: Future Enhancements

5. 📊 Configurability - Let users tune concurrency
6. 🐛 Panic logging - Aid debugging
7. 📈 Performance metrics - Optimize pool settings

Overall Assessment

Current Usage: ✅ EXCELLENT

The project uses conc correctly and idiomatically. The implementation:

• Follows best practices from the library documentation
• Configures pool properly before use
• Handles appropriate task durations for pool overhead
• Captures loop variables correctly
• Uses type-safe generics for result collection

Key Strengths:

• Single, focused usage reduces complexity
• Network-bound tasks are perfect for this library
• Bounded concurrency prevents resource exhaustion
• Type-safe result collection with generics

Suggested Enhancements:

• Context support for better lifecycle management
• Configurability for power users
• Optional fail-fast behavior for CI environments

Next Steps

1. ✅ No immediate changes required - Current implementation is solid
2. 🔄 Consider enhancements - Context support and configurability add value
3. 👀 Monitor for v1.0 - Review breaking changes when library reaches 1.0

---

Module summary saved to: specs/mods/conc.md

AI generated by Go Fan

[github-actions[bot]]

⚓ Avast! This discussion be marked as outdated by Go Fan.
🗺️ A newer treasure map awaits ye at Discussion #5271.
Fair winds, matey! 🏴‍☠️