Agent Performance Report - Week of 2026-03-11

[github-actions[bot]]

Executive Summary

• Agents analyzed: 166 workflows (114 Copilot · 40 Claude · 11 Codex · 1 Gemini)
• Compilation rate: 166/166 (100%) — all workflows compiled ✅
• Agent Quality Score: 85/100 (↑1 from 84 — Codex recovery lifts baseline)
• Agent Effectiveness Score: 84/100 (→ stable)
• Workflow Health Score: 72/100 (↑2 from 70 — two recoveries!)
• Ecosystem Health: 97% (159/166 workflows fully operational)
• Top performers: The Great Escapi, Daily Safe Outputs Conformance, Contribution Check
• Needs attention: Smoke Gemini (new failure), Safe Output Health Monitor, Lockdown workflows

---

🎉 Significant Recoveries Since Last Report

Two workflows recovered this week, improving the overall baseline:

Workflow	Status	Tracking Issue	Duration Down
Smoke Codex	✅ RECOVERED (Mar 11)	#20285 can be closed	~2 weeks (OpenAI restriction)
Duplicate Code Detector	✅ RECOVERED (Mar 11)	#20304 can be closed	~1 week

The OpenAI cybersecurity restriction that blocked gpt-5.3-codex engine workflows has been resolved. Smoke Codex run #2215 succeeded on March 11.

---

Performance Rankings

Top Performing Agents 🏆

Rank	Agent	Quality	Effectiveness	Notes
1	The Great Escapi	95/100	95/100	Ultra-efficient, ~75K tokens/run, consistent ✅
2	Daily Safe Outputs Conformance	93/100	92/100	~164K tokens, catches conformance regressions
3	Contribution Check	92/100	91/100	~301K tokens, PR quality gating at scale
4	Smoke Claude / Smoke Copilot	90/100	90/100	Both passing — ecosystem stability indicators
5	Agent Container Smoke Test	88/100	87/100	~139K tokens, reliable container validation
6	Metrics Collector	85/100	82/100	Recovered in prior period, now stable ✅
7	AI Moderator	84/100	83/100	Mostly healthy after Codex recovery

View Engine Performance Breakdown

Copilot Engine (114 workflows — 69% of ecosystem)

• Largest engine fleet, strongest coverage across scheduled + event workflows
• Coverage: daily/scheduled (majority), slash commands, PR review, reaction-triggered
• Typical characteristics: Good at structured reporting, analysis, code quality tasks
• No new engine-level issues detected

Claude Engine (40 workflows — 24% of ecosystem)

• Strong performance on documentation, deep analysis, multi-step reasoning workflows
• Notable workflows: daily-doc-healer, daily-security-red-team, copilot-agent-analysis
• All Claude workflows currently operational ✅

Codex Engine (11 workflows — 7% of ecosystem)

• RECOVERED from 2-week OpenAI restriction ✅
• Smoke Codex (run Add daily instructions janitor workflow to maintain github-agentic-workflows.instructions.md #2215) and Duplicate Code Detector back online March 11
• Remaining Codex workflows: ai-moderator (healthy), daily-fact, deep-report, issue-arborist, etc.
• Issues [aw] Smoke Codex failed #20285 and [aw] Duplicate Code Detector failed #20304 can now be closed

Gemini Engine (1 workflow — <1% of ecosystem)

• Smoke Gemini: 100% schedule failure — new issue this period
• Error: add_comment context error — likely missing GitHub context for issue/PR targeting
• Needs investigation and tracking issue

---

Agents Needing Improvement 📉

P1 Infrastructure: Lockdown Token (4 workflows failing)

Not an agent quality issue — these agents are correctly implemented but blocked by missing token.

• Issue Monster — 100% failure rate, tracked by [P1] Lockdown token failures: Issue Monster, PR Triage Agent, Daily Issues Report #20315
• PR Triage Agent — 100% failure rate, tracked by [P1] Lockdown token failures: Issue Monster, PR Triage Agent, Daily Issues Report #20315
• Daily Issues Report — 100% failure rate, tracked by [P1] Lockdown token failures: Issue Monster, PR Triage Agent, Daily Issues Report #20315
• Org Health Report — weekly failure, no dedicated issue

Status: All programmatic fix paths closed (#17414, #17807 — closed "not_planned"). Manual intervention by a repo admin required to provide GH_AW_GITHUB_TOKEN.

Effectiveness impact: These 4 workflows represent ~2.4% of the ecosystem. The issues they produce (triage, daily reports) go ungenerated, creating manual workload.

P2 Warning: New/Ongoing Failures

Workflow	Failure Rate	Issue	Root Cause	Severity
Smoke Gemini	100% schedule	None	add_comment context error	P2
Safe Output Health Monitor	2 consecutive	#20305	Under investigation	P2
Smoke Update Cross-Repo PR	100%	#20288	Pre-agent failure	P2

Smoke Gemini analysis: The add_comment tool requires GitHub context (issue or PR number) when called without an explicit item_number. Smoke tests running on a schedule trigger lack this context. The workflow likely needs to either: (a) specify an explicit item_number for its test comment, or (b) switch from add_comment to create_issue for schedule-based smoke validation.

---

Quality Analysis

Output Quality Distribution

Based on observable metrics (compilation rate, run success, recovery patterns, safe output conformance):

Tier	Score Range	Count	%
Excellent	80–100	~82	~49%
Good	60–79	~68	~41%
Fair	40–59	~12	~7%
Poor	<40	~4	~2%

Poor tier: lockdown-affected workflows (infrastructure, not quality) + Smoke Gemini (new failure).

Common quality patterns observed:

1. ✅ Well-structured issue/discussion reports with markdown formatting
2. ✅ Appropriate use of collapsible sections (<details>)
3. ✅ Consistent tracking issue management (creating, updating, referencing)
4. ⚠️ Some agents creating issues without expiry management (Org Health Report gap)
5. ⚠️ Smoke tests on schedule triggers not always handling missing GitHub context

---

Behavioral Patterns

Productive Patterns ✅

• Infrastructure issue tracking: Agents consistently create and update tracking issues with expiry dates, then escalate when unresolved — healthy accountability loop
• Recovery detection: Workflow Health Manager correctly identified Smoke Codex + Duplicate Code Detector recoveries and updated related issues
• Cross-orchestrator coordination: Health/Performance/Campaign managers sharing memory files reduces duplicate analysis work
• Outdated lock file resolution: The 12 outdated lock files flagged in the last report (blog-auditor, cloclo, etc.) are now resolved — make recompile was executed ✅

Problematic Patterns ⚠️

• Missing event context in smoke tests: Smoke Gemini demonstrates a pattern where schedule-triggered smoke tests assume GitHub event context (issue/PR) that isn't present — other smoke tests may have similar latent risk
• Stale tracking issues: Issues [aw] Smoke Codex failed #20285 (Smoke Codex) and [aw] Duplicate Code Detector failed #20304 (Duplicate Code Detector) are still open despite the workflows recovering — agents should track their own issues to closure
• Lockdown escalation ceiling: 4 workflows hitting the same infrastructure wall with no resolution path — "closed as not_planned" fix attempts create fatigue

---

Coverage Analysis

Coverage by Domain

Well-covered areas:

• Code quality (code-simplifier, daily-code-metrics, duplicate-code-detector, ci-coach)
• Documentation (daily-doc-healer, daily-doc-updater, developer-docs-consolidator)
• PR lifecycle (contribution-check, auto-triage-issues, copilot-pr-merged-report)
• Security (daily-security-red-team, code-scanning-fixer)
• Smoke testing (smoke-claude, smoke-copilot, smoke-codex ✅, agent-container-smoke-test)

Coverage gaps identified:

• Gemini engine validation: Single smoke test, currently failing — no backup validation path
• Org Health Report: Failing without dedicated tracking issue
• Cross-repo PR workflows: Smoke-update-cross-repo-PR failing with no investigation

Trigger type distribution:

• Scheduled (117) — 70% of workflows
• Workflow dispatch (140) — available for manual triggers on nearly all
• Slash commands (16) — interactive user triggers
• PR-triggered (15) — reactive PR quality
• Reaction-triggered (13) — user engagement workflows

---

Recommendations

High Priority

1. Create tracking issue for Smoke Gemini — The add_comment context error needs a fix: specify explicit item_number or use create_issue for schedule smoke tests

   • Estimated effort: 30 min
   • Impact: Restore Gemini engine validation coverage

2. Close stale tracking issues — Issues [aw] Smoke Codex failed #20285 (Smoke Codex) and [aw] Duplicate Code Detector failed #20304 (Duplicate Code Detector) are open but both workflows recovered on March 11

   • Action: Close both issues with "resolved" status

3. Create Org Health Report tracking issue — One of the four lockdown-affected workflows has no dedicated tracking issue unlike the others

   • Estimated effort: 5 min

Medium Priority

4. Audit smoke test workflows for event-context assumptions — Smoke Gemini revealed a pattern: schedule-triggered tests calling add_comment without explicit item_number will fail. Other smoke tests may have the same latent risk.

   • Review: smoke-agent, smoke-codex, smoke-claude, smoke-copilot for add_comment usage

5. Escalate Safe Output Health Monitor — Two consecutive failures tracked in [aw] Safe Output Health Monitor failed #20305. Needs root cause investigation before becoming a longer-running issue.

Low Priority

6. Lockdown token resolution — Continue monitoring [P1] Lockdown token failures: Issue Monster, PR Triage Agent, Daily Issues Report #20315. If the manual intervention path remains blocked by end of month, consider agent-prompt alternatives that don't require GH_AW_GITHUB_TOKEN
7. Investigate Smoke Update Cross-Repo PR — Pre-agent failure ([aw] Smoke Update Cross-Repo PR failed #20288) — may be a configuration or permission issue

---

Trends

Metric	2026-03-01	2026-03-07	2026-03-11	Trend
Agent Quality	85/100	84/100	85/100	↑ (Codex recovery)
Agent Effectiveness	85/100	84/100	84/100	→ Stable
Workflow Health	73/100	74/100	72/100	↓ (new P2s)
Total Workflows	162	165	166	↑ Growing
Compiled	162/162	165/165	166/166	✅ 100%
P1 Failures	3	6	4	↓ (Codex resolved)
P2 Failures	1	2	3	↑ (new issues)

---

Actions Taken This Run

• ✅ Analyzed 166 workflows across 4 engine types
• ✅ Detected 2 recoveries (Smoke Codex, Duplicate Code Detector)
• ✅ Identified 1 new failure (Smoke Gemini)
• ✅ Confirmed 0 outdated lock files (prior 12 resolved ✅)
• ✅ Updated agent-performance-latest.md shared memory
• ✅ Updated shared-alerts.md coordination notes

Next Steps

1. Track down and fix Smoke Gemini add_comment context error
2. Close resolved issues [aw] Smoke Codex failed #20285 and [aw] Duplicate Code Detector failed #20304
3. Monitor Safe Output Health Monitor ([aw] Safe Output Health Monitor failed #20305)
4. Continue lockdown escalation path via [P1] Lockdown token failures: Issue Monster, PR Triage Agent, Daily Issues Report #20315
5. Audit other schedule-based smoke tests for event-context assumptions

---

Analysis period: 2026-03-07 to 2026-03-11
Next report: 2026-03-18
Run: §22965967194

References:

• §22965967194 — This run
• §22941596501 — Workflow Health (Mar 11)
• §22803659178 — Previous Performance (Mar 7)

AI generated by Agent Performance Analyzer - Meta-Orchestrator · history

• expires on Mar 12, 2026, 5:44 PM UTC

[github-actions[bot]]

🤖 Beep boop! The smoke test agent stopped by to say hello! 👋

I was here at 2026-03-11T18:07Z running the Copilot smoke test for run §22967098653. Everything's looking healthy in the gh-aw universe! ✨

poof 💨 — the smoke test agent vanishes into the GitHub Actions ether...

📰 BREAKING: Report filed by Smoke Copilot · ◷

[github-actions[bot]]

🎉 The smoke test agent returns with a GRAND FINALE comment! 🎉

After conducting extensive research (clicking buttons and reading pages), I can confirm that:

🌟 gh-aw builds like a dream
🚀 GitHub loads (revolutionary finding)
📝 Files can be written (groundbreaking)
🔍 PRs do indeed exist

This has been your friendly neighborhood smoke test agent. I'll see myself out. 🤖💨

(P.S. Serena, if you're reading this — please start your MCP server next time 🙏)

📰 BREAKING: Report filed by Smoke Copilot · ◷

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-03-12T17:44:47.369Z.

Closed by Workflow