Live-stack failure: Workflow did not complete within 25 minute polling timeout on ubuntu-latest-l

[a5c-ai]

Automated live-stack failure triage.

Scenario/job: Workflow did not complete within 25 minute polling timeout
OS: ubuntu-latest-l
Conclusion: timed_out
Run: https://github.com/a5c-ai/babysitter/actions/runs/31230474327
Job: https://github.com/a5c-ai/babysitter/actions/runs/31230474327
Affected component: live-stack / ci

Matrix:

[
  {
    "agent": "codex",
    "model": "foundry-gpt55",
    "mode": "ni",
    "install": "bp",
    "process_mode": "predefined",
    "live": true
  }
]

[a5c-ai]

Triage Summary

• One-line: This is a QA/live-stack monitoring timeout issue, not evidence that the referenced live-stack scenario itself failed.

Critical Evaluation

• Reporter's analysis needs correction: the 25 minute timeout belongs to the QA polling instruction, not the live-stack scenario execution. When inspected, run 31230474327 was still in_progress; Build All had succeeded and the selected Live Stack job was still running its E2E step.

Root Cause Analysis

• .github/workflows/qa-daily.yml:96-98 tells the QA agent to poll dispatched live-stack runs every 60 seconds with a 25 minute timeout.
• .github/workflows/live-stack.yml:436-465 gives the selected live-stack E2E step a 90 minute Actions timeout and sets LIVE_STACK_TEST_TIMEOUT_MS=4860000 / LIVE_STACK_COMMAND_TIMEOUT_MS=4800000.
• The referenced run is a workflow_dispatch run for staging; the live-stack job only reached Run selected live stack E2E after build/setup time, so the 25 minute poller can expire before the actual live-stack scenario reaches a terminal result.
• The actual bug is premature/ambiguous failure reporting from the monitor path.

Affected Components

• Packages/components: ci, triggers
• Effort: effort:small

Recommended Approach

Treat this as a QA dispatch / live-stack monitoring bug:

• Update the QA polling logic/prompt so a non-terminal dispatched run is reported as monitoring-timeout or needs-follow-up, not as a completed live-stack failure.
• Prefer waiting for a terminal workflow conclusion before creating live-stack-failure issues.
• If the polling budget is increased, account for queue/build time separately from scenario execution time.
• Add a focused guardrail using mocked gh run list/view output for an in-progress run exceeding the poll window.

Reproduction

Dispatch live-stack.yml on staging with:

[{"agent":"codex","model":"foundry-gpt55","mode":"ni","install":"bp","process_mode":"predefined","live":true}]

Then let queue/build time plus E2E startup exceed the QA monitor's 25 minute polling window. The monitor can report timed_out while the live-stack workflow is still running.

Related Issues/PRs

• Similar historical live-stack timeout/failure issues exist ([live-stack] sdk: ubuntu-latest-l Live Stack (ubuntu-latest-l, vanilla, hermes/claude-sonnet-4-6, bridged-interactive) #1602, [live-stack] sdk: ubuntu-latest-l Live Stack (ubuntu-latest-l, vanilla, hermes/gpt-5.5, bridged-interactive) #1599, live-stack failure: run-timeout on ubuntu-latest-l #1063, live-stack failure: ubuntu-latest-l Live Stack (ubuntu-latest-l, bp/predefined, codex/gpt-5.5, bridged-hooks) #1429), but none matched this exact run/scenario as a duplicate.
• Recent live-stack-adjacent changes include f37319247 and fe0e313bd; neither clearly fixes this monitoring timeout behavior.

Priority: medium

This creates false or premature live-stack-failure tracking and can waste triage time, but it does not prove the underlying live-stack scenario failed.

Risk Analysis

• Risk Level: risk:low
• Increasing the polling window too much could slow QA feedback.
  • Mitigation: separate queued/build time from scenario execution time and cap total wait with clear in-progress reporting.
• Changing labels/comments could hide genuine live-stack failures.
  • Mitigation: only classify as monitoring timeout while the dispatched run is non-terminal; keep failure issue creation for completed runs with failure/timed_out conclusions.
• Fixing live-stack.yml budgets alone would miss the source.
  • Mitigation: change the QA polling/triage path and verify against mocked/in-progress run data.

[a5c-ai]

Triage Summary

• One-line: Daily QA is creating a live-stack timed_out issue after its own 25 minute polling window, while the dispatched live-stack workflow is still legitimately running under a much longer workflow/test budget.

Critical Evaluation

• The issue is valid, but the reported conclusion needs correction: this is not evidence that the selected live-stack scenario reached a terminal failure.
• gh run view 31230474327 showed the workflow still in_progress; Build All ran from 00:34:29 to 00:56:57 UTC, and the selected live-stack job did not start until 01:06:01 UTC.
• The issue body says install: bp, but the GitHub job displays vanilla because bp + ni is routed through the vanilla_ni job group while retaining install_mode: babysitter-plugin in the matrix payload.

Root Cause Analysis

• Affected files/functions:
  • .github/workflows/qa-daily.yml: dispatch/poll instructions, especially the 25 minute per-run timeout.
  • .github/workflows/live-stack.yml: workflow_dispatch matrix grouping and live E2E timeout/budget.
  • packages/adapters/cli/tests/live-stack/scenario-contract.test.ts: covers live-stack step timeout alignment, but not qa-daily polling alignment.
  • packages/adapters/cli/tests/live-stack/pipeline-scenario.test.ts and primary-live-runner.ts: live-stack command/test timeout execution path.
• Actual root cause:
  • qa-daily.yml instructs the QA agent to poll dispatched live-stack.yml runs for only 25 minutes.
  • live-stack.yml allows Run selected live stack E2E up to 90 minutes, with LIVE_STACK_TEST_TIMEOUT_MS=4860000 and LIVE_STACK_COMMAND_TIMEOUT_MS=4800000.
  • The 25 minute poll window can expire before the dispatched workflow reaches the selected scenario, especially after build/queue time.
• Code evidence:
  • .github/workflows/qa-daily.yml lines 91-98 dispatch live-stack.yml and set the 25 minute polling timeout.
  • .github/workflows/live-stack.yml lines 436-464 define the 90 minute live E2E step and 81/80 minute test/command budgets.
  • .github/workflows/live-stack.yml lines 214-225 route bp + ni into vanillaNi; line 318 hardcodes that job display name as vanilla.

Affected Components

• Packages/components: ci, live-stack
• Effort: effort:small

Recommended Approach

1. Update the qa-daily.yml live-stack polling contract so its timeout is derived from, or at least aligned with, the live-stack workflow budget plus expected build/queue overhead.
2. Distinguish in_progress after polling budget from terminal failed/timed_out conclusions. If the workflow is still running, report it as deferred/still-running or continue with a larger hard cap.
3. Add a regression test that checks qa-daily’s polling timeout cannot be shorter than live-stack’s selected job budget.
4. Clean up the bp + ni matrix routing/job naming mismatch, or explicitly document/test that vanilla_ni can carry install_mode: babysitter-plugin so future triage does not misread it.

Reproduction

Dispatch live-stack.yml from daily QA with:

[{"agent":"codex","model":"foundry-gpt55","mode":"ni","install":"bp","process_mode":"predefined","live":true}]

If build time + queue time + scenario time exceeds 25 minutes, the QA agent can create a timed_out issue even though the live-stack workflow is still in progress and has a 90 minute live E2E step budget.

Related Issues/PRs

• Related live-stack timeout family: Live-stack failure: Live Stack (ubuntu-latest-l, bp/predefined, codex/gpt-5.5, interactive) #1645, but that is already marked duplicate of [live-stack] sdk: ubuntu-latest-l Live Stack (ubuntu-latest-l, vanilla, hermes/claude-sonnet-4-6, bridged-interactive) #1602 and covers an interactive case.
• Related CI/matrix triage pattern: Live-stack failure: ubuntu-latest-l Compute-Matrix #1586, but this issue has a different root cause: qa-daily polling timeout vs live-stack workflow budget mismatch.

Priority: medium

This degrades the daily live-stack QA signal and creates noisy issues, but it is not production data loss and manual workflow inspection remains a workaround.

Risk Analysis

• Risk Level: risk:medium
• Longer poll timeout can delay true failure reports.
  • Mitigation: keep a hard cap and use explicit still-running/deferred status before creating terminal failure issues.
• Matrix/job naming changes can affect artifact aggregation.
  • Mitigation: add workflow contract tests for bp + ni grouping, job display names, and artifact globs.
• Treating in-progress runs as non-failures can hide stuck jobs.
  • Mitigation: make the hard cap larger than the actual live-stack budget plus queue/build allowance, then create a distinct stale-run issue only after that cap.