fix(ci): unstick WarpBuild nightly release builds (#1183)

* fix(ci): use supported warp-windows-2022 runner label for nightly

* feat(ci): fail nightly fast when WarpBuild runners never pick up jobs

* docs(ci): WarpBuild one-time setup and stuck-queue runbook

* fix(ci): harden nightly queue watchdog per review

* docs(ci): verify runner-group contents before allowing public repos

* docs(ci): correct concurrency pending-run wording

Author: shadowfax92

.github/workflows/nightly-release.yml

@@ -59,7 +59,7 @@ jobs:
 
           matrix='[
             {"platform":"linux","arch":"x64","runner":"warp-ubuntu-2204-x64-32x","config":"release.linux.ci.yaml","timeout":660},
-            {"platform":"windows","arch":"x64","runner":"warp-windows-2025-x64-32x","config":"release.windows.ci.yaml","timeout":780},
+            {"platform":"windows","arch":"x64","runner":"warp-windows-2022-x64-32x","config":"release.windows.ci.yaml","timeout":780},
             {"platform":"macos","arch":"arm64","runner":"warp-macos-15-arm64-12x","config":"release.macos.arm64.ci.yaml","timeout":720}
           ]'
           if [ "$PLATFORMS" != "all" ]; then
@@ -80,6 +80,66 @@ jobs:
             echo "publish=$publish"
           } >> "$GITHUB_OUTPUT"
 
+  # WarpBuild provisions runners on demand via webhook; when none arrive
+  # (org runner-group policy, bad label, account issue) the build jobs sit
+  # queued until GitHub discards them 24h later, holding the
+  # nightly-release concurrency group all day. Fail fast instead — see the
+  # troubleshooting section of
+  # packages/browseros/build/docs/nightly-warpbuild-ci.md.
+  queue-watchdog:
+    needs: plan
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      actions: write
+    steps:
+      - name: Fail fast when no runner picks up the builds
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          set -euo pipefail
+          deadline=$(( $(date +%s) + 20 * 60 ))
+          failures=0
+          while :; do
+            if jobs="$(gh api "repos/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID/jobs?per_page=100" \
+              --jq '[.jobs[] | select(.name | startswith("build (")) | {name, status}]')"; then
+              failures=0
+            else
+              failures=$(( failures + 1 ))
+              if [ "$failures" -ge 3 ]; then
+                echo "::error::queue-watchdog could not list this run's jobs (3 consecutive failures); builds are unwatched this run."
+                exit 1
+              fi
+              sleep 120
+              continue
+            fi
+            total="$(jq 'length' <<<"$jobs")"
+            queued="$(jq '[.[] | select(.status == "queued")] | length' <<<"$jobs")"
+            in_progress="$(jq '[.[] | select(.status == "in_progress")] | length' <<<"$jobs")"
+            if [ "$total" -gt 0 ] && [ "$queued" -eq 0 ]; then
+              echo "All $total build jobs picked up by runners."
+              exit 0
+            fi
+            if [ "$(date +%s)" -ge "$deadline" ]; then
+              doc="packages/browseros/build/docs/nightly-warpbuild-ci.md"
+              if [ "$total" -eq 0 ]; then
+                echo "::error::queue-watchdog matched no 'build (' jobs — were the matrix job names changed? Fix the filter; not cancelling."
+                exit 1
+              fi
+              stuck="$(jq -r '[.[] | select(.status == "queued") | .name] | join(", ")' <<<"$jobs")"
+              # Cancel only when nothing is live: a queued job is then the
+              # only thing keeping the run (and concurrency group) pinned.
+              if [ "$in_progress" -eq 0 ]; then
+                echo "::error::No build job is running and these never left the queue: $stuck. Check the runner-group public-repo setting, runner labels, and the WarpBuild dashboard — see $doc. Cancelling the run to free the nightly-release concurrency group."
+                gh api -X POST "repos/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID/cancel"
+              else
+                echo "::error::Still queued after 20 min: $stuck. In-progress builds keep running — see $doc."
+              fi
+              exit 1
+            fi
+            sleep 120
+          done
+
   build:
     needs: plan
     strategy:

packages/browseros/build/docs/nightly-warpbuild-ci.md

@@ -12,7 +12,7 @@ up here, that workflow can be retired.
 | Platform | Label | Specs | Disk |
 | --- | --- | --- | --- |
 | Linux x64 | `warp-ubuntu-2204-x64-32x` | 32 vCPU / 128 GB | 150 GB |
-| Windows x64 | `warp-windows-2025-x64-32x` | 32 vCPU / 128 GB | 256 GB |
+| Windows x64 | `warp-windows-2022-x64-32x` | 32 vCPU / 128 GB | 256 GB |
 | macOS arm64 | `warp-macos-15-arm64-12x` | M4 Pro, 12 vCPU / 44 GB | 270 GB |
 
 There is no 32-core macOS tier; 12x is WarpBuild's largest Mac. WarpBuild
@@ -21,6 +21,51 @@ apply — but `timeout-minutes` must be set explicitly (the implicit default is
 360). Linux's 150 GB disk is the tightest fit: ~60-75 GB checkout +
 ~25-40 GB out dir + OS image. The workflow prints `df -h` after each build.
 
+## One-time setup (WarpBuild)
+
+The `warpbuildbot` GitHub app is installed org-wide on `browseros-ai`
+(since 2026-06-11). Two more things must be true before any `warp-*` job
+leaves `queued`:
+
+1. **The org must allow self-hosted runners on public repos.** WarpBuild
+   runners register as org-level self-hosted runners, and GitHub blocks
+   those on public repositories by default
+   (https://www.warpbuild.com/docs/ci/public-repos). BrowserOS is public,
+   so an org admin must check: Organization Settings → Actions → Runner
+   groups → Default → "Allow public repositories". Via API (needs
+   `admin:org` scope):
+
+   ```bash
+   gh auth refresh -h github.com -s admin:org
+   gh api orgs/browseros-ai/actions/runner-groups \
+     --jq '.runner_groups[] | {id, name, allows_public_repositories}'
+   gh api -X PATCH "orgs/browseros-ai/actions/runner-groups/<id>" \
+     -F allows_public_repositories=true
+   ```
+
+   Before flipping the toggle, check what else lives in that group — it
+   widens exposure for every runner in it:
+
+   ```bash
+   gh api "orgs/browseros-ai/actions/runner-groups/<id>/runners" \
+     --jq '.runners[] | {name, status, labels: [.labels[].name]}'
+   ```
+
+   Expect only ephemeral `warp-*` runners (usually none while idle). The
+   signed-nightly Mac (`browseros-builder`) is registered at the repo
+   level, so this org-group toggle does not change its exposure. If the
+   group ever holds other persistent org-level runners, give WarpBuild a
+   dedicated runner group instead of widening Default.
+
+2. **The WarpBuild org must be active**: sign in at
+   https://app.warpbuild.com/, confirm the `browseros-ai` connection and
+   that billing/credits are set up — runners are not provisioned without
+   an active account.
+
+Smoke test after changing either:
+`gh workflow run "Nightly Release Build" -f platforms=linux`, then watch
+the build job leave `queued` within ~5 minutes (`gh run watch`).
+
 ## Per-night pipeline (per platform)
 
 1. `actions/checkout` + `astral-sh/setup-uv`.
@@ -127,3 +172,44 @@ The first run per platform is the cache warm-up; expect cold timings. If a
 pin bump lands, the next night is cold again for that version. To force a
 fresh checkout, bump the `v1` in the cache key (workflow) — for Windows also
 delete the old object under `ci-cache/chromium/` in R2.
+
+## Troubleshooting: jobs stuck in `queued`
+
+A job no runner ever picked up shows `runner_id: 0` and empty steps:
+
+```bash
+gh run view <run-id> --json jobs --jq '.jobs[] | {name, status}'
+gh api repos/browseros-ai/BrowserOS/actions/jobs/<job-id> \
+  --jq '{status, runner_id, runner_name, labels}'
+```
+
+Causes, in the order to check:
+
+1. **Runner group blocks public repos** — see one-time setup above. This
+   stalls all platforms at once.
+2. **Label not in WarpBuild's catalog** — supported images: Ubuntu
+   22.04/24.04 (x64, arm64), macOS 14/15/26 (arm64), Windows Server 2022
+   (x64) (https://www.warpbuild.com/docs/ci/preinstalled-software). An
+   unsupported label queues forever (this workflow originally shipped a
+   `windows-2025` label that WarpBuild does not image); WarpBuild reports
+   no error back to GitHub.
+3. **WarpBuild account** — org connection or billing lapsed
+   (https://app.warpbuild.com/).
+4. **WarpBuild capacity or incident** — rare; check their dashboard.
+
+Mechanics worth knowing:
+
+- GitHub discards self-hosted jobs queued for more than 24h, and the
+  workflow's `nightly-release` concurrency group
+  (`cancel-in-progress: false`) makes the next run wait (newer pending
+  runs supersede older pending ones) — one stuck night delays the next
+  by a full day (runs 27367077749 → 27407228486 did exactly this). The `queue-watchdog` job therefore steps in at the
+  20-minute mark: it cancels the run when no build job is actually
+  running (everything stuck in queue or already finished), and fails
+  loudly without cancelling while any build is in progress. In that
+  mixed case, cancel the run manually once the live builds finish — a
+  still-queued job otherwise pins the group for up to 24h with no
+  watcher left.
+- Fixing the root cause does not revive already-queued jobs: WarpBuild
+  provisions on the `workflow_job.queued` webhook, which has already
+  fired. Cancel the stuck run and re-dispatch.