Cross-Platform Tests (windows-latest) has been failing on main for ~3 months, invisible in PR checks

[Pedrovaleriolopez]

Summary

Cross-Platform Tests (windows-latest) — all four Node matrices (18, 20, 22, 24) — has been failing on main for ~3 months. The same job on ubuntu-latest and macos-latest passes every time in the same runs; Pro Machine ID Stability (windows-latest) (a different, narrower Windows job) also passes. This is not "Windows is broken" — it's these specific tests, on this specific job, and the job's own scope makes the failure invisible to normal review.

Why this is worse than it looks: the job never runs where anyone would see it

# .github/workflows/ci.yml
cross-platform:
  name: Cross-Platform Tests
  if: github.ref == 'refs/heads/main' && github.event_name == 'push'

This job only runs on push to main. On pull_request events it renders as skipping in PR checks (confirmed on PRs #823, #825, #826, #828, #829, #830 today — all show Cross-Platform Tests: skipping). Reviewers never see it red. It only shows up if someone specifically goes looking at push-triggered workflow runs on main, which is not part of normal PR review.

Consequence: main has been carrying ~3 months of Windows-specific test failures with zero visibility at the point where merge decisions get made.

Cost already materialized: on 2026-08-15 a student in production could not activate Pro because of a bug specific to Windows npm invocation (npm.cmd via cmd.exe, broken path quoting — see #823 and #827). The one CI job with real per-platform Windows coverage had been blind for months by the time that shipped.

Failure history (push-triggered CI workflow, main branch)

Confirmed continuous failure of the Cross-Platform Tests (windows-latest, *) jobs specifically, going back to 2026-05-21. Representative recent runs, all main, all push:

Date	Run	Cross-Platform (windows-latest) result
2026-08-15 (today, x5)	e.g. 31900293948	failure (all 4 Node versions)
2026-08-10	—	failure
2026-07-13, 07-11 (x2), 07-10, 07-09 (x2)	—	failure
2026-05-21 (x6)	—	failure
2026-05-18	—	(last plausible green window; not confirmed at the job level — see "What I could not determine" below)

Root cause — determined for today's failures, with log evidence

Pulled the full job log for 95050203547 (Cross-Platform Tests (windows-latest, 20), run 31900293948) and read past the Jest console-warning noise to the actual summary and per-test failures:

Test Suites: 3 failed, 12 skipped, 374 passed, 377 of 389 total
Tests:       10 failed, 172 skipped, 9028 passed, 9210 total

Same 3 suites fail identically across all four Node versions (18/20/22/24) on windows-latest; zero failures on ubuntu-latest/macos-latest in the same run.

Cause A — CodeRabbit requires WSL on Windows, quality-gate tests don't account for it

tests/unit/quality-gates/layer2-pr-automation.test.js and tests/integration/quality-gate-pipeline.test.js — multiple assertions fail because on Windows CI runners, CodeRabbit invocation short-circuits with:

"CodeRabbit CLI requires WSL on Windows hosts. Install WSL via `wsl --install`
(https://learn.microsoft.com/windows/wsl/install), then install the CodeRabbit
CLI inside the WSL distribution. See docs/guides/installation-troubleshooting.md
Issue 10. To bypass this check, set coderabbit.installation_mode='native' in
your config."

The tests expect either a specific non-zero exit code (CodeRabbit CLI exited with code 137) or a 3-layer pipeline result (quality-gate-pipeline.test.js: expect(result.layers.length).toBe(3) → received 2, expect(result.exitCode).toBe(0) → received 1). Since windows-latest GitHub-hosted runners don't have WSL configured, CodeRabbit's Windows-specific guard fires immediately with the message above instead of the behavior the tests were written against — the tests were evidently written/validated against a Linux/macOS runner (or a Windows box with WSL already set up), not the bare windows-latest GHA image.

Cause B — separate: .grok managed-content checksum drift on Windows, likely .gitattributes line-ending gap

tests/unit/grok/grok-skills-sync.test.js › validates the committed .grok tree (strict) fails with result.ok === false. The test's own diagnostic dump shows "Managed content drift" for ~25 files, all .toml:

Managed content drift: config.toml
Managed content drift: personas/aiox-analyst.toml
Managed content drift: personas/aiox-architect.toml
... (22 more, all personas/*.toml and roles/*.toml)

Working hypothesis, not fully proven (flagging the distinction explicitly, per the standard we're holding this issue to): .gitattributes normalizes line endings (eol=lf) for .js .mjs .cjs .ts .jsx .tsx .json .md .yml .yaml .css .scss .html .xml .svg .txt .csv — .toml is not in that list. It's still covered by the blanket * text=auto, but without an explicit eol=lf override, Windows git checkouts are exposed to core.autocrlf behavior converting LF→CRLF on checkout. If the "managed content" checksums the validator compares against were computed from LF-normalized source, a CRLF checkout on windows-latest would produce a byte-level mismatch on every .toml file — matching the blanket, uniform drift seen across ~25 unrelated files. I have not done a byte-level diff of an actual checked-out .toml file on a Windows runner to confirm this with certainty; treat it as the most likely explanation given the evidence, not a proven root cause.

Note this suite (grok-skills-sync.test.js) was only added 2026-08-10 (#822) — well after the 2026-05-21 onset of Cause A. It's a second, independent failure that landed into an already-red pipeline; nobody could tell, because the pipeline was already invisible-red for Cause A.

What I could not determine with reasonable effort

The exact commit/PR that first broke Cross-Platform Tests (windows-latest) around 2026-05-21 is not pinned down. The whole-workflow CI run conclusion (not the specific job) is noisy around that date — it shows a mix of success/failure on both sides of 2026-05-18/05-21 for unrelated reasons, and the Cross-Platform Tests job-level history doesn't cleanly resolve via gh run view --json jobs for the older runs (job naming/structure may have changed since then). Bisecting job-level conclusions across 3 months of history to find the exact introducing commit is a real effort beyond what's reasonable for this pass — flagging it as undetermined rather than guessing. Cause A (CodeRabbit/WSL) is a strong candidate for the original regression given the message text explicitly names a Windows-specific guard, but I can't confirm whether that guard existed since before 05-21 and something else changed runner behavior, or whether the guard itself was added around then.

Suggested remediation (options, not prescribing)

• Run Cross-Platform Tests on pull_request too (at least for windows-latest), or scope it to PRs touching platform-sensitive paths (packages/installer/, .github/workflows/, anything invoking subprocesses) if running it on every PR is too expensive — the point is: it must be visible somewhere a merge decision gets made, not exclusively on push to main.
• Cause A: either install/configure WSL on the windows-latest runner image before this job's npm test step, or make the CodeRabbit-dependent tests skip/mock on Windows explicitly (the same way pro-setup-auth.test.js's machine-id suite already conditionally skips when a prerequisite isn't available) instead of asserting behavior that requires an environment the runner doesn't have.
• Cause B: add .toml (and audit for other extensions used by managed-content systems) to .gitattributes' eol=lf list if the hypothesis above holds; verify with an actual Windows checkout + hash comparison before merging that fix.
• Process: treat main red on push as a condition to fix, not a background state — e.g. a scheduled check or Slack/issue-bot that opens automatically after N consecutive red push runs on a specific job, so the next 3-month silent failure doesn't happen again.

Related

• Release chain silently skips publishing aiox-core: lockstep version sync is never persisted to main #827 — documents failures in the release pipeline (the package doesn't reach the registry correctly). This issue is about the product CI silently failing to catch a platform-specific regression before it reached users — different failure class, same underlying theme of "green run, real problem hidden inside."
• fix(installer): surface real npm errors and stop leaking reg.exe stderr in Pro install #823 — the PR whose Windows-specific npm-invocation bug this same job should have had a chance to catch, had it been visible.