Fix flaky RB SUnit integration tests: 60s timeout + deterministic run-limit guard - #403
Fix flaky RB SUnit integration tests: 60s timeout + deterministic run-limit guard#403ericwinger wants to merge 2 commits into
Conversation
The MAX_RUN_CLASSES guard (runFailingTests) was only "tested" by an on-demand gci smoke test that branched on the live image's size and, on the under-cap branch, ran every discovered suite in one ~46s un-interruptible GCI call — too slow for CI (vitest's 5s timeout can't kill it), so it was quarantined out of CI and never actually ran. Replace it with two fast, deterministic integration tests (default project → run in CI) that trip the guard via a bounded fixture of throwaway TestCase subclasses with NO test methods, so the guard fires BEFORE any suite runs (nothing blocks): - delete client/src/__tests__/gci/querySunitRunLimit.smoke.test.ts - add installGuardProbeClasses + SUNIT_GUARD_PROBE_PATTERN to sunitProbeFixture.ts - add 'refuses an oversized run' (limit+1 -> throws the guard) and 'at the cap boundary' (limit -> no throw) to querySunit.integration.test.ts; drop the stale "parked in gci" note Transient (useIntegrationTest per-test abort; nothing committed to the stone). Same pattern as the already-merged 066c24d (gci -> CI integration migration). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… retry-flake) The ~15 refactoring *.integration.test.ts each file in the (growing) engine-tests.gs payload and run an engine GS SUnit suite in-stone. Alone that's ~3s, but under full-`npm test` contention it's 8-11s — brushing vitest's 5s default timeout, so those tests fail on the first run and pass on retry (the classic flake). Three files already carried an explicit timeout (split/extract-superclass 60_000, change-signature 30_000) and did not flake; the rest ran on the 5s default. Give every in-stone suite-run `it` a uniform 60s timeout (~6x the worst observed contended run), matching the existing 60_000 convention. No production code, no logic change — purely the timeout argument. (change-signature normalized 30_000 -> 60_000 for uniformity.) A cleaner long-term fix — running the engine SUnit ONCE instead of every refactoring re-filing the payload and running its own suite — is left as a follow-up (it needs a stone to verify the consolidated discovery/run, which this environment can't provide). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
b0d9747 to
4ec7bd6
Compare
|
Data point from hitting this repeatedly over the last couple of days, in case it's useful while Mode 1 — the timeout. Exactly as described above: the in-stone suite run brushes vitest's 5s Mode 2 — session exhaustion. Separately, the integration tests fail at login, before any A longer timeout can't help this one — the login fails immediately, not slowly. It shows up as Workaround that reliably clears it: cap vitest's worker count for the run, which reduces env -u GEMSTONE_NRS_ALL VITEST_MAX_WORKERS=3 VITEST_MIN_WORKERS=1 npm test
# same for the pre-push gate:
env -u GEMSTONE_NRS_ALL VITEST_MAX_WORKERS=3 VITEST_MIN_WORKERS=1 git push origin <branch>Without it I saw 1, 2, 4 and 7 files fail across different full runs; with it, repeated full runs were green (client 351 Not suggesting this PR should grow to cover mode 2 — the timeout fix stands on its own and is |
Two fixes for the flaky RB SUnit integration tests, on one branch.
1. The retry-flake (the main one) — timeout bump
The ~15 refactoring
*.integration.test.tseach file in the (growing)engine-tests.gspayload andrun an engine GS SUnit suite in-stone. Alone that's ~3s, but under full-
npm testcontention it's8–11s — which brushes vitest's 5s default timeout, so those tests fail on the first run and pass
on retry (the classic flake). Three files already carried an explicit timeout
(split / extract-superclass
60_000, change-signature30_000) and did not flake; the rest ranon the 5s default.
Give every in-stone suite-run
ita uniform 60s timeout (~6× the worst observed contendedrun), matching the existing
60_000convention. Purely the timeout argument — no production code, nologic change. (13 files, 15 line changes; change-signature normalized
30_000 → 60_000.) A scanconfirms no in-stone suite-run is left on the default, including the easy-to-miss ones in
refactoringMethod(two suite tests) andrefactoringPushMethod(push-up + push-down).2. The run-limit guard test — deterministic + CI-runnable
runFailingTests'MAX_RUN_CLASSESguard was only "tested" by an on-demand gci smoke test thatbranched on the live image's size and, on the under-cap branch, ran every discovered suite in one
~46s un-interruptible GCI call — too slow for CI, so it was quarantined and never actually ran.
Replaced with two fast, deterministic integration tests (default project → run in CI) that trip the
guard via a bounded fixture of throwaway
TestCasesubclasses with no test methods, so the guardfires before any suite runs (nothing blocks):
limit + 1 →throws the guard;limit →doesn't.Transient (
useIntegrationTestper-test abort). Mirrors the already-merged066c24d0(gci → CIintegration migration).
Follow-up (not in this PR)
The cleaner long-term fix — run the engine SUnit once instead of every refactoring re-filing the
payload and running its own suite — is deferred; it needs a stone to verify the consolidated
discovery/run and is better done as its own change.
Verification
Full
npm testgreen via the pre-push hook against a freshly-provisioned 3.7.5 stone with the RBplugin installed (so the gated integration tests exercised, not skipped): client + server (322) +
mcp (92). CI will confirm across the version matrix.
🤖 Generated with Claude Code