Skip to content

Commit 3985216

Browse files
chaxusclaude
andcommitted
ci(nightly): two cadences, a gate for quiet nights, and the @serial split
The nightly had been red every morning for over a month with almost the same list each time, which is a nightly nobody reads -- and a shipped Safari defect sat inside it the whole while. Three changes so a red morning means something again. The corpus matrix moves to Sundays. It is 80 minutes against a corpus that never changes; what varies is this repository, and a week of it at a time keeps the findings table current. The cross-browser sweep stays nightly: it is this repository's only WebKit/Firefox coverage, the gate being Chromium throughout. Neither runs when nothing was committed since the last time it did -- `schedule` fires whether or not there is anything new to say. The gate job itself does run every night, cheaply: a quiet night should still leave a run explaining why it was quiet, because a workflow that simply stops appearing looks exactly like one that broke. Its two decisions are written as `if` blocks rather than `a || b && c`, which evaluates to non-zero when both tests fail and would end a quiet night as a failed gate under `bash -e`. The cross-browser job also gains the @serial split ci.yml's e2e job already has, and needed it more. sw-silent-update.spec.ts stands in for a deploy by rewriting the served dist/sw.js -- visible to the whole origin, which its own header says -- so run beside others it replaces THEIR worker mid-test. That is where sw-warm read a vendor stamp belonging to it, and where the cache-first probe lost the entry it had just planted: the last two flakes in the suite, and they go away when the pass is split. Both halves are pinned, since dropping the `--grep-invert` puts the hazard back and dropping the second pass means nothing runs those cases at all while the job stays green. Reverse-verified: each pin fails on its own when the half it guards is removed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent d7266c2 commit 3985216

3 files changed

Lines changed: 129 additions & 7 deletions

File tree

‎.github/workflows/nightly-corpus.yml‎

Lines changed: 84 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,19 @@
1-
# Nightly real-document regression matrix (docs/superpowers/plans/2026-08-15-v9-test-coverage-strategy.md,
2-
# section 2 tier 2 / section 6). Pulls a public corpus of real Office files
3-
# (never committed here), drives the real editor through open -> edit -> save
4-
# for each, and publishes the findings table. Red is a signal, not a gate.
1+
# What the pull-request gate does not cover, run on a clock instead of on a
2+
# diff: the PR-time specs on the two engines the gate never launches
3+
# (playwright.browsers.config.ts), the interaction sweeps and throttled-network
4+
# budgets, and the real-document matrix over a public corpus of real Office
5+
# files (docs/superpowers/plans/2026-08-15-v9-test-coverage-strategy.md, section
6+
# 2 tier 2 / section 6). Red here is a signal, not a gate.
7+
#
8+
# Two cadences, one nightly trigger (see the `gate` job):
9+
# - cross-browser + sweeps run nightly. They are this repository's ONLY
10+
# WebKit/Firefox coverage, and on 2026-08-25 that is what surfaced a
11+
# shipped Safari defect the Chromium gate cannot see.
12+
# - the corpus matrix runs on Sundays. It is 80 minutes against a corpus that
13+
# never changes; what varies is this repository, and a week of it at a time
14+
# is enough to keep the findings table current.
15+
# Neither runs when nothing has been committed since the last time it did --
16+
# `schedule` fires whether or not there is anything new to say.
517
name: Nightly corpus
618

719
on:
@@ -20,10 +32,63 @@ on:
2032
default: "1"
2133

2234
jobs:
35+
# Which of tonight's jobs have anything to say. Cheap (one checkout, no
36+
# toolchain) and it runs every night, so the run itself is the record of why
37+
# a quiet night was quiet -- a workflow that simply stops appearing is
38+
# indistinguishable from one that broke.
39+
gate:
40+
name: What to run tonight
41+
runs-on: ubuntu-latest
42+
timeout-minutes: 5
43+
outputs:
44+
browsers: ${{ steps.decide.outputs.browsers }}
45+
corpus: ${{ steps.decide.outputs.corpus }}
46+
47+
steps:
48+
- name: Checkout code
49+
uses: actions/checkout@v7
50+
51+
- name: Decide
52+
id: decide
53+
env:
54+
MANUAL: ${{ github.event_name == 'workflow_dispatch' }}
55+
run: |
56+
age_hours=$(( ( $(date -u +%s) - $(git log -1 --format=%ct) ) / 3600 ))
57+
# 1..7 with 7 = Sunday. The cron fires at 19:17 UTC, so this is the
58+
# UTC weekday of the trigger, not of the Asia/Shanghai morning it
59+
# lands in -- which is what the comment above the crons means.
60+
weekday=$(date -u +%u)
61+
62+
browsers=false
63+
corpus=false
64+
# A cycle plus an hour of slack on each window: a commit landing just
65+
# before the run it belongs to would otherwise wait a whole cycle.
66+
# Written as `if` blocks rather than `a || b && c` -- that list
67+
# evaluates to non-zero when both tests fail, and the step runs under
68+
# `bash -e`, so a quiet night would end as a failed gate.
69+
if [ "$MANUAL" = "true" ] || [ "$age_hours" -lt 25 ]; then
70+
browsers=true
71+
fi
72+
if [ "$MANUAL" = "true" ] || { [ "$weekday" -eq 7 ] && [ "$age_hours" -lt 193 ]; }; then
73+
corpus=true
74+
fi
75+
76+
{
77+
echo "HEAD is ${age_hours}h old, UTC weekday ${weekday}, manual=${MANUAL}"
78+
echo ""
79+
echo "- cross-browser + sweeps: ${browsers}"
80+
echo "- corpus matrix: ${corpus}"
81+
} >> "$GITHUB_STEP_SUMMARY"
82+
83+
echo "browsers=${browsers}" >> "$GITHUB_OUTPUT"
84+
echo "corpus=${corpus}" >> "$GITHUB_OUTPUT"
85+
2386
corpus:
2487
name: Real-document matrix (Apache POI test-data)
2588
runs-on: ubuntu-latest
2689
timeout-minutes: 300
90+
needs: gate
91+
if: needs.gate.outputs.corpus == 'true'
2792

2893
steps:
2994
- name: Checkout code
@@ -83,6 +148,8 @@ jobs:
83148
name: Cross-browser (WebKit + Firefox)
84149
runs-on: ubuntu-latest
85150
timeout-minutes: 90
151+
needs: gate
152+
if: needs.gate.outputs.browsers == 'true'
86153

87154
steps:
88155
- name: Checkout code
@@ -96,7 +163,17 @@ jobs:
96163
# The same PR-time suites (minus the opt-in sweeps) on the two engines
97164
# the gate does not cover. Same-realm/L0 rules apply unchanged.
98165
- name: Run E2E on WebKit and Firefox
99-
run: pnpm exec playwright test -c playwright.browsers.config.ts --grep-invert "api surface|corpus" --reporter=list
166+
run: pnpm exec playwright test -c playwright.browsers.config.ts --grep-invert "api surface|corpus|@serial" --reporter=list
167+
168+
# The other half of that `--grep-invert`, exactly as ci.yml's e2e job
169+
# pairs them: an @serial case is one that cannot share the run. Without
170+
# this split sw-silent-update.spec.ts rewrites the served sw.js while
171+
# other specs are being served from it, replacing THEIR worker mid-test
172+
# -- which is where sw-warm's "runtime-e2e-next" and cache-first's
173+
# missing sentinel came from on 2026-08-25. Without the second pass
174+
# nothing runs them at all and the job stays green regardless.
175+
- name: Run the cases that cannot share the run
176+
run: pnpm exec playwright test -c playwright.browsers.config.ts --grep @serial --workers=1 --reporter=list
100177

101178
- name: Upload report
102179
if: always()
@@ -110,6 +187,8 @@ jobs:
110187
name: Slow-network budgets
111188
runs-on: ubuntu-latest
112189
timeout-minutes: 30
190+
needs: gate
191+
if: needs.gate.outputs.browsers == 'true'
113192

114193
steps:
115194
- name: Checkout code

‎CLAUDE.md‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -224,9 +224,10 @@ test/setup/vitest.ts # 全局 mock:matchMedia、URL.createObjectURL
224224
`CORPUS_LIMIT=<上限,截断会打印>` / `CORPUS_VISUAL=1`(L3:原始 vs 存回再
225225
打开的渲染逐像素比对);每条用例结束即追加一行到
226226
`test-results/corpus-rows-<worker>.jsonl`,`node bin/corpus-report.mjs [dir]`
227-
合并出 JSON + markdown 表(含 L2 内容、L3 视觉、open/save p50/p95);**夜间 CI** `.github/workflows/nightly-corpus.yml`
227+
合并出 JSON + markdown 表(含 L2 内容、L3 视觉、open/save p50/p95);**CI 每周日**
228+
由 `.github/workflows/nightly-corpus.yml`
228229
拉 Apache POI test-data 公开语料跑同一套(红=信号非门禁,可
229-
workflow_dispatch 指定 limit/filter);对目录下每个 docx/doc/xlsx/xls/pptx/ppt/csv 走
230+
workflow_dispatch 随时手动触发并指定 limit/filter);对目录下每个 docx/doc/xlsx/xls/pptx/ppt/csv 走
230231
打开 → 监听致命弹窗/asc_onError → 编辑 → 保存 → 产物 sanity,输出汇总
231232
报告。语料留在测试机上不入库;未设 `CORPUS_DIR` 整套 skip,CI 保持绿。
232233
第一天就抓出 P0(非 ASCII 文件名导致 -82 打开失败 + 永久转圈),见
@@ -251,6 +252,19 @@ test/setup/vitest.ts # 全局 mock:matchMedia、URL.createObjectURL
251252
(WebKit + Firefox,夜间)、`playwright.prod.config.ts`(打线上/preview,
252253
`prod-smoke.yml` 与 PR 门禁 `preview-smoke.yml` 用)。
253254

255+
**夜间那个 workflow 有两档节奏,且没有新提交就整晚不跑**(`nightly-corpus.yml`
256+
的 `gate` job,2026-08-25 起):cross-browser + sweeps **每天**(它是本仓库唯一
257+
的 WebKit/Firefox 覆盖,PR 门禁全是 Chromium),corpus 矩阵**每周日**(80 分钟,
258+
而语料是静态的,变的是本仓库);`workflow_dispatch` 无视这两个判断。判据是
259+
HEAD 的提交时间,窗口各留一个周期加一小时的余量。**gate 本身每晚都跑**——安静
260+
的夜晚也要留下一条 run 说明为什么安静,否则"没有 run"和"workflow 坏了"长得一样。
261+
**cross-browser 那个 job 必须和 ci.yml 一样把 `@serial` 分成两趟**
262+
(`--grep-invert "api surface|corpus|@serial"` + `--grep @serial --workers=1`):
263+
`sw-silent-update.spec.ts` 靠改写被服务的 `dist/sw.js` 来模拟一次部署,那是**整个
264+
源站可见**的,跟别的 spec 并排跑会把它们的 worker 也换掉——2026-08-25 之前
265+
`sw-warm` 读到 `runtime-e2e-next`、`sw-vendor-cache-first` 找不到自己刚种下的
266+
哨兵,都是这个。两半必须成对,`workflow-contract.test.ts` 钉住。
267+
254268
**L0 全局 fixture(`test/e2e/lib/l0.ts`,2026-08-15 起所有 spec 从它
255269
导入 `test`/`expect`)**:自动把 `asc_onError`、厂商致命弹窗、编辑器 iframe
256270
内的 `unhandledrejection`/`error`、pageerror、非白名单 console.error 判为

‎test/unit/workflow-contract.test.ts‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -377,6 +377,35 @@ describe('.github/workflows/nightly-corpus.yml', () => {
377377
expect(readFileSync(resolve(ROOT, 'test/e2e/lib/l0.ts'), 'utf8')).toMatch(/projectName: \[/);
378378
});
379379

380+
it('keeps the cases that cannot share the run out of the parallel pass, and still runs them', () => {
381+
// The same pairing ci.yml's e2e job has, for the same reason and with the
382+
// same failure mode when either half goes missing. Here the case that
383+
// needs it is sw-silent-update.spec.ts: it rewrites the served sw.js to
384+
// stand in for a deploy, which replaces the worker under every other spec
385+
// on the origin -- sw-warm read a vendor stamp that belonged to it, and
386+
// the cache-first probe lost the entry it had just planted.
387+
expect(src).toMatch(/--grep-invert "api surface\|corpus\|@serial"/);
388+
expect(src).toMatch(/-c playwright\.browsers\.config\.ts --grep @serial --workers=1/);
389+
});
390+
391+
it('runs each half of the night only when it has something to say', () => {
392+
// `schedule` fires whether or not anything was committed, and the corpus
393+
// is static: what varies is this repository. Both windows are decided in
394+
// one cheap gate job so a quiet night still leaves a run explaining itself.
395+
const jobs = parseJobs(src);
396+
expect(jobs.map(({ name }) => name)).toContain('gate');
397+
for (const name of ['browsers', 'budgets']) {
398+
const job = jobs.find((candidate) => candidate.name === name);
399+
expect(job, name).toBeDefined();
400+
expect(job!.body, name).toMatch(/^ {4}if: needs\.gate\.outputs\.browsers == 'true'$/m);
401+
expect(job!.body, name).toMatch(/^ {4}needs: gate$/m);
402+
}
403+
const corpus = jobs.find(({ name }) => name === 'corpus');
404+
expect(corpus!.body).toMatch(/^ {4}if: needs\.gate\.outputs\.corpus == 'true'$/m);
405+
// Manual runs always go through, whatever the clock says.
406+
expect(src).toMatch(/MANUAL: \$\{\{ github\.event_name == 'workflow_dispatch' \}\}/);
407+
});
408+
380409
it('drops the corpus fuzzer output without dropping real bug-report files', () => {
381410
// Byte soup minimized by a fuzzer until it broke a parser: "this editor
382411
// will not open it either" is not a finding, and 15 such files were red on

0 commit comments

Comments
 (0)