Commit 54735f9
authored
test(sdk): stop re-transforming built sibling packages, instead of warming the cost (#2946)
* test(sdk): warm every lazy namespace import once, instead of per file
The SDK loads five namespace implementations with a dynamic import() on
first use, so whichever test runs first pays the whole cold-import cost
inside its own 5000ms budget. That has now been fixed three times, once
per namespace, each time by warming that one file: ids.test.ts
(3a00b5e), list.test.ts (#2935, after five unrelated PRs failed at
once between 5005ms and 5056ms and main went red on it), and
parquet-geometry.test.ts (#2248) for the same shape elsewhere.
Replace the per-file patches with one setup file covering all six
lazily-imported packages, so drawing, bcf and sandbox cannot become the
fourth instance.
Measured, warm cache, running list.test.ts and ids.test.ts together:
without setup cold tests 469ms and 491ms, setup 0ms, tests 968ms
with setup cold tests 2ms and 2ms, setup 1.20s, tests 8ms
The cost moves out of every test's budget rather than being absorbed by
a larger one. Deliberately NOT raising testTimeout, which is how
packages/data handles its own cold-transform cost: a blanket raise buys
the same green at the price of every genuine hang in this package taking
the new timeout to report.
allSettled rather than all, because @ifc-lite/sandbox is deliberately not
a dependency of the SDK and its import rejects here exactly as it does
for a consumer who has not installed it. The warm-up is an optimisation;
every test still passes without it, just closer to the timeout.
* test(sdk): make the warm-up list check itself against the source
Review of the previous commit found the arrangement had no way to notice
its own drift. The list of lazily-imported packages is maintained by hand
against src/namespaces, so a seventh dynamic import added later gets no
warm-up and no signal -- the failure surfaces months on, as a flake under
CI load, in whichever unrelated test happened to run first. An un-warmed
package is indistinguishable from a warmed one until that happens.
Add a test that reads both sides as text and compares them, in both
directions: every specifier src imports lazily must be warmed, and every
entry in the list must still be imported lazily. Deliberately not by
importing vitest.setup.ts, which would execute the warm-up and would let
a rename keep the test green while the list no longer matched src.
It carries its own control: if the loader idiom is ever refactored away,
the scan finds nothing and both comparisons pass vacuously against two
empty sets, so a third assertion fails loudly instead.
Mutation-verified, each applied from a snapshot and asserted to have
landed before the run:
new lazy import in src, not added to the list -> FAILS
package removed from the list -> FAILS
stale entry left in the list -> FAILS
Also from review: repoint ids.test.ts's comment, which still described a
warm-up that this change had deleted from that file; state in the setup
docblock that the ten files touching none of these packages now pay for
it too (types.test.ts 76ms -> 680ms, full suite at --maxWorkers=2 1.61s
-> 2.26s), since the tradeoff against testTimeout is not honest without
it; and record that @ifc-lite/sandbox can never resolve here because it
depends on @ifc-lite/sdk, so its rejection is inert by construction
rather than a missing dependency somebody should fix.
* test(sdk): make the warm-up drift check a lint, not a test
The drift check landed as packages/sdk/src/namespaces/lazy-imports.test.ts,
which reads two source files and compares their text. The repo bans that
shape and gates on it: AGENTS.md "Never assert on a source file's text",
enforced by scripts/check-source-text-assertions.mjs at test.yml:415. Run
against the previous commit it exits 1, so the branch was red as it stood.
The gate is right. A claim that one file's list matches another file's
declarations can only be made by reading both sources, and that is exactly
the shape that certifies a string exists rather than that the code works.
check-clash-degenerate-reason-parity.mjs was moved out of a test file for
the same reason and its header says so; this follows it, including the
--root flag and the vacuity guard.
Two things the move buys beyond satisfying the gate:
- comments are now stripped from both sides symmetrically, so a package
named only in prose cannot stand in for a real one. Both files discuss
these package names at length, so that was a live hole.
- the checker gets a regression harness, which is how the sibling checks
prove they can fire. Seven cases against mutated copies of the real
sources in a temp tree, each mutation anchor asserted present in the
real input first so a drifted anchor fails loudly instead of quietly
testing nothing: unwarmed new import, package dropped, stale entry,
loader idiom refactored away (the vacuity mode -- two empty sets
"agree"), and a package named only in a comment.
Wired as two steps of the CI node-test job, matching the sibling pair.
* test(sdk): anchor the warm-up scan on the array, trim the restated prose
From cleanup review, each item verified rather than taken on the claim:
- warmedPackages anchored on the array literal via one regex instead of
indexOf plus a fragile indexOf('];'). The reviewer tested the naive
whole-file alternative and it is wrong: a straight-quoted specifier in
prose satisfies it, and a renamed constant leaves it returning all six
while the list it names no longer exists. Anchoring makes a rename
yield the empty set, so every package reads as unwarmed and it fails
loudly. Pinned as an eighth harness case.
- the cold-import justification was written five times across four
files. It now lives once, in vitest.setup.ts. vitest.config.ts and
ids.test.ts point at it, and the checker header keeps only what is
specific to checking rather than to warming.
- the three-fix history compressed to its evidence (3a00b5e, #2935,
#2248). "This replaces the per-file patches" was already said by the
diff deleting them.
Kept, against the finding: the `import(` guard in the source scan is
inert on today's tree. It states the contract that only DYNAMIC imports
cost anything to warm, so a future static-import binding is not demanded
here. The comment now says it is inert and why it stays.
* test(sdk): stop re-transforming built sibling packages
The cold-import cost this branch was warming is not module loading. pnpm
links @ifc-lite/* as symlinks, so the specifier resolves to
packages/<name>/dist/index.js -- a real path inside the project root --
and vite treats that built file as SOURCE and runs it through its SSR
transform on every run. Node imports the same files in 12-40ms.
It surfaced as a timeout rather than as slowness because the namespaces
load lazily, so the transform landed inside the 5000ms budget of
whichever test touched one first.
One config line removes it. Full sdk suite, --maxWorkers=2, 3 reps,
173/173 green in every column:
before 1.60s (transform 1.38s)
warming the imports 2.28s (transform 3.41s, setup 2.50s)
externalising built dist 0.61s (transform 129ms)
The two tests that flaked land at 29ms and 38ms with nothing warmed at
all. So this deletes the mechanism the earlier commits on this branch
built: vitest.setup.ts, the hand-maintained package list, the lint that
checked the list against the source, and that lint's regression harness.
The warm-up was a correct fix one level above the cost, and it made the
suite slower than doing nothing.
Verified rather than assumed, because this changes how every sdk test
resolves its dependencies: same dist file either way, so nothing about
what is under test changes; vi.mock('@ifc-lite/sandbox') still applies
(src/sandbox.test.ts, 4/4); and the siblings must be built, which turbo
test already guarantees via dependsOn ["^build"].
The same transform tax is paid by every package in this monorepo that
imports a sibling. Filed separately rather than widened into here.
* test(sdk): anchor the external pattern, cover pkg/, assert it took effect
Review findings on the previous commit, each checked against the tree
rather than taken on the claim:
- the pattern was an unanchored substring, so it would also match a
checkout living beneath some other packages/<x>/dist/ -- and then it
matches this package's own src/*.ts, which Node cannot execute. Now
anchored at the repo root.
- it covered dist/ only, missing @ifc-lite/wasm, which builds to
packages/wasm/pkg/ifc-lite.js (turbo build outputs are dist/** and
pkg/**) and is reachable here via @ifc-lite/clash/wasm. A large
generated bundle is exactly what must not be re-transformed inside a
lazy import's budget.
- the comment cited turbo's test task as dependsOn ["^build"]. It is
["build"], with build itself ["^build"], so the guarantee holds
transitively but the citation was wrong. AGENTS.md line 13 carries the
same inaccuracy; not touched here.
- "nothing about what is under test changes" was false in one respect:
externalised modules are native ESM, so vi.spyOn on a sibling's
namespace now throws "Module namespace is not configurable". vi.mock
is unaffected. Nothing spies that way today; the comment now names it
rather than overstating.
And the finding that mattered most: nothing made a regression visible.
The warm-ups were self-evidencing, sitting in the file they protected;
a config is action-at-a-distance, and if it stops applying the tests do
not fail -- they quietly go back to a ~2s transform inside a 5s budget
and the flake returns on somebody else's PR.
So it is asserted, on the shape of the module namespace rather than on a
timing that would be flaky by construction: a natively imported ES module
namespace is sealed, a vite-built one is an ordinary object. Measured
both ways -- configurable/extensible false with the rule, true without.
Mutation-verified:
external rule deleted -> FAILS
deps.inline overrides it (path-shaped) -> FAILS
deps.inline: [/@ifc-lite/] -> passes, correctly:
inline matches resolved paths too, so that pattern overrides
nothing. The mutation never applied; it is not a surviving guard.
* test(sdk): cover every heavy sibling, not the lightest one
The guard added in the previous commit sampled @ifc-lite/lists alone.
That is the SMALLEST of the eight packages externalised here at 168K;
the cost is @ifc-lite/data at 9.8M and @ifc-lite/parser at 5.5M.
Adding `inline: [/packages\/(data|parser)\/dist\//]` -- precisely the
"later inline rule added for an unrelated reason" the guard's own
comment names as the threat -- left it green while suite transform went
120ms to 944ms. The safety net had a hole in the exact shape of the
threat it documents, which is the failure mode it exists to prevent.
Now it.each over the five that matter. Re-run against the same
mutation: 2 tests fail, transform 173ms -> 1.23s, so the condition is
observable where before it was not.
Two more from the same review:
- the discriminator is stated as "today's vite" rather than as a
definition. Measured, the ONLY differing bit between a native and a
transformed namespace is sealed-ness; proto and Symbol.toStringTag are
identical in both. So if vite ever seals its namespaces this goes
green while inlining, and per-package coverage is the real belt.
- repoRoot is normalised to forward slashes. On win32 fileURLToPath
yields C:\repo\ while the pattern tail and the ids vitest matches
against use /, so the regex could never match and externalisation
would silently vanish.
The `pkg` half of (dist|pkg) is now labelled UNTESTED insurance rather
than left to read as covered: instrumenting the pattern over the whole
suite shows eight ids submitted, every one a dist/index.js, none under
pkg. It stays for @ifc-lite/wasm, which is reachable via
@ifc-lite/clash/wasm and is exactly the kind of large generated bundle
that must not be re-transformed inside a lazy import's budget.1 parent 989ee2c commit 54735f9
4 files changed
Lines changed: 148 additions & 50 deletions
File tree
- packages/sdk
- src
- namespaces
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | | - | |
17 | | - | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
27 | 20 | | |
28 | 21 | | |
29 | 22 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | 25 | | |
64 | 26 | | |
65 | 27 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
0 commit comments