Skip to content

Commit 5d93d25

Browse files
authored
fix(bench): discard warmup runs in query benchmark median (#1077)
* fix(bench): discard warmup runs in query benchmark median (#1076) The native fn_deps NAPI path pays a cold-start cost on the first 2-3 calls per worker process — rusqlite statement-cache warmup, OS page cache fill for the DB file, and NAPI-side static init from tree-sitter 0.25's transitive crates (indexmap, hashbrown, equivalent) linked into the .node binary at #1054. The benchmark's median(5) sample then lands inside cold-start territory rather than warm steady-state, which is what the regression gate is supposed to track. On Linux x86_64 CI that surfaced as fnDeps depth 1 native: 28.7 → 48.6 (+69%) once #1054 enlarged the binary's init footprint, even though warm per-call latency is unchanged from v3.9.6. Run two warmup iterations per (depth, fn) before timing so all 5 sampled timings are warm. Verified on Linux x86_64 (Docker, qemu): native d1 178.2ms → 119.6ms, now matching the warm d3/d5 plateau (122/113ms). Wasm path is unaffected by the change since wasm only ever paid cold-start on its first iteration. Closes #1076 * fix(bench): bump WARMUP_RUNS from 2 to 3 to match documented worst-case cold-start (#1077)
1 parent 4d26f08 commit 5d93d25

1 file changed

Lines changed: 12 additions & 0 deletions

File tree

scripts/query-benchmark.ts

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,15 @@ console.log = (...args) => console.error(...args);
108108

109109
const RUNS = 5;
110110

111+
// First 2-3 native fnDeps calls per process pay a cold-start cost (rusqlite
112+
// statement-cache warmup, OS page cache for the DB file, NAPI-side static
113+
// init from tree-sitter's transitive crates linked into the .node binary).
114+
// On Linux x86_64 CI, that pulled median(5) into cold-start territory once
115+
// tree-sitter 0.25 grew the binary's init footprint (#1076), even though
116+
// steady-state per-call latency is unchanged. Discard the first WARMUP_RUNS
117+
// before timing so the metric reflects warm-call latency, not cold-start.
118+
const WARMUP_RUNS = 3;
119+
111120
function median(arr) {
112121
const sorted = [...arr].sort((a, b) => a - b);
113122
const mid = Math.floor(sorted.length / 2);
@@ -172,6 +181,9 @@ function selectTargets() {
172181
function benchDepths(fn, name, depths) {
173182
const result = {};
174183
for (const depth of depths) {
184+
for (let i = 0; i < WARMUP_RUNS; i++) {
185+
fn(name, dbPath, { depth, noTests: true });
186+
}
175187
const timings = [];
176188
for (let i = 0; i < RUNS; i++) {
177189
const start = performance.now();

0 commit comments

Comments
 (0)