Skip to content

perf(bench): exclude resolution-benchmark fixtures from dogfooding sweep #1134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
carlos-alm merged 7 commits into from
May 17, 2026
Merged

perf(bench): exclude resolution-benchmark fixtures from dogfooding sweep #1134

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5f96cac
perf(bench): exclude resolution-benchmark fixtures from dogfooding sweep
carlos-alm May 15, 2026
b008ffb
test(exclude): add incremental round-trip tests for opts.exclude (#1134)
carlos-alm May 15, 2026
e2346e0
fix(bench): exempt 3.10.0:DB bytes/file from regression guard (#1134)
carlos-alm May 15, 2026
8cc86cf
fix: resolve merge conflicts with main
carlos-alm May 17, 2026
d5b354a
fix: remove duplicate exclude field from BuildGraphOpts (#1134)
carlos-alm May 17, 2026
e6bf5a1
fix: skip BENCHMARK_EXCLUDES in --npm mode for corpus parity (#1134)
carlos-alm May 17, 2026
10c1812
Merge branch 'main' into perf/1112-exclude-bench-fixtures
carlos-alm May 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions scripts/benchmark.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import path from 'node:path';
import { performance } from 'node:perf_hooks';
import { fileURLToPath } from 'node:url';
import Database from 'better-sqlite3';
import { resolveBenchmarkSource, srcImport } from './lib/bench-config.js';
import { BENCHMARK_EXCLUDES, resolveBenchmarkSource, srcImport } from './lib/bench-config.js';
import { isWorker, workerEngine, workerTargets, forkEngines } from './lib/fork-engine.js';

// ── Parent process: fork one child per engine, assemble final output ─────
Expand Down Expand Up @@ -130,7 +130,7 @@ console.log = (...args) => console.error(...args);
if (fs.existsSync(dbPath)) fs.unlinkSync(dbPath);

const buildStart = performance.now();
const buildResult = await buildGraph(root, { engine, incremental: false });
const buildResult = await buildGraph(root, { engine, incremental: false, exclude: [...BENCHMARK_EXCLUDES] });
const buildTimeMs = performance.now() - buildStart;

const queryStart = performance.now();
Expand All @@ -150,7 +150,7 @@ try {
const noopTimings = [];
for (let i = 0; i < INCREMENTAL_RUNS; i++) {
const start = performance.now();
await buildGraph(root, { engine, incremental: true });
await buildGraph(root, { engine, incremental: true, exclude: [...BENCHMARK_EXCLUDES] });
noopTimings.push(performance.now() - start);
}
noopRebuildMs = Math.round(median(noopTimings));
Expand All @@ -167,7 +167,7 @@ try {
for (let i = 0; i < INCREMENTAL_RUNS; i++) {
fs.writeFileSync(PROBE_FILE, original + `\n// probe-${i}\n`);
const start = performance.now();
const res = await buildGraph(root, { engine, incremental: true });
const res = await buildGraph(root, { engine, incremental: true, exclude: [...BENCHMARK_EXCLUDES] });
oneFileRuns.push({ ms: performance.now() - start, phases: res?.phases || null });
}
oneFileRuns.sort((a, b) => a.ms - b.ms);
Expand All @@ -179,7 +179,7 @@ try {
} finally {
fs.writeFileSync(PROBE_FILE, original);
try {
await buildGraph(root, { engine, incremental: true });
await buildGraph(root, { engine, incremental: true, exclude: [...BENCHMARK_EXCLUDES] });
} catch {
// Cleanup rebuild failed — probe file is already restored, move on
}
Expand Down
14 changes: 7 additions & 7 deletions scripts/incremental-benchmark.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import fs from 'node:fs';
import path from 'node:path';
import { performance } from 'node:perf_hooks';
import { fileURLToPath } from 'node:url';
import { resolveBenchmarkSource, srcImport } from './lib/bench-config.js';
import { BENCHMARK_EXCLUDES, resolveBenchmarkSource, srcImport } from './lib/bench-config.js';
import { isWorker, workerEngine, forkEngines } from './lib/fork-engine.js';

// ── Parent process: fork one child per engine, assemble final output ─────
Expand Down Expand Up @@ -194,7 +194,7 @@ const fullTimings = [];
for (let i = 0; i < RUNS; i++) {
if (fs.existsSync(dbPath)) fs.unlinkSync(dbPath);
const start = performance.now();
await buildGraph(root, { engine, incremental: false });
await buildGraph(root, { engine, incremental: false, exclude: [...BENCHMARK_EXCLUDES] });
fullTimings.push(performance.now() - start);
}
const fullBuildMs = Math.round(median(fullTimings));
Expand All @@ -203,12 +203,12 @@ const fullBuildMs = Math.round(median(fullTimings));
let noopRebuildMs = null;
try {
for (let i = 0; i < WARMUP_RUNS; i++) {
await buildGraph(root, { engine, incremental: true });
await buildGraph(root, { engine, incremental: true, exclude: [...BENCHMARK_EXCLUDES] });
}
const noopTimings = [];
for (let i = 0; i < RUNS; i++) {
const start = performance.now();
await buildGraph(root, { engine, incremental: true });
await buildGraph(root, { engine, incremental: true, exclude: [...BENCHMARK_EXCLUDES] });
noopTimings.push(performance.now() - start);
}
noopRebuildMs = Math.round(median(noopTimings));
Expand All @@ -223,13 +223,13 @@ let oneFilePhases = null;
try {
for (let i = 0; i < WARMUP_RUNS; i++) {
fs.writeFileSync(PROBE_FILE, original + `\n// warmup-${i}\n`);
await buildGraph(root, { engine, incremental: true });
await buildGraph(root, { engine, incremental: true, exclude: [...BENCHMARK_EXCLUDES] });
}
const oneFileRuns = [];
for (let i = 0; i < RUNS; i++) {
fs.writeFileSync(PROBE_FILE, original + `\n// probe-${i}\n`);
const start = performance.now();
const res = await buildGraph(root, { engine, incremental: true });
const res = await buildGraph(root, { engine, incremental: true, exclude: [...BENCHMARK_EXCLUDES] });
oneFileRuns.push({ ms: performance.now() - start, phases: res?.phases || null });
}
oneFileRuns.sort((a, b) => a.ms - b.ms);
Expand All @@ -241,7 +241,7 @@ try {
} finally {
fs.writeFileSync(PROBE_FILE, original);
try {
await buildGraph(root, { engine, incremental: true });
await buildGraph(root, { engine, incremental: true, exclude: [...BENCHMARK_EXCLUDES] });
} catch {
// Cleanup rebuild failed — probe file is already restored, move on
}
Expand Down
15 changes: 15 additions & 0 deletions scripts/lib/bench-config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,21 @@ import { pathToFileURL } from 'node:url';

import { getBenchmarkVersion } from '../bench-version.js';

/**
* Globs excluded from every benchmark's `buildGraph(root, ...)` invocation.
*
* Resolution-benchmark fixtures (`tests/benchmarks/resolution/fixtures/**`)
* are hand-annotated scaffolding for the static-resolution test suite, not
* representative source code. They inflate dogfooding timing measurements
* disproportionately whenever a new-language PR lands a heavyweight grammar
* (e.g. tree-sitter-verilog added ~850ms to native fullBuildMs in #1107).
* Excluding them here keeps build/query/incremental benchmarks measuring
* codegraph's own source rather than its test fixtures.
*/
export const BENCHMARK_EXCLUDES: readonly string[] = [
'tests/benchmarks/resolution/fixtures/**',
];

// On Windows, `npm` is `npm.cmd` and Node refuses to spawn `.cmd`/`.bat`
// without `shell: true` (since Node 18.20 / 20.15). When `shell: true`, the
// Windows `cmd.exe` shell resolves bare `npm` to `npm.cmd` automatically, so
Expand Down
4 changes: 2 additions & 2 deletions scripts/query-benchmark.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import path from 'node:path';
import { performance } from 'node:perf_hooks';
import { fileURLToPath } from 'node:url';
import Database from 'better-sqlite3';
import { resolveBenchmarkSource, srcImport } from './lib/bench-config.js';
import { BENCHMARK_EXCLUDES, resolveBenchmarkSource, srcImport } from './lib/bench-config.js';
import { isWorker, workerEngine, workerTargets, forkEngines } from './lib/fork-engine.js';

// ── Parent process: fork one child per engine, assemble final output ─────
Expand Down Expand Up @@ -254,7 +254,7 @@ function benchDiffImpact(hubName) {

// Build graph for this engine
if (fs.existsSync(dbPath)) fs.unlinkSync(dbPath);
await buildGraph(root, { engine, incremental: false });
await buildGraph(root, { engine, incremental: false, exclude: [...BENCHMARK_EXCLUDES] });

const targets = workerTargets() || selectTargets();
console.error(`Targets: hub=${targets.hub}, mid=${targets.mid}, leaf=${targets.leaf}`);
Expand Down
7 changes: 7 additions & 0 deletions src/domain/graph/builder/pipeline.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,13 @@ function setupPipeline(ctx: PipelineContext): void {
initSchema(ctx.db);

ctx.config = loadConfig(ctx.rootDir);
// Merge caller-supplied excludes on top of the file-config excludes so
// programmatic callers (e.g. benchmark scripts) can extend exclusion
// without mutating .codegraphrc.json. Native orchestrator picks this up
// automatically — it reads exclude off the serialized ctx.config below.
if (ctx.opts.exclude?.length) {
ctx.config.exclude = [...(ctx.config.exclude ?? []), ...ctx.opts.exclude];
}
ctx.incremental =
ctx.opts.incremental !== false && ctx.config.build && ctx.config.build.incremental !== false;

Expand Down
7 changes: 7 additions & 0 deletions src/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1063,6 +1063,13 @@ export interface BuildGraphOpts {
complexity?: boolean;
cfg?: boolean;
scope?: string[];
/**
* Glob patterns merged on top of `config.exclude` for this build only.
* Lets callers extend exclusion programmatically without writing a config
* file — used by the benchmark scripts to skip resolution-benchmark
* fixtures that aren't representative of real code.
*/
exclude?: string[];
skipRegistry?: boolean;
}

Expand Down
13 changes: 0 additions & 13 deletions tests/benchmarks/regression-guard.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,18 +176,6 @@
* absolute delta 10.4ms exactly at the MIN_ABSOLUTE_DELTA floor. Exempt
* this release; remove once 3.11.0+ data confirms stabilization.
*
* - 3.10.0:Full build — adding native Verilog support (#1107) pulled the
* 4 `.v` resolution-benchmark fixtures into the corpus the incremental
* benchmark sweeps (it runs against the repo root). tree-sitter-verilog
* is a large grammar (SystemVerilog is one of the heaviest in the
* tree-sitter ecosystem) so each file costs noticeably more than the
* other fixture languages. Local measurement: 1959 → 2809 (+43%, run
* 25716010487). The cost is real and structural — not a regression in
* shared code paths. Resolution: either exclude `tests/benchmarks/
* resolution/fixtures/verilog/**` from the benchmark sweep or accept the
* one-time bump as the cost of supporting Verilog. Tracked separately;
* exempt this release.
*
* - 3.10.0:Query time — cumulative effect of adding two native extractors
* (Solidity #1100 + R #1102) in quick succession. Neither tripped the
* threshold individually (Solidity PR's Query time stayed at 49ms, R PR
Expand Down Expand Up @@ -230,7 +218,6 @@
'3.10.0:fnDeps depth 1',
'3.10.0:fnDeps depth 3',
'3.10.0:fnDeps depth 5',
'3.10.0:Full build',
'3.10.0:Query time',
]);

Expand Down Expand Up @@ -447,7 +434,7 @@
` ${r.label}: ${r.previous} → ${r.current} (+${Math.round(r.pctChange * 100)}%, threshold ${Math.round(thresholdFor(r.label) * 100)}%)`,
)
.join('\n');
expect.fail(`Benchmark regressions exceed threshold:\n${details}`);

Check failure on line 437 in tests/benchmarks/regression-guard.test.ts

View workflow job for this annotation

GitHub Actions / Pre-publish benchmark gate

tests/benchmarks/regression-guard.test.ts > Benchmark regression guard > build benchmarks > native engine — dev vs 3.10.0 

AssertionError: Benchmark regressions exceed threshold:
  DB bytes/file: 41614 → 52211 (+25%, threshold 25%)
 ❯ assertNoRegressions tests/benchmarks/regression-guard.test.ts:437:12
 ❯ tests/benchmarks/regression-guard.test.ts:577:9
}
}

Expand Down
33 changes: 33 additions & 0 deletions tests/integration/config-include-exclude.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,4 +178,37 @@ describe('config.include / config.exclude (issue #981)', () => {
// Paths are already relative to each run's own tmpDir so they compare directly.
expect(nativeFiles).toEqual(wasmFiles);
});

// ── opts.exclude (programmatic, no on-disk config) ───────────────

async function buildWithOptsExclude(
root: string,
engine: EngineName,
optsExclude: string[],
): Promise<string[]> {
clearConfigCache();
const dbDir = path.join(root, '.codegraph');
if (fs.existsSync(dbDir)) fs.rmSync(dbDir, { recursive: true, force: true });
await buildGraph(root, { engine, exclude: optsExclude, skipRegistry: true });
const files = readFileRows(path.join(dbDir, 'graph.db'));
return files.map((f) => f.replace(/\\/g, '/')).sort();
}

it('wasm: opts.exclude rejects matching files without writing config', async () => {
const root = fs.mkdtempSync(path.join(tmpDir, 'opts-wasm-'));
writeFixture(root);
const files = await buildWithOptsExclude(root, 'wasm', ['**/*.test.js', '**/*.spec.js']);
expect(files).toContain('src/math.js');
expect(files).not.toContain('src/math.test.js');
expect(files).not.toContain('src/util.spec.js');
});

itNative('native: opts.exclude rejects matching files without writing config', async () => {
const root = fs.mkdtempSync(path.join(tmpDir, 'opts-native-'));
writeFixture(root);
const files = await buildWithOptsExclude(root, 'native', ['**/*.test.js', '**/*.spec.js']);
expect(files).toContain('src/math.js');
expect(files).not.toContain('src/math.test.js');
expect(files).not.toContain('src/util.spec.js');
});
});
Comment on lines +196 to 280
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 opts.exclude incremental coverage gap

Both new tests always wipe the DB before building, so they only exercise the fresh-build path. The scenario where files that were previously indexed become excluded on a subsequent incremental run (i.e. opts.exclude changes between builds against the same DB) is untested. In practice this path should work — collectFiles would omit the newly-excluded files and detectChanges would surface them as removals — but a short test covering one incremental round trip would lock in that behaviour and guard against regressions in the collect/detect stages.

Fix in Claude Code

Copy link
Copy Markdown
Contributor Author

@carlos-alm carlos-alm May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b008ffb — added wasm + native parity tests opts.exclude introduced on second incremental build drops previously-indexed files that build the fixture twice against the same DB (first without exclude, then with). The second build observes the previously-indexed test files as removals via detectChanges, and they are dropped from file_hashes. Verified locally: first build indexes 5 files, second build reports "0 changed, 2 removed", and asserts the test/spec files no longer appear in file_hashes.

Loading