Skip to content

perf: exclude verilog benchmark fixtures from incremental benchmark sweep #1112

@carlos-alm

Description

@carlos-alm

Deferred from PR #1107.

Context: Adding native Verilog support pulled the 4 .v resolution-benchmark
fixtures (tests/benchmarks/resolution/fixtures/verilog/*.v) into the corpus
that scripts/incremental-benchmark.ts builds (it walks the entire repo root).
tree-sitter-verilog is a large grammar, so parsing those fixtures adds ~850ms
to the native fullBuildMs measurement (1959 → 2809ms, +43% vs the 3.10.0
baseline). The PR landed with 3.10.0:Full build added to KNOWN_REGRESSIONS
so the per-PR benchmark gate is unblocked.

Fix options:

  1. Add an exclude glob (e.g. tests/benchmarks/resolution/fixtures/**) to
    the buildGraph invocation in scripts/incremental-benchmark.ts so fixture
    files don't inflate the timing for any new-language PR going forward. This
    is the principled fix — resolution-benchmark fixtures are not representative
    of real codebases the benchmark is supposed to measure.
  2. Alternatively, accept the structural cost as the price of supporting Verilog
    and remove the 3.10.0:Full build exemption once 3.11.0+ data establishes
    the new baseline.

Once resolved, the 3.10.0:Full build entry in tests/benchmarks/regression-guard.test.ts
should be removed.

Originally raised by: PR #1107 CI failure (run 25716010487).

Metadata

Metadata

Assignees

No one assigned

    Labels

    follow-upDeferred work from PR reviews that needs tracking

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions