Skip to content

duckdb-bench --baseline option#8030

Closed
myrrc wants to merge 1 commit into
developfrom
myrrc/duckdb-bench-baseline
Closed

duckdb-bench --baseline option#8030
myrrc wants to merge 1 commit into
developfrom
myrrc/duckdb-bench-baseline

Conversation

@myrrc

@myrrc myrrc commented May 20, 2026

Copy link
Copy Markdown
Contributor

This isn't a replacement for scripts/compare-benchmark-jsons.py but a way to
quickly compare benchmark query results with baseline.

Add a --baseline parameter for duckdb-bench to use previously exported
--gh-json-v3 file as a baseline

Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
@myrrc myrrc force-pushed the myrrc/duckdb-bench-baseline branch from bb38fc7 to c99b46b Compare May 20, 2026 10:26
@myrrc myrrc added the changelog/feature A new feature label May 20, 2026
@myrrc myrrc requested a review from 0ax1 May 20, 2026 10:26
@myrrc myrrc enabled auto-merge (squash) May 20, 2026 10:26
@0ax1 0ax1 requested a review from AdamGS May 20, 2026 10:27
@AdamGS

AdamGS commented May 20, 2026

Copy link
Copy Markdown
Contributor

this functionality exists in the python tool (vx-bench), I would really love to avoid adding more features to the underlying binaries.

@codspeed-hq

codspeed-hq Bot commented May 20, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 19.71%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 1 improved benchmark
✅ 1236 untouched benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation chunked_varbinview_opt_canonical_into[(1000, 10)] 224.6 µs 187.6 µs +19.71%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing myrrc/duckdb-bench-baseline (c99b46b) with develop (f97805d)

Open in CodSpeed

@myrrc myrrc closed this May 20, 2026
auto-merge was automatically disabled May 20, 2026 10:35

Pull request was closed

@0ax1

0ax1 commented May 20, 2026

Copy link
Copy Markdown
Contributor

--gh-json-v3 file as a baseline An interesting addition would be to be able to pass a URL from a PR which the Python runner compares against.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants