Skip to content

feat(vortex-bench): support vortex native geo types into SpatialBench, and wire into benchmark orchestrator#8623

Open
HarukiMoriarty wants to merge 2 commits into
nemo/geo-multipolyginfrom
nemo/duckdb-native-geometry
Open

feat(vortex-bench): support vortex native geo types into SpatialBench, and wire into benchmark orchestrator#8623
HarukiMoriarty wants to merge 2 commits into
nemo/geo-multipolyginfrom
nemo/duckdb-native-geometry

Conversation

@HarukiMoriarty

@HarukiMoriarty HarukiMoriarty commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Wires vortex geo native format into SpatialBench, and wire into the vx-bench / bench-orchestrator pipeline so it can be run end-to-end like the other benchmarks.

Running command:

vx-bench run spatialbench -e duckdb -f parquet,vortex,vortex-native --opt scale-factor=1.0 --queries 1,2,3,4,5,6,7,8,9

Limitation

  • DuckDB-only as before. For now SpatialBench queries use DuckDB-specific ST_* spatial SQL that DataFusion has no functions for yet. There is a a single ad-hoc entry in BENCHMARK_ENGINES = { SPATIALBENCH: {DUCKDB} }.
  • Queries 10, 11, 12 is timeout simply because DuckDB poorly support on Spatial index.

Performance

SF=1.0

┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ Query ┃ duckdb:parquet (base) ┃   duckdb:vortex ┃ duckdb:vortex-native ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
│ 1     │                40.8ms │  13.8ms (0.34x) │       23.2ms (0.57x) │
│ 2     │               181.2ms │  26.3ms (0.14x) │       35.3ms (0.20x) │
│ 3     │                57.8ms │  19.8ms (0.34x) │       30.9ms (0.53x) │
│ 4     │               331.9ms │  61.8ms (0.19x) │      111.0ms (0.33x) │
│ 5     │               356.1ms │ 294.3ms (0.83x) │      299.3ms (0.84x) │
│ 6     │               441.9ms │  86.3ms (0.20x) │      135.1ms (0.31x) │
│ 7     │               157.2ms │  73.7ms (0.47x) │       93.1ms (0.59x) │
│ 8     │               197.8ms │  78.1ms (0.39x) │       91.9ms (0.46x) │
│ 9     │                20.3ms │  18.6ms (0.91x) │       20.7ms (1.02x) │
└───────┴───────────────────────┴─────────────────┴──────────────────────┘

SF=10

┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ Query ┃ duckdb:parquet (base) ┃   duckdb:vortex ┃ duckdb:vortex-native ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
│ 1     │               160.4ms │ 128.5ms (0.80x) │      214.8ms (1.34x) │
│ 2     │               254.3ms │ 239.8ms (0.94x) │      325.9ms (1.28x) │
│ 3     │               231.0ms │ 198.7ms (0.86x) │      284.4ms (1.23x) │
│ 4     │               188.0ms │ 124.2ms (0.66x) │      147.9ms (0.79x) │
│ 5     │                 3.14s │   3.05s (0.97x) │        2.92s (0.93x) │
│ 6     │               480.5ms │ 361.7ms (0.75x) │      467.7ms (0.97x) │
│ 7     │               992.8ms │   1.02s (1.02x) │      915.2ms (0.92x) │
│ 8     │                 1.07s │ 961.8ms (0.90x) │        1.02s (0.95x) │
│ 9     │                34.2ms │  34.1ms (1.00x) │       43.5ms (1.27x) │
└───────┴───────────────────────┴─────────────────┴──────────────────────┘

Takeaways: for now we do not have only pushdown to execute predicate in vortex kernel, so geo native types pay the tax of converting to geo-binary (talk to duckdb) but no gains, so in SF=10 vortex-native is less efficient compared with vortex-wkb.

@codspeed-hq

codspeed-hq Bot commented Jun 29, 2026 •
edited
Loading

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 1595 untouched benchmarks
⏩ 4 skipped benchmarks1

Comparing nemo/duckdb-native-geometry (bc74778) with nemo/geo-multipolygin (80e39d5)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@HarukiMoriarty HarukiMoriarty force-pushed the nemo/geo-multipolygin branch from a3c046b to 790a0d7 Compare July 1, 2026 15:28
@HarukiMoriarty HarukiMoriarty force-pushed the nemo/duckdb-native-geometry branch from 69a4f90 to bc74778 Compare July 1, 2026 17:48
@HarukiMoriarty HarukiMoriarty added the changelog/feature A new feature label Jul 1, 2026
@HarukiMoriarty
feat: wire vortex-native into duckdb-bench
1048d01 
Signed-off-by: Nemo Yu <zyu379@wisc.edu>
@HarukiMoriarty
feat: wire geo native into vx-bench
8e0738a 
Signed-off-by: Nemo Yu <zyu379@wisc.edu>
@HarukiMoriarty HarukiMoriarty force-pushed the nemo/duckdb-native-geometry branch from bc74778 to 8e0738a Compare July 1, 2026 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

changelog/feature A new feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@HarukiMoriarty