Update ClickBench benchmarks with DataFusion `43.0.0`

[alamb]

Is your feature request related to a problem or challenge?

Like #11567

Requires

• Release DataFusion 43.0.0 #12470

Once DataFusion 43.0.0 is released, It would be great to update ClickBench https://benchmark.clickhouse.com/ with runs from the latest version. It looks like we are still reporting numbers for DataFusion 40 and there have been significant improvements since then. See for more details:

• [DISCUSSION] Make DataFusion the fastest engine for querying parquet data in ClickBench #12821

Describe the solution you'd like

Perhaps we can follow the model of ClickHouse/ClickBench#210 (thanks @pmcgleenon )

We will also need to update DataFusion to apply the new binary_as_string option added by @goldmedal in #12816. TLDR is that we need to update the create table statements to have the OPTIONS ('binary_as_string' 'true') clause

https://github.com/ClickHouse/ClickBench/blob/main/datafusion/create_partitioned.sql

CREATE EXTERNAL TABLE hits
STORED AS PARQUET
LOCATION 'partitioned'
OPTIONS ('binary_as_string' 'true');

Note this is the same as the DuckDB runner, as explained in #12788

Describe alternatives you've considered

No response

Additional context

No response

[pmcgleenon]

Hi @alamb are we good to proceed with this now that 43.0 has been released (#13254)?

[alamb]

Hi @pmcgleenon

I just double checked and this should be good to go now -- the clickbench runner uses datafusion-cli rather than the python bindings (which haven't yet released a version 43.0.0)

Thank you so much

[pmcgleenon]

take

[pmcgleenon]

Hi @alamb here are the initial results.

I've created a composite html file with results for DataFusion releases 33, 34, 36, 40 and 43 so we can compare across the previous releases.

clickbench.html.zip

Single results

Partitioned results

Before running the tests I've updated the SQL to include OPTIONS ('binary_as_string' 'true')
ClickHouse/ClickBench@30fa096

It looks like we've seen some significant gains across a number of queries. Are these results in line with your expectations?

If so I can update the Clickbench to update it with the latest results.

[alamb]

It looks like we've seen some significant gains across a number of queries. Are these results in line with your expectations?

Yes, indeed, @korowa @Rachelint @jayzhan211 @XiangpengHao and many others have been hard at work improving the speed. For an idea of what was done: #12821 (comment)

It is sad that we are not yet the fastest engine but interestingly it looks like many places where we are the farthest from the top are the really fast queries (that take 100s of ms)

Thank you again @pmcgleenon

[Dandandan]

Looks like we're the fastest for Parquet on c6a.4xlarge? The difference is 16 cores vs 192 cores (and not shared/virtualized).

[alamb]

YES! you are totally right @Dandandan

With some more finagling, I filtered for only c6a.metal 500gb gp2:

WOOHOO -- once this gets published it will be time to write a blog post !

[Rachelint]

After profiling, I believe we still can get obvious improvement after finishing #11943 and its related epic!

[pmcgleenon]

It is sad that we are not yet the fastest engine but interestingly it looks like many places where we are the farthest from the top are the really fast queries (that take 100s of ms)

That's an interesting observation. Maybe we can look at benchmarking on c6a.metal at some point - it might highlight some more areas which could be improved and would show how DataFusion scales out to 192 vCPUs