add benchmarks using pytest-benchmark and codspeed

[d-v-b]

since #3554 was an unpopular direction I'm going instead with codspeed + pytest-benchmark. Opening as a draft because I haven't looked into how codspeed works at all, but I'd like people to weigh in on whether these initial benchmarks make sense. Naturally we can add more specific ones later, but I figured just some bulk array read / write workloads would be a good start.

[d-v-b]

@zarr-developers/steering-council I don't have permission to register this repo with codspeed. I submitted a request to register it, could someone approve it?

[normanrz]

@zarr-developers/steering-council I don't have permission to register this repo with codspeed. I submitted a request to register it, could someone approve it?

done

[d-v-b]

does anyone have opinions about benchmarks? feel free to suggest something concrete. Otherwise, I think we should take this as-is and deal with later benchmarks (like partial shard read / writes) in a subsequent pr

[codspeed-hq]

CodSpeed Performance Report

Congrats! CodSpeed is installed 🎉

🆕 30 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks

• test_write_array[memory-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-None] (WallTime): 11.3 s
• test_write_array[memory-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-gzip] (WallTime): 2.2 s
• test_write_array[memory-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-None] (WallTime): 1.7 s
• test_write_array[memory-Layout(shape=(1000000,), chunks=(1000,), shards=None)-None] (WallTime): 552.5 ms
• test_slice_indexing[(slice(None, 10, None), slice(None, 10, None), slice(None, 10, None))-memory] (WallTime): 981.2 µs
• test_slice_indexing[(slice(10, -10, 4), slice(10, -10, 4), slice(10, -10, 4))-memory] (WallTime): 274.7 ms
• test_slice_indexing[(slice(0, None, 4), slice(0, None, 4), slice(0, None, 4))-memory] (WallTime): 502.4 ms
• test_write_array[local-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-gzip] (WallTime): 3.3 s
• test_slice_indexing[(0, 0, 0)-memory] (WallTime): 924.2 µs
• test_read_array[local-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-gzip] (WallTime): 6.9 s
• test_write_array[local-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-None] (WallTime): 2.9 s
• test_write_array[memory-Layout(shape=(1000000,), chunks=(1000,), shards=None)-gzip] (WallTime): 1 s
• test_slice_indexing[(slice(None, None, None), slice(None, None, None), slice(None, None, None))-memory] (WallTime): 507.2 ms
• test_write_array[memory-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-gzip] (WallTime): 15.4 s
• test_write_array[local-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-gzip] (WallTime): 15.4 s
• test_slice_indexing[(slice(None, None, None), slice(0, 3, 2), slice(0, 10, None))-memory] (WallTime): 4.7 ms
• test_read_array[memory-Layout(shape=(1000000,), chunks=(1000,), shards=None)-gzip] (WallTime): 681.1 ms
• test_read_array[memory-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-None] (WallTime): 3.9 s
• test_read_array[memory-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-gzip] (WallTime): 6.9 s
• test_write_array[local-Layout(shape=(1000000,), chunks=(1000,), shards=None)-None] (WallTime): 1.2 s
• ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

[dcherian]

can we test the latest instead? seems more appropriate...

[d-v-b]

The latest version of python? What's the reasoning? I'd rather update this file when we drop a supported version vs when a new version of python comes out.

[dcherian]

Because we'd want to catch a perf regression from upstream changes too? I'm suggested latest version of released libraries py=3.13, np=2.2

[d-v-b]

we don't have an upper bound on numpy versions, so I don't think this particular workflow will help us catch regressions from upstream changes -- we would need to update this workflow every time a new version of numpy is released. IMO that's something we should do in a separate benchmark workflow. This workflow here will run on every PR, and in that case the oldest version of numpy we support seems better.

we also don't have to use a pre-baked hatch environment here, we could define a dependency set specific to benchmarking. but my feeling is that benchmarking against older versions of stuff gives us a better measure of what users will actually experience.

[dcherian]

feel free to suggest something concrete

indexing please. that'll exercise the codec pipeline too.

a peakmem metric would be good to track also, if possible.

[d-v-b]

feel free to suggest something concrete

indexing please. that'll exercise the codec pipeline too.

a peakmem metric would be good to track also, if possible.

I don't think codspeed or pytest-benchmark do memory profiling. we would need https://pytest-memray.readthedocs.io/en/latest/ or something equivalent for that.

and an indexing benchmark sounds like a great idea but I don't think I have the bandwidth for it in this pr right now

[joshmoore]

(Enabled the app)

[maxrjones]

IMO it'd be better to skip the tests/benchmarks during regular test runs in the interest of speed

[d-v-b]

IMO it'd be better to skip the tests/benchmarks during regular test runs in the interest of speed

i think this makes sense -- on my workstation the current benchmark suite takes 40s to run as regular tests, which is a big addition to our total test runtime. The latest changes to this branch skip the test/benchmarks folder by default when running our main test suite and the gpu tests.

[d-v-b]

I'm seeing some flakiness in the codspeed CI run: https://github.com/zarr-developers/zarr-python/actions/runs/21025818794/job/60449955113#step:5:253. Since this is a codspeed-specific issue (from what I can tell), and it doesn't detract from the utility of the benchmark suite for local benchmarking, I'm going to work around this failure by ignoring this warning.