Handle nested subchunks

[kylebarron]

+1 to moving fast! my two questions after a cursory read -- how are nested subchunks handled, and what's the contract between an arbitrary array-bytes codec and the array that allows the array to use the cache? I assume there's a trait in zarrs for "contains subchunks".

Originally posted by @d-v-b in #137 (comment)

[kylebarron]

Making an issue to make it easier to continue discussion.

I don't think Zarrs' primary API here supports nested subchunks.

The API I exposed is a very straightforward binding to the underlying Zarrs API.

I assume there's a trait in zarrs for "contains subchunks".

Yeah the trait ArrayShardedExt has ArrayShardedExt::is_sharded which lets you check if there's a sharding codec.

We already expose those APIs on both Array and AsyncArray:

zarrista/src/array/shared.rs

Lines 88 to 93 in 2cdc5f7

	#[getter]
	fn is_sharded(&self) -> bool {
	use zarrs::array::ArrayShardedExt;
	self.inner.is_sharded()
	}

zarrista/src/array/shared.rs

Lines 118 to 137 in 2cdc5f7

	#[getter]
	fn subchunk_grid(&self) -> $crate::array::PyChunkGrid {
	use zarrs::array::ArrayShardedExt;
	self.inner.subchunk_grid().into()
	}
	#[getter]
	fn subchunk_grid_shape(&self) -> $crate::array::PyArrayShape {
	use zarrs::array::ArrayShardedExt;
	self.inner.subchunk_grid_shape()
	}
	#[getter]
	fn subchunk_shape(&self) -> Option<$crate::array::PyChunkShape> {
	use zarrs::array::ArrayShardedExt;
	self.inner.subchunk_shape()
	}

[kylebarron]

what's the contract between an arbitrary array-bytes codec and the array that allows the array to use the cache?

I'm not 100% sure I understand this, but I think the answer is that the cache is entirely specific to one Array.

[LDeakin]

The API for subchunks has changed quite a bit on main, so this'll need some adaptation with the next release.

Subchunking is generalised now, not sharding specific. Nested subchunk grids are supported, and even custom codecs can do subchunking. This might seem niche, but I have data I want to virtualise with a custom codec with subchunks.

[d-v-b]

This might seem niche, but I have data I want to virtualise with a custom codec with subchunks.

doesn't seem niche to me! sounds like a very useful abstraction.