Test ManifestStore against S3 and fix key parsing

[maxrjones]

This PR test the ManifestStore against S3 using MinIO and fixes a small issue with translating ManifestArray keys to the appropriate key for obstore.get().

• Closes Test against S3 using minio #97
• Tests added
• Tests passing
• Full type hint coverage
• Changes are documented in docs/releases.rst
• New functions/methods are listed in api.rst
• New functionality has documentation

[TomNicholas]

This is cool, but I'm not totally sure I understand what you're intending to test here, or how it relates to the ManifestStore work. Surely dealing with remote object storage is the responsibility of obstore and icechunk?

[maxrjones]

This is cool, but I'm not totally sure I understand what you're intending to test here, or how it relates to the ManifestStore work. Surely dealing with remote object storage is the responsibility of obstore and icechunk?

I would like to test that ManifestStore.get() and ManifestStore.list_prefix() return the same expected results for files on both local and object storage. Since ManifestStore requires some URL manipulation, it is possible that we make assumptions valid for local filesystems but not object storage as mentioned as a concern in #490 (comment).

I built this as a separate PR under the assumption that PR scope should be as constrained as possible and that there were some Icechunk tests that would benefit from minio., but didn't pick a good test to start from. I'll modify this PR to test

VirtualiZarr/virtualizarr/tests/test_backend.py

Lines 206 to 232 in 206277f

	@requires_network
	@requires_s3fs
	class TestReadFromS3:
	@pytest.mark.parametrize(
	"indexes",
	[
	None,
	pytest.param({}, marks=pytest.mark.xfail(reason="not implemented")),
	],
	ids=["None index", "empty dict index"],
	)
	@parametrize_over_hdf_backends
	def test_anon_read_s3(self, indexes, hdf_backend):
	"""Parameterized tests for empty vs supplied indexes and filetypes."""
	# TODO: Switch away from this s3 url after minIO is implemented.
	fpath = "s3://carbonplan-share/virtualizarr/local.nc"
	with open_virtual_dataset(
	fpath,
	indexes=indexes,
	reader_options={"storage_options": {"anon": True}},
	backend=hdf_backend,
	) as vds:
	assert vds.dims == {"time": 2920, "lat": 25, "lon": 53}
	assert isinstance(vds["air"].data, ManifestArray)
	for name in ["time", "lat", "lon"]:
	assert isinstance(vds[name].data, np.ndarray)

instead, which is explicitly testing for S3 compatibilty.

[TomNicholas]

I would like to test that ManifestStore.get() and ManifestStore.list_prefix() return the same expected results for files on both local and object storage.

Thanks for the explanation, that makes a lot more sense.

I just want to avoid the rabbit hole of adding tests to VirtualiZarr that are really just testing that Icechunk does what it says it does.

I built this as a separate PR under the assumption that PR scope should be as constrained as possible

FWIW I would be happy to merge your ManifestStore as it is and iterate on testing that.

[maxrjones]

FWIW I would be happy to merge your ManifestStore as it is and iterate on testing that.

Oh wow, that'd make this so much easier!

[maxrjones]

This is the only non-test related change in the PR, which makes the ManifestStore work for both local and object stores

[TomNicholas]

Nice! This does seem useful.

[TomNicholas]

Why does this test not require obstore?

[maxrjones]

These markers don't make any impact on fixtures, so I removed it. The fixture is generated if any tests using the fixtures are captured by pytest, so the markers should all go on tests rather than fixtures.

[TomNicholas]

You only did one put, but now there are 4 chunks in there?

[maxrjones]

I added a docstring explaining the approach in 874c6df (#507). This is kinda similar to libraries that concurrently compress each chunk (or not for uncompressed data), concatenate them, and then write then to a file in one go. At least, I think that's a thing.

[TomNicholas]

Should the mark be requires_minio?

[maxrjones]

They're synonymous, but I will re-add requires_minio since it's more descriptive

[TomNicholas]

Is there a way to split this fixture up a bit, to more clearly delineate the creation of data in a (minio) s3-like obstore from the creation of the ManifestStore object which contains the byte references pointing to it?

[maxrjones]

good call, done in 874c6df (#507)