Add CLI for converting v2 metadata to v3

[K-Meech]

For #1798

Adds a CLI using typer to convert v2 metadata (.zarray / .zattrs...) to v3 metadata zarr.json.

To test, you will need to install the new optional cli dependency e.g.
pip install -e ".[remote,cli]"

This should make the zarr-converter command available e.g. try:

zarr-converter --help
zarr-converter convert --help
zarr-converter clear --help

convert adds zarr.json files to every group / array, leaving the v2 metadata as-is. A zarr with both sets of metadata can still be opened with zarr.open, but will give a UserWarning: Both zarr.json (Zarr format 3) and .zarray (Zarr format 2) metadata objects exist... Zarr v3 will be used.. This can be avoided by passing zarr_format=3 to zarr.open, or by using the clear command to remove the v2 metadata.

clear can also remove v3 metadata. This is useful if the conversion fails part way through e.g. if one of the arrays uses a codec with no v3 equivalent.

All code for the cli is in src/zarr/core/metadata/converter/cli.py, with the actual conversion functions in src/zarr/core/metadata/converter/converter_v2_v3.py. These functions can be called directly, for those who don't want to use the CLI (although currently they are part of /core which is considered private API, so it may be best to move them elsewhere in the package).

Some points to consider:

• I had to modify set_path from test_dtype_registry.py and test_codec_entrypoints.py, as they were causing the CLI tests to fail if they were run after. This seems to be due to the lazy_load_list of the numcodecs codecs registries being cleared, meaning they were no longer available in my code which finds the numcodecs.zarr3 equivalent of a numcodecs codec.
• I tested this on local zarr images, so it would be great if someone with access to s3 / google cloud etc., could try it out on some small example images there.
• I'm happy to add docs about how to use the CLI, but wanted to get feedback on the general structure first

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[K-Meech]

I've updated the structure of the CLI - hopefully this addresses both @dstansby and @d-v-b 's comments! You should be able to test with:

zarr --help
zarr migrate --help
zarr remove-metadata --help

I haven't addressed this comment yet, but will do so in the next round of changes (I know @dstansby had some additional comments to make on the converter implementation).

One known issue:

• If you run zarr migrate --dry-run with a non-existent location e.g. zarr migrate v3 data/images/example-1.zarr --dry-run, it will create an empty directory at that location. This is related to r+ opens - reported on this issue: Using r+ mode with zarr.open creates an empty directory #3295

[dstansby]

Very nice! I had a play with it locally and it worked well. I'll do a fuller review of the code now.

[dstansby]

Looking great! I've reviewed the implementation, and left some comments; I'll move on to reviewing the tests next, but thought I'd post these comments first.

[dstansby]

Suggested change

	logging.basicConfig(level=lvl, format=fmt)
	logger.basicConfig(level=lvl, format=fmt)

I think you want to configure the logger instance, not global settings?

[K-Meech]

I could do something like:

if verbose:
        logger.setLevel(logging.INFO)
    else:
        logger.setLevel(logging.WARNING)
logger.addHandler(logging.StreamHandler())

but the issue is this will only affect logs directly from the cli.py file. When using --dry-run, I also want to see the log of created / deleted files from migrate_to_v3.py, which won't be shown with this setting alone.

I could add a similar setup to the migrate_to_v3.py file, but this could be annoying for downstream code that uses these functions and wants to use a different logging level / setup e.g. to a file rather than the console.

It could work if I created the migrate_to_v3.py logger as an explicit child of the cli.py logger though i.e.

# in cli.py
logger = logging.getLogger("cli")

# in migrate_to_v3.py
logger = logging.getLogger("cli.migrate")

What do you think?

[dstansby]

Suggested change

	logging.getLogger().setLevel(logging.INFO)
	logger.setLevel(logging.INFO)

Same reason as above

[dstansby]

Suggested change

	"Output location to write generated metadata (no chunks will be copied). If not provided, "
	"Output location to write generated metadata (no array data will be copied). If not provided, "

[dstansby]

Suggested change

	input_store: StoreLike,
	*,
	input_store: StoreLike,

Going keyword only here should avoid accidentally hading the input/ouput stores the wrong way round.

[dstansby]

To save having to deal with storage_options here, I would instead just accept an actual Store as input to this funciton. Doing this would also avoid having to deal with the case where one wants different storage_options for both input and output stores.

[dstansby]

I think it would make slightly more sense to pass filters, instead of the whole metadata here.

Suggested change

	def _convert_filters(metadata_v2: ArrayV2Metadata) -> list[ArrayArrayCodec]:
	def _convert_filters(filters) -> list[ArrayArrayCodec]:

(I probably got the typing wrong)

[dstansby]

Suggested change

	def _convert_compressor(metadata_v2: ArrayV2Metadata) -> BytesBytesCodec | None:
	def _convert_compressor(compressor, dtype) -> BytesBytesCodec | None:

Again, I think makes for a clearer API if you explicitly just pass the compressor and type instead of the whole metadata.

[dstansby]

Suggested change

	compressor_name = metadata_v2.compressor.codec_id
	match compressor_name:
	match metadata_v2.compressor.codec_id:

Since you don't use the variable again

[TomNicholas]

If you have the patience for it, please keep track of the stuff we could change in the core library to make this kind of tool easier to write, and feel free to open issues to track these features.

For example, is there currently a way to use zarr-python to convert v2 to v3 metadata without the CLI? If so then that would be independently useful (e.g. in VirtualiZarr). If not then in an ideal world that would exist before a CLI is added to wrap it.

[emmanuelmathot]

Just FYI, I made this converter for ESA EOPF datasets (Zarr v2) to EOPF GeoZarr (V3). I adopted a recursive approach to copy and keep the datatree as well as the consolidated metadata.

[K-Meech]

@TomNicholas - this PR will add both a CLI + functions to convert v2 to v3 metadata. Everything possible with the CLI is also available via direct function calls.

Although, at the moment these are added to src/zarr/core/metadata which I don't think is part of the public API - @d-v-b / @dstansby, any suggestions for where these could be moved to make them easy to use by downstream scripts / packages?

[TomNicholas]

@K-Meech thanks. That's actually okay for my use case - I'm already importing other private metadata-related internals. But I agree a later PR to make these public would be great.

[dstansby]

I'd suggest putting the metadata converter python API in a new zarr.metadata submodule (which can contain other bits of metadata migrated from zarr.core.metadata later), and hiding the the CLI wrapper in a new private zarr._cli submodule (or zarr._cli.py file)