feat/batch creation

[d-v-b]

This PR adds a few routines for creating a collection of arrays and groups (i.e., a dict with path-like keys and ArrayMetadata / GroupMetadata values) in storage concurrently.

• create_hierarchy takes a dict representation of a hierarchy, parses that dict to ensure that there are no implicit groups (creating group metadata documents as needed), then invokes create_nodes and yields the results
• create_nodes concurrently writes metadata documents to storage, and yields the created AsyncArray / AsyncGroup instances.

I still need to wire up concurrency limits, and test them.

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[d-v-b]

this is now working, so I would appreciate some feedback on the design.

The basic design is the same as what I outlined earlier in this PR: there are two new functions that take a dict[path, GroupMetadata | ArrayMetadata] like {'a': GroupMetadata(zarr_format=3), 'a/b': ArrayMetadata(...)} and concurrently persist those metadata documents to storage, resulting in a hierarchy on disk that looks like the dict.

approach

basically the same as concurrent group members listing, except we don't need any recursion. I'm scheduling writes and using as_completed to yield Arrays / Groups when they are available.

new functions

• create_nodes is low-level and doesn't do any checking of its input, so it will happily create invalid hierarchies, e.g. nesting groups inside arrays, or mixing v2 and v3 metadata, and it won't create intermediate groups, either.

• create_hierarchy is higher level, it parses the input, checking it for invalid hierarchies, and inserting implicit groups as needed.

• Group.create_hierarchy is a new method on the Group / AsyncGroup classes that takes a hierarchy dict and creates the nodes specified in that dict at locations relative to the path of the group instance. the return value is dict[str, AsyncGroup | AsyncArray], but I guess it also doesn't have tor return anything, or it could be an async iterator, so that you can interact with the nodes as they are formed. This is flexible right now, but I think the iterator idea is nice.

• _from_flat (names welcome) is a new function that creates a group entirely from a hierarchy dict + a store. that dict must specify a root group, otherwise an exception is raised. We could revise this to create a root group if one is not specified. Open to suggestions here.

Implicit groups

Partial hierarchies like {'a': GroupMetadata(), 'a/b/c': ArrayMetadata(...)} implicitly denote a group at a/b. When creating such a hierarchy, if we find an existing group at a/b, then we don't need to create a new one. So in the context of modeling a hierarchy, implicit groups are a little special -- by not specifying the properties of the group, the user / application is tolerant of any group being there. So I introduced a subclass of GroupMetadata called _ImplicitGroupMetadata, which can be inserted into a hierarchy dict to explicitly denote groups that don't need to be written if one already exists. _ImplicitGroupMetadata is just like GroupMetadata except it errors if you try to set any parameter except zarr_format.

streaming v2 vs v3 node creation

creating v3 arrays / groups requires writing 1 metadata document, but v2 requires 2. To get the most concurrency I await the write of each metadata document separately, which means that foo/.zattrs might resolve before foo/.zarray. So in the v2 case I only yield up an array / group when both documents were written.

Overlap with metadata consolidation logic

there's a lot of similarity between the stuff in this PR and routines used for consolidated metadata. it would be great to find ways to factor out some of the overlap areas

still to do:

• write some more tests (checking that implicit groups don't get written if a group already exists)
• handle overwriting. I think the plan here is, if overwrite is False, then we do a check before any writing to ensure that there are no conflicts between the proposed hierarchy and the stuff that actually exists in storage. this check will involve more IO.

[d-v-b]

@TomNicholas you should have a look at some of these new functions / methods. I'd be happy to change things if you have some datatree conventions you'd like to suggest

[dcherian]

A note about when to use this would be great

[d-v-b]

good point, would something like "use this method to create an entire tree of sub-groups and / or sub-arrays efficiently." suffice?

[dcherian]

hierarchy is fine to me.

[dcherian]

why do we need None as an option here? And why is it "no semaphore" instead of "default concurrency semaphore"?

[d-v-b]

create_nodes is low-level, kind of dangerous, and the expectation is that the user of this function knows what they are doing. So it doesn't default with any concurrency limit. It also doesn't check if nodes already exist, or if the user wants to nest arrays inside arrays. A higher level function (create_hierarchy) does all that checking, and that's where the concurrency limit defaults to the value in the config.

[dcherian]

so why not require the semaphore if it's an advanced user?

[d-v-b]

the semaphore is only necessary if you want some concurrency limiting mechanism, but I think that's a special case. A default of None means that users of the function don't need to run from asyncio import Semaphore before creating some nodes. E.g., there are a lot of places in the tests where I would want to use create_nodes, and in basically none of those places would a semaphore be necessary.

[dcherian]

can we short-circuit inside the loop and raise on the first instance of zarr_format that is not equal to next(data.items())[1].zarr_format?

[d-v-b]

we definitely could, but we get a much nicer error message if we can traverse the entire proposed hierarchy and identify all the problematic nodes.

[dcherian]

i think it's better to just grab all 3 at once honestly

[dcherian]

seems like this would duplicate logic form elsewhere? Is there nowhere else this function could be used?

[d-v-b]

this should be duplicated, but sadly these functions don't exist in the codebase yet! I add some functions like this in another PR: #2761

[d-v-b]

question: should we export a sync version of create_hierarchy from the top-level zarr namespace?

[dcherian]

question: should we export a sync version of create_hierarchy from the top-level zarr namespace?

Yes, this would be used in Xarray.

[d-v-b]

this PR adds a few functions that have async implementations and sync wrappers, like create_hierarchy_a (async) and create_hierarchy. Instead of putting (async, sync) pairs in the same module with slightly different names, I wonder if we should split the group module into an async-only namespace and a sync namespace (that imports stuff from the async namespace)? Then we don't need to mangle function names.