Move MemoryStore to a persistent store (e.g. LocalStore)

[kevinli1993]

Hi, once I put data inside a group on a MemoryStore, how do I copy the group over to a persistent store?

root = zarr.create_group(store={})
# put data in root

# Now I want to move this to a persistent store
store = LocalStore("....")
store.MAGIC_COPY_COMMAND(from=root)

I see there is zarr.copy() and zarr.copy_group(), but they are not implemented; reading the linked issue, it is also unclear whether they would be removed.

Curious if the use-case above would be supported? Thanks.

[d-v-b]

hi @kevinli1993 this is a great question and we should absolutely support this with a stand-alone convenience function. Until we get around to writing such a convenience function ,here's a script that illustrates how you can do this today:

# /// script
# requires-python = ">=3.11"
# dependencies = [
#     "zarr>3",
# ]
# ///

import zarr
from zarr.storage import MemoryStore
from zarr.core.buffer import default_buffer_prototype
import asyncio

mem_a = {}
mem_b = {}

store_a = MemoryStore(mem_a)
store_b = MemoryStore(mem_b)

# create an array and fill it with values
array = zarr.create_array(store_a, name='foo', shape=(10,), dtype='float32')
array[:] = 1

# define an async function that lists the contents of the source store and copies each key, value pair to the dest store
async def copy_store(store_a, store_b):
    async for x in store_a.list():
        await store_b.set(x, await store_a.get(x, prototype=default_buffer_prototype()))
        print(f'setting {x} ')
    return None

# run that async function
asyncio.run(copy_store(store_a, store_b))

# check that the two dictionaries are identical, i.e. everything from mem_a is in mem_b
assert mem_a == mem_b

[jhamman]

Noting that we have an open issue for zarr.copy_store: #2407

[kevinli1993]

Thanks @d-v-b for the example snippet -- it works well and also taught me a bit about .set() and .get() methods in the process. Also thanks @jhamman for the reference.