Feat: improves delete_dir for s3fs-backed FsspecStore #2661
Conversation
- override Store.delete_dir default method, which deletes keys one by one, to support bulk deletion for fsspec implementations that support a list of paths in the fs._rm method. - This can greatly reduce the number of requests to S3, which reduces likelihood of running into throttling errors and improves delete performance. - Currently, only s3fs is supported.
…shadi/zarr-python into feat-fsspecstore-bulk-delete
dstansby
added
the
needs release notes
github-actions
bot
removed
the
needs release notes
github-actions
bot
added
the
needs release notes
Co-authored-by: Joe Hamman <[email protected]>
There was a problem hiding this comment.
Looks good here!
@carshadi - the only thing this needs is a release note.
@martindurant - care to do the final review and/or merge?
tests/test_store/test_fsspec.py
Outdated
| """The local fs is not async so we should expect it to be wrapped automatically""" | ||
| from fsspec.implementations.asyn_wrapper import AsyncFileSystemWrapper | ||
|
|
||
| store = FsspecStore.from_url(tmpdir / "test/path", storage_options={"auto_mkdir": True}) |
There was a problem hiding this comment.
tmpdir / "test/path"
This is not a URL. I don't know what from_url demands, but f"{tmpdir}/test/path" seems better to me.
There was a problem hiding this comment.
I used that method since it wraps the filesystem automatically, and figured that would be the entry point most people try first. FWIW, it seems to work with a local path. I could construct the filesystem manually if preferrable, e.g.,
sync_fs = LocalFileSystem(auto_mkdir=True)
async_fs = AsyncFileSystemWrapper(sync_fs)
store = FsspecStore(async_fs, read_only=False, path=f"{tmpdir}/test/path")There was a problem hiding this comment.
it seems to work with a local path
on windows?? :)
There was a problem hiding this comment.
I'm running Windows 11, so apparently :)
Improves performance of
FsspecStore.delete_dirwhen underlyingfsiss3fs, by passing a list of filepaths to be removed in bulk instead of one-by-one.Resolves #2659
TODO: