Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrap sync fs for xarray.to_zarr #2533

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

moradology
Copy link

@moradology moradology commented Dec 4, 2024

This PR automatically wraps synchronous filesystems in the RemoteStore.from_url function. This is necessary unless we provide a force_async argument on fsspecs url_to_fs function.

I'm not sure adding more flags (even if we sugar them with default values) is the best move. Another option might be to implement a url_to_asyncfs method that simply wraps url_to_fs and wraps any synchronous filesystems that come out automatically but that, too, would require changes here.

This PR is in service of downstream kerchunk compatibility: fsspec/kerchunk#516

It depends on: fsspec/filesystem_spec#1755

@moradology moradology force-pushed the fix/wrap_sync_in_remote branch 2 times, most recently from 8cd084d to 38a841c Compare December 18, 2024 18:39
@moradology moradology marked this pull request as ready for review December 18, 2024 18:40
@moradology
Copy link
Author

moradology commented Dec 18, 2024

This fix is implicated in the issue here: #2554. Opening for review. Merge (and CI) is blocked pending a release of fsspec with async wrapper and it may be sensible to wait on that release for a bit to ensure compatibility with other projects as there are a lot of moving pieces across the various libraries being updated

@jhamman
Copy link
Member

jhamman commented Jan 4, 2025

@moradology - is there interest in finishing this up? The proposed changes look good but we'll want a test to make sure this is working as expected.

@moradology
Copy link
Author

Yeah, definite interest in getting back to this - just at the tail end of a long period of vacation/travel and will have time in the coming week!

@moradology moradology force-pushed the fix/wrap_sync_in_remote branch 2 times, most recently from 563e99d to c7e0f59 Compare January 8, 2025 20:28
@jhamman jhamman requested a review from martindurant January 8, 2025 21:22
@dstansby dstansby added the needs release notes Automatically applied to PRs which haven't added release notes label Jan 9, 2025
@@ -3,6 +3,8 @@
import warnings
from typing import TYPE_CHECKING, Any

from fsspec.implementations.asyn_wrapper import AsyncFileSystemWrapper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the whole module depend on fsspec. I'm OK with that.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is definitely unfortunate but a bit difficult to avoid

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why can't this import be done in or around line 162?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to do that (and add a bit of caution around the possibly failing import depending on fsspec versions)

@github-actions github-actions bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Jan 13, 2025
Copy link

codecov bot commented Jan 13, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.27%. Comparing base (eaf5d7a) to head (4b8338f).
Report is 426 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2533      +/-   ##
==========================================
- Coverage   99.98%   90.27%   -9.72%     
==========================================
  Files          38       59      +21     
  Lines       14718     6468    -8250     
==========================================
- Hits        14716     5839    -8877     
- Misses          2      629     +627     
Files with missing lines Coverage Δ
src/zarr/storage/_fsspec.py 84.48% <ø> (ø)

... and 96 files with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants