Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement our own url_to_fs for fsspec storage #2673

Closed
d-v-b opened this issue Jan 8, 2025 · 1 comment
Closed

implement our own url_to_fs for fsspec storage #2673

d-v-b opened this issue Jan 8, 2025 · 1 comment

Comments

@d-v-b
Copy link
Contributor

d-v-b commented Jan 8, 2025

we are currently using fsspec.url_to_fs to convert string URLs to fsspec filesystems. url_to_fs can be very slow (~25s in some cases), which adds unnecessary latency given the simplicity of the URLs we need to handle. So we should implement our own version of this function that runs in much less time.

I opened an issue about this upstream: fsspec/filesystem_spec#1768

@d-v-b
Copy link
Contributor Author

d-v-b commented Jan 8, 2025

looks like the actual problem is the time required for gcsfs to resolve credentials, not url_to_fs itself. I will close this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant