fsspec is a storage abstraction library. It is used in dask to provide abstraction for different (cloud) storages, providying a unified API for working with different kinds of remote data. It can be used in notebooks or other python projects.
We could write a role that writes default fsspec config files for different kinds of storage:
- webdav / researchdrive
- irods / yoda
- dcache
- s3
- ...
The idea would be to provide an easy way to connect workspaces to storage -- perhaps for all users on the system, so you can have e.g. a project s3 bucket that all users can work with in their python code.
This could work by setting CO Secrets containing tokens (preferably macaroons or tokens that are only valid for a specific file or folder, since we don't want to share personal access tokens with all users on a workspace!) for authentication with different storage backends.
To investigate:
- manually test fsspec with different storage backends
- can we provide a global fsspec config file for all users on the workspace?
- a separate secret per possible storage backend (e.g.
FSSPEC_S3_SECRET) or one big secret with all info (FSSPEC_SECRETS)?
fsspec is a storage abstraction library. It is used in
daskto provide abstraction for different (cloud) storages, providying a unified API for working with different kinds of remote data. It can be used in notebooks or other python projects.We could write a role that writes default fsspec config files for different kinds of storage:
The idea would be to provide an easy way to connect workspaces to storage -- perhaps for all users on the system, so you can have e.g. a project s3 bucket that all users can work with in their python code.
This could work by setting CO Secrets containing tokens (preferably macaroons or tokens that are only valid for a specific file or folder, since we don't want to share personal access tokens with all users on a workspace!) for authentication with different storage backends.
To investigate:
FSSPEC_S3_SECRET) or one big secret with all info (FSSPEC_SECRETS)?