-
Notifications
You must be signed in to change notification settings - Fork 373
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
http: allow put_file()ing file-like objects #764
Conversation
This seems like it might be a useful thing to do for all filesystems. Possibly related: #723 |
I agree. It might require a bigger design discussion, but something I wondered when I started to the migration to the fsspec was why this was not supported natively (since that would potentially render the requirement to the callbacks redundant, since we could simply wrap |
ddcddcb
to
702c64f
Compare
Should be |
fsspec/implementations/http.py
Outdated
@@ -231,21 +232,21 @@ async def _cat_file(self, url, start=None, end=None, **kwargs): | |||
return out | |||
|
|||
async def _get_file( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this one was right!
It says, "get file from remote to local", and should be the revers of put_file. That's how we have it in spec.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, missed that! Thanks for the correction.
This reverts commit df58af7.
Leftover from fsspec#764
Since writing to
HTTPFile
is not supported, currently you can not upload arbitrary streams (unlike other filesystems where you could open a new file in writing mode throughfs.open(path, 'wb')
and copy the original stream throughshutil.copy_file_obj()
to the new file). Not sure if this would be worth documenting at the moment, but would be extremely useful to compensate for the missingw
method at the moment while we are thinking about other solutions.