Add support for async API calls

[mariosasko]

It would be great to have the async versions of snapshot_download/HfApi to enable async usage of hfh.

(Link to the internal Slack thread in which this has been discussed)

[TheSeriousProgrammer]

Since hffs is archived, is there any possible work around to call methods such as upload file from HfAPi in async
It will be very usefull in bulk uploads, or is there a method to upload a bunch of files in bulk

Thanks for the phenomenal work you guys do

[Wauplin]

Hey @TheSeriousProgrammer no, at the moment there is no async version of HfApi. We thought about it but there has not been much demand yet for it and unfortunately Python is not very well designed to have both a sync and an async client without duplicating code (hence duplicating maintenance). We did it for the AsyncInferenceClient because in that case it makes sense to parallelize calls to an inference server.

However for a commit to a repo I'm not sure to see the point. Each upload_file creates a commit in the history so it's usually not massively used in parallel. If you want to upload many files, I'd suggest you to use create_commit or upload_folder which uses multithreading internally (not asyncio). Finally, if you want to upload something in the background without blocking the main thread, you can pass run_as_future=True (but commits will still be made sequentially in a separate thread if you call it multiple times). Check out the upload guide for more details: https://huggingface.co/docs/huggingface_hub/guides/upload.

Please let me know if you're thinking of something else.

[TheSeriousProgrammer]

create_commit sounds good let me try using the same, i read through the docs you provided, is there a way I could create a commit with a batch of files together just like in git

[Wauplin]

Yes of course! Just provide a list of CommitOperationAdd to the call and everything will be taken care of in a single commit! 🤗