Skip to content

Commit

Permalink
Extend Retry logic to address OSErrors.
Browse files Browse the repository at this point in the history
I've encountered a case where large bulk deletions throws an `OSError`. The GCS documentation points out that this is a general problem (https://cloud.google.com/storage/docs/deleting-objects#delete-objects-in-bulk). I did not expect, however, that this would throw an OSError instead of a 500 error.

In any case, this PR extends `gcsfs`'s retry logic to include internal errors that throw `OSError`s. My hope is that this fixes an associated issue in Pangeo Forge Recipes (pangeo-forge/pangeo-forge-recipes#406).
  • Loading branch information
alxmrs committed Aug 29, 2022
1 parent ba716f4 commit f21c6eb
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions gcsfs/retry.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ class ChecksumError(Exception):
google.auth.exceptions.RefreshError,
aiohttp.client_exceptions.ClientError,
ChecksumError,
OSError,
)


Expand Down Expand Up @@ -119,6 +120,7 @@ async def retry_request(func, retries=6, *args, **kwargs):
google.auth.exceptions.GoogleAuthError,
ChecksumError,
aiohttp.client_exceptions.ClientError,
OSError,
) as e:
if (
isinstance(e, HttpError)
Expand Down

0 comments on commit f21c6eb

Please sign in to comment.