8.0.5 retries can take a very long time #1411
Description
Retry logic introduced by #1390 can cause surprisingly long delays in uploads.
This means that a request that would previously have failed ~immediately could now take 60+ minutes.
I would see this as a significant change to the API, warranting a change in major version number.
Also it would be helpful to allow for configuration of backoff time and retry count for a minio client, and for individual commands.
Example:
07:45:25 Retrying request (attempt 1/10) after 3439.900516157544ms due to: Error: Retryable HTTP status: 500
07:45:28 Retrying request (attempt 2/10) after 6413.11554101859ms due to: Error: Retryable HTTP status: 500
07:45:35 Retrying request (attempt 3/10) after 8294.534844609287ms due to: Error: Retryable HTTP status: 500
07:45:43 Retrying request (attempt 4/10) after 22798.412321911434ms due to: Error: Retryable HTTP status: 500
07:46:06 Retrying request (attempt 5/10) after 43566.78475639889ms due to: Error: Retryable HTTP status: 500
07:46:49 Retrying request (attempt 6/10) after 109948.62006674097ms due to: Error: Retryable HTTP status: 500
07:48:39 Retrying request (attempt 7/10) after 173239.9702801027ms due to: Error: Retryable HTTP status: 500
07:51:33 Retrying request (attempt 8/10) after 410686.5376373904ms due to: Error: Retryable HTTP status: 500
07:58:23 Retrying request (attempt 9/10) after 916907.7173781618ms due to: Error: Retryable HTTP status: 500
08:13:40 Retrying request (attempt 10/10) after 1337530.4101596891ms due to: Error: Retryable HTTP status: 500
Total wait time: 50 minutes.