Use a dedicated `retry` crate

[Mirko-von-Leipzig]

As per #1495 (comment), we have a bunch of retry logic sprinkled about various places.

It would be good to abstract this away somewhat, I suggest we use a dedicated crate for this as there are always subtle changes one wants in various places.

backon looks nice api-wise at a glance, but there are other options. I would prefer a suffix trait extension and not a closure style API if possible. i.e.

do_thing.retry()
// and not
retry(async move || do_thing));