teach evaluator to conditionally retry provider invocation

[nyobe]

Adds a RetryableError error that providers can return to opt into being retried by the evaluator when they encounter a temporary error (such as a timeout).

Closes https://github.com/pulumi/pulumi-service/issues/25444

[pgavlin]

so at a high level I'm wondering if this is kind of a footgun.

if a rotator gets halfway through a rotation, it'll need to have some way to understand the state it is in when it is next retried.

I don't think that we have a solution for that at the moment.

I wonder if instead of handling this in the evaluator we should just add a helper function that provider authors can use for retries.

[pgavlin]

nit: since we have Unwrap, I'd make this field unexported

[pgavlin]

also--let's add a factory for these

func Retryable(err error) RetryableError { return RetryableError{err} }

[nyobe]

33eda08

[nyobe]

if a rotator gets halfway through a rotation, it'll need to have some way to understand the state it is in when it is next retried.

When providers have access to managing creds they'll usually have the authority to overwrite whatever they find, so keeping intermediate state isn't as necessary like it would be if we tried to use the rotated credential to manage itself.

I wonder if instead of handling this in the evaluator we should just add a helper function that provider authors can use for retries.

We sort of do have a helper available with the retry.Until utility. Also, many sdks (like aws's) do their own builtin retries. Since the provider itself has the most context about the errors it encounters, maybe we should just let the provider itself make the decision on how it wants to retry itself, rather than trying to put that in the evaluator? (So just close this for now?)

[pgavlin]

Since the provider itself has the most context about the errors it encounters, maybe we should just let the provider itself make the decision on how it wants to retry itself, rather than trying to put that in the evaluator? (So just close this for now?)

Yeah, this is what I'm thinking tbh. That allows us to sidestep questions of state, etc.