Survive minor db drops

[adamruzicka]

Includes #462

Essentially it means wrapping all db interactions in the persistence adapter with retries. We don't really need the retries, but we need to wrap the db-specific errors with general dynflow persistence errors which the rest of the codebase expects.

[ofedoren]

Thanks, @adamruzicka !

IIUC, this will help if a DB won't be available for cca 1 minute, and if more it will just kill the process now? If so, apologies for the newbie question, but looking at the related tickets, how will that help users to recover from the DB drop for e.g. 15 minutes? Is it a clear state that dynflow process is dead and needs to be started, is it that something will automatically try to restart the process since it's dead now?

[adamruzicka]

IIUC, this will help if a DB won't be available for cca 1 minute, and if more it will just kill the process now?

Yup, that's about right. The timing isn't exact, but the breaking point between carrying on and crashing hard is somewhere around that 1 minute mark.

is it that something will automatically try to restart the process since it's dead now?

In Foreman deployment, systemd should attempt to restart it, once we move to containers I'd expect again systemd or the container runtime to monitor it and restart if needed. In worst case without anything restarting it, this would at least clearly mark it as being down, rather than leaving it "up" in a broken state.

[ofedoren]

Given the explanation in #427 (comment) and that I'm not much familiar with dynflow internals, I don't have any concerns looking at the code changes themselves.

In other words, LGTM, but how would you like to proceed with the commits before merge?

[adamruzicka]

In other words, LGTM, but how would you like to proceed with the commits before merge?

I'd prefer to get your blessing on #462 , squash and merge that and then rebase this one on top

[ofedoren]

as soon as it's green