/kind feature
Describe the solution you'd like
The goal is to fix certain edge cases where the Hetzner server doesn't boot or boots wrongly and we don't manage to reach the rescue system in time via ssh.
If we don't manage, then we want to try again by doing a cycle of ensuring the rescue system is activated, and then powering the server off and back on.
Ideally, this fixes problems that the server had previously - basically the classic IT solution of restarting.
There are two different triggers for this cycle: first, we reach a timeout of e.g. 45 sec, and second, we reach the server via ssh but are not in the rescue system.
Both have nothing to do with each other. The first is currently handled in a way that we wait for 6 minutes and then mark the machine as unhealthy, the second is handled in a way that we directly mark the machine as unhealthy.
Instead of doing that, which would lead to a remediation, we can react faster (in the first case) and smarter (in both cases).
I think it is safe to assume that we'll always manage to boot the server into rescue when we are in the second case.
In the first case, if we were not successful and waited e.g. for another 45 sec, we can mark the machine as unhealthy.
Some implementation notes:
I would try to use the same pattern we already have. First, we do the action, which should be the powering off of the server. Then, we change the state to another one (a new one), requeue and once we reconcile, go to the handle function of the new state.
This state should first check whether the server is powered off (via action or getServer?), then ensure rescue is enabled on the server (via getServer), and then power the server on again. We should be able to handle everything easily in one state, because we can store in the status the actionIDs and therefore know whether e.g. we are currently activating rescue or powering off the server.
After we triggered the powering on, we should again go to the existing state "handleStateBootingToRescue".
Anything else you would like to add:
There is a PR (#2207) that attempted to implement something similar. It doesn't follow the same mechanism described in this issue and should not be taken as source of truth. However, it might give an idea for the implementation.
/kind feature
Describe the solution you'd like
The goal is to fix certain edge cases where the Hetzner server doesn't boot or boots wrongly and we don't manage to reach the rescue system in time via ssh.
If we don't manage, then we want to try again by doing a cycle of ensuring the rescue system is activated, and then powering the server off and back on.
Ideally, this fixes problems that the server had previously - basically the classic IT solution of restarting.
There are two different triggers for this cycle: first, we reach a timeout of e.g. 45 sec, and second, we reach the server via ssh but are not in the rescue system.
Both have nothing to do with each other. The first is currently handled in a way that we wait for 6 minutes and then mark the machine as unhealthy, the second is handled in a way that we directly mark the machine as unhealthy.
Instead of doing that, which would lead to a remediation, we can react faster (in the first case) and smarter (in both cases).
I think it is safe to assume that we'll always manage to boot the server into rescue when we are in the second case.
In the first case, if we were not successful and waited e.g. for another 45 sec, we can mark the machine as unhealthy.
Some implementation notes:
I would try to use the same pattern we already have. First, we do the action, which should be the powering off of the server. Then, we change the state to another one (a new one), requeue and once we reconcile, go to the handle function of the new state.
This state should first check whether the server is powered off (via action or getServer?), then ensure rescue is enabled on the server (via getServer), and then power the server on again. We should be able to handle everything easily in one state, because we can store in the status the actionIDs and therefore know whether e.g. we are currently activating rescue or powering off the server.
After we triggered the powering on, we should again go to the existing state "handleStateBootingToRescue".
Anything else you would like to add:
There is a PR (#2207) that attempted to implement something similar. It doesn't follow the same mechanism described in this issue and should not be taken as source of truth. However, it might give an idea for the implementation.