[Promotion] HTTP Step retry behavior with polling

[AlanLonguet]

Hello !

We're trying to implement an http step in our promotion tasks that do polling following this documentation but we would like to be able to call this http endpoint for example every 30seconds. In our tests we observed that the HTTP call was retried only when a new reconciliation happened which can take up to 5min.

Is there something we're missing ? Or is this intended behavior ?

For context we're using Kargo to promote non-kubernetes workloads and would like to use this http-step to check that the deployment has in fact happened, in a similar fashion to what Kargo is doing out of the box for ArgoCD Applications. But we're hitting this limitation on the promotion process and had to resort to moving this to the verification process which is also a bit sub-optimal for us :/

[krancour]

You're not missing anything. Retry happens on next reconciliation.

In general, our steps, including this one, don't do any internal waiting/polling. The reason for this is so steps that may need to poll for a very long time to find a definitive result aren't bogarting controller resources.

I realize the problem this may pose for some use cases and am open to discussing what can possibly be done about it, but it's definitely not anything as simple as letting the step poll internally.

[AlanLonguet]

Thanks for the swift answer I'm reassured we didn't miss anything !

I understand the concerns for long running steps that could potentially pile up and hold too much resources. In our case we really can't use this polling mechanism as it would add a lot of waiting time to our CD process, the workloads in question are deployed between 15 to 20 times per day which can result in a 1h/1h30 of waiting time in a day.

In terms of solutions we discussed 2 internally that could accommodate us :

• A short-term solution that is still not ideal would be to be able to modify the default requeue delay of each controller : Make Stage controller requeue delay configurable #4969
• In the long run we would like to be able to add a "frequency/period" field inside the top-level retry of every step so we can retry more aggressively on specific steps, I don't see all the implications this change would have but I guess the controllers could instead of the default 5min requeue, requeue the Stage/Promotion resource based on this field WDYT ?

[AlanLonguet]

@krancour I'd be done to implement either if this was something you had in mind for Kargo