Should Stage health evaluated be a ERROR in the kargo-controller?

[nissessenap]

Hi

Once again, thanks for a great project.

I'm not enabling the ArgoCD integration with Kargo, mostly due to I have environments that are air-gapped, so I can't reach my ArgoCD instances. So instead I just see Kargo as a GitOps tools.

The kargo-controller shows me lots of errors like this:

time="2025-09-10T08:09:13Z" level=error msg="Reconciler error" Stage="{customer-exports-staging customer-exports}" controller=stage controllerGroup=kargo.akuity.io controllerKind=Stage error="Stage health evaluated to Unknown" name=customer-exports-staging namespace=customer-exports reconcileID="\"ad800301-2af5-45d0-9eaa-22a51483ff61\""

I assume it is since I don't use the argocd-update feature.
https://docs.kargo.io/user-guide/reference-docs/promotion-steps/argocd-update/#health-checks

In my this should be an info, or even if the ArgoCD feature isn't enabled, it should only be debug.
Happy to create a PR around this if you agree.

[krancour]

It's been the case for some time that if Kargo doesn't know how to check on the Stage's health in some way, it treats the health as Unkown. Without Argo CD in the picture, there really is no way to know.

It was fairly recently that we began treating Unkown health as an error under some specific circumstances.

If it was Unkown and unknowable, it was not considered an error. It it was Unkown, but knowable, (i.e. we just failed to conclusively make a determination) we treat it as an error because that gives us automatic retries with a progressive backoff for free.

I think the problem here is that we failed to treat the case you're describing as one where health is unknowable and therefor not an error.

I will create an issue for this.

[krancour]

Closing as it is now tracked in #5045.

[krancour]

Re-opening this discussion. Explained in detail here: #5045 (comment)

I weighted this statement in #5024 too heavily:

I assume it is since I don't use the argocd-update feature.

Which led me to misspeak:

It's been the case for some time that if Kargo doesn't know how to check on the Stage's health in some way, it treats the health as Unkown. Without Argo CD in the picture, there really is no way to know.

This is actually not true. Had I double-checked the code, I'd have been reminded that Stages without any health checks defined (i.e. none created as a side effect of the argocd-update step) actually default to Healthy. (They must, or else verification is blocked indefinitely.)

Since this whole issue was based on a false assumption, I'm closing this as well as #5093.

I will re-open #5024 and see if we can't get to the bottom of what the real problem was.

@nissessenap there must have been some other issue at work. If you're still experiencing this problem, could you please examine the Stage's status to find more details about why health is unknown? It shouldn't be unknown without a detailed reason accompanying it.

[nissessenap]

I currently don't have any kargo instance up and running. But I will try to find an easy way to reproduce the error as soon as I get the time.