How do you all manage kargo at scale and during batch releases?

[JessicaLight]

So we have a regular window of releases around 8am, where all the renovate prs are merged and a flood of releases come through Kargo.

We have noticed that there is only one replica of the kargo-controller. Does anyone know why?

We were struggling with the load due to some polling http calls that got struck for a long time and luckily we found the reconciler values to change it from the default of 4 to much more which has really helped. We did find this discussion - #5092 which would make our lives a lot easier. But wanted to get some tips about how others are managing load at scale? We want to migrate everything to kargo

Reconciler-specific settings

reconcilers:
  maxConcurrentReconciles: 4
  controlFlowStages:
    maxConcurrentReconciles: 10
  promotions:
    maxConcurrentReconciles: 100
  stages:
    maxConcurrentReconciles: 100
  warehouses:
    maxConcurrentReconciles: 10
    minReconciliationInterval: "5m0s"