Guidance needed for High Availability deployment with two Clusters using Kargo

[WZHMIJJ]

Hello Kargo Team,

I'm seeking assistance with setting up a high availability deployment using the Akuity Kargo tool across two Kubernetes clusters, HA1 and HA2. We are using the Control Flow pattern with stages diverging from service-uat to service-uat-ha1 and service-uat-ha2, eventually leading to production.

Current Setup:

• Clusters: HA1 and HA2
• Deployment Pattern: Control Flow with sharding
• Promotion Task: Deploys first to the passive cluster, then to the active cluster

Issue: The deployment fails during the argocd-update step with the following error:
step "task-1::step-12" met error threshold of 0: error running step "task-1::step-12": error getting Argo CD Application "service-uat-ha2" in namespace "mercury-argocd": unable to find Argo CD Application "service-uat-ha2" in namespace "platform-argocd"

The application appears to be correctly named, but the sharding setup seems to prevent access to the second HA cluster in the ArgoCD step.

Promotion Task Example:

apiVersion: kargo.akuity.io/v1alpha1
kind: PromotionTask
metadata:
  name: ha-promotion
spec:
  vars:
    - name: stageName
    - name: argoCDAppName
    - name: repoURL
    - name: warehouseName
    - name: imageURL
    - name: accountName
    - name: clusterName
    - name: mode # active or passive
    - name: srcPath
    - name: outPath
  steps:
    - uses: git-clone
      config:
        repoURL: ${{ vars.repoURL }}
        checkout:
          - commit: ${{ commitFrom(vars.repoURL, warehouse(vars.warehouseName)).ID }}
            path: ${{ vars.srcPath }}
          - branch: stage/${{ vars.stageName }}
            create: true
            path: ${{ vars.outPath }}
    - uses: git-clear
      config:
        path: ${{ vars.outPath }}
    - uses: yaml-update
      as: update-image
      config:
        path: ${{ vars.srcPath }}/deployment/values.yaml
        updates:
          - key: app.deployment.appImageTag
            value: ${{ imageFrom(vars.imageURL).Tag }}
    - uses: yaml-update
      if: ${{ vars.mode == "active" }}
      as: set-active
      config:
        path: ${{ vars.srcPath }}/deployment/stages/${{ vars.stageName }}/values.yaml
        updates:
          - key: replicaCount
            value: 1
          - key: networkPolicies.enabled
            value: true
    - uses: yaml-update
      if: ${{ vars.mode == "passive" }}
      as: set-passive
      config:
        path: ${{ vars.srcPath }}/deployment/stages/${{ vars.stageName }}/values.yaml
        updates:
          - key: replicaCount
            value: 0
          - key: networkPolicies.enabled
            value: false
    - uses: helm-template
      config:
        releaseName: ${{ vars.warehouseName }}
        path: ${{ vars.srcPath }}/deployment
        valuesFiles:
          - ${{ vars.srcPath }}/deployment/accounts/${{ vars.accountName }}/values.yaml
          - ${{ vars.srcPath }}/deployment/clusters/${{ vars.clusterName }}/values.yaml
          - ${{ vars.srcPath }}/deployment/stages/${{ vars.stageName }}/values.yaml
        outPath: ./tmp/helm/all.yaml
    - uses: copy
      config:
        inPath: ./tmp/helm/all.yaml
        outPath: ${{ vars.outPath }}/all.yaml
    - uses: git-commit
      as: commit
      config:
        path: ${{ vars.outPath }}
        message: rendered manifests for ${{ commitFrom(vars.repoURL).Message }}
    - uses: git-push
      as: push
      config:
        path: ${{ vars.outPath }}
        generateTargetBranch: true

    - uses: git-open-pr
      as: openpr
      config:
        repoURL: ${{ vars.repoURL }}
        provider: github
        createTargetBranch: true
        sourceBranch: ${{ task.outputs.push.branch }}
        targetBranch: stage/${{ vars.stageName }}
    - uses: git-wait-for-pr
      as: wait-for-pr
      config:
        repoURL: ${{ vars.repoURL }}
        provider: github
        prNumber: ${{ task.outputs.openpr.pr.id }}
    - uses: argocd-update
      config:
        apps:
          - name: ${{ vars.argoCDAppName }}

Stage Example:

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: uat-ha1-service

  labels:
    alias: uat-ha1-service
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
    kargo.akuity.io/color: yellow
spec:
  shard: ha1-test
  requestedFreight:
    - origin:
        kind: Warehouse
        name: service-name
      sources:
        stages:
          - uat-service

  promotionTemplate:
    spec:
      vars:
        - name: repoURL
          value: https://www.github.com/project/service-manifest.git
        - name: imageURL
          value: docker.com/docker/project/service-application/service-application
        - name: warehouseName
          value: service
      steps:
        - task:
            name: ha-promotion
          vars:
            - name: accountName
              value: account-test
            - name: clusterName
              value: ha2-test
            - name: argoCDAppName
              value: uat-ha2-service
            - name: stageName
              value: uat-ha2
            - name: mode
              value: passive
            - name: srcPath
              value: ./src-ha2
            - name: outPath
              value: ./out-ha2
        - task:
            name: ha-promotion
          vars:
            - name: accountName
              value: account-test
            - name: clusterName
              value: ha1-test
            - name: argoCDAppName
              value: uat-service
            - name: stageName
              value: uat
            - name: mode
              value: active
            - name: srcPath
              value: ./src-ha1
            - name: outPath
              value: ./out-ha1

Questions:

1. How can I resolve the error related to accessing the Argo CD Application in the second HA cluster?
2. Are there specific configurations or settings required for sharding that might help me solve the issue?
3. Are there any additional best practices for ensuring seamless failover and deployment between two clusters using Kargo?
4. Are there best practices for deploying to one stage and automatically deploying the passive setup to the second cluster?
5. Is there a recommended approach to achieve automatic deployment to the passive cluster in this setup?