promotion debugging

[xakraz]

Hello there 👋

I have a question regarding promotionTemplate / promotionStep debugging 😁

I have a promotionTemplate which stays running while all steps are reported done and the last one as failure ...

How is it possible to get more information about what is going on?

[krancour]

What version are you on?

Have you looked at the Promotion's status? Or are you basing your observations only on the UI?

Posting YAML of the Promotion, including it's status would help.

[xakraz]

I am on version 1.7.3 and I have also noticed that the status of the last step is reported as failed in the yaml output of the promotion.

I will reproduce and paste the yaml tomorrow.

Thanks for answering!

[krancour]

fwiw, there was a bug in 1.7.3 that might have made Stages appear to be "reconciling" when they ought not to have been -- especially in the wake of a failed Promotion. It was especially visible in the UI. I mention it due to the likelihood that you saw that and mistook it for a Promotion still being in Progress, when in fact, the Promotion had completed (failed), and the status you were seeing was the status of the Stage.

[xakraz]

I have upgraded to v1.7.4

Here is the status of my Promotion

freight:
  commits:
    - author: **************
      committer: **************
      id: ****************
      message: 'chore: update cert-manager helm chart version from v1.16.0 to v1.17.0'
      repoURL: https://github.com/********
      tag: cert-manager-pr-10
  name: 01a1532de4e5dffd9ddc3c88ba2e7fd6ca12ae20
  origin:
    kind: Warehouse
    name: cert-manager-pr
freightCollection:
  id: 90470f5fbc7a8100f03fd03402c13da718e9fca7
  items:
    Warehouse/cert-manager-pr:
      commits:
        - author: **************
          committer: **************
          id: *********************
          message: 'chore: update cert-manager helm chart version from v1.16.0 to v1.17.0'
          repoURL: https://github.com/*********************
          tag: cert-manager-pr-10
      name: 01a1532de4e5dffd9ddc3c88ba2e7fd6ca12ae20
      origin:
        kind: Warehouse
        name: cert-manager-pr

phase: Running

startedAt: "2025-09-08T10:13:30Z"
state: {}
stepExecutionMetadata:
  - alias: promotion::github-pr-lookup
    startedAt: "2025-09-08T10:13:30Z"
    status: Failed

And the UI reported the Stage as running for 5 minutes ...
=> ❓ Is it possible to tune that duration
=> ❓ Where can I look to see what is going on? Which controllers do what?

[xakraz]

Or the Promotion in "nominal mode"

1st the Promotion looks failed

currentStep: 5
freight:
  commits:
    - author: *********************
      committer: *********************
      id: *********************
      message: 'ci: Add a "global" PR status check for "gating"'
      repoURL: https://github.com/*********************
  name: 3a41a3c07ab7224f2b2916674cce74f36554f606
  origin:
    kind: Warehouse
    name: cert-manager-pr
freightCollection:
  id: 426c1b88d7b77ecdf6d1980d19ed583e7eff91ef
  items:
    Warehouse/cert-manager-pr:
      commits:
        - author: *********************
          committer: *********************
          id: *********************
          message: 'ci: Add a "global" PR status check for "gating"'
          repoURL: https://github.com/*********************
      name: 3a41a3c07ab7224f2b2916674cce74f36554f606
      origin:
        kind: Warehouse
        name: cert-manager-pr
healthChecks:
  - config:
      apps:
        - desiredRevisions:
            - *********************
          name: cert-manager-k3d-testing-1
          namespace: argocd
    uses: argocd-update
phase: Running
startedAt: "2025-09-08T10:35:06Z"
state:
  promotion::github-pr-lookup:
    prId: 2787870082
    prNumber: 15
  promotion::github-ref-lookup:
    commitSha: *********************
  promotion::github-set-commit-status-success: {}
stepExecutionMetadata:
  - alias: promotion::github-pr-lookup
    finishedAt: "2025-09-08T10:35:07Z"
    startedAt: "2025-09-08T10:35:06Z"
    status: Succeeded
  - alias: promotion::github-ref-lookup
    finishedAt: "2025-09-08T10:35:07Z"
    startedAt: "2025-09-08T10:35:07Z"
    status: Succeeded
  - alias: promotion::argocd-update
    finishedAt: "2025-09-08T10:35:32Z"
    startedAt: "2025-09-08T10:35:07Z"
    status: Succeeded
  - alias: promotion::github-set-commit-status-success
    finishedAt: "2025-09-08T10:35:33Z"
    startedAt: "2025-09-08T10:35:32Z"
    status: Succeeded
  - alias: promotion::github-set-commit-status-failure
    status: Skipped
  - alias: promotion::github-post-comment
    startedAt: "2025-09-08T10:35:33Z"
    status: Failed

And finally, it is reported as OK

...
healthChecks:
  - config:
      apps:
        - desiredRevisions:
            - *********************
          name: cert-manager-k3d-testing-1
          namespace: argocd
    uses: argocd-update
phase: Succeeded
startedAt: "2025-09-08T10:35:06Z"
state:
  promotion:
    environmentTargetCommitSha: *********************
    prId: 2787870082
    prNumber: 15
  promotion::github-pr-lookup:
    prId: 2787870082
    prNumber: 15
  promotion::github-ref-lookup:
    commitSha: *********************
  promotion::github-set-commit-status-success: {}
  promotion::output:
    environmentTargetCommitSha: *********************
    prId: 2787870082
    prNumber: 15
stepExecutionMetadata:
  - alias: promotion::github-pr-lookup
    finishedAt: "2025-09-08T10:35:07Z"
    startedAt: "2025-09-08T10:35:06Z"
    status: Succeeded
  - alias: promotion::github-ref-lookup
    finishedAt: "2025-09-08T10:35:07Z"
    startedAt: "2025-09-08T10:35:07Z"
    status: Succeeded
  - alias: promotion::argocd-update
    finishedAt: "2025-09-08T10:35:32Z"
    startedAt: "2025-09-08T10:35:07Z"
    status: Succeeded
  - alias: promotion::github-set-commit-status-success
    finishedAt: "2025-09-08T10:35:33Z"
    startedAt: "2025-09-08T10:35:32Z"
    status: Succeeded
  - alias: promotion::github-set-commit-status-failure
    status: Skipped
  - alias: promotion::github-post-comment
    startedAt: "2025-09-08T10:35:33Z"
    status: Skipped
  - alias: promotion::output
    finishedAt: "2025-09-08T10:40:07Z"
    startedAt: "2025-09-08T10:40:07Z"
    status: Succeeded

The failed step is now reported as "skipped"

Despite having a condition set to if: ${{ success() }} like the other one before it...

Here is the full Promotion

apiVersion: kargo.akuity.io/v1alpha1
kind: Promotion
metadata:
  creationTimestamp: 2025-09-08T10:35:06Z
  generation: 1
  name: cert-manager-testing.01k4me9jcs9j8nbvhhqdazca67.3a41a3c
  namespace: kargo-cluster-components-tooling
  ownerReferences:
    - apiVersion: kargo.akuity.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Stage
      name: cert-manager-testing
      uid: 7c491be5-1e0b-4762-ae57-597d578f767c
  resourceVersion: "3826"
  uid: d3d87598-8531-4c9b-b0d6-0adcd806d0ec
spec:
  freight: 3a41a3c07ab7224f2b2916674cce74f36554f606
  stage: cert-manager-testing
  steps:
    - as: promotion::github-pr-lookup
      config:
        headers:
          - name: Authorization
            value: Bearer ${{ vars.githubToken }}
          - name: Accept
            value: application/vnd.github+json
        method: GET
        outputs:
          - fromExpression: response.body[0].number
            name: prNumber
          - fromExpression: response.body[0].id
            name: prId
        url: https://api.github.com/repos/${{ vars.repoOwner }}/${{ vars.repoName
          }}/commits/${{ vars.parentPrCommitId }}/pulls
      uses: http
      vars:
        - name: repoOwner
          value: ${{ vars.repoOwner }}
        - name: repoName
          value: ${{ vars.repoName }}
        - name: githubToken
          value: ${{ vars.githubToken }}
        - name: parentPrCommitId
          value: ${{ commitFrom( vars.repoUrl ).ID }}
        - name: envBranch
          value: ${{ vars.envBranch }}
        - name: appName
          value: ${{ vars.appName }}
        - name: repoUrl
          value: ${{ vars.repoUrl }}
        - name: gitHubStatusTargetUrl
          value: ${{ vars.gitHubStatusTargetUrl }}
        - name: githubStatusContext
          value: ${{ vars.githubStatusContext }}
    - as: promotion::github-ref-lookup
      config:
        headers:
          - name: Authorization
            value: Bearer ${{ vars.githubToken }}
          - name: Accept
            value: application/vnd.github+json
        method: GET
        outputs:
          - fromExpression: response.body.object.sha
            name: commitSha
        url: https://api.github.com/repos/${{ vars.repoOwner }}/${{ vars.repoName
          }}/git/refs/heads/${{ vars.envBranch }}_pr-${{
          task.outputs['github-pr-lookup'].prNumber }}
      uses: http
      vars:
        - name: repoOwner
          value: ${{ vars.repoOwner }}
        - name: repoName
          value: ${{ vars.repoName }}
        - name: githubToken
          value: ${{ vars.githubToken }}
        - name: parentPrCommitId
          value: ${{ commitFrom( vars.repoUrl ).ID }}
        - name: envBranch
          value: ${{ vars.envBranch }}
        - name: appName
          value: ${{ vars.appName }}
        - name: repoUrl
          value: ${{ vars.repoUrl }}
        - name: gitHubStatusTargetUrl
          value: ${{ vars.gitHubStatusTargetUrl }}
        - name: githubStatusContext
          value: ${{ vars.githubStatusContext }}
    - as: promotion::argocd-update
      config:
        apps:
          - name: ${{ vars.appName }}
            sources:
              - desiredRevision: ${{ task.outputs['github-ref-lookup'].commitSha }}
                repoURL: ${{ vars.repoUrl }}
                updateTargetRevision: true
      uses: argocd-update
      vars:
        - name: repoOwner
          value: ${{ vars.repoOwner }}
        - name: repoName
          value: ${{ vars.repoName }}
        - name: githubToken
          value: ${{ vars.githubToken }}
        - name: parentPrCommitId
          value: ${{ commitFrom( vars.repoUrl ).ID }}
        - name: envBranch
          value: ${{ vars.envBranch }}
        - name: appName
          value: ${{ vars.appName }}
        - name: repoUrl
          value: ${{ vars.repoUrl }}
        - name: gitHubStatusTargetUrl
          value: ${{ vars.gitHubStatusTargetUrl }}
        - name: githubStatusContext
          value: ${{ vars.githubStatusContext }}
    - as: promotion::github-set-commit-status-success
      config:
        body: |
          ${{ quote({
            "context": vars.githubStatusContext,
            "target_url": vars.gitHubStatusTargetUrl,
            "state": "success",
            "description": "Promotion succeeded"
          }) }}
        headers:
          - name: Authorization
            value: Bearer ${{ vars.githubToken }}
          - name: Accept
            value: application/vnd.github+json
        method: POST
        url: https://api.github.com/repos/${{ vars.repoOwner }}/${{ vars.repoName
          }}/statuses/${{ vars.parentPrCommitId }}
      if: ${{ success() }}
      uses: http
      vars:
        - name: repoOwner
          value: ${{ vars.repoOwner }}
        - name: repoName
          value: ${{ vars.repoName }}
        - name: githubToken
          value: ${{ vars.githubToken }}
        - name: parentPrCommitId
          value: ${{ commitFrom( vars.repoUrl ).ID }}
        - name: envBranch
          value: ${{ vars.envBranch }}
        - name: appName
          value: ${{ vars.appName }}
        - name: repoUrl
          value: ${{ vars.repoUrl }}
        - name: gitHubStatusTargetUrl
          value: ${{ vars.gitHubStatusTargetUrl }}
        - name: githubStatusContext
          value: ${{ vars.githubStatusContext }}
    - as: promotion::github-set-commit-status-failure
      config:
        body: |
          ${{ quote({
            "context": vars.githubStatusContext,
            "target_url": vars.gitHubStatusTargetUrl,
            "state": "failure",
            "description": "Promotion failed"
          }) }}
        headers:
          - name: Authorization
            value: Bearer ${{ vars.githubToken }}
          - name: Accept
            value: application/vnd.github+json
        method: POST
        url: https://api.github.com/repos/${{ vars.repoOwner }}/${{ vars.repoName
          }}/statuses/${{ vars.parentPrCommitId }}
      if: ${{ failure() }}
      uses: http
      vars:
        - name: repoOwner
          value: ${{ vars.repoOwner }}
        - name: repoName
          value: ${{ vars.repoName }}
        - name: githubToken
          value: ${{ vars.githubToken }}
        - name: parentPrCommitId
          value: ${{ commitFrom( vars.repoUrl ).ID }}
        - name: envBranch
          value: ${{ vars.envBranch }}
        - name: appName
          value: ${{ vars.appName }}
        - name: repoUrl
          value: ${{ vars.repoUrl }}
        - name: gitHubStatusTargetUrl
          value: ${{ vars.gitHubStatusTargetUrl }}
        - name: githubStatusContext
          value: ${{ vars.githubStatusContext }}
    - as: promotion::github-post-comment
      config:
        body: >-
          Promotion stage '${{ ctx.stage }}' has been applied successfully!


          You can run the following command to promote the changes to the next
          stage:

          ```

          /kargo promote --project ${{ ctx.project }} --downstream-from=${{
          ctx.stage }} --freight=${{ ctx.targetFreight.name }}

          ```
        headers:
          - name: Authorization
            value: Bearer ${{ vars.githubToken }}
          - name: Accept
            value: application/vnd.github+json
        method: POST
        url: https://api.github.com/repos/${{ vars.repoOwner }}/${{ vars.repoName
          }}/issues/${{ task.outputs['github-pr-lookup'].prNumber }}/comments \
      if: ${{ success() }}
      uses: http
      vars:
        - name: repoOwner
          value: ${{ vars.repoOwner }}
        - name: repoName
          value: ${{ vars.repoName }}
        - name: githubToken
          value: ${{ vars.githubToken }}
        - name: parentPrCommitId
          value: ${{ commitFrom( vars.repoUrl ).ID }}
        - name: envBranch
          value: ${{ vars.envBranch }}
        - name: appName
          value: ${{ vars.appName }}
        - name: repoUrl
          value: ${{ vars.repoUrl }}
        - name: gitHubStatusTargetUrl
          value: ${{ vars.gitHubStatusTargetUrl }}
        - name: githubStatusContext
          value: ${{ vars.githubStatusContext }}
    - as: promotion::output
      config:
        environmentTargetCommitSha: ${{ task.outputs['github-ref-lookup'].commitSha }}
        prId: ${{ task.outputs['github-pr-lookup'].prId }}
        prNumber: ${{ task.outputs['github-pr-lookup'].prNumber }}
      uses: compose-output
      vars:
        - name: repoOwner
          value: ${{ vars.repoOwner }}
        - name: repoName
          value: ${{ vars.repoName }}
        - name: githubToken
          value: ${{ vars.githubToken }}
        - name: parentPrCommitId
          value: ${{ commitFrom( vars.repoUrl ).ID }}
        - name: envBranch
          value: ${{ vars.envBranch }}
        - name: appName
          value: ${{ vars.appName }}
        - name: repoUrl
          value: ${{ vars.repoUrl }}
        - name: gitHubStatusTargetUrl
          value: ${{ vars.gitHubStatusTargetUrl }}
        - name: githubStatusContext
          value: ${{ vars.githubStatusContext }}
  vars:
    - name: argoCdServerURL
      value: https://argocd.tooling.datadome.internal
    - name: appName
      value: cert-manager-k3d-testing-1
    - name: repoOwner
      value: DataDome
    - name: repoName
      value: test-infra-eks-updates-promotion
    - name: githubToken
      value: ${{ secret('kargo-ghapp-token').token }}
    - name: envBranch
      value: environments/tooling/k3d-testing-1
    - name: repoUrl
      value: https://github.com/${{ vars.repoOwner }}/${{ vars.repoName }}.git
    - name: component
      value: ${{ replace(ctx.stage, "-testing", "") }}
    - name: environment
      value: ${{ replace(vars.envBranch, "environments/", "") }}
    - name: githubStatusContext
      value: Promotion/${{ vars.component }}/${{ vars.environment }}
    - name: gitHubStatusTargetUrl
      value: ${{ vars.argoCdServerURL }}/applications/${{ vars.appName }}
status:
  currentStep: 6
  finishedAt: 2025-09-08T10:40:07Z
  freight:
    commits:
      - author: *********************
        committer: *********************
        id: *********************
        message: 'ci: Add a "global" PR status check for "gating"'
        repoURL: https://github.com/*********************
        tag: cert-manager-pr-15
    name: 3a41a3c07ab7224f2b2916674cce74f36554f606
    origin:
      kind: Warehouse
      name: cert-manager-pr
  freightCollection:
    id: 426c1b88d7b77ecdf6d1980d19ed583e7eff91ef
    items:
      Warehouse/cert-manager-pr:
        commits:
          - author: *********************
            committer: *********************
            id: *********************
            message: 'ci: Add a "global" PR status check for "gating"'
            repoURL: https://github.com/*********************
            tag: cert-manager-pr-15
        name: 3a41a3c07ab7224f2b2916674cce74f36554f606
        origin:
          kind: Warehouse
          name: cert-manager-pr
  healthChecks:
    - config:
        apps:
          - desiredRevisions:
              - *********************
            name: cert-manager-k3d-testing-1
            namespace: argocd
      uses: argocd-update
  phase: Succeeded
  startedAt: 2025-09-08T10:35:06Z
  state:
    promotion:
      environmentTargetCommitSha: *********************
      prId: 2787870082
      prNumber: 15
    promotion::github-pr-lookup:
      prId: 2787870082
      prNumber: 15
    promotion::github-ref-lookup:
      commitSha: *********************
    promotion::github-set-commit-status-success: {}
    promotion::output:
      environmentTargetCommitSha: *********************
      prId: 2787870082
      prNumber: 15
  stepExecutionMetadata:
    - alias: promotion::github-pr-lookup
      finishedAt: 2025-09-08T10:35:07Z
      startedAt: 2025-09-08T10:35:06Z
      status: Succeeded
    - alias: promotion::github-ref-lookup
      finishedAt: 2025-09-08T10:35:07Z
      startedAt: 2025-09-08T10:35:07Z
      status: Succeeded
    - alias: promotion::argocd-update
      finishedAt: 2025-09-08T10:35:32Z
      startedAt: 2025-09-08T10:35:07Z
      status: Succeeded
    - alias: promotion::github-set-commit-status-success
      finishedAt: 2025-09-08T10:35:33Z
      startedAt: 2025-09-08T10:35:32Z
      status: Succeeded
    - alias: promotion::github-set-commit-status-failure
      status: Skipped
    - alias: promotion::github-post-comment
      startedAt: 2025-09-08T10:35:33Z
      status: Skipped
    - alias: promotion::output
      finishedAt: 2025-09-08T10:40:07Z
      startedAt: 2025-09-08T10:40:07Z
      status: Succeeded

It is a bit confusing 😅

[xakraz]

@krancour , any thoughts?

[xakraz]

After some digging, I was able to fix my issue 🎉

1 - Increase logs verbosity on the kargo-controller

📝 I had to update the Kargo-controller logLevel from INFO → TRACE to be able to get useful information regarding the PromotionStep process

2 - Fix your problem 😅

Once the log level increased, I got the error message

time="2025-09-16T10:46:43Z" level=trace msg="HTTP request response" body="{\"message\":\"Problems parsing JSON\",\"documentation_url\":\"https://docs.github.com/rest/issues/comments#create-an-issue-comment\",\"status\":\"400\"}" freight=**************** header="{\"Access-Control-Allow-Origin\":[\"*\"],\"Access-Control-Expose-Headers\":[\"ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset\"],\"Content-Length\":[\"141\"],\"Content-Security-Policy\":[\"default-src 'none'\"],\"Content-Type\":[\"application/json; charset=utf-8\"],\"Date\":[\"Tue, 16 Sep 2025 10:46:43 GMT\"],\"Referrer-Policy\":[\"origin-when-cross-origin, strict-origin-when-cross-origin\"],\"Server\":[\"github.com\"],\"Strict-Transport-Security\":[\"max-age=31536000; includeSubdomains; preload\"],\"Vary\":[\"Accept-Encoding, Accept, X-Requested-With\"],\"X-Accepted-Github-Permissions\":[\"issues=write; pull_requests=write\"],\"X-Content-Type-Options\":[\"nosniff\"],\"X-Frame-Options\":[\"deny\"],\"X-Github-Api-Version-Selected\":[\"2022-11-28\"],\"X-Github-Media-Type\":[\"github.v3; format=json\"],\"X-Github-Request-Id\":[\"9D3D:3D11E5:324CFE6:2F903D0:68C94013\"],\"X-Ratelimit-Limit\":[\"15000\"],\"X-Ratelimit-Remaining\":[\"14988\"],\"X-Ratelimit-Reset\":[\"1758020951\"],\"X-Ratelimit-Resource\":[\"core\"],\"X-Ratelimit-Used\":[\"12\"],\"X-Xss-Protection\":[\"0\"]}" namespace=kargo-cluster-components-tooling promotion=cert-manager-testing.01k58yv4ej4sx1rk6zf4ekn7kz.2a5846f stage=cert-manager-testing status=400

And was able to fix my PromotionTask

[xakraz]

Nobody has a hint to suggest 😢