Allow non active frights to run analysis Runs in the background.

[tal-hason]

When a freight is promoted to a stage and the testing fails, and during this time, a new freight arrives, the previous freight can't be checked again if the test is restarted.

The idea is to enable a freight in a stage to run analysis Runs, even if the freight isn't active, and if the analysis passes, approve the freight for promotion

[tal-hason]

@krancour, any opinion on the idea?

[krancour]

@tal-hason what is a "non-active Freight?"

My only guess is that you mean Freight that isn't currently used by some Stage.

If I'm understanding you correctly, I'm not sure how what you're proposing could possibly work. I avoid using the word "deploy," because Kargo is about promotions, but I'll break my rule for a moment: Assuming a piece of Freight is no longer in use (i.e. its been "promoted over" by something newer) then whatever artifacts are referenced by that piece of Freight are not longer deployed anywhere. How exactly could tests continue to execute against them?

[tal-hason]

HI @krancour , we are using the stage freight history as a magazine. Our QE's automated testing is geting the version that they need to deploy and run tests on them, then they need to promote the build that passes to the next stage.

So while the latest freight is the active one, in real life, we sometimes have 2-3 builds running at the test lanes, which have been promoted to testing.

The automated test can take from 1-2 hours up to 12 hours, and if they need to manually restart a test due to automation failure, this can also take more time.

So, the freights that are in the QE stage that did get “deployed/delivered for testing”, if I could run a background check to see if the failures were analyzed, basically, instead of the promotion, will need to wait for a manual waiver of this test. The freight analysis run can check the test results to see what the state is, and then approve, and maybe promote to the next stage....

[krancour]

None of this answers the question of how you expect to run tests against something not currently deployed.

[tal-hason]

To clarify, our testing process is a deployment. For every test run, we spin up a full cluster from scratch (including operators) to validate the artifact.

Because these test environments are ephemeral and isolated, we can process multiple pieces of Freight in parallel. For example, in a 24-hour period, we might have 2-4 different artifacts running tests simultaneously on their own clusters.

As soon as an artifact passes, it gets promoted to the next stage. This means that while the 'Active Freight' shows the most recent artifact entering the pipeline, there may be several earlier artifacts that have already been promoted and are still finishing their tests in parallel.

[krancour]

I'm sorry. None of this makes any sense.

[tal-hason]

Apologies for the confusion. Let me try to clarify the architecture with a diagram, as it differs significantly from deploying to a single, static cluster.

We are using this Kargo Stage to manage Parallel Ephemeral Environments.

The critical difference is that our "deployment" action doesn't sync to one persistent cluster. Instead, it triggers external automation that spins up a brand new, isolated cluster just for that specific Freight. Because these clusters are isolated, multiple Freights can be running their tests at the same time without conflict.

We want to leverage Kargo's existing AnalysisRun feature to poll the status of these ephemeral clusters to determine when a Freight is ready for promotion.

Here is the flow of parallel deployments over time:

%%{init: { 'theme': 'base', 'themeVariables': { 'textColor': '#000000', 'actorTextColor': '#000000', 'loopTextColor': '#000000', 'noteTextColor': '#000000', 'messageTextColor': '#000000' } } }%%
sequenceDiagram
    autonumber
    participant W as Warehouse
    participant KS as Kargo Stage
    participant Auto as Infrastructure Automation
    participant C1 as Ephemeral Cluster 1 (Freight A)
    participant C2 as Ephemeral Cluster 2 (Freight B)
    participant AR as Kargo AnalysisRun

    rect rgb(240, 248, 255)
    note right of W: **Time: 10:00 AM**
    W->>KS: Freight A detected
    KS->>Auto: Trigger test environment for A
    par Provisioning A
        Auto->>C1: Provision Cluster & Deploy A
        activate C1
    and Kargo Checks A
        KS->>AR: Start AnalysisRun for A
        activate AR
        AR->>C1: Poll status... (Running)
    end
    end

    rect rgb(255, 250, 240)
    note right of W: **Time: 10:30 AM** (Cluster 1 still busy)
    W->>KS: Freight B detected
    KS->>Auto: Trigger test environment for B
    par Provisioning B
        Auto->>C2: Provision Cluster & Deploy B
        activate C2
    and Kargo Checks B
        KS->>AR: Start AnalysisRun for B
        activate AR
        AR->>C2: Poll status... (Running)
    end
    end
    
    rect rgb(255, 255, 255)
    note right of W: **Parallel State:** Two separate clusters running simultaneously.
    C1-->>C1: Running Tests (Freight A)
    C2-->>C2: Running Tests (Freight B)
    end

    rect rgb(240, 255, 240)
    note right of W: **Time: 11:00 AM**
    C1->>C1: Tests Passed!
    AR->>C1: Poll status... (Success)
    C1-->>Auto: Deprovision Cluster 1
    deactivate C1
    AR->>KS: Report Freight A success
    deactivate AR
    Note over KS: Freight A promoted to next stage
    end
    
    rect rgb(240, 255, 240)
    note right of W: **Time: 11:30 AM**
    C2->>C2: Tests Passed!
    AR->>C2: Poll status... (Success)
    C2-->>Auto: Deprovision Cluster 2
    deactivate C2
    AR-->>KS: Report Freight B success
    deactivate AR
    Note over KS: Freight B promoted to next stage
    end

Loading

The issue we are facing is that when Freight B arrives (at 10:30 AM), Kargo marks it as the "current" freight, and seems to lose track of the fact that Freight A is still actively running in its own environment and needs its AnalysisRun to complete.

[krancour]

This falls so very far outside the range of what we built for that I frankly don't see anything realistic that can be done for you here.