Switch test run to ai-json parsed set of applications

[ScrewTSW]

What does this PR do?

• Adds new approach to running tests based on ai.json
• Fixes model download timeouts inconsistency

Screenshot / video of UI

What issues does this PR fix or reference?

Closes #3333

How to test this PR?

[jeffmaury]

See comments

[cbr7]

I'm not sure I understand what this PR is about? What is in the ai.json file that is being read?

[ScrewTSW]

I'm not sure I understand what this PR is about? What is in the ai.json file that is being read?

All of the available inference servers, models and demo apps

[odockal]

This just creates objects with models, services, etc. It does not yet uses them.?

[cbr7]

I'm with @odockal on this, the data doesn't seem to be getting used for anything currently.

Also, what is the end goal of the data that will be read from the file? What should it be used for?

[ScrewTSW]

This just creates objects with models, services, etc. It does not yet uses them.?

good catch, not sure how, but it slipped out when I was rewriting it from a diff

[odockal]

I do not have review like itself atm. It is a breaking change, we have discussed maybe even better approach to start with models first aside of running recipes which "contains" almost identical e2e functionality we might also cover individually (download model + test service is also done when running whole recipe, etc).
I am also waiting for PR check to pass reliably.
BUT. It is quite great to see the amount of tests now being run vs. before

before:

Let's proceed with the PR. and talk about possible follow ups.

[ScrewTSW]

@odockal fully passing now, runtime 34 minutes pr-check


e2e-test-main


passed on Windows too, but it's painfully slow.
the runner section on Azure took 1 hour 8 minutes

[danivilla9]

LGTM, I can't find anything to improve. My test run on a windows machine timed out when trying to download a model, but I see that you managed to run them properly on the CI, so I suppose it's some setup issue on my end.
One tiny nitpick, any reason not to use states.ts from the PD core? Not sure if it's possible to do it from an extension repo, but I think it should.

[bmahabirbu]

Nice work! Id like to be a part of the conversation for enhancing the ai lab tests as well

[ScrewTSW]

LGTM, I can't find anything to improve. My test run on a windows machine timed out when trying to download a model, but I see that you managed to run them properly on the CI, so I suppose it's some setup issue on my end. One tiny nitpick, any reason not to use states.ts from the PD core? Not sure if it's possible to do it from an extension repo, but I think it should.

Model download speeds can fluctuate a lot indeed, it's just something that seems to be a limitation of huggingface.
Thank you for the note about states.ts, would it be sufficient to implement that change in the subsequent PRs?

#3334
#3335
#3336