Introduce new scenarios testing framework#4895
Conversation
|
|
||
| ext.outputChannel = new TestOutputChannel(); | ||
|
|
||
| registerAppServiceExtensionVariables(ext); |
There was a problem hiding this comment.
TODO: Remove when migrating to es build
| const folderName = path.basename(workspacePath); | ||
| assert.equal(folderName, String(i), `Unexpected workspace folder name "${folderName}".`); | ||
| // const folderName = path.basename(workspacePath); | ||
| // assert.equal(folderName, String(i), `Unexpected workspace folder name "${folderName}".`); |
There was a problem hiding this comment.
Is this assertion needed?
| * │ | ||
| * ┌────────┴────────┐ | ||
| * │ Create New │ | ||
| * │ Project │ |
There was a problem hiding this comment.
I had Copilot help me draft a lot of the documentation / comments. Let me know if it's too overkill, I'm happy to pare it down if needed.
| "AZFUNC_UPDATE_BACKUP_TEMPLATES": "", | ||
| "AzCode_EnableLongRunningTestsLocal": "", | ||
| "AzCode_EnableLongRunningTestsLocal": "true", | ||
| "AzCode_RunScenarioExtended": "durable-azurestorage-jsnode" |
There was a problem hiding this comment.
Todo: Unset these before merging
There was a problem hiding this comment.
The default nightly run would test with AzCode_RunScenarioExtended not set to anything**
|
|
||
| const testScenarios: AzExtFunctionsParallelTestScenario[] = generateParallelScenarios(); | ||
|
|
||
| suite.only('Scenarios', async function (this: Mocha.Suite) { |
There was a problem hiding this comment.
TODO: Remove .only before merging
| /** | ||
| * A wrapper for {@link AzExtFunctionsTestScenario}. Prepares a scenario for concurrent test execution. | ||
| */ | ||
| export interface AzExtFunctionsParallelTestScenario { |
There was a problem hiding this comment.
Not truly "parallel", but felt like just following the naming convention we're already using for concurrent tests. I think it's implicitly understood.
| this.updateTestStatus(deployFunctionAppTest, 'fail', error); | ||
| } | ||
|
|
||
| report(): void { |
There was a problem hiding this comment.
Why not use a standard test reporter? Mocha has many options for that
There was a problem hiding this comment.
Good call, I'll take a closer look at leveraging the Mocha test reporter
2 participants
Overview
This introduces a new test framework that I’d like us to start leveraging to gain better visibility into how the wide range of app configurations impacts deployment health across different project types.
For a deeper dive into the what and why, see the full write-up here:
https://github.com/microsoft/azcode-internal/issues/231
If you’d rather jump straight to a high-level overview of the design, this diagram and the accompanying type definitions capture the core flow of the implementation:
test/nightly/scenarios/testScenarios/AzExtFunctionsTestScenario.tsAs far as I can tell, this change doesn’t introduce any breaking changes to existing tests. That said, there are already some failing tests, so it’s difficult to be completely certain. I think we should definitely do a deeper pass on the older tests—both to fix what’s broken and to evaluate which ones are still necessary versus which could be effectively replaced by the new coverage introduced here.
Output
Basic
Extended
Todos