|
| 1 | +# Swift for Visual Studio Code test strategy |
| 2 | + |
| 3 | +## Overview |
| 4 | + |
| 5 | +The recommended way for [testing extensions](https://code.visualstudio.com/api/working-with-extensions/testing-extension) involves using either the new [vscode-test-cli](https://github.com/microsoft/vscode-test-cli) or creating your [own mocha test runner](https://code.visualstudio.com/api/working-with-extensions/testing-extension#advanced-setup-your-own-runner). Either approach results in Visual Studio Code getting downloaded, and a window spawned. This is necessary to have access the the APIs of the `vscode` namespace, to stimulate behaviour (ex. `vscode.tasks.executeTasks`) and obtain state (ex. `vscode.languages.getDiagnostics`). |
| 6 | + |
| 7 | +There are some testing gaps when only using this approach. Relying on using the `vscode` APIs makes it difficult to easily write unit tests. It ends up testing the communication between a lot of components in the `vscode-swift` extension and associated dependencies. Additionally, there is a lot of code that remains unverified. This code may get executed so that it shows up in the coverage report, but the behaviour is unobserved. Some examples of behaviour that is not observed, includes prompting the user for input, or verifying if a notification gets shown. See https://devblogs.microsoft.com/ise/testing-vscode-extensions-with-typescript/ for a more detailed overview. |
| 8 | + |
| 9 | +In addition to gaps in testing, the current approach tests at the integration level which results in slower and more brittle tests, as they rely on the communication between several components. |
| 10 | + |
| 11 | +## Unit testing |
| 12 | + |
| 13 | +For unit testing [ts-mockito](https://github.com/NagRock/ts-mockito) is used for mocking out user interaction and observing behaviour that cannot otherwise be observed. Many helpful utility functions exist in [MockUtils.ts](../../test/unit-tests/MockUtils.ts). These utility methods take care of the setup and teardown of the mocks so the developer does not need to remember to do this for each suite/test. |
| 14 | + |
| 15 | +### Mocking `vscode` namespace |
| 16 | + |
| 17 | +The `mockNamespace` function provides a mocked implementation of one of the `vscode` API namespaces. Below is an example of how to employ mocking to test if the [showReloadExtensionNotification](../../src/ui/ReloadExtension.ts) function shows a notification and mock the button click. |
| 18 | + |
| 19 | +```ts |
| 20 | +suite("ReloadExtension Unit Test Suite", async function () { |
| 21 | + const windowMock = mockNamespace(vscode, "window"); |
| 22 | + const commandsMock = mockNamespace(vscode, "commands"); |
| 23 | + |
| 24 | + test('"Reload Extensions" is clicked', async () => { |
| 25 | + // What happens if they click this button? |
| 26 | + when(windowMock.showWarningMessage(anyString(), "Reload Extensions")).thenReturn( |
| 27 | + Promise.resolve("Reload Extensions") |
| 28 | + ); |
| 29 | + await showReloadExtensionNotification("Want to reload?"); |
| 30 | + verify(commandsMock.executeCommand("workbench.action.reloadWindow")).called(); |
| 31 | + }); |
| 32 | +}); |
| 33 | +``` |
| 34 | + |
| 35 | +### Mocking event emitter |
| 36 | + |
| 37 | +The `eventListenerMock` function captures components listening for a given event and fires the event emitter with the provided test data. Below is an example of mocking the `onDidStartTask` event. |
| 38 | + |
| 39 | +```ts |
| 40 | +suite("Event emitter example", async function () { |
| 41 | + const listenerMock = eventListenerMock(vscode.tasks, "onDidStartTask"); |
| 42 | + |
| 43 | + test("Fire event", async () => { |
| 44 | + const mockedTask = mock(vscode.Task); |
| 45 | + mockedTaskExecution = { task: instance(mockedTask), terminate: () => {} }; |
| 46 | + |
| 47 | + listenerMock.notifyAll({ execution: mockedTaskExecution }); |
| 48 | + }); |
| 49 | +}); |
| 50 | +``` |
| 51 | + |
| 52 | +### Mocking global variables |
| 53 | + |
| 54 | +The `globalVariableMock` function allows for overriding the value for some global constant. |
| 55 | + |
| 56 | +```ts |
| 57 | +suite("Environment variable example", async function () { |
| 58 | + const envMock = globalVariableMock(process, "env"); |
| 59 | + |
| 60 | + test("Linux", async () => { |
| 61 | + env.setValue({ DEVELOPER_DIR: '/path/to/Xcode.app' }); |
| 62 | + |
| 63 | + // Test DEVELOPER_DIR usage |
| 64 | + }); |
| 65 | +}); |
| 66 | +``` |
| 67 | + |
| 68 | +It can also be used to mock the extension [configuration](../../src/configuration.ts). |
| 69 | + |
| 70 | +```ts |
| 71 | +import configuration from "../../../src/configuration"; |
| 72 | +suite("SwiftBuildStatus Unit Test Suite", async function () { |
| 73 | + const statusConfig = globalVariableMock(configuration, "showBuildStatus"); |
| 74 | + |
| 75 | + test("Shows notification", async () => { |
| 76 | + statusConfig.setValue("notification"); |
| 77 | + |
| 78 | + // Test shows as notification |
| 79 | + }); |
| 80 | + |
| 81 | + test("Shows status bar", async () => { |
| 82 | + statusConfig.setValue("swiftStatus"); |
| 83 | + |
| 84 | + // Test shows in status bar |
| 85 | + }); |
| 86 | +}); |
| 87 | +``` |
| 88 | + |
| 89 | +## Test Pyramid |
| 90 | + |
| 91 | +Tests are grouped into 3 levels. The biggest distinguishing factors between the various levels will be the runtime of the test, and the number of "real" vs. mocked dependencies. |
| 92 | + |
| 93 | +### 1. Unit (`/test/unit`) |
| 94 | + |
| 95 | +- Employ stubbing or mocking techniques to allow for user interaction, AND to mock slow APIs like `executeTask` |
| 96 | +- Mocked SwiftPM commands return hardcoded output, such as compile errors |
| 97 | +- Any sourcekit-lsp interaction is mocked, with hardcoded responses |
| 98 | +- Runs with a fast timeout of 100ms |
| 99 | +- No usages of assets/test projects |
| 100 | + - Use [mock-fs](https://www.npmjs.com/package/mock-fs) for testing fs usage |
| 101 | +- Run in CI build for new PRs |
| 102 | +- Ideally the vast majority of tests are at this level |
| 103 | + |
| 104 | +### 2. Integration (`/test/integration`) |
| 105 | + |
| 106 | +- Tests interaction between components, with some mocking for slow or fragile dependencies |
| 107 | +- Stimulate actions using the VS Code APIs |
| 108 | +- Use actual output from SwiftPM |
| 109 | +- Use actual responses from sourcekit-lsp |
| 110 | +- Use a moderate maximum timeout of up to 30s |
| 111 | + - The CI job timeout is 15 minutes |
| 112 | +- Use curated `assets/test` projects |
| 113 | +- Run in CI and nightly builds |
| 114 | +- Test key integrations with the VS Code API and key communication between our components |
| 115 | + |
| 116 | +### 3. Smoke (`/test/smoke`) |
| 117 | + |
| 118 | +- No mocking at all |
| 119 | +- For now only stimulate actions using the VS Code APIs, testing via the UI is a different beast |
| 120 | +- Use curated `assets/test` projects |
| 121 | +- No need to enforce a maximum timeout (per test) |
| 122 | +- Only run in nightly build |
| 123 | +- Should only have a handful of these tests, for complex features |
| 124 | + |
| 125 | +## Test Matrix |
| 126 | + |
| 127 | +### CI Build |
| 128 | + |
| 129 | +- Run for new PRs (`@swift-server-bot test this please`) |
| 130 | +- Run macOS, Linux and Windows |
| 131 | + - Currently only Linux, macOS and Windows is being explored |
| 132 | + - Expect Windows to fail short term, annotate to disable these tests |
| 133 | +- Ideally run against Swift versions 5.6 - 6.0 + main |
| 134 | +- Run `unit` and `integration` test suites |
| 135 | +- Run test against latest `stable` VS Code |
| 136 | + |
| 137 | +### Nightly Build |
| 138 | + |
| 139 | +- Run macOS, Linux and Windows |
| 140 | + - Currently only Linux, macOS and Windows is being explored |
| 141 | +- Ideally run against Swift versions 5.6 - 6.0 + main |
| 142 | +- Run `integration` and `smoke` test suites |
| 143 | +- Run test against latest `stable` and `insiders` VS Code |
0 commit comments