VS Code UI with vscode-test for End-to-End Integration testing

[lukehinds]

Summary

We want to investigate how to automate Visual Studio Code for end-to-end testing scenarios. Specifically, we need to confirm it’s possible to simulate user interactions such as using Chat / Composer boxes, F-I-M etc, with perhaps something like VS Code’s official Extension Testing framework. This will help us ensure our core functionality works.

This is course needs to work in a headless environment as it will be run in CI as an integration test(s)

Background and Motivation

• Our current tests primarily validate functionality via direct command calls or API usage, without mimicking a real user’s workflow.
• End-to-end flows which are triggered via a user typing something into an IDE are often critical for an assistant extension works as expected in actual user scenarios with CodeGate inline
• By leveraging vscode-test, perhaps can launch a headless (or GUI) instance of VS Code, install extension, and drive UI interactions via commands and the built-in "type" command via CodeGate

References

1. Official vscode-test Repository
2. VS Code Extension Testing Documentation
3. Example usage of typing into the Command Palette in tests:

await vscode.commands.executeCommand('workbench.action.showCommands');
// wait or ensure UI focus
await vscode.commands.executeCommand('type', { text: 'Hello World' });

4. Example usage of typing text into an editor:

const doc = await vscode.workspace.openTextDocument({ content: '' });
await vscode.window.showTextDocument(doc);
await vscode.commands.executeCommand('type', { text: 'Automated text' });

5. Potential approach for testing a feature like GitHub Copilot Chat (requires that the extension expose a command to open and focus the chat input).

Goals for This Research

• Confirm which built-in or extension-specific commands are needed to open and interact with UI elements (like the Command Palette, custom panels, or input boxes).
• Test whether the "type" command correctly routes keystrokes to the relevant input area (or if a custom solution is needed).
• Establish best practices (e.g., waiting for focus, verifying state changes) for robust end-to-end tests.
• Document how to handle scenarios that don’t expose commands (i.e., pure UI interactions that may require deeper automation tooling).

Proposed Tasks

1. Identify Commands: Collect the command IDs (workbench.action.showCommands, github.copilot.chat.open, etc.) that will be needed for our scenarios.
2. Create a Prototype Test: Write a minimal test using vscode-test to open the Command Palette, type a query, and confirm the expected behavior.
3. Focus Handling: Investigate how focus is handled when opening panels or editors. Document any timeouts/delays or other steps needed to ensure "type" commands work.
4. Evaluate Limitations: Determine if there are UI elements that cannot be accessed purely by commands. If so, explore alternative solutions or see if the extension can provide a command-based API.
5. Documentation: Compile findings into a reference doc for the team, detailing how to set up end-to-end tests in CI, potential pitfalls, and known workarounds.

Importance

• Ensures real user workflows are validated, reducing regressions that occur at the UI integration layer.
• Helps catch UI or focus-related bugs that pure unit or integration tests might miss.
• Provides confidence that our extension(s) and any external services (e.g., Copilot, proxies) behave as intended for end users.

Other points

We may need to write some mock APIs that replicate the various providers (copilot, antropic etc) as I doubt we want to hit the real services.

[alex-mcgovern]

Have done some initial work on this — it's "fiddly", but have made some progress.

The outputs of this spike so far are this document, and these 2 repos:

• spike-codegate-vscode-test which uses microsoft/vscode-test
• spike-codegate-vscode-extension-tester which uses redhat-developer/vscode-extension-tester

Will continue trying, would like to get a "happy" path in place for at least 1 provider locally, and begin forming thoughts on how we might run it in CI with CodeGate.