[Feature]: Load Playwright settings programmatically as an alternative to a .runsettings file

[0xced]

🚀 Feature Request

Currently, setting the BrowserTypeLaunchOptions when running tests with the Microsoft.Playwright.MSTest, Microsoft.Playwright.NUnit or Microsoft.Playwright.Xunit pacakge is controlled through the .runsettings configuration file.

It would be nice to be able to control the launch options programmatically, just like BrowserNewContextOptions can be controlled programmatically by overriding the ContextOptions() method.

Example

using System;
using System.Threading.Tasks;
using Microsoft.Playwright;
using Microsoft.Playwright.TestAdapter;
using Microsoft.Playwright.Xunit;
using Xunit;

public class IntegrationTests(WebAppFixture webAppFixture) : PageTest, IClassFixture<WebAppFixture>
{
    // 👇 Proposed new virtual method where browser, launch options and more can be controlled programmatically
    protected override PlaywrightSettings PlaywrightSettings()
    {
        return new PlaywrightSettings
        {
            Browser = PlaywrightBrowser.Firefox,
            LaunchOptions = new BrowserTypeLaunchOptions
            {
                Headless = false,
                SlowMo = 100,
                Timeout = 20_000,
            },
            ExpectTimeout = TimeSpan.FromSeconds(10),
        };
    }

    [Fact]
    public async Task TestSearchPage()
    {
        await Page.GotoAsync(webAppFixture.Url("search"));
        await Expect(Page).ToHaveTitleAsync("Search");
    }
}

This example is given with Xunit but would work the same with MSTest or NUnit.

Motivation

Controlling options programatically opens up new possibilities that are not achievable with a .runsettings file. For example, one could decide to run tests on Firefox based on some runtime conditions or change the value of the Timeout based on what hardware the tests are running on. This would give a lot of power to Playwright users.

Also, having a new PlaywrightSettings type is great for discoverability. Typing properties with autocompletion support from your IDE is a much better experience than writing XML elements in a .runsettings file without any autocompletion.

Finally, .runsettings support in Rider is pretty broken, I was not able to get the Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ISettingsProvider.Load method to be called, rendring Playwright .runsettings unusable with Rider. See also TestCaseFilter in .runsettings ignored.

[mxschmitt]

We intentionally had these settings configured on a global level before - this would require us to diverge from what we do in Node.js or Python. Usually for end-to-end tests, you write a test once and run them on all the browsers. For that you run dotnet test 3 times. I recommend to reach out to Jetbrains to fix the issue with runsettings - in the CLI it seems to work all expected, so its an issue with the IDE. Do you mind elaborating why you want to run some test files only in a specific browser? We recommend having multiple runsettings files if you want to specify custom timeouts etc.

[Xen0byte]

Hi @0xced, if it helps with what you need, I usually prefer to have a JSON configuration file and define my browser options in there under different profiles, then I just deserialise as needed and pass that object to the browser on initialisation. The profile is selected by name by an environment variable of sorts, so it can be dynamically set, e.g. .runsettings, CLI command argument, or pipeline variable. Some of the reasons for which I prefer this method are that 1) it keeps the config out of the code, which is what I believe that the official .runsettings approach is aiming for, but 2) is more powerful, flexible, reusable, and transparent than .runsetting config.