dotnet test Microsoft.Testing.Platform experience

[Evangelink]

Summary

Introduce a new dotnet test experience that is focused toward the new testing platform Microsoft.Testing.Platform (aka MTP).

This ticket will act as the hub of all implementations related to the new dotnet test experience.

Background and Motivation

We have introduced a new testing platform as an alternative to VSTest, this new testing platform, named Microsoft.Testing.Platform changes fundamentally the way the test applications are structured.

VSTest:

Microsoft.Testing.Platform:

Currently, Microsoft.Testing.Platform allows tests to be run through dotnet test using some workaround by injecting a named target that will be loaded by dotnet test code in place of the one from VSTest. This behavior is fine when doing basic dotnet test call but is showing problem when we want to use more advanced features as the options are statically declared and tuned for VSTest. For example, we need to do dotnet test -- --minimum-expected-tests 10 or dotnet test -p:TestingPlatformCommandLineArguments="--minimum-expected-tests 10". This is also causing confusions to users for some arguments, for example, doing dotnet test --filter A will not forward the filter down to the new platform but will eat the argument for VSTest (which is not plugged in that context) and users should instead do dotnet test -- --filter A or dotnet test -p:TestingPlatformCommandLineArguments="--filter A".

One downside to the new testing platform is that each test application is working in "isolation" to other test applications while some features are expected to be provided as cross-test apps (e.g. users would expect at least one merge trx report and not only 1 trx per test application).

Proposed Feature

Instead of shipping an extra client executable, we will use dotnet process to act as a client and offer pre and post run capabilities.

The new testing platform offers the possibility to extend the supported command line arguments so we no longer can simply rely on a static way but instead either ask test applications for their supported options (e.g. when user will do dotnet test --help) or forward the extra parameters down to the test applications (e.g. dotnet test --coverage, the --coverage needs to be forward to each test application).

We should allow users to select the dotnet test mode (see #45471) to be either the new experience or the old one.

Under discussions

• We are currently discussing what to do for the case of "mixed mode", at least one project using VSTest and one project using MTP.

Limitations

Other

Note that I will edit this main design as we progress through the design and implementation and incorporate user's comments.

Pinging main test framework authors:

• MSTest: @Evangelink @Youssef1313
• NUnit: @OsirisTerje @rprouse
• TUnit: @thomhurst
• xUnit: @bradwilson

[dotnet-policy-service]

Added needs-breaking-change-doc-created label because this issue has the breaking-change label.

1. Create and link to this issue a matching issue in the dotnet/docs repo using the breaking change documentation template, then remove this needs-breaking-change-doc-created label.

Tagging @dotnet/compat for awareness of the breaking change.

[bradwilson]

Perhaps a nitpick, but your MTP diagram shows loading xunit.runner.visualstudio, but that is only ever used for VSTest. Our MTP support is baked into the test executable by default automatically.

[Evangelink]

Thanks @bradwilson! I have indeed reused old diagram before you actually did the implementation. I'll update/fix that.

[thomhurst]

How does this work with older SDKs? Do they get a new patch/minor version? Or will this only be in new .NET 9/10 ones?

[MarcoRossignoli]

Thanks @bradwilson! I have indeed reused old diagram before you actually did the implementation. I'll update/fix that.

If I recall is loading xunit.core right @bradwilson?

[bradwilson]

If I recall is loading xunit.core right @bradwilson?

If you mean which assembly/package does the M.T.P. code live in, it's xunit.v3.runner.inproc.console (which is responsible for the command line dotnet run experience for v3): https://github.com/xunit/xunit/tree/main/src/xunit.v3.runner.inproc.console/TestingPlatform

[Evangelink]

How does this work with older SDKs? Do they get a new patch/minor version? Or will this only be in new .NET 9/10 ones?

It would be available only for NET 10+. We could discuss backporting to NET 9 but I think we would need to have a really strong justification.