Cannot run .Net Framework Assemblies from nunit.engine.netstandard

[clydeigc]

I have a compiled set of .Net Framework 4.7.1 NUnit3 Tests as a DLL and would like to run them from a .Net Core 2.0 project through the NUnit.Engine.NetStandard.

What I am trying to do is:

1. Point the TestPackage to the test assembly using:

TestPackage testPackage = new TestPackage(testAssemblyPaths);

2. Initializing the TestEngine:

_testEngine = new TestEngine();

3. Creating an instance of the TestRunner and call the Run() method:

using (ITestRunner testRunner = _testEngine.GetRunner(testPackage))
{
    TestFilter categoryTestFilter = builder.GetFilter();

    /* 
    NUnitTestEventMapper and NUnitEventSplitter are custom classes unrelated to this question, 
    the latter of which implements ITestEventListener.
    */
    NUnitTestEventMapper nUnitTestEventMapper = new NUnitTestEventMapper();
    testRunner.Run(new NUnitEventSplitter(subscribers, nUnitTestEventMapper), categoryTestFilter);
}

On Run, I'm getting the following Exception:

Exception Message - 'Could not load file or assembly 'MyTestAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.'

Exception Stack Trace -

   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks, IntPtr ptrLoadContextBinder)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, IntPtr ptrLoadContextBinder)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at NUnit.Engine.Drivers.NUnitNetStandardDriver.Load(String testAssembly, IDictionary`2 settings)
   at NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
   at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
   at NUnit.Engine.Runners.DirectTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.TestExecutionTask.Execute()
   at NUnit.Engine.Runners.AggregatingTestRunner.RunTestsSequentially(ITestEventListener listener, TestFilter filter, List`1 results, Boolean disposeRunners)
   at NUnit.Engine.Runners.AggregatingTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.MasterTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.MasterTestRunner.Run(ITestEventListener listener, TestFilter filter)

Should it be possible at all to create a TestPackage pointing to test assemblies using .Net Framework and .Net Core and running everything using a single .Net Standard Engine?

[CharliePoole]

@clydeigc Personal opinion here, so please wait for the team members to weigh in.

The .NET Core engine is intended as a subset of the original engine and probably should have had a different name. It might be possible to fix it to do what you want to do provided the supported subset is what you want. I'm personally doubtful that it's worth the effort.

OTOH it seems much more feasible to do the reverse: run .NET standard tests on a platform (.NET Core for example) that supports them using the full (.NET Framework) engine. Is there reason that won't work for you?

[clydeigc]

@CharliePoole Thanks for your reply.

My use case is that I am creating a test running library which creates a layer over other test runners (like the NUnit engine). The idea is for users of the library to create a console executable project and through a uniform api run their test assemblies (NUnit, XUnit etc) without worrying on how to call and configure the runner of each testing framework.

Since I don't want to bind users of the library to use .NET Framework in their console executable project, I would like to keep my library as a .Net Standard library, thus limiting me to use the .Net Standard engine offered by NUnit if I understood correctly.

Other than this issue, the subset of features provided by the .NET Standard engine seem to be enough for me so far.

[CharliePoole]

Since I'm retiring from this project, let's wait for the other team members to weigh in. 😄

Your project sounds a little like the old Gallio project.

[ChrisMaddock]

Unfortunately this isn't possible yet - the current .NET Standard engine can only run tests in process - so couldn't load a .NET Framework test suite if it was being ran on .NET Core I don't think.

In the long run, I think this roughly fits with out ambitions. @rprouse is doing some work on bringing the .NET Standard and .NET Framework engines close together right now, then I think we need to re-asses where we are at.

[clydeigc]

For now we have opted to stick with the .NET Framework engine due to the limitations of the .NET Standard engine. The main limitation affecting us is that from what we saw, presently, it is only possible to run everything in a single process without memory isolation. Ideally the process model behavior would be similar to that of the .NET Framework engine eventually.

Would it be possible to provide some visibility on the progress of @rprouse 's work?

[ChrisMaddock]

@clydeigc This is probably the best source right now: https://github.com/orgs/nunit/teams/engine-team/discussions/4

I'm going to label this issue up and keep it open, as it's a feature request that isn't tracked anywhere else yet, I don't think, and something that I agree should happen. Just be aware, there are several blockers on the way - as discussed in the above. 🙂

[clydeigc]

@ChrisMaddock

It would be great to see the NUnit .NET Standard engine get the process-level memory-isolation functionality in the near future.

Unfortunately I am unable to track the progress on that link since I do not have access to the NUnit Github organisation pages. Seeing as people might be interested in this project to migrate to .NET Standard, it might be worth offering some form of public visibility through a Github project.

Anyways, thanks for your time and detailed replies :)

[ChrisMaddock]

Sorry - I thought that page was public. I plan to create the relevant github issues soon, which I guess will serve as a better 'definitive source'. In the meanwhile - here's a preview! I've trimmed it down to the important bits - consider this a 'high-level current thoughts'.

Motivation
Our current support for running .NET Core tests is only possible through dotnet test. This presents various usability constraints - limitations including passing in command line test parameters and getting NUnit xml results files. Both could be worked around, however, I think it would be good to have our own tooling support, which cuts out any limitations of the dotnet CLI.
Overall vision
We should have a runner that can run on the .NET Core platform, for when the .NET Framework/Mono is not installed. In my opinion, our current .NET 2.0 runner should also support the running of .NET Core tests, for cases where both framework/core frameworks are available, and tests wish to be ran together.
Steps I think this should involve:
.NET Core nunit3-console.exe
Would could multi-target the current console runner to use the current .NET Standard engine used by the VS adapter, and remove the console options that aren't available in the reduced functionality engine. (Of course, adding them back as the .NET Standard engine gains more functionality.) This would give us a console runner which can be used where no Mono/.NET Framework are available relatively quickly.
Support for running .NET Core tests in the .NET 2.0 nunit3-console.exe
I have projects that multi-target both .NET Framework and .NET Core. I'd like to be able to pass these projects to a single test runner, which would test on both platforms - much like I could currently do if I had e.g. .NET 2.0 and .NET 4.0 builds.
This should be fairly possible to add to the current console runner, if we can create an nunit-agent.exe which can run .NET Core tests. I believe the big issue blocking that functionality would be: #266 - we'd then also need to implement framework detection of the test assembly, and launching the correct agent.
Other considerations
dotnet nunit tool
For a long time, we talked about creating a dotnet nunit tool, similar to dotnet xunit - which would make the running of NUnit .NET Core tests possible, without the constraints of dotnet test, and using the .NET Core CLI.
My understanding, is that this would be just be packaging around an nunit3-console tool. We should also be aware that XUnit have just discontinued dotnet xunit citing stability issues with integrating with the .NET Core CLI - it would be interesting to learn more on that before working on this further. As I understand however - this would effectively be a layer on top of a .NET Core nunit-console, and shouldn't affect any of the above proposed ideas.

[CharliePoole]

Moved to the 4.0 milestone but leaving it blocked by #1045 for the moment.