System.OutOfMemoryException running NET6 unit testing with Console.Runner 3.16.3

[ilCosmico]

I'm running approximately 2600 unit tests for my custom control in both NetFramework 4.7.2 and NET6 via TeamCity.
With net472, all works fine.

However, I encountered issues with net6. Initially, I tried using Console runner 3.16.2, but I received the error System.NotSupportedException: Cannot read resources that depend on serialization.
This error can also be reproduced when running tests on my local machine.

I then attempted to use the latest version, v3.16.3, and it seemed to work fine until I encountered an OutOfMemoryException after approximately 10 minutes (it does not crash always with the same test).

When running the single test, it succeeds, but running all the tests on my local machine gives the System.Net.Sockets.SocketException instead (the same described here)

The command line used is:
"C:\Users\xxx\.nuget\packages\nunit.consolerunner\3.16.3\tools\nunit3-console.exe" UnitTesting.dll --where cat!=Wpf --labels=BeforeAndAfter --trace=Error

Unfortunately, I can't run with the --inprocess flag due to the reasons explained here.

[OsirisTerje]

Can you try with 3.15.4. That is the version we use in the adapter, and then what is used in dotnet and Visual Studio

[ilCosmico]

Can you try with 3.15.4. That is the version we use in the adapter, and then what is used in dotnet and Visual Studio

I get the below error. It seems it does not support NET6.

Could not load file or assembly 'System.Windows.Forms, Version=6.0.2.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. The system cannot find the file specified.

[CharliePoole]

@OsirisTerje Support for Windows Forms was one of the things added in the 3.16 series. I think you may need to merge changes made in 3.15 and 3.16 into a new release.

[OsirisTerje]

@CharliePoole Thanks!
I had hoped to be able to move up to version 4, but unsure how much undone there is to get there. There are different loading issues on all the 3.16 versions.

@ilCosmico Any chance you could provide a small repro for this?

[ilCosmico]

@OsirisTerje it's very difficult because, as I wrote, it happens randomly when running about 2600 tests.
Anyway I will try and, in case, I will let you know.

[OsirisTerje]

@ilCosmico Way back I had a need to create a huge amount of tests, and I used a .tt script (a T4 template). You could possibly use the same approach, have a look at https://github.com/OsirisTerje/NUnit.VisualStudioAdapter.TestPrograms/tree/master/TestNUnitMultipleTest/NUnitTest1
Look in the file script1.tt, and make a similar one suited more to your kind of test.

[ilCosmico]

@OsirisTerje I was able to isolate a bunch of tests for reproducing the issue, but the whole package I can share with you is cca 15MB so I prefer to share it here.

Command line
"C:\nunit.consolerunner\3.16.3\tools\nunit3-console.exe" UnitTesting.dll --test=UnitTests.BrepBooleans1 --test=UnitTests.BrepBooleans2 --test=UnitTests.BrepBooleans3 --test=UnitTests.BrepBooleans4 --test=UnitTests.BrepBooleans5 --labels=BeforeAndAfter

As I wrote, when I run it via the command line it gives the below error

The same run via TeamCity gives OutOfMemoryException

[OsirisTerje]

I do need to see the source code for this, otherwise hard to debug it.
So if this is propriety code, please create a source code repro that you can upload.

[ilCosmico]

I do need to see the source code for this, otherwise hard to debug it. So if this is propriety code, please create a source code repro that you can upload.

Do you mean the source code of UnitTesting.dll?

In the meantime, please download the content of the link and let me know if you are able to reproduce the issue on your machine too.

[OsirisTerje]

I have done that, but I dont need to doubt your findings. In orderen to figure out what is happening here, i need to see the sourced code of what you're trying to test. Preferably a repro.

[ilCosmico]

Ok, here you can download zip file containing also src folder with the source code of the tests.

[ilCosmico]

@OsirisTerje did you download the project? Is it what you needed to investigate the issue?

[OsirisTerje]

Yes, I did, and I did experience an OOM after 6 minutes run, in a file CreateNewEyeshotControl.
But, the test is so intermixed with the software that is to be tested (SUT), that it is very hard to say anything about this.
I do see the project mixes engines, you should not include the console runner together with the adapter without the versions matching according to this list https://docs.nunit.org/articles/vs-test-adapter/Adapter-Engine-Compatibility.html
Further, it is possible to mix and match MSTest and NUnit, but I am not sure if this will help in this case.

The major thing here is that there are a lot of calculations being done, and at the same time the UI is present. That doesn't seem like a good choice, and will stress the system further.

Looking at the memory consumption, the memory is being deallocated too, but I did see peaks at around 1GB memory. There might be higher peaks and then the question is what that comes from. This might come from anything really, and in particular, any winform or WPF involved here will be high memory objects.

When the OOM happened the memory consumption was only around 400 MB.

When the test code, the UI code and the simulation (?) algorithm (?) (assuming a bit here) is intermixed like this, It is hard to say what is causing what. I don't see any NUnit code in the OOM stacktrace.

Just to be clear, I run this in Visual Studio, so the console runner is not involved, this is the adapter and engine only.

And, looking at the stacktrace I see that it is using the MSTest adapter, not the NUnit one.

So there is nothing from NUnit involved here.

That said: I would suggest that you simplify this code:

1. Use only one test framework
2. Split the test code and the simulation code
3. Avoid testing with the UI
4. Assuming you're doing some kind of integration tests here, separate these from the pure unit tests so that you get better control over which are doing what.

In short Divide and Conquer!

[ilCosmico]

Hi @OsirisTerje , thanks for the feedback.

I prepared a new project without MS.TestAdapter and console.runner as suggested, but I get the same issue.
Here you can download the new source code sample.

Simulation is a WinForms custom control for which I need to perform tests, for this reason, I can't split testing with UI, sorry.

Last but not least, please, don't forget that the main reason for the opening of this issue is that the same testing code run in Net Framework works like a charm.

[OsirisTerje]

Since the OOM happens regardless of whether you run with NUnit or MSTest, it is not a problem in the test system/test frameworks. It obviously do have something to do with the framework you run, since it works in Net Framework. You need to run this with a memory debugger and diagnose from there.

[ilCosmico]

Ok, thanks for pointing me in the right direction.

[ilCosmico]

@OsirisTerje I've managed to identify the cause behind the out-of-memory issue. It turns out to be a known bug in .NET6, which has already been documented here.

P.S. If you could spare a moment, your upvote on the issue would be greatly appreciated as well. Thank you! :)