ReqnRoll is unable to take any input from xunit.runner.json

[archie098]

Reqnroll Version

2.3.0

Which test runner are you using?

xUnit

Test Runner Version Number

3.0.2

.NET Implementation

.NET 8.0

Test Execution Method

Visual Studio Test Explorer

Content of reqnroll.json configuration file

No response

Issue Description

After migrating from Specflow to ReqnRoll everything is working fine except xunit parallel thread count is not being considered from xunit.runner.json file., What it looks to me is ReqnRoll project is not able to recognize the file. It used to work perfectly fine with Specflow.

This is the content of the json file which is placed in the project directory and used to work fine with Specflow. It used to open only 4 thread.
{
"parallelizeAssembly": true,
"parallelizeTestCollections": true,
"maxParallelThreads": 4,
"parallelAlgorithm": "aggressive"
}

Even if I specify the number here, all the feature files are starting to run in paralle.

Steps to Reproduce

This is my project file -

net8.0 enable enable all runtime; build; native; contentfiles; analyzers; buildtransitive

I am running test via VS Test Explorer.

Link to Repro Project

No response

[gasparnagy]

I think this is not related to accessing the xunit.runner.json, but the way how xunit handles maxParallelThreads. Could you please try it with any other xUnit setting that is not related to parallel execution?

For handling the maxParallelThreads, see the discussion: https://github.com/orgs/reqnroll/discussions/328

[archie098]

@gasparnagy Thank you so much for replying. I have had a detailed look at the discussion when I was implementing parallel run with xunit using xunit.runner.json in Specflow few weeks back.

Apparently xunit team have implemented a new algorithm where parallelAlgorithm now accepts another input which is aggressive. By using this I was able to open the exact number of threads I specified in the xunit.runner.json file in Specflow. Please see the algorithm section where they have mentioned about this - https://xunit.net/docs/running-tests-in-parallel.html

Now the problem in ReqnRoll is not only with maxParallelThreads. It is not able to read any value from the xunit.runner. json file.

For example - the below code in xunit.runner.json does not supposed executes feature files in parallel because I have made parallelizeTestCollections and parallelizeAssembly as false (You will get more information here about the Configuration file values - https://xunit.net/docs/configuration-files)

{
"parallelizeAssembly": false,
"parallelizeTestCollections": false,
"maxParallelThreads": 3,
"parallelAlgorithm": "aggressive"
}

The same code is respected by my Specflow project and it is starting one thread only as the parallelization is disabled . But the same code is not working in ReqNRoll. It is still executing all the feature files in parallel which is a default behavior of xunit. This confirms me that somehow ReqnRoll is not able to identify the xunit.runner.json file or unable to read any value from the json, not only maxParallelThreads.

I will try to find any other xunit settings which is not related to parallel but as of now there is no such feature in my project which requires any other settings file from xunit. If I get any, I will update here.

[gasparnagy]

Please do so. I have used xunit json config a few weeks ago and it worked for me. Reqnroll generates plain xunit test classes as code, so we don't interfere with the config handling of xunit at all.

Oh. I just see the version in your issue. Unfortunately xUnit 3 is not supported yet, so maybe this is just a duplicate of #405. But I'm not sure.

[DerAlbertCom]

@gasparnagy the xunit runner 3 runs also xunit v2 test.

@archie098 if your are using xunit v2, you may like #491.