NUnit Engine - Missing TestEvent with overall testinformation after Update from 3.10 to 3.11.1

[DrHardReset]

I'm trying to update my testenvironment and painfully noticed a missing testevent.
I'm using my own resultwriter which implements ITestEventListener.OnTestEvent(string report).

1. With NUnit Engine 3.10 after each tested assembly the last Event the listener received was a test-suite report of type="Assembly" which contained all test results (TestSuiteResults and TestCaseResults).

2. With NUnit Engine 3.11.1 this Event is not showing up. Instead there is a new event "test-run" which contains all test results. This event is only showing up at the end of a testrun, not after each assembly.

The problem is, that my resultwriter is relying on the event of 1) as it shall process the result information after each tested assembly. My testenvironment is heavily using NUnitProject to test multiple assemblies at once (sequentially).

Is it expected behaviour that the final test-suite report after each assembly is not showing up any more?

events_3.10.txt
events_3.11.1.txt

[ChrisMaddock]

@CharliePoole - do you think this could be related to #603? Any thoughts on this?

[CharliePoole]

@ChrisMaddock Could be... I'll take a look.

@DrHardReset Can you provide a simple example that reproduces this? Also, can you check to see if the problem only occurs when running an NUnit project? The latter could be significant because of #382

[DrHardReset]

@CharliePoole This would be an example that produces the problem, both with single assembly and with NUnit project.

using NUnit.Framework;

namespace MyTest
{
    [TestFixture]
    public class Class1
    {
        [Test]
        public void Test()
        {

        }
    }
}

<NUnitProject>
  <Settings activeconfig="Release"/>
  <Config name="Release">
	<assembly path="bin\Debug\MyTest.dll"/>
	<assembly path="bin\Debug\MyTest.dll"/>
  </Config>
</NUnitProject>

Find attached a prepared C# test project with captured events for 3.10 and 3.11.1.
MyTest.zip

@ChrisMaddock transferred this issue into nunit-console. I'm facing the problem while using nunit-gui and am not very familiar with (the output of) the console. So I'm not sure if the problem occurs with the console, too.

[ChrisMaddock]

Hi @DrHardReset - this repository contains two NUnit components: the NUnit Console and the NUnit Engine. The repository name doesn't make that clear, I'm afraid!

Your problem will most likely be with the engine, which is the central component used by both the GUI and the Console Runner.

[ChrisMaddock]

FYI both, I'm not planning to hold the v3.12 release for this issue to be investigated, but please let me know if something comes up which we think needs to be squeezed in quickly. 🙂

[Snuxx]

Hi, is there any progress regarding this issue? @ChrisMaddock @CharliePoole
We had to update NUnit from 3.10 to 3.13.2 and now our listener logic wont work anymore, since it it relies on the <test-suite> event to extract neccessary information about errors and general data.
I noticed that the result for the <test-suite> gets set properly in the RunTestsCallbackHandler as the final result but this result never gets reported anywhere, except as part of the <test-run> event.
I attached two files which show the <test-suite> event for NUnit 3.10 and 3.13.2.
Imho this result should get reported as an event because its part of the "progress" to the final <test-run> event.
test-suite-event-3-13-2.txt
test-suite-event-3-10.txt

[Snuxx]

I worked around this issue by changing the RaiseCallbackEvent method like this:

        public void RaiseCallbackEvent(string eventArgument)
        {
            if (IsFinalResult(eventArgument))
            {
                Result = eventArgument;
                ReportProgress(eventArgument);
            }
            else
            {
                ReportProgress(eventArgument);
            }
        }

Now my listener can check if an environment/settings node exists in the <test-suite> event.
Thats possibly not the right place to put it and now two <test-suite> events get fired.

[CharliePoole]

@Snuxx As you know, the callbacks come from the framework. The point of the existing engine code, was that the callback was occurring twice, once for the final result and once for the progress event. I created the heuristics in IsFinalResult in order to distinguish the two identical calls by the content of the XML delivered as an argument.

In order to keep the console running correctly, it would be useful to know what - if anything - changed in the nunit 13.2 release. Are you able to see that? Since you say two test-suite events are generated, we are apparently still being called twice. Is the data the same in both cases? (It should not be) Why does the eventArgument appear to be a final result to us in both calls?

Your change is a good one for the framework release you are using, but the engine needs to continue working with all releases >= 3.0. For that reason, I'd rather see an improvement to IsFinalResult.

[CharliePoole]

I'm inclined to close this issue as being by design.

All test-case and test-suite events received from the tests are intended to contain minimal information needed to identify the particular test, which is starting, ending or producing output. The final result from the assembly is held by the engine until the entire run, potentially containing multiple assemblies, is complete.

It is conceivable for us to provide a way to get at the aggregate result for a suite within the event handler but we would need some motivating use case to spend time on it. Can you explain why using the final test-run result isn't satisfactory in your case?

[CharliePoole]

@DrHardReset @Snuxx
The 3.16 release will come out sometime this month. Pending further information, I will be removing the issue from the milestone as explained in my earlier comment.

For that reason, if this issue still presents a problem for you, it may be a good idea to reply to my earlier request for a motivating use-case for an enhancement, which could resolve it. Since there is nobody yet lined up to take over this project, the 3.16 release could conceivably be the last release for some time.

[DrHardReset]

@CharliePoole In the meantime I changed my resultwriter/eventlistener to parse all single incoming events. So for me this issue can be closed, as I am no longer relying on the aggregated results.