[SUREFIRE-1710] skipAfterFailureCount in JUnit5 provider

[Tibor17]

This is my old change from my shelf. Providing it open. The ITs are broken and the impl should be fixed.
Implements the config parameter for JUnit5 skipAfterFailure.

Following this checklist to help us incorporate your
contribution quickly and easily:

• Make sure there is a JIRA issue filed
  for the change (usually before you start working on it). Trivial changes like typos do not
  require a JIRA issue. Your pull request should address just this issue, without
  pulling in other changes.
• Each commit in the pull request should have a meaningful subject line and body.
• Format the pull request title like [SUREFIRE-XXX] - Fixes bug in ApproximateQuantiles,
  where you replace SUREFIRE-XXX with the appropriate JIRA issue. Best practice
  is to use the JIRA issue title in the pull request title and in the first line of the
  commit message.
• Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
• Run mvn clean install to make sure basic checks pass. A more thorough check will
  be performed on your pull request automatically.
• You have run the integration tests successfully (mvn -Prun-its clean install).

If your pull request is about ~20 lines of code you don't need to sign an
Individual Contributor License Agreement if you are unsure
please ask on the developers list.

To make clear that you license your contribution under
the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

• I hereby declare this contribution to be licenced under the Apache License Version 2.0, January 2004

• In any other case, please file an Apache Individual Contributor License Agreement.

[Tibor17]

@andyRokit
@mpkorstanje
This is the implementation for https://issues.apache.org/jira/browse/SUREFIRE-1743 and https://issues.apache.org/jira/browse/SUREFIRE-1710 but it still does not work properly. Whatever value is set in skipAfterFailure, it still stops the execution with 2 failures. If you guys have a guess, pls ping us.

[Tibor17]

The class FailFastJUnit5IT should be renamed...

[mpkorstanje]

Cheers. I don't mind taking a look but at a glance this will only work with JUnit Jupiter rather then every TestEngine on the JUnit Platform. The current implementation is tied to JUnit Jupiter APIs.

It won't work for example with JUnit 4 (through JUnit 5s Vintage module) or Cucumber.

[mpkorstanje]

Overall I think it would be more effective to first resolve junit-team/junit-framework#1880 before trying to make it work in surefire.

[Tibor17]

Yes, it would be worth to have the fix of JUnit5 issue I reported.
Unfortunately, the JUnit5 team does not care ;-)

[mpkorstanje]

I believe the JUnit team uses community engagement as a measure of importance and interest.

Now it seems the conversation never moved on after the brainstorming. I think they are waiting for other people - possibly me, you or someone else - to reply to the brainstorm suggestions or provide their own. And then move on towards an implementation.

I've had no problems getting other features merged provided I did the heavy lifting. So I don't think this is a lack of care or interest. It is just not the most important thing compared to everything else. Especially with no or limited apparent community interest.

I do suspect that if someone has a sufficient need or interest in this feature it will be implemented. With that in mind I think it'd be better not to implement this in an ad-hoc fashionen. It will hopefully make some contributor come out of the woodwork.

Or if not, the feature wasn't really needed.

On a personal note: Due to circumstances I've had to significantly reduce the scope of my involvement. I think it is unlikely I'll be able to invest time getting this implemented on the JUnit side.

[Tibor17]

@marcphilipp
Hi Marc,

I have tried to implement the feature pleaseStop() by other way than we would expect in junit-team/junit-framework#1880
I have no idea what's wrong with my two SPIs. But I guess the native support would be booletproof in JUnit5 engines and API.

[marcphilipp]

Overall I think it would be more effective to first resolve junit-team/junit5#1880 before trying to make it work in surefire.

I agree. The only other solution I see is to do this from the Maven JVM, i.e. terminating the fork, instead of doing it in the provider.

[Tibor17]

This feature is implemented in all providers JUnit4 and TestNG. The same principle - we are counting the errors inside and outside of the JVM. If you have one fork then of course you have to count the failures/errors inside. The first happens in the concurrent model that wins and notification is sent and the framework like JUnit4 or TestNG is called with the method pleaseStop() and the framework stops "scheduling" new tests. Stopping the JVM is too naive.

[Vampire]

If this is going to make it into surefire without junit-team/junit-framework#1880 being fixed and used, I think stuff you called "junit5" should be renamed to "jupiter".

Spock 2+ for example is also a JUnit 5 engine, but would not be handled by your solution as far as I have seen from a very quick look. Probably the same for other engines like Vintage that runs JUnit 4 tests as JUnit 5 engine. At least providing some Jupiter services in common-junit5 makes me think it only supports the Jupiter engine and should thus be common-jupiter. :-)

[jira-importer]

Resolve #2038

[marcphilipp]

I think this PR can be closed as it's superseded by #3155

[Bukama]

Thanks for the information Marc! I'll close.

[Tibor17]

Hey guys. Sorry I am back after so long time when started the covid era.
I am open to help you out but first I need to talk with the colleagues in the ASF.
Basically, the idea behind this PR is that the the plugin counts the events concurrently from the JVM forks and calls a command to the JUnit to stop the execution of next test runs.
This is intentional because the architecture is built on the rules that the plugin is a manager of the JUnit, and so the plugin controls the JUnit framework. If we turn this around then the Apache would have a big problem because we will be a vassal of third party delivery (means JUnit provider, TextNG, whatever framework). The external deliveries would then control or dictate the code changes in the plugin and the plugin will have finally only one delivery contract with one framework which is very high risk for the ASF.
Therefore the architecture must always obey the principles plugin controls the provider ext delivery, and not opposite. So the plugin must send a command to the provider and transparently down to the JUnit framework, and the JUnit framework is the slave which must notify the sender of the command by an event. So this is event-command architecture, where plugin is the master and the framework is the slave. Basically, all the frameworks like JUnit or TestNG also they were built on this philosophy.
I remember that the JUnit team had a technical problem where to put the command in their API which stops or interrupts the execution. This was only a technical issue. Shortly, run() is a command, so it is event-command arch already and it should not be any problem to add commands and find a nice way and API to traverse down to the code which does this.

[marcphilipp]

So the plugin must send a command to the provider and transparently down to the JUnit framework, and the JUnit framework is the slave which must notify the sender of the command by an event. So this is event-command architecture, where plugin is the master and the framework is the slave. Basically, all the frameworks like JUnit or TestNG also they were built on this philosophy.
I remember that the JUnit team had a technical problem where to put the command in their API which stops or interrupts the execution.

JUnit Platform now provides such an API and #3155 makes use of it. While this PR only worked for JUnit Jupiter, the solution in the other PR supports all TestEngine implementations. Therefore, I think this PR should be closed in favor of #3155.