8247690: RunTest does not support running of JTREG manual tests

[mpdonova]

This PR adds a make keyword, MANUAL, to run manual tests with the make command e.g.:

make test JTREG=MANUAL=true TEST=... 

I've gotten into the habit of running tests with make and although this doesn't come up often, I have wanted the functionality for a while.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8247690: RunTest does not support running of JTREG manual tests (Bug - P4)

Reviewers

• Erik Joelsson (@erikj79 - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/28381/head:pull/28381
$ git checkout pull/28381

Update a local copy of the PR:
$ git checkout pull/28381
$ git pull https://git.openjdk.org/jdk.git pull/28381/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 28381

View PR using the GUI difftool:
$ git pr show -t 28381

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/28381.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back mdonovan! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@mpdonova This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8247690: RunTest does not support running of JTREG manual tests

Reviewed-by: erikj

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 33 new commits pushed to the master branch:

• 7b11bd1: 8372047: ClassTransform.transformingMethodBodies andThen composes incorrectly
• c419dda: 8372163: G1: Remove unused G1HeapRegion::remove_code_root
• ad38a12: 8371557: java/net/httpclient/http3/H3RequestRejectedTest.java: javax.net.ssl.SSLHandshakeException: local endpoint (wildcard) and remote endpoint (loopback) ports conflict
• ... and 30 more: https://git.openjdk.org/jdk/compare/4a975637a144fa8aa449a1419e656721833513b5...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[openjdk]

@mpdonova The following label will be automatically applied to this pull request:

• build

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 04: Full - Incremental (69b50cb9)
• 03: Full - Incremental (f650c58c)
• 02: Full - Incremental (7cbb1ddb)
• 01: Full - Incremental (1cfa515d)
• 00: Full (6e1a5a9a)

[erikj79]

I think this looks ok. Please update doc/testing.[md|html] with documentation describing this new option.

[sendaoYan]

Seems that it's no need to distinguish manual tests and automatic tests. Just remove '-automatic' will make the original make command can run all the jtreg tests except the test has mark as '@ignore'

[mpdonova]

Seems that it's no need to distinguish manual tests and automatic tests. Just remove '-automatic' will make the original make command can run all the jtreg tests except the test has mark as '@ignore'

I reverted my change and removed the '-automatic' flag. However, I can see an argument for having the flag. When specifying a test directory (instead of a specific test) the flag will only run the automatic or manual tests. That can be useful if you have to do manual test verification.

[sendaoYan]

However, I can see an argument for having the flag. When specifying a test directory (instead of a specific test) the flag will only run the automatic or manual tests. That can be useful if you have to do manual test verification.

Do you mean run all the manual tests for the specific directory. Maybe pass parameter JTREG="OPTIONS=-m" can make this thing work, such as run all the manual tests in test/jdk directory, after your new propose patch integrated.

make test TEST=test/jdk/ JTREG="OPTIONS=-m"

[erikj79]

Seems that it's no need to distinguish manual tests and automatic tests. Just remove '-automatic' will make the original make command can run all the jtreg tests except the test has mark as '@ignore'

What makes you say this? We have tons of tests with the /manual option specified. Those are currently filtered out when running tests through the makefiles. Removing -automatic will make those suddenly run everywhere. The original patch made sense, this does not.

[sendaoYan]

Seems that it's no need to distinguish manual tests and automatic tests. Just remove '-automatic' will make the original make command can run all the jtreg tests except the test has mark as '@ignore'

What makes you say this? We have tons of tests with the /manual option specified. Those are currently filtered out when running tests through the makefiles. Removing -automatic will make those suddenly run everywhere. The original patch made sense, this does not.

I think you are right, we are relay on the default '-automatic' option to exclude the '/manual' tests, especial the CI? It's not a good idea to change the long history habit

[erikj79]

I believe this will execute only manual tests.

-m | -manual Only tests with /manual will be run

[erikj79]

It looks like an accidental extra newline was added here.

[mpdonova]

/integrate

[openjdk]

Going to push as commit f125c76.
Since your change was applied there have been 33 commits pushed to the master branch:

• 7b11bd1: 8372047: ClassTransform.transformingMethodBodies andThen composes incorrectly
• c419dda: 8372163: G1: Remove unused G1HeapRegion::remove_code_root
• ad38a12: 8371557: java/net/httpclient/http3/H3RequestRejectedTest.java: javax.net.ssl.SSLHandshakeException: local endpoint (wildcard) and remote endpoint (loopback) ports conflict
• ... and 30 more: https://git.openjdk.org/jdk/compare/4a975637a144fa8aa449a1419e656721833513b5...master

Your commit was automatically rebased without conflicts.

[openjdk]

@mpdonova Pushed as commit f125c76.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.