8319055: JCMD should not buffer the whole output of commands

[alexmenkov]

The fix implements streaming output support for attach protocol.
See JBS issue for evaluation, summary of the changes in the 1st comment.
Testing: tier1..4,hs-tier5-svc

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8319055: JCMD should not buffer the whole output of commands (Enhancement - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/23405/head:pull/23405
$ git checkout pull/23405

Update a local copy of the PR:
$ git checkout pull/23405
$ git pull https://git.openjdk.org/jdk.git pull/23405/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 23405

View PR using the GUI difftool:
$ git pr show -t 23405

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/23405.diff

Using Webrev

Link to Webrev Comment

[alexmenkov]

The changes:
Attach API protocol:

• attach operation options are added for ATTACH_API_V2 requests.
  The options are passed as part of command name; old clients old clients enquere command without options, new client adds options only if target VM support them, so there is no need to bump attach api version;
• "getversion" command supports new "options" argument. with this argument VM reports supported options.
  The only supported option for now is "streaming" which allows client to control streaming/buffered output mode.

AttachListener:

• implemented attachStream class to support both streaming and buffered output;
• all attach command handler are updated to use attachStream instead of bufferedStream and start streaming (if enabled) after parsing command arguments;
• default streaming mode is controlled by "jdk.attach.vm.streaming" system properties; this value is used when clients does not specify "streaming" option in the request (for example when tools from older releases attach to the current VM). For safety now default value is "false" (i.e. buffered output). The plan is to make the default "true" later;
• platform implementations (windows/posix) updated accordingly shared code changes;

DCmd framework:

• added a way to start streaming output after command/argument parsing, but before executing the command;

Client side:

• added option support;
• added system property "jdk.attach.allowStreamingOutput" to set streaming mode;
  For now default value is "false" (i.e. to turn on streaming mode user need something like "JAVA_TOOL_OPTIONS=-Djdk.attach.allowStreamingOutput=true jcmd")
  The plan is to make the default "true" after fixing tests which fail with streaming output (or maybe add command line option for jcmd and other tools).

Test CodeHeapAnalyticsParams.java

• 'Compiler.CodeHeap_Analytics' jcmd command is the only command which thrown AttachOperationFailedException exception if argument is out of range (and this causes jcmd exits with exit code 1).
  All other jcmd commands just writes exception message in the output.
  Now 'Compiler.CodeHeap_Analytics' works like other commands, so the test is updated to expect exception message.

[bridgekeeper]

👋 Welcome back amenkov! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@alexmenkov The following labels will be automatically applied to this pull request:

• hotspot-runtime
• serviceability

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 00: Full (052319f9)

[AlanBateman]

Do you expect there will be follow up work at some point for the commands that upcall to Java and produce a byte[] to return to the tool?