8351040: [REDO] Protection zone for easier detection of accidental zero-nKlass use

[tstuefe]

Please consider this second attempt at fixing https://bugs.openjdk.org/browse/JDK-8330174.

JDK-8330174 broke Windows and AIX (see breakage issue, https://bugs.openjdk.org/browse/JDK-8350768). The Windows issue happened in MetaspaceShared::map_archives for ArchiveRelocationMode=0 or ArchiveRelocationMode=2 (use_requested_addr=true). In those cases, we (A) delete the initial combined mapping for the CDS archive and then (B) mmap the individual archive regions separately into their respective, now vacated, address spaces. The protection zone is also part of the combined CDS archive mapping, so it gets released at (A). Since the protection zone is not part of the archive, it is not reinstated like the other regions at step (B).
Happily, that caused the canary assertion whose purpose was to catch such errors to segfault, so we noticed. Without assert, since the mapping is released, the OS may at some later time put another mapping into that region. So we have to make sure the mapping for the protection zone gets re-reserved after being released at (A).

The fix for the windows error is in commit 504931d .

The AIX error, in comparison, is easy. On AIX we cannot mprotect System V shared memory (or better, we cannot mprotect 64K pages, @JoKern65 or @TheRealMDoerr ?). Using 64K pages for such frequently accessed memory as CDS and class space is more beneficial than protecting the zero nklass page. As a fallback, on AIX, we still leave the page, but we fill it with a marker value ('P', 0x50). Now, if you accidentally dereference a zero nKlass, you will not crash immediately. But at least later crashes will probably contain register values like '0x5050505050505050', so its a hint.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8351040: [REDO] Protection zone for easier detection of accidental zero-nKlass use (Bug - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/23912/head:pull/23912
$ git checkout pull/23912

Update a local copy of the PR:
$ git checkout pull/23912
$ git pull https://git.openjdk.org/jdk.git pull/23912/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 23912

View PR using the GUI difftool:
$ git pr show -t 23912

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/23912.diff

[bridgekeeper]

👋 Welcome back stuefe! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@tstuefe The following label will be automatically applied to this pull request:

• hotspot

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[JoKern65]

Hi Thomas,
mprotect supports System V shared memory, but only if running in an environment where the MPROTECT_SHM=ON environmental variable is defined, which is not the case in the jdk. So we can fairly say System V shared memory cannot be mprotected by us.

The documentation says:
The mprotect subroutine can only be used on shared memory regions backed with 4 KB or 64 KB pages;
So we can mprotect 64K pages and mmap supports 64K pages beginning with AIX 7.3 TL1.
With JDK-8334371 we favor the use of mmap 64K pages over System V shared memory if running on a system with AIX 7.3 TL1 or higher. But as long as we allow lower os versions the system V shared memory is still in place, and the mprotect restriction stays valid.