8354922: ZGC: Use MAP_FIXED_NOREPLACE when reserving memory

[sercher]

Hi all,

Please review the backport of JDK-8354922 to JDK 21.

There are versions of the Linux kernel that do not honor the address hint when mmapping memory without MAP_FIXED, that clobbers the older mappings overlapped with the requested range. A safer MAP_FIXED_NOREPLACE flag is used with ZGC since Java 25.

JDK 21 crashes with -XX:+UseZGC with RANDMMAP [1] kernel patch enabled, while JDK 25 works as expected.

$ ./25.0.1/bin/java -XX:+UseZGC -version
openjdk version "25.0.1" 2025-10-21 LTS
OpenJDK Runtime Environment (build 25.0.1+11-LTS)
OpenJDK 64-Bit Server VM (build 25.0.1+11-LTS, mixed mode, sharing)

$ ./21.0.9/bin/java -XX:+UseZGC -version
[0.070s][error][gc] Failed to reserve enough address space for Java heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

ZGC relies on kernel that respects address hints in mmap system call, which is not the case with RANDMMAP enabled kernels. ZVirtualMemoryManager::pd_reserve() always returns false at line 50 because the mmap returned value never matches the requested address.

jdk21u-dev/src/hotspot/os/posix/gc/z/zVirtualMemory_posix.cpp

Lines 40 to 51 in 634d3fa

	bool ZVirtualMemoryManager::pd_reserve(zaddress_unsafe addr, size_t size) {
	void* const res = mmap((void*)untype(addr), size, PROT_NONE, MAP_ANONYMOUS|MAP_PRIVATE|MAP_NORESERVE, -1, 0);
	if (res == MAP_FAILED) {
	// Failed to reserve memory
	return false;
	}
	if (res != (void*)untype(addr)) {
	// Failed to reserve memory at the requested address
	munmap(res, size);
	return false;
	}

Since Linux 4.17 there's MAP_FIXED_NOREPLACE flag in mmap, that satisfies the addr requests and reports failure when the requested range overlaps a pre-existing mapping.

The backport isn't clean. JDK 21 doesn't have JDK-8350441 that renamed zVirtualMemory_posix.cpp to zVirtualMemoryManager_posix.cpp, and JDK-8341692 that removed non-generational mode in ZGC. In absense of JDK-8341692 the same approach was taken on gc/x/ version of pd_reserve().

It is also proposed to backport JDK-8313319 that prevents unnecessary mmap-munmap cycle, that will follow in a separate PR (#2476 ).

[1] https://pax.grsecurity.net/docs/randmmap.txt

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• JDK-8354922 needs maintainer approval

Issue

• JDK-8354922: ZGC: Use MAP_FIXED_NOREPLACE when reserving memory (Enhancement - P4 - Approved)

Reviewers

• Stefan Karlsson (@stefank - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk21u-dev.git pull/2475/head:pull/2475
$ git checkout pull/2475

Update a local copy of the PR:
$ git checkout pull/2475
$ git pull https://git.openjdk.org/jdk21u-dev.git pull/2475/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 2475

View PR using the GUI difftool:
$ git pr show -t 2475

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk21u-dev/pull/2475.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back schernyshev! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@sercher This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8354922: ZGC: Use MAP_FIXED_NOREPLACE when reserving memory

Reviewed-by: stefank

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 36 new commits pushed to the master branch:

• 7c99320: 8354145: G1: UseCompressedOops boundary is calculated on maximum heap region size instead of maxiumum ergonomic heap region size
• 0f2ffb5: 8357277: Update OpenSSL library for interop tests
• 12f09b7: 8345668: ZoneOffset.ofTotalSeconds performance regression
• ... and 33 more: https://git.openjdk.org/jdk21u-dev/compare/634d3fa7f147fc6fb1a7765755a3feed82ffaf50...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@stefank) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

[openjdk]

This backport pull request has now been updated with issue from the original commit.

[mlbridge]

Webrevs

• 00: Full (9c950cf0)

[stefank]

This looks good to me.

[openjdk]

⚠️ @sercher This change is now ready for you to apply for maintainer approval. This can be done directly in each associated issue or by using the /approval command.

[sercher]

Thanks @stefank

[sercher]

/approval request This fixes ZGC on systems that do not honor the address hint when mmapping memory without MAP_FIXED

[openjdk]

@sercher
8354922: The approval request has been created successfully.

[bridgekeeper]

@sercher This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply issue a /touch or /keepalive command to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

[sercher]

/integrate

[openjdk]

@sercher
Your change (at version 9c950cf) is now ready to be sponsored by a Committer.

[phohensee]

/sponsor

[openjdk]

Going to push as commit ab298c2.
Since your change was applied there have been 37 commits pushed to the master branch:

• b17eef0: 8313319: [linux] mmap should use MAP_FIXED_NOREPLACE if available
• 7c99320: 8354145: G1: UseCompressedOops boundary is calculated on maximum heap region size instead of maxiumum ergonomic heap region size
• 0f2ffb5: 8357277: Update OpenSSL library for interop tests
• ... and 34 more: https://git.openjdk.org/jdk21u-dev/compare/634d3fa7f147fc6fb1a7765755a3feed82ffaf50...master

Your commit was automatically rebased without conflicts.

[openjdk]

@phohensee @sercher Pushed as commit ab298c2.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.