8354922: ZGC: Use MAP_FIXED_NOREPLACE when reserving memory

[sercher]

Hi all,

Please review the backport of JDK-8354922 to JDK 21.

There are versions of the Linux kernel that do not honor the address hint when mmapping memory without MAP_FIXED, that clobbers the older mappings overlapped with the requested range. A safer MAP_FIXED_NOREPLACE flag is used with ZGC since Java 25.

JDK 21 crashes with -XX:+UseZGC with RANDMMAP [1] kernel patch enabled, while JDK 25 works as expected.

$ ./25.0.1/bin/java -XX:+UseZGC -version
openjdk version "25.0.1" 2025-10-21 LTS
OpenJDK Runtime Environment (build 25.0.1+11-LTS)
OpenJDK 64-Bit Server VM (build 25.0.1+11-LTS, mixed mode, sharing)

$ ./21.0.9/bin/java -XX:+UseZGC -version
[0.070s][error][gc] Failed to reserve enough address space for Java heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

ZGC relies on kernel that respects address hints in mmap system call, which is not the case with RANDMMAP enabled kernels. ZVirtualMemoryManager::pd_reserve() always returns false at line 50 because the mmap returned value never matches the requested address.

jdk21u-dev/src/hotspot/os/posix/gc/z/zVirtualMemory_posix.cpp

Lines 40 to 51 in 634d3fa

	bool ZVirtualMemoryManager::pd_reserve(zaddress_unsafe addr, size_t size) {
	void* const res = mmap((void*)untype(addr), size, PROT_NONE, MAP_ANONYMOUS|MAP_PRIVATE|MAP_NORESERVE, -1, 0);
	if (res == MAP_FAILED) {
	// Failed to reserve memory
	return false;
	}
	if (res != (void*)untype(addr)) {
	// Failed to reserve memory at the requested address
	munmap(res, size);
	return false;
	}

Since Linux 4.17 there's MAP_FIXED_NOREPLACE flag in mmap, that satisfies the addr requests and reports failure when the requested range overlaps a pre-existing mapping.

The backport isn't clean. JDK 21 doesn't have JDK-8350441 that renamed zVirtualMemory_posix.cpp to zVirtualMemoryManager_posix.cpp, and JDK-8341692 that removed non-generational mode in ZGC. In absense of JDK-8341692 the same approach was taken on gc/x/ version of pd_reserve().

It is also proposed to backport JDK-8313319 that prevents unnecessary mmap-munmap cycle, that will follow in a separate PR (#7).

[1] https://pax.grsecurity.net/docs/randmmap.txt