Large Pages for OpenJDK on Windows
I decided to take a quick peek into [JDK-8296467] Large Pages fails when allocating 8GB+ on Windows/Java 19 – Java Bug System (openjdk.org) using this command. See resultant output below.
/c/java/binaries/jdk/x64/jdk-19.0.1+10/bin/java.exe -Xms8G -Xmx8G -XX:+UseLargePages -Xlog:gc+init --versionThe local build I used to investigate this issue was from this commit.
OpenJDK 64-Bit Server VM warning: JVM cannot use large page memory because it does not have enough privilege to lock pages in memory.
[0.099s][info][gc,init] CardTable entry size: 512
[0.229s][info][gc,init] Version: 19.0.1+10 (release)
[0.230s][info][gc,init] CPUs: 8 total, 8 available
[0.230s][info][gc,init] Memory: 16301M
[0.230s][info][gc,init] Large Page Support: Disabled
[0.231s][info][gc,init] NUMA Support: Disabled
[0.231s][info][gc,init] Compressed Oops: Enabled (Zero based)
[0.231s][info][gc,init] Heap Region Size: 4M
[0.231s][info][gc,init] Heap Min Capacity: 8G
[0.232s][info][gc,init] Heap Initial Capacity: 8G
[0.232s][info][gc,init] Heap Max Capacity: 8G
[0.232s][info][gc,init] Pre-touch: Disabled
[0.232s][info][gc,init] Parallel Workers: 8
[0.232s][info][gc,init] Concurrent Workers: 2
[0.233s][info][gc,init] Concurrent Refinement Workers: 8
[0.233s][info][gc,init] Periodic GC: Disabled
openjdk 19.0.1 2022-10-18
OpenJDK Runtime Environment Temurin-19.0.1+10 (build 19.0.1+10)
OpenJDK 64-Bit Server VM Temurin-19.0.1+10 (build 19.0.1+10, mixed mode, sharing)That error message comes from the large_page_init_decide_size() function. It calls request_lock_memory_privilege(), which:
- Calls OpenProcessToken to open the access token associated with the process.
- Calls LookupPrivilegeValue to determine the locally unique identifier (LUID) used to locally represent the SeLockMemoryPrivilege privilege name.
- Calls AdjustTokenPrivileges to enable the SeLockMemoryPrivilege in the process access token.
On my system, AdjustTokenPrivileges failed with ERROR_NOT_ALL_ASSIGNED. This is the stack when the failure occurs.
jvm.dll!large_page_init_decide_size() C++
jvm.dll!os::large_page_init() C++
jvm.dll!os::init_before_ergo() C++
jvm.dll!Threads::create_vm(JavaVMInitArgs * args, bool * canTryAgain) C++
jvm.dll!JNI_CreateJavaVM_inner(JavaVM_ * * vm, void * * penv, void * args) C++
jvm.dll!JNI_CreateJavaVM(JavaVM_ * * vm, void * * penv, void * args) C++
jli.dll!InitializeJVM(const JNIInvokeInterface_ * * * pvm, const JNINativeInterface_ * * * penv, InvocationFunctions * ifn) C
jli.dll!JavaMain(void * _args) C
jli.dll!ThreadJavaMain(void * args) C
The reason for the error is that the Lock pages in memory policy did not have any allowed users. As documented at AdjustTokenPrivileges, use the Local Security Policy app to change the security policy.
Adding myself to this list didn’t appear to have any effect. I came back to this experiment a few weeks later and found that the issue was gone. Perhaps a reboot was necessary to pick up the change. I also found useful these remarks on VirtualAlloc and MEM_LARGE_PAGES – The Old New Thing (microsoft.com) and the article on Large-Page Support.
The output I got once the privilege issue was addressed is in the snippet below. To find these message in the source code, run grep -Rin --include *.cpp "Failed to reserve and commit" ./src/.
OpenJDK 64-Bit Server VM warning: Failed to reserve and commit memory using large pages. req_addr: 0x0000000000000000 bytes: 251658240
[0.249s][info][gc,init] CardTable entry size: 512
OpenJDK 64-Bit Server VM warning: Failed to reserve and commit memory using large pages. req_addr: 0x0000000600000000 bytes: 8589934592
OpenJDK 64-Bit Server VM warning: Failed to reserve and commit memory using large pages. req_addr: 0x0000000000000000 bytes: 134217728
OpenJDK 64-Bit Server VM warning: Failed to reserve and commit memory using large pages. req_addr: 0x0000000000000000 bytes: 134217728
[0.374s][info][gc,init] Version: 19.0.1+10 (release)
[0.375s][info][gc,init] CPUs: 12 total, 12 available
[0.375s][info][gc,init] Memory: 32487M
[0.375s][info][gc,init] Large Page Support: Enabled
[0.376s][info][gc,init] NUMA Support: Disabled
[0.376s][info][gc,init] Compressed Oops: Enabled (Zero based)
[0.376s][info][gc,init] Heap Region Size: 4M
[0.377s][info][gc,init] Heap Min Capacity: 8G
[0.377s][info][gc,init] Heap Initial Capacity: 8G
[0.377s][info][gc,init] Heap Max Capacity: 8G
[0.378s][info][gc,init] Pre-touch: Disabled
[0.378s][info][gc,init] Parallel Workers: 10
[0.378s][info][gc,init] Concurrent Workers: 3
[0.379s][info][gc,init] Concurrent Refinement Workers: 10
[0.379s][info][gc,init] Periodic GC: Disabled
openjdk 19.0.1 2022-10-18
OpenJDK Runtime Environment Temurin-19.0.1+10 (build 19.0.1+10)
OpenJDK 64-Bit Server VM Temurin-19.0.1+10 (build 19.0.1+10, mixed mode, sharing)Here is the call stack where the actual memory reservation happens (the call to the VirtualAlloc function):
jvm.dll!virtualAlloc(void * lpAddress, unsigned __int64 dwSize, unsigned long flAllocationType, unsigned long flProtect) C++
jvm.dll!reserve_large_pages_single_range(unsigned __int64 size, char * req_addr, bool exec) C++
jvm.dll!reserve_large_pages(unsigned __int64 size, char * req_addr, bool exec) C++
jvm.dll!os::pd_reserve_memory_special(unsigned __int64 bytes, unsigned __int64 alignment, unsigned __int64 page_size, char * addr, bool exec) C++
jvm.dll!os::reserve_memory_special(unsigned __int64 size, unsigned __int64 alignment, unsigned __int64 page_size, char * addr, bool executable) C++
jvm.dll!reserve_memory_special(char * requested_address, const unsigned __int64 size, const unsigned __int64 alignment, const unsigned __int64 page_size, bool exec) C++
jvm.dll!ReservedSpace::reserve(unsigned __int64 size, unsigned __int64 alignment, unsigned __int64 page_size, char * requested_address, bool executable) C++
jvm.dll!ReservedSpace::initialize(unsigned __int64 size, unsigned __int64 alignment, unsigned __int64 page_size, char * requested_address, bool executable) C++
jvm.dll!ReservedCodeSpace::ReservedCodeSpace(unsigned __int64 r_size, unsigned __int64 rs_align, unsigned __int64 rs_page_size) C++
jvm.dll!CodeCache::reserve_heap_memory(unsigned __int64 size) C++
jvm.dll!CodeCache::initialize_heaps() C++
jvm.dll!CodeCache::initialize() C++
jvm.dll!codeCache_init() C++
jvm.dll!init_globals() C++
jvm.dll!Threads::create_vm(JavaVMInitArgs * args, bool * canTryAgain) C++
jvm.dll!JNI_CreateJavaVM_inner(JavaVM_ * * vm, void * * penv, void * args) C++
jvm.dll!JNI_CreateJavaVM(JavaVM_ * * vm, void * * penv, void * args) C++
jli.dll!InitializeJVM(const JNIInvokeInterface_ * * * pvm, const JNINativeInterface_ * * * penv, InvocationFunctions * ifn) C
jli.dll!JavaMain(void * _args) C
jli.dll!ThreadJavaMain(void * args) CIn the reserve_memory_special and reserve_large_pages_single_range functions, I notice that there is additional logging available using the pagesize tag (trace is the most verbose product build log level as per JEP 158: Unified JVM Logging (openjdk.org)). Here’s how to write the pagesize logs to disk:
/c/java/binaries/jdk/x64/jdk-19.0.1+10/bin/java.exe -Xms8G -Xmx8G -XX:+UseLargePages -Xlog:gc+init -Xlog:pagesize=trace:file=pagesize.txt::filecount=0 --versionThe virtualAlloc function uses a logging wrapper to capture and log the result of the VirtualAlloc Windows function. It uses a separate logging tag (os). To capture both OS and pagesize logs to disk, use this command:
/c/java/binaries/jdk/x64/jdk-19.0.1+10/bin/java.exe -Xms8G -Xmx8G -XX:+UseLargePages -Xlog:gc+init -Xlog:pagesize=trace:file=pagesize.txt::filecount=0 -Xlog:os=trace:file=os.txt::filecount=0 --versionHere’s a snippet of the beginning of pagesize.txt.
[0.070s][trace][pagesize] Attempt special mapping: size: 240M, alignment: 2M
[0.071s][debug][pagesize] Reserving large pages in a single large chunk.
[0.320s][info ][pagesize] CodeHeap 'non-nmethods': min=4M max=6M base=0x000002a9c5c00000 page_size=2M size=6M
[0.320s][info ][pagesize] CodeHeap 'profiled nmethods': min=4M max=116M base=0x000002a9be800000 page_size=2M size=116M
[0.320s][info ][pagesize] CodeHeap 'non-profiled nmethods': min=4M max=118M base=0x000002a9c6200000 page_size=2M size=118M
[0.324s][trace][pagesize] Attempt special mapping: size: 8G, alignment: 4M
[0.324s][debug][pagesize] Reserving large pages in a single large chunk.
[0.948s][info ][pagesize] Heap: min=8G max=8G base=0x0000000600000000 page_size=4K size=8G
And below is the beginning of the os.txt log. Notice that the arguments to VirtualAlloc are logged, as well as its return value and the error code when a failure occurs. For example, the 2nd-to-last line shows that the 8GB allocation with large pages failed with error code 1450 (ERROR_NO_SYSTEM_RESOURCES aka Insufficient system resources exist to complete the requested service).
[0.052s][debug][os] Initial active processor count set to 12
[0.059s][trace][os] VirtualAlloc(0x0000000000000000, 8192, 2000, 4) returned 0x000002a9b5c60000.
[0.060s][trace][os] VirtualAlloc(0x000002a9b5c60000, 8192, 1000, 4) returned 0x000002a9b5c60000.
[0.060s][info ][os] SafePoint Polling address, bad (protected) page:0x000002a9b5c60000, good (unprotected) page:0x000002a9b5c61000
[0.065s][trace][os] VirtualAlloc(0x000000bc22600000, 16384, 1000, 4) returned 0x000000bc22600000.
[0.066s][info ][os] attempting shared library load of C:\java\binaries\jdk\x64\jdk-19.0.1+10\bin\java.dll
[0.069s][info ][os] shared library load of C:\java\binaries\jdk\x64\jdk-19.0.1+10\bin\java.dll was successful
[0.319s][trace][os] VirtualAlloc(0x0000000000000000, 251658240, 20003000, 40) returned 0x000002a9be800000.
[0.320s][trace][os] VirtualAlloc(0x0000000000000000, 65536, 2000, 4) returned 0x000002a9b5c80000.
[0.320s][trace][os] VirtualAlloc(0x000002a9b5c80000, 32768, 1000, 4) returned 0x000002a9b5c80000.
[0.320s][trace][os] VirtualAlloc(0x0000000000000000, 983040, 2000, 4) returned 0x000002a9b5c90000.
[0.320s][trace][os] VirtualAlloc(0x000002a9b5c90000, 32768, 1000, 4) returned 0x000002a9b5c90000.
[0.320s][trace][os] VirtualAlloc(0x0000000000000000, 983040, 2000, 4) returned 0x000002a9b5d80000.
[0.320s][trace][os] VirtualAlloc(0x000002a9b5d80000, 32768, 1000, 4) returned 0x000002a9b5d80000.
[0.944s][info ][os] VirtualAlloc(0x0000000600000000, 8589934592, 20003000, 4) failed (1450).
[0.947s][trace][os] VirtualAlloc(0x0000000600000000, 8589934592, 2000, 4) returned 0x0000000600000000.The JBS bug made it sound like this feature worked in JDK 17 so I ran the same command using JDK 17.0.5.
/c/java/binaries/jdk/x64/jdk-17.0.5+8/bin/java.exe -Xms8G -Xmx8G -XX:+UseLargePages -Xlog:gc+init -Xlog:pagesize=trace:file=pagesize17.txt::filecount=0 -Xlog:os=trace:file=os17.txt::filecount=0 --versionThe head of pagesize17.txt looks quite similar to its JDK 19 counterpart.
[0.122s][trace][pagesize] Attempt special mapping: size: 240M, alignment: 2M
[0.122s][debug][pagesize] Reserving large pages in a single large chunk.
[0.434s][info ][pagesize] CodeHeap 'non-nmethods': min=4M max=6M base=0x000001c237e00000 page_size=2M size=6M
[0.434s][info ][pagesize] CodeHeap 'profiled nmethods': min=4M max=116M base=0x000001c238400000 page_size=2M size=116M
[0.434s][info ][pagesize] CodeHeap 'non-profiled nmethods': min=4M max=118M base=0x000001c23f800000 page_size=2M size=118M
[0.435s][trace][pagesize] Attempt special mapping: size: 8G, alignment: 4M
[0.435s][debug][pagesize] Reserving large pages in a single large chunk.
[1.039s][info ][pagesize] Heap: min=8G max=8G base=0x0000000600000000 page_size=4K size=8GInterestingly, the os17.txt shows the same failure! The difference is that there is no warning message output to the console.
[0.078s][debug][os] Initial active processor count set to 12
[0.083s][trace][os] VirtualAlloc(0x0000000000000000, 8192, 2000, 4) returned 0x000001c22fa90000.
[0.083s][trace][os] VirtualAlloc(0x000001c22fa90000, 8192, 1000, 4) returned 0x000001c22fa90000.
[0.083s][info ][os] SafePoint Polling address, bad (protected) page:0x000001c22fa90000, good (unprotected) page:0x000001c22fa91000
[0.086s][trace][os] VirtualAlloc(0x000000307c500000, 16384, 1000, 4) returned 0x000000307c500000.
[0.086s][info ][os] attempting shared library load of C:\java\binaries\jdk\x64\jdk-17.0.5+8\bin\java.dll
[0.121s][info ][os] shared library load of C:\java\binaries\jdk\x64\jdk-17.0.5+8\bin\java.dll was successful
[0.433s][trace][os] VirtualAlloc(0x0000000000000000, 251658240, 20003000, 40) returned 0x000001c237e00000.
[0.434s][trace][os] VirtualAlloc(0x0000000000000000, 65536, 2000, 4) returned 0x000001c22fab0000.
[0.434s][trace][os] VirtualAlloc(0x000001c22fab0000, 32768, 1000, 4) returned 0x000001c22fab0000.
[0.434s][trace][os] VirtualAlloc(0x0000000000000000, 983040, 2000, 4) returned 0x000001c22fac0000.
[0.434s][trace][os] VirtualAlloc(0x000001c22fac0000, 32768, 1000, 4) returned 0x000001c22fac0000.
[0.434s][trace][os] VirtualAlloc(0x0000000000000000, 983040, 2000, 4) returned 0x000001c22fbb0000.
[0.434s][trace][os] VirtualAlloc(0x000001c22fbb0000, 32768, 1000, 4) returned 0x000001c22fbb0000.
[1.038s][info ][os] VirtualAlloc(0x0000000600000000, 8589934592, 20003000, 4) failed (1450).
[1.039s][trace][os] VirtualAlloc(0x0000000600000000, 8589934592, 2000, 4) returned 0x0000000600000000.The git blame view of the JDK 19 warning message shows that it was introduced by 8271195: Use largest available large page size smaller than LargePage… · openjdk/jdk@08cadb4 (github.com). The corresponding JBS issue is [JDK-8271195] Use largest available large page size smaller than LargePageSizeInBytes when available – Java Bug System (openjdk.org) and it confirms that the message was introduced in JDK 19. So the problem indeed exists in both builds but JDK 19 provides a better experience because it warns you that it could not use large pages as you requested.
Outstanding Items
- How do you enable the Local Security Policy on Windows 10 Home Edition? secpol.msc windows home