Saint's Log

hsdis LLVM backend for Windows ARM64

8253757: Add LLVM-based backend for hsdis by magicus · Pull Request #7531 makes it possible to easily use LLVM as the hsdis backend. An LLVM installation is required for this. The official LLVM builds for the Windows platform do not work for building hsdis because they do not have all the prerequisite LLVM include files. See Building LLVM for Windows ARM64 – Saint’s Log (swesonga.org) for instructions on how to build LLVM for ARM64 Windows (on an x64 Windows host). To configure OpenJDK for LLVM as an hsdis backend on Windows ARM64, use this command:

bash configure --openjdk-target=aarch64-unknown-cygwin \
 --with-hsdis=llvm \
 --with-llvm=/cygdrive/d/dev/software/llvm-aarch64/

The JDK and hsdis can then be built as usual with these commands:

make images
make build-hsdis
make install-hsdis
cp /cygdrive/d/dev/software/llvm-aarch64/bin/LLVM-C.dll build/windows-aarch64-server-slowdebug/jdk/bin/

The generated JDK can then be deployed to an ARM64 machine like the Surface Pro X. To test LLVM’s disassembly, use the -XX:CompileCommand flag on the ARM64 machine:

/java -XX:CompileCommand="print java.lang.String::checkIndex" -version

Behind the Scenes

Missing Include File that Exists?

The path given to --with-llvm needs to be a Cygwin path if building in Cygwin. Otherwise, the build-hsdis target will fail with this error: c:\...\jdk\src\utils\hsdis\llvm\hsdis-llvm.cpp(58): fatal error C1083: Cannot open include file: 'llvm-c/Disassembler.h': No such file or directory. I caught this by inspecting build\windows-aarch64-server-release\make-support\failure-logs\support_hsdis_hsdis-llvm.obj.cmdline after the build failed. This was the only include that didn’t have Cygwin paths: -IC:/dev/repos/llvm-project/build_llvm_AArch64/install_local/include

Investigating Missing Disassembly

My first disassembly attempt did not work – only abstract disassembly was displayed:

...
  # {method} {0x000002ca9940f2e8} 'checkIndex' '(II)V' in 'java/lang/String'
  # parm0:    c_rarg1   = int
  # parm1:    c_rarg2   = int
  #           [sp+0x30]  (sp of caller)
  0x000002ca87ad3940: 1f20 03d5 | e953 40d1 | 3f01 00f9 | ffc3 00d1 | fd7b 02a9 | a201 f837 | 3f00 026b | e200 0054
  0x000002ca87ad3960: fd7b 42a9 | ffc3 0091
...

I verified that hsdis-aarch64.dll was present in the JDK’s bin folder. That was the only issue I had seen before that caused this behavior so I dug around to find the code that loads the hsdis DLL. A search for the “hsdis-” DLL prefix in the sources reveals the hsdis_library_name string used in the Disassembler::dll_load method. Notice that there is a Verbose flag that can display what is happening when loading the hsdis DLL!

void* Disassembler::dll_load(char* buf, int buflen, int offset, char* ebuf, int ebuflen, outputStream* st) {
  int sz = buflen - offset;
  int written = jio_snprintf(&buf[offset], sz, "%s%s", hsdis_library_name, os::dll_file_extension());
  if (written < sz) { // written successfully, not truncated.
    if (Verbose) st->print_cr("Trying to load: %s", buf);
    return os::dll_load(buf, ebuf, ebuflen);
  } else if (Verbose) {
    st->print_cr("Try to load hsdis library failed: the length of path is beyond the OS limit");
  }
  return NULL;
}

This turns out to be a JVM flag! I try passing it to java.exe but -Verbose doesn’t do anything. Learn from HotSpot Command-Line Flags Overhaul – Design Doc – OpenJDK Wiki (java.net) that it’s a -XX: flag. Trying to use it causes JVM to complain that it is a develop-only flag.

Error: VM option 'Verbose' is develop and is available only in debug version of VM.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

The –enable-debug flag documented at Building OpenJDK (java.net) is required to generate a debug VM.

bash configure --openjdk-target=aarch64-unknown-cygwin \
 --with-hsdis=llvm \
 --with-llvm=/cygdrive/d/dev/software/llvm-aarch64/ \
 --enable-debug

Running the debug JVM with the verbose flag now displays some diagnostic information:

.\java.exe -XX:CompileCommand="print java.lang.String::checkIndex" -XX:+Verbose -XX:+PrintMiscellaneous
CompileCommand: print java/lang/String.checkIndex bool print = true


============================= C1-compiled nmethod ==============================
----------------------------------- Assembly -----------------------------------
Trying to load: C:\dev\software\jdk-aarch64\jdk\bin\server\hsdis-aarch64.dll
Trying to load: C:\dev\software\jdk-aarch64\jdk\bin\server\hsdis-aarch64.dll
Trying to load: C:\dev\software\jdk-aarch64\jdk\bin\hsdis-aarch64.dll
Trying to load: hsdis-aarch64.dll
Could not load hsdis-aarch64.dll; Can't find dependent libraries; PrintAssembly defaults to abstract disassembly.

...

The error message substring “find dependent libraries” appears only once in the hotspot source code – in os::dll_load (which is called by Disassembler::dll_load). This error is displayed because LoadLibrary returns ERROR_MOD_NOT_FOUND.

The case of the DLL that refuses to load – The Old New Thing (microsoft.com) mentions loader snaps. The loader snaps are an option for the gflags tool found in the Windows Kits folder. The docs explain that GFlags is included in the Debugging Tools for Windows 10 (WinDbg) so a search for “Debugging Tools for Windows arm64” leads to Debugging ARM64 – Windows drivers. This says to install the Windows SDK, after which I now have the gflags binary (in the x86 folder)!

C:\Program Files (x86)\Windows Kits\10\Debuggers\arm64\gflags.exe

I still wasn’t sure how to see the snaps output. Show Loader Snaps in GFlags.exe, fails to capture any output in WinDbg – Stack Overflow implies that I should be able to use WinDbg to see what is failing to load.

Turns out the loader snaps aren’t really necessary. There is some critical info in the WinDbg diagnostic output:

2698:2e5c @ 03908953 - LdrpResolveDllName - ENTER: DLL name: .\LLVM-C.dll
2698:2e5c @ 03908953 - LdrpResolveDllName - RETURN: Status: 0xc0000135
...
2698:2e5c @ 03908968 - LdrpResolveDllName - ENTER: DLL name: C:\WINDOWS\LLVM-C.dll
2698:2e5c @ 03908968 - LdrpResolveDllName - RETURN: Status: 0xc0000135
...
2698:2e5c @ 03908968 - LdrpSearchPath - RETURN: Status: 0xc0000135
2698:2e5c @ 03908968 - LdrpProcessWork - ERROR: Unable to load DLL: "LLVM-C.dll", Parent Module: "C:\dev\software\jdk-aarch64\jdk\bin\hsdis-aarch64.dll", Status: 0xc0000135

hsdis-aarch64.dll is not being loaded because LLVM-C.dll cannot be found! Still learning the need for reading the full instructions to avoid unnecessary pain.