Saint's Log

Troubleshooting hsdis LLVM backend MSVC Linker Errors

The post about Exploring the hsdis LLVM Support PR mentioned link errors when building hsdis using an LLVM backend on Windows (x86-64 host building JDK for the x86-64 platform). Before we look at why linking fails, we can get a simple repro for the error from the Cygwin logs. To get the command line used to invoke the linker, run make LOG=debug build-hsdis. Search the output for link.exe to find the failing command or open build\windows-x86_64-server-release\support\hsdis\BUILD_HSDIS_link.cmdline. Change the path from Cygwin to Windows style so that the command can be run in the x64 Native Tools Command Prompt.

cd C:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\

c:\progra~2\micros~3\2019\enterp~1\vc\tools\msvc\1429~1.301\bin\hostx86\x64\link.exe -nologo -libpath:c:\dev\repos\llvm-project\build_llvm\install_local\\lib -dll -debug "-pdb:c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.pdb" "-map:c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.map" "-implib:c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.lib" -libpath:c:\progra~2\micros~3\2019\enterp~1\vc\tools\msvc\1429~1.301\atlmfc\lib\x64 -libpath:c:\progra~2\micros~3\2019\enterp~1\vc\tools\msvc\1429~1.301\lib\x64 -libpath:c:\progra~2\wi3cf2~1\netfxsdk\4.8\lib\um\x64 -libpath:c:\progra~2\wi3cf2~1\10\lib\100190~1.0\ucrt\x64 -libpath:c:\progra~2\wi3cf2~1\10\lib\100190~1.0\um\x64 -out:c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.dll c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis-llvm.obj c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.dll.res

These are the resulting link errors mentioned in Exploring the hsdis LLVM Support PR.

   Creating library c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.lib and object c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.exp
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMCreateDisasm referenced in function "public: __cdecl hsdis_backend::hsdis_backend(unsigned __int64,unsigned __int64,unsigned char *,unsigned __int64,void * (__cdecl*)(void *,char const *,void *),void *,int (__cdecl*)(void *,char const *,...),void *,char const *,int)" (??0hsdis_backend@@QEAA@_K0PEAE0P6APEAXPEAXPEBD2@Z2P6AH23ZZ23H@Z)
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMSetDisasmOptions referenced in function "public: __cdecl hsdis_backend::hsdis_backend(unsigned __int64,unsigned __int64,unsigned char *,unsigned __int64,void * (__cdecl*)(void *,char const *,void *),void *,int (__cdecl*)(void *,char const *,...),void *,char const *,int)" (??0hsdis_backend@@QEAA@_K0PEAE0P6APEAXPEAXPEBD2@Z2P6AH23ZZ23H@Z)
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMDisasmDispose referenced in function "public: __cdecl hsdis_backend::~hsdis_backend(void)" (??1hsdis_backend@@QEAA@XZ)
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMDisasmInstruction referenced in function "protected: virtual unsigned __int64 __cdecl hsdis_backend::decode_instruction(unsigned __int64,unsigned __int64,unsigned __int64)" (?decode_instruction@hsdis_backend@@MEAA_K_K00@Z)
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86TargetInfo referenced in function LLVMInitializeNativeTarget
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86Target referenced in function LLVMInitializeNativeTarget
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86TargetMC referenced in function LLVMInitializeNativeTarget
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86AsmPrinter referenced in function LLVMInitializeNativeAsmPrinter
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86Disassembler referenced in function LLVMInitializeNativeDisassembler
c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.dll : fatal error LNK1120: 9 unresolved externals

The hsdis_backend class uses functions in the LLVM libraries that cannot be resolved:

• LLVMCreateDisasm, LLVMSetDisasmOptions, LLVMDisasmDispose, LLVMDisasmInstruction
• LLVMInitializeX86TargetInfo, LLVMInitializeX86Target, LLVMInitializeX86TargetMC, LLVMInitializeX86AsmPrinter, LLVMInitializeX86Disassembler

The X86 specific symbols are referenced by the calls to LLVMInitializeNativeTarget, LLVMInitializeNativeAsmPrinter, and LLVMInitializeNativeDisassembler.

Tracking Down the Linker Issues

We can use the DUMPBIN tool to inspect the LLVM libraries.

cd c:\dev\repos\llvm-project\build_llvm\install_local\lib
dumpbin LLVMX86Disassembler.lib
dumpbin /symbols /out:LLVMX86Disassembler.txt LLVMX86Disassembler.lib

The forfiles command is useful for dumping the symbols from all the libraries (forfiles was suggested at How to do something to each file in a directory with a batch script). I thought forfiles would work without the “cmd /c” prefix but that only resulted in dumpbin /summary output!

cd c:\dev\repos\llvm-project\build_llvm\install_local\lib
forfiles /m *.lib /c "cmd /c dumpbin /symbols /out:@fname.txt @file"

Now we can easily search for the symbols of interest, e.g.

> findstr /sipnc:"LLVMInitializeX86Disassembler" *.txt
LLVMX86Disassembler.txt:151:090 00000000 SECT2C notype ()    External     | LLVMInitializeX86Disassembler
LLVMX86Disassembler.txt:926:397 00000000 SECT6B notype       Static       | $unwind$LLVMInitializeX86Disassembler
LLVMX86Disassembler.txt:929:39A 00000000 SECT6C notype       Static       | $pdata$LLVMInitializeX86Disassembler

So there really is no such symbol in this lib folder! I’m guessing I need to add another lib folder to the path. A quick search for LLVMInitializeX86Disassembler leads to this post on Using the LLVM MC Disassembly API. It mentions using llvm-config to set the linker flags. Shouldn’t running the bash configure command take care of this? Let’s see what’s in the configure output:

...
checking what hsdis backend to use... 'llvm'
checking for LLVM_CONFIG... C:/dev/repos/llvm-project/build_llvm/install_local/bin [user supplied]
/cygdrive/c/dev/repos/java/forks/jdk/build/.configure-support/generated-configure.sh: line 135451: C:/dev/repos/llvm-project/build_llvm/install_local/bin: Is a directory
/cygdrive/c/dev/repos/java/forks/jdk/build/.configure-support/generated-configure.sh: line 135452: C:/dev/repos/llvm-project/build_llvm/install_local/bin: Is a directory
/cygdrive/c/dev/repos/java/forks/jdk/build/.configure-support/generated-configure.sh: line 135453: C:/dev/repos/llvm-project/build_llvm/install_local/bin: Is a directory
...

Well, that could be the problem! I think I need to fix the llvm-config path in Cygwin by appending /llvm-config to LLVM_CONFIG.

bash configure --with-hsdis=llvm LLVM_CONFIG=C:/dev/repos/llvm-project/build_llvm/install_local/bin/llvm-config --with-llvm=C:/dev/repos/llvm-project/build_llvm/install_local/

Sure enough, that was the problem! The bash configure output (below) now looks good and make build-hsdis now works. The fix for this would be to ensure bash configure fails if LLVM_CONFIG is set to the directory instead of the executable!

checking what hsdis backend to use... 'llvm'
checking for LLVM_CONFIG... C:/dev/repos/llvm-project/build_llvm/install_local/bin/llvm-config [user supplied]
checking for number of cores... 8
...

$ make build-hsdis
Building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Creating support/hsdis/hsdis.dll from 1 file(s)
Finished building target 'build-hsdis' in configuration 'windows-x86_64-server-release'

Notice from the new build command line in build\windows-x86_64-server-release\support\hsdis\BUILD_HSDIS_link.cmdline that there are now many .lib files supplied to the linker! These are the lib files that I was inspecting with dumpbin so my earlier hypothesis was wrong (there were no additional .lib files required, the ones I was looking at were simply not being passed to the linker).

/cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/fixpath exec
 /cygdrive/c/progra~2/micros~3/2019/enterp~1/vc/tools/msvc/1429~1.301/bin/hostx86/x64/link.exe
 -nologo
 -libpath:/cygdrive/c/dev/repos/llvm-project/build_llvm/install_local//lib
 -dll
 -debug
 "-pdb:/cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.pdb"
 "-map:/cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.map"
 "-implib:/cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.lib"
 -libpath:/cygdrive/c/progra~2/micros~3/2019/enterp~1/vc/tools/msvc/1429~1.301/atlmfc/lib/x64
 -libpath:/cygdrive/c/progra~2/micros~3/2019/enterp~1/vc/tools/msvc/1429~1.301/lib/x64
 -libpath:/cygdrive/c/progra~2/wi3cf2~1/netfxsdk/4.8/lib/um/x64
 -libpath:/cygdrive/c/progra~2/wi3cf2~1/10/lib/100190~1.0/ucrt/x64
 -libpath:/cygdrive/c/progra~2/wi3cf2~1/10/lib/100190~1.0/um/x64
 -out:/cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.dll 
 /cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis-llvm.obj
 /cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.dll.res
 llvmx86targetmca.lib llvmmca.lib llvmx86disassembler.lib llvmx86asmparser.lib llvmx86codegen.lib llvmcfguard.lib llvmglobalisel.lib llvmx86desc.lib llvmx86info.lib llvmmcdisassembler.lib llvmselectiondag.lib llvminstrumentation.lib llvmasmprinter.lib llvmdebuginfomsf.lib llvmcodegen.lib llvmtarget.lib llvmscalaropts.lib llvminstcombine.lib llvmaggressiveinstcombine.lib llvmtransformutils.lib llvmbitwriter.lib llvmanalysis.lib llvmprofiledata.lib llvmdebuginfodwarf.lib llvmobject.lib llvmtextapi.lib llvmmcparser.lib llvmmc.lib llvmdebuginfocodeview.lib llvmbitreader.lib llvmcore.lib llvmremarks.lib llvmbitstreamreader.lib llvmbinaryformat.lib llvmsupport.lib llvmdemangle.lib

Now running make install-hsdis copies hsdis-amd64.dll into /build/windows-x86_64-server-release/jdk/bin. The LLVM hsdis backend can now be used to disassemble instructions:

$ ./java -XX:CompileCommand="print java.lang.String::checkIndex" -version
CompileCommand: print java/lang/String.checkIndex bool print = true

============================= C2-compiled nmethod ==============================
----------------------------------- Assembly -----------------------------------

Compiled method (c2)    5912   60       4       java.lang.String::checkIndex (10 bytes)
 total in heap  [0x00000162f39e3090,0x00000162f39e3308] = 632
 relocation     [0x00000162f39e31e8,0x00000162f39e3200] = 24
 main code      [0x00000162f39e3200,0x00000162f39e3280] = 128
 stub code      [0x00000162f39e3280,0x00000162f39e3298] = 24
 oops           [0x00000162f39e3298,0x00000162f39e32a0] = 8
 metadata       [0x00000162f39e32a0,0x00000162f39e32a8] = 8
 scopes data    [0x00000162f39e32a8,0x00000162f39e32c0] = 24
 scopes pcs     [0x00000162f39e32c0,0x00000162f39e3300] = 64
 dependencies   [0x00000162f39e3300,0x00000162f39e3308] = 8

[Disassembly]
--------------------------------------------------------------------------------
[Constant Pool (empty)]

--------------------------------------------------------------------------------

[Verified Entry Point]
  # {method} {0x000001628800f2f0} 'checkIndex' '(II)V' in 'java/lang/String'
  # parm0:    rdx       = int
  # parm1:    r8        = int
  #           [sp+0x30]  (sp of caller)
  0x00000162f39e3200:           movl    %eax, -0x7000(%rsp)
  0x00000162f39e3207:           pushq   %rbp
  0x00000162f39e3208:           subq    $0x20, %rsp
  0x00000162f39e320c:           testl   %r8d, %r8d
  0x00000162f39e320f:           jl      0x2f
  0x00000162f39e3211:           cmpl    %r8d, %edx
  0x00000162f39e3214:           jae     0x16
  0x00000162f39e3216:           vzeroupper
  0x00000162f39e3219:           addq    $0x20, %rsp
  0x00000162f39e321d:           popq    %rbp
  0x00000162f39e321e:           cmpq    0x338(%r15), %rsp   ;   {poll_return}
  0x00000162f39e3225:           ja      0x29
  0x00000162f39e322b:           retq
  0x00000162f39e322c:           movl    %edx, %ebp
  0x00000162f39e322e:           movl    %r8d, (%rsp)
  0x00000162f39e3232:           movl    $0xffffffe4, %edx
  0x00000162f39e3237:           nop
  0x00000162f39e3238:           vzeroupper
  0x00000162f39e323b:           callq   -0x7a80f40          ; ImmutableOopMap {}
                                                            ;*invokestatic checkIndex {reexecute=0 rethrow=0 return_oop=0}
                                                            ; - java.lang.String::checkIndex@5 (line 4554)
                                                            ;   {runtime_call UncommonTrapBlob}
  0x00000162f39e3240:           movl    %edx, %ebp
  0x00000162f39e3242:           movl    %r8d, (%rsp)
  0x00000162f39e3246:           movl    $0xffffffcc, %edx
...

References

Here are some of the bugs/questions I looked at when investigating these failures. Stack overflow taught me about dumpbin and C++ decorated names/ the undname tool.

• unresolved external symbols in iosfwd with _HAS_CONSTEXPR_CHAR_TRAITS and __builtin_XXX functions when compiling with clang/LLVM in C++17 – Visual Studio Feedback
• unable to use clang-cl + coroutines due to unresolved external symbol _coro_resume – Visual Studio Feedback
• c++ – LLVM 3.4 linker errors on VS 2012 – Stack Overflow
• Building LLVM with CMake — LLVM 15.0.0git documentation (I wondered if perhaps I had built LLVM incorrectly)
• visual studio – Clang+LLVM static library linking error on Windows – Why would the symbols be different? – Stack Overflow
• Unresolved Externals in C++: Visual C++ mangles method signature differently from mangled method in dll – Stack Overflow

Exploring the hsdis LLVM Support PR

The previous post described how LLVM can be configured as the disassembly backend for hsdis. Here, I explain the process it took for me to figure out the details of the change adding support for LLVM. One of the first things to do when learning these details of this change is to build it. Since I’m using my own fork of the OpenJDK repo, I need to add the upstream repo to my remotes. This makes it possible to fetch commits from PRs submitted to the upstream repo.

cd ~/repos/forks/jdk
git remote add upstream https://github.com/openjdk/jdk
git fetch upstream

The LLVM-backend PR has only 1 commit (as of this writing). Create a new branch then cherry-pick that commit (I was on commit 77757ba9 when I wrote this.

git checkout -b hsdis-backend-llvm
git cherry-pick effac9b87ecb3cdc8d3d149b9dcd72ee1ea88fec

Some conflicts need to be resolved:

Performing inexact rename detection: 100% (88356/88356), done.
Auto-merging make/autoconf/spec.gmk.in
Auto-merging make/autoconf/jdk-options.m4
CONFLICT (content): Merge conflict in make/autoconf/jdk-options.m4
Auto-merging make/Hsdis.gmk
error: could not apply effac9b87ec... Create hsdis backend using LLVM

The files view of PR 5920 shows that the change to make/autoconf/jdk-options.m4 is mostly adding another branch to the if-else statements checking the hsdis backend. Lines 841-854 of PR 5920 can therefore be added just before the else on line 890 to resolve the conflict. The diff from my branch can be seen here.

Building the Changes on macOS ARM64

Install LLVM using homebrew (if it is not already installed).

brew install llvm

Set the the LDFLAGS and CPPFLAGS environment variables then run printenv | grep -i flags to verify that the flags have been set correctly. Exporting CC and CXX is crucial since that is how to let bash configure know that we need a custom compiler for the build!

# export LDFLAGS="-L/opt/homebrew/opt/llvm/lib"
# export CFLAGS="-I/opt/homebrew/opt/llvm/include"
export CC=/opt/homebrew/opt/llvm/bin/clang
export CXX=$(CC)++
bash configure --with-hsdis=llvm LLVM_CONFIG=/opt/homebrew/opt/llvm/bin

Run make build-hsdis in the root folder of the jdk repo.

If the proper flags have not been set, make will fail with the error below. Run make --debug=v for additional information on what make is doing.

saint@Saints-MBP-2021 jdk % make build-hsdis
Building target 'build-hsdis' in configuration 'macosx-aarch64-server-release'
/Users/saint/repos/java/forks/jdk/src/utils/hsdis/llvm/hsdis-llvm.cpp:58:10: fatal error: 'llvm-c/Disassembler.h' file not found
#include <llvm-c/Disassembler.h>
         ^~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[3]: *** [/Users/saint/repos/java/forks/jdk/build/macosx-aarch64-server-release/support/hsdis/hsdis-llvm.o] Error 1
make[2]: *** [build-hsdis] Error 2

ERROR: Build failed for target 'build-hsdis' in configuration 'macosx-aarch64-server-release' (exit code 2)

After all that fidgeting around, the fix is as simple as updating your path to include LLVM <insert facepalm / clown>. This is what installing LLVM using brew ends with:

...
==> llvm
To use the bundled libc++ please add the following LDFLAGS:
  LDFLAGS="-L/opt/homebrew/opt/llvm/lib -Wl,-rpath,/opt/homebrew/opt/llvm/lib"

llvm is keg-only, which means it was not symlinked into /opt/homebrew,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have llvm first in your PATH, run:
  echo 'export PATH="/opt/homebrew/opt/llvm/bin:$PATH"' >> ~/.zshrc

For compilers to find llvm you may need to set:
  export LDFLAGS="-L/opt/homebrew/opt/llvm/lib"
  export CPPFLAGS="-I/opt/homebrew/opt/llvm/include"

My MacBook didn’t even have a ~/.zshrc file. Setting the PATH using the suggestion above fixed the build errors!

echo 'export PATH="/opt/homebrew/opt/llvm/bin:$PATH"' >> ~/.zshrc

Now open a new terminal and configure the repo (no need for LLVM_CONFIG).

% bash configure --with-hsdis=llvm
% make build-hsdis

Interestingly, running make images does not work on subsequent attempts?! After further investigation, it turns out that the clang compiler installed by brew cannot successfully compile the OpenJDK sources. Why does it issue warnings that Apple’s clang compiler does not?

In file included from /Users/saint/repos/java/forks/jdk/src/hotspot/cpu/aarch64/abstractInterpreter_aarch64.cpp:31:
In file included from /Users/saint/repos/java/forks/jdk/src/hotspot/share/runtime/frame.inline.hpp:42:
In file included from /Users/saint/repos/java/forks/jdk/src/hotspot/cpu/aarch64/frame_aarch64.inline.hpp:31:
In file included from /Users/saint/repos/java/forks/jdk/src/hotspot/cpu/aarch64/pauth_aarch64.hpp:28:
/Users/saint/repos/java/forks/jdk/src/hotspot/os_cpu/bsd_aarch64/pauth_bsd_aarch64.inline.hpp:29:10: fatal error: 'ptrauth.h' file not found
#include <ptrauth.h>
         ^~~~~~~~~~~
1 error generated.
make[3]: *** [/Users/saint/repos/java/forks/jdk/build/macosx-aarch64-server-release/hotspot/variant-server/libjvm/objs/abstractInterpreter_aarch64.o] Error 1
m

To work around this, first build the JDK using Apple’s clang. Next, add brew’s LLVM installation to the PATH, then configure for hsdis. Finally, build hsdis:

# Warning: ensure /opt/homebrew/opt/llvm/bin is not in the PATH
cd ~/repos/java/forks/jdk
bash configure
make images

# Now add brew's LLVM to the PATH before running bash configure
export OLDPATH=$PATH
export PATH="/opt/homebrew/opt/llvm/bin:$PATH"

bash configure --with-hsdis=llvm
make build-hsdis
make install-hsdis
export PATH=$OLDPATH

# Why doesn't install-hsdis do this?
cd build/macosx-aarch64-server-release
cp support/hsdis/libhsdis.dylib jdk/bin/

The JVM did not appear to be generating the disassembly even with this approach. A quick search for hsdis not printing assembly macOS leads to this post mentioning the error Could not load hsdis-amd64.dylib; library not loadable; PrintAssembly is disabled. This reminds me that theRealAph had pointed out that the library seems to be built with the wrong name, so the runtime doesn’t find it. So I just needed to specify that file name when copying the hsdis dylib in the last step!

cp support/hsdis/libhsdis.dylib jdk/bin/hsdis-aarch64.dylib

Building the Changes on Windows x86-64

Install the 64-bit Windows LLVM. Configure the OpenJDK repo using both the --with-hsdis and LLVM_CONFIG options as shown. I needed to use the 8.3 path name (obtained using the command suggested on StackOverflow) for value of the LLVM_CONFIG parameter.

bash configure --with-hsdis=llvm LLVM_CONFIG=C:/PROGRA~1/LLVM/bin

Unfortunately, this is not sufficient to enable building on Windows as detailed by this error:

$ make build-hsdis
Building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Creating support/hsdis/hsdis.dll from 1 file(s)
/usr/bin/bash: x86_64-w64-mingw32-g++: command not found
make[3]: *** [Hsdis.gmk:135: /..../build/windows-x86_64-server-release/support/hsdis/hsdis-llvm.obj] Error 127
make[2]: *** [make/Main.gmk:530: build-hsdis] Error 2

ERROR: Build failed for target 'build-hsdis' in configuration 'windows-x86_64-server-release' (exit code 2)

Jorn fixed this so we can add Jorn’s upstream JDK, fetch its commits, then cherry pick the commit with the fix.

git remote add jorn https://github.com/JornVernee/jdk/
git fetch jorn
git cherry-pick 8de8b763c9159f84bcc044c04ee2fac9f2390774

Some conflicts in make/Hsdis.gmk need to be resolved. This is straightforward since Jorn’s change splits the existing binutils Windows code into the first branch of an if-statement then adds support for the LLVM backend in the else case. The resolved conflicts are in my fork in the branch. The repo should now be configured with the additional --with-llvm option added by Jorn.

bash configure --with-hsdis=llvm LLVM_CONFIG=C:/PROGRA~1/LLVM/bin --with-llvm=C:/PROGRA~1/LLVM

Running make build-hsdis results in errors about missing LLVM includes.

$ make build-hsdis
Building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Creating support/hsdis/hsdis.dll from 1 file(s)
d:\.....\jdk\src\utils\hsdis\llvm\hsdis-llvm.cpp(58): fatal error C1083: Cannot open include file: 'llvm-c/Disassembler.h': No such file or directory
make[3]: *** [Hsdis.gmk:142: /cygdrive/d/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis-llvm.obj] Error 1
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [make/Main.gmk:530: build-hsdis] Error 2

Let’s try setting CC and CXX then rerunning the above configure command.

export CC=C:/PROGRA~1/LLVM/bin/clang.exe
export CXX=C:/PROGRA~1/LLVM/bin/clang++.exe

Turns out a Microsoft compiler is required!

configure: Will use user supplied compiler CC=C:/PROGRA~1/LLVM/bin/clang.exe
checking resolved symbolic links for CC... no symlink
configure: The C compiler (located as C:/PROGRA~1/LLVM/bin/clang.exe) does not seem to be the required microsoft compiler.
configure: The result from running it was: "clang: error: no input files"
configure: error: A microsoft compiler is required. Try setting --with-tools-dir.
configure exiting with result code 1

But let’s see what happens if we change the toolchain type to clang:

# This command does not work
bash configure --with-hsdis=llvm LLVM_CONFIG=C:/PROGRA~1/LLVM/bin --with-llvm=C:/PROGRA~1/LLVM --with-toolchain-type=clang

I guess they were serious about that since clang is not valid on this platform.

configure: Toolchain type clang is not valid on this platform.
configure: Valid toolchains: microsoft.
configure: error: Cannot continue.
configure exiting with result code 1

Indeed, clang is not a valid toolchain for Windows as declared in make/autoconf/toolchain.m4. Open question: how is the VALID_TOOLCHAIN_windows actually checked? So we can now unset the environment variables.

unset CC
unset CXX

This brought me back to the first thing I should have done when I saw the “No such file or directory” error – verifying that the file existed on disk! This is all there is there:

$ ls C:/PROGRA~1/LLVM/include/llvm-c
Remarks.h  lto.h

Well, turns out this is the issue that led Jorn to build LLVM manually. I now know what the needed header files being referred to are. So let’s build LLVM using Jorn’s steps.

git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build_llvm
cd build_llvm
cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=X86" -D"CMAKE_BUILD_TYPE:STRING=Release" -D"CMAKE_INSTALL_PREFIX=install_local" -A x64 -T host=x64
cmake --build . --config Release --target install

The last command fails with the error below!??? Why can’t anything just simply work?

  Building Opts.inc...
  '..\..\RelWithDebInfo\bin\llvm-tblgen.exe' is not recognized as an internal or external command,
  operable program or batch file.
C:\Program Files\Microsoft Visual Studio\2022\Preview\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(243,5): error MSB8066: Custom build for 'D:\dev\repos\llvm-project\build_llvm\CMakeFiles\dd1f7b42098
1667d7f617e96802947d3\Opts.inc.rule;D:\dev\repos\llvm-project\build_llvm\CMakeFiles\9fbf2dc5caba7f0c75934f43d12abdf5\RcOptsTableGen.rule;D:\dev\repos\llvm-project\llvm\tools\llvm-rc\CMakeLists.txt' exited wit
h code 9009. [D:\dev\repos\llvm-project\build_llvm\tools\llvm-rc\RcOptsTableGen.vcxproj]

Switch to my Surface Book 2 and LLVM builds just fine!

bash configure --with-hsdis=llvm LLVM_CONFIG=C:/dev/repos/llvm-project/build_llvm/install_local/bin --with-llvm=C:/dev/repos/llvm-project/build_llvm/install_local/

Interestingly, this fails with the same errors I saw on macOS:

$ make build-hsdis
Building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Creating support/hsdis/hsdis.dll from 1 file(s)
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMCreateDisasm referenced in function "public: __cdecl hsdis_backend::hsdis_backend(unsigned __int64,unsi...,char const *,int)" (??0hsdis_backend@@QEAA@_K0PEAE0P6APEAXPEAXPEBD2@Z2P6AH23ZZ23H@Z)
...
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86Disassembler referenced in function LLVMInitializeNativeDisassembler
c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.dll : fatal error LNK1120: 9 unresolved externals
make[3]: *** [Hsdis.gmk:142: /cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.dll] Error 1

The PATH environment variable probably needs to be adjusted to work around this.

Update 2022-02-08: the problem above is that bash configure is invoked with the wrong LLVM_CONFIG option – the actual llvm-config executable name is missing. See Troubleshooting hsdis LLVM backend MSVC Linker Errors for details.