—Categories: hsdis, OpenJDK

Troubleshooting hsdis LLVM backend MSVC Linker Errors

The post about Exploring the hsdis LLVM Support PR mentioned link errors when building hsdis using an LLVM backend on Windows (x86-64 host building JDK for the x86-64 platform). Before we look at why linking fails, we can get a simple repro for the error from the Cygwin logs. To get the command line used to invoke the linker, run make LOG=debug build-hsdis. Search the output for link.exe to find the failing command or open build\windows-x86_64-server-release\support\hsdis\BUILD_HSDIS_link.cmdline. Change the path from Cygwin to Windows style so that the command can be run in the x64 Native Tools Command Prompt.

cd C:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\

c:\progra~2\micros~3\2019\enterp~1\vc\tools\msvc\1429~1.301\bin\hostx86\x64\link.exe -nologo -libpath:c:\dev\repos\llvm-project\build_llvm\install_local\\lib -dll -debug "-pdb:c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.pdb" "-map:c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.map" "-implib:c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.lib" -libpath:c:\progra~2\micros~3\2019\enterp~1\vc\tools\msvc\1429~1.301\atlmfc\lib\x64 -libpath:c:\progra~2\micros~3\2019\enterp~1\vc\tools\msvc\1429~1.301\lib\x64 -libpath:c:\progra~2\wi3cf2~1\netfxsdk\4.8\lib\um\x64 -libpath:c:\progra~2\wi3cf2~1\10\lib\100190~1.0\ucrt\x64 -libpath:c:\progra~2\wi3cf2~1\10\lib\100190~1.0\um\x64 -out:c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.dll c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis-llvm.obj c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.dll.res

These are the resulting link errors mentioned in Exploring the hsdis LLVM Support PR.

   Creating library c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.lib and object c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.exp
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMCreateDisasm referenced in function "public: __cdecl hsdis_backend::hsdis_backend(unsigned __int64,unsigned __int64,unsigned char *,unsigned __int64,void * (__cdecl*)(void *,char const *,void *),void *,int (__cdecl*)(void *,char const *,...),void *,char const *,int)" (??0hsdis_backend@@QEAA@_K0PEAE0P6APEAXPEAXPEBD2@Z2P6AH23ZZ23H@Z)
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMSetDisasmOptions referenced in function "public: __cdecl hsdis_backend::hsdis_backend(unsigned __int64,unsigned __int64,unsigned char *,unsigned __int64,void * (__cdecl*)(void *,char const *,void *),void *,int (__cdecl*)(void *,char const *,...),void *,char const *,int)" (??0hsdis_backend@@QEAA@_K0PEAE0P6APEAXPEAXPEBD2@Z2P6AH23ZZ23H@Z)
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMDisasmDispose referenced in function "public: __cdecl hsdis_backend::~hsdis_backend(void)" (??1hsdis_backend@@QEAA@XZ)
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMDisasmInstruction referenced in function "protected: virtual unsigned __int64 __cdecl hsdis_backend::decode_instruction(unsigned __int64,unsigned __int64,unsigned __int64)" (?decode_instruction@hsdis_backend@@MEAA_K_K00@Z)
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86TargetInfo referenced in function LLVMInitializeNativeTarget
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86Target referenced in function LLVMInitializeNativeTarget
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86TargetMC referenced in function LLVMInitializeNativeTarget
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86AsmPrinter referenced in function LLVMInitializeNativeAsmPrinter
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86Disassembler referenced in function LLVMInitializeNativeDisassembler
c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.dll : fatal error LNK1120: 9 unresolved externals

The hsdis_backend class uses functions in the LLVM libraries that cannot be resolved:

The X86 specific symbols are referenced by the calls to LLVMInitializeNativeTarget, LLVMInitializeNativeAsmPrinter, and LLVMInitializeNativeDisassembler.

Tracking Down the Linker Issues

We can use the DUMPBIN tool to inspect the LLVM libraries.

cd c:\dev\repos\llvm-project\build_llvm\install_local\lib
dumpbin LLVMX86Disassembler.lib
dumpbin /symbols /out:LLVMX86Disassembler.txt LLVMX86Disassembler.lib

The forfiles command is useful for dumping the symbols from all the libraries (forfiles was suggested at How to do something to each file in a directory with a batch script). I thought forfiles would work without the “cmd /c” prefix but that only resulted in dumpbin /summary output!

cd c:\dev\repos\llvm-project\build_llvm\install_local\lib
forfiles /m *.lib /c "cmd /c dumpbin /symbols /out:@fname.txt @file"

Now we can easily search for the symbols of interest, e.g.

> findstr /sipnc:"LLVMInitializeX86Disassembler" *.txt
LLVMX86Disassembler.txt:151:090 00000000 SECT2C notype ()    External     | LLVMInitializeX86Disassembler
LLVMX86Disassembler.txt:926:397 00000000 SECT6B notype       Static       | $unwind$LLVMInitializeX86Disassembler
LLVMX86Disassembler.txt:929:39A 00000000 SECT6C notype       Static       | $pdata$LLVMInitializeX86Disassembler

So there really is no such symbol in this lib folder! I’m guessing I need to add another lib folder to the path. A quick search for LLVMInitializeX86Disassembler leads to this post on Using the LLVM MC Disassembly API. It mentions using llvm-config to set the linker flags. Shouldn’t running the bash configure command take care of this? Let’s see what’s in the configure output:

...
checking what hsdis backend to use... 'llvm'
checking for LLVM_CONFIG... C:/dev/repos/llvm-project/build_llvm/install_local/bin [user supplied]
/cygdrive/c/dev/repos/java/forks/jdk/build/.configure-support/generated-configure.sh: line 135451: C:/dev/repos/llvm-project/build_llvm/install_local/bin: Is a directory
/cygdrive/c/dev/repos/java/forks/jdk/build/.configure-support/generated-configure.sh: line 135452: C:/dev/repos/llvm-project/build_llvm/install_local/bin: Is a directory
/cygdrive/c/dev/repos/java/forks/jdk/build/.configure-support/generated-configure.sh: line 135453: C:/dev/repos/llvm-project/build_llvm/install_local/bin: Is a directory
...

Well, that could be the problem! I think I need to fix the llvm-config path in Cygwin by appending /llvm-config to LLVM_CONFIG.

bash configure --with-hsdis=llvm LLVM_CONFIG=C:/dev/repos/llvm-project/build_llvm/install_local/bin/llvm-config --with-llvm=C:/dev/repos/llvm-project/build_llvm/install_local/

Sure enough, that was the problem! The bash configure output (below) now looks good and make build-hsdis now works. The fix for this would be to ensure bash configure fails if LLVM_CONFIG is set to the directory instead of the executable!

checking what hsdis backend to use... 'llvm'
checking for LLVM_CONFIG... C:/dev/repos/llvm-project/build_llvm/install_local/bin/llvm-config [user supplied]
checking for number of cores... 8
...

$ make build-hsdis
Building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Creating support/hsdis/hsdis.dll from 1 file(s)
Finished building target 'build-hsdis' in configuration 'windows-x86_64-server-release'

Notice from the new build command line in build\windows-x86_64-server-release\support\hsdis\BUILD_HSDIS_link.cmdline that there are now many .lib files supplied to the linker! These are the lib files that I was inspecting with dumpbin so my earlier hypothesis was wrong (there were no additional .lib files required, the ones I was looking at were simply not being passed to the linker).

/cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/fixpath exec
 /cygdrive/c/progra~2/micros~3/2019/enterp~1/vc/tools/msvc/1429~1.301/bin/hostx86/x64/link.exe
 -nologo
 -libpath:/cygdrive/c/dev/repos/llvm-project/build_llvm/install_local//lib
 -dll
 -debug
 "-pdb:/cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.pdb"
 "-map:/cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.map"
 "-implib:/cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.lib"
 -libpath:/cygdrive/c/progra~2/micros~3/2019/enterp~1/vc/tools/msvc/1429~1.301/atlmfc/lib/x64
 -libpath:/cygdrive/c/progra~2/micros~3/2019/enterp~1/vc/tools/msvc/1429~1.301/lib/x64
 -libpath:/cygdrive/c/progra~2/wi3cf2~1/netfxsdk/4.8/lib/um/x64
 -libpath:/cygdrive/c/progra~2/wi3cf2~1/10/lib/100190~1.0/ucrt/x64
 -libpath:/cygdrive/c/progra~2/wi3cf2~1/10/lib/100190~1.0/um/x64
 -out:/cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.dll 
 /cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis-llvm.obj
 /cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.dll.res
 llvmx86targetmca.lib llvmmca.lib llvmx86disassembler.lib llvmx86asmparser.lib llvmx86codegen.lib llvmcfguard.lib llvmglobalisel.lib llvmx86desc.lib llvmx86info.lib llvmmcdisassembler.lib llvmselectiondag.lib llvminstrumentation.lib llvmasmprinter.lib llvmdebuginfomsf.lib llvmcodegen.lib llvmtarget.lib llvmscalaropts.lib llvminstcombine.lib llvmaggressiveinstcombine.lib llvmtransformutils.lib llvmbitwriter.lib llvmanalysis.lib llvmprofiledata.lib llvmdebuginfodwarf.lib llvmobject.lib llvmtextapi.lib llvmmcparser.lib llvmmc.lib llvmdebuginfocodeview.lib llvmbitreader.lib llvmcore.lib llvmremarks.lib llvmbitstreamreader.lib llvmbinaryformat.lib llvmsupport.lib llvmdemangle.lib

Now running make install-hsdis copies hsdis-amd64.dll into /build/windows-x86_64-server-release/jdk/bin. The LLVM hsdis backend can now be used to disassemble instructions:

$ ./java -XX:CompileCommand="print java.lang.String::checkIndex" -version
CompileCommand: print java/lang/String.checkIndex bool print = true

============================= C2-compiled nmethod ==============================
----------------------------------- Assembly -----------------------------------

Compiled method (c2)    5912   60       4       java.lang.String::checkIndex (10 bytes)
 total in heap  [0x00000162f39e3090,0x00000162f39e3308] = 632
 relocation     [0x00000162f39e31e8,0x00000162f39e3200] = 24
 main code      [0x00000162f39e3200,0x00000162f39e3280] = 128
 stub code      [0x00000162f39e3280,0x00000162f39e3298] = 24
 oops           [0x00000162f39e3298,0x00000162f39e32a0] = 8
 metadata       [0x00000162f39e32a0,0x00000162f39e32a8] = 8
 scopes data    [0x00000162f39e32a8,0x00000162f39e32c0] = 24
 scopes pcs     [0x00000162f39e32c0,0x00000162f39e3300] = 64
 dependencies   [0x00000162f39e3300,0x00000162f39e3308] = 8

[Disassembly]
--------------------------------------------------------------------------------
[Constant Pool (empty)]

--------------------------------------------------------------------------------

[Verified Entry Point]
  # {method} {0x000001628800f2f0} 'checkIndex' '(II)V' in 'java/lang/String'
  # parm0:    rdx       = int
  # parm1:    r8        = int
  #           [sp+0x30]  (sp of caller)
  0x00000162f39e3200:           movl    %eax, -0x7000(%rsp)
  0x00000162f39e3207:           pushq   %rbp
  0x00000162f39e3208:           subq    $0x20, %rsp
  0x00000162f39e320c:           testl   %r8d, %r8d
  0x00000162f39e320f:           jl      0x2f
  0x00000162f39e3211:           cmpl    %r8d, %edx
  0x00000162f39e3214:           jae     0x16
  0x00000162f39e3216:           vzeroupper
  0x00000162f39e3219:           addq    $0x20, %rsp
  0x00000162f39e321d:           popq    %rbp
  0x00000162f39e321e:           cmpq    0x338(%r15), %rsp   ;   {poll_return}
  0x00000162f39e3225:           ja      0x29
  0x00000162f39e322b:           retq
  0x00000162f39e322c:           movl    %edx, %ebp
  0x00000162f39e322e:           movl    %r8d, (%rsp)
  0x00000162f39e3232:           movl    $0xffffffe4, %edx
  0x00000162f39e3237:           nop
  0x00000162f39e3238:           vzeroupper
  0x00000162f39e323b:           callq   -0x7a80f40          ; ImmutableOopMap {}
                                                            ;*invokestatic checkIndex {reexecute=0 rethrow=0 return_oop=0}
                                                            ; - java.lang.String::checkIndex@5 (line 4554)
                                                            ;   {runtime_call UncommonTrapBlob}
  0x00000162f39e3240:           movl    %edx, %ebp
  0x00000162f39e3242:           movl    %r8d, (%rsp)
  0x00000162f39e3246:           movl    $0xffffffcc, %edx
...

References

Here are some of the bugs/questions I looked at when investigating these failures. Stack overflow taught me about dumpbin and C++ decorated names/ the undname tool.

Article info

Categories

Tags

Leave a Reply

Your email address will not be published. Required fields are marked *