—Categories: Compilers, LLVM

Building LLVM for Windows ARM64

I was trying to test using LLVM as a backend for hsdis on the Windows ARM64 platform as implemented in PR 5920. I downloaded LLVM 13 and tried to use it in the build. Unfortunately, it didn’t have all the prerequisite include files and so building your own LLVM installation was the approach suggested for Windows. This post explicitly outlines the instructions needed to build LLVM for the Windows ARM64 platform on a Windows x64 host machine.

The first requirement is an LLVM build with native llvm-nm.exe and llvm-tblgen.exe binaries. These can be downloaded (I think) or generated by building LLVM for the native x64 platform as specified in the instructions from Jorn.

git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build_llvm
cd build_llvm
cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=X86" -D"CMAKE_BUILD_TYPE:STRING=Release" -D"CMAKE_INSTALL_PREFIX=install_local" -A x64 -T host=x64
cmake --build . --config Release --target install

Once that build successfully completes, we can then build LLVM for the Windows ARM64 platform with the commands below. Notice that we specify paths to the native llvm-nm and llvm-tblgen binaries to prevent the build from trying to use their ARM64 equivalents (which won’t run on the host).

cd llvm-project
mkdir build_llvm_AArch64
cd build_llvm_Aarch64

cmake ../llvm -DLLVM_TARGETS_TO_BUILD:STRING=AArch64 \
 -DCMAKE_BUILD_TYPE:STRING=Release \
 -DCMAKE_INSTALL_PREFIX=install_local \
 -DCMAKE_CROSSCOMPILING=True \
 -DLLVM_TARGET_ARCH=AArch64 \
 -DLLVM_NM=C:/repos/llvm-project/build_llvm/install_local/bin/llvm-nm.exe \
 -DLLVM_TABLEGEN=C:/repos/llvm-project/build_llvm/install_local/bin/llvm-tblgen.exe \
 -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-win32-msvc \
 -A ARM64 \
 -T host=x64

date; time \
 cmake --build . --config Release --target install ; \
 date

Once the build completes, the LLVM ARM64 files will be in the build_llvm_AArch64/install_local folder in the llvm-project repo. That build should have all the necessary header files and static libraries required for LLVM projects targeting Windows on ARM64. See the general cmake options and the LLVM-specific cmake options for details on the various flags and variables.

Behind the Scenes: Cross-Compiling LLVM

I naively started out by adding AArch64 to the list of LLVM_TARGETS_TO_BUILD, then using only AArch64 in the list. Trying to use the generated build would still fail with errors about mismatched platforms so I knew some cross compilation specific flags would be needed. How do I cross-compile LLVM/Clang for AArch64 on x64 host? – Stack Overflow and How To Cross-Compile Clang/LLVM using Clang/LLVM — LLVM 15.0.0git documentation were handy references. They didn’t have anything windows specific but got me walking down the right path (e.g. the importance of the native LLVM_TABLEGEN). I tried something along these lines:

cd llvm-project
mkdir build_llvm_AArch64
cd build_llvm_AArch64

cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=AArch64" \
 -D"CMAKE_BUILD_TYPE:STRING=Release" \
 -D"CMAKE_INSTALL_PREFIX=install_local_AArch64" \
 -D"CMAKE_CROSSCOMPILING=True" \
 -D"LLVM_TARGET_ARCH=AArch64" \
 -A x64 \
 -T host=x64

cmake --build . --config Release --target install

This still results in errors about conflicting machine types:

c:\...\llvm-project\build_llvm_aarch64\install_local_aarch64\\lib\llvmaarch64disassembler.lib : warning LNK4272: library machine type 'x64' conflicts with target machine type 'ARM64'

That’s when I tried adding the LLVM_TABLE_GEN from a Windows x64 LLVM build I had generated earlier. I accidentally omitted the options prefixed with a # below because I didn’t include the trailing slash after adding the llvm-tblgen.exe option.

cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=AArch64" \
 -D"CMAKE_BUILD_TYPE:STRING=Release" \
 -D"CMAKE_INSTALL_PREFIX=install_local_AArch64_2" \
 -D"CMAKE_CROSSCOMPILING=True" \
 -D"LLVM_TARGET_ARCH=AArch64" \
 -D"LLVM_TABLEGEN=C:\dev\repos\llvm-project\build_llvm\install_local\bin\llvm-tblgen.exe" \
 #-D"LLVM_DEFAULT_TARGET_TRIPLE=aarch64-win32-msvc" \
 #-A x64 \
 #-T host=x64

date; time \
 cmake --build . --config Release --target install; \
 date

The build still succeeded and generated AArch64 .lib files in the LLVM installation! Interestingly, they still had the x64 machine type in the header.

$ dumpbin /headers build_llvm_AArch64_2\install_local_AArch64_2\lib\LLVMAArch64AsmParser.lib
Microsoft (R) COFF/PE Dumper Version 14.29.30133.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file build_llvm_AArch64_2\install_local_AArch64_2\lib\LLVMAArch64AsmParser.lib

File Type: LIBRARY

FILE HEADER VALUES
            8664 machine (x64)
...

I had no choice but to reexamine my understanding of what the -A flag does. It is used to specify the platform name but it’s only after digging into the CMAKE_GENERATOR_PLATFORM docs that I noticed that this was the target platform! This also made me realize that I hadn’t noticed that the x64 C++ compiler was being used all along!

-- The C compiler identification is MSVC 19.29.30133.0
-- The CXX compiler identification is MSVC 19.29.30133.0
-- The ASM compiler identification is MSVC
-- Found assembler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - works

Some references to LLVM triples led back to the clang cross-compilation docs and the llvm::Triple source code so I tried again with the triple set and with -A now set to AArch64.

cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=AArch64" \
 -D"CMAKE_BUILD_TYPE:STRING=Release" \
 -D"CMAKE_INSTALL_PREFIX=install_local_AArch64_3" \
 -D"CMAKE_CROSSCOMPILING=True" \
 -D"LLVM_TARGET_ARCH=AArch64" \
 -D"LLVM_TABLEGEN=C:\dev\repos\llvm-project\build_llvm\install_local\bin\llvm-tblgen.exe" \
 -D"LLVM_DEFAULT_TARGET_TRIPLE=aarch64-win32-msvc" \
 -A AArch64 \
 -T host=x64

Setting -A to AArch64 causes MSBuild to fail with an error about an unknown platform. So -A just might be the argument I need to get ARM64 libraries built.

"C:\dev\repos\llvm-project\build_llvm_AArch64_3\CMakeFiles\3.17.3\VCTargetsPath.vcxproj" (default target) (1) ->
    (_CheckForInvalidConfigurationAndPlatform target) ->
      C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(820,5): error : The BaseOutputPath/OutputPath property is not set for project 'VCTargetsPath.vcxproj'.  Please check to make sure that you have specified a valid combination of Configuration and Platform for this project.  Configuration='Debug'  Platform='AArch64'.  You may be seeing this message because you are trying to build a project without a solution file, and have specified a non-default Configuration or Platform that doesn't exist for this project. [C:\dev\repos\llvm-project\build_llvm_AArch64_3\CMakeFiles\3.17.3\VCTargetsPath.vcxproj]

So I tried using -A ARM64 instead. I noticed that we now have the ARM64 C++ compiler selected! This is something I should have been paying attention to from the beginning, crucial for cross-compilation.

-- The C compiler identification is MSVC 19.29.30133.0
-- The CXX compiler identification is MSVC 19.29.30133.0
-- The ASM compiler identification is MSVC
-- Found assembler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe - works

Unfortunately, the build still failed with an error from gen-msvc-exports.py. Taking a look at gen-msvc-exports.py, it looks like it is trying to run llvm-nm.exe (for the target platform).

  Generating export list for LLVM-C
  Traceback (most recent call last):
    File "C:/dev/repos/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 116, in <module>
      main()
    File "C:/dev/repos/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 112, in main
      gen_llvm_c_export(ns.output, ns.underscore, libs, ns.nm)
    File "C:/dev/repos/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 72, in gen_llvm_c_export
      check_call([nm, '-g', lib], stdout=dumpout_f)
    File "C:\dev\tools\Anaconda3\lib\subprocess.py", line 359, in check_call
      retcode = call(*popenargs, **kwargs)
    File "C:\dev\tools\Anaconda3\lib\subprocess.py", line 340, in call
      with Popen(*popenargs, **kwargs) as p:
    File "C:\dev\tools\Anaconda3\lib\subprocess.py", line 854, in __init__
      self._execute_child(args, executable, preexec_fn, close_fds,
    File "C:\dev\tools\Anaconda3\lib\subprocess.py", line 1307, in _execute_child
      hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
  OSError: [WinError 216] This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(241,5): error MSB8066: Custom build for 'C:\dev\repos\llvm-project\build_llvm_AArch64_3\CMakeFi
les\02a88fa656bb9cf8b9ffd0e0debe57ae\libllvm-c.exports.rule;C:\dev\repos\llvm-project\build_llvm_AArch64_3\CMakeFiles\8ebc0efbf04134b25d0f37561fba0d55\LLVM-C.def.rule;C:\dev\repos\llvm-project\build_llvm_AArch64_
3\CMakeFiles\509fcb3f8bb132e9c560e15e8d25cb45\LLVM-C_exports.rule;C:\dev\repos\llvm-project\llvm\tools\llvm-shlib\CMakeLists.txt' exited with code 1. [C:\dev\repos\llvm-project\build_llvm_AArch64_3\tools\llvm-shl
ib\LLVM-C_exports.vcxproj]

A quick search for the general message (Generating export list for LLVM-C) reveals that it is from llvm-shlib/CMakeLists.txt. Looks like we just need to set LLVM_NM as per llvm-shlib/CMakeLists.txt.

date; time cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=AArch64" \
 -D"CMAKE_BUILD_TYPE:STRING=Release" \
 -D"CMAKE_INSTALL_PREFIX=install_local" \
 -D"CMAKE_CROSSCOMPILING=True" \
 -D"LLVM_TARGET_ARCH=AArch64" \
 -D"LLVM_NM=C:\dev\repos\llvm-project\build_llvm\install_local\bin\llvm-nm.exe" \
 -D"LLVM_TABLEGEN=C:\dev\repos\llvm-project\build_llvm\install_local\bin\llvm-tblgen.exe" \
 -D"LLVM_DEFAULT_TARGET_TRIPLE=aarch64-win32-msvc" \
 -A ARM64 \
 -T host=x64

date; time \
 cmake --build . --config Release --target install; \
 date

These build commands work! Dumpbin shows that the generated .lib files have ARM64 headers!

$ dumpbin /headers C:\dev\repos\llvm-project\build_llvm_AArch64_3\Release\lib\LLVMAArch64Disassembler.lib
Microsoft (R) COFF/PE Dumper Version 14.29.30133.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file C:\dev\repos\llvm-project\build_llvm_AArch64_3\Release\lib\LLVMAArch64Disassembler.lib

File Type: LIBRARY

FILE HEADER VALUES
            AA64 machine (ARM64)

Unfortunately, the JDK project that got me started down this path still doesn’t build. Cygwin shows defines like -DLLVM_DEFAULT_TRIPLET='”aarch64-pc-windows-msvc”‘ being passed to the compiler, which then complains:

C:/.../src/utils/hsdis/llvm/hsdis-llvm.cpp(217): error C2015: too many characters in constant

The quotes in the commands therefore needed to be dropped. This caused build failures since the paths used back-slashes!

Building Opts.inc...
  'C:devreposllvm-projectbuild_llvminstall_localbinllvm-tblgen.exe' is not recognized as an internal or external command,
  operable program or batch file

This is now the part where I find a nice document on the LLVM site with the 3-liner for this task 😀

Article info

Categories

Tags

Leave a Reply

Your email address will not be published. Required fields are marked *