Categories: Compilers, LLVM, Visual C++

Trying to Build LLVM for Windows using the Release Script

The post on Tracking Down Missing Headers in LLVM for Windows identified which headers were missing from the LLVM for Windows builds. Let us fix this. Clone the LLVM repo and install the prerequisites as explained at Getting Started with the LLVM System using Microsoft Visual Studio — LLVM 18.0.0git documentation.

The build_llvm_release.bat script in the LLVM repo is used to create a release build for Windows. Running it without any arguments displays a sample command line to use to build LLVM.

cd \repos && mkdir llvm
cd \repos\llvm\llvm-project\llvm\utils\release

C:\> cd \repos\llvm\llvm-project\llvm\utils\release
C:\repos\llvm\llvm-project\llvm\utils\release> build_llvm_release.bat
--version option is required
=============================
Script for building the LLVM installer on Windows,
used for the releases at https://github.com/llvm/llvm-project/releases

Usage: build_llvm_release.bat --version <version> [--x86,--x64, --arm64]

Options:
--version: [required] version to build
--help: display this help
--x86: build and test x86 variant
--x64: build and test x64 variant
--arm64: build and test arm64 variant

Note: At least one variant to build is required.

Example: build_llvm_release.bat --version 15.0.0 --x86 --x64

Let us build version 17.0.1, which is the latest LLVM release. Open an administrator Developer Command Prompt then run:

:: set PATH=%PATH%;C:\Program Files\7-Zip
build_llvm_release.bat --version 17.0.1 --x64

The first error is caused by mv not being a standard command in the Windows command prompt. There are ways to create aliases, e.g. alias – Aliases in Windows command prompt – Stack Overflow but given that this is a batch file, why not just use the built-in function? I fix this in the script.

C:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_17.0.1> mv llvm-project-* llvm-project   || exit /b 1
'mv' is not recognized as an internal or external command,
operable program or batch file.

The build directory needs to be deleted before restarting the build with the fix. Otherwise, the script will fail.

rmdir /s /q llvm_package_17.0.1

Installing Python

The next things that fails on my machine is setting the python environment. I should have started with Getting Started with the LLVM System using Microsoft Visual Studio — LLVM 18.0.0git documentation. I head to Welcome to Python.org and download Python 3.11.5. The installer says I already have it:

Python 3.11.5 (64-bit) Setup

This command confirms that it is installed.

C:\repos\llvm\llvm-project\llvm\utils\release> \Python311\python.exe --version
Python 3.11.0

However, from a new command window, it looks like it hasn’t been set up.

C:\repos> python --version
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.

Just running python loads the Microsoft Store:

I proceed with the Customize Installation option in the open Python installer, this time selecting “for all users (requires admin privileges)”.

I’m not sure why the “Install Python 3.11 for all users” option is not checked on the “Advanced Options” page.

Setup completes successfully. The What’s New In Python 3.11 — Python 3.11.5 documentation page is interesting.

Opening a new command prompt then running python still gives the same behavior as before. where python shows that the new installation appears 2nd.

C:\repos> where python
C:\Users\saint\AppData\Local\Microsoft\WindowsApps\python.exe
C:\Python311\python.exe

The PATH environment variable has these paths in reverse order so I don’t understand what is happening. Looks like the easiest way forward is to just install python from the Microsoft Store – this at least ensures that python scripts can run. After a failure due to the python command not being found, closer inspection of build_llvm_release.bat reveals that it needs the PYTHON_HOME environment variable to be set. However, that is set on line 358 using the python_dir variable, which is in turn the first argument to :set_environment. The call to set_environment is from :do_build_32 (or do_build_64), which uses a hard-coded python path! The script should first validate the python installation and exit early if it cannot be found! I should have reported/fixed these when I worked on Tracking Down Missing Headers in LLVM for Windows – Saint’s Log (swesonga.org) last year.

Build Errors

With the Python path fixed, the build proceeds but the C++ compilation fails. A few months ago I got the error below build version 15.0.7. Unfortunately, I didn’t explicitly note the compiler version.

[5834/6907] Building CXX object tools\lldb\source\Plugins\InstrumentationRuntime\MainThreadChecker\CMakeFiles\lldbPluginInstrumentationRuntimeMainThreadChecker.dir\InstrumentationRuntimeMainThreadChecker.cpp.obj
FAILED: tools/lldb/source/Plugins/InstrumentationRuntime/MainThreadChecker/CMakeFiles/lldbPluginInstrumentationRuntimeMainThreadChecker.dir/InstrumentationRuntimeMainThreadChecker.cpp.obj
C:\PROGRA~1\MICROS~3\2022\COMMUN~1\VC\Tools\MSVC\1437~1.328\bin\Hostx64\x64\cl.exe 
 /nologo
 /TP
 -DGTEST_HAS_RTTI=0
 -DUNICODE
 -D_CRT_NONSTDC_NO_DEPRECATE
 -D_CRT_NONSTDC_NO_WARNINGS
 -D_CRT_SECURE_NO_DEPRECATE
 -D_CRT_SECURE_NO_WARNINGS
 -D_ENABLE_EXTENDED_ALIGNED_STORAGE
 -D_HAS_EXCEPTIONS=0
 -D_SCL_SECURE_NO_DEPRECATE
 -D_SCL_SECURE_NO_WARNINGS
 -D_UNICODE
 -D__STDC_CONSTANT_MACROS
 -D__STDC_FORMAT_MACROS
 -D__STDC_LIMIT_MACROS
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\build64_stage0\tools\lldb\source\Plugins\InstrumentationRuntime\MainThreadChecker
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\llvm-project\lldb\source\Plugins\InstrumentationRuntime\MainThreadChecker
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\llvm-project\lldb\include
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\build64_stage0\tools\lldb\include
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\build64_stage0\include
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\llvm-project\llvm\include
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\llvm-project\llvm\..\clang\include
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\build64_stage0\tools\lldb\..\clang\include
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\llvm-project\lldb\source
 -IC:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\build64_stage0\tools\lldb\source
 -DLIBXML_STATIC
 /Zc:inline
 /Zc:__cplusplus
 /Oi
 /bigobj
 /permissive-
 /W4
 -wd4141
 -wd4146
 -wd4244
 -wd4267
 -wd4291
 -wd4351
 -wd4456
 -wd4457
 -wd4458
 -wd4459
 -wd4503
 -wd4624
 -wd4722
 -wd4100
 -wd4127
 -wd4512
 -wd4505
 -wd4610
 -wd4510
 -wd4702
 -wd4245
 -wd4706
 -wd4310
 -wd4701
 -wd4703
 -wd4389
 -wd4611
 -wd4805
 -wd4204
 -wd4577
 -wd4091
 -wd4592
 -wd4319
 -wd4709
 -wd4324
 -w14062
 -we4238
 /Gw
 /MD
 /O2
 /Ob2
 /DNDEBUG  
 -wd4018
 -wd4068
 -wd4150
 -wd4201
 -wd4251
 -wd4521
 -wd4530 
 /EHs-c-
 /GR-
 -std:c++14
 /showIncludes
 /Fotools\lldb\source\Plugins\InstrumentationRuntime\MainThreadChecker\CMakeFiles\lldbPluginInstrumentationRuntimeMainThreadChecker.dir\InstrumentationRuntimeMainThreadChecker.cpp.obj
 /Fdtools\lldb\source\Plugins\InstrumentationRuntime\MainThreadChecker\CMakeFiles\lldbPluginInstrumentationRuntimeMainThreadChecker.dir\lldbPluginInstrumentationRuntimeMainThreadChecker.pdb
 /FS
 -c C:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\llvm-project\lldb\source\Plugins\InstrumentationRuntime\MainThreadChecker\InstrumentationRuntimeMainThreadChecker.cpp
C:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_15.0.7\llvm-project\lldb\source\Plugins\InstrumentationRuntime\MainThreadChecker\InstrumentationRuntimeMainThreadChecker.cpp : fatal error C1083: Cannot open compiler generated file: '': Invalid argument

Today, I got a similar error in InstrumentationRuntimeMainThreadChecker.cpp AND and this internal compiler error using Microsoft (R) C/C++ Optimizing Compiler Version 19.38.33133 for x64!

FAILED: tools/clang/lib/Analysis/FlowSensitive/CMakeFiles/obj.clangAnalysisFlowSensitive.dir/HTMLLogger.cpp.obj
C:\PROGRA~1\MICROS~3\2022\COMMUN~1\VC\Tools\MSVC\1438~1.331\bin\Hostx64\x64\cl.exe  /nologo /TP ... -c C:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_17.0.1\llvm-project\clang\lib\Analysis\FlowSensitive\HTMLLogger.cpp
C:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_17.0.1\llvm-project\clang\lib\Analysis\FlowSensitive\HTMLLogger.cpp(332): fatal error C1001: Internal compiler error.
(compiler file 'msc1.cpp', line 1587)
 To work around this problem, try simplifying or changing the program near the locations listed above.
If possible please provide a repro here: https://developercommunity.visualstudio.com
Please choose the Technical Support command on the Visual C++
 Help menu, or open the Technical Support help file for more information

Additional troubleshooting is obviously required to make progress here. I might try to create a narrowed down testcase for the internal compiler error or perhaps just report it as is and let the Visual C++ team figure that out.


Categories: Compilers, Fortran, LLVM

Building Flang with Visual C++ [2022/10]

Note: I wrote this in October 2022 but never got around to posting it. Doing so now to enable comparisons with the latest sources and compilers.

In Failing to Build Flang with Visual C++, the C++ compiler had bugs that prevented compilation from succeeding. I just tried a new version of the Visual C++ compiler (Microsoft (R) C/C++ Optimizing Compiler Version 19.34.31823.3 for x64). This failed to build the same commit with the following error.

D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-implementation.h(1585): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-implementation.h(1585): warning C4189: 'ctx': local variable is initialized but not referenced
...

This line was last modified by [flang][msvc] Fix lambda capture ambiguity. NFC. · llvm/llvm-project@0b671a4 (github.com). Can the entire workaround be removed? Some of it was introduced by [flang][msvc] Rework a MSVC work-around to avoid clang warning · llvm/llvm-project@75a5ec1 (github.com). I get the same warning as error if I rename ctx to context. I undo the MSVC workaround and get this error now:

D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-implementation.h(1578): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-implementation.h(1578): warning C4101: 'buffer': unreferenced local variable
...

Moving that variable deeper into the if-statement addresses that issue. I end up having to suppress C4661 again.

D:\dev\repos\llvm-project\flang\include\flang\Evaluate\expression.h(101): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\include\flang\Evaluate\expression.h(101): warning C4661: 'std::optional<Fortran::evaluate::DynamicType> Fortran::evaluate::ExpressionBase<Fortran::evaluate::SomeDerived>::GetType(void) const': no suitable definition provided for explicit template instantiation request
...

These are the next failures I run into:

C:\PROGRA~1\MIB055~1\2022\Preview\VC\Tools\MSVC\1434~1.318\bin\Hostx64\x64\cl.exe  /nologo /TP -DFLANG_LITTLE_ENDIAN=1 -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\include -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\include -ID:\dev\repos\llvm-project\build-nowarn\include -ID:\dev\repos\llvm-project\llvm\include -external:ID:\dev\repos\llvm-project\llvm\..\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\clang\include -external:ID:\dev\repos\llvm-project\llvm\..\clang\include -external:W0 /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4661 -wd4324 -w14062 -we4238 /Gw /WX /MD /O2 /Ob2  /EHs-c- /GR- -UNDEBUG -std:c++17 /showIncludes /Fotools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\mod-file.cpp.obj /Fdtools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\ /FS -c D:\dev\repos\llvm-project\flang\lib\Semantics\mod-file.cpp
D:\dev\repos\llvm-project\flang\lib\Semantics\mod-file.cpp(962): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Semantics\mod-file.cpp(962): warning C4834: discarding return value of function with 'nodiscard' attribute

c++ – What’s the reason for not using C++17’s [[nodiscard]] almost everywhere in new code? – Software Engineering Stack Exchange has a link to the proposal of [[nodiscard]] and other attributes. This is easy to resolve by casting the result to (void) as per Compiler warning (Level 1) C4834 | Microsoft Docs.


[4578/5288] Building CXX object tools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\expression.cpp.obj

C:\PROGRA~1\MIB055~1\2022\Preview\VC\Tools\MSVC\1434~1.318\bin\Hostx64\x64\cl.exe  /nologo /TP -DFLANG_LITTLE_ENDIAN=1 -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\include -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\include -ID:\dev\repos\llvm-project\build-nowarn\include -ID:\dev\repos\llvm-project\llvm\include -external:ID:\dev\repos\llvm-project\llvm\..\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\clang\include -external:ID:\dev\repos\llvm-project\llvm\..\clang\include -external:W0 /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4661 -wd4324 -w14062 -we4238 /Gw /WX /MD /O2 /Ob2  /EHs-c- /GR- -UNDEBUG -std:c++17 /showIncludes /Fotools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\expression.cpp.obj /Fdtools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\ /FS -c D:\dev\repos\llvm-project\flang\lib\Semantics\expression.cpp
D:\dev\repos\llvm-project\flang\lib\Semantics\expression.cpp(1192): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Semantics\expression.cpp(1192): warning C4927: illegal conversion; more than one user-defined conversion has been implicitly applied
D:\dev\repos\llvm-project\flang\lib\Semantics\expression.cpp(1192): note: while calling the constructor 'Fortran::semantics::Symbol::Symbol(const Fortran::semantics::Symbol &) noexcept(false)'
D:\dev\repos\llvm-project\flang\include\flang\Semantics\symbol.h(734): note: compiler has generated 'Fortran::semantics::Symbol::Symbol' here

C:\PROGRA~1\MIB055~1\2022\Preview\VC\Tools\MSVC\1434~1.318\bin\Hostx64\x64\cl.exe  /nologo /TP -DFLANG_LITTLE_ENDIAN=1 -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\include -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\include -ID:\dev\repos\llvm-project\build-nowarn\include -ID:\dev\repos\llvm-project\llvm\include -external:ID:\dev\repos\llvm-project\llvm\..\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\clang\include -external:ID:\dev\repos\llvm-project\llvm\..\clang\include -external:W0 /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4661 -wd4324 -w14062 -we4238 /Gw /WX /MD /O2 /Ob2  /EHs-c- /GR- -UNDEBUG -std:c++17 /showIncludes /Fotools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\resolve-names.cpp.obj /Fdtools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\ /FS -c D:\dev\repos\llvm-project\flang\lib\Semantics\resolve-names.cpp
D:\dev\repos\llvm-project\flang\lib\Semantics\resolve-names.cpp(3151): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Semantics\resolve-names.cpp(3151): warning C4927: illegal conversion; more than one user-defined conversion has been implicitly applied
D:\dev\repos\llvm-project\flang\lib\Semantics\resolve-names.cpp(3151): note: while calling the constructor 'Fortran::semantics::Symbol::Symbol(const Fortran::semantics::Symbol &) noexcept(false)'
D:\dev\repos\llvm-project\flang\include\flang\Semantics\symbol.h(734): note: compiler has generated 'Fortran::semantics::Symbol::Symbol' here

This same warning appears in more files:

C:\PROGRA~1\MIB055~1\2022\Preview\VC\Tools\MSVC\1434~1.318\bin\Hostx64\x64\cl.exe  /nologo /TP -DFLANG_LITTLE_ENDIAN=1 -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\include -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\include -ID:\dev\repos\llvm-project\build-nowarn\include -ID:\dev\repos\llvm-project\llvm\include -external:ID:\dev\repos\llvm-project\llvm\..\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\clang\include -external:ID:\dev\repos\llvm-project\llvm\..\clang\include -external:W0 /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4661 -wd4324 -w14062 -we4238 /Gw /WX /MD /O2 /Ob2  /EHs-c- /GR- -UNDEBUG -std:c++17 /showIncludes /Fotools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\semantics.cpp.obj /Fdtools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\ /FS -c D:\dev\repos\llvm-project\flang\lib\Semantics\semantics.cpp
D:\dev\repos\llvm-project\flang\lib\Semantics\semantics.cpp(511): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Semantics\semantics.cpp(511): warning C4927: illegal conversion; more than one user-defined conversion has been implicitly applied
D:\dev\repos\llvm-project\flang\lib\Semantics\semantics.cpp(511): note: while calling the constructor 'Fortran::semantics::Symbol::Symbol(const Fortran::semantics::Symbol &) noexcept(false)'
D:\dev\repos\llvm-project\flang\include\flang\Semantics\symbol.h(734): note: compiler has generated 'Fortran::semantics::Symbol::Symbol' here

This last error is different because it comes from a generated source file.

C:\PROGRA~1\MIB055~1\2022\Preview\VC\Tools\MSVC\1434~1.318\bin\Hostx64\x64\cl.exe  /nologo /TP -DFLANG_LITTLE_ENDIAN=1 -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\lib\Optimizer\Dialect -ID:\dev\repos\llvm-project\flang\lib\Optimizer\Dialect -ID:\dev\repos\llvm-project\flang\include -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\include -ID:\dev\repos\llvm-project\build-nowarn\include -ID:\dev\repos\llvm-project\llvm\include -external:ID:\dev\repos\llvm-project\llvm\..\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\clang\include -external:ID:\dev\repos\llvm-project\llvm\..\clang\include -external:W0 /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4661 -wd4324 -w14062 -we4238 /Gw /WX /MD /O2 /Ob2  /EHs-c- /GR- -UNDEBUG -std:c++17 /showIncludes /Fotools\flang\lib\Optimizer\Dialect\CMakeFiles\obj.FIRDialect.dir\FIROps.cpp.obj /Fdtools\flang\lib\Optimizer\Dialect\CMakeFiles\obj.FIRDialect.dir\ /FS -c D:\dev\repos\llvm-project\flang\lib\Optimizer\Dialect\FIROps.cpp
D:\dev\repos\llvm-project\build-nowarn\tools\flang\include\flang/Optimizer/Dialect/CanonicalizationPatterns.inc(386): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\build-nowarn\tools\flang\include\flang/Optimizer/Dialect/CanonicalizationPatterns.inc(386): warning C4927: illegal conversion; more than one user-defined conversion has been implicitly applied

Next error is because resolve-names.cpp has a function that doesn’t return a value in some path. Did this compile before?

C:\PROGRA~1\MIB055~1\2022\Preview\VC\Tools\MSVC\1434~1.318\bin\Hostx64\x64\cl.exe  /nologo /TP -DFLANG_LITTLE_ENDIAN=1 -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\lib\Semantics -ID:\dev\repos\llvm-project\flang\include -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\include -ID:\dev\repos\llvm-project\build-nowarn\include -ID:\dev\repos\llvm-project\llvm\include -external:ID:\dev\repos\llvm-project\llvm\..\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\clang\include -external:ID:\dev\repos\llvm-project\llvm\..\clang\include -external:W0 /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4661 -wd4927 -wd4324 -w14062 -we4238 /Gw /WX /MD /O2 /Ob2  /EHs-c- /GR- -UNDEBUG -std:c++17 /showIncludes /Fotools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\resolve-names.cpp.obj /Fdtools\flang\lib\Semantics\CMakeFiles\obj.FortranSemantics.dir\ /FS -c D:\dev\repos\llvm-project\flang\lib\Semantics\resolve-names.cpp
D:\dev\repos\llvm-project\flang\lib\Semantics\resolve-names.cpp(2796) : error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Semantics\resolve-names.cpp(2796) : warning C4715: '`Fortran::semantics::ModuleVisitor::DoAddUse'::`2'::<lambda_1>::operator()': not all control paths return a value

Oh boy, them errors keep coming:

C:\PROGRA~1\MIB055~1\2022\Preview\VC\Tools\MSVC\1434~1.318\bin\Hostx64\x64\cl.exe  /nologo /TP -DFLANG_LITTLE_ENDIAN=1 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\unittests\Evaluate -ID:\dev\repos\llvm-project\flang\unittests\Evaluate -ID:\dev\repos\llvm-project\flang\include -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\include -ID:\dev\repos\llvm-project\build-nowarn\include -ID:\dev\repos\llvm-project\llvm\include -external:ID:\dev\repos\llvm-project\llvm\..\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\clang\include -external:ID:\dev\repos\llvm-project\llvm\..\clang\include -external:W0 /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4661 -wd4927 -wd4324 -w14062 -we4238 /Gw /WX /MD /O2 /Ob2 -UNDEBUG -std:c++17 /showIncludes /Fotools\flang\unittests\Evaluate\CMakeFiles\FortranEvaluateTesting.dir\fp-testing.cpp.obj /Fdtools\flang\unittests\Evaluate\CMakeFiles\FortranEvaluateTesting.dir\FortranEvaluateTesting.pdb /FS -c D:\dev\repos\llvm-project\flang\unittests\Evaluate\fp-testing.cpp
C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31823\include\xtree(1664): error C2220: the following warning is treated as an error
C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31823\include\xtree(1626): note: see reference to function template instantiation 'std::_Tree_node<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>,std::_Default_allocator_traits<_Alloc>::void_pointer> *std::_Tree<std::_Tmap_traits<_Kty,_Ty,_Pr,_Alloc,false>>::_Copy_nodes<std::_Tree<std::_Tmap_traits<_Kty,_Ty,_Pr,_Alloc,false>>::_Strategy::_Copy>(std::_Tree_node<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>,std::_Default_allocator_traits<_Alloc>::void_pointer> *,std::_Tree_node<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>,std::_Default_allocator_traits<_Alloc>::void_pointer> *)' being compiled
        with
        [
            _Alloc=std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>,
            _Kty=Fortran::parser::CharBlock,
            _Ty=Fortran::common::ConstantSubscript,
            _Pr=std::less<Fortran::parser::CharBlock>
        ]
...
C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31823\include\map(109): note: while compiling class template member function 'std::map<Fortran::parser::CharBlock,Fortran::common::ConstantSubscript,std::less<Fortran::parser::CharBlock>,std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>>::map(const std::map<Fortran::parser::CharBlock,Fortran::common::ConstantSubscript,std::less<Fortran::parser::CharBlock>,std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>> &)'
D:\dev\repos\llvm-project\flang\include\flang/Evaluate/common.h(243): note: see reference to function template instantiation 'std::map<Fortran::parser::CharBlock,Fortran::common::ConstantSubscript,std::less<Fortran::parser::CharBlock>,std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>>::map(const std::map<Fortran::parser::CharBlock,Fortran::common::ConstantSubscript,std::less<Fortran::parser::CharBlock>,std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>> &)' being compiled
D:\dev\repos\llvm-project\flang\include\flang/Evaluate/common.h(291): note: see reference to class template instantiation 'std::map<Fortran::parser::CharBlock,Fortran::common::ConstantSubscript,std::less<Fortran::parser::CharBlock>,std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>>' being compiled
C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31823\include\xtree(1664): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc

I try setting this flag just at the end of the if(MSVC) block in HandleLLVMOptions.cmake. Unfortunately, the build still fails and that’s when I notice that the earlier command lines in this post have the /EHs-c- flag. Looks like HandleLLVMOptions.cmake is explicitly removing such flags. The next logical step is to search for /EH in the codebase to see how it’s getting set for the commands that have it. That’s when I notice clang-cl – looks like a cl.exe immitation. Maybe this is how they’ve been building flang for windows without addressing all these Visual C++ issues?

https://github.com/llvm/llvm-project/blob/c0702ac07b8e206f424930ff0331151954fb821c/libc/cmake/modules/LLVMLibCObjectRules.cmake#L25

There is talk of using clang-cl on non-Windows hosts to target MSVC in llvm-project/WinMsvc.cmake at 00874c48ea4d291908517afaab50d1dcbfb016c3 · llvm/llvm-project (github.com). Discovered this when investigating the packing issue in the LLVM missing headers post.

Back to the exception handling issue: searching for /EHsc leads me to AddLLVM.cmake which references the LLVM_ENABLE_EH variable. Looks like that defaults to OFF in HandleLLVMOptions.cmake and it also needs LLVM_ENABLE_RTTI to be enabled.

cmake -G Ninja ../llvm -DCMAKE_BUILD_TYPE=Release -DFLANG_ENABLE_WERROR=On -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_TARGETS_TO_BUILD=host -DCMAKE_INSTALL_PREFIX=../install -DLLVM_LIT_ARGS=-v -DLLVM_ENABLE_PROJECTS="clang;mlir;flang" -DLLVM_ENABLE_RUNTIMES="compiler-rt" -DLLVM_ENABLE_EH=ON -DLLVM_ENABLE_RTTI=ON

The fact that this still doesn’t address the warning is the red flag forcing me to notice that this is a unit test and we probably don’t want to turn on RTTI anyway. That would be a huge change! Let’s drop these flags and instead modify the flang/unittests/CMakeLists.txt to add the -EHsc flag. That turns out to be exactly the right fix! Building with this command line takes about 1hr 55min on my desktop.

mkdir build
cd build
cmake -G Ninja ../llvm -DCMAKE_BUILD_TYPE=Release -DFLANG_ENABLE_WERROR=On -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_TARGETS_TO_BUILD=host -DCMAKE_INSTALL_PREFIX=../install -DLLVM_LIT_ARGS=-v -DLLVM_ENABLE_PROJECTS="clang;mlir;flang" -DLLVM_ENABLE_RUNTIMES="compiler-rt" && ninja

At this point, the commit I’ve been using and the Visual Studio build (17.4.0 Preview 3 with C++ compiler version 19.34.31823.3) are outdated. Let’s see what happens with the latest code and Visual Studio 17.4.0 Preview 4 (with C++ compiler version 19.34.31931). Bad timing on my part though, the installer now says Preview 5.0 is available and it comes with Microsoft (R) C/C++ Optimizing Compiler Version 19.34.31932 for x86. Not sure why these new preview builds are showing the x86 host when launching the developer command prompt from the start menu. Doesn’t happen when using the drop down in Windows Terminal.

Outstanding Tasks

  1. Expand on std::vector /EHsc – Search (bing.com) and visual c++ – std::vector without exceptions: warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc – Stack Overflow

Categories: LLVM, Windows

Cmd.exe File System Frustration

When working on Tracking Down Missing Headers in LLVM for Windows, I kept running into these access denied failures when running the LLVM build script:

D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2> move llvm-project-* llvm-project   || exit /b 1
D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2\llvm-project-llvmorg-15.0.2
Access is denied.
        0 dir(s) moved.

Before retrying the script, I tried to clean up using rmdir since the script requires the directory to not exist.

 rmdir /s /q llvm_package_15.0.2 && build_llvm_release.bat 15.0.2

Strangely enough, rmdir failed with this error:

llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\try_lock_shared_for.pass.cpp - The system cannot find the path specified.
llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\try_lock_shared_until.pass.cpp - The system cannot find the path specified.
llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\try_lock_until_deadlock_bug.pass.cpp - The system cannot find the path specified.

These files still exist on disk though! They are displayed if you dir their containing directory but are not found if you dir their full paths! They cannot be deleted using del either. Interestingly, pressing tab after the directory path will autocomplete the file names.

C:\> dir D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\
 Volume in drive D is DATAVOL1
 Volume Serial Number is 8800-8693

 Directory of D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class

10/16/2022  01:22 PM    <DIR>          .
10/04/2022  03:29 AM    <DIR>          ..
10/04/2022  03:29 AM             2,461 try_lock_shared_for.pass.cpp
10/04/2022  03:29 AM             2,423 try_lock_shared_until.pass.cpp
10/04/2022  03:29 AM             2,146 try_lock_until_deadlock_bug.pass.cpp
               3 File(s)          7,030 bytes

C:\> dir D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\try_lock_shared_for.pass.cpp
 Volume in drive D is DATADRIVE1
 Volume Serial Number is 548C-FFC9

 Directory of D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class

File Not Found

C:\> del D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\try_lock_shared_for.pass.cpp
The system cannot find the path specified.

These files can be viewed in file explorer. Something that caught my eye when examining their properties is that their locations started with the \\?\ prefix! That seems unusual for files on my local drive.

File Locations Starting with \\?\

Resource monitor does not show any images with associated handles when searching for “try_lock”. Neither does searching for “\?\D:\dev\repos\llvm\”. I tried using Process Explorer’s “Find Handle or DLL…” command as well. There also don’t appear to be any child processes for the cmd.exe process I was using (a Developer 2019 Command Prompt).

Next idea, open Process Monitor and see what’s happening when dir and rmdir are executed. I used the Path contains thread.sharedtimedmutex.class filter. The deletes are showing up as SetDispositionInformationFile events and seem to be using the RemoveDirectoryW function.

The RemoveDirectory function marks a directory for deletion on close. Therefore, the directory is not removed until the last handle to the directory is closed.

RemoveDirectoryW function (fileapi.h)
Process Monitor View of RemoveDirectoryW Call
SetDispositionInformationFile Event Info

Notice the NOT EMPTY result of the SetDispositionInformationFile event. I believe this comes from RemoveDirectoryW. There’s the question of how the 3 files are printed to the command line. The FindNextFile API is used to search for files.

Ah, in the middle of this investigation, PowerShell.exe dies and so does Windows Terminal. All my tabs, everything, gone! Aaaargh… Windows Event Viewer has an Information level event showing that powershell.exe crashed due to a System.InvalidOperationException. This is then followed by another Information event with the WER source and P1 problem signature Microsoft.WindowsTerminal_1.15.2713.0_x64__8wekyb3d8bbwe. Then comes the Error level event with the Application Hang source and General explanation that “The program WindowsTerminal.exe version 1.15.2209.28003 stopped interacting with Windows and was closed. To see if more information about the problem is available, check the problem history in the Security and Maintenance control panel.” The ExeFileName is cut off below but simply append “\WindowsTerminal.exe” to the package name to reconstruct it. Looks like I need to avoid PowerShell. And why is there no crash dump created for it???

I use the Feedback Hub for the first time, trying to see whether there is a way to prevent Windows Terminal from dying with child processes. Windows Terminal crashes immmediately when launched from Win+X menu · Issue #13108 · microsoft/terminal (github.com) seems to suggest that Feedback Hub is the right way to do this.

One upside of this crash is that it lets me confirm that it is not the cmd.exe process that is hanging onto those files. I terminate explorer.exe and when I run new task in Task Manager, it asks me to create a Windows Hello pin. What is happening?? Now moving on to opening these files: Notepad++ acts as though nothing happened when you File->Open and select one of them. Notepad opens it though! Running cat in Git Bash also dumps its contents:

cat /d/dev/repos/llvm/dups/llvm-project/llvm/utils/release/llvm_package_15.0.2/llvm-project-llvmorg-15.0.2/libcxx/test/std/thread/thread.mutex/thread.mutex.requirements/thread.sharedtimedmutex.requirements/thread.sharedtimedmutex.class/try_lock_shared_for.pass.cpp
//===----------------------------------------------------------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
...

I’m suspecting Windows Defender but don’t have any definitive proof. According to Enable attack surface reduction (ASR) rules, this command in an admin powershell should do the trick if the problem was Windows Defender:

Add-MpPreference -AttackSurfaceReductionOnlyExclusions D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release

Well, looks like rm from Git Bash works just fine as does deleting from file explorer. Unfortunately, this unsolved issue is probably going to continue to cause pain in command prompt batch files like the LLVM build script.

rm /d/dev/repos/llvm/dups/llvm-project/llvm/utils/release/llvm_package_15.0.2/llvm-project-llvmorg-15.0.2/libcxx/test/std/thread/thread.mutex/thread.mutex.requirements/thread.sharedtimedmutex.requirements/thread.sharedtimedmutex.class/try_lock_shared_for.pass.cpp

Categories: hsdis, Installers, LLVM, Windows

Tracking Down Missing Headers in LLVM for Windows

hsdis is a plugin for disassembling code dynamically generated by the Java Virtual Machine. On Linux & MacOS, it uses GNU binutils. Support for the LLVM disassembly backend was recently added to hsdis in https://github.com/openjdk/jdk/pull/7531. This was motivated by the fact that GNU binutils is not distributed with the JDK (due to licensing reasons mentioned at https://github.com/openjdk/jdk/pull/5920#issuecomment-942398786) and the LLVM disassembly may be preferrable in certain circumstances. Unfortunately, the official Windows LLVM distribution does not have the header files necessary to build the hotspot disassembler. This prevents Windows developers from easily using the LLVM disassembler backend because they now have to build LLVM themselves as well – see hsdis LLVM backend for Windows ARM64 and Building LLVM for Windows ARM64, for example. In this post, we investigate why the LLVM Windows build does not have the necessary header files. The llvm-c directory in Windows build contains these 2 files only:

C:\Program Files\LLVM\include\llvm-c>dir
 Volume in drive C is OSDisk
 Volume Serial Number is c070-2ac0

 Directory of C:\Program Files\LLVM\include\llvm-c

01/08/2022  11:54 AM    <DIR>          .
01/08/2022  11:54 AM    <DIR>          ..
09/24/2021  10:18 AM            29,760 lto.h
09/24/2021  10:18 AM             9,632 Remarks.h
               2 File(s)         39,392 bytes
               2 Dir(s)  62,273,200,128 bytes free

I created a local LLVM build (see Building LLVM with CMake) and confirmed that it has all the header files.

C:\dev\repos\llvm-project\build_llvm\install_local\include\llvm-c>dir /w
 Volume in drive C is OSDisk
 Volume Serial Number is 0087-4c48

 Directory of C:\dev\repos\llvm-project\build_llvm\install_local\include\llvm-c

[.]                   [..]                  Analysis.h
BitReader.h           BitWriter.h           blake3.h
Comdat.h              Core.h                DataTypes.h
DebugInfo.h           Deprecated.h          Disassembler.h
DisassemblerTypes.h   Error.h               ErrorHandling.h
ExecutionEngine.h     ExternC.h             Initialization.h
IRReader.h            Linker.h              LLJIT.h
lto.h                 Object.h              Orc.h
OrcEE.h               Remarks.h             Support.h
Target.h              TargetMachine.h       [Transforms]
Types.h
              28 File(s)        382,361 bytes
               3 Dir(s)  59,158,138,880 bytes free

Does this problem still exist in the latest Windows LLVM release? I went to Releases · llvm/llvm-project (github.com) to find the latest LLVM installer for Windows but couldn’t find it. Turns out it’s because the 15.0.1 release is only 14 hours old so some of the assets probably haven’t been uploaded. Notice that 15.0.0 has 47 assets. I can successfully download and install LLVM-15.0.0-win64.exe and see that the header files are still missing.

Interestingly, trying to install LLVM-15.0.0-win32.exe before uninstalling LLVM-15.0.0-win64.exe gives this dialog and clicking Yes uninstalls before the actual installation of the 32-bit build starts!

LLVM is already installed.

I assumed that would happen at this stage:

All the same, these dialogs have strings that can lead us to the sources that create the installer! The installer looks very similar to the one from Building the Elmer Install Folder so searching the llvm codebase for “ncis ” gives only a handful of hits leading to the key discovery of build_llvm_release.bat! (later learn that this needs to be executed in a (2019) developer command prompt so that the ninja command can be found). That script requires 7zip though. The script fails on my machine because it can’t find 7zip. Failure seems to be coming from the for-statement (see for | Microsoft Learn for usage). The for command uses the escape character (^) as explained at set | Microsoft Learn.

C:\dev\repos\llvm-project\llvm\utils\release> build_llvm_release.bat 15.0.0
Check 7-zip version and/or administrator permissions.
'7z.exe' is not recognized as an internal or external command,
operable program or batch file.
You need to modify the paths below:
Revision: llvmorg-15.0.0
Package version: 15.0.0
Build dir: C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0

Press any key to continue . . .

Why does the script not exit gracefully now? Git blame shows that the checking code was added by Update the Windows packaging script. · llvm/llvm-project@df7c577 (github.com). Adding the 7-Zip installation path to my user environment variables addresses this issue but the script should check for this!

Installing 7-Zip allows me to check the syntax of the command used by the script to ensure that it will work.

C:\Program Files\7-Zip> 7z.exe | findstr /r "2[1-9].[0-9][0-9]"
7-Zip 22.01 (x64) : Copyright (c) 1999-2022 Igor Pavlov : 2022-07-15

Surprisingly, the script still fails, this time because

'mv' is not recognized as an internal or external command,
operable program or batch file.

This must be line 81 introduced by Update the Windows packaging script. · llvm/llvm-project@83e9225 (github.com). Changing it to “move” now displays an error but the script continues executing until this error:

-- Looking for CrashReporterClient.h
-- Looking for CrashReporterClient.h - not found
-- Looking for pfm_initialize in pfm
-- Looking for pfm_initialize in pfm - not found
-- Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)
CMake Error at C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find LibXml2 (missing: LIBXML2_INCLUDE_DIR)
Call Stack (most recent call first):
  C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/FindLibXml2.cmake:108 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  cmake/config-ix.cmake:156 (find_package)
  CMakeLists.txt:774 (include)


-- Configuring incomplete, errors occurred!
See also "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/CMakeFiles/CMakeOutput.log".
See also "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/CMakeFiles/CMakeError.log".

The script downloads and extracts LibXml in the build directory. It also changes the libxmldir path separators from \ to /. To see the exact command failing, comment out the echo off line.

cmake
 -GNinja
 -DCMAKE_BUILD_TYPE=Release
 -DLLVM_ENABLE_ASSERTIONS=OFF
 -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON
 -DLLVM_BUILD_LLVM_C_DYLIB=ON
 -DCMAKE_INSTALL_UCRT_LIBRARIES=ON
 -DPython3_FIND_REGISTRY=NEVER
 -DPACKAGE_VERSION=15.0.0
 -DLLDB_RELOCATABLE_PYTHON=1
 -DLLDB_EMBED_PYTHON_HOME=OFF
 -DCMAKE_CL_SHOWINCLUDES_PREFIX="Note: including file: "
 -DLLVM_ENABLE_LIBXML2=FORCE_ON
 -DLLDB_ENABLE_LIBXML2=OFF
 -DCMAKE_C_FLAGS="-DLIBXML_STATIC"
 -DCMAKE_CXX_FLAGS="-DLIBXML_STATIC"
 -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld;compiler-rt;lldb;openmp"
 -DLLDB_TEST_COMPILER=C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0/build32_stage0/bin/clang.exe
 -DPYTHON_HOME=C:\Users\saint\AppData\Local\Programs\Python\Python310-32
 -DPython3_ROOT_DIR=C:\Users\saint\AppData\Local\Programs\Python\Python310-32
 -DLIBXML2_INCLUDE_DIRS=C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/libxmlbuild/install/include/libxml2
 -DLIBXML2_LIBRARIES=C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/libxmlbuild/install/lib/libxml2s.lib ..\llvm-project\llvm

Looking through FindPackageHandleStandardArgs.cmake leads me to the simple realization that the wrong define is being used on the command line. Could this be because I’m using a newer CMake? I’ve been using the VS 2022 Preview Developer Command Prompt thus far. My VS 2019 (16.11.19) installation uses CMake 3.20. Both FindLibXml2.cmake in 3.20 and FindLibXml2.cmake in 3.24 require the LIBXML2_INCLUDE_DIR variable. However, they also claim (at the top) to set these variables.

A quick review of the history of build_llvm_release.bat shows that Build Windows releases with libxml enabled, to unbreak llvm-mt · llvm/llvm-project@145835c (github.com) introduced -DLIBXML2_INCLUDE_DIR but the next commit Pass -DLIBXML2_INCLUDE_DIRS in the Windows release package script · llvm/llvm-project@7735019 (github.com) changed it to plural. Adding the singular form to the script finally unblocks the build. Now to see how packing happens..

Packaging a Regular LLVM Build

in my build folder (build_llvm), there is a CPackConfig.cmake file that sets variables like CPACK_PACKAGE_FILE_NAME and CPACK_NSIS_DISPLAY_NAME. Since it is NSIS Wiki (sourceforge.io) in use, I wonder about running the package target myself in a manner similar to that used to create my local build. I switch back to a previous build directory (created without the build_llvm_release.bat) and run:

cmake --build . --config Release --target package

The resulting failure below indicates that NSIS is required.

MSBuild version 17.4.0-preview-22466-03+48ab5664b for .NET Framework
  PipSqueak.vcxproj -> C:\dev\repos\llvm-project\build_llvm\unittests\Support\DynamicLibrary\Release\PipSqueak.dll
  SecondLib.vcxproj -> C:\dev\repos\llvm-project\build_llvm\unittests\Support\DynamicLibrary\Release\SecondLib.dll
  obj.llvm-tblgen.vcxproj -> C:\dev\repos\llvm-project\build_llvm\utils\TableGen\obj.llvm-tblgen.dir\Release\obj.llvm-tblgen.lib
  LLVMDemangle.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\lib\LLVMDemangle.lib
...
  verify-uselistorder.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\verify-uselistorder.exe
  yaml-bench.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\yaml-bench.exe
  yaml2obj.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\yaml2obj.exe
EXEC : CPack error : Cannot find NSIS compiler makensis: likely it is not installed, or not in your PATH [C:\dev\repos\llvm-project\build_llvm\package.vcxproj]
EXEC : CPack error : Could not read NSIS registry value. This is usually caused by NSIS not being installed. Please install NSIS from http://nsis.sourceforge.net [C:\dev\repos\llvm-proje
ct\build_llvm\package.vcxproj]
EXEC : CPack error : Cannot initialize the generator NSIS [C:\dev\repos\llvm-project\build_llvm\package.vcxproj]

After installing NSIS, the previous command successfully creates an LLVM for Windows installer.

...
  verify-uselistorder.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\verify-uselistorder.exe
  yaml-bench.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\yaml-bench.exe
  yaml2obj.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\yaml2obj.exe
  CPack: Create package using NSIS
  CPack: Install projects
  CPack: - Install project: LLVM [Release]
  CMake Warning (dev) at C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/GNUInstallDirs.cmake:243 (messa
  ge):
    Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
    target architecture is known.  Please enable at least one language before
    including GNUInstallDirs.
  Call Stack (most recent call first):
    C:/dev/repos/llvm-project/llvm/cmake/modules/LLVMInstallSymlink.cmake:5 (include)
    C:/dev/repos/llvm-project/build_llvm/tools/llvm-ar/cmake_install.cmake:48 (include)
    C:/dev/repos/llvm-project/build_llvm/tools/cmake_install.cmake:39 (include)
    C:/dev/repos/llvm-project/build_llvm/cmake_install.cmake:71 (include)
  This warning is for project developers.  Use -Wno-dev to suppress it.

  CPack: Create package
  CPack: - package: C:/dev/repos/llvm-project/build_llvm/LLVM-16.0.0git-win64.exe generated.

This installer generates the LLVM includes on disk as expected. The issue must therefore be confined to the installer generated by the script.

Reviewing Ninja NSIS Packaging

At this point, I ran build_llvm_release.bat to create an installer. Once packaging is complete, the install_manifest.txt file can be used to determine which files are in the installer. The batch file also runs lots of tests and this was annoying when trying to generate installers. Once the tests failed on the build I was creating and I had CTRL+C’d a couple of times, I ran ninja package myself (taken from the batch file)

C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0>ninja package
[0/1] Run CPack packaging tool...CPack: Create package using NSIS
CPack: Install projects
CPack: - Install project: LLVM []
CMake Warning (dev) at C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/GNUInstallDirs.cmake:243 (message):
  Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
  target architecture is known.  Please enable at least one language before
  including GNUInstallDirs.
Call Stack (most recent call first):
  C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/llvm-project/llvm/cmake/modules/LLVMInstallSymlink.cmake:5 (include)
  C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/tools/llvm-ar/cmake_install.cmake:40 (include)
  C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/tools/cmake_install.cmake:39 (include)
  C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/cmake_install.cmake:114 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

CPack: Create package
CPack: - package: C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/LLVM-15.0.0-win32.exe generated.

I was curious about these warnings but found it really annoying that I can’t open these paths by pasting them into the VS Code File/Open dialog. Looks like Windows: Allow to open file paths that contain slashes · Issue #15270 · microsoft/vscode (github.com) but that points to [Windows] Bug with open file dialog with forward slash (`file_dialog::ShowOpenDialog`) · Issue #7954 · electron/electron (github.com). Unfortunately, this sounds like a Windows Open dialog limitation since Notepad is not using the same dialog as Notepad++.

I then try to find a package target in build.ninja. Search for CMakeFiles\package.util.+ include since we’re interested in include files. There are some interesting differences in the include directories of the build created manually from the local install and the one created by the script, e.g.

Directory of C:\dev\repos\llvm-project\build_llvm\include\llvm\Support
[.]
[..]
[CMakeFiles]
cmake_install.cmake
Extension.def
INSTALL.vcxproj
INSTALL.vcxproj.filters
llvm_vcsrevision_h.vcxproj
llvm_vcsrevision_h.vcxproj.filters
PACKAGE.vcxproj
PACKAGE.vcxproj.filters
VCSRevision.h
[x64]
               9 File(s)         47,877 bytes
               4 Dir(s)  34,980,511,744 bytes free
Directory of C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0\include\llvm\Support

[.]                   [..]                  [CMakeFiles]
cmake_install.cmake   Extension.def         VCSRevision.h
               3 File(s)          1,293 bytes
               3 Dir(s)  34,981,122,048 bytes free

Try searching in build.ninja for the 2 header files the installer creates in the (broken) shipping LLVM for Windows build. Nothing there but searching the file system for remarks.h gives interesting results, e.g. the existence of an NSIS project file: project.nsi. Looks like there are some tutorials showing how to create .nsi files at Invoking NSIS run-time commands on compile-time – NSIS (sourceforge.io). The way NSIS is used with CPack when building is documented at Packaging With CPack — Mastering CMake

Directory of C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0\_CPack_Packages\win32\NSIS

09/21/2022  06:41 PM    <DIR>          .
09/21/2022  06:41 PM    <DIR>          ..
09/21/2022  06:41 PM    <DIR>          LLVM-15.0.0-win32
09/21/2022  06:54 PM       256,557,945 LLVM-15.0.0-win32.exe
09/21/2022  06:41 PM               631 NSIS.InstallOptions.ini
09/21/2022  06:41 PM            55,204 project.nsi
               3 File(s)    256,613,780 bytes
               3 Dir(s)  35,416,317,952 bytes free

Directory of C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0\_CPack_Packages\win32\NSIS\LLVM-15.0.0-win32\include\llvm-c

09/21/2022  06:41 PM    <DIR>          .
09/21/2022  06:41 PM    <DIR>          ..
09/05/2022  03:48 AM            30,109 lto.h
09/05/2022  03:48 AM             9,632 Remarks.h
               2 File(s)         39,741 bytes
               2 Dir(s)  35,416,289,280 bytes free

The natural hypothesis is that NSIS is simply packing the whole LLVM-15.0.0-win32 directory into the installer. I had been comparing these two files earlier…

C:\dev\repos\llvm-project\build_llvm\cmake_install.cmake
C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0\cmake_install.cmake

… but I completely missed the fact that the 2nd didn’t have these lines from the first.

if(CMAKE_INSTALL_COMPONENT STREQUAL "llvm-headers" OR NOT CMAKE_INSTALL_COMPONENT)
  file(INSTALL DESTINATION "${CMAKE_INSTALL_PREFIX}/include" TYPE DIRECTORY FILES
    "C:/dev/repos/llvm-project/llvm/include/llvm"
    "C:/dev/repos/llvm-project/llvm/include/llvm-c"
    FILES_MATCHING REGEX "/[^/]*\\.def$" REGEX "/[^/]*\\.h$" REGEX "/[^/]*\\.td$" REGEX "/[^/]*\\.inc$" REGEX "/license\\.txt$")
endif()

if(CMAKE_INSTALL_COMPONENT STREQUAL "llvm-headers" OR NOT CMAKE_INSTALL_COMPONENT)
  file(INSTALL DESTINATION "${CMAKE_INSTALL_PREFIX}/include" TYPE DIRECTORY FILES
    "C:/dev/repos/llvm-project/build_llvm/include/llvm"
    "C:/dev/repos/llvm-project/build_llvm/include/llvm-c"
    FILES_MATCHING REGEX "/[^/]*\\.def$" REGEX "/[^/]*\\.h$" REGEX "/[^/]*\\.gen$" REGEX "/[^/]*\\.inc$" REGEX "/cmakefiles$" EXCLUDE REGEX "/config\\.h$" EXCLUDE)
endif()

Search the codebase for “llvm-headers” and find the llvm-header component definition. That whole code block is gated by the LLVM_INSTALL_TOOLCHAIN_ONLY variable! This is explicitly turned off in build_llvm_release.bat! I rerun the batch file and see tests failing after the build succeeds. CTRL+C to kill the processes so that I can get to the root issue: does turning off that flag fix the includes? makensis fails, probably because I killed the build and some things might still have been in use?

C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0>ninja package
[0/1] Run CPack packaging tool...CPack: Create package using NSIS
CPack: Install projects
CPack: - Install project: LLVM []
CMake Warning (dev) at C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/GNUInstallDirs.cmake:243 (message):
  Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
  target architecture is known.  Please enable at least one language before
  including GNUInstallDirs.
Call Stack (most recent call first):
  C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/llvm-project/llvm/cmake/modules/LLVMInstallSymlink.cmake:5 (include)
  C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/tools/llvm-ar/cmake_install.cmake:40 (include)
  C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/tools/cmake_install.cmake:39 (include)
  C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/cmake_install.cmake:128 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

CPack: Create package
CPack Error: Problem running NSIS command: "C:/Program Files (x86)/NSIS/makensis.exe" "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/_CPack_Packages/win32/NSIS/project.nsi"
Please check C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/_CPack_Packages/win32/NSIS/NSISOutput.log for errors
CPack Error: Problem compressing the directory
CPack Error: Error when generating package: LLVM

FAILED: CMakeFiles/package.util
cmd.exe /C "cd /D C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0 && "C:\Program Files\Microsoft Visual Studio\2022\Preview\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cpack.exe" --config ./CPackConfig.cmake"
ninja: build stopped: subcommand failed.

NSISOutput.log failed due to an “Internal compiler error #12345: error mmapping datablock to 17235001.” However, the include files are now present in the source directory being packaged by NSIS.

Turning Off Tests

There are many tests that the build script runs and some of them are failing. Testing is not on my critical path since all I need is to generate installers so I modify the scripts to enable me to package the build without running all the tests. I then start my build without tests and go to bed only to wake up the next morning to find that I need to rerun it because there are no running programs when I log in. Event Viewer doesn’t show any reboot-related events and sure enough, Task Manager shows over 9 days of uptime still. Turns out the Desktop Window Manager crashed (C:\WINDOWS\system32\dwm.exe)! Curse you dwmcore.dll. Well, time to install those updates I’ve been putting off and reboot before jumping back in. Now on the new Windows 10.0.22621.521. The build still fails:

-- LLVM host triple: i686-pc-windows-msvc
-- LLVM default target triple: i686-pc-windows-msvc
-- Using Release VC++ CRT: MD
-- Looking for os_signpost_interval_begin
-- Looking for os_signpost_interval_begin - not found
CMake Error at C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python3 (missing: Python3_EXECUTABLE Interpreter) (Required
  is at least version "3.6")

      Reason given by package:
          Interpreter: Cannot use the interpreter "C:/Python310/python.exe"

Call Stack (most recent call first):
  C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPython/Support.cmake:3165 (find_package_handle_standard_args)
  C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPython3.cmake:485 (include)
  CMakeLists.txt:817 (find_package)


-- Configuring incomplete, errors occurred!
See also "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/CMakeFiles/CMakeOutput.log".
See also "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/CMakeFiles/CMakeError.log".

When I interrupted the tests before modifying the batch file to skip them, I noticed that they were being run by %LOCALAPPDATA%\Microsoft\WindowsApps\python3.9.exe. This is still present on my machine. Ah, turns out I’m now using the 2019 developer command prompt (and therefore an older CMake). The only difference between CMake 3.20 FindPython3.cmake and CMake 3.24 FindPython3.cmake is a comment about static libraries, so this failure is a mystery.

Diagnosing Build Failures

Since this issue also bit me when I moved to my Surface Book, it is worth understanding why it happens.

Missing CMake in Visual Studio 17.3.4 Developer Command Prompt

Here is the VS 2022 Preview vs VS 2022 Enterprise path to CMake:

C:\Program Files (x86)\Microsoft Visual Studio\Installer> where cmake
C:\Program Files\Microsoft Visual Studio\2022\Preview\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe

C:\dev\repos\llvm-project\llvm\utils\release> where cmake
INFO: Could not find files for the given pattern(s).

Wait… why is there no CMake in VS 2022 Enterprise on my desktop? The Visual Studio Installer shows cmake to be installed. It also warnings and it’s only now that I’m learning that there are not just troubleshooting tips but also ways to create a local layout from the command line Create an offline installation – Visual Studio (Windows) | Microsoft Learn

The View Logs link opens the Documents folder under This PC – not particularly useful. Interestingly though, clicking on the Modify button shows a Total space required 1.63 GB. How is there space required before I’ve selected anything? Something similar happens with 16.11.19 though. Without making any individual component selections, I start the install process. CMake gets (re-?)installed as shown below. This fixes the setup warnings as well and cmake is now usable in the VS2022 command prompt.

Missing Python3 in VS 17.3.4 Developer Command Prompt

This is the error I got when trying to build LLVM on my Surface Book 2 in the VS 2022 developer command prompt:

CMake Error at C:/Program Files/CMake/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:164 (message):
  Could NOT find Python3 (missing: Python3_EXECUTABLE Interpreter) (Required
  is at least version "3.6")

      Reason given by package:
          Interpreter: Cannot use the interpreter "C:/Python310/python.exe"

Call Stack (most recent call first):
  C:/Program Files/CMake/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:445 (_FPHSA_FAILURE_MESSAGE)
  C:/Program Files/CMake/share/cmake-3.17/Modules/FindPython/Support.cmake:2437 (find_package_handle_standard_args)
  C:/Program Files/CMake/share/cmake-3.17/Modules/FindPython3.cmake:309 (include)
  CMakeLists.txt:817 (find_package)

Here is the (fixed up) output from where python:

C:\Python310\python.exe
%LOCALAPPDATA%\Microsoft\WindowsApps\python.exe

I modify build_llvm_release.bat to pass the --trace-expand --trace-redirect=cmake_trace.txt CMake option as recommended by cmake Python: Cannot use the interpreter – Stack Overflow. That’s when I notice that the list of python versions CMake is looking for does not contain 3.10: Modules/FindPython/Support.cmake · v3.17.5 · CMake / CMake · GitLab (kitware.com). My suspicion is that this is the cause of the above error. It looks like I installed CMake a while back on this laptop.

Uninstalling CMake enables the command line to pick up the CMake distributed with Visual Studio. Python3 is now found successfully in the path below (I’ve shortened it using %LOCALAPPDATA%).

-- Found Python3: %LOCALAPPDATA%/Microsoft/WindowsApps/python3.8.exe (found suitable version "3.8.10", minimum required is "3.6") found components: Interpreter

Missing Python3 in VS 16.11.19 Developer Command Prompt

Interestingly, I still get the same error in VS 2019 despite uninstalling CMake 3.17. My earlier hypothesis is therefore invalid.

CMake Error at C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python3 (missing: Python3_EXECUTABLE Interpreter) (Required
  is at least version "3.6")

      Reason given by package:
          Interpreter: Cannot use the interpreter "C:/Python310/python.exe"

Call Stack (most recent call first):
  C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPython/Support.cmake:3165 (find_package_handle_standard_args)
  C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPython3.cmake:485 (include)
  CMakeLists.txt:817 (find_package)

Using --trace-expand --trace-redirect=cmake_trace.txt again (and searching for “execute_process“) reveals how the python version is determined using the execute_process cmake command in Modules/FindPython/Support.cmake · v3.20.0:

C:/Python310/python.exe -c "import sys; sys.stdout.write('.'.join([str(x) for x in sys.version_info[:3]]))"

I comment out the ERROR_QUIET line to reveal the stdout and stderr output from python since the return code from the python process is causing the CMake error to be raised. Running with --trace-expand --trace-redirect=cmake_trace.txt now reveals the root cause (paths below cleaned up using %LOCALAPPDATA%):

Python path configuration:
  PYTHONHOME = '%LOCALAPPDATA%\Programs\Python\Python310-32'
  PYTHONPATH = (not set)
  program name = 'C:/Python310/python.exe'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = 'C:\\Python310\\python.exe'
  sys.base_prefix = '%LOCALAPPDATA%\\Programs\\Python\\Python310-32'
  sys.base_exec_prefix = '%LOCALAPPDATA%\\Programs\\Python\\Python310-32'
  sys.platlibdir = 'lib'
  sys.executable = 'C:\\Python310\\python.exe'
  sys.prefix = '%LOCALAPPDATA%\\Programs\\Python\\Python310-32'
  sys.exec_prefix = '%LOCALAPPDATA%\\Programs\\Python\\Python310-32'
  sys.path = [
    'C:\\Python310\\python310.zip',
    '%LOCALAPPDATA%\\Programs\\Python\\Python310-32\\DLLs',
    '%LOCALAPPDATA%\\Programs\\Python\\Python310-32\\lib',
    'C:\\Python310',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00003174 (most recent call first):
  <no Python frame>

django – init_fs_encoding: failed to get the Python codec of the filesystem encoding – Stack Overflow is a hint that the PYTHONHOME is wrong. Sure enough, I didn’t change it in build_llvm_release.bat so the paths in the configuration above do not exist! This now raises another question: how on earth does this work in VS 2022? I notice on my desktop that python.exe does not even appear in the CMake tracing output! The difference in behavior stems from the fact that the find_program command in Modules/FindPython/Support.cmake · v3.20.0 finds python 3.10 first in the VS 2019 environment. This path is then assigned to _Python3_EXECUTABLE, preventing the 3.8 path from being used. One important difference between CMake 3.20 and 3.23 that I notice is FindPython: fix typo error (fff8d5b2) · Commits · CMake / CMake · GitLab (kitware.com). Since the fix for the build_llvm_release.bat script is straightforward and it is clear that there are some CMake implementation differences at work, we no longer need to dig into why this behavior could be happening.

Python Hangs

One of my build attempts successfully completes stage0 but hangs when CMake tries to detect the python version. Manually running the same command (copied from Process Explorer) also hangs. Even %LOCALAPPDATA%/Microsoft/WindowsApps/python3.9.exe --version hangs. Inspecting the full dump created by Task Manager reveals that python3.9.exe made a call to get (what looks like) the Package.InstalledLocation Property (Windows.ApplicationModel) – Windows UWP applications | Microsoft Learn

...
-- Looking for os_signpost_interval_begin
-- Looking for os_signpost_interval_begin - not found

Windows becomes pretty unusable as I investigate this behavior (mouse doesn’t work, changes program in focus but can’t click on anything). A reboot fixes these issues (e.g. version now works). Can’t believe we have to deal with this in 2022???

The support link is https://www.python.org/doc/ and the product link is https://www.python.org/. The privacy policy is https://www.python.org/privacy/ and the license terms link is https://docs.python.org/3.9/license.html.

I’m tempted to just remove this store app but also curious about how to get symbols and see exactly where it hang.

Comparison with macOS/Linux includes

On Windows, it is easy to get the Linux and macOS LLVM builds using curl (added to Windows in build 17063 as per Tar and Curl Come to Windows! | Microsoft Learn).

curl -L https://github.com/llvm/llvm-project/releases/download/llvmorg-15.0.1/clang+llvm-15.0.1-aarch64-linux-gnu.tar.xz -o clang+llvm-15.0.1-aarch64-linux-gnu.tar.xz

curl -L https://github.com/llvm/llvm-project/releases/download/llvmorg-15.0.1/clang+llvm-15.0.1-arm64-apple-darwin21.0.tar.xz -o clang+llvm-15.0.1-arm64-apple-darwin21.0.tar.xz

curl -L https://github.com/llvm/llvm-project/releases/download/llvmorg-15.0.1/clang+llvm-15.0.1-x86_64-apple-darwin.tar.xz -o clang+llvm-15.0.1-x86_64-apple-darwin.tar.xz

To use 7zip to extract these XZ Files:

7z x clang+llvm-15.0.1-aarch64-linux-gnu.tar.xz
7z x clang+llvm-15.0.1-arm64-apple-darwin21.0.tar.xz
7z x clang+llvm-15.0.1-x86_64-apple-darwin.tar.xz

tar xf clang+llvm-15.0.1-aarch64-linux-gnu.tar
tar xf clang+llvm-15.0.1-arm64-apple-darwin21.0.tar
tar xf clang+llvm-15.0.1-x86_64-apple-darwin.tar

To use XZ Utils (tukaani.org) to extract these XZ Files, run these commands but note that they remove the .xz files!

xz -d clang+llvm-15.0.1-aarch64-linux-gnu.tar.xz
xz -d clang+llvm-15.0.1-arm64-apple-darwin21.0.tar.xz
xz -d clang+llvm-15.0.1-x86_64-apple-darwin.tar.xz

tar xf clang+llvm-15.0.1-aarch64-linux-gnu.tar
tar xf clang+llvm-15.0.1-arm64-apple-darwin21.0.tar
tar xf clang+llvm-15.0.1-x86_64-apple-darwin.tar

Here are the ARM64 llvm include directory listings for macOS and Linux LLVM builds.

.../Downloads/clang+llvm-15.0.1-arm64-apple-darwin21.0/include
c++
clang
clang-c
clang-tidy
flang
lld
lldb
llvm
llvm-c
mlir
mlir-c
polly

.../Downloads/clang+llvm-15.0.1-aarch64-linux-gnu/include
aarch64-unknown-linux-gnu
c++
clang
clang-c
clang-tidy
flang
lld
lldb
llvm
llvm-c
mlir
mlir-c
ompt-multiplex.h
polly

Here are the directories in the include folder before the installer is created. There are also 28 include files in the include/llvm-c/ directory as desired.

Directory of llvm\utils\release\llvm_package_15.0.0\build32_stage0\_CPack_Packages\win64\NSIS\LLVM-15.0.0-win64\include
 clang
 clang-c
 clang-tidy
 lld
 lldb
 llvm
 llvm-c

Outstanding Questions

  1. Why does the NSIS project fail to build? Why are there test failures and build errors?
  2. Why does the Linux build have ompt-multiplex.h and the aarch64-unknown-linux-gnu directory?
  3. How is the Windows ARM64 installer generated?
  4. Why doesn’t the Windows build have c++, flang, mlir, mlir-c, and polly?
  5. How do we get symbols to the Python app in the Microsoft Store?

Categories: Compilers, Fortran, LLVM

Failing to Build Flang with Visual C++

Background

Elmer is the first codebase that I have dug into that has a substantial (or really any) amount of Fortran code. I used GFortran to build it but went digging around for a clang based compiler. I found llvm-project/flang and since I had been building LLVM earlier this year, I figured it should be straightforward to build flang and perhaps explore it in a debugger.

My first attempt to build flang (on Windows, my primary OS) resulted in many build errors. Unfortunately, I was using a preview Visual Studio build, so I didn’t want to compare the errors with those from a different machine because it wasn’t the same compiler version in use. I decided to use an RTM Visual Studio compiler (VS 17.2.5) to avoid possible compiler bugs present only in VS preview builds since most people would not be using preview VS builds anyway.

Without giving it much thought, my suspicion was that any build failures probably arose from not using the correct C++ version. The source code I was trying to build (commit c0702ac0) states that it uses C++17. I set this in CMake by defining the CXX_STANDARD property. Here is the full cmake command line I used to set up the build.

cd llvm-project
mkdir build
cd build

cmake \
  -G Ninja \
  ../llvm \
  -DCMAKE_BUILD_TYPE=Release \
  -DFLANG_ENABLE_WERROR=On \
  -DLLVM_ENABLE_ASSERTIONS=ON \
  -DLLVM_TARGETS_TO_BUILD=host \
  -DCMAKE_INSTALL_PREFIX=../install
  -DLLVM_LIT_ARGS=-v \
  -DLLVM_ENABLE_PROJECTS="clang;mlir;flang" \
  -DLLVM_ENABLE_RUNTIMES="compiler-rt" \
  -DCXX_STANDARD=17

# Shown here without \ to be executable in cmd.exe
cmake -G Ninja ../llvm -DCMAKE_BUILD_TYPE=Release -DFLANG_ENABLE_WERROR=On -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_TARGETS_TO_BUILD=host -DCMAKE_INSTALL_PREFIX=../install -DLLVM_LIT_ARGS=-v -DLLVM_ENABLE_PROJECTS="clang;mlir;flang" -DLLVM_ENABLE_RUNTIMES="compiler-rt" -DCXX_STANDARD=17

That took about 2 minutes on my machine after which I ran ninja to start the build

ninja

Unfortunately, the build failed! The first error I encountered was in fold-real.cpp. Here is the command line used to invoke the compiler (shown with newlines to simplify interpretation, see Compiler options listed alphabetically | Microsoft Docs for the complete list of compiler options).

 C:\PROGRA~1\MIB055~1\2022\ENTERP~1\VC\Tools\MSVC\1432~1.313\bin\Hostx64\x64\cl.exe
 /nologo
 /TP
 -DFLANG_LITTLE_ENDIAN=1
 -DGTEST_HAS_RTTI=0 -DUNICODE
 -D_CRT_NONSTDC_NO_DEPRECATE
 ...
 -D__STDC_LIMIT_MACROS
 -ID:\dev\repos\llvm-project\build-cpp17\tools\flang\lib\Evaluate
 ...
 -ID:\dev\repos\llvm-project\llvm\include
 -external:I D:\dev\repos\llvm-project\llvm\..\mlir\include
 ...
 -external:I D:\dev\repos\llvm-project\llvm\..\clang\include
 -external:W0
 /DWIN32
 /D_WINDOWS
 /Zc:inline
 /Zc:__cplusplus
 /Oi
 /bigobj
 /permissive-
 /W4
 -wd4141
 ...
 -wd4324
 -w14062
 -we4238
 /Gw
 /WX
 /MD
 /O2
 /Ob2
 /EHs-c-
 /GR-
 -UNDEBUG
 -std:c++17
 /showIncludes
 /Fotools\flang\lib\Evaluate\CMakeFiles\obj.FortranEvaluate.dir\fold-real.cpp.obj
 /Fdtools\flang\lib\Evaluate\CMakeFiles\obj.FortranEvaluate.dir\
 /FS
 -c
 D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-real.cpp

I had to look up the meaning of the C++ syntax on that line to understand why the build could be failing. Turns out to be a lambda, as explained at c++ – What is a lambda expression in C++11? – Stack Overflow.

I tried manually creating a repro for this compiler issue by creating a new Visual C++ project in Visual Studio and recreating the structure of the code failing to build. One of the questions I had was how to set conformance mode in a Visual Studio Cmake project. I still haven’t yet figured this out. However, one of the issues I ran into was that my cmake project was building the code without the /permissive- flag! I ended up switching to a regular Visual C++ project (.vcxproj) since I knew how to change the compiler options reliably for such projects. After struggling with recreating the code, I realized that I would make more progress removing code from flang’s fold-real.cpp instead. Here are some of the other searches and concepts I had to look up to understand the code while trying to create a minimal repro of the build failure.

  1. c++11 – how to remove error : X is not a class template – Stack Overflow
  2. c++ – What does template <unsigned int N> mean? – Stack Overflow
  3. what does this … (three dots) means in c++ – Stack Overflow
  4. create a reference in c++ -> Reference of Reference in C++ – Stack Overflow
  5. c++ using namespace -> Namespaces (C++) | Microsoft Docs
  6. using typedef -> Aliases and typedefs (C++) | Microsoft Docs
  7. move mechanics -> C++ Move Semantics Introduction | hacking C++ (hackingcpp.com)
  8. c++ – What is std::decay and when it should be used? – Stack Overflow

I was eventually able to create a simpler test case showing that the flang code could not build with my RTM compiler.

cl /std:c++17 /permissive- flang-msvc-clang-test.cpp

Microsoft (R) C/C++ Optimizing Compiler Version 19.32.31332 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

flang-msvc-clang-test.cpp
flang-msvc-clang-test.cpp(159): error C2065: 'T': undeclared identifier
flang-msvc-clang-test.cpp(48): note: see reference to function template instantiation 'auto FoldIntrinsicFunction::<lambda_1>::operator ()<_First>(const _T1 &) const' being compiled
        with
        [
            _First=Expr<Type<TypeCategory::Real,1>>,
            _T1=Expr<Type<TypeCategory::Real,1>>
        ]
flang-msvc-clang-test.cpp(171): note: see reference to function template instantiation 'Expr<Type<TypeCategory::Real,2>> FoldIntrinsicFunction<2>(FoldingContext &,FunctionRef<Type<TypeCategory::Real,2>> &&)' being compiled
flang-msvc-clang-test.cpp(159): error C2923: 'Scalar': 'T' is not a valid template type argument for parameter 'T'
flang-msvc-clang-test.cpp(159): note: see declaration of 'T'

So after all that, the RTM LTS Visual C++ compiler turned out to have a bug. Turns out the Visual C++ folks had already fixed this issue so the way to unblock myself was to switch to the preview Visual Studio build :(! The irony…

Suppressing Warnings

Armed with a preview build that correctly compiled the test case, the next obstacle in the build process was a set of warnings that were treated as errors: C4661 and C4101.

FAILED: tools/flang/lib/Evaluate/CMakeFiles/obj.FortranEvaluate.dir/fold.cpp.obj
C:\...\cl.exe ... -c D:\dev\repos\llvm-project\flang\lib\Evaluate\fold.cpp
D:\dev\repos\llvm-project\flang\include\flang\Evaluate\expression.h(101): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\include\flang\Evaluate\expression.h(101): warning C4661: 'std::optional<Fortran::evaluate::DynamicType> Fortran::evaluate::ExpressionBase<Fortran::evaluate::SomeDerived>::GetType(void) const': no suitable definition provided for explicit template instantiation request
...
FAILED: tools/flang/lib/Evaluate/CMakeFiles/obj.FortranEvaluate.dir/fold-complex.cpp.obj
C:\...\cl.exe ... -c D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-complex.cpp
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-implementation.h(1583): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-implementation.h(1583): warning C4101: 'buffer': unreferenced local variable

I tried to suppressed them to keep marching forward:

cd \dev\repos\llvm-project
mkdir build-nowarn
cd build-nowarn

cmake -G Ninja ../llvm -DCMAKE_BUILD_TYPE=Release -DFLANG_ENABLE_WERROR=On -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_TARGETS_TO_BUILD=host -DCMAKE_INSTALL_PREFIX=../install -DLLVM_LIT_ARGS=-v -DLLVM_ENABLE_PROJECTS="clang;mlir;flang" -DLLVM_ENABLE_RUNTIMES="compiler-rt" -DCXX_STANDARD=17 -DCXX_FLAGS="-wd4661 -wd4101"

ninja

Defining CXX_FLAGS like that did not work so I end up looking around for how to disable warnings in cmake. This was when I discovered that CMAKE_CXX_STANDARD is not necessary on the command line because flang/CMakeLists.txt already requires C++17. Trying to append the warning disable option /wdXXXX to that file didn’t work either. However, the comment on line 329 made me explore HandleLLVMOptions.cmake. There, I discovered support for setting the number of parallel jobs (via /MP for Visual C++). This file also contained the code that sets up most of the compiler options used when building! Closer to the task at hand is the discover of the LLVM_ENABLE_WARNINGS option and the hard-coded list of MSVC warning flags! I therefore made this change (before running cmake and ninja) to get the warning flags to be respected:

diff --git a/llvm/cmake/modules/HandleLLVMOptions.cmake b/llvm/cmake/modules/HandleLLVMOptions.cmake
index 56d05f5b5fce..589281b232f1 100644
--- a/llvm/cmake/modules/HandleLLVMOptions.cmake
+++ b/llvm/cmake/modules/HandleLLVMOptions.cmake
@@ -648,6 +648,8 @@ if (MSVC)
           # v15.8.8. Re-evaluate the usefulness of this diagnostic when the bug
           # is fixed.
       -wd4709 # Suppress comma operator within array index expression
+      -wd4101  # Suppress ...
+      -wd4661  # Suppress ...

       # Ideally, we'd like this warning to be enabled, but even MSVC 2019 doesn't
       # support the 'aligned' attribute in the way that clang sources requires (for
cmake -G Ninja ../llvm -DCMAKE_BUILD_TYPE=Release -DFLANG_ENABLE_WERROR=On -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_TARGETS_TO_BUILD=host -DCMAKE_INSTALL_PREFIX=../install -DLLVM_LIT_ARGS=-v -DLLVM_ENABLE_PROJECTS="clang;mlir;flang" -DLLVM_ENABLE_RUNTIMES="compiler-rt"

Another Compiler Failure

With the aforementioned change, the build proceeded to a different build failure, this time in fold-integer.cpp.

FAILED: tools/flang/lib/Evaluate/CMakeFiles/obj.FortranEvaluate.dir/fold-integer.cpp.obj
C:\PROGRA~1\MIB055~1\2022\Preview\VC\Tools\MSVC\1433~1.316\bin\Hostx64\x64\cl.exe  /nologo /TP -DFLANG_LITTLE_ENDIAN=1 -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -ID:\dev\repos\llvm-project\build-vsmain\tools\flang\lib\Evaluate -ID:\dev\repos\llvm-project\flang\lib\Evaluate -ID:\dev\repos\llvm-project\flang\include -ID:\dev\repos\llvm-project\build-vsmain\tools\flang\include -ID:\dev\repos\llvm-project\build-vsmain\include -ID:\dev\repos\llvm-project\llvm\include -external:ID:\dev\repos\llvm-project\llvm\..\mlir\include -external:ID:\dev\repos\llvm-project\build-vsmain\tools\mlir\include -external:ID:\dev\repos\llvm-project\build-vsmain\tools\clang\include -external:ID:\dev\repos\llvm-project\llvm\..\clang\include -external:W0 /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4101 -wd4661 -wd4324 -w14062 -we4238 /Gw /WX /MD /O2 /Ob2  /EHs-c- /GR- -UNDEBUG -std:c++17 /showIncludes /Fotools\flang\lib\Evaluate\CMakeFiles\obj.FortranEvaluate.dir\fold-integer.cpp.obj /Fdtools\flang\lib\Evaluate\CMakeFiles\obj.FortranEvaluate.dir\ /FS -c D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-integer.cpp
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-integer.cpp(771): error C2672: 'invoke': no matching overloaded function found
C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.33.31627\include\type_traits(1552): note: could be 'unknown-type std::invoke(_Callable &&,_Ty1 &&,_Types2 &&...) noexcept(<expr>)'
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-integer.cpp(771): note: Failed to specialize function template 'unknown-type std::invoke(_Callable &&,_Ty1 &&,_Types2 &&...) noexcept(<expr>)'
C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.33.31627\include\type_traits(1552): note: see declaration of 'std::invoke'
...

By this point, I knew that simplifying the function containing the error was the fastest path to a repro. One of the little problems I ran into was how to figure out the type of fptr since it is declared using the auto keyword. I ended up assigning it to a new temporary variable of a different type, e.g. char et voila!

D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-integer.cpp(505): error C2440: 'initializing': cannot convert from 'int (__cdecl Fortran::evaluate::value::Integer<8,true,8,unsigned char,unsigned short>::* )(void) const' to 'char'

I then removed the temporary assignment and explicitly specified this type as the type of fptr:

using T2 = int (__cdecl Fortran::evaluate::value::Integer<8,true,8,unsigned char,unsigned short>::* )(void) const;

T2 fptr{&Scalar<TI>::LEADZ};

The build then failed because the function pointer types are not the same, which was really confusing given that I had just checked the type of fptr.

D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-integer.cpp(504): error C2440: 'initializing': cannot convert from
'int (__cdecl Fortran::evaluate::value::Integer<16,true,16,unsigned short,unsigned int>::* )(void) const' to
'int (__cdecl Fortran::evaluate::value::Integer<8,true,8,unsigned char,unsigned short>::* )(void) const'

I switched the type of fptr and got a different error:

D:\dev\repos\llvm-project\flang\include\flang\Evaluate\integer.h(66): error C2607: static assertion failed
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-integer.cpp(490): note: see reference to class template instantiation 'Fortran::evaluate::value::Integer<16,true,16,unsigned char,unsigned short>' being compiled

Here is a different change I tried:

using T2 = int (__cdecl Fortran::evaluate::value::Integer<8>::* )(void) const;

T2 fptr{&Scalar<TI>::LEADZ};

That still failed with the following error:

D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-integer.cpp(504): error C2440: 'initializing': cannot convert from
'int (__cdecl Fortran::evaluate::value::Integer<8,true,8,unsigned char,unsigned short>::* )(void) const' to
'int (__cdecl Fortran::evaluate::value::Integer<32,true,32,unsigned int,unsigned __int64>::* )(void) const'

It was at this point that I realized that it was time to learn a bit more about decay. What is decay and array-to-pointer conversion? | C++ FAQ (64.github.io) had a good explanation of why the term decay is used. Perhaps a reexamination of std::decay – cppreference.com might lead to some insight. I wasn’t sure what Result referred to in the statement using TI = typename std::decay_t<decltype(n)>::Result; One idea I got was to append a number to the typename and examine the compiler error. Here’s the new line 752 of llvm-project/fold-integer.cpp and the resulting compiler error showing that this name cannot be arbitrary.

using TI = typename std::decay_t<decltype(n)>::Result3;


C:\dev\repos\llvm-project\flang\lib\Evaluate\fold-integer.cpp(502): error C2039: 'Result3': is not a member of 'Fortran::evaluate::Expr<Fortran::evaluate::Type<Fortran::common::TypeCategory::Integer,1>>'

Aha, so what it was referring to is the using statement in llvm-project/expression.h!

template <int KIND>
class Expr<Type<TypeCategory::Integer, KIND>>
    : public ExpressionBase<Type<TypeCategory::Integer, KIND>> {
public:
  using Result = Type<TypeCategory::Integer, KIND>;

...

The problematic lambda is therefore expecting a Scalar<Type<TypeCategory::Integer, KIND>>. Scalar is defined using decay and Type<TypeCategory::Integer, KIND>::Scalar is defined in llvm-project/type.h as the type value::Integer<8 * KIND>. This is when I see the reason for the previous build errors about mismatched Integer sizes no matter which size I picked – the fixed type I was using didn’t allow for the different template instantiations! Note that the problematic lambda is defined as a ScalarFunc.

By this point, I had a self-contained repro of the compiler bug, which ironically, compiled successfully on the RTM C++ compiler so I could use neither the preview nor the RTM to build the flang code.

cl /c /TP /std:c++17 /permissive- flang-msvc-clang-test-02.cpp

This compiler invocation gives the same error seen when compiling the flang code:

Microsoft (R) C/C++ Optimizing Compiler Version 19.33.31627.1 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

flang-msvc-clang-test-02.cpp
flang-msvc-clang-test-02.cpp(193): error C2672: 'invoke': no matching overloaded function found
C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31721\include\type_traits(1552): note: could be 'unknown-type std::invoke(_Callable &&,_Ty1 &&,_Types2 &&...) noexcept(<expr>)'
flang-msvc-clang-test-02.cpp(193): note: Failed to specialize function template 'unknown-type std::invoke(_Callable &&,_Ty1 &&,_Types2 &&...) noexcept(<expr>)'

I ended up reporting this compiler bug via the Visual Studio feedback system – see C++17 lambda fails to compile on latest VS preview compiler.


Categories: hsdis, LLVM

hsdis LLVM backend for Windows ARM64

8253757: Add LLVM-based backend for hsdis by magicus · Pull Request #7531 makes it possible to easily use LLVM as the hsdis backend. An LLVM installation is required for this. The official LLVM builds for the Windows platform do not work for building hsdis because they do not have all the prerequisite LLVM include files. See Building LLVM for Windows ARM64 – Saint’s Log (swesonga.org) for instructions on how to build LLVM for ARM64 Windows (on an x64 Windows host). To configure OpenJDK for LLVM as an hsdis backend on Windows ARM64, use this command:

bash configure --openjdk-target=aarch64-unknown-cygwin \
 --with-hsdis=llvm \
 --with-llvm=/cygdrive/d/dev/software/llvm-aarch64/

The JDK and hsdis can then be built as usual with these commands:

make images
make build-hsdis
make install-hsdis
cp /cygdrive/d/dev/software/llvm-aarch64/bin/LLVM-C.dll build/windows-aarch64-server-slowdebug/jdk/bin/

The generated JDK can then be deployed to an ARM64 machine like the Surface Pro X. To test LLVM’s disassembly, use the -XX:CompileCommand flag on the ARM64 machine:

/java -XX:CompileCommand="print java.lang.String::checkIndex" -version

Behind the Scenes

Missing Include File that Exists?

The path given to --with-llvm needs to be a Cygwin path if building in Cygwin. Otherwise, the build-hsdis target will fail with this error: c:\...\jdk\src\utils\hsdis\llvm\hsdis-llvm.cpp(58): fatal error C1083: Cannot open include file: 'llvm-c/Disassembler.h': No such file or directory. I caught this by inspecting build\windows-aarch64-server-release\make-support\failure-logs\support_hsdis_hsdis-llvm.obj.cmdline after the build failed. This was the only include that didn’t have Cygwin paths: -IC:/dev/repos/llvm-project/build_llvm_AArch64/install_local/include

Investigating Missing Disassembly

My first disassembly attempt did not work – only abstract disassembly was displayed:

...
  # {method} {0x000002ca9940f2e8} 'checkIndex' '(II)V' in 'java/lang/String'
  # parm0:    c_rarg1   = int
  # parm1:    c_rarg2   = int
  #           [sp+0x30]  (sp of caller)
  0x000002ca87ad3940: 1f20 03d5 | e953 40d1 | 3f01 00f9 | ffc3 00d1 | fd7b 02a9 | a201 f837 | 3f00 026b | e200 0054
  0x000002ca87ad3960: fd7b 42a9 | ffc3 0091
...

I verified that hsdis-aarch64.dll was present in the JDK’s bin folder. That was the only issue I had seen before that caused this behavior so I dug around to find the code that loads the hsdis DLL. A search for the “hsdis-” DLL prefix in the sources reveals the hsdis_library_name string used in the Disassembler::dll_load method. Notice that there is a Verbose flag that can display what is happening when loading the hsdis DLL!

void* Disassembler::dll_load(char* buf, int buflen, int offset, char* ebuf, int ebuflen, outputStream* st) {
  int sz = buflen - offset;
  int written = jio_snprintf(&buf[offset], sz, "%s%s", hsdis_library_name, os::dll_file_extension());
  if (written < sz) { // written successfully, not truncated.
    if (Verbose) st->print_cr("Trying to load: %s", buf);
    return os::dll_load(buf, ebuf, ebuflen);
  } else if (Verbose) {
    st->print_cr("Try to load hsdis library failed: the length of path is beyond the OS limit");
  }
  return NULL;
}

This turns out to be a JVM flag! I try passing it to java.exe but -Verbose doesn’t do anything. Learn from HotSpot Command-Line Flags Overhaul – Design Doc – OpenJDK Wiki (java.net) that it’s a -XX: flag. Trying to use it causes JVM to complain that it is a develop-only flag.

Error: VM option 'Verbose' is develop and is available only in debug version of VM.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

The –enable-debug flag documented at Building OpenJDK (java.net) is required to generate a debug VM.

bash configure --openjdk-target=aarch64-unknown-cygwin \
 --with-hsdis=llvm \
 --with-llvm=/cygdrive/d/dev/software/llvm-aarch64/ \
 --enable-debug

Running the debug JVM with the verbose flag now displays some diagnostic information:

.\java.exe -XX:CompileCommand="print java.lang.String::checkIndex" -XX:+Verbose -XX:+PrintMiscellaneous
CompileCommand: print java/lang/String.checkIndex bool print = true


============================= C1-compiled nmethod ==============================
----------------------------------- Assembly -----------------------------------
Trying to load: C:\dev\software\jdk-aarch64\jdk\bin\server\hsdis-aarch64.dll
Trying to load: C:\dev\software\jdk-aarch64\jdk\bin\server\hsdis-aarch64.dll
Trying to load: C:\dev\software\jdk-aarch64\jdk\bin\hsdis-aarch64.dll
Trying to load: hsdis-aarch64.dll
Could not load hsdis-aarch64.dll; Can't find dependent libraries; PrintAssembly defaults to abstract disassembly.

...

The error message substring “find dependent libraries” appears only once in the hotspot source code – in os::dll_load (which is called by Disassembler::dll_load). This error is displayed because LoadLibrary returns ERROR_MOD_NOT_FOUND.

The case of the DLL that refuses to load – The Old New Thing (microsoft.com) mentions loader snaps. The loader snaps are an option for the gflags tool found in the Windows Kits folder. The docs explain that GFlags is included in the Debugging Tools for Windows 10 (WinDbg) so a search for “Debugging Tools for Windows arm64” leads to Debugging ARM64 – Windows drivers. This says to install the Windows SDK, after which I now have the gflags binary (in the x86 folder)!

C:\Program Files (x86)\Windows Kits\10\Debuggers\arm64\gflags.exe

I still wasn’t sure how to see the snaps output. Show Loader Snaps in GFlags.exe, fails to capture any output in WinDbg – Stack Overflow implies that I should be able to use WinDbg to see what is failing to load.

Turns out the loader snaps aren’t really necessary. There is some critical info in the WinDbg diagnostic output:

2698:2e5c @ 03908953 - LdrpResolveDllName - ENTER: DLL name: .\LLVM-C.dll
2698:2e5c @ 03908953 - LdrpResolveDllName - RETURN: Status: 0xc0000135
...
2698:2e5c @ 03908968 - LdrpResolveDllName - ENTER: DLL name: C:\WINDOWS\LLVM-C.dll
2698:2e5c @ 03908968 - LdrpResolveDllName - RETURN: Status: 0xc0000135
...
2698:2e5c @ 03908968 - LdrpSearchPath - RETURN: Status: 0xc0000135
2698:2e5c @ 03908968 - LdrpProcessWork - ERROR: Unable to load DLL: "LLVM-C.dll", Parent Module: "C:\dev\software\jdk-aarch64\jdk\bin\hsdis-aarch64.dll", Status: 0xc0000135

hsdis-aarch64.dll is not being loaded because LLVM-C.dll cannot be found! Still learning the need for reading the full instructions to avoid unnecessary pain.


Categories: Compilers, LLVM

Building LLVM for Windows ARM64

I was trying to test using LLVM as a backend for hsdis on the Windows ARM64 platform as implemented in PR 5920. I downloaded LLVM 13 and tried to use it in the build. Unfortunately, it didn’t have all the prerequisite include files and so building your own LLVM installation was the approach suggested for Windows. This post explicitly outlines the instructions needed to build LLVM for the Windows ARM64 platform on a Windows x64 host machine.

The first requirement is an LLVM build with native llvm-nm.exe and llvm-tblgen.exe binaries. These can be downloaded (I think) or generated by building LLVM for the native x64 platform as specified in the instructions from Jorn.

git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build_llvm
cd build_llvm
cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=X86" -D"CMAKE_BUILD_TYPE:STRING=Release" -D"CMAKE_INSTALL_PREFIX=install_local" -A x64 -T host=x64
cmake --build . --config Release --target install

Once that build successfully completes, we can then build LLVM for the Windows ARM64 platform with the commands below. Notice that we specify paths to the native llvm-nm and llvm-tblgen binaries to prevent the build from trying to use their ARM64 equivalents (which won’t run on the host).

cd llvm-project
mkdir build_llvm_AArch64
cd build_llvm_Aarch64

cmake ../llvm -DLLVM_TARGETS_TO_BUILD:STRING=AArch64 \
 -DCMAKE_BUILD_TYPE:STRING=Release \
 -DCMAKE_INSTALL_PREFIX=install_local \
 -DCMAKE_CROSSCOMPILING=True \
 -DLLVM_TARGET_ARCH=AArch64 \
 -DLLVM_NM=C:/repos/llvm-project/build_llvm/install_local/bin/llvm-nm.exe \
 -DLLVM_TABLEGEN=C:/repos/llvm-project/build_llvm/install_local/bin/llvm-tblgen.exe \
 -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-win32-msvc \
 -A ARM64 \
 -T host=x64

date; time \
 cmake --build . --config Release --target install ; \
 date

Once the build completes, the LLVM ARM64 files will be in the build_llvm_AArch64/install_local folder in the llvm-project repo. That build should have all the necessary header files and static libraries required for LLVM projects targeting Windows on ARM64. See the general cmake options and the LLVM-specific cmake options for details on the various flags and variables.

Behind the Scenes: Cross-Compiling LLVM

I naively started out by adding AArch64 to the list of LLVM_TARGETS_TO_BUILD, then using only AArch64 in the list. Trying to use the generated build would still fail with errors about mismatched platforms so I knew some cross compilation specific flags would be needed. How do I cross-compile LLVM/Clang for AArch64 on x64 host? – Stack Overflow and How To Cross-Compile Clang/LLVM using Clang/LLVM — LLVM 15.0.0git documentation were handy references. They didn’t have anything windows specific but got me walking down the right path (e.g. the importance of the native LLVM_TABLEGEN). I tried something along these lines:

cd llvm-project
mkdir build_llvm_AArch64
cd build_llvm_AArch64

cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=AArch64" \
 -D"CMAKE_BUILD_TYPE:STRING=Release" \
 -D"CMAKE_INSTALL_PREFIX=install_local_AArch64" \
 -D"CMAKE_CROSSCOMPILING=True" \
 -D"LLVM_TARGET_ARCH=AArch64" \
 -A x64 \
 -T host=x64

cmake --build . --config Release --target install

This still results in errors about conflicting machine types:

c:\...\llvm-project\build_llvm_aarch64\install_local_aarch64\\lib\llvmaarch64disassembler.lib : warning LNK4272: library machine type 'x64' conflicts with target machine type 'ARM64'

That’s when I tried adding the LLVM_TABLE_GEN from a Windows x64 LLVM build I had generated earlier. I accidentally omitted the options prefixed with a # below because I didn’t include the trailing slash after adding the llvm-tblgen.exe option.

cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=AArch64" \
 -D"CMAKE_BUILD_TYPE:STRING=Release" \
 -D"CMAKE_INSTALL_PREFIX=install_local_AArch64_2" \
 -D"CMAKE_CROSSCOMPILING=True" \
 -D"LLVM_TARGET_ARCH=AArch64" \
 -D"LLVM_TABLEGEN=C:\dev\repos\llvm-project\build_llvm\install_local\bin\llvm-tblgen.exe" \
 #-D"LLVM_DEFAULT_TARGET_TRIPLE=aarch64-win32-msvc" \
 #-A x64 \
 #-T host=x64

date; time \
 cmake --build . --config Release --target install; \
 date

The build still succeeded and generated AArch64 .lib files in the LLVM installation! Interestingly, they still had the x64 machine type in the header.

$ dumpbin /headers build_llvm_AArch64_2\install_local_AArch64_2\lib\LLVMAArch64AsmParser.lib
Microsoft (R) COFF/PE Dumper Version 14.29.30133.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file build_llvm_AArch64_2\install_local_AArch64_2\lib\LLVMAArch64AsmParser.lib

File Type: LIBRARY

FILE HEADER VALUES
            8664 machine (x64)
...

I had no choice but to reexamine my understanding of what the -A flag does. It is used to specify the platform name but it’s only after digging into the CMAKE_GENERATOR_PLATFORM docs that I noticed that this was the target platform! This also made me realize that I hadn’t noticed that the x64 C++ compiler was being used all along!

-- The C compiler identification is MSVC 19.29.30133.0
-- The CXX compiler identification is MSVC 19.29.30133.0
-- The ASM compiler identification is MSVC
-- Found assembler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - works

Some references to LLVM triples led back to the clang cross-compilation docs and the llvm::Triple source code so I tried again with the triple set and with -A now set to AArch64.

cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=AArch64" \
 -D"CMAKE_BUILD_TYPE:STRING=Release" \
 -D"CMAKE_INSTALL_PREFIX=install_local_AArch64_3" \
 -D"CMAKE_CROSSCOMPILING=True" \
 -D"LLVM_TARGET_ARCH=AArch64" \
 -D"LLVM_TABLEGEN=C:\dev\repos\llvm-project\build_llvm\install_local\bin\llvm-tblgen.exe" \
 -D"LLVM_DEFAULT_TARGET_TRIPLE=aarch64-win32-msvc" \
 -A AArch64 \
 -T host=x64

Setting -A to AArch64 causes MSBuild to fail with an error about an unknown platform. So -A just might be the argument I need to get ARM64 libraries built.

"C:\dev\repos\llvm-project\build_llvm_AArch64_3\CMakeFiles\3.17.3\VCTargetsPath.vcxproj" (default target) (1) ->
    (_CheckForInvalidConfigurationAndPlatform target) ->
      C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(820,5): error : The BaseOutputPath/OutputPath property is not set for project 'VCTargetsPath.vcxproj'.  Please check to make sure that you have specified a valid combination of Configuration and Platform for this project.  Configuration='Debug'  Platform='AArch64'.  You may be seeing this message because you are trying to build a project without a solution file, and have specified a non-default Configuration or Platform that doesn't exist for this project. [C:\dev\repos\llvm-project\build_llvm_AArch64_3\CMakeFiles\3.17.3\VCTargetsPath.vcxproj]

So I tried using -A ARM64 instead. I noticed that we now have the ARM64 C++ compiler selected! This is something I should have been paying attention to from the beginning, crucial for cross-compilation.

-- The C compiler identification is MSVC 19.29.30133.0
-- The CXX compiler identification is MSVC 19.29.30133.0
-- The ASM compiler identification is MSVC
-- Found assembler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe - works

Unfortunately, the build still failed with an error from gen-msvc-exports.py. Taking a look at gen-msvc-exports.py, it looks like it is trying to run llvm-nm.exe (for the target platform).

  Generating export list for LLVM-C
  Traceback (most recent call last):
    File "C:/dev/repos/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 116, in <module>
      main()
    File "C:/dev/repos/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 112, in main
      gen_llvm_c_export(ns.output, ns.underscore, libs, ns.nm)
    File "C:/dev/repos/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 72, in gen_llvm_c_export
      check_call([nm, '-g', lib], stdout=dumpout_f)
    File "C:\dev\tools\Anaconda3\lib\subprocess.py", line 359, in check_call
      retcode = call(*popenargs, **kwargs)
    File "C:\dev\tools\Anaconda3\lib\subprocess.py", line 340, in call
      with Popen(*popenargs, **kwargs) as p:
    File "C:\dev\tools\Anaconda3\lib\subprocess.py", line 854, in __init__
      self._execute_child(args, executable, preexec_fn, close_fds,
    File "C:\dev\tools\Anaconda3\lib\subprocess.py", line 1307, in _execute_child
      hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
  OSError: [WinError 216] This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(241,5): error MSB8066: Custom build for 'C:\dev\repos\llvm-project\build_llvm_AArch64_3\CMakeFi
les\02a88fa656bb9cf8b9ffd0e0debe57ae\libllvm-c.exports.rule;C:\dev\repos\llvm-project\build_llvm_AArch64_3\CMakeFiles\8ebc0efbf04134b25d0f37561fba0d55\LLVM-C.def.rule;C:\dev\repos\llvm-project\build_llvm_AArch64_
3\CMakeFiles\509fcb3f8bb132e9c560e15e8d25cb45\LLVM-C_exports.rule;C:\dev\repos\llvm-project\llvm\tools\llvm-shlib\CMakeLists.txt' exited with code 1. [C:\dev\repos\llvm-project\build_llvm_AArch64_3\tools\llvm-shl
ib\LLVM-C_exports.vcxproj]

A quick search for the general message (Generating export list for LLVM-C) reveals that it is from llvm-shlib/CMakeLists.txt. Looks like we just need to set LLVM_NM as per llvm-shlib/CMakeLists.txt.

date; time cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=AArch64" \
 -D"CMAKE_BUILD_TYPE:STRING=Release" \
 -D"CMAKE_INSTALL_PREFIX=install_local" \
 -D"CMAKE_CROSSCOMPILING=True" \
 -D"LLVM_TARGET_ARCH=AArch64" \
 -D"LLVM_NM=C:\dev\repos\llvm-project\build_llvm\install_local\bin\llvm-nm.exe" \
 -D"LLVM_TABLEGEN=C:\dev\repos\llvm-project\build_llvm\install_local\bin\llvm-tblgen.exe" \
 -D"LLVM_DEFAULT_TARGET_TRIPLE=aarch64-win32-msvc" \
 -A ARM64 \
 -T host=x64

date; time \
 cmake --build . --config Release --target install; \
 date

These build commands work! Dumpbin shows that the generated .lib files have ARM64 headers!

$ dumpbin /headers C:\dev\repos\llvm-project\build_llvm_AArch64_3\Release\lib\LLVMAArch64Disassembler.lib
Microsoft (R) COFF/PE Dumper Version 14.29.30133.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file C:\dev\repos\llvm-project\build_llvm_AArch64_3\Release\lib\LLVMAArch64Disassembler.lib

File Type: LIBRARY

FILE HEADER VALUES
            AA64 machine (ARM64)

Unfortunately, the JDK project that got me started down this path still doesn’t build. Cygwin shows defines like -DLLVM_DEFAULT_TRIPLET='”aarch64-pc-windows-msvc”‘ being passed to the compiler, which then complains:

C:/.../src/utils/hsdis/llvm/hsdis-llvm.cpp(217): error C2015: too many characters in constant

The quotes in the commands therefore needed to be dropped. This caused build failures since the paths used back-slashes!

Building Opts.inc...
  'C:devreposllvm-projectbuild_llvminstall_localbinllvm-tblgen.exe' is not recognized as an internal or external command,
  operable program or batch file

This is now the part where I find a nice document on the LLVM site with the 3-liner for this task 😀