The build_llvm_release.bat script in the LLVM repo is used to create a release build for Windows. Running it without any arguments displays a sample command line to use to build LLVM.
cd \repos && mkdir llvmcd \repos\llvm\llvm-project\llvm\utils\release
C:\> cd \repos\llvm\llvm-project\llvm\utils\release
C:\repos\llvm\llvm-project\llvm\utils\release> build_llvm_release.bat
--version option is required
=============================
Script for building the LLVM installer on Windows,
used for the releases at https://github.com/llvm/llvm-project/releases
Usage: build_llvm_release.bat --version <version> [--x86,--x64, --arm64]
Options:
--version: [required] version to build
--help: display this help
--x86: build and test x86 variant
--x64: build and test x64 variant
--arm64: build and test arm64 variant
Note: At least one variant to build is required.
Example: build_llvm_release.bat --version 15.0.0 --x86 --x64
Let us build version 17.0.1, which is the latest LLVM release. Open an administrator Developer Command Prompt then run:
:: set PATH=%PATH%;C:\Program Files\7-Zip
build_llvm_release.bat --version 17.0.1 --x64
The first error is caused by mv not being a standard command in the Windows command prompt. There are ways to create aliases, e.g. alias – Aliases in Windows command prompt – Stack Overflow but given that this is a batch file, why not just use the built-in function? I fix this in the script.
C:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_17.0.1> mv llvm-project-* llvm-project || exit /b 1
'mv' is not recognized as an internal or external command,
operable program or batch file.
The build directory needs to be deleted before restarting the build with the fix. Otherwise, the script will fail.
However, from a new command window, it looks like it hasn’t been set up.
C:\repos> python --version
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.
Just running python loads the Microsoft Store:
I proceed with the Customize Installation option in the open Python installer, this time selecting “for all users (requires admin privileges)”.
I’m not sure why the “Install Python 3.11 for all users” option is not checked on the “Advanced Options” page.
Opening a new command prompt then running python still gives the same behavior as before. where python shows that the new installation appears 2nd.
C:\repos> where python
C:\Users\saint\AppData\Local\Microsoft\WindowsApps\python.exe
C:\Python311\python.exe
The PATH environment variable has these paths in reverse order so I don’t understand what is happening. Looks like the easiest way forward is to just install python from the Microsoft Store – this at least ensures that python scripts can run. After a failure due to the python command not being found, closer inspection of build_llvm_release.bat reveals that it needs the PYTHON_HOME environment variable to be set. However, that is set on line 358 using the python_dir variable, which is in turn the first argument to :set_environment. The call to set_environment is from :do_build_32 (or do_build_64), which uses a hard-coded python path! The script should first validate the python installation and exit early if it cannot be found! I should have reported/fixed these when I worked on Tracking Down Missing Headers in LLVM for Windows – Saint’s Log (swesonga.org) last year.
Build Errors
With the Python path fixed, the build proceeds but the C++ compilation fails. A few months ago I got the error below build version 15.0.7. Unfortunately, I didn’t explicitly note the compiler version.
FAILED: tools/clang/lib/Analysis/FlowSensitive/CMakeFiles/obj.clangAnalysisFlowSensitive.dir/HTMLLogger.cpp.obj
C:\PROGRA~1\MICROS~3\2022\COMMUN~1\VC\Tools\MSVC\1438~1.331\bin\Hostx64\x64\cl.exe /nologo /TP ... -c C:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_17.0.1\llvm-project\clang\lib\Analysis\FlowSensitive\HTMLLogger.cpp
C:\repos\llvm\llvm-project\llvm\utils\release\llvm_package_17.0.1\llvm-project\clang\lib\Analysis\FlowSensitive\HTMLLogger.cpp(332): fatal error C1001: Internal compiler error.
(compiler file 'msc1.cpp', line 1587)
To work around this problem, try simplifying or changing the program near the locations listed above.
If possible please provide a repro here: https://developercommunity.visualstudio.com
Please choose the Technical Support command on the Visual C++
Help menu, or open the Technical Support help file for more information
Additional troubleshooting is obviously required to make progress here. I might try to create a narrowed down testcase for the internal compiler error or perhaps just report it as is and let the Visual C++ team figure that out.
Note: I wrote this in October 2022 but never got around to posting it. Doing so now to enable comparisons with the latest sources and compilers.
In Failing to Build Flang with Visual C++, the C++ compiler had bugs that prevented compilation from succeeding. I just tried a new version of the Visual C++ compiler (Microsoft (R) C/C++ Optimizing Compiler Version 19.34.31823.3 for x64). This failed to build the same commit with the following error.
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-implementation.h(1585): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-implementation.h(1585): warning C4189: 'ctx': local variable is initialized but not referenced
...
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-implementation.h(1578): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-implementation.h(1578): warning C4101: 'buffer': unreferenced local variable
...
Moving that variable deeper into the if-statement addresses that issue. I end up having to suppress C4661 again.
D:\dev\repos\llvm-project\flang\include\flang\Evaluate\expression.h(101): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\include\flang\Evaluate\expression.h(101): warning C4661: 'std::optional<Fortran::evaluate::DynamicType> Fortran::evaluate::ExpressionBase<Fortran::evaluate::SomeDerived>::GetType(void) const': no suitable definition provided for explicit template instantiation request
...
C:\PROGRA~1\MIB055~1\2022\Preview\VC\Tools\MSVC\1434~1.318\bin\Hostx64\x64\cl.exe /nologo /TP -DFLANG_LITTLE_ENDIAN=1 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\unittests\Evaluate -ID:\dev\repos\llvm-project\flang\unittests\Evaluate -ID:\dev\repos\llvm-project\flang\include -ID:\dev\repos\llvm-project\build-nowarn\tools\flang\include -ID:\dev\repos\llvm-project\build-nowarn\include -ID:\dev\repos\llvm-project\llvm\include -external:ID:\dev\repos\llvm-project\llvm\..\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\mlir\include -external:ID:\dev\repos\llvm-project\build-nowarn\tools\clang\include -external:ID:\dev\repos\llvm-project\llvm\..\clang\include -external:W0 /DWIN32 /D_WINDOWS /Zc:inline /Zc:__cplusplus /Oi /bigobj /permissive- /W4 -wd4141 -wd4146 -wd4244 -wd4267 -wd4291 -wd4351 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4709 -wd4661 -wd4927 -wd4324 -w14062 -we4238 /Gw /WX /MD /O2 /Ob2 -UNDEBUG -std:c++17 /showIncludes /Fotools\flang\unittests\Evaluate\CMakeFiles\FortranEvaluateTesting.dir\fp-testing.cpp.obj /Fdtools\flang\unittests\Evaluate\CMakeFiles\FortranEvaluateTesting.dir\FortranEvaluateTesting.pdb /FS -c D:\dev\repos\llvm-project\flang\unittests\Evaluate\fp-testing.cpp
C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31823\include\xtree(1664): error C2220: the following warning is treated as an error
C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31823\include\xtree(1626): note: see reference to function template instantiation 'std::_Tree_node<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>,std::_Default_allocator_traits<_Alloc>::void_pointer> *std::_Tree<std::_Tmap_traits<_Kty,_Ty,_Pr,_Alloc,false>>::_Copy_nodes<std::_Tree<std::_Tmap_traits<_Kty,_Ty,_Pr,_Alloc,false>>::_Strategy::_Copy>(std::_Tree_node<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>,std::_Default_allocator_traits<_Alloc>::void_pointer> *,std::_Tree_node<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>,std::_Default_allocator_traits<_Alloc>::void_pointer> *)' being compiled
with
[
_Alloc=std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>,
_Kty=Fortran::parser::CharBlock,
_Ty=Fortran::common::ConstantSubscript,
_Pr=std::less<Fortran::parser::CharBlock>
]
...
C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31823\include\map(109): note: while compiling class template member function 'std::map<Fortran::parser::CharBlock,Fortran::common::ConstantSubscript,std::less<Fortran::parser::CharBlock>,std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>>::map(const std::map<Fortran::parser::CharBlock,Fortran::common::ConstantSubscript,std::less<Fortran::parser::CharBlock>,std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>> &)'
D:\dev\repos\llvm-project\flang\include\flang/Evaluate/common.h(243): note: see reference to function template instantiation 'std::map<Fortran::parser::CharBlock,Fortran::common::ConstantSubscript,std::less<Fortran::parser::CharBlock>,std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>>::map(const std::map<Fortran::parser::CharBlock,Fortran::common::ConstantSubscript,std::less<Fortran::parser::CharBlock>,std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>> &)' being compiled
D:\dev\repos\llvm-project\flang\include\flang/Evaluate/common.h(291): note: see reference to class template instantiation 'std::map<Fortran::parser::CharBlock,Fortran::common::ConstantSubscript,std::less<Fortran::parser::CharBlock>,std::allocator<std::pair<const Fortran::parser::CharBlock,Fortran::common::ConstantSubscript>>>' being compiled
C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31823\include\xtree(1664): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
I try setting this flag just at the end of the if(MSVC) block in HandleLLVMOptions.cmake. Unfortunately, the build still fails and that’s when I notice that the earlier command lines in this post have the /EHs-c- flag. Looks like HandleLLVMOptions.cmake is explicitly removing such flags. The next logical step is to search for /EH in the codebase to see how it’s getting set for the commands that have it. That’s when I notice clang-cl – looks like a cl.exe immitation. Maybe this is how they’ve been building flang for windows without addressing all these Visual C++ issues?
Back to the exception handling issue: searching for /EHsc leads me to AddLLVM.cmake which references the LLVM_ENABLE_EH variable. Looks like that defaults to OFF in HandleLLVMOptions.cmake and it also needs LLVM_ENABLE_RTTI to be enabled.
The fact that this still doesn’t address the warning is the red flag forcing me to notice that this is a unit test and we probably don’t want to turn on RTTI anyway. That would be a huge change! Let’s drop these flags and instead modify the flang/unittests/CMakeLists.txt to add the -EHsc flag. That turns out to be exactly the right fix! Building with this command line takes about 1hr 55min on my desktop.
At this point, the commit I’ve been using and the Visual Studio build (17.4.0 Preview 3 with C++ compiler version 19.34.31823.3) are outdated. Let’s see what happens with the latest code and Visual Studio 17.4.0 Preview 4 (with C++ compiler version 19.34.31931). Bad timing on my part though, the installer now says Preview 5.0 is available and it comes with Microsoft (R) C/C++ Optimizing Compiler Version 19.34.31932 for x86. Not sure why these new preview builds are showing the x86 host when launching the developer command prompt from the start menu. Doesn’t happen when using the drop down in Windows Terminal.
llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\try_lock_shared_for.pass.cpp - The system cannot find the path specified.
llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\try_lock_shared_until.pass.cpp - The system cannot find the path specified.
llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\try_lock_until_deadlock_bug.pass.cpp - The system cannot find the path specified.
These files still exist on disk though! They are displayed if you dir their containing directory but are not found if you dir their full paths! They cannot be deleted using del either. Interestingly, pressing tab after the directory path will autocomplete the file names.
C:\> dir D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\
Volume in drive D is DATAVOL1
Volume Serial Number is 8800-8693
Directory of D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class
10/16/2022 01:22 PM <DIR> .
10/04/2022 03:29 AM <DIR> ..
10/04/2022 03:29 AM 2,461 try_lock_shared_for.pass.cpp
10/04/2022 03:29 AM 2,423 try_lock_shared_until.pass.cpp
10/04/2022 03:29 AM 2,146 try_lock_until_deadlock_bug.pass.cpp
3 File(s) 7,030 bytes
C:\> dir D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\try_lock_shared_for.pass.cpp
Volume in drive D is DATADRIVE1
Volume Serial Number is 548C-FFC9
Directory of D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class
File Not Found
C:\> del D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\try_lock_shared_for.pass.cpp
The system cannot find the path specified.
These files can be viewed in file explorer. Something that caught my eye when examining their properties is that their locations started with the \\?\ prefix! That seems unusual for files on my local drive.
File Locations Starting with \\?\
Resource monitor does not show any images with associated handles when searching for “try_lock”. Neither does searching for “\?\D:\dev\repos\llvm\”. I tried using Process Explorer’s “Find Handle or DLL…” command as well. There also don’t appear to be any child processes for the cmd.exe process I was using (a Developer 2019 Command Prompt).
Next idea, open Process Monitor and see what’s happening when dir and rmdir are executed. I used the Path contains thread.sharedtimedmutex.class filter. The deletes are showing up as SetDispositionInformationFile events and seem to be using the RemoveDirectoryW function.
The RemoveDirectory function marks a directory for deletion on close. Therefore, the directory is not removed until the last handle to the directory is closed.
Notice the NOT EMPTY result of the SetDispositionInformationFile event. I believe this comes from RemoveDirectoryW. There’s the question of how the 3 files are printed to the command line. The FindNextFile API is used to search for files.
Ah, in the middle of this investigation, PowerShell.exe dies and so does Windows Terminal. All my tabs, everything, gone! Aaaargh… Windows Event Viewer has an Information level event showing that powershell.exe crashed due to a System.InvalidOperationException. This is then followed by another Information event with the WER source and P1 problem signature Microsoft.WindowsTerminal_1.15.2713.0_x64__8wekyb3d8bbwe. Then comes the Error level event with the Application Hang source and General explanation that “The program WindowsTerminal.exe version 1.15.2209.28003 stopped interacting with Windows and was closed. To see if more information about the problem is available, check the problem history in the Security and Maintenance control panel.” The ExeFileName is cut off below but simply append “\WindowsTerminal.exe” to the package name to reconstruct it. Looks like I need to avoid PowerShell. And why is there no crash dump created for it???
One upside of this crash is that it lets me confirm that it is not the cmd.exe process that is hanging onto those files. I terminate explorer.exe and when I run new task in Task Manager, it asks me to create a Windows Hello pin. What is happening?? Now moving on to opening these files: Notepad++ acts as though nothing happened when you File->Open and select one of them. Notepad opens it though! Running cat in Git Bash also dumps its contents:
cat /d/dev/repos/llvm/dups/llvm-project/llvm/utils/release/llvm_package_15.0.2/llvm-project-llvmorg-15.0.2/libcxx/test/std/thread/thread.mutex/thread.mutex.requirements/thread.sharedtimedmutex.requirements/thread.sharedtimedmutex.class/try_lock_shared_for.pass.cpp
//===----------------------------------------------------------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
...
I’m suspecting Windows Defender but don’t have any definitive proof. According to Enable attack surface reduction (ASR) rules, this command in an admin powershell should do the trick if the problem was Windows Defender:
Well, looks like rm from Git Bash works just fine as does deleting from file explorer. Unfortunately, this unsolved issue is probably going to continue to cause pain in command prompt batch files like the LLVM build script.
hsdis is a plugin for disassembling code dynamically generated by the Java Virtual Machine. On Linux & MacOS, it uses GNU binutils. Support for the LLVM disassembly backend was recently added to hsdis in https://github.com/openjdk/jdk/pull/7531. This was motivated by the fact that GNU binutils is not distributed with the JDK (due to licensing reasons mentioned at https://github.com/openjdk/jdk/pull/5920#issuecomment-942398786) and the LLVM disassembly may be preferrable in certain circumstances. Unfortunately, the official Windows LLVM distribution does not have the header files necessary to build the hotspot disassembler. This prevents Windows developers from easily using the LLVM disassembler backend because they now have to build LLVM themselves as well – see hsdis LLVM backend for Windows ARM64 and Building LLVM for Windows ARM64, for example. In this post, we investigate why the LLVM Windows build does not have the necessary header files. The llvm-c directory in Windows build contains these 2 files only:
C:\Program Files\LLVM\include\llvm-c>dir
Volume in drive C is OSDisk
Volume Serial Number is c070-2ac0
Directory of C:\Program Files\LLVM\include\llvm-c
01/08/2022 11:54 AM <DIR> .
01/08/2022 11:54 AM <DIR> ..
09/24/2021 10:18 AM 29,760 lto.h
09/24/2021 10:18 AM 9,632 Remarks.h
2 File(s) 39,392 bytes
2 Dir(s) 62,273,200,128 bytes free
I created a local LLVM build (see Building LLVM with CMake) and confirmed that it has all the header files.
C:\dev\repos\llvm-project\build_llvm\install_local\include\llvm-c>dir /w
Volume in drive C is OSDisk
Volume Serial Number is 0087-4c48
Directory of C:\dev\repos\llvm-project\build_llvm\install_local\include\llvm-c
[.] [..] Analysis.h
BitReader.h BitWriter.h blake3.h
Comdat.h Core.h DataTypes.h
DebugInfo.h Deprecated.h Disassembler.h
DisassemblerTypes.h Error.h ErrorHandling.h
ExecutionEngine.h ExternC.h Initialization.h
IRReader.h Linker.h LLJIT.h
lto.h Object.h Orc.h
OrcEE.h Remarks.h Support.h
Target.h TargetMachine.h [Transforms]
Types.h
28 File(s) 382,361 bytes
3 Dir(s) 59,158,138,880 bytes free
Does this problem still exist in the latest Windows LLVM release? I went to Releases · llvm/llvm-project (github.com) to find the latest LLVM installer for Windows but couldn’t find it. Turns out it’s because the 15.0.1 release is only 14 hours old so some of the assets probably haven’t been uploaded. Notice that 15.0.0 has 47 assets. I can successfully download and install LLVM-15.0.0-win64.exe and see that the header files are still missing.
Interestingly, trying to install LLVM-15.0.0-win32.exe before uninstalling LLVM-15.0.0-win64.exe gives this dialog and clicking Yes uninstalls before the actual installation of the 32-bit build starts!
LLVM is already installed.
I assumed that would happen at this stage:
All the same, these dialogs have strings that can lead us to the sources that create the installer! The installer looks very similar to the one from Building the Elmer Install Folder so searching the llvm codebase for “ncis ” gives only a handful of hits leading to the key discovery of build_llvm_release.bat! (later learn that this needs to be executed in a (2019) developer command prompt so that the ninja command can be found). That script requires 7zip though. The script fails on my machine because it can’t find 7zip. Failure seems to be coming from the for-statement (see for | Microsoft Learn for usage). The for command uses the escape character (^) as explained at set | Microsoft Learn.
C:\dev\repos\llvm-project\llvm\utils\release> build_llvm_release.bat 15.0.0
Check 7-zip version and/or administrator permissions.
'7z.exe' is not recognized as an internal or external command,
operable program or batch file.
You need to modify the paths below:
Revision: llvmorg-15.0.0
Package version: 15.0.0
Build dir: C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0
Press any key to continue . . .
-- Looking for CrashReporterClient.h
-- Looking for CrashReporterClient.h - not found
-- Looking for pfm_initialize in pfm
-- Looking for pfm_initialize in pfm - not found
-- Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)
CMake Error at C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find LibXml2 (missing: LIBXML2_INCLUDE_DIR)
Call Stack (most recent call first):
C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/FindLibXml2.cmake:108 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
cmake/config-ix.cmake:156 (find_package)
CMakeLists.txt:774 (include)
-- Configuring incomplete, errors occurred!
See also "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/CMakeFiles/CMakeOutput.log".
See also "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/CMakeFiles/CMakeError.log".
Looking through FindPackageHandleStandardArgs.cmake leads me to the simple realization that the wrong define is being used on the command line. Could this be because I’m using a newer CMake? I’ve been using the VS 2022 Preview Developer Command Prompt thus far. My VS 2019 (16.11.19) installation uses CMake 3.20. Both FindLibXml2.cmake in 3.20 and FindLibXml2.cmake in 3.24 require the LIBXML2_INCLUDE_DIR variable. However, they also claim (at the top) to set these variables.
in my build folder (build_llvm), there is a CPackConfig.cmake file that sets variables like CPACK_PACKAGE_FILE_NAME and CPACK_NSIS_DISPLAY_NAME. Since it is NSIS Wiki (sourceforge.io) in use, I wonder about running the package target myself in a manner similar to that used to create my local build. I switch back to a previous build directory (created without the build_llvm_release.bat) and run:
cmake --build . --config Release --target package
The resulting failure below indicates that NSIS is required.
MSBuild version 17.4.0-preview-22466-03+48ab5664b for .NET Framework
PipSqueak.vcxproj -> C:\dev\repos\llvm-project\build_llvm\unittests\Support\DynamicLibrary\Release\PipSqueak.dll
SecondLib.vcxproj -> C:\dev\repos\llvm-project\build_llvm\unittests\Support\DynamicLibrary\Release\SecondLib.dll
obj.llvm-tblgen.vcxproj -> C:\dev\repos\llvm-project\build_llvm\utils\TableGen\obj.llvm-tblgen.dir\Release\obj.llvm-tblgen.lib
LLVMDemangle.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\lib\LLVMDemangle.lib
...
verify-uselistorder.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\verify-uselistorder.exe
yaml-bench.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\yaml-bench.exe
yaml2obj.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\yaml2obj.exe
EXEC : CPack error : Cannot find NSIS compiler makensis: likely it is not installed, or not in your PATH [C:\dev\repos\llvm-project\build_llvm\package.vcxproj]
EXEC : CPack error : Could not read NSIS registry value. This is usually caused by NSIS not being installed. Please install NSIS from http://nsis.sourceforge.net [C:\dev\repos\llvm-proje
ct\build_llvm\package.vcxproj]
EXEC : CPack error : Cannot initialize the generator NSIS [C:\dev\repos\llvm-project\build_llvm\package.vcxproj]
After installing NSIS, the previous command successfully creates an LLVM for Windows installer.
...
verify-uselistorder.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\verify-uselistorder.exe
yaml-bench.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\yaml-bench.exe
yaml2obj.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\yaml2obj.exe
CPack: Create package using NSIS
CPack: Install projects
CPack: - Install project: LLVM [Release]
CMake Warning (dev) at C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/GNUInstallDirs.cmake:243 (messa
ge):
Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
target architecture is known. Please enable at least one language before
including GNUInstallDirs.
Call Stack (most recent call first):
C:/dev/repos/llvm-project/llvm/cmake/modules/LLVMInstallSymlink.cmake:5 (include)
C:/dev/repos/llvm-project/build_llvm/tools/llvm-ar/cmake_install.cmake:48 (include)
C:/dev/repos/llvm-project/build_llvm/tools/cmake_install.cmake:39 (include)
C:/dev/repos/llvm-project/build_llvm/cmake_install.cmake:71 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
CPack: Create package
CPack: - package: C:/dev/repos/llvm-project/build_llvm/LLVM-16.0.0git-win64.exe generated.
This installer generates the LLVM includes on disk as expected. The issue must therefore be confined to the installer generated by the script.
Reviewing Ninja NSIS Packaging
At this point, I ran build_llvm_release.bat to create an installer. Once packaging is complete, the install_manifest.txt file can be used to determine which files are in the installer. The batch file also runs lots of tests and this was annoying when trying to generate installers. Once the tests failed on the build I was creating and I had CTRL+C’d a couple of times, I ran ninja package myself (taken from the batch file)
C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0>ninja package
[0/1] Run CPack packaging tool...CPack: Create package using NSIS
CPack: Install projects
CPack: - Install project: LLVM []
CMake Warning (dev) at C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/GNUInstallDirs.cmake:243 (message):
Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
target architecture is known. Please enable at least one language before
including GNUInstallDirs.
Call Stack (most recent call first):
C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/llvm-project/llvm/cmake/modules/LLVMInstallSymlink.cmake:5 (include)
C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/tools/llvm-ar/cmake_install.cmake:40 (include)
C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/tools/cmake_install.cmake:39 (include)
C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/cmake_install.cmake:114 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
CPack: Create package
CPack: - package: C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/LLVM-15.0.0-win32.exe generated.
I then try to find a package target in build.ninja. Search for CMakeFiles\package.util.+ include since we’re interested in include files. There are some interesting differences in the include directories of the build created manually from the local install and the one created by the script, e.g.
Try searching in build.ninja for the 2 header files the installer creates in the (broken) shipping LLVM for Windows build. Nothing there but searching the file system for remarks.h gives interesting results, e.g. the existence of an NSIS project file: project.nsi. Looks like there are some tutorials showing how to create .nsi files at Invoking NSIS run-time commands on compile-time – NSIS (sourceforge.io). The way NSIS is used with CPack when building is documented at Packaging With CPack — Mastering CMake
… but I completely missed the fact that the 2nd didn’t have these lines from the first.
if(CMAKE_INSTALL_COMPONENT STREQUAL "llvm-headers" OR NOT CMAKE_INSTALL_COMPONENT)
file(INSTALL DESTINATION "${CMAKE_INSTALL_PREFIX}/include" TYPE DIRECTORY FILES
"C:/dev/repos/llvm-project/llvm/include/llvm"
"C:/dev/repos/llvm-project/llvm/include/llvm-c"
FILES_MATCHING REGEX "/[^/]*\\.def$" REGEX "/[^/]*\\.h$" REGEX "/[^/]*\\.td$" REGEX "/[^/]*\\.inc$" REGEX "/license\\.txt$")
endif()
if(CMAKE_INSTALL_COMPONENT STREQUAL "llvm-headers" OR NOT CMAKE_INSTALL_COMPONENT)
file(INSTALL DESTINATION "${CMAKE_INSTALL_PREFIX}/include" TYPE DIRECTORY FILES
"C:/dev/repos/llvm-project/build_llvm/include/llvm"
"C:/dev/repos/llvm-project/build_llvm/include/llvm-c"
FILES_MATCHING REGEX "/[^/]*\\.def$" REGEX "/[^/]*\\.h$" REGEX "/[^/]*\\.gen$" REGEX "/[^/]*\\.inc$" REGEX "/cmakefiles$" EXCLUDE REGEX "/config\\.h$" EXCLUDE)
endif()
Search the codebase for “llvm-headers” and find the llvm-header component definition. That whole code block is gated by the LLVM_INSTALL_TOOLCHAIN_ONLY variable! This is explicitly turned off in build_llvm_release.bat! I rerun the batch file and see tests failing after the build succeeds. CTRL+C to kill the processes so that I can get to the root issue: does turning off that flag fix the includes? makensis fails, probably because I killed the build and some things might still have been in use?
C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0>ninja package
[0/1] Run CPack packaging tool...CPack: Create package using NSIS
CPack: Install projects
CPack: - Install project: LLVM []
CMake Warning (dev) at C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/GNUInstallDirs.cmake:243 (message):
Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
target architecture is known. Please enable at least one language before
including GNUInstallDirs.
Call Stack (most recent call first):
C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/llvm-project/llvm/cmake/modules/LLVMInstallSymlink.cmake:5 (include)
C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/tools/llvm-ar/cmake_install.cmake:40 (include)
C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/tools/cmake_install.cmake:39 (include)
C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/cmake_install.cmake:128 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
CPack: Create package
CPack Error: Problem running NSIS command: "C:/Program Files (x86)/NSIS/makensis.exe" "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/_CPack_Packages/win32/NSIS/project.nsi"
Please check C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/_CPack_Packages/win32/NSIS/NSISOutput.log for errors
CPack Error: Problem compressing the directory
CPack Error: Error when generating package: LLVM
FAILED: CMakeFiles/package.util
cmd.exe /C "cd /D C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0 && "C:\Program Files\Microsoft Visual Studio\2022\Preview\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cpack.exe" --config ./CPackConfig.cmake"
ninja: build stopped: subcommand failed.
NSISOutput.log failed due to an “Internal compiler error #12345: error mmapping datablock to 17235001.” However, the include files are now present in the source directory being packaged by NSIS.
Turning Off Tests
There are many tests that the build script runs and some of them are failing. Testing is not on my critical path since all I need is to generate installers so I modify the scripts to enable me to package the build without running all the tests. I then start my build without tests and go to bed only to wake up the next morning to find that I need to rerun it because there are no running programs when I log in. Event Viewer doesn’t show any reboot-related events and sure enough, Task Manager shows over 9 days of uptime still. Turns out the Desktop Window Manager crashed (C:\WINDOWS\system32\dwm.exe)! Curse you dwmcore.dll. Well, time to install those updates I’ve been putting off and reboot before jumping back in. Now on the new Windows 10.0.22621.521. The build still fails:
-- LLVM host triple: i686-pc-windows-msvc
-- LLVM default target triple: i686-pc-windows-msvc
-- Using Release VC++ CRT: MD
-- Looking for os_signpost_interval_begin
-- Looking for os_signpost_interval_begin - not found
CMake Error at C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find Python3 (missing: Python3_EXECUTABLE Interpreter) (Required
is at least version "3.6")
Reason given by package:
Interpreter: Cannot use the interpreter "C:/Python310/python.exe"
Call Stack (most recent call first):
C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPython/Support.cmake:3165 (find_package_handle_standard_args)
C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPython3.cmake:485 (include)
CMakeLists.txt:817 (find_package)
-- Configuring incomplete, errors occurred!
See also "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/CMakeFiles/CMakeOutput.log".
See also "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/CMakeFiles/CMakeError.log".
When I interrupted the tests before modifying the batch file to skip them, I noticed that they were being run by %LOCALAPPDATA%\Microsoft\WindowsApps\python3.9.exe. This is still present on my machine. Ah, turns out I’m now using the 2019 developer command prompt (and therefore an older CMake). The only difference between CMake 3.20 FindPython3.cmake and CMake 3.24 FindPython3.cmake is a comment about static libraries, so this failure is a mystery.
Diagnosing Build Failures
Since this issue also bit me when I moved to my Surface Book, it is worth understanding why it happens.
Missing CMake in Visual Studio 17.3.4 Developer Command Prompt
Here is the VS 2022 Preview vs VS 2022 Enterprise path to CMake:
C:\Program Files (x86)\Microsoft Visual Studio\Installer> where cmake
C:\Program Files\Microsoft Visual Studio\2022\Preview\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe
C:\dev\repos\llvm-project\llvm\utils\release> where cmake
INFO: Could not find files for the given pattern(s).
The View Logs link opens the Documents folder under This PC – not particularly useful. Interestingly though, clicking on the Modify button shows a Total space required 1.63 GB. How is there space required before I’ve selected anything? Something similar happens with 16.11.19 though. Without making any individual component selections, I start the install process. CMake gets (re-?)installed as shown below. This fixes the setup warnings as well and cmake is now usable in the VS2022 command prompt.
Missing Python3 in VS 17.3.4 Developer Command Prompt
This is the error I got when trying to build LLVM on my Surface Book 2 in the VS 2022 developer command prompt:
CMake Error at C:/Program Files/CMake/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:164 (message):
Could NOT find Python3 (missing: Python3_EXECUTABLE Interpreter) (Required
is at least version "3.6")
Reason given by package:
Interpreter: Cannot use the interpreter "C:/Python310/python.exe"
Call Stack (most recent call first):
C:/Program Files/CMake/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:445 (_FPHSA_FAILURE_MESSAGE)
C:/Program Files/CMake/share/cmake-3.17/Modules/FindPython/Support.cmake:2437 (find_package_handle_standard_args)
C:/Program Files/CMake/share/cmake-3.17/Modules/FindPython3.cmake:309 (include)
CMakeLists.txt:817 (find_package)
Uninstalling CMake enables the command line to pick up the CMake distributed with Visual Studio. Python3 is now found successfully in the path below (I’ve shortened it using %LOCALAPPDATA%).
-- Found Python3: %LOCALAPPDATA%/Microsoft/WindowsApps/python3.8.exe (found suitable version "3.8.10", minimum required is "3.6") found components: Interpreter
Missing Python3 in VS 16.11.19 Developer Command Prompt
Interestingly, I still get the same error in VS 2019 despite uninstalling CMake 3.17. My earlier hypothesis is therefore invalid.
CMake Error at C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find Python3 (missing: Python3_EXECUTABLE Interpreter) (Required
is at least version "3.6")
Reason given by package:
Interpreter: Cannot use the interpreter "C:/Python310/python.exe"
Call Stack (most recent call first):
C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPython/Support.cmake:3165 (find_package_handle_standard_args)
C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPython3.cmake:485 (include)
CMakeLists.txt:817 (find_package)
C:/Python310/python.exe -c "import sys; sys.stdout.write('.'.join([str(x) for x in sys.version_info[:3]]))"
I comment out the ERROR_QUIET line to reveal the stdout and stderr output from python since the return code from the python process is causing the CMake error to be raised. Running with --trace-expand --trace-redirect=cmake_trace.txt now reveals the root cause (paths below cleaned up using %LOCALAPPDATA%):
Python path configuration:
PYTHONHOME = '%LOCALAPPDATA%\Programs\Python\Python310-32'
PYTHONPATH = (not set)
program name = 'C:/Python310/python.exe'
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = 'C:\\Python310\\python.exe'
sys.base_prefix = '%LOCALAPPDATA%\\Programs\\Python\\Python310-32'
sys.base_exec_prefix = '%LOCALAPPDATA%\\Programs\\Python\\Python310-32'
sys.platlibdir = 'lib'
sys.executable = 'C:\\Python310\\python.exe'
sys.prefix = '%LOCALAPPDATA%\\Programs\\Python\\Python310-32'
sys.exec_prefix = '%LOCALAPPDATA%\\Programs\\Python\\Python310-32'
sys.path = [
'C:\\Python310\\python310.zip',
'%LOCALAPPDATA%\\Programs\\Python\\Python310-32\\DLLs',
'%LOCALAPPDATA%\\Programs\\Python\\Python310-32\\lib',
'C:\\Python310',
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
Current thread 0x00003174 (most recent call first):
<no Python frame>
django – init_fs_encoding: failed to get the Python codec of the filesystem encoding – Stack Overflow is a hint that the PYTHONHOME is wrong. Sure enough, I didn’t change it in build_llvm_release.bat so the paths in the configuration above do not exist! This now raises another question: how on earth does this work in VS 2022? I notice on my desktop that python.exe does not even appear in the CMake tracing output! The difference in behavior stems from the fact that the find_program command in Modules/FindPython/Support.cmake · v3.20.0 finds python 3.10 first in the VS 2019 environment. This path is then assigned to _Python3_EXECUTABLE, preventing the 3.8 path from being used. One important difference between CMake 3.20 and 3.23 that I notice is FindPython: fix typo error (fff8d5b2) · Commits · CMake / CMake · GitLab (kitware.com). Since the fix for the build_llvm_release.bat script is straightforward and it is clear that there are some CMake implementation differences at work, we no longer need to dig into why this behavior could be happening.
Python Hangs
One of my build attempts successfully completes stage0 but hangs when CMake tries to detect the python version. Manually running the same command (copied from Process Explorer) also hangs. Even %LOCALAPPDATA%/Microsoft/WindowsApps/python3.9.exe --version hangs. Inspecting the full dump created by Task Manager reveals that python3.9.exe made a call to get (what looks like) the Package.InstalledLocation Property (Windows.ApplicationModel) – Windows UWP applications | Microsoft Learn
...
-- Looking for os_signpost_interval_begin
-- Looking for os_signpost_interval_begin - not found
Windows becomes pretty unusable as I investigate this behavior (mouse doesn’t work, changes program in focus but can’t click on anything). A reboot fixes these issues (e.g. version now works). Can’t believe we have to deal with this in 2022???
7z x clang+llvm-15.0.1-aarch64-linux-gnu.tar.xz
7z x clang+llvm-15.0.1-arm64-apple-darwin21.0.tar.xz
7z x clang+llvm-15.0.1-x86_64-apple-darwin.tar.xz
tar xf clang+llvm-15.0.1-aarch64-linux-gnu.tar
tar xf clang+llvm-15.0.1-arm64-apple-darwin21.0.tar
tar xf clang+llvm-15.0.1-x86_64-apple-darwin.tar
Here are the directories in the include folder before the installer is created. There are also 28 include files in the include/llvm-c/ directory as desired.
Directory of llvm\utils\release\llvm_package_15.0.0\build32_stage0\_CPack_Packages\win64\NSIS\LLVM-15.0.0-win64\include
clang
clang-c
clang-tidy
lld
lldb
llvm
llvm-c
Outstanding Questions
Why does the NSIS project fail to build? Why are there test failures and build errors?
Why does the Linux build have ompt-multiplex.h and the aarch64-unknown-linux-gnu directory?
How is the Windows ARM64 installer generated?
Why doesn’t the Windows build have c++, flang, mlir, mlir-c, and polly?
How do we get symbols to the Python app in the Microsoft Store?
Elmer is the first codebase that I have dug into that has a substantial (or really any) amount of Fortran code. I used GFortran to build it but went digging around for a clang based compiler. I found llvm-project/flang and since I had been building LLVM earlier this year, I figured it should be straightforward to build flang and perhaps explore it in a debugger.
My first attempt to build flang (on Windows, my primary OS) resulted in many build errors. Unfortunately, I was using a preview Visual Studio build, so I didn’t want to compare the errors with those from a different machine because it wasn’t the same compiler version in use. I decided to use an RTM Visual Studio compiler (VS 17.2.5) to avoid possible compiler bugs present only in VS preview builds since most people would not be using preview VS builds anyway.
Without giving it much thought, my suspicion was that any build failures probably arose from not using the correct C++ version. The source code I was trying to build (commit c0702ac0) states that it uses C++17. I set this in CMake by defining the CXX_STANDARD property. Here is the full cmake command line I used to set up the build.
cd llvm-project
mkdir build
cd build
cmake \
-G Ninja \
../llvm \
-DCMAKE_BUILD_TYPE=Release \
-DFLANG_ENABLE_WERROR=On \
-DLLVM_ENABLE_ASSERTIONS=ON \
-DLLVM_TARGETS_TO_BUILD=host \
-DCMAKE_INSTALL_PREFIX=../install
-DLLVM_LIT_ARGS=-v \
-DLLVM_ENABLE_PROJECTS="clang;mlir;flang" \
-DLLVM_ENABLE_RUNTIMES="compiler-rt" \
-DCXX_STANDARD=17
# Shown here without \ to be executable in cmd.exe
cmake -G Ninja ../llvm -DCMAKE_BUILD_TYPE=Release -DFLANG_ENABLE_WERROR=On -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_TARGETS_TO_BUILD=host -DCMAKE_INSTALL_PREFIX=../install -DLLVM_LIT_ARGS=-v -DLLVM_ENABLE_PROJECTS="clang;mlir;flang" -DLLVM_ENABLE_RUNTIMES="compiler-rt" -DCXX_STANDARD=17
That took about 2 minutes on my machine after which I ran ninja to start the build
ninja
Unfortunately, the build failed! The first error I encountered was in fold-real.cpp. Here is the command line used to invoke the compiler (shown with newlines to simplify interpretation, see Compiler options listed alphabetically | Microsoft Docs for the complete list of compiler options).
I tried manually creating a repro for this compiler issue by creating a new Visual C++ project in Visual Studio and recreating the structure of the code failing to build. One of the questions I had was how to set conformance mode in a Visual Studio Cmake project. I still haven’t yet figured this out. However, one of the issues I ran into was that my cmake project was building the code without the /permissive- flag! I ended up switching to a regular Visual C++ project (.vcxproj) since I knew how to change the compiler options reliably for such projects. After struggling with recreating the code, I realized that I would make more progress removing code from flang’s fold-real.cpp instead. Here are some of the other searches and concepts I had to look up to understand the code while trying to create a minimal repro of the build failure.
I was eventually able to create a simpler test case showing that the flang code could not build with my RTM compiler.
cl /std:c++17 /permissive- flang-msvc-clang-test.cpp
Microsoft (R) C/C++ Optimizing Compiler Version 19.32.31332 for x64
Copyright (C) Microsoft Corporation. All rights reserved.
flang-msvc-clang-test.cpp
flang-msvc-clang-test.cpp(159): error C2065: 'T': undeclared identifier
flang-msvc-clang-test.cpp(48): note: see reference to function template instantiation 'auto FoldIntrinsicFunction::<lambda_1>::operator ()<_First>(const _T1 &) const' being compiled
with
[
_First=Expr<Type<TypeCategory::Real,1>>,
_T1=Expr<Type<TypeCategory::Real,1>>
]
flang-msvc-clang-test.cpp(171): note: see reference to function template instantiation 'Expr<Type<TypeCategory::Real,2>> FoldIntrinsicFunction<2>(FoldingContext &,FunctionRef<Type<TypeCategory::Real,2>> &&)' being compiled
flang-msvc-clang-test.cpp(159): error C2923: 'Scalar': 'T' is not a valid template type argument for parameter 'T'
flang-msvc-clang-test.cpp(159): note: see declaration of 'T'
So after all that, the RTM LTS Visual C++ compiler turned out to have a bug. Turns out the Visual C++ folks had already fixed this issue so the way to unblock myself was to switch to the preview Visual Studio build :(! The irony…
Suppressing Warnings
Armed with a preview build that correctly compiled the test case, the next obstacle in the build process was a set of warnings that were treated as errors: C4661 and C4101.
FAILED: tools/flang/lib/Evaluate/CMakeFiles/obj.FortranEvaluate.dir/fold.cpp.obj
C:\...\cl.exe ... -c D:\dev\repos\llvm-project\flang\lib\Evaluate\fold.cpp
D:\dev\repos\llvm-project\flang\include\flang\Evaluate\expression.h(101): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\include\flang\Evaluate\expression.h(101): warning C4661: 'std::optional<Fortran::evaluate::DynamicType> Fortran::evaluate::ExpressionBase<Fortran::evaluate::SomeDerived>::GetType(void) const': no suitable definition provided for explicit template instantiation request
...
FAILED: tools/flang/lib/Evaluate/CMakeFiles/obj.FortranEvaluate.dir/fold-complex.cpp.obj
C:\...\cl.exe ... -c D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-complex.cpp
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-implementation.h(1583): error C2220: the following warning is treated as an error
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-implementation.h(1583): warning C4101: 'buffer': unreferenced local variable
I tried to suppressed them to keep marching forward:
Defining CXX_FLAGS like that did not work so I end up looking around for how to disable warnings in cmake. This was when I discovered that CMAKE_CXX_STANDARD is not necessary on the command line because flang/CMakeLists.txt already requires C++17. Trying to append the warning disable option /wdXXXX to that file didn’t work either. However, the comment on line 329 made me explore HandleLLVMOptions.cmake. There, I discovered support for setting the number of parallel jobs (via /MP for Visual C++). This file also contained the code that sets up most of the compiler options used when building! Closer to the task at hand is the discover of the LLVM_ENABLE_WARNINGS option and the hard-coded list of MSVC warning flags! I therefore made this change (before running cmake and ninja) to get the warning flags to be respected:
diff --git a/llvm/cmake/modules/HandleLLVMOptions.cmake b/llvm/cmake/modules/HandleLLVMOptions.cmake
index 56d05f5b5fce..589281b232f1 100644
--- a/llvm/cmake/modules/HandleLLVMOptions.cmake
+++ b/llvm/cmake/modules/HandleLLVMOptions.cmake
@@ -648,6 +648,8 @@ if (MSVC)
# v15.8.8. Re-evaluate the usefulness of this diagnostic when the bug
# is fixed.
-wd4709 # Suppress comma operator within array index expression
+ -wd4101 # Suppress ...
+ -wd4661 # Suppress ...
# Ideally, we'd like this warning to be enabled, but even MSVC 2019 doesn't
# support the 'aligned' attribute in the way that clang sources requires (for
By this point, I knew that simplifying the function containing the error was the fastest path to a repro. One of the little problems I ran into was how to figure out the type of fptr since it is declared using the auto keyword. I ended up assigning it to a new temporary variable of a different type, e.g. char et voila!
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-integer.cpp(505): error C2440: 'initializing': cannot convert from 'int (__cdecl Fortran::evaluate::value::Integer<8,true,8,unsigned char,unsigned short>::* )(void) const' to 'char'
I then removed the temporary assignment and explicitly specified this type as the type of fptr:
using T2 = int (__cdecl Fortran::evaluate::value::Integer<8,true,8,unsigned char,unsigned short>::* )(void) const;
T2 fptr{&Scalar<TI>::LEADZ};
The build then failed because the function pointer types are not the same, which was really confusing given that I had just checked the type of fptr.
I switched the type of fptr and got a different error:
D:\dev\repos\llvm-project\flang\include\flang\Evaluate\integer.h(66): error C2607: static assertion failed
D:\dev\repos\llvm-project\flang\lib\Evaluate\fold-integer.cpp(490): note: see reference to class template instantiation 'Fortran::evaluate::value::Integer<16,true,16,unsigned char,unsigned short>' being compiled
Here is a different change I tried:
using T2 = int (__cdecl Fortran::evaluate::value::Integer<8>::* )(void) const;
T2 fptr{&Scalar<TI>::LEADZ};
It was at this point that I realized that it was time to learn a bit more about decay. What is decay and array-to-pointer conversion? | C++ FAQ (64.github.io) had a good explanation of why the term decay is used. Perhaps a reexamination of std::decay – cppreference.com might lead to some insight. I wasn’t sure what Result referred to in the statement using TI = typename std::decay_t<decltype(n)>::Result; One idea I got was to append a number to the typename and examine the compiler error. Here’s the new line 752 of llvm-project/fold-integer.cpp and the resulting compiler error showing that this name cannot be arbitrary.
using TI = typename std::decay_t<decltype(n)>::Result3;
C:\dev\repos\llvm-project\flang\lib\Evaluate\fold-integer.cpp(502): error C2039: 'Result3': is not a member of 'Fortran::evaluate::Expr<Fortran::evaluate::Type<Fortran::common::TypeCategory::Integer,1>>'
template <int KIND>
class Expr<Type<TypeCategory::Integer, KIND>>
: public ExpressionBase<Type<TypeCategory::Integer, KIND>> {
public:
using Result = Type<TypeCategory::Integer, KIND>;
...
The problematic lambda is therefore expecting a Scalar<Type<TypeCategory::Integer, KIND>>. Scalar is defined using decay and Type<TypeCategory::Integer, KIND>::Scalar is defined in llvm-project/type.h as the type value::Integer<8 * KIND>. This is when I see the reason for the previous build errors about mismatched Integer sizes no matter which size I picked – the fixed type I was using didn’t allow for the different template instantiations! Note that the problematic lambda is defined as a ScalarFunc.
By this point, I had a self-contained repro of the compiler bug, which ironically, compiled successfully on the RTM C++ compiler so I could use neither the preview nor the RTM to build the flang code.
This compiler invocation gives the same error seen when compiling the flang code:
Microsoft (R) C/C++ Optimizing Compiler Version 19.33.31627.1 for x64
Copyright (C) Microsoft Corporation. All rights reserved.
flang-msvc-clang-test-02.cpp
flang-msvc-clang-test-02.cpp(193): error C2672: 'invoke': no matching overloaded function found
C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31721\include\type_traits(1552): note: could be 'unknown-type std::invoke(_Callable &&,_Ty1 &&,_Types2 &&...) noexcept(<expr>)'
flang-msvc-clang-test-02.cpp(193): note: Failed to specialize function template 'unknown-type std::invoke(_Callable &&,_Ty1 &&,_Types2 &&...) noexcept(<expr>)'
8253757: Add LLVM-based backend for hsdis by magicus · Pull Request #7531 makes it possible to easily use LLVM as the hsdis backend. An LLVM installation is required for this. The official LLVM builds for the Windows platform do not work for building hsdis because they do not have all the prerequisite LLVM include files. See Building LLVM for Windows ARM64 – Saint’s Log (swesonga.org) for instructions on how to build LLVM for ARM64 Windows (on an x64 Windows host). To configure OpenJDK for LLVM as an hsdis backend on Windows ARM64, use this command:
The JDK and hsdis can then be built as usual with these commands:
make images
make build-hsdis
make install-hsdis
cp /cygdrive/d/dev/software/llvm-aarch64/bin/LLVM-C.dll build/windows-aarch64-server-slowdebug/jdk/bin/
The generated JDK can then be deployed to an ARM64 machine like the Surface Pro X. To test LLVM’s disassembly, use the -XX:CompileCommand flag on the ARM64 machine:
The path given to --with-llvm needs to be a Cygwin path if building in Cygwin. Otherwise, the build-hsdis target will fail with this error: c:\...\jdk\src\utils\hsdis\llvm\hsdis-llvm.cpp(58): fatal error C1083: Cannot open include file: 'llvm-c/Disassembler.h': No such file or directory. I caught this by inspecting build\windows-aarch64-server-release\make-support\failure-logs\support_hsdis_hsdis-llvm.obj.cmdline after the build failed. This was the only include that didn’t have Cygwin paths: -IC:/dev/repos/llvm-project/build_llvm_AArch64/install_local/include
Investigating Missing Disassembly
My first disassembly attempt did not work – only abstract disassembly was displayed:
I verified that hsdis-aarch64.dll was present in the JDK’s bin folder. That was the only issue I had seen before that caused this behavior so I dug around to find the code that loads the hsdis DLL. A search for the “hsdis-” DLL prefix in the sources reveals the hsdis_library_name string used in the Disassembler::dll_load method. Notice that there is a Verbose flag that can display what is happening when loading the hsdis DLL!
void* Disassembler::dll_load(char* buf, int buflen, int offset, char* ebuf, int ebuflen, outputStream* st) {
int sz = buflen - offset;
int written = jio_snprintf(&buf[offset], sz, "%s%s", hsdis_library_name, os::dll_file_extension());
if (written < sz) { // written successfully, not truncated.
if (Verbose) st->print_cr("Trying to load: %s", buf);
return os::dll_load(buf, ebuf, ebuflen);
} else if (Verbose) {
st->print_cr("Try to load hsdis library failed: the length of path is beyond the OS limit");
}
return NULL;
}
Error: VM option 'Verbose' is develop and is available only in debug version of VM.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
hsdis-aarch64.dll is not being loaded because LLVM-C.dll cannot be found! Still learning the need for reading the full instructions to avoid unnecessary pain.
I was trying to test using LLVM as a backend for hsdis on the Windows ARM64 platform as implemented in PR 5920. I downloaded LLVM 13 and tried to use it in the build. Unfortunately, it didn’t have all the prerequisite include files and so building your own LLVM installation was the approach suggested for Windows. This post explicitly outlines the instructions needed to build LLVM for the Windows ARM64 platform on a Windows x64 host machine.
The first requirement is an LLVM build with native llvm-nm.exe and llvm-tblgen.exe binaries. These can be downloaded (I think) or generated by building LLVM for the native x64 platform as specified in the instructions from Jorn.
git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build_llvm
cd build_llvm
cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=X86" -D"CMAKE_BUILD_TYPE:STRING=Release" -D"CMAKE_INSTALL_PREFIX=install_local" -A x64 -T host=x64
cmake --build . --config Release --target install
Once that build successfully completes, we can then build LLVM for the Windows ARM64 platform with the commands below. Notice that we specify paths to the native llvm-nm and llvm-tblgen binaries to prevent the build from trying to use their ARM64 equivalents (which won’t run on the host).
cd llvm-project
mkdir build_llvm_AArch64
cd build_llvm_Aarch64
cmake ../llvm -DLLVM_TARGETS_TO_BUILD:STRING=AArch64 \
-DCMAKE_BUILD_TYPE:STRING=Release \
-DCMAKE_INSTALL_PREFIX=install_local \
-DCMAKE_CROSSCOMPILING=True \
-DLLVM_TARGET_ARCH=AArch64 \
-DLLVM_NM=C:/repos/llvm-project/build_llvm/install_local/bin/llvm-nm.exe \
-DLLVM_TABLEGEN=C:/repos/llvm-project/build_llvm/install_local/bin/llvm-tblgen.exe \
-DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-win32-msvc \
-A ARM64 \
-T host=x64
date; time \
cmake --build . --config Release --target install ; \
date
Once the build completes, the LLVM ARM64 files will be in the build_llvm_AArch64/install_local folder in the llvm-project repo. That build should have all the necessary header files and static libraries required for LLVM projects targeting Windows on ARM64. See the general cmake options and the LLVM-specific cmake options for details on the various flags and variables.
cd llvm-project
mkdir build_llvm_AArch64
cd build_llvm_AArch64
cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=AArch64" \
-D"CMAKE_BUILD_TYPE:STRING=Release" \
-D"CMAKE_INSTALL_PREFIX=install_local_AArch64" \
-D"CMAKE_CROSSCOMPILING=True" \
-D"LLVM_TARGET_ARCH=AArch64" \
-A x64 \
-T host=x64
cmake --build . --config Release --target install
This still results in errors about conflicting machine types:
c:\...\llvm-project\build_llvm_aarch64\install_local_aarch64\\lib\llvmaarch64disassembler.lib : warning LNK4272: library machine type 'x64' conflicts with target machine type 'ARM64'
That’s when I tried adding the LLVM_TABLE_GEN from a Windows x64 LLVM build I had generated earlier. I accidentally omitted the options prefixed with a # below because I didn’t include the trailing slash after adding the llvm-tblgen.exe option.
The build still succeeded and generated AArch64 .lib files in the LLVM installation! Interestingly, they still had the x64 machine type in the header.
$ dumpbin /headers build_llvm_AArch64_2\install_local_AArch64_2\lib\LLVMAArch64AsmParser.lib
Microsoft (R) COFF/PE Dumper Version 14.29.30133.0
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file build_llvm_AArch64_2\install_local_AArch64_2\lib\LLVMAArch64AsmParser.lib
File Type: LIBRARY
FILE HEADER VALUES
8664 machine (x64)
...
I had no choice but to reexamine my understanding of what the -A flag does. It is used to specify the platform name but it’s only after digging into the CMAKE_GENERATOR_PLATFORM docs that I noticed that this was the target platform! This also made me realize that I hadn’t noticed that the x64 C++ compiler was being used all along!
-- The C compiler identification is MSVC 19.29.30133.0
-- The CXX compiler identification is MSVC 19.29.30133.0
-- The ASM compiler identification is MSVC
-- Found assembler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - works
Setting -A to AArch64 causes MSBuild to fail with an error about an unknown platform. So -A just might be the argument I need to get ARM64 libraries built.
"C:\dev\repos\llvm-project\build_llvm_AArch64_3\CMakeFiles\3.17.3\VCTargetsPath.vcxproj" (default target) (1) ->
(_CheckForInvalidConfigurationAndPlatform target) ->
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(820,5): error : The BaseOutputPath/OutputPath property is not set for project 'VCTargetsPath.vcxproj'. Please check to make sure that you have specified a valid combination of Configuration and Platform for this project. Configuration='Debug' Platform='AArch64'. You may be seeing this message because you are trying to build a project without a solution file, and have specified a non-default Configuration or Platform that doesn't exist for this project. [C:\dev\repos\llvm-project\build_llvm_AArch64_3\CMakeFiles\3.17.3\VCTargetsPath.vcxproj]
So I tried using -A ARM64 instead. I noticed that we now have the ARM64 C++ compiler selected! This is something I should have been paying attention to from the beginning, crucial for cross-compilation.
-- The C compiler identification is MSVC 19.29.30133.0
-- The CXX compiler identification is MSVC 19.29.30133.0
-- The ASM compiler identification is MSVC
-- Found assembler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe - works
Unfortunately, the build still failed with an error from gen-msvc-exports.py. Taking a look at gen-msvc-exports.py, it looks like it is trying to run llvm-nm.exe (for the target platform).
Generating export list for LLVM-C
Traceback (most recent call last):
File "C:/dev/repos/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 116, in <module>
main()
File "C:/dev/repos/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 112, in main
gen_llvm_c_export(ns.output, ns.underscore, libs, ns.nm)
File "C:/dev/repos/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 72, in gen_llvm_c_export
check_call([nm, '-g', lib], stdout=dumpout_f)
File "C:\dev\tools\Anaconda3\lib\subprocess.py", line 359, in check_call
retcode = call(*popenargs, **kwargs)
File "C:\dev\tools\Anaconda3\lib\subprocess.py", line 340, in call
with Popen(*popenargs, **kwargs) as p:
File "C:\dev\tools\Anaconda3\lib\subprocess.py", line 854, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\dev\tools\Anaconda3\lib\subprocess.py", line 1307, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
OSError: [WinError 216] This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(241,5): error MSB8066: Custom build for 'C:\dev\repos\llvm-project\build_llvm_AArch64_3\CMakeFi
les\02a88fa656bb9cf8b9ffd0e0debe57ae\libllvm-c.exports.rule;C:\dev\repos\llvm-project\build_llvm_AArch64_3\CMakeFiles\8ebc0efbf04134b25d0f37561fba0d55\LLVM-C.def.rule;C:\dev\repos\llvm-project\build_llvm_AArch64_
3\CMakeFiles\509fcb3f8bb132e9c560e15e8d25cb45\LLVM-C_exports.rule;C:\dev\repos\llvm-project\llvm\tools\llvm-shlib\CMakeLists.txt' exited with code 1. [C:\dev\repos\llvm-project\build_llvm_AArch64_3\tools\llvm-shl
ib\LLVM-C_exports.vcxproj]
date; time cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=AArch64" \
-D"CMAKE_BUILD_TYPE:STRING=Release" \
-D"CMAKE_INSTALL_PREFIX=install_local" \
-D"CMAKE_CROSSCOMPILING=True" \
-D"LLVM_TARGET_ARCH=AArch64" \
-D"LLVM_NM=C:\dev\repos\llvm-project\build_llvm\install_local\bin\llvm-nm.exe" \
-D"LLVM_TABLEGEN=C:\dev\repos\llvm-project\build_llvm\install_local\bin\llvm-tblgen.exe" \
-D"LLVM_DEFAULT_TARGET_TRIPLE=aarch64-win32-msvc" \
-A ARM64 \
-T host=x64
date; time \
cmake --build . --config Release --target install; \
date
These build commands work! Dumpbin shows that the generated .lib files have ARM64 headers!
$ dumpbin /headers C:\dev\repos\llvm-project\build_llvm_AArch64_3\Release\lib\LLVMAArch64Disassembler.lib
Microsoft (R) COFF/PE Dumper Version 14.29.30133.0
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file C:\dev\repos\llvm-project\build_llvm_AArch64_3\Release\lib\LLVMAArch64Disassembler.lib
File Type: LIBRARY
FILE HEADER VALUES
AA64 machine (ARM64)
Unfortunately, the JDK project that got me started down this path still doesn’t build. Cygwin shows defines like -DLLVM_DEFAULT_TRIPLET='”aarch64-pc-windows-msvc”‘ being passed to the compiler, which then complains:
C:/.../src/utils/hsdis/llvm/hsdis-llvm.cpp(217): error C2015: too many characters in constant
The quotes in the commands therefore needed to be dropped. This caused build failures since the paths used back-slashes!
Building Opts.inc...
'C:devreposllvm-projectbuild_llvminstall_localbinllvm-tblgen.exe' is not recognized as an internal or external command,
operable program or batch file
This is now the part where I find a nice document on the LLVM site with the 3-liner for this task 😀
