Saint's Log

Tracking Down Missing Headers in LLVM for Windows

hsdis is a plugin for disassembling code dynamically generated by the Java Virtual Machine. On Linux & MacOS, it uses GNU binutils. Support for the LLVM disassembly backend was recently added to hsdis in https://github.com/openjdk/jdk/pull/7531. This was motivated by the fact that GNU binutils is not distributed with the JDK (due to licensing reasons mentioned at https://github.com/openjdk/jdk/pull/5920#issuecomment-942398786) and the LLVM disassembly may be preferrable in certain circumstances. Unfortunately, the official Windows LLVM distribution does not have the header files necessary to build the hotspot disassembler. This prevents Windows developers from easily using the LLVM disassembler backend because they now have to build LLVM themselves as well – see hsdis LLVM backend for Windows ARM64 and Building LLVM for Windows ARM64, for example. In this post, we investigate why the LLVM Windows build does not have the necessary header files. The llvm-c directory in Windows build contains these 2 files only:

C:\Program Files\LLVM\include\llvm-c>dir
 Volume in drive C is OSDisk
 Volume Serial Number is c070-2ac0

 Directory of C:\Program Files\LLVM\include\llvm-c

01/08/2022  11:54 AM    <DIR>          .
01/08/2022  11:54 AM    <DIR>          ..
09/24/2021  10:18 AM            29,760 lto.h
09/24/2021  10:18 AM             9,632 Remarks.h
               2 File(s)         39,392 bytes
               2 Dir(s)  62,273,200,128 bytes free

I created a local LLVM build (see Building LLVM with CMake) and confirmed that it has all the header files.

C:\dev\repos\llvm-project\build_llvm\install_local\include\llvm-c>dir /w
 Volume in drive C is OSDisk
 Volume Serial Number is 0087-4c48

 Directory of C:\dev\repos\llvm-project\build_llvm\install_local\include\llvm-c

[.]                   [..]                  Analysis.h
BitReader.h           BitWriter.h           blake3.h
Comdat.h              Core.h                DataTypes.h
DebugInfo.h           Deprecated.h          Disassembler.h
DisassemblerTypes.h   Error.h               ErrorHandling.h
ExecutionEngine.h     ExternC.h             Initialization.h
IRReader.h            Linker.h              LLJIT.h
lto.h                 Object.h              Orc.h
OrcEE.h               Remarks.h             Support.h
Target.h              TargetMachine.h       [Transforms]
Types.h
              28 File(s)        382,361 bytes
               3 Dir(s)  59,158,138,880 bytes free

Does this problem still exist in the latest Windows LLVM release? I went to Releases · llvm/llvm-project (github.com) to find the latest LLVM installer for Windows but couldn’t find it. Turns out it’s because the 15.0.1 release is only 14 hours old so some of the assets probably haven’t been uploaded. Notice that 15.0.0 has 47 assets. I can successfully download and install LLVM-15.0.0-win64.exe and see that the header files are still missing.

Interestingly, trying to install LLVM-15.0.0-win32.exe before uninstalling LLVM-15.0.0-win64.exe gives this dialog and clicking Yes uninstalls before the actual installation of the 32-bit build starts!

I assumed that would happen at this stage:

All the same, these dialogs have strings that can lead us to the sources that create the installer! The installer looks very similar to the one from Building the Elmer Install Folder so searching the llvm codebase for “ncis ” gives only a handful of hits leading to the key discovery of build_llvm_release.bat! (later learn that this needs to be executed in a (2019) developer command prompt so that the ninja command can be found). That script requires 7zip though. The script fails on my machine because it can’t find 7zip. Failure seems to be coming from the for-statement (see for | Microsoft Learn for usage). The for command uses the escape character (^) as explained at set | Microsoft Learn.

C:\dev\repos\llvm-project\llvm\utils\release> build_llvm_release.bat 15.0.0
Check 7-zip version and/or administrator permissions.
'7z.exe' is not recognized as an internal or external command,
operable program or batch file.
You need to modify the paths below:
Revision: llvmorg-15.0.0
Package version: 15.0.0
Build dir: C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0

Press any key to continue . . .

Why does the script not exit gracefully now? Git blame shows that the checking code was added by Update the Windows packaging script. · llvm/llvm-project@df7c577 (github.com). Adding the 7-Zip installation path to my user environment variables addresses this issue but the script should check for this!

Installing 7-Zip allows me to check the syntax of the command used by the script to ensure that it will work.

C:\Program Files\7-Zip> 7z.exe | findstr /r "2[1-9].[0-9][0-9]"
7-Zip 22.01 (x64) : Copyright (c) 1999-2022 Igor Pavlov : 2022-07-15

Surprisingly, the script still fails, this time because

'mv' is not recognized as an internal or external command,
operable program or batch file.

This must be line 81 introduced by Update the Windows packaging script. · llvm/llvm-project@83e9225 (github.com). Changing it to “move” now displays an error but the script continues executing until this error:

-- Looking for CrashReporterClient.h
-- Looking for CrashReporterClient.h - not found
-- Looking for pfm_initialize in pfm
-- Looking for pfm_initialize in pfm - not found
-- Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)
CMake Error at C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find LibXml2 (missing: LIBXML2_INCLUDE_DIR)
Call Stack (most recent call first):
  C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/FindLibXml2.cmake:108 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  cmake/config-ix.cmake:156 (find_package)
  CMakeLists.txt:774 (include)


-- Configuring incomplete, errors occurred!
See also "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/CMakeFiles/CMakeOutput.log".
See also "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/CMakeFiles/CMakeError.log".

The script downloads and extracts LibXml in the build directory. It also changes the libxmldir path separators from \ to /. To see the exact command failing, comment out the echo off line.

cmake
 -GNinja
 -DCMAKE_BUILD_TYPE=Release
 -DLLVM_ENABLE_ASSERTIONS=OFF
 -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON
 -DLLVM_BUILD_LLVM_C_DYLIB=ON
 -DCMAKE_INSTALL_UCRT_LIBRARIES=ON
 -DPython3_FIND_REGISTRY=NEVER
 -DPACKAGE_VERSION=15.0.0
 -DLLDB_RELOCATABLE_PYTHON=1
 -DLLDB_EMBED_PYTHON_HOME=OFF
 -DCMAKE_CL_SHOWINCLUDES_PREFIX="Note: including file: "
 -DLLVM_ENABLE_LIBXML2=FORCE_ON
 -DLLDB_ENABLE_LIBXML2=OFF
 -DCMAKE_C_FLAGS="-DLIBXML_STATIC"
 -DCMAKE_CXX_FLAGS="-DLIBXML_STATIC"
 -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld;compiler-rt;lldb;openmp"
 -DLLDB_TEST_COMPILER=C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0/build32_stage0/bin/clang.exe
 -DPYTHON_HOME=C:\Users\saint\AppData\Local\Programs\Python\Python310-32
 -DPython3_ROOT_DIR=C:\Users\saint\AppData\Local\Programs\Python\Python310-32
 -DLIBXML2_INCLUDE_DIRS=C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/libxmlbuild/install/include/libxml2
 -DLIBXML2_LIBRARIES=C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/libxmlbuild/install/lib/libxml2s.lib ..\llvm-project\llvm

Looking through FindPackageHandleStandardArgs.cmake leads me to the simple realization that the wrong define is being used on the command line. Could this be because I’m using a newer CMake? I’ve been using the VS 2022 Preview Developer Command Prompt thus far. My VS 2019 (16.11.19) installation uses CMake 3.20. Both FindLibXml2.cmake in 3.20 and FindLibXml2.cmake in 3.24 require the LIBXML2_INCLUDE_DIR variable. However, they also claim (at the top) to set these variables.

A quick review of the history of build_llvm_release.bat shows that Build Windows releases with libxml enabled, to unbreak llvm-mt · llvm/llvm-project@145835c (github.com) introduced -DLIBXML2_INCLUDE_DIR but the next commit Pass -DLIBXML2_INCLUDE_DIRS in the Windows release package script · llvm/llvm-project@7735019 (github.com) changed it to plural. Adding the singular form to the script finally unblocks the build. Now to see how packing happens..

Packaging a Regular LLVM Build

in my build folder (build_llvm), there is a CPackConfig.cmake file that sets variables like CPACK_PACKAGE_FILE_NAME and CPACK_NSIS_DISPLAY_NAME. Since it is NSIS Wiki (sourceforge.io) in use, I wonder about running the package target myself in a manner similar to that used to create my local build. I switch back to a previous build directory (created without the build_llvm_release.bat) and run:

cmake --build . --config Release --target package

The resulting failure below indicates that NSIS is required.

MSBuild version 17.4.0-preview-22466-03+48ab5664b for .NET Framework
  PipSqueak.vcxproj -> C:\dev\repos\llvm-project\build_llvm\unittests\Support\DynamicLibrary\Release\PipSqueak.dll
  SecondLib.vcxproj -> C:\dev\repos\llvm-project\build_llvm\unittests\Support\DynamicLibrary\Release\SecondLib.dll
  obj.llvm-tblgen.vcxproj -> C:\dev\repos\llvm-project\build_llvm\utils\TableGen\obj.llvm-tblgen.dir\Release\obj.llvm-tblgen.lib
  LLVMDemangle.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\lib\LLVMDemangle.lib
...
  verify-uselistorder.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\verify-uselistorder.exe
  yaml-bench.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\yaml-bench.exe
  yaml2obj.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\yaml2obj.exe
EXEC : CPack error : Cannot find NSIS compiler makensis: likely it is not installed, or not in your PATH [C:\dev\repos\llvm-project\build_llvm\package.vcxproj]
EXEC : CPack error : Could not read NSIS registry value. This is usually caused by NSIS not being installed. Please install NSIS from http://nsis.sourceforge.net [C:\dev\repos\llvm-proje
ct\build_llvm\package.vcxproj]
EXEC : CPack error : Cannot initialize the generator NSIS [C:\dev\repos\llvm-project\build_llvm\package.vcxproj]

After installing NSIS, the previous command successfully creates an LLVM for Windows installer.

...
  verify-uselistorder.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\verify-uselistorder.exe
  yaml-bench.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\yaml-bench.exe
  yaml2obj.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\yaml2obj.exe
  CPack: Create package using NSIS
  CPack: Install projects
  CPack: - Install project: LLVM [Release]
  CMake Warning (dev) at C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/GNUInstallDirs.cmake:243 (messa
  ge):
    Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
    target architecture is known.  Please enable at least one language before
    including GNUInstallDirs.
  Call Stack (most recent call first):
    C:/dev/repos/llvm-project/llvm/cmake/modules/LLVMInstallSymlink.cmake:5 (include)
    C:/dev/repos/llvm-project/build_llvm/tools/llvm-ar/cmake_install.cmake:48 (include)
    C:/dev/repos/llvm-project/build_llvm/tools/cmake_install.cmake:39 (include)
    C:/dev/repos/llvm-project/build_llvm/cmake_install.cmake:71 (include)
  This warning is for project developers.  Use -Wno-dev to suppress it.

  CPack: Create package
  CPack: - package: C:/dev/repos/llvm-project/build_llvm/LLVM-16.0.0git-win64.exe generated.

This installer generates the LLVM includes on disk as expected. The issue must therefore be confined to the installer generated by the script.

Reviewing Ninja NSIS Packaging

At this point, I ran build_llvm_release.bat to create an installer. Once packaging is complete, the install_manifest.txt file can be used to determine which files are in the installer. The batch file also runs lots of tests and this was annoying when trying to generate installers. Once the tests failed on the build I was creating and I had CTRL+C’d a couple of times, I ran ninja package myself (taken from the batch file)

C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0>ninja package
[0/1] Run CPack packaging tool...CPack: Create package using NSIS
CPack: Install projects
CPack: - Install project: LLVM []
CMake Warning (dev) at C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/GNUInstallDirs.cmake:243 (message):
  Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
  target architecture is known.  Please enable at least one language before
  including GNUInstallDirs.
Call Stack (most recent call first):
  C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/llvm-project/llvm/cmake/modules/LLVMInstallSymlink.cmake:5 (include)
  C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/tools/llvm-ar/cmake_install.cmake:40 (include)
  C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/tools/cmake_install.cmake:39 (include)
  C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/cmake_install.cmake:114 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

CPack: Create package
CPack: - package: C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/LLVM-15.0.0-win32.exe generated.

I was curious about these warnings but found it really annoying that I can’t open these paths by pasting them into the VS Code File/Open dialog. Looks like Windows: Allow to open file paths that contain slashes · Issue #15270 · microsoft/vscode (github.com) but that points to [Windows] Bug with open file dialog with forward slash (`file_dialog::ShowOpenDialog`) · Issue #7954 · electron/electron (github.com). Unfortunately, this sounds like a Windows Open dialog limitation since Notepad is not using the same dialog as Notepad++.

I then try to find a package target in build.ninja. Search for CMakeFiles\package.util.+ include since we’re interested in include files. There are some interesting differences in the include directories of the build created manually from the local install and the one created by the script, e.g.

Directory of C:\dev\repos\llvm-project\build_llvm\include\llvm\Support
[.]
[..]
[CMakeFiles]
cmake_install.cmake
Extension.def
INSTALL.vcxproj
INSTALL.vcxproj.filters
llvm_vcsrevision_h.vcxproj
llvm_vcsrevision_h.vcxproj.filters
PACKAGE.vcxproj
PACKAGE.vcxproj.filters
VCSRevision.h
[x64]
               9 File(s)         47,877 bytes
               4 Dir(s)  34,980,511,744 bytes free

Directory of C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0\include\llvm\Support

[.]                   [..]                  [CMakeFiles]
cmake_install.cmake   Extension.def         VCSRevision.h
               3 File(s)          1,293 bytes
               3 Dir(s)  34,981,122,048 bytes free

Try searching in build.ninja for the 2 header files the installer creates in the (broken) shipping LLVM for Windows build. Nothing there but searching the file system for remarks.h gives interesting results, e.g. the existence of an NSIS project file: project.nsi. Looks like there are some tutorials showing how to create .nsi files at Invoking NSIS run-time commands on compile-time – NSIS (sourceforge.io). The way NSIS is used with CPack when building is documented at Packaging With CPack — Mastering CMake

Directory of C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0\_CPack_Packages\win32\NSIS

09/21/2022  06:41 PM    <DIR>          .
09/21/2022  06:41 PM    <DIR>          ..
09/21/2022  06:41 PM    <DIR>          LLVM-15.0.0-win32
09/21/2022  06:54 PM       256,557,945 LLVM-15.0.0-win32.exe
09/21/2022  06:41 PM               631 NSIS.InstallOptions.ini
09/21/2022  06:41 PM            55,204 project.nsi
               3 File(s)    256,613,780 bytes
               3 Dir(s)  35,416,317,952 bytes free

Directory of C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0\_CPack_Packages\win32\NSIS\LLVM-15.0.0-win32\include\llvm-c

09/21/2022  06:41 PM    <DIR>          .
09/21/2022  06:41 PM    <DIR>          ..
09/05/2022  03:48 AM            30,109 lto.h
09/05/2022  03:48 AM             9,632 Remarks.h
               2 File(s)         39,741 bytes
               2 Dir(s)  35,416,289,280 bytes free

The natural hypothesis is that NSIS is simply packing the whole LLVM-15.0.0-win32 directory into the installer. I had been comparing these two files earlier…

C:\dev\repos\llvm-project\build_llvm\cmake_install.cmake
C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0\cmake_install.cmake

… but I completely missed the fact that the 2nd didn’t have these lines from the first.

if(CMAKE_INSTALL_COMPONENT STREQUAL "llvm-headers" OR NOT CMAKE_INSTALL_COMPONENT)
  file(INSTALL DESTINATION "${CMAKE_INSTALL_PREFIX}/include" TYPE DIRECTORY FILES
    "C:/dev/repos/llvm-project/llvm/include/llvm"
    "C:/dev/repos/llvm-project/llvm/include/llvm-c"
    FILES_MATCHING REGEX "/[^/]*\\.def$" REGEX "/[^/]*\\.h$" REGEX "/[^/]*\\.td$" REGEX "/[^/]*\\.inc$" REGEX "/license\\.txt$")
endif()

if(CMAKE_INSTALL_COMPONENT STREQUAL "llvm-headers" OR NOT CMAKE_INSTALL_COMPONENT)
  file(INSTALL DESTINATION "${CMAKE_INSTALL_PREFIX}/include" TYPE DIRECTORY FILES
    "C:/dev/repos/llvm-project/build_llvm/include/llvm"
    "C:/dev/repos/llvm-project/build_llvm/include/llvm-c"
    FILES_MATCHING REGEX "/[^/]*\\.def$" REGEX "/[^/]*\\.h$" REGEX "/[^/]*\\.gen$" REGEX "/[^/]*\\.inc$" REGEX "/cmakefiles$" EXCLUDE REGEX "/config\\.h$" EXCLUDE)
endif()

Search the codebase for “llvm-headers” and find the llvm-header component definition. That whole code block is gated by the LLVM_INSTALL_TOOLCHAIN_ONLY variable! This is explicitly turned off in build_llvm_release.bat! I rerun the batch file and see tests failing after the build succeeds. CTRL+C to kill the processes so that I can get to the root issue: does turning off that flag fix the includes? makensis fails, probably because I killed the build and some things might still have been in use?

C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0>ninja package
[0/1] Run CPack packaging tool...CPack: Create package using NSIS
CPack: Install projects
CPack: - Install project: LLVM []
CMake Warning (dev) at C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/GNUInstallDirs.cmake:243 (message):
  Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
  target architecture is known.  Please enable at least one language before
  including GNUInstallDirs.
Call Stack (most recent call first):
  C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/llvm-project/llvm/cmake/modules/LLVMInstallSymlink.cmake:5 (include)
  C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/tools/llvm-ar/cmake_install.cmake:40 (include)
  C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/tools/cmake_install.cmake:39 (include)
  C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/cmake_install.cmake:128 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

CPack: Create package
CPack Error: Problem running NSIS command: "C:/Program Files (x86)/NSIS/makensis.exe" "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/_CPack_Packages/win32/NSIS/project.nsi"
Please check C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/_CPack_Packages/win32/NSIS/NSISOutput.log for errors
CPack Error: Problem compressing the directory
CPack Error: Error when generating package: LLVM

FAILED: CMakeFiles/package.util
cmd.exe /C "cd /D C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0 && "C:\Program Files\Microsoft Visual Studio\2022\Preview\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cpack.exe" --config ./CPackConfig.cmake"
ninja: build stopped: subcommand failed.

NSISOutput.log failed due to an “Internal compiler error #12345: error mmapping datablock to 17235001.” However, the include files are now present in the source directory being packaged by NSIS.

Turning Off Tests

There are many tests that the build script runs and some of them are failing. Testing is not on my critical path since all I need is to generate installers so I modify the scripts to enable me to package the build without running all the tests. I then start my build without tests and go to bed only to wake up the next morning to find that I need to rerun it because there are no running programs when I log in. Event Viewer doesn’t show any reboot-related events and sure enough, Task Manager shows over 9 days of uptime still. Turns out the Desktop Window Manager crashed (C:\WINDOWS\system32\dwm.exe)! Curse you dwmcore.dll. Well, time to install those updates I’ve been putting off and reboot before jumping back in. Now on the new Windows 10.0.22621.521. The build still fails:

-- LLVM host triple: i686-pc-windows-msvc
-- LLVM default target triple: i686-pc-windows-msvc
-- Using Release VC++ CRT: MD
-- Looking for os_signpost_interval_begin
-- Looking for os_signpost_interval_begin - not found
CMake Error at C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python3 (missing: Python3_EXECUTABLE Interpreter) (Required
  is at least version "3.6")

      Reason given by package:
          Interpreter: Cannot use the interpreter "C:/Python310/python.exe"

Call Stack (most recent call first):
  C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPython/Support.cmake:3165 (find_package_handle_standard_args)
  C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPython3.cmake:485 (include)
  CMakeLists.txt:817 (find_package)


-- Configuring incomplete, errors occurred!
See also "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/CMakeFiles/CMakeOutput.log".
See also "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/CMakeFiles/CMakeError.log".

When I interrupted the tests before modifying the batch file to skip them, I noticed that they were being run by %LOCALAPPDATA%\Microsoft\WindowsApps\python3.9.exe. This is still present on my machine. Ah, turns out I’m now using the 2019 developer command prompt (and therefore an older CMake). The only difference between CMake 3.20 FindPython3.cmake and CMake 3.24 FindPython3.cmake is a comment about static libraries, so this failure is a mystery.

Diagnosing Build Failures

Since this issue also bit me when I moved to my Surface Book, it is worth understanding why it happens.

Missing CMake in Visual Studio 17.3.4 Developer Command Prompt

Here is the VS 2022 Preview vs VS 2022 Enterprise path to CMake:

C:\Program Files (x86)\Microsoft Visual Studio\Installer> where cmake
C:\Program Files\Microsoft Visual Studio\2022\Preview\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe

C:\dev\repos\llvm-project\llvm\utils\release> where cmake
INFO: Could not find files for the given pattern(s).

Wait… why is there no CMake in VS 2022 Enterprise on my desktop? The Visual Studio Installer shows cmake to be installed. It also warnings and it’s only now that I’m learning that there are not just troubleshooting tips but also ways to create a local layout from the command line Create an offline installation – Visual Studio (Windows) | Microsoft Learn

The View Logs link opens the Documents folder under This PC – not particularly useful. Interestingly though, clicking on the Modify button shows a Total space required 1.63 GB. How is there space required before I’ve selected anything? Something similar happens with 16.11.19 though. Without making any individual component selections, I start the install process. CMake gets (re-?)installed as shown below. This fixes the setup warnings as well and cmake is now usable in the VS2022 command prompt.

Missing Python3 in VS 17.3.4 Developer Command Prompt

This is the error I got when trying to build LLVM on my Surface Book 2 in the VS 2022 developer command prompt:

CMake Error at C:/Program Files/CMake/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:164 (message):
  Could NOT find Python3 (missing: Python3_EXECUTABLE Interpreter) (Required
  is at least version "3.6")

      Reason given by package:
          Interpreter: Cannot use the interpreter "C:/Python310/python.exe"

Call Stack (most recent call first):
  C:/Program Files/CMake/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:445 (_FPHSA_FAILURE_MESSAGE)
  C:/Program Files/CMake/share/cmake-3.17/Modules/FindPython/Support.cmake:2437 (find_package_handle_standard_args)
  C:/Program Files/CMake/share/cmake-3.17/Modules/FindPython3.cmake:309 (include)
  CMakeLists.txt:817 (find_package)

Here is the (fixed up) output from where python:

C:\Python310\python.exe
%LOCALAPPDATA%\Microsoft\WindowsApps\python.exe

I modify build_llvm_release.bat to pass the --trace-expand --trace-redirect=cmake_trace.txt CMake option as recommended by cmake Python: Cannot use the interpreter – Stack Overflow. That’s when I notice that the list of python versions CMake is looking for does not contain 3.10: Modules/FindPython/Support.cmake · v3.17.5 · CMake / CMake · GitLab (kitware.com). My suspicion is that this is the cause of the above error. It looks like I installed CMake a while back on this laptop.

Uninstalling CMake enables the command line to pick up the CMake distributed with Visual Studio. Python3 is now found successfully in the path below (I’ve shortened it using %LOCALAPPDATA%).

-- Found Python3: %LOCALAPPDATA%/Microsoft/WindowsApps/python3.8.exe (found suitable version "3.8.10", minimum required is "3.6") found components: Interpreter

Missing Python3 in VS 16.11.19 Developer Command Prompt

Interestingly, I still get the same error in VS 2019 despite uninstalling CMake 3.17. My earlier hypothesis is therefore invalid.

CMake Error at C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python3 (missing: Python3_EXECUTABLE Interpreter) (Required
  is at least version "3.6")

      Reason given by package:
          Interpreter: Cannot use the interpreter "C:/Python310/python.exe"

Call Stack (most recent call first):
  C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPython/Support.cmake:3165 (find_package_handle_standard_args)
  C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPython3.cmake:485 (include)
  CMakeLists.txt:817 (find_package)

Using --trace-expand --trace-redirect=cmake_trace.txt again (and searching for “execute_process“) reveals how the python version is determined using the execute_process cmake command in Modules/FindPython/Support.cmake · v3.20.0:

C:/Python310/python.exe -c "import sys; sys.stdout.write('.'.join([str(x) for x in sys.version_info[:3]]))"

I comment out the ERROR_QUIET line to reveal the stdout and stderr output from python since the return code from the python process is causing the CMake error to be raised. Running with --trace-expand --trace-redirect=cmake_trace.txt now reveals the root cause (paths below cleaned up using %LOCALAPPDATA%):

Python path configuration:
  PYTHONHOME = '%LOCALAPPDATA%\Programs\Python\Python310-32'
  PYTHONPATH = (not set)
  program name = 'C:/Python310/python.exe'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = 'C:\\Python310\\python.exe'
  sys.base_prefix = '%LOCALAPPDATA%\\Programs\\Python\\Python310-32'
  sys.base_exec_prefix = '%LOCALAPPDATA%\\Programs\\Python\\Python310-32'
  sys.platlibdir = 'lib'
  sys.executable = 'C:\\Python310\\python.exe'
  sys.prefix = '%LOCALAPPDATA%\\Programs\\Python\\Python310-32'
  sys.exec_prefix = '%LOCALAPPDATA%\\Programs\\Python\\Python310-32'
  sys.path = [
    'C:\\Python310\\python310.zip',
    '%LOCALAPPDATA%\\Programs\\Python\\Python310-32\\DLLs',
    '%LOCALAPPDATA%\\Programs\\Python\\Python310-32\\lib',
    'C:\\Python310',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00003174 (most recent call first):
  <no Python frame>

django – init_fs_encoding: failed to get the Python codec of the filesystem encoding – Stack Overflow is a hint that the PYTHONHOME is wrong. Sure enough, I didn’t change it in build_llvm_release.bat so the paths in the configuration above do not exist! This now raises another question: how on earth does this work in VS 2022? I notice on my desktop that python.exe does not even appear in the CMake tracing output! The difference in behavior stems from the fact that the find_program command in Modules/FindPython/Support.cmake · v3.20.0 finds python 3.10 first in the VS 2019 environment. This path is then assigned to _Python3_EXECUTABLE, preventing the 3.8 path from being used. One important difference between CMake 3.20 and 3.23 that I notice is FindPython: fix typo error (fff8d5b2) · Commits · CMake / CMake · GitLab (kitware.com). Since the fix for the build_llvm_release.bat script is straightforward and it is clear that there are some CMake implementation differences at work, we no longer need to dig into why this behavior could be happening.

Python Hangs

One of my build attempts successfully completes stage0 but hangs when CMake tries to detect the python version. Manually running the same command (copied from Process Explorer) also hangs. Even %LOCALAPPDATA%/Microsoft/WindowsApps/python3.9.exe --version hangs. Inspecting the full dump created by Task Manager reveals that python3.9.exe made a call to get (what looks like) the Package.InstalledLocation Property (Windows.ApplicationModel) – Windows UWP applications | Microsoft Learn

...
-- Looking for os_signpost_interval_begin
-- Looking for os_signpost_interval_begin - not found

Windows becomes pretty unusable as I investigate this behavior (mouse doesn’t work, changes program in focus but can’t click on anything). A reboot fixes these issues (e.g. version now works). Can’t believe we have to deal with this in 2022???

The support link is https://www.python.org/doc/ and the product link is https://www.python.org/. The privacy policy is https://www.python.org/privacy/ and the license terms link is https://docs.python.org/3.9/license.html.

I’m tempted to just remove this store app but also curious about how to get symbols and see exactly where it hang.

Comparison with macOS/Linux includes

On Windows, it is easy to get the Linux and macOS LLVM builds using curl (added to Windows in build 17063 as per Tar and Curl Come to Windows! | Microsoft Learn).

curl -L https://github.com/llvm/llvm-project/releases/download/llvmorg-15.0.1/clang+llvm-15.0.1-aarch64-linux-gnu.tar.xz -o clang+llvm-15.0.1-aarch64-linux-gnu.tar.xz

curl -L https://github.com/llvm/llvm-project/releases/download/llvmorg-15.0.1/clang+llvm-15.0.1-arm64-apple-darwin21.0.tar.xz -o clang+llvm-15.0.1-arm64-apple-darwin21.0.tar.xz

curl -L https://github.com/llvm/llvm-project/releases/download/llvmorg-15.0.1/clang+llvm-15.0.1-x86_64-apple-darwin.tar.xz -o clang+llvm-15.0.1-x86_64-apple-darwin.tar.xz

To use 7zip to extract these XZ Files:

7z x clang+llvm-15.0.1-aarch64-linux-gnu.tar.xz
7z x clang+llvm-15.0.1-arm64-apple-darwin21.0.tar.xz
7z x clang+llvm-15.0.1-x86_64-apple-darwin.tar.xz

tar xf clang+llvm-15.0.1-aarch64-linux-gnu.tar
tar xf clang+llvm-15.0.1-arm64-apple-darwin21.0.tar
tar xf clang+llvm-15.0.1-x86_64-apple-darwin.tar

To use XZ Utils (tukaani.org) to extract these XZ Files, run these commands but note that they remove the .xz files!

xz -d clang+llvm-15.0.1-aarch64-linux-gnu.tar.xz
xz -d clang+llvm-15.0.1-arm64-apple-darwin21.0.tar.xz
xz -d clang+llvm-15.0.1-x86_64-apple-darwin.tar.xz

tar xf clang+llvm-15.0.1-aarch64-linux-gnu.tar
tar xf clang+llvm-15.0.1-arm64-apple-darwin21.0.tar
tar xf clang+llvm-15.0.1-x86_64-apple-darwin.tar

Here are the ARM64 llvm include directory listings for macOS and Linux LLVM builds.

.../Downloads/clang+llvm-15.0.1-arm64-apple-darwin21.0/include
c++
clang
clang-c
clang-tidy
flang
lld
lldb
llvm
llvm-c
mlir
mlir-c
polly

.../Downloads/clang+llvm-15.0.1-aarch64-linux-gnu/include
aarch64-unknown-linux-gnu
c++
clang
clang-c
clang-tidy
flang
lld
lldb
llvm
llvm-c
mlir
mlir-c
ompt-multiplex.h
polly

Here are the directories in the include folder before the installer is created. There are also 28 include files in the include/llvm-c/ directory as desired.

Directory of llvm\utils\release\llvm_package_15.0.0\build32_stage0\_CPack_Packages\win64\NSIS\LLVM-15.0.0-win64\include
 clang
 clang-c
 clang-tidy
 lld
 lldb
 llvm
 llvm-c

Outstanding Questions

1. Why does the NSIS project fail to build? Why are there test failures and build errors?
2. Why does the Linux build have ompt-multiplex.h and the aarch64-unknown-linux-gnu directory?
3. How is the Windows ARM64 installer generated?
4. Why doesn’t the Windows build have c++, flang, mlir, mlir-c, and polly?
5. How do we get symbols to the Python app in the Microsoft Store?