I was recently implementing matrix multiplication on the GPU (using CUDA). For my application, I was generating random numbers and generating statistics about the performance of matrix multiplication variants (e.g. using shared memory vs naive multiplication). Some of the results tended to differ from the CPU’s results. Therefore, I decided to use deterministic matrices for the inputs to ensure my algorithm is correct. What I needed was a neutral (3rd party) matrix multiplication algorithm. This seems like a job for MATLAB. Unfortunately, my license expired a few years ago. My robotics professor at the University of Washington was a fan of Octave because it is open source and free. Here is the script I created to generate matrices with the positive integers.
A = 1:10000;
B = 10001:20000;
A = reshape(A, [100,100]);
B = reshape(B, [100,100]);
A = transpose(A);
B = transpose(B);
C = A * B;
# format short;
save 'mmult100x100.txt' C;
Backstory
It has been a while since I used MATLAB. Here are the searches I used to create the script.
This process of using Nsight Compute to profile CUDA kernels is documented in detail at Nsight Compute :: Nsight Compute Documentation (nvidia.com). Here are the screenshots with the “quick start” steps without all the verbosity of the documentation.
I need to understand what happens if we build the jdk master branch (at commit 18cd16d2 when I started) without any ABI-specific changes. To do so, we need JDK 18 or later as a boot JDK to build the latest code, e.g. Oracle’s JDK 18 Windows x64 Installer. Here are the commands I used in Cygwin:
git clone https://github.com/swesonga/jdk
cd jdk
bash configure --openjdk-target=aarch64-unknown-cygwin --with-debug-level=slowdebug --with-boot-jdk=/cygdrive/d/dev/repos/java/infra/binaries/jdk-18.0.2
make images LOG=debug > build/abi-20220802-1500.txt
make build-test-jdk-jtreg-native LOG=debug > build/test-20220802-1500.txt
Once the build complete, create the artifacts for an AArch64 Windows device. These build and archive steps are available as the build-aarch64.sh script.
cd build/windows-aarch64-server-slowdebug/jdk
zip -qru jdk-20220802-1500-master.zip .
mv jdk-20220802-1500-master.zip ..
cd ..
zip -qru test-jdk-20220802-1500-master.zip support/test
Copy the two zip files to the 64-bit ARM device (e.g. by sharing folders or using OneDrive). I used a Surface Pro X device running Windows 11 build 22000.795. I unzipped the 2 files into these paths:
I later discovered that unzip is available in the Git Bash terminal! These commands can be used to unzip the files:
mkdir -p /c/dev/java/abi/devbranch/jdk
cd /c/dev/java/abi/devbranch/jdk
unzip -q /c/dev/java/builds/debug/jdk-20220802-1500-devbranch.zip
cd ..
unzip -q test-jdk-20220802-1500-master.zip
I also downloaded jtreg and placed it in this path (note that it might be easier to extract the .tar.gz on the Windows x64 build machine then share it).
C:\dev\java\jtreg\
Finish setting up the Windows AArch64 device to run the ABI jtreg tests by cloning the OpenJDK repo onto it. The jtreg tests will be run from the root of the OpenJDK repo.
cd \dev\java\repos\forks
git clone https://github.com/swesonga/jdk
cd jdk
We’ll run VaListTest.java to see how it fails on Windows AArch64.
--------------------------------------------------
TEST: java/foreign/valist/VaListTest.java
TEST JDK: C:\dev\java\abi\master\jdk
ACTION: build -- Passed. All files up to date
REASON: Named class compiled on demand
TIME: 0.069 seconds
messages:
command: build VaListTest
reason: Named class compiled on demand
elapsed time (seconds): 0.069
ACTION: testng -- Failed. Execution failed: `main' threw exception: org.testng.TestNGException: An error occurred while instantiating class VaListTest: null
REASON: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED VaListTest
TIME: 12.557 seconds
messages:
command: testng --enable-native-access=ALL-UNNAMED VaListTest
reason: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED VaListTest
Mode: othervm [/othervm specified]
Additional options from @modules: --add-modules java.base --add-exports java.base/jdk.internal.foreign=ALL-UNNAMED --add-exports java.base/jdk.internal.foreign.abi=ALL-UNNAMED --add-exports java.base/jdk.internal.foreign.abi.x64=ALL-UNNAMED --add-exports java.base/jdk.internal.foreign.abi.x64.sysv=ALL-UNNAMED --add-exports java.base/jdk.internal.foreign.abi.x64.windows=ALL-UNNAMED --add-exports java.base/jdk.internal.foreign.abi.aarch64=ALL-UNNAMED --add-exports java.base/jdk.internal.foreign.abi.aarch64.linux=ALL-UNNAMED --add-exports java.base/jdk.internal.foreign.abi.aarch64.macos=ALL-UNNAMED --add-exports java.base/jdk.internal.foreign.abi.aarch64.windows=ALL-UNNAMED
elapsed time (seconds): 12.557
configuration:
Boot Layer
add modules: java.base
add exports: java.base/jdk.internal.foreign ALL-UNNAMED
java.base/jdk.internal.foreign.abi ALL-UNNAMED
java.base/jdk.internal.foreign.abi.aarch64 ALL-UNNAMED
java.base/jdk.internal.foreign.abi.aarch64.linux ALL-UNNAMED
java.base/jdk.internal.foreign.abi.aarch64.macos ALL-UNNAMED
java.base/jdk.internal.foreign.abi.aarch64.windows ALL-UNNAMED
java.base/jdk.internal.foreign.abi.x64 ALL-UNNAMED
java.base/jdk.internal.foreign.abi.x64.sysv ALL-UNNAMED
java.base/jdk.internal.foreign.abi.x64.windows ALL-UNNAMED
STDOUT:
STDERR:
WARNING: package jdk.internal.foreign.abi.aarch64.windows not in java.base
org.testng.TestNGException:
An error occurred while instantiating class VaListTest: null
at org.testng.internal.InstanceCreator.createInstanceUsingObjectFactory(InstanceCreator.java:123)
at org.testng.internal.InstanceCreator.createInstance(InstanceCreator.java:79)
...
I expected Bernhard’s code to be the one introducing Windows AArch64 ABI clean-up code. So why are there failures about the aarch64.windows foreign abi package missing? This requirement is from VaListTest.java and was introduced by the Foreign Function & Memory API (Preview) PR (it added the java.base/jdk.internal.foreign.abi.aarch64.windows module to the failing test).
Porting the Changes
I worked on porting Bernhard’s code on a Windows x64 machine.
# Switch the the OpenJDK repo directory
cd jdk
# This was the tip of the upstream master branch
# git checkout 18cd16d2eae2ee624827eb86621f3a4ffd98fe8c
git switch -c WinAArch64ABI
git remote add lewurm https://github.com/lewurm/openjdk
git fetch lewurm
git switch foreign-windows-aarch64
git rebase WinAArch64ABI
The files he modified have been deleted in the current repo:
$ git log --full-history -2 -- src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java
commit 2c5d136260fa717afa374db8b923b7c886d069b7
Author: Maurizio Cimadamore <mcimadamore@openjdk.org>
Date: Thu May 12 16:17:45 2022 +0000
8282191: Implementation of Foreign Function & Memory API (Preview)
Reviewed-by: erikj, jvernee, psandoz, dholmes, mchung
The deleted files moved to src/java.base/share/classes/jdk/internal/foreign. Bernhard’s changes are small enough that I manually port them (copy/paste) into the files in the new locations in the tree. It’s interesting seeing the newer Java language features in use, e.g. the permits keyword. Now build the changes using the build-aarch64.sh script:
$ find build/windows-aarch64-server-slowdebug/jdk/ -name "WindowsAArch64CallArranger*"
...
build/windows-aarch64-server-slowdebug/jdk/modules/java.base/jdk/internal/foreign/abi/aarch64/windows/WindowsAArch64CallArranger.class
# Verify last modification time
$ ls -l build/windows-aarch64-server-slowdebug/jdk/./modules/java.base/jdk/internal/foreign/abi/aarch64/windows/WindowsAArch64CallArranger.class
Need to create a WindowsAArch64CallArranger to match the current structure of the foreign ABI. With these changes, VaListTest.java now passes. However, StdLibTest.java and TestVarArgs.java fail.
TEST: java/foreign/StdLibTest.java
TEST JDK: C:\dev\java\abi\devbranch\jdk
ACTION: build -- Passed. All files up to date
REASON: Named class compiled on demand
TIME: 0.039 seconds
messages:
command: build StdLibTest
reason: Named class compiled on demand
elapsed time (seconds): 0.039
ACTION: testng -- Failed. Unexpected exit from test [exit code: -1073741819]
REASON: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED StdLibTest
TIME: 15.02 seconds
messages:
command: testng --enable-native-access=ALL-UNNAMED StdLibTest
reason: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED StdLibTest
Mode: othervm [/othervm specified]
elapsed time (seconds): 15.02
configuration:
STDOUT:
test StdLibTest.test_printf([STRING]): failure
java.lang.AssertionError: expected [11] but found [14]
at org.testng.Assert.fail(Assert.java:99)
...
at org.testng.Assert.assertEquals(Assert.java:917)
at StdLibTest.test_printf(StdLibTest.java:135)
...
at org.testng.TestNG.run(TestNG.java:1037)
...
at java.base/java.lang.Thread.run(Thread.java:1589)
test StdLibTest.test_printf(java.util.ArrayList@5499b7af): success
test StdLibTest.test_printf([DOUBLE, DOUBLE, CHAR]): success
TEST: java/foreign/TestVarArgs.java
TEST JDK: C:\dev\java\abi\devbranch\jdk
ACTION: build -- Passed. All files up to date
REASON: Named class compiled on demand
TIME: 0.031 seconds
messages:
command: build TestVarArgs
reason: Named class compiled on demand
elapsed time (seconds): 0.031
ACTION: testng -- Failed. Unexpected exit from test [exit code: 1]
REASON: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED -Dgenerator.sample.factor=17 TestVarArgs
TIME: 17.52 seconds
messages:
command: testng --enable-native-access=ALL-UNNAMED -Dgenerator.sample.factor=17 TestVarArgs
reason: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED -Dgenerator.sample.factor=17 TestVarArgs
Mode: othervm [/othervm specified]
elapsed time (seconds): 17.52
configuration:
STDOUT:
test TestVarArgs.testVarArgs(0, "f0_V__", VOID, [], []): success
STDERR:
java.lang.RuntimeException: java.lang.IllegalStateException: java.lang.AssertionError: expected [24.0] but found [8.135772792034E-312]
at TestVarArgs.check(TestVarArgs.java:134)
...
at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:758)
at TestVarArgs.testVarArgs(TestVarArgs.java:104)
...
at org.testng.TestNG.runSuites(TestNG.java:1069)
at org.testng.TestNG.run(TestNG.java:1037)
...
The data for these tests is supplied by a testngdataProvider that returns an array of arrays of objects. As per the dataProvider docs, the first dimension’s size is the number of times the test method will be invoked and the second dimension size contains an array of objects that must be compatible with the parameter types of the test method.
Java Concepts in the Tests
As per the article Enum Types, enums implicitly extend java.lang.Enum and cannot extend anything else because Java does not support multiple inheritance. The Enum class docs also point out that all the constants of an enum class can be obtained by calling the implicit public static T[] values() method of that class and that more information about enums, including descriptions of the implicitly declared methods synthesized by the compiler, can be found in section 8.9 of The Java Language Specification. Section 8.9 explains that an enum constant may be followed by arguments, which are passed to the constructor of the enum when the constant is created during class initialization as described later in this section. The constructor to be invoked is chosen using the normal rules of overload resolution (§15.12.2). If the arguments are omitted, an empty argument list is assumed. This is helpful for understanding all the code I’m seeing in the PrintfArg enum!
The printfArgs dataProvider permutes the values of the PrintfArg enum. The implementation uses streams, which are new to me since I last wrote Java before JDK 8 was released. The overview of streams on Oracle’s technical resources website is helpful in coming up to speed with streams. TODO: the implementation of the permutation is mysterious to me, need to study it closely. It uses List.of(), Set.of(), and Collections.shuffle().
Try blocks without catch or finally blocks is a try-with-resources statement. This helps prevent leaks of native resources.
StdLibTest.java uses functionality from JEP 424: Foreign Function & Memory API (Preview). This JEP provides a good overview of why we need a supported API for accessing off-heap data (i.e. foreign memory) designed from the ground up to be safe and with JIT optimizations in mind.
Creates a memory segment on line 312 using the allocateUtf8String method of the MemorySession‘s SegmentAllocator base interface. This method “converts a Java string into a UTF-8 encoded, null-terminated C string, storing the result into a memory segment.”
Create a variable argument list using the VaList.make() method. This invokes SharedUtils.newVaList, which we modified to support Windows on AArch64.
Invoke the native vprintf function via its method handle: final static MethodHandle vprintf = abi.downcallHandle(abi.defaultLookup().lookup("vprintf").get(), FunctionDescriptor.of(C_INT, C_POINTER, C_POINTER));.
The value of the abi variable is computed by the SharedUtils.getSystemLinker method, hence the need for creating a WindowsAArch64Linker here. As explained at JEP 424: Foreign Function & Memory API (Preview), abi.defaultLookup() “creates a default lookup, which locates all the symbols in libraries that are commonly used on the OS and processor combination associated with the Linker instance.” defaultLookup() returns a SymbolLookup on which the lookup(“vprintf”) method is invoked. Note that Optional<T>.get() will throw a NoSuchElementException if no value is present. Otherwise, it will return the zero-length MemorySegment whose base address indicates the address of the vprintf function.
As per JEP 424, the Linker interface enables both downcalls (calls from Java code to native code) and upcalls (calls from native code back to Java code). The MemorySegment associated with the address of the vprintf function and a FunctionDescriptor (created by the static FunctionDescriptor.of method) are passed to Linker.downcallHandle to create a MethodHandle which can be used to call vprintf. The arguments to FunctionDescriptor.of are the MemoryLayouts representing the return type (int), the format string, and the format arguments. MethodHandle.invoke() is the how the native vprintf gets, well, invoked, with the format string and the variable argument list. Here’s the Java vprint method.
Inlining the code invoked by test_printf here for easy reference. See the docs for the printf function and the printf format specification for additional information about printf. Line 20 of specializedPrintf creates a MethodType for a method returning an int and taking a single pointer (MemoryAddress). appendParameterTypes is used to add all the other printf parameter types to the MethodType. The MemoryLayouts of the arguments are also accumulated into a list. It doesn’t look like we do anything with the method type (mt) though! Looks like dead code from this PR.
That PR also changed from invokeExact to invoke. Why?
As an aside, notice that the test_time test (and every other test) passed when we disabled test_printf. test_time calls gmtime, which returns a tm struct so that side of things is working fine.
Makes an array-spreading method handle, which accepts an array argument at a given position and spreads its elements as positional arguments in place of the array. The new method handle adapts, as its target, the current method handle. The type of the adapter will be the same as the type of the target, except that the arrayLength parameters of the target’s type, starting at the zero-based position spreadArgPos, are replaced by a single array parameter of type arrayType.
CallArranger.classifyLayout() will return either INTEGER, FLOAT, or POINTER for the case I’m interested in. These cases in UnboxBindingCalculator.getBindings call storageCalculator.nextStorage. DIving into that implementation reveals that we don’t want adjustForVarArgs() to be called! Hmm, after looking at the optimized code in my post on “Building & Disassembling ARM64 Code using Visual C++”, I notice FMOV being used to load general purpose registers x1-x3 with the IEEE double! This looks idfferent from the getBindings implementation, which gets the next storage for FLOATs from the vector registers! et voila! The contradiction I’ve been waiting for: now the addendum on variadic functions at Overview of ARM64 ABI conventions makes sense.
Clone the JitWatch repo. Download the mvn binaries. Set JAVA_HOME to the path of our custom JDK (with hsdis) then start JitWatch. Errors running it though.
No Windows AArch64 binaries at Adoptium or Oracle though.
Let’s just try on x64. Might gain some insight:
cd /d/dev/repos/java/AdoptOpenJDK/jitwatch
/d/dev/repos/java/infra/binaries/jdk-19+34/bin/java --enable-preview -jar ./ui/target/jitwatch-ui-shaded.jar
Looking at these options, I wonder if manually setting the Compile Threshold could show more disassembly:
Update JitWatch to support preview features then change JAVA_HOME. This doesn’t make mvn clean package use my latest JDK…
I can get the JIT to assemble for the main method. Why doesn’t this work on Windows for ARM64? Perhaps I should try a non-debug configuration by configuring as follows before running the build-aarch64.sh script:
I get the same results with the release build – no native code for my printf function! I wonder about downloading something heavier and seeing if anything interesting gets compiled to native code. How about Eclipse? Interestingly, there is no Eclipse build for Windows on ARM64!
Examining this reduced output now helps me realize that the double keyword is what I should have been looking for all along! Look at this snippet with arguments that look similar to my modified test case (where I call with a char, a double, and an integer).
I’m still unsure what the parm fields mean but I’m assuming that the double is still being passed in a vector register! Sure enough, I changed the BoxBindingCalculator instead of the UnboxBindingCalculator. Fixed that then reran the test:
The test fails but this time there is a fatal error! Feels like progress.
Note: C:\dev\repos\java\forks\jdk\test\jdk\java\foreign\StdLibTest.java uses preview features of Java SE 20.
Note: Recompile with -Xlint:preview for details.
ACTION: testng -- Failed. Unexpected exit from test [exit code: 1]
REASON: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED StdLibTest
TIME: 4.783 seconds
messages:
command: testng --enable-native-access=ALL-UNNAMED StdLibTest
reason: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED StdLibTest
Mode: othervm [/othervm specified]
elapsed time (seconds): 4.783
configuration:
STDOUT:
test StdLibTest.test_printf([INTEGRAL, STRING, CHAR, CHAR]): success
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (assembler_aarch64.hpp:253), pid=11060, tid=5996
# guarantee(val < (1ULL << nbits)) failed: Field too big for insn
#
# JRE version: OpenJDK Runtime Environment (20.0) (build 20-internal-adhoc.sawesong.jdk)
# Java VM: OpenJDK 64-Bit Server VM (20-internal-adhoc.sawesong.jdk, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, windows-aarch64)
# No core dump will be written.Minidumps are not enabled by default on client versions of Windows
#
# An error report file with more information is saved as:
# C:\dev\repos\java\forks\jdk\JTwork\scratch\0\hs_err_pid11060.log
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
hello(42,str,h,h)
Since the fatal error in the JRE states that Minidumps are not enabled by default on client versions of Windows, I enabled collection of dump files using the enable-crash-dumps.bat script. Now we see a minidump written to disk:
C:\dev\java\abi\devbranch5\jdk\bin\java.exe --enable-preview MinimizedStdLibTest
WARNING: A restricted method in java.lang.foreign.Linker has been called
WARNING: java.lang.foreign.Linker::nativeLinker has been called by the unnamed module
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc: SuppressErrorAt=\vmreg_aarch64.hpp:48
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (c:\dev\repos\java\forks\jdk\src\hotspot\cpu\aarch64\vmreg_aarch64.hpp:48), pid=14728, tid=11380
# assert(is_FloatRegister() && is_even(value())) failed: must be
#
# JRE version: OpenJDK Runtime Environment (20.0) (slowdebug build 20-internal-adhoc.sawesong.jdk)
# Java VM: OpenJDK 64-Bit Server VM (slowdebug 20-internal-adhoc.sawesong.jdk, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, windows-aarch64)
# Core dump will be written. Default location: C:\dev\java\abi\tests\hs_err_pid14728.mdmp
#
# An error report file with more information is saved as:
# C:\dev\java\abi\tests\hs_err_pid14728.log
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
Decide to run java under the debugger and see what happens.
Launch WinDbg and go to File > Open Executable…
Browse to the java.exe path.
Specify the starting directory containing the compiled MinimizedStdLibTest file.
Specify these arguments: --enable-preview MinimizedStdLibTest then click Open.
Press F5 to start the program.
After a few breaks due to unhandled exceptions, I decide to look up the warnings in the text on-screen when a foreign function API is invoked. These messages are from Reflection.ensureNativeAccess and are called by …
WARNING: A restricted method in java.lang.foreign.Linker has been called
WARNING: java.lang.foreign.Linker::nativeLinker has been called by the unnamed module
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
Debugging in Visual Studio 2019
Create a C++ Console Application then open its Configuration Properties. On the Debug page, change the command, command arguments, and working directory to that of the newly built java.exe. Here are some interesting methods based on exploring after setting breakpoints in methodHandles.cpp:
There are threads with native code (such as the methods above) but no method info. I think those are Java methods. I end up stepping through the code on x64 to gain a better understanding of how the native code stubs are generated. VZEROUPPER motivates a quick detour into AVX-512 just to get a better feel of what it’s about. The instruction set reference (from Intel® 64 and IA-32 Architectures Software Developer Manuals) explains that in 64-bit mode, VZEROUPPER zeroes the bits in positions 128 and higher in YMM0-YMM15 and ZMM0-ZMM15.
I end up updating the test to have a single MethodHandle.invoke() call on its own line to simplify narrowing down the call in the disassembly. To simplify debugging even further, I create another test (MinimizedStdLibTest20Args) with 20 arguments (most of them doubles) that need to be formatted. This should make it easier to identify the code I am interested in and how these arguments are passed. I have a better grasp of x86-64 architecture so that seems like a better place to start examining to better understanding how this native call is handled.
amd64 Disassembly
There are several verified entry points with these many parameters. Why? Here’s the last one on my Intel(R) Xeon(R) W-2133 CPU.
The string “MemberName required for invokeVirtual etc” looks like a unique string and is therefore a reasonable one to use to find the code that set up the entry point. It comes from the generate_method_handle_dispatch method. Placing a breakpoint here reveals an interesting stack:
jvm.dll!MethodHandles::generate_method_handle_dispatch(MacroAssembler * _masm, vmIntrinsicID iid, RegisterImpl * receiver_reg, RegisterImpl * member_reg, bool for_compiler_entry) Line 364 C++
jvm.dll!gen_special_dispatch(MacroAssembler * masm, const methodHandle & method, const BasicType * sig_bt, const VMRegPair * regs) Line 1508 C++
jvm.dll!SharedRuntime::generate_native_wrapper(MacroAssembler * masm, const methodHandle & method, int compile_id, BasicType * in_sig_bt, VMRegPair * in_regs, BasicType ret_type) Line 1572 C++
jvm.dll!AdapterHandlerLibrary::create_native_wrapper(const methodHandle & method) Line 3159 C++
jvm.dll!SystemDictionary::find_method_handle_intrinsic(vmIntrinsicID iid, Symbol * signature, JavaThread * __the_thread__) Line 2017 C++
jvm.dll!LinkResolver::lookup_polymorphic_method(const LinkInfo & link_info, Handle * appendix_result_or_null, JavaThread * __the_thread__) Line 446 C++
jvm.dll!LinkResolver::resolve_method(const LinkInfo & link_info, Bytecodes::Code code, JavaThread * __the_thread__) Line 756 C++
jvm.dll!LinkResolver::linktime_resolve_static_method(const LinkInfo & link_info, JavaThread * __the_thread__) Line 1106 C++
jvm.dll!LinkResolver::resolve_static_call(CallInfo & result, const LinkInfo & link_info, bool initialize_class, JavaThread * __the_thread__) Line 1072 C++
jvm.dll!MethodHandles::resolve_MemberName(Handle mname, Klass * caller, int lookup_mode, bool speculative_resolve, JavaThread * __the_thread__) Line 777 C++
jvm.dll!MHN_resolve_Mem(JNIEnv_ * env, _jobject * igcls, _jobject * mname_jh, _jclass * caller_jh, long lookup_mode, unsigned char speculative_resolve) Line 1252 C++
0000020a0a26fb92() Unknown
0000020a0058eb00() Unknown
0000005f992fd040() Unknown
0000005f992fd010() Unknown
This is essentially all the interesting action I have been searching for! Especially AdapterHandlerLibrary::create_native_wrapper, which calls SharedRuntime::java_calling_convention and SharedRuntime::generate_native_wrapper. The latter are exactly what I’ve been seeking!
The VerifyOops flag is off by default so the verify_oop doesn’t generate any code. The testptr is therefore the first MacroAssembler code to be generated. Notice that the code jumps to the MemberName required for invokeVirtual etc label if rcx is zero – that must be error-handling code. The jz mnemonic would be preferrable to je (see assembly – Difference between JE/JNE and JZ/JNZ – Stack Overflow) but they are identical opcodes. Here is the listing with links to the methods that generated them.
class oopDesc {
friend class VMStructs;
friend class JVMCIVMStructs;
private:
volatile markWord _mark;
union _metadata {
Klass* _klass;
narrowKlass _compressed_klass;
} _metadata;
The first movabsq instruction loads (int64_t)CompressedKlassPointers::base() into the temporary register r10. As per NarrowPtrStruct._base, this is the base address for oop-within-java-object materialization. Not yet exactly sure whether that means an offset to add to the klass* to get the virtual address of the object since this base is added to the klass* in rdi. That addition ends the MacroAssembler::load_klass call.
The 2nd movabsq instruction loads the external klass address of the klass with vmClassID java_lang_invoke_MemberName. This value is then compared with the computed klass address in r10. If these 2 values are equal, then all is well and the CPU will branch to L_ok. If this branch is not taken, then the super_check_offset of the MemberName Klass is computed by Klass::super_check_offset. This offset indicates where to look to observe a supertype. So for my purposes, everything in the ;; verify_klass {... ;; } verify_klass section can be ignored since it is MemberName validation.
Without looking at the rest of the assembly code, the key thing to notice is that rcx was assumed to have a MemberName, meaning that by the time all these instructions execute, all the arguments I passed to printf are already in registers/on the stack. A quick detour into the method header is in order though. Here’s the first instance of that signature.
Here is a particularly interesting callstack showing how NEP_makeDowncallStub ends up calling the DowncallStubGenerator.
> jvm.dll!DowncallStubGenerator::generate() Line 142 C++
jvm.dll!DowncallLinker::make_downcall_stub(BasicType * signature, int num_args, BasicType ret_bt, const ABIDescriptor & abi, const GrowableArray<VMRegImpl *> & input_registers, const GrowableArray<VMRegImpl *> & output_registers, bool needs_return_buffer) Line 101 C++
jvm.dll!NEP_makeDowncallStub(JNIEnv_ * env, _jclass * _unused, _jobject * method_type, _jobject * jabi, _jobjectArray * arg_moves, _jobjectArray * ret_moves, unsigned char needs_return_buffer) Line 77 C++
0000017244641db1() Unknown
...
What is interesting about this? The DowncallStubGenerator is not only generating assembly instructions that are most likely what I have been searching for, it also has logging code that is being skipped. That looks like unified logging code! Therefore, using +PrintAssembly was not sufficient to generate the code I wanted to see! Here’s an updated command line after which downcall.txt will contain the results of argument shuffling.
Here is a stack revealing a bit more detail about how the arguments are set up.
jvm.dll!SharedRuntime::java_calling_convention(const BasicType * sig_bt, VMRegPair * regs, int total_args_passed) Line 505 C++
jvm.dll!JavaCallingConvention::calling_convention(BasicType * sig_bt, VMRegPair * regs, int num_args) Line 66 C++
jvm.dll!ArgumentShuffle::ArgumentShuffle(BasicType * in_sig_bt, int num_in_args, BasicType * out_sig_bt, int num_out_args, const CallingConventionClosure * input_conv, const CallingConventionClosure * output_conv, VMRegImpl * shuffle_temp) Line 328 C++
jvm.dll!DowncallStubGenerator::generate() Line 141 C++
jvm.dll!DowncallLinker::make_downcall_stub(BasicType * signature, int num_args, BasicType ret_bt, const ABIDescriptor & abi, const GrowableArray<VMRegImpl *> & input_registers, const GrowableArray<VMRegImpl *> & output_registers, bool needs_return_buffer) Line 101 C++
jvm.dll!NEP_makeDowncallStub(JNIEnv_ * env, _jclass * _unused, _jobject * method_type, _jobject * jabi, _jobjectArray * arg_moves, _jobjectArray * ret_moves, unsigned char needs_return_buffer) Line 77 C++
0000017244641db1() Unknown
More questions about how all this works:
What happens after all the hsdis code is executed? Is the final jump to the native code?
Where is rbx loaded (since that’s what we’re jumping to)?
AArch64 Disassembly
Having now understood that I can log the downcall stubs using the unified logging flags, this is the stub I get on the Surface Pro X (generated by DowncallStubGenerator::generate)
Argument shuffle {
Move a double from ([-1137525940],[-1137525936]) to ([-1137525916],[-1137525912])
Move a double from ([-1137525948],[-1137525944]) to ([-1137525924],[-1137525920])
Move a double from ([-1137525956],[-1137525952]) to ([-1137525932],[-1137525928])
Move a double from ([-1137525964],[-1137525960]) to ([-1137525940],[-1137525936])
Move a double from ([-1137525972],[-1137525968]) to ([-1137525948],[-1137525944])
Move a double from ([-1137525980],[-1137525976]) to ([-1137525956],[-1137525952])
Move a double from ([-1137525988],[-1137525984]) to ([-1137525964],[-1137525960])
Move a double from ([-1137525996],[-1137525992]) to ([-1137525972],[-1137525968])
Move a double from ([-1137526004],[-1137526000]) to ([-1137525980],[-1137525976])
Move a double from ([-1137526012],[-1137526008]) to ([-1137525988],[-1137525984])
Move a double from (v7,v7) to ([-1137525996],[-1137525992])
Move a double from (v6,v6) to ([-1137526004],[-1137526000])
Move a double from (v5,v5) to ([-1137526012],[-1137526008])
Move a double from (v4,v4) to (c_rarg7,c_rarg7)
Move a double from (v3,v3) to (c_rarg6,c_rarg6)
Move a double from (v2,v2) to (c_rarg5,c_rarg5)
Move a long from (c_rarg1,c_rarg1) to (rscratch2,rscratch2)
Move a byte from (c_rarg3,BAD!) to (c_rarg1,BAD!)
Move a int from (c_rarg4,BAD!) to (c_rarg3,BAD!)
Move a double from (v1,v1) to (c_rarg4,c_rarg4)
Move a long from (c_rarg2,c_rarg2) to (c_rarg0,c_rarg0)
Move a double from (v0,v0) to (c_rarg2,c_rarg2)
Stack argument slots: 26
}
It is immediately evident that there are BAD! registers. Why isn’t there more output as one would expect from looking at the additional logging in DowncallStubGenerator::generate? Well, the JVM crash might have something to do with it…
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc: SuppressErrorAt=\vmreg_aarch64.hpp:48
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (c:\dev\repos\java\forks\jdk\src\hotspot\cpu\aarch64\vmreg_aarch64.hpp:48), pid=11888, tid=18884
# assert(is_FloatRegister() && is_even(value())) failed: must be
#
# JRE version: OpenJDK Runtime Environment (20.0) (slowdebug build 20-internal-adhoc.sawesong.jdk)
# Java VM: OpenJDK 64-Bit Server VM (slowdebug 20-internal-adhoc.sawesong.jdk, compiled mode, tiered, compressed oops, compressed class ptrs, g1 gc, windows-aarch64)
# Core dump will be written. Default location: C:\dev\repos\scratchpad\compilers\tests\aarch64\abi\printf\java\hs_err_pid11888.mdmp
#
# An error report file with more information is saved as:
# C:\dev\repos\scratchpad\compilers\tests\aarch64\abi\printf\java\hs_err_pid11888.log
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
NEP_makeDowncallStub calls ForeignGlobals::parse_vmstorage, which in turn defers to the architecture-specific ForeignGlobals::vmstorage_to_vmreg implementation. This code returns the BAD register if the VMStorage type and does not match the register type! This must be the culprit! How do I log the asString output?
Rexamining the x64 foreign downcall log below, I notice the BAD registers there too! Perhaps this is not an oddity after all. Could it be NativeCallingConvention::calling_convention marking half slots as bad? Actually, notice that in both x64 and AArch64 logs, only the byte and int have these BAD! entries. This must be the other 32-bit slot for the arguments! This means that the AArch64 log is actually fine!
Argument shuffle {
Move a double from ([79203860],[79203864]) to ([79203908],[79203912])
Move a double from ([79203852],[79203856]) to ([79203900],[79203904])
Move a double from ([79203844],[79203848]) to ([79203892],[79203896])
Move a double from ([79203836],[79203840]) to ([79203884],[79203888])
Move a double from ([79203828],[79203832]) to ([79203876],[79203880])
Move a double from ([79203820],[79203824]) to ([79203868],[79203872])
Move a double from ([79203812],[79203816]) to ([79203860],[79203864])
Move a double from ([79203804],[79203808]) to ([79203852],[79203856])
Move a double from ([79203796],[79203800]) to ([79203844],[79203848])
Move a double from ([79203788],[79203792]) to ([79203836],[79203840])
Move a double from ([79203780],[79203784]) to ([79203828],[79203832])
Move a double from (xmm7,xmm7) to ([79203820],[79203824])
Move a double from (xmm6,xmm6) to ([79203812],[79203816])
Move a double from (xmm5,xmm5) to ([79203804],[79203808])
Move a double from (xmm4,xmm4) to ([79203796],[79203800])
Move a double from (xmm3,xmm3) to ([79203788],[79203792])
Move a double from (xmm2,xmm2) to ([79203780],[79203784])
Move a long from (rdx,rdx) to (r10,r10)
Move a byte from (r9,BAD!) to (rdx,BAD!)
Move a int from (rdi,BAD!) to (r9,BAD!)
Move a double from (xmm1,xmm1) to (xmm2,xmm2)
Move a long from (r8,r8) to (rcx,rcx)
Move a double from (xmm0,xmm0) to (r8,r8)
Stack argument slots: 34
}
Back to the MacroAssembler’s and float_move methods… I think the fmovd instruction I seek is this one with a general purpose register operand. After changing double_move to support fmovd between general purpose and floating point registers, rerunning the test on AArch64 does not give any additional output in the downcall log file. Very strange since I don’t see an assertion failure preventing the logging code from running…
I realize though that instead of trying to mess with WinDbg, I can simply write to the unified logging stream (to which output is already successfully being written). Making the LogStream creation unconditional enables me to verify that the code is indeed being executed. __ flush looks like AbstractAssembler::flush. It is only now that I realize that this is not flushing the output stream of the assembler – it is instead invalidating the CPU’s instruction cache! This is done by callingFlushInstructionCache on Windows.
After fixing the assertion failure by now checking the register types for fmovd, I get an OOM. Lots of output in the hotspot.log as well. paste it here. The hsdis output ends with this:
The Chunk::new string is from Chunk::operator new. Before debugging this, I try adding a delay to the NEP.make call to see if the logs I want will be written to disk before the process dies but I still get the OOM without additional logging output.
Next idea, terminate the program with an assertion failure to see if the output will be written to disk at termination. _wassert – Search (bing.com) -> c – Why is `_wassert` wrapped in `(..,0)`? – Stack Overflow. The hotspot asserts appear to be defines for the CRT _assert function. The latter calls abort, which on Windows, lets a custom abort signal handler function to run (enabling cleanup of resources or log information). Does the JVM use this?
I sprinkle DowncallLinker::generate with this logging code: ls.print_cr("Returning stub after %d", __LINE__); The output shows that the generate method completes executing successfully. However, I don’t get any output from logging calls one level below it in the callstack – in DowncallLinker::make_downcall_stub. Commenting out the creation of the new RuntimeStub (by using the aforemention logging call then returning nullptr on the previous line) shows that execution makes it to that point successfully. That has got to be the culprint since logging messages after that stub do not appear in the logs. And now looking at the RuntimeStub class, it is evident that it has an operator new implementation!
Let’s take a look at happens in WinDbg. The bp, bu, bm (Set Breakpoint) and x (Examine Symbols) are quite useful. x * shows the local variables and their values. I didn’t have the matching sources on the Surface Pro when trying to step into DowncallLinker::make_downcall_stub so I cleaned up all the custom logging, committed my changes, and rebuilt the JDK.
bp jvm!NEP_makeDowncallStub
g
x *
Surprisingly, the newly built JDK successfully passes the StdLibTest.java. Unfortunately, it regresses VaListTest.java and still fails TestVarArgs.java. The error from VaListTest is surprising since that was passing before I began but it looks like a compiler error:
--------------------------------------------------
TEST: java/foreign/valist/VaListTest.java
TEST JDK: C:\dev\java\abi\devbranch5\jdk
ACTION: build -- Failed. Compilation failed: Compilation failed
REASON: Named class compiled on demand
TIME: 32.591 seconds
messages:
command: build VaListTest
reason: Named class compiled on demand
Test directory:
compile: VaListTest
elapsed time (seconds): 32.591
ACTION: compile -- Failed. Compilation failed: Compilation failed
REASON: .class file out of date or does not exist
TIME: 32.384 seconds
messages:
command: compile C:\dev\repos\java\forks\jdk\test\jdk\java\foreign\valist\VaListTest.java
reason: .class file out of date or does not exist
...
direct:
C:\dev\repos\java\forks\jdk\test\jdk\java\foreign\valist\VaListTest.java:153: error: cannot find symbol
= (builder, scope) -> WindowsAArch64Linker.newVaList(builder, scope.scope());
^
symbol: method scope()
location: variable scope of type MemorySession
Note: C:\dev\repos\java\forks\jdk\test\jdk\java\foreign\valist\VaListTest.java uses preview features of Java SE 20.
Note: Recompile with -Xlint:preview for details.
1 error
...
The rvalue in the failing assignment needs to match the other lines (simply replace with WindowsAArch64Linker.newVaList). Then get this:
test VaListTest.testCopy(VaListTest$$Lambda$125/0x000000080013cb10@1156402a, i32): success
test VaListTest.testCopy(): failure
org.testng.internal.reflect.MethodMatcherException:
[public void VaListTest.testCopy(java.util.function.BiFunction,java.lang.foreign.ValueLayout$OfInt)] has no parameters defined but was found to be using a data provider (either explicitly specified or inherited from class level annotation).
Data provider mismatch
Method: testCopy([Parameter{index=0, type=java.util.function.BiFunction, declaredAnnotations=[]}, Parameter{index=1, type=java.lang.foreign.ValueLayout$OfInt, declaredAnnotations=[]}])
Arguments: [(VaListTest$$Lambda$120/0x000000080013c000) VaListTest$$Lambda$120/0x000000080013c000@6a8ce624,(java.lang.foreign.ValueLayout$OfInt) i32]
at org.testng.internal.reflect.DataProviderMethodMatcher.getConformingArguments(DataProviderMethodMatcher.java:43)
at org.testng.internal.Parameters.injectParameters(Parameters.java:905)
at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:34)
at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.testng.TestRunner.privateRun(TestRunner.java:764)
at org.testng.TestRunner.run(TestRunner.java:585)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
at org.testng.SuiteRunner.run(SuiteRunner.java:286)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
at org.testng.TestNG.runSuites(TestNG.java:1069)
at org.testng.TestNG.run(TestNG.java:1037)
at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:93)
at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:53)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:578)
at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:125)
at java.base/java.lang.Thread.run(Thread.java:1589)
Turns out to be a porting bug in which copy() used winAArch64VaListFactory instead of winAArch64VaListScopedFactory. Thankfully the test passes after this fix. Unfortunately, TestVaArgs.java still fails:
STDOUT:
test TestVarArgs.testVarArgs(0, "f0_V__", VOID, [], []): success
test TestVarArgs.testVarArgs(17, "f0_V_S_DI", VOID, [STRUCT], [DOUBLE, INT]): success
test TestVarArgs.testVarArgs(34, "f0_V_S_IDF", VOID, [STRUCT], [INT, DOUBLE, FLOAT]): success
test TestVarArgs.testVarArgs(51, "f0_V_S_FDD", VOID, [STRUCT], [FLOAT, DOUBLE, DOUBLE]): success
test TestVarArgs.testVarArgs(68, "f0_V_S_DDP", VOID, [STRUCT], [DOUBLE, DOUBLE, POINTER]): success
test TestVarArgs.testVarArgs(85, "f0_V_S_PPI", VOID, [STRUCT], [POINTER, POINTER, INT]): success
test TestVarArgs.testVarArgs(102, "f0_V_IS_FF", VOID, [INT, STRUCT], [FLOAT, FLOAT]): failure
java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
at java.base/jdk.internal.foreign.abi.aarch64.windows.WindowsAArch64CallArranger$StorageCalculator.regAlloc(WindowsAArch64CallArranger.java:230)
at java.base/jdk.internal.foreign.abi.aarch64.windows.WindowsAArch64CallArranger$UnboxBindingCalculator.getBindings(WindowsAArch64CallArranger.java:369)
at java.base/jdk.internal.foreign.abi.aarch64.windows.WindowsAArch64CallArranger.getBindings(WindowsAArch64CallArranger.java:150)
at java.base/jdk.internal.foreign.abi.aarch64.windows.WindowsAArch64CallArranger.arrangeDowncall(WindowsAArch64CallArranger.java:157)
at java.base/jdk.internal.foreign.abi.aarch64.windows.WindowsAArch64Linker.arrangeDowncall(WindowsAArch64Linker.java:85)
at java.base/jdk.internal.foreign.abi.AbstractLinker.lambda$downcallHandle$0(AbstractLinker.java:53)
at java.base/jdk.internal.foreign.abi.SoftReferenceCache$Node.get(SoftReferenceCache.java:52)
at java.base/jdk.internal.foreign.abi.SoftReferenceCache.get(SoftReferenceCache.java:38)
at java.base/jdk.internal.foreign.abi.AbstractLinker.downcallHandle(AbstractLinker.java:51)
at java.base/java.lang.foreign.Linker.downcallHandle(Linker.java:221)
at TestVarArgs.testVarArgs(TestVarArgs.java:97)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at ...
at java.base/java.lang.Thread.run(Thread.java:1589)
test TestVarArgs.testVarArgs(119, "f0_V_IS_IFD", VOID, [INT, STRUCT], [INT, FLOAT, DOUBLE]): success
test TestVarArgs.testVarArgs(136, "f0_V_IS_FFP", VOID, [INT, STRUCT], [FLOAT, FLOAT, POINTER]): success
test TestVarArgs.testVarArgs(153, "f0_V_IS_DDI", VOID, [INT, STRUCT], [DOUBLE, DOUBLE, INT]): success
test TestVarArgs.testVarArgs(170, "f0_V_IS_PDF", VOID, [INT, STRUCT], [POINTER, DOUBLE, FLOAT]): success
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc: SuppressErrorAt=\code/vmreg.hpp:147
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (c:\dev\repos\java\forks\jdk\src\hotspot\share\code/vmreg.hpp:147), pid=10580, tid=10896
# assert(is_stack()) failed: Not a stack-based register
#
# JRE version: OpenJDK Runtime Environment (20.0) (slowdebug build 20-internal-adhoc.sawesong.jdk)
# Java VM: OpenJDK 64-Bit Server VM (slowdebug 20-internal-adhoc.sawesong.jdk, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, windows-aarch64)
# Core dump will be written. Default location: C:\dev\repos\java\forks\jdk\JTwork\scratch\0\hs_err_pid10580.mdmp
#
# An error report file with more information is saved as:
# C:\dev\repos\java\forks\jdk\JTwork\scratch\0\hs_err_pid10580.log
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
The problem turns out to be the fact that I had removed the vector registers from the list of input registers but the HFA code expects these to exist. The Windows AArch64 ABI also expected these vector registers to be used in this scenario. Restoring them addresses this bug, getting us back to the original failure (before I made any changes):
--------------------------------------------------
TEST: java/foreign/TestVarArgs.java
TEST JDK: C:\dev\java\abi\devbranch6\jdk
ACTION: build -- Passed. All files up to date
REASON: Named class compiled on demand
TIME: 0.015 seconds
messages:
command: build TestVarArgs
reason: Named class compiled on demand
elapsed time (seconds): 0.015
ACTION: testng -- Failed. Unexpected exit from test [exit code: 1]
REASON: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED -Dgenerator.sample.factor=17 TestVarArgs
TIME: 18.911 seconds
messages:
command: testng --enable-native-access=ALL-UNNAMED -Dgenerator.sample.factor=17 TestVarArgs
reason: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED -Dgenerator.sample.factor=17 TestVarArgs
Mode: othervm [/othervm specified]
elapsed time (seconds): 18.911
configuration:
STDOUT:
test TestVarArgs.testVarArgs(0, "f0_V__", VOID, [], []): success
test TestVarArgs.testVarArgs(17, "f0_V_S_DI", VOID, [STRUCT], [DOUBLE, INT]): success
test TestVarArgs.testVarArgs(34, "f0_V_S_IDF", VOID, [STRUCT], [INT, DOUBLE, FLOAT]): success
test TestVarArgs.testVarArgs(51, "f0_V_S_FDD", VOID, [STRUCT], [FLOAT, DOUBLE, DOUBLE]): success
test TestVarArgs.testVarArgs(68, "f0_V_S_DDP", VOID, [STRUCT], [DOUBLE, DOUBLE, POINTER]): success
test TestVarArgs.testVarArgs(85, "f0_V_S_PPI", VOID, [STRUCT], [POINTER, POINTER, INT]): success
STDERR:
java.lang.RuntimeException: java.lang.IllegalStateException: java.lang.AssertionError: expected [12.0] but found [2.8E-45]
at TestVarArgs.check(TestVarArgs.java:134)
at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:758)
at TestVarArgs.testVarArgs(TestVarArgs.java:104)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:578)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:599)
at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.testng.TestRunner.privateRun(TestRunner.java:764)
at org.testng.TestRunner.run(TestRunner.java:585)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
at org.testng.SuiteRunner.run(SuiteRunner.java:286)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
at org.testng.TestNG.runSuites(TestNG.java:1069)
at org.testng.TestNG.run(TestNG.java:1037)
at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:93)
at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:53)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:578)
at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:125)
at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: java.lang.IllegalStateException: java.lang.AssertionError: expected [12.0] but found [2.8E-45]
at CallGeneratorHelper.lambda$initStruct$10(CallGeneratorHelper.java:443)
at TestVarArgs.lambda$check$4(TestVarArgs.java:132)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at TestVarArgs.check(TestVarArgs.java:132)
... 32 more
Caused by: java.lang.AssertionError: expected [12.0] but found [2.8E-45]
at org.testng.Assert.fail(Assert.java:99)
at org.testng.Assert.failNotEquals(Assert.java:1037)
at org.testng.Assert.assertEqualsImpl(Assert.java:140)
at org.testng.Assert.assertEquals(Assert.java:122)
at org.testng.Assert.assertEquals(Assert.java:617)
at CallGeneratorHelper.lambda$makeArg$8(CallGeneratorHelper.java:413)
at CallGeneratorHelper.lambda$initStruct$10(CallGeneratorHelper.java:441)
... 35 more
Examining the test source shows that upcalls can also be traced using -XX:+TraceOptimizedUpcallStubs. I wonder how many other tests are failing though since I didn’t expect this failure. Rerunning them all results in these failures:
The bug is that reg2offset_out is called on a single physical register on line 5894! This happens because the src.is_single_phys_reg returns false. I break out the local variables to get an explicit breakdown in the debugger:
// A float arg may have to do float reg int reg conversion
void MacroAssembler::float_move(VMRegPair src, VMRegPair dst, Register tmp) {
VMReg src_first = src.first();
VMReg dst_first = dst.first();
if (src_first->is_stack()) {
if (dst_first->is_stack()) {
ldrw(tmp, Address(rfp, reg2offset_in(src.first())));
strw(tmp, Address(sp, reg2offset_out(dst_first)));
} else {
ldrs(dst.first()->as_FloatRegister(), Address(rfp, reg2offset_in(src_first)));
}
} else if (src_first != dst_first) {
bool src_is_single_phys_reg = src.is_single_phys_reg();
bool dst_is_single_phys_reg = dst.is_single_phys_reg();
bool src_is_float_reg = src_first->is_FloatRegister();
bool src_is_reg = src_first->is_Register();
bool dst_is_float_reg = dst_first->is_FloatRegister();
bool dst_is_reg = dst_first->is_Register();
if (src_is_single_phys_reg && dst_is_single_phys_reg)
fmovs(dst_first->as_FloatRegister(), src_first->as_FloatRegister());
else
strs(src_first->as_FloatRegister(), Address(sp, reg2offset_out(dst_first)));
}
}
Interestingly, the src register is a floating point register but the name is c_arg0. It is confusing to me that the regName field in both the source’s _first and _second fields point to the same location as the destination’s _first and _second VMRegImpl::regName pointers. Looking at the source, this makes sense because the regName pointer is a static field (missed this in WinDbg) and is set by the staticset_regName method.
Notice that ArgumentShuffle::ArgumentShuffle calls NativeCallingConvention::calling_convention, which in turn calls out_regs[i].set1(reg). The set1 method explicitly sets _second to BAD (which is first() – 1). set2() on the other hand sets _second to first() + 1. The solution is then to simply check whether the dst is a register since it will not be a single physical register in this scenario. This fix addresses the assertion failure. We should now be able to get downcall logging.
java.lang.Exception: Expected 2 but found 4621819117588971520
java.lang.Exception: Expected 0 but found 2
java.lang.Exception: Expected 13 but found 0
java.lang.Exception: Expected a but found
4621819117588971520 is 0x4024000000000000, nothing revealing about that value. The native functions that were invoked must be invoke_high_arity2, invoke_high_arity4, invoke_high_arity5 , and invoke_high_arity6 since they are the only ones that match those expected return values. I remove the loop to run invoke_high_arity2 only. Here’s a snippet of the downcall log:
Argument shuffle {
Move a int from (c_rarg2,BAD!) to (c_rarg0,BAD!)
Move a long from (c_rarg3,c_rarg3) to (c_rarg2,c_rarg2)
Move a float from (v1,BAD!) to (c_rarg3,BAD!)
Move a long from (c_rarg1,c_rarg1) to (rscratch2,rscratch2)
Move a double from (v0,v0) to (c_rarg1,c_rarg1)
Stack argument slots: 0
}
[CodeBlob (0x00000259e688df90)]
Framesize: 4
Runtime Stub (0x00000259e688df90): nep_invoker_blob
--------------------------------------------------------------------------------
Decoding CodeBlob, name: nep_invoker_blob, at [0x00000259e688e040, 0x00000259e688e118] 216 bytes
0x00000259e688e040: stp x29, x30, [sp, #-0x10]!
0x00000259e688e044: mov x29, sp
0x00000259e688e048: sub sp, x29, #0x10
0x00000259e688e04c: adr x9, #0x0
0x00000259e688e050: str x9, [x28, #0x318]
0x00000259e688e054: mov x9, sp
0x00000259e688e058: str x9, [x28, #0x310]
0x00000259e688e05c: str x29, [x28, #0x320]
;; 0x4
0x00000259e688e060: orr x9, xzr, #0x4
0x00000259e688e064: add x10, x28, #0x3c4
0x00000259e688e068: stlr w9, [x10]
;; { argument shuffle
;; bt=int
0x00000259e688e06c: sxtw x0, w2
;; bt=long
0x00000259e688e070: mov x2, x3
;; bt=float
0x00000259e688e074: fmov w3, s1
;; bt=long
0x00000259e688e078: mov x9, x1
;; bt=double
0x00000259e688e07c: fmov x1, d0
;; } argument shuffle
0x00000259e688e080: blr x9
Notice that the instructions correctly load the registers x0-x3. The question now is where the return value is used after this function. Here are the rest of the instructions:
I needed to search for B.cond in the ARM Architecture Reference Manual for A-profile architecture PDF. The HI mnemonic in b.hi means unsigned higher and is equivalent to the condition flags C==1 && Z == 0. This branch is to the safepoint poll slow path, which is the label immediately following the L_safepoint_poll_slow_path comment. I found it strange that 0x00000259e688e0a0 + #0x3c = 0x259E688E0DC, which is the 2nd instruction after the L_safepoint_poll_slow_path label. However, the B.cond documentation states that the program label to be conditionally branched to is given by an offset from the address of the branch instruction.
Looks like most of the above code is not relevant because it doesn’t touch x0. At this point, it seems like the problem could be in the native code we’re branching into. I set a breakpoint in invoke but the code doesn’t seem to make much sense:
bp intrinsics!invoke_high_arity2
Let us disassemble support/test/jdk/jtreg/native/lib/Intrinsics.dll and see what the compiler generated.
cd build\windows-aarch64-server-slowdebug\support\test\jdk\jtreg\native\support\libIntrinsics\
dumpbin /disasm /out:Intrinsics.asm libIntrinsics.obj
dumpbin /all /out:Intrinsics.txt libIntrinsics.obj
Here is the relevant code, which makes it apparent that libIntrinsics is not expecting floating point parameters in general purpose registers!
I update the WindowsAArch64CallArranger to specifically use general purpose registers for floating point data only for variadic FunctionDescriptors. This fixes both TestIntrinsics and TestUpcallHighArity but not TestVarArgs so I create a self contained test for it: MinimizedTestVarArgs.
TestVarArgs
This test depends on the native varargs.dll (built from libVarArgs.c). This DLL can be found in the build/windows-x86_64-server-slowdebug/support/test/jdk/jtreg/native/lib/ directory.
How does the test work?
It uses upcalls, how do they work?
Here’s how the native upcall linker is invoked to create an upcall stub:
These logging options generate argument shuffling output only. I expected to see comments like on_entry.
[8.157s][trace][foreign,upcall] Argument shuffle {
[8.157s][trace][foreign,upcall] Move a long from (c_rarg1,c_rarg1) to (c_rarg3,c_rarg3)
[8.157s][trace][foreign,upcall] Move a int from (c_rarg0,BAD!) to (c_rarg2,BAD!)
[8.157s][trace][foreign,upcall] Stack argument slots: 0
[8.158s][trace][foreign,upcall] }
[8.860s][trace][foreign,downcall] Argument shuffle {
[8.860s][trace][foreign,downcall] Move a long from (c_rarg1,c_rarg1) to (rscratch2,rscratch2)
[8.860s][trace][foreign,downcall] Move a int from (c_rarg3,BAD!) to (c_rarg1,BAD!)
[8.860s][trace][foreign,downcall] Move a long from (c_rarg2,c_rarg2) to (c_rarg0,c_rarg0)
[8.862s][trace][foreign,downcall] Stack argument slots: 0
[8.862s][trace][foreign,downcall] }
[8.862s][trace][foreign,downcall] [CodeBlob (0x0000027b876f0810)]
[8.862s][trace][foreign,downcall] Framesize: 2
[8.862s][trace][foreign,downcall] Runtime Stub (0x0000027b876f0810): nep_invoker_blob
[8.862s][trace][foreign,downcall] --------------------------------------------------------------------------------
[8.862s][trace][foreign,downcall] Decoding CodeBlob, name: nep_invoker_blob, at [0x0000027b876f08c0, 0x0000027b876f0980] 192 bytes
[8.879s][trace][foreign,downcall] 0x0000027b876f08c0: stp x29, x30, [sp, #-0x10]!
[8.879s][trace][foreign,downcall] 0x0000027b876f08c4: mov x29, sp
...
That is not sufficient though. Simply outputs this to the command prompt:
[CodeBlob (0x0000025291ffe090)]
Framesize: 0
UpcallStub (0x0000025291ffe090) used for upcall_stub_(Ljava/lang/Object;IJ)V
[CodeBlob (0x0000025291ffe090)]
Framesize: 0
UpcallStub (0x0000025291ffe090) used for upcall_stub_(Ljava/lang/Object;IJ)V
...
The UpcallStub constructor turns out to have the UpcallStub tracing code (notice the stub name “UpcallStub”). It expects the PrintStubCode flag. This outputs the disassembly as I expected but does so for just about everything – 10MB of text. The stub name can be used to narrow down the calls we’re interested in.
cd build\windows-aarch64-server-slowdebug\support\test\jdk\jtreg\native\support\libVarArgs\
dumpbin /disasm /out:libVarArgs.asm libVarArgs.obj
dumpbin /all /out:libVarArgs.txt libVarArgs.obj
Setting aside all this learning and simply reviewing the Overview of ARM64 ABI conventions, the statement that floating-point values are returned in s0, d0, or v0, as appropriate should be enough to track down the bug. The change I made to the CallArranger switched the floating point storage to a general purpose register whenever floating point storage was requested for a variadic function. However, this doesn’t fix the test, thereby showing the value of understanding exactly how things are flowing through registers!
Understanding libVarArgs
The varargs function does not return a value. Here is an interpretation of the disassembly:
;$LN2:
;;
;; i++
;;
0000000000000044: B9400BE8 ldr w8,[sp,#8]
0000000000000048: 11000508 add w8,w8,#1
000000000000004C: B9000BE8 str w8,[sp,#8]
$LN4:
;;
;; i < num
;;
0000000000000050: B9401FE9 ldr w9,[sp,#0x1C]
0000000000000054: B9400BE8 ldr w8,[sp,#8]
0000000000000058: 6B09011F cmp w8,w9
000000000000005C: 5400F66A bge $LN3
;;
;; x8 = info
;;
0000000000000060: F9401FE8 ldr x8,[sp,#0x38]
;;
;; x10 = &info->argids
;;
0000000000000064: 9100210A add x10,x8,#8
;;
;; x9 = i * 4
;;
0000000000000068: B9400BE8 ldr w8,[sp,#8]
000000000000006C: 93407D09 sxtw x9,w8
0000000000000070: D2800088 mov x8,#4
0000000000000074: 9B087D29 mul x9,x9,x8
;;
;; Get the pointer from the call_info
;;
0000000000000078: F9400148 ldr x8,[x10]
;;
;; computer the offset of element [i]
;;
000000000000007C: 8B090108 add x8,x8,x9
;;
;; w8 = info->argids[i];
;;
0000000000000080: B9400108 ldr w8,[x8]
0000000000000084: B90023E8 str w8,[sp,#0x20]
0000000000000088: B94023E8 ldr w8,[sp,#0x20]
000000000000008C: B9001BE8 str w8,[sp,#0x18]
0000000000000090: B9401BE8 ldr w8,[sp,#0x18]
;;
;; There are 88 (0x58) enums.
;;
0000000000000094: 71015D1F cmp w8,#0x57
;;
;; Go to default case if not one of the defined enums
;;
0000000000000098: 5400F3E8 bhi $LN95
;;
;; w10 = info->argids[i];
;;
000000000000009C: B9401BEA ldr w10,[sp,#0x18]
;;
;; x9 = PC-relative address of $LN100
;;
00000000000000A0: 1000F509 adr x9,$LN100
;;
;; uxtw: unsigned word extend
;; load a signed offset from the table at $LN100
;; x8 = sign-extend([x9 + w10 * 4])
;;
00000000000000A4: B8AA5928 ldrsw x8,[x9,w10 uxtw #2]
;;
;; x9 = PC-relative address of $LN51 (half-way point in the switch/45th label from here)
;;
00000000000000A8: 10007969 adr x9,$LN51
;;
;; x8 = address of the case statement to jump to
;; why the left shift though?
;;
00000000000000AC: 8B080928 add x8,x9,x8,lsl #2
00000000000000B0: D61F0100 br x8
...
$LN95:
0000000000001F14: 12800000 mov w0,#-1
0000000000001F18: 90000008 adrp x8,__imp_exit
0000000000001F1C: F9400108 ldr x8,[x8,__imp_exit]
0000000000001F20: D63F0100 blr x8
$LN188:
0000000000001F24: 17FFF848 b $LN2
;; va_end(a_list);
;; This expands to ((void)(a_list = (va_list)0))
;;
$LN3:
0000000000001F28: D2800008 mov x8,#0
0000000000001F2C: F90003E8 str x8,[sp]
;;
;; cleanup before returning
;;
0000000000001F30: 9132C3FF add sp,sp,#0xCB0
0000000000001F34: 94000000 bl __security_pop_cookie
0000000000001F38: A8C47BFD ldp fp,lr,[sp],#0x40
0000000000001F3C: D65F03C0 ret
$LN100:
0000000000001F40: FFFFFC38
$LN101:
0000000000001F44: FFFFFC49
The unconditional branch to the address in x8 is to the upcall stub.Notice from the setup for the branch that the target is invoked by the blr.
Stepping through the code, I decide to look up the void* parameter that was passed into the upcall stub (just before the last instruction of preserve_callee_saved_regs – str d24, [sp, #0xd0]). Perhaps a more reasonable point would be at the end of the argument shuffle but the values will be the same ones below:
The 64-bit value is 0x4038000000000000. The program below confirms this value to be 24.0. Therefore, everything has been correctly set up for the upcall.
#include <stdio.h>
int main()
{
__int64 i = 0x4038000000000000;
double* d = (double*)&i;
printf("%f", *d);
}
Review earlier 0x4024 value.
Review set of volatile registers defined by the ABI since that’s what ends up in the upcall stub.
A good place to break is jvm!UpcallLinker::on_entry
Why don’t we review how these cases are handled in the native code? Here is the definition of va_arg from C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31823\include\vadefs.h:
Below is the disassembly for the first case in libVarArgs.c. The 2nd definition of __crt_va_arg is used on ARM64. The _SLOTSIZEOF evaluates to 8 for both int and double. TODO: finish explaining this assembly.
So why does TestUpcallArity pass? It does not use variadic functions! I update MinimizedTestVarArgs to show the function signature codes when it fails. From the resulting log, a struct is being passed to the downcall.
f0_V_S_F java.lang.Exception: Expected 12.0 but found 7.95336E-11
f0_V_S_D java.lang.Exception: Expected 24.0 but found 9.022351855793E-312
f0_V_S_FF java.lang.Exception: Expected 12.0 but found 2.2120472E-11
f0_V_S_FF java.lang.Exception: Expected 12.0 but found 5.96E-43
f0_V_S_DD java.lang.Exception: Expected 24.0 but found 9.02227530708E-312
f0_V_S_DD java.lang.Exception: Expected 24.0 but found 4.9E-324
f0_V_S_FFF java.lang.Exception: Expected 12.0 but found 2.384152E-12
f0_V_S_FFF java.lang.Exception: Expected 12.0 but found 5.96E-43
f0_V_S_FFF java.lang.Exception: Expected 12.0 but found 1.4E-45
f0_V_S_DDD java.lang.Exception: Expected 24.0 but found 9.020261611475E-312
f0_V_S_DDD java.lang.Exception: Expected 24.0 but found 9.02168631996E-312
f0_V_S_DDD java.lang.Exception: Expected 24.0 but found 1.8075E-319
f0_V_IS_F java.lang.Exception: Expected 12.0 but found 2.8E-45
f0_V_IS_D java.lang.Exception: Expected 24.0 but found 9.9E-324
f0_V_IS_FF java.lang.Exception: Expected 12.0 but found 2.8E-45
f0_V_IS_FF java.lang.Exception: Expected 12.0 but found 0.0
f0_V_IS_DD java.lang.Exception: Expected 24.0 but found 9.9E-324
f0_V_IS_DD java.lang.Exception: Expected 24.0 but found 2.08E-322
f0_V_IS_FFF java.lang.Exception: Expected 12.0 but found 2.8E-45
f0_V_IS_FFF java.lang.Exception: Expected 12.0 but found 0.0
f0_V_IS_FFF java.lang.Exception: Expected 12.0 but found 5.9E-44
These signatures remind me of seeing 24.0 in d0 when debugging. I didn’t think about this as much as I should have. Breaking on the branch to the address from the table is the best way to examine the state of the registers and notice 24.0 in d0. Interestingly, only the general purpose registers are shown. See r (Registers) – Windows drivers | Microsoft Docs for details on how to view and modify additional registers.
bp VarArgs!varargs+0xb0
r
rF
The pattern in the above failing signatures implies that the UnboxBindingCalculator is using the STRUCT_HFA case to place them in floating point registers. Changing the code to use the STRUCT_REGISTER case for these causes some of the cases to pass (updated MinimizedTestVarArgs as well). The last case doesn’t work though..
Starting test 6 for f0_V_S_F ... Finished test 6 for f0_V_S_F
Starting test 7 for f0_V_S_D ... Finished test 7 for f0_V_S_D
Starting test 14 for f0_V_S_FF ... Finished test 14 for f0_V_S_FF
Starting test 19 for f0_V_S_DD ... Finished test 19 for f0_V_S_DD
Starting test 46 for f0_V_S_FFF ... Finished test 46 for f0_V_S_FFF
Starting test 67 for f0_V_S_DDD ...
My initial hypothesis is that there weren’t enough registers, but if that’s the case then why does the 3 floats case work? The above bp command in the debugger shows that $LN73 of VarArgs.dll is executed and that the integer registers contain the 4 floating point values (why 5 and not 3)? Turns out the reason the test failed to be complete is because there was an AccessViolation when loading the pair x8 and x9 from [x10].
At this point, my curiosity about the correct solution for these registers leads me to create a self-contained varargs test SimpleVarArgs.c. The disassembly of call_S_DDD shows the struct being placed on the stack and a pointer to it being passed to varargs.
Volatile registers are scratch registers presumed by the caller to be destroyed across a call. Nonvolatile registers are required to retain their values across a function call and must be saved by the callee if used.
Just when I think I’m done fixing up the CallArranger so that all the Windows AArch64 floating point ABI changes are in there, I realize when going through the other changes in the PR I would open that I don’t understand exactly what WindowsAArch64VaList is used for. I based it on the MacOsAArch64VaList class but perhaps WinVaList would be more appropriate.
While reviewing all this, I take a peek at the CallArranger tests. All but one of them use CallArranger.LINUX. This means I need to create a test for Windows. After replacing LINUX with WINDOWS, I run the test on the Surface Pro X and it passes, even though it should definitely fail! Oh boy, this turns out to be a copy/paste issue – I hadn’t updated the @run testng ClassName to the new class name so a different test was running!
Structure of CallArranger Tests
testStructHFA1 creates a struct with 2 floats for a downcall. One of the arrays it passes to checkArgumentBindings starts off with the dup() binding, which “duplicates the value on the top of the operand stack (without popping it!), and pushes the duplicate onto the operand stack.“
Breaking Down WinVaList
As part of this port, I needed to implement VaList. Understanding the Windows x64 implementation (WinVaList) is helpful. The skip() method repeatedly calls MemorySegment.asSlice() to create a memory segment offset by VA_SLOT_SIZE_BYTES. WinVaList.Builder also uses VA_SLOT_SIZE_BYTES for each argument whereas MacOsAArch64VaList.Builder uses the sizeOf method to compute the slot sizes for the arguments. The definition of Utils.alignUp (shown below) is what I thought the builder was using but it is actually SharedUtils.alignUp.
// Utils.alignUp
public static long alignUp(long n, long alignment) {
return (n + alignment - 1) & -alignment;
}
// SharedUtils.alignUp
public static long alignUp(long addr, long alignment) {
return ((addr - 1) | (alignment - 1)) + 1;
}
// Compare these to _SLOTSIZEOF(t) in vadefs.h
#define _SLOTSIZEOF(t) ((sizeof(t) + _VA_ALIGN - 1) & ~(_VA_ALIGN - 1))
This enables the AArch64 implementation to align up the size required for STRUCT_REGISTER and STRUCT_HFA layouts. This also matches the definition of Visual Studio’s __crt_va_arg in vadefs.h. The Builder.build() method uses MemorySegment.copyFrom().
It’s only when I start preparing to engage the OpenJDK mailing lists about a PR that I discover that there’s a separate repo for the Foreign Function & Memory API development so I need to apply my changes onto my new fork of the panama-foreign repo.
There were some conflicts to resolve after cherry-picking but nothing too bad. Looks like I didn’t have the commits starting from July when I was changing the TestAArch64CallArranger.
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc: SuppressErrorAt=\foreignGlobals_aarch64.cpp:181
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (d:\dev\repos\java\forks\panama-foreign\src\hotspot\cpu\aarch64\foreignGlobals_aarch64.cpp:181), pid=18972, tid=18908
# Error: ShouldNotReachHere()
#
# JRE version: OpenJDK Runtime Environment (20.0) (slowdebug build 20-internal-adhoc.sawesong.panama-foreign)
# Java VM: OpenJDK 64-Bit Server VM (slowdebug 20-internal-adhoc.sawesong.panama-foreign, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, windows-aarch64)
# Core dump will be written. Default location: C:\dev\repos\java\forks\panama-foreign\JTwork\scratch\0\hs_err_pid18972.mdmp
#
# An error report file with more information is saved as:
# C:\dev\repos\java\forks\panama-foreign\JTwork\scratch\0\hs_err_pid18972.log
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
The minimized tests I created are now out of date as well, e.g. History for test/jdk/java/foreign/TestIntrinsics.java – openjdk/panama-foreign (github.com) has 2 commits showing the changes I need to make in addition to copying the DLL from support\test\jdk\jtreg\native\lib. Suprisingly, WinDbg cannot open the executable as it did earlier. I’m launching it from C:\Program Files (x86)\Windows Kits\10\Debuggers\arm64\windbg.exe.
WinDbg could not create process
Perhaps it’s the wrong one for the current Windows version? Search for “debugger” in the store and install the WinDbg Preview app.
WinDbg Preview
Now we can set the breakpoint in foreignGlobals_aarch64.cpp:
bp jvm!move_v128
g
u jvm!move_v128
Here is the call stack when the breakpoint is hit:
Why isn’t using fmovd only failing for some test using a floating point argument?
Are my macroAssembler instructions really necessary?
Where is a test showing these instructions in use? MinimizedTestIntrinsics (run above)
Building on macOS
A newer boot JDK is required once again as explained by the error message when running bash configure. Download and install the macOS .pkg installer for JDK 19 from the adoptium site.
checking for java... /usr/bin/java
configure: Found potential Boot JDK using java(c) in PATH
configure: Potential Boot JDK found at /usr is incorrect JDK version (openjdk version "17.0.1" 2021-10-19 LTS OpenJDK Runtime Environment Microsoft-28056 (build 17.0.1+12-LTS) OpenJDK 64-Bit Server VM Microsoft-28056 (build 17.0.1+12-LTS, mixed mode)); ignoring
configure: (Your Boot JDK version must be one of: 19 20)
Testing 4-Float HFAs
I was reviewing the tests I added and realized that I wasn’t testing the variadic HFAs. Sure enough, I couldn’t get the tests for variadic HFA structs with 4 floats to pass. My code was assigning 2 64-bit general purpose registers to such a struct. Why isn’t this caught by one of the existing tests? TestVarArgs appears to simply pass the struct to the native code in the downcall and the native code passes it back in the upcall. Shouldn’t there be additional validation? testFloatStruct in VaListTest also looks like it should catch this. Is the problem that it only uses structs on the stack? Disassemble libVaList to find out:
cd build\windows-aarch64-server-slowdebug\support\test\jdk\jtreg\native\support\libVaList\
dumpbin /disasm /out:libVaList.asm libVaList.obj
dumpbin /all /out:libVaList.txt libVaList.obj
When the debugger was done loading, I ran these commands to set a breakpoint in the native code invoked by VaListTest. Unfortunately, the breakpoint was not hit. Why this happens is still a mystery.
bp VaList!sumFloatStruct
g
Adding the HFA Field Values
The function descriptor for the downcall to the native sum_struct_hfa_floats function is created by calling FunctionDescriptor.of with C_FLOAT as the first argument. This allows the result of the invokeWithArguments method of the downcall’s MethodHandle to be cast to a float. Using C_INT, for example, results in this error: ClassCastException: java.lang.Integer cannot be cast to class java.lang.Float.
Validating the HFA Field Values
Although the existing varargs tests passed, they looked like they checked round-tripping of a single value. Adding the components of the HFA seemed like a better idea because it verified that all the values were delivered correctly. This caught a bug in my implementation – when there aren’t enough registers for a HFA being passed to a variadic function, the struct was partially loaded into the available registers and then the rest of the struct was spilled onto the stack. This behavior differs from the macOS & Linux environments and wasn’t caught by any of the existing tests.
In the process of testing these changes, I deployed the locally built JDK to the Surface Pro X and got this cryptic error message:
C:\dev\java\abi\devbranch35\jdk\bin\java.exe --enable-preview SumVariadicStructHfa
WARNING: A restricted method in java.lang.foreign.Linker has been called
WARNING: java.lang.foreign.Linker::nativeLinker has been called by the unnamed module
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
Exception in thread "main" java.lang.UnsatisfiedLinkError: C:\dev\repos\scratchpad\compilers\tests\aarch64\abi\varargs\VarArgs.dll: Can't load ARM 64-bit .dll on a AMD 64-bit platform
at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:331)
at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:197)
at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:139)
at java.base/jdk.internal.loader.NativeLibraries.findFromPaths(NativeLibraries.java:259)
at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:251)
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2437)
at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:873)
at java.base/java.lang.System.loadLibrary(System.java:2047)
at SumVariadicStructHfa.<clinit>(SumVariadicStructHfa.java:61)
Turns out I deployed x64 binaries to the Surface Pro X and launched Java in a folder containing the prior ARM64 varargs test DLL. The solution was to delete that DLL and copy the DLL from the new build. The test passed successfully and it’s only then that I realized that x64 binaries run successfully on this ARM64 platform. Getting the correct ARM64 binaries in place without replacing the x64 varargs will give a similar error Exception in thread "main" java.lang.UnsatisfiedLinkError: C:\dev\repos\scratchpad\compilers\tests\aarch64\abi\varargs\VarArgs.dll: Can't load AMD 64-bit .dll on a ARM 64-bit platform.
Outstanding Questions
Why invoke and instead of invokeExact in the tests?
What happens if we return the method handle without the .asSpreader call?
Why do we need to shuffle the PrintfArgs?
Remove dead code
Show how to debug (VS/VS Code) into the native code (on Windows x64 first, then ARM64).
Generate logs showing the wrong downcall registers in use without my changes
Generate logs showing the wrong upcall registers in use without my changes
Make foreign+upcalls log the upcall stub details as is done for the downcall stubs.
Why does using r10 as the retBufAddrStorage field work on Windows? Is there not test for returning a struct?
Create test that returns a 16-byte result and verify that it is in x1:x0 (no tests failed with this change).
Create test that returns result in address stored in x8 – see Return Values: For types greater than 16 bytes, the caller shall reserve a block of memory of sufficient size and alignment to hold the result. The address of the memory block shall be passed as an additional argument to the function in x8. The callee may modify the result memory block at any point during the execution of the subroutine. The callee isn’t required to preserve the value stored in x8. How does this compare to the comments in assembler_aarch64.hpp, downcallLinker_aarch64.cpp, stubGenerator_aarch64.cpp?
Create test that uses r16-r17 and v24 and verify that they really are volatile.
Fix d24 not being a volatile register
Why doesn’t any test fail without the cursor update in MacOsAArch64VaList.Builder.read?
I recently had to investigate an OpenJDK google test. To run the test locally, I needed to ensure that configure is aware of my intent. As documented at jdk/building.md · openjdk/jdk (github.com), we need to pass the --with-gtest option to configure. We first need to get the appropriate googletest sources, e.g (in Git Bash):
cd /c/dev/repos
git clone -b release-1.8.1 https://github.com/google/googletest
Then in Cygwin:
cd /cygdrive/d/java/forks/jdk
bash configure --with-gtest=/cygdrive/c/dev/repos/googletest --with-debug-level=slowdebug
Once this is done, the OpenJDK repo can be built using this script. I use the time command to get statistics on how long the build took. I also only just discovered that the prompt can be configured to include the time.
time /cygdrive/d/dev/repos/scratchpad/scripts/java/cygwin/build-jdk.sh
The googletest launcher is in the images folder of the build configuration:
An interesting observation is that the JVM test code is in build/windows-x86_64-server-slowdebug/images/test/hotspot/gtest/server/jvm.dll, which is just over 5 MB larger than build/windows-x86_64-server-slowdebug/jdk/bin/server/jvm.dll. Here’s a snippet of the call stack showing how the tests get kicked off.
jvm.dll!JVMInitializerListener::OnTestStart(const testing::TestInfo & test_info) Line 129
...
jvm.dll!RUN_ALL_TESTS() Line 2342 C++
jvm.dll!runUnitTestsInner(int argc, char * * argv) Line 289 C++
jvm.dll!runUnitTests(int argc, char * * argv) Line 370 C++
gtestLauncher.exe!main(int argc, char * * argv) Line 40 C++
[Inline Frame] gtestLauncher.exe!invoke_main() Line 78 C++
gtestLauncher.exe!__scrt_common_main_seh() Line 288 C++
kernel32.dll!BaseThreadInitThunk...
Behind the Scenes
My first attempt at running the gtests was to launch them using the gtestLauncher from a build I was testing but using a locally built JDK:
The logging I added to my local gtest was not showing up in the output. Naturally, the question that arose was how do I know which binaries it is running against since I don’t see the logging I expected? Process Explorer and Process Monitor did not seem to have a way to show me all the DLLs in the process (before it terminated). I end up creating a dump file using Process Explorer. Here are the non-Windows binaries – a mix of local build and CI build DLLS.
DLLs Loaded in gTestLauncher.exe
This was what inspired me to figure out how to run the whole show with locally built binaries as described in the main section of this post.
llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\try_lock_shared_for.pass.cpp - The system cannot find the path specified.
llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\try_lock_shared_until.pass.cpp - The system cannot find the path specified.
llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\try_lock_until_deadlock_bug.pass.cpp - The system cannot find the path specified.
These files still exist on disk though! They are displayed if you dir their containing directory but are not found if you dir their full paths! They cannot be deleted using del either. Interestingly, pressing tab after the directory path will autocomplete the file names.
C:\> dir D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\
Volume in drive D is DATAVOL1
Volume Serial Number is 8800-8693
Directory of D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class
10/16/2022 01:22 PM <DIR> .
10/04/2022 03:29 AM <DIR> ..
10/04/2022 03:29 AM 2,461 try_lock_shared_for.pass.cpp
10/04/2022 03:29 AM 2,423 try_lock_shared_until.pass.cpp
10/04/2022 03:29 AM 2,146 try_lock_until_deadlock_bug.pass.cpp
3 File(s) 7,030 bytes
C:\> dir D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\try_lock_shared_for.pass.cpp
Volume in drive D is DATADRIVE1
Volume Serial Number is 548C-FFC9
Directory of D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class
File Not Found
C:\> del D:\dev\repos\llvm\dups\llvm-project\llvm\utils\release\llvm_package_15.0.2\llvm-project-llvmorg-15.0.2\libcxx\test\std\thread\thread.mutex\thread.mutex.requirements\thread.sharedtimedmutex.requirements\thread.sharedtimedmutex.class\try_lock_shared_for.pass.cpp
The system cannot find the path specified.
These files can be viewed in file explorer. Something that caught my eye when examining their properties is that their locations started with the \\?\ prefix! That seems unusual for files on my local drive.
File Locations Starting with \\?\
Resource monitor does not show any images with associated handles when searching for “try_lock”. Neither does searching for “\?\D:\dev\repos\llvm\”. I tried using Process Explorer’s “Find Handle or DLL…” command as well. There also don’t appear to be any child processes for the cmd.exe process I was using (a Developer 2019 Command Prompt).
Next idea, open Process Monitor and see what’s happening when dir and rmdir are executed. I used the Path contains thread.sharedtimedmutex.class filter. The deletes are showing up as SetDispositionInformationFile events and seem to be using the RemoveDirectoryW function.
The RemoveDirectory function marks a directory for deletion on close. Therefore, the directory is not removed until the last handle to the directory is closed.
Notice the NOT EMPTY result of the SetDispositionInformationFile event. I believe this comes from RemoveDirectoryW. There’s the question of how the 3 files are printed to the command line. The FindNextFile API is used to search for files.
Ah, in the middle of this investigation, PowerShell.exe dies and so does Windows Terminal. All my tabs, everything, gone! Aaaargh… Windows Event Viewer has an Information level event showing that powershell.exe crashed due to a System.InvalidOperationException. This is then followed by another Information event with the WER source and P1 problem signature Microsoft.WindowsTerminal_1.15.2713.0_x64__8wekyb3d8bbwe. Then comes the Error level event with the Application Hang source and General explanation that “The program WindowsTerminal.exe version 1.15.2209.28003 stopped interacting with Windows and was closed. To see if more information about the problem is available, check the problem history in the Security and Maintenance control panel.” The ExeFileName is cut off below but simply append “\WindowsTerminal.exe” to the package name to reconstruct it. Looks like I need to avoid PowerShell. And why is there no crash dump created for it???
One upside of this crash is that it lets me confirm that it is not the cmd.exe process that is hanging onto those files. I terminate explorer.exe and when I run new task in Task Manager, it asks me to create a Windows Hello pin. What is happening?? Now moving on to opening these files: Notepad++ acts as though nothing happened when you File->Open and select one of them. Notepad opens it though! Running cat in Git Bash also dumps its contents:
cat /d/dev/repos/llvm/dups/llvm-project/llvm/utils/release/llvm_package_15.0.2/llvm-project-llvmorg-15.0.2/libcxx/test/std/thread/thread.mutex/thread.mutex.requirements/thread.sharedtimedmutex.requirements/thread.sharedtimedmutex.class/try_lock_shared_for.pass.cpp
//===----------------------------------------------------------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
...
I’m suspecting Windows Defender but don’t have any definitive proof. According to Enable attack surface reduction (ASR) rules, this command in an admin powershell should do the trick if the problem was Windows Defender:
Well, looks like rm from Git Bash works just fine as does deleting from file explorer. Unfortunately, this unsolved issue is probably going to continue to cause pain in command prompt batch files like the LLVM build script.
I had previously installed the 11.1 toolkit on my Surface Book so I started by uninstalling all apps that showed up when searching for “nvidia” under “Installed Apps” except NVIDIA Graphics Driver 461 and NVIDIA Update 38.0.2.0. I then got the new installer from Installation Guide Windows :: CUDA Toolkit Documentation (nvidia.com) and installed every component presented by the installer. Note that older builds can be found at the CUDA Toolkit Archive.
Installed NVIDIA CUDA Components
You can now create a new CUDA project in Visual Studio:
Visual Studio’s Create New Project Dialog
Surface Book 2 CUDA Issues
Creating and running a CUDA 11.8 Runtime project on my Surface Book 2 fails with the error cudaSetDevice failed! Do you have a CUDA-capable GPU installed?addWithCuda failed! A search for using nvidia GPU on surface book 2 leads to suggestions that involve the NVIDIA Control Panel. Unfortunately, it doesn’t start on my laptop. A peek at the event viewer reveals why:
Opening the dump file in Visual Studio to see what’s going on is not helpful because there are no symbols available for the NVIDIA binaries. The NVIDIA Driver Symbol Server even says that it does not have PDBs (even though that’s for drivers) so this is not an optimistic path. The trimmed callstack of the main thread from the dump is shown below though. The paths to the NVIDIA binaries are C:\Program Files\WindowsApps\NVIDIACorp.NVIDIAControlPanel_8.1.962.0_x64__56jybvy8sckqj\nvcplui.exe and C:\Windows\System32\DriverStore\FileRepository\nvmsoui.inf_amd64_8fd9664c41d93f19\nvgames.dll
> nvcplui.exe!00007ff756d547f5 Unknown
nvcplui.exe!00007ff756d529c7 Unknown
nvcplui.exe!00007ff756d09f57 Unknown
KERNELBASE.dll!UnhandledExceptionFilter C
[Inline Frame] ntdll.dll!RtlpThreadExceptionFilter C
...
ntdll.dll!RtlRaiseException C
[External Code]
nvgames.dll!00007ffd372ba7d2 Unknown
...
nvgames.dll!00007ffd36ffd59f Unknown
combase.dll!???::CreateInstance C++
...
[Inline Frame] combase.dll!CoCreateInstanceEx C++
combase.dll!CoCreateInstance C++
nvcplui.exe!00007ff756afdf63 Unknown
...
nvcplui.exe!00007ff756d08f63 Unknown
kernel32.dll!BaseThreadInitThunk C
ntdll.dll!RtlUserThreadStart C
Launching it again errors with a dialog claiming that an NVIDIA graphics card was not detected in my system. Check out the language too…
Sure enough, device manager no long shows the GTX 1060 in the list of display adapters.
Rebooting restores the GTX 1060 but doesn’t address the crash in the NVIDIA Control Panel so I decide to move to my workstation and everything is much smoother there. The new Visual Studio CUDA project runs to completion so I turn my attention back to the CUDA installer to work on resolving the Surface Book 2 issues. The first thing I notice is that the installer is not keyboard accessible, so here’s a detour…
NVIDIA Installer Accessibility Issues
Is the NVIDIA Installer narrator-friendly? Narrator informs me that there are new natural voices available so I install them (Microsoft Aria, Guy, and Jenny).
CUDA 11.6 Driver Components
Looks like narrator works with the installer. However, the installer cannot be used via keyboard alone due to these issues:
You cannot TAB out of the NVIDIA software license agreement.
Narrator doesn’t read the captions below the Express and Custom radio buttons on the Installation Options page.
You cannot TAB into the components tree to select them via keyboard.
Keyboard navigation works after clicking on a component but the focus goes back to the NEXT button after using ALT+TAB to switch to another program then back.
Narrator reads the individual components, e.g. “NSight Systems, Selected” regardless of whether the checkbox is ticked or not. How does one know it’s a checkbox?
The custom installation components columns are not resizable (Component, New Version, and Current Version). For example, what NVIDIA GeForce Experience compo…
Why isn’t it resizable?
A general usability issue: why do all the NVIDIA components need to be uninstalled individually instead of having an option to remove everything?
Outstanding Questions
How do we figure out which component installed the NVIDIA Control Panel? One approach is to uninstall the existing components until the control panel binary from the dump file is deleted on disk. Removing NVIDIA NSight Systems 2022.4.2 removed the C:\Program Files\WindowsApps\NVIDIACorp.NVIDIAControlPanel_8.1.962.0_x64__56jybvy8sckqj\ directory. However, installing only this component in 11.6 did not bring back the NVIDIA control panel!
The installer asks for a path to a temp directory to unpack setup file into. Could examining that folder help determine where the control panel is coming from?
Was this installer generated by NSIS?
Resolution
I end up uninstalling all “nvidia” components on the Installed Apps page except NVIDIA Graphics Driver 461.40 then installing all components from CUDA 11.6. This finally has a working control panel!
NVIDIA Control Panel from CUDA 11.6 Installer
Surprisingly, this executable is in C:\Program Files\WindowsApps\NVIDIACorp.NVIDIAControlPanel_8.1.962.0_x64__56jybvy8sckqj, the same directory as 11.8! This must not have been the buggy component! Here is the version info for the 2 NVIDIA binaries in the earlier crash dump (nvgames.dll is now in C:\Windows\System32\DriverStore\FileRepository\nvmsoui.inf_amd64_ed4d74dfae95b5e6):
nvcplui.exe Properties
Visual Studio 2022 does not have the new CUDA project option though. However, changing the paths (in the .vcxproj) for my new project created using the 11.8 tools on my VS 2022 desktop makes the program work. Looks like I need to use 11.7 instead so I uninstall all the “nvidia” components except the NVIDIA Control Panel and the NVIDIA Graphics Driver 511.23 before installing 11.7. Thankfully, 11.7 works just fine!
Once these are installed, run the commands below in a Visual Studio 2022 Developer Command Prompt to build zlib for Windows. The x64 directory will contain zlibwapi.dll, which can be renamed to zlib.dll according to zlib/readme.txt (from the latest commit as of this post).
Unfortunately (or maybe fortunately?), I didn’t see any binaries at zlib. There is a link to the zlib GitHub repo though and zlib/DLL_FAQ.txt at master · madler/zlib (github.com) says to review the zlib site for an alternative download location. Sure enough, it does have a link to zlib for Windows 9x/NT/2000/XP/2003 (DLL version, plus related utilities). That doesn’t inspire much confidence in the binaries though… Might as well build them myself.
Investigating How to Build zlib
Open a Visual Studio Developer Command Prompt then build the project I saw in the docs:
git clone https://github.com/madler/zlib
cd zlib/contrib/vstudio/vc14
msbuild zlibvc.vcxproj
There are other prereqs, apparently:
MSBuild version 17.3.1+2badb37d1 for .NET Framework
Build started 9/29/2022 11:01:01 AM.
Project "D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj" on node 1 (default targets).
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(460,5): error MSB8020: The build tools for Visual Studio 2015 (Platform Toolset = 'v140') cannot b
e found. To build using the v140 build tools, please install Visual Studio 2015 build tools. Alternatively, you may upgrade to the current Visual Studio tools by selecting the Project menu or right-click the
solution, and then selecting "Retarget solution". [D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj]
Done Building Project "D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj" (default targets) -- FAILED.
Build FAILED.
D:\dev\repos\zlib\contrib\vstudio\vc14> msbuild zlibvc.vcxproj
MSBuild version 17.3.1+2badb37d1 for .NET Framework
Build started 9/29/2022 11:23:13 AM.
Project "D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj" on node 1 (default targets).
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Platforms\Win32\PlatformToolsets\v140\Toolset.targets(34,5): error MSB8036: The Windows SDK version 8.1 was not found. Install the required version of Wi
ndows SDK or change the SDK version in the project property pages or by right-clicking the solution and selecting "Retarget solution". [D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj]
Done Building Project "D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj" (default targets) -- FAILED.
Build FAILED.
D:\dev\repos\zlib\contrib\vstudio\vc14> msbuild zlibvc.vcxproj
MSBuild version 17.3.1+2badb37d1 for .NET Framework
Build started 9/29/2022 11:51:10 AM.
Project "D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj" on node 1 (default targets).
PrepareForBuild:
Creating directory "x86\ZlibDllDebug\Tmp\".
Creating directory "x86\ZlibDllDebug\Tmp\zlibvc.tlog\".
InitializeBuildStatus:
Creating "x86\ZlibDllDebug\Tmp\zlibvc.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
PreBuildEvent:
cd ..\..\masmx86
bld_ml32.bat
:VCEnd
The system cannot find the path specified.
'bld_ml32.bat' is not recognized as an internal or external command,
operable program or batch file.
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(123,5): error MSB3073: The command "cd ..\..\masmx86 [D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj]
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(123,5): error MSB3073: bld_ml32.bat [D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj]
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(123,5): error MSB3073: :VCEnd" exited with code 9009. [D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj]
Done Building Project "D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj" (default targets) -- FAILED.
Build FAILED.
This looks like really bad development on the zlib repo. Removing scripts without removing outdated documentation, much less documenting the new way to build. This is Please fix 1.2.12 compile · Issue #631 · madler/zlib (github.com). Instead of using the workarounds there, just build 1.2.11 and let the zlib folks deal with that mess.
git switch --detach v1.2.11
cd zlib/contrib/vstudio/vc14
msbuild zlibvc.vcxproj
hsdis is a plugin for disassembling code dynamically generated by the Java Virtual Machine. On Linux & MacOS, it uses GNU binutils. Support for the LLVM disassembly backend was recently added to hsdis in https://github.com/openjdk/jdk/pull/7531. This was motivated by the fact that GNU binutils is not distributed with the JDK (due to licensing reasons mentioned at https://github.com/openjdk/jdk/pull/5920#issuecomment-942398786) and the LLVM disassembly may be preferrable in certain circumstances. Unfortunately, the official Windows LLVM distribution does not have the header files necessary to build the hotspot disassembler. This prevents Windows developers from easily using the LLVM disassembler backend because they now have to build LLVM themselves as well – see hsdis LLVM backend for Windows ARM64 and Building LLVM for Windows ARM64, for example. In this post, we investigate why the LLVM Windows build does not have the necessary header files. The llvm-c directory in Windows build contains these 2 files only:
C:\Program Files\LLVM\include\llvm-c>dir
Volume in drive C is OSDisk
Volume Serial Number is c070-2ac0
Directory of C:\Program Files\LLVM\include\llvm-c
01/08/2022 11:54 AM <DIR> .
01/08/2022 11:54 AM <DIR> ..
09/24/2021 10:18 AM 29,760 lto.h
09/24/2021 10:18 AM 9,632 Remarks.h
2 File(s) 39,392 bytes
2 Dir(s) 62,273,200,128 bytes free
I created a local LLVM build (see Building LLVM with CMake) and confirmed that it has all the header files.
C:\dev\repos\llvm-project\build_llvm\install_local\include\llvm-c>dir /w
Volume in drive C is OSDisk
Volume Serial Number is 0087-4c48
Directory of C:\dev\repos\llvm-project\build_llvm\install_local\include\llvm-c
[.] [..] Analysis.h
BitReader.h BitWriter.h blake3.h
Comdat.h Core.h DataTypes.h
DebugInfo.h Deprecated.h Disassembler.h
DisassemblerTypes.h Error.h ErrorHandling.h
ExecutionEngine.h ExternC.h Initialization.h
IRReader.h Linker.h LLJIT.h
lto.h Object.h Orc.h
OrcEE.h Remarks.h Support.h
Target.h TargetMachine.h [Transforms]
Types.h
28 File(s) 382,361 bytes
3 Dir(s) 59,158,138,880 bytes free
Does this problem still exist in the latest Windows LLVM release? I went to Releases · llvm/llvm-project (github.com) to find the latest LLVM installer for Windows but couldn’t find it. Turns out it’s because the 15.0.1 release is only 14 hours old so some of the assets probably haven’t been uploaded. Notice that 15.0.0 has 47 assets. I can successfully download and install LLVM-15.0.0-win64.exe and see that the header files are still missing.
Interestingly, trying to install LLVM-15.0.0-win32.exe before uninstalling LLVM-15.0.0-win64.exe gives this dialog and clicking Yes uninstalls before the actual installation of the 32-bit build starts!
LLVM is already installed.
I assumed that would happen at this stage:
All the same, these dialogs have strings that can lead us to the sources that create the installer! The installer looks very similar to the one from Building the Elmer Install Folder so searching the llvm codebase for “ncis ” gives only a handful of hits leading to the key discovery of build_llvm_release.bat! (later learn that this needs to be executed in a (2019) developer command prompt so that the ninja command can be found). That script requires 7zip though. The script fails on my machine because it can’t find 7zip. Failure seems to be coming from the for-statement (see for | Microsoft Learn for usage). The for command uses the escape character (^) as explained at set | Microsoft Learn.
C:\dev\repos\llvm-project\llvm\utils\release> build_llvm_release.bat 15.0.0
Check 7-zip version and/or administrator permissions.
'7z.exe' is not recognized as an internal or external command,
operable program or batch file.
You need to modify the paths below:
Revision: llvmorg-15.0.0
Package version: 15.0.0
Build dir: C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0
Press any key to continue . . .
-- Looking for CrashReporterClient.h
-- Looking for CrashReporterClient.h - not found
-- Looking for pfm_initialize in pfm
-- Looking for pfm_initialize in pfm - not found
-- Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)
CMake Error at C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find LibXml2 (missing: LIBXML2_INCLUDE_DIR)
Call Stack (most recent call first):
C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/FindLibXml2.cmake:108 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
cmake/config-ix.cmake:156 (find_package)
CMakeLists.txt:774 (include)
-- Configuring incomplete, errors occurred!
See also "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/CMakeFiles/CMakeOutput.log".
See also "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/CMakeFiles/CMakeError.log".
Looking through FindPackageHandleStandardArgs.cmake leads me to the simple realization that the wrong define is being used on the command line. Could this be because I’m using a newer CMake? I’ve been using the VS 2022 Preview Developer Command Prompt thus far. My VS 2019 (16.11.19) installation uses CMake 3.20. Both FindLibXml2.cmake in 3.20 and FindLibXml2.cmake in 3.24 require the LIBXML2_INCLUDE_DIR variable. However, they also claim (at the top) to set these variables.
in my build folder (build_llvm), there is a CPackConfig.cmake file that sets variables like CPACK_PACKAGE_FILE_NAME and CPACK_NSIS_DISPLAY_NAME. Since it is NSIS Wiki (sourceforge.io) in use, I wonder about running the package target myself in a manner similar to that used to create my local build. I switch back to a previous build directory (created without the build_llvm_release.bat) and run:
cmake --build . --config Release --target package
The resulting failure below indicates that NSIS is required.
MSBuild version 17.4.0-preview-22466-03+48ab5664b for .NET Framework
PipSqueak.vcxproj -> C:\dev\repos\llvm-project\build_llvm\unittests\Support\DynamicLibrary\Release\PipSqueak.dll
SecondLib.vcxproj -> C:\dev\repos\llvm-project\build_llvm\unittests\Support\DynamicLibrary\Release\SecondLib.dll
obj.llvm-tblgen.vcxproj -> C:\dev\repos\llvm-project\build_llvm\utils\TableGen\obj.llvm-tblgen.dir\Release\obj.llvm-tblgen.lib
LLVMDemangle.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\lib\LLVMDemangle.lib
...
verify-uselistorder.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\verify-uselistorder.exe
yaml-bench.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\yaml-bench.exe
yaml2obj.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\yaml2obj.exe
EXEC : CPack error : Cannot find NSIS compiler makensis: likely it is not installed, or not in your PATH [C:\dev\repos\llvm-project\build_llvm\package.vcxproj]
EXEC : CPack error : Could not read NSIS registry value. This is usually caused by NSIS not being installed. Please install NSIS from http://nsis.sourceforge.net [C:\dev\repos\llvm-proje
ct\build_llvm\package.vcxproj]
EXEC : CPack error : Cannot initialize the generator NSIS [C:\dev\repos\llvm-project\build_llvm\package.vcxproj]
After installing NSIS, the previous command successfully creates an LLVM for Windows installer.
...
verify-uselistorder.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\verify-uselistorder.exe
yaml-bench.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\yaml-bench.exe
yaml2obj.vcxproj -> C:\dev\repos\llvm-project\build_llvm\Release\bin\yaml2obj.exe
CPack: Create package using NSIS
CPack: Install projects
CPack: - Install project: LLVM [Release]
CMake Warning (dev) at C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/GNUInstallDirs.cmake:243 (messa
ge):
Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
target architecture is known. Please enable at least one language before
including GNUInstallDirs.
Call Stack (most recent call first):
C:/dev/repos/llvm-project/llvm/cmake/modules/LLVMInstallSymlink.cmake:5 (include)
C:/dev/repos/llvm-project/build_llvm/tools/llvm-ar/cmake_install.cmake:48 (include)
C:/dev/repos/llvm-project/build_llvm/tools/cmake_install.cmake:39 (include)
C:/dev/repos/llvm-project/build_llvm/cmake_install.cmake:71 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
CPack: Create package
CPack: - package: C:/dev/repos/llvm-project/build_llvm/LLVM-16.0.0git-win64.exe generated.
This installer generates the LLVM includes on disk as expected. The issue must therefore be confined to the installer generated by the script.
Reviewing Ninja NSIS Packaging
At this point, I ran build_llvm_release.bat to create an installer. Once packaging is complete, the install_manifest.txt file can be used to determine which files are in the installer. The batch file also runs lots of tests and this was annoying when trying to generate installers. Once the tests failed on the build I was creating and I had CTRL+C’d a couple of times, I ran ninja package myself (taken from the batch file)
C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0>ninja package
[0/1] Run CPack packaging tool...CPack: Create package using NSIS
CPack: Install projects
CPack: - Install project: LLVM []
CMake Warning (dev) at C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/GNUInstallDirs.cmake:243 (message):
Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
target architecture is known. Please enable at least one language before
including GNUInstallDirs.
Call Stack (most recent call first):
C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/llvm-project/llvm/cmake/modules/LLVMInstallSymlink.cmake:5 (include)
C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/tools/llvm-ar/cmake_install.cmake:40 (include)
C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/tools/cmake_install.cmake:39 (include)
C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/cmake_install.cmake:114 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
CPack: Create package
CPack: - package: C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/LLVM-15.0.0-win32.exe generated.
I then try to find a package target in build.ninja. Search for CMakeFiles\package.util.+ include since we’re interested in include files. There are some interesting differences in the include directories of the build created manually from the local install and the one created by the script, e.g.
Try searching in build.ninja for the 2 header files the installer creates in the (broken) shipping LLVM for Windows build. Nothing there but searching the file system for remarks.h gives interesting results, e.g. the existence of an NSIS project file: project.nsi. Looks like there are some tutorials showing how to create .nsi files at Invoking NSIS run-time commands on compile-time – NSIS (sourceforge.io). The way NSIS is used with CPack when building is documented at Packaging With CPack — Mastering CMake
… but I completely missed the fact that the 2nd didn’t have these lines from the first.
if(CMAKE_INSTALL_COMPONENT STREQUAL "llvm-headers" OR NOT CMAKE_INSTALL_COMPONENT)
file(INSTALL DESTINATION "${CMAKE_INSTALL_PREFIX}/include" TYPE DIRECTORY FILES
"C:/dev/repos/llvm-project/llvm/include/llvm"
"C:/dev/repos/llvm-project/llvm/include/llvm-c"
FILES_MATCHING REGEX "/[^/]*\\.def$" REGEX "/[^/]*\\.h$" REGEX "/[^/]*\\.td$" REGEX "/[^/]*\\.inc$" REGEX "/license\\.txt$")
endif()
if(CMAKE_INSTALL_COMPONENT STREQUAL "llvm-headers" OR NOT CMAKE_INSTALL_COMPONENT)
file(INSTALL DESTINATION "${CMAKE_INSTALL_PREFIX}/include" TYPE DIRECTORY FILES
"C:/dev/repos/llvm-project/build_llvm/include/llvm"
"C:/dev/repos/llvm-project/build_llvm/include/llvm-c"
FILES_MATCHING REGEX "/[^/]*\\.def$" REGEX "/[^/]*\\.h$" REGEX "/[^/]*\\.gen$" REGEX "/[^/]*\\.inc$" REGEX "/cmakefiles$" EXCLUDE REGEX "/config\\.h$" EXCLUDE)
endif()
Search the codebase for “llvm-headers” and find the llvm-header component definition. That whole code block is gated by the LLVM_INSTALL_TOOLCHAIN_ONLY variable! This is explicitly turned off in build_llvm_release.bat! I rerun the batch file and see tests failing after the build succeeds. CTRL+C to kill the processes so that I can get to the root issue: does turning off that flag fix the includes? makensis fails, probably because I killed the build and some things might still have been in use?
C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0>ninja package
[0/1] Run CPack packaging tool...CPack: Create package using NSIS
CPack: Install projects
CPack: - Install project: LLVM []
CMake Warning (dev) at C:/Program Files/Microsoft Visual Studio/2022/Preview/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.24/Modules/GNUInstallDirs.cmake:243 (message):
Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
target architecture is known. Please enable at least one language before
including GNUInstallDirs.
Call Stack (most recent call first):
C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/llvm-project/llvm/cmake/modules/LLVMInstallSymlink.cmake:5 (include)
C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/tools/llvm-ar/cmake_install.cmake:40 (include)
C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/tools/cmake_install.cmake:39 (include)
C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/cmake_install.cmake:128 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
CPack: Create package
CPack Error: Problem running NSIS command: "C:/Program Files (x86)/NSIS/makensis.exe" "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/_CPack_Packages/win32/NSIS/project.nsi"
Please check C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/_CPack_Packages/win32/NSIS/NSISOutput.log for errors
CPack Error: Problem compressing the directory
CPack Error: Error when generating package: LLVM
FAILED: CMakeFiles/package.util
cmd.exe /C "cd /D C:\dev\repos\llvm-project\llvm\utils\release\llvm_package_15.0.0\build32_stage0 && "C:\Program Files\Microsoft Visual Studio\2022\Preview\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cpack.exe" --config ./CPackConfig.cmake"
ninja: build stopped: subcommand failed.
NSISOutput.log failed due to an “Internal compiler error #12345: error mmapping datablock to 17235001.” However, the include files are now present in the source directory being packaged by NSIS.
Turning Off Tests
There are many tests that the build script runs and some of them are failing. Testing is not on my critical path since all I need is to generate installers so I modify the scripts to enable me to package the build without running all the tests. I then start my build without tests and go to bed only to wake up the next morning to find that I need to rerun it because there are no running programs when I log in. Event Viewer doesn’t show any reboot-related events and sure enough, Task Manager shows over 9 days of uptime still. Turns out the Desktop Window Manager crashed (C:\WINDOWS\system32\dwm.exe)! Curse you dwmcore.dll. Well, time to install those updates I’ve been putting off and reboot before jumping back in. Now on the new Windows 10.0.22621.521. The build still fails:
-- LLVM host triple: i686-pc-windows-msvc
-- LLVM default target triple: i686-pc-windows-msvc
-- Using Release VC++ CRT: MD
-- Looking for os_signpost_interval_begin
-- Looking for os_signpost_interval_begin - not found
CMake Error at C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find Python3 (missing: Python3_EXECUTABLE Interpreter) (Required
is at least version "3.6")
Reason given by package:
Interpreter: Cannot use the interpreter "C:/Python310/python.exe"
Call Stack (most recent call first):
C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPython/Support.cmake:3165 (find_package_handle_standard_args)
C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPython3.cmake:485 (include)
CMakeLists.txt:817 (find_package)
-- Configuring incomplete, errors occurred!
See also "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/CMakeFiles/CMakeOutput.log".
See also "C:/dev/repos/llvm-project/llvm/utils/release/llvm_package_15.0.0/build32_stage0/CMakeFiles/CMakeError.log".
When I interrupted the tests before modifying the batch file to skip them, I noticed that they were being run by %LOCALAPPDATA%\Microsoft\WindowsApps\python3.9.exe. This is still present on my machine. Ah, turns out I’m now using the 2019 developer command prompt (and therefore an older CMake). The only difference between CMake 3.20 FindPython3.cmake and CMake 3.24 FindPython3.cmake is a comment about static libraries, so this failure is a mystery.
Diagnosing Build Failures
Since this issue also bit me when I moved to my Surface Book, it is worth understanding why it happens.
Missing CMake in Visual Studio 17.3.4 Developer Command Prompt
Here is the VS 2022 Preview vs VS 2022 Enterprise path to CMake:
C:\Program Files (x86)\Microsoft Visual Studio\Installer> where cmake
C:\Program Files\Microsoft Visual Studio\2022\Preview\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe
C:\dev\repos\llvm-project\llvm\utils\release> where cmake
INFO: Could not find files for the given pattern(s).
The View Logs link opens the Documents folder under This PC – not particularly useful. Interestingly though, clicking on the Modify button shows a Total space required 1.63 GB. How is there space required before I’ve selected anything? Something similar happens with 16.11.19 though. Without making any individual component selections, I start the install process. CMake gets (re-?)installed as shown below. This fixes the setup warnings as well and cmake is now usable in the VS2022 command prompt.
Missing Python3 in VS 17.3.4 Developer Command Prompt
This is the error I got when trying to build LLVM on my Surface Book 2 in the VS 2022 developer command prompt:
CMake Error at C:/Program Files/CMake/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:164 (message):
Could NOT find Python3 (missing: Python3_EXECUTABLE Interpreter) (Required
is at least version "3.6")
Reason given by package:
Interpreter: Cannot use the interpreter "C:/Python310/python.exe"
Call Stack (most recent call first):
C:/Program Files/CMake/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:445 (_FPHSA_FAILURE_MESSAGE)
C:/Program Files/CMake/share/cmake-3.17/Modules/FindPython/Support.cmake:2437 (find_package_handle_standard_args)
C:/Program Files/CMake/share/cmake-3.17/Modules/FindPython3.cmake:309 (include)
CMakeLists.txt:817 (find_package)
Uninstalling CMake enables the command line to pick up the CMake distributed with Visual Studio. Python3 is now found successfully in the path below (I’ve shortened it using %LOCALAPPDATA%).
-- Found Python3: %LOCALAPPDATA%/Microsoft/WindowsApps/python3.8.exe (found suitable version "3.8.10", minimum required is "3.6") found components: Interpreter
Missing Python3 in VS 16.11.19 Developer Command Prompt
Interestingly, I still get the same error in VS 2019 despite uninstalling CMake 3.17. My earlier hypothesis is therefore invalid.
CMake Error at C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find Python3 (missing: Python3_EXECUTABLE Interpreter) (Required
is at least version "3.6")
Reason given by package:
Interpreter: Cannot use the interpreter "C:/Python310/python.exe"
Call Stack (most recent call first):
C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPython/Support.cmake:3165 (find_package_handle_standard_args)
C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/FindPython3.cmake:485 (include)
CMakeLists.txt:817 (find_package)
C:/Python310/python.exe -c "import sys; sys.stdout.write('.'.join([str(x) for x in sys.version_info[:3]]))"
I comment out the ERROR_QUIET line to reveal the stdout and stderr output from python since the return code from the python process is causing the CMake error to be raised. Running with --trace-expand --trace-redirect=cmake_trace.txt now reveals the root cause (paths below cleaned up using %LOCALAPPDATA%):
Python path configuration:
PYTHONHOME = '%LOCALAPPDATA%\Programs\Python\Python310-32'
PYTHONPATH = (not set)
program name = 'C:/Python310/python.exe'
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = 'C:\\Python310\\python.exe'
sys.base_prefix = '%LOCALAPPDATA%\\Programs\\Python\\Python310-32'
sys.base_exec_prefix = '%LOCALAPPDATA%\\Programs\\Python\\Python310-32'
sys.platlibdir = 'lib'
sys.executable = 'C:\\Python310\\python.exe'
sys.prefix = '%LOCALAPPDATA%\\Programs\\Python\\Python310-32'
sys.exec_prefix = '%LOCALAPPDATA%\\Programs\\Python\\Python310-32'
sys.path = [
'C:\\Python310\\python310.zip',
'%LOCALAPPDATA%\\Programs\\Python\\Python310-32\\DLLs',
'%LOCALAPPDATA%\\Programs\\Python\\Python310-32\\lib',
'C:\\Python310',
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
Current thread 0x00003174 (most recent call first):
<no Python frame>
django – init_fs_encoding: failed to get the Python codec of the filesystem encoding – Stack Overflow is a hint that the PYTHONHOME is wrong. Sure enough, I didn’t change it in build_llvm_release.bat so the paths in the configuration above do not exist! This now raises another question: how on earth does this work in VS 2022? I notice on my desktop that python.exe does not even appear in the CMake tracing output! The difference in behavior stems from the fact that the find_program command in Modules/FindPython/Support.cmake · v3.20.0 finds python 3.10 first in the VS 2019 environment. This path is then assigned to _Python3_EXECUTABLE, preventing the 3.8 path from being used. One important difference between CMake 3.20 and 3.23 that I notice is FindPython: fix typo error (fff8d5b2) · Commits · CMake / CMake · GitLab (kitware.com). Since the fix for the build_llvm_release.bat script is straightforward and it is clear that there are some CMake implementation differences at work, we no longer need to dig into why this behavior could be happening.
Python Hangs
One of my build attempts successfully completes stage0 but hangs when CMake tries to detect the python version. Manually running the same command (copied from Process Explorer) also hangs. Even %LOCALAPPDATA%/Microsoft/WindowsApps/python3.9.exe --version hangs. Inspecting the full dump created by Task Manager reveals that python3.9.exe made a call to get (what looks like) the Package.InstalledLocation Property (Windows.ApplicationModel) – Windows UWP applications | Microsoft Learn
...
-- Looking for os_signpost_interval_begin
-- Looking for os_signpost_interval_begin - not found
Windows becomes pretty unusable as I investigate this behavior (mouse doesn’t work, changes program in focus but can’t click on anything). A reboot fixes these issues (e.g. version now works). Can’t believe we have to deal with this in 2022???
7z x clang+llvm-15.0.1-aarch64-linux-gnu.tar.xz
7z x clang+llvm-15.0.1-arm64-apple-darwin21.0.tar.xz
7z x clang+llvm-15.0.1-x86_64-apple-darwin.tar.xz
tar xf clang+llvm-15.0.1-aarch64-linux-gnu.tar
tar xf clang+llvm-15.0.1-arm64-apple-darwin21.0.tar
tar xf clang+llvm-15.0.1-x86_64-apple-darwin.tar
Here are the directories in the include folder before the installer is created. There are also 28 include files in the include/llvm-c/ directory as desired.
Directory of llvm\utils\release\llvm_package_15.0.0\build32_stage0\_CPack_Packages\win64\NSIS\LLVM-15.0.0-win64\include
clang
clang-c
clang-tidy
lld
lldb
llvm
llvm-c
Outstanding Questions
Why does the NSIS project fail to build? Why are there test failures and build errors?
Why does the Linux build have ompt-multiplex.h and the aarch64-unknown-linux-gnu directory?
How is the Windows ARM64 installer generated?
Why doesn’t the Windows build have c++, flang, mlir, mlir-c, and polly?
How do we get symbols to the Python app in the Microsoft Store?
This path C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build has various scripts to set up a command window as documented at Use the Microsoft C++ toolset from the command line | Microsoft Docs. If vcvarsx86_arm64.bat and vcvarsamd64_arm64.bat are missing in that folder on your Windows x64 machine, install the MSVC v143 – VS 2022 C++ ARM64 build tools (Latest) component in the Visual Studio 2022 installer.
Selection ARM64 Build Tools in VS Installer
Once it is installed, open a new cmd.exe window and run this command to set up the build environment:
To see the command Visual Studio uses to build the project, create a C++ console application and use the Configuration Manager to change the Active solution platform to ARM64. Next, go to Tools > Options then expand the Projects and Solutions node. Select Build And Run then change the MSBuild project build output verbosity to Detailed. Building the project should now show the full command line used to invoke the compiler, for example here are the command lines used in the Debug and Release configurations respectively.
Notice the /O2 flag (maximize speed) in the release build instead of the /Od flag (no optimizations) above. The debug build also uses the just my code /JMC, runtime error checks /RTC1, and debug multithread-specific version of the run-time library /MDd flags. For our testing purposes, we can ignore most of these flags.
Calling Printf
Here is a simple program, aarch64-abi-test-printf.cpp, which calls printf with a format specifier and 4 additional arguments.
#include <stdio.h>
int main()
{
int result = printf("%.4f,%.4f,%.4f,%s", 1.2345, 1.2345, 1.2345, "str");
}
In the disassembly generated by dumpbin (printf-abi.asm), notice that all 5 arguments to printf are passed in registers! x0 contains a pointer to the format string, x1-x3 contain the address of the $LN3 label. The 64-bits at that label are the IEEE double floating point representation of 1.2345. x4 contains a pointer to the null-terminated string “str“.
Which are the printf String Arguments?
To determine what symbols in instructions like adrp x8,$SG5571 mean, we use the output of dumpbin /all. The RELOCATIONS section shows $SG5571 to have symbol index 8. The COFF SYMBOL TABLE shows this symbol index 8 to be in SECT3. The raw data for section 3 contains the format string and the single string parameter passed to printf. I’m still not sure how the assembler knows the difference in offsets between these 2 strings?
.
.
.
SECTION HEADER #3
.rdata name
0 physical address
0 virtual address
1A size of raw data
31A file pointer to raw data (0000031A to 00000333)
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
40400040 flags
Initialized Data
8 byte align
Read Only
RAW DATA #3
00000000: 73 74 72 00 00 00 00 00 25 2E 34 66 2C 25 2E 34 str.....%.4f,%.4
00000010: 66 2C 25 2E 34 66 2C 25 73 00 f,%.4f,%s.
.
.
.
RELOCATIONS #4
Symbol Symbol
Offset Type Applied To Index Name
-------- ---------------- ----------------- -------- ------
00000008 PAGEBASE_REL21 90000008 8 $SG5571
0000000C PAGEOFFSET_12A 91000104 8 $SG5571
0000001C PAGEBASE_REL21 90000008 9 $SG5572
00000020 PAGEOFFSET_12A 91000100 9 $SG5572
00000024 BRANCH26 94000000 16 printf
.
.
.
COFF SYMBOL TABLE
000 01057A64 ABS notype Static | @comp.id
001 80010190 ABS notype Static | @feat.00
002 00000000 SECT1 notype Static | .drectve
Section length 62, #relocs 0, #linenums 0, checksum 0
004 00000000 SECT2 notype Static | .debug$S
Section length 9C, #relocs 0, #linenums 0, checksum 0
006 00000000 SECT3 notype Static | .rdata
Section length 1A, #relocs 0, #linenums 0, checksum B99D9667
008 00000000 SECT3 notype Static | $SG5571
009 00000008 SECT3 notype Static | $SG5572
00A 00000000 SECT4 notype Static | .text$mn
Compiling an Optimized Build
Specifying the /O2 flag for speed generates optimized code.
I have been toying around with the idea of doing a fluid dynamics or crystal growth simulation using nVidia CUDA. I decided to try out nVidia’s cuda samples to see what their approach looks like, in particular when rendering using OpenGL. I am using Visual Studio 2022 so I simply cloned the cuda samples repo, opened the fluidsGL_vs2022.sln solution, right click on the fluidsGL project, then selected Build.
Build started...
1>------ Build started: Project: fluidsGL, Configuration: Debug x64 ------
1>D:\dev\...\cuda-samples\Samples\5_Domain_Specific\fluidsGL\fluidsGL_vs2022.vcxproj(37,5): error MSB4019: The imported project "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\BuildCustomizations\CUDA 11.6.props" was not found. Confirm that the expression in the Import declaration "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\\BuildCustomizations\CUDA 11.6.props" is correct, and that the file exists on disk.
1>Done building project "fluidsGL_vs2022.vcxproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========
The prerequisites section does mention that the CUDA Toolkit 11.6 is required, so I close VS and install it. I end up with version 11.7 though:
When reopening the fluidsGL solution, I still get the same error about CUDA 11.6.props not being found. A quick look at the directory this file is expected to be in reveals that this is a simple version mismatch problem – see the hard coded version in the fluidsGL.vcxproj file. Instead of fixing every example .vcxproj file to match CUDA 11.7, we can patch the VS folder by running these commands from an admin command prompt:
The code now builds in Visual Studio and I can now oooh, aaaah over the demo. Visual Studio does seem a bit sluggish at opening the entire samples solution though… I get this information about my device in the console window after the demo launches:
GPU Device 0: "Pascal" with compute capability 6.1
CUDA device [Quadro P1000] has 5 Multi-Processors
