Saint's Log

Java’s Foreign Function API vs the Windows AArch64 ABI

I just opened PR 8295290: Add Windows ARM64 ABI support to the Foreign Function & Memory API · Pull Request #754 · openjdk/panama-foreign (github.com) (almost) completing some work that Bernhard had started to properly support the Windows ARM64 ABI in the JDK’s Foreign Function & Memory API. This post documents how I learned about the feature and its implementation. I picked up from where Bernhard left off… here is how my investigation proceeded.

I need to understand what happens if we build the jdk master branch (at commit 18cd16d2 when I started) without any ABI-specific changes. To do so, we need JDK 18 or later as a boot JDK to build the latest code, e.g. Oracle’s JDK 18 Windows x64 Installer. Here are the commands I used in Cygwin:

git clone https://github.com/swesonga/jdk
cd jdk

bash configure --openjdk-target=aarch64-unknown-cygwin --with-debug-level=slowdebug --with-boot-jdk=/cygdrive/d/dev/repos/java/infra/binaries/jdk-18.0.2

make images LOG=debug > build/abi-20220802-1500.txt
make build-test-jdk-jtreg-native LOG=debug > build/test-20220802-1500.txt

Once the build complete, create the artifacts for an AArch64 Windows device. These build and archive steps are available as the build-aarch64.sh script.

cd build/windows-aarch64-server-slowdebug/jdk
zip -qru jdk-20220802-1500-master.zip .
mv jdk-20220802-1500-master.zip ..

cd ..
zip -qru test-jdk-20220802-1500-master.zip support/test

Copy the two zip files to the 64-bit ARM device (e.g. by sharing folders or using OneDrive). I used a Surface Pro X device running Windows 11 build 22000.795. I unzipped the 2 files into these paths:

C:\dev\java\abi\master\jdk\
C:\dev\java\abi\master\support\test\..

I later discovered that unzip is available in the Git Bash terminal! These commands can be used to unzip the files:

mkdir -p /c/dev/java/abi/devbranch/jdk
cd /c/dev/java/abi/devbranch/jdk
unzip -q /c/dev/java/builds/debug/jdk-20220802-1500-devbranch.zip
cd ..
unzip -q test-jdk-20220802-1500-master.zip

I also downloaded jtreg and placed it in this path (note that it might be easier to extract the .tar.gz on the Windows x64 build machine then share it).

C:\dev\java\jtreg\

Finish setting up the Windows AArch64 device to run the ABI jtreg tests by cloning the OpenJDK repo onto it. The jtreg tests will be run from the root of the OpenJDK repo.

cd \dev\java\repos\forks
git clone https://github.com/swesonga/jdk
cd jdk

We’ll run VaListTest.java to see how it fails on Windows AArch64.

C:\dev\java\abi\master\jdk\bin\java.exe -jar C:\dev\java\jtreg\lib\jtreg.jar -agentvm -timeoutFactor:4 -concurrency:4 -verbose:fail,error,summary -nativepath:C:\dev\java\abi\master\support\test\jdk\jtreg\native\lib test/jdk/java/foreign/valist/VaListTest.java

Test fails:

--------------------------------------------------
TEST: java/foreign/valist/VaListTest.java
TEST JDK: C:\dev\java\abi\master\jdk

ACTION: build -- Passed. All files up to date
REASON: Named class compiled on demand
TIME:   0.069 seconds
messages:
command: build VaListTest
reason: Named class compiled on demand
elapsed time (seconds): 0.069

ACTION: testng -- Failed. Execution failed: `main' threw exception: org.testng.TestNGException: An error occurred while instantiating class VaListTest: null
REASON: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED VaListTest
TIME:   12.557 seconds
messages:
command: testng --enable-native-access=ALL-UNNAMED VaListTest
reason: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED VaListTest
Mode: othervm [/othervm specified]
Additional options from @modules: --add-modules java.base --add-exports java.base/jdk.internal.foreign=ALL-UNNAMED --add-exports java.base/jdk.internal.foreign.abi=ALL-UNNAMED --add-exports java.base/jdk.internal.foreign.abi.x64=ALL-UNNAMED --add-exports java.base/jdk.internal.foreign.abi.x64.sysv=ALL-UNNAMED --add-exports java.base/jdk.internal.foreign.abi.x64.windows=ALL-UNNAMED --add-exports java.base/jdk.internal.foreign.abi.aarch64=ALL-UNNAMED --add-exports java.base/jdk.internal.foreign.abi.aarch64.linux=ALL-UNNAMED --add-exports java.base/jdk.internal.foreign.abi.aarch64.macos=ALL-UNNAMED --add-exports java.base/jdk.internal.foreign.abi.aarch64.windows=ALL-UNNAMED
elapsed time (seconds): 12.557
configuration:
Boot Layer
  add modules: java.base
  add exports: java.base/jdk.internal.foreign                     ALL-UNNAMED
               java.base/jdk.internal.foreign.abi                 ALL-UNNAMED
               java.base/jdk.internal.foreign.abi.aarch64         ALL-UNNAMED
               java.base/jdk.internal.foreign.abi.aarch64.linux   ALL-UNNAMED
               java.base/jdk.internal.foreign.abi.aarch64.macos   ALL-UNNAMED
               java.base/jdk.internal.foreign.abi.aarch64.windows ALL-UNNAMED
               java.base/jdk.internal.foreign.abi.x64             ALL-UNNAMED
               java.base/jdk.internal.foreign.abi.x64.sysv        ALL-UNNAMED
               java.base/jdk.internal.foreign.abi.x64.windows     ALL-UNNAMED

STDOUT:
STDERR:
WARNING: package jdk.internal.foreign.abi.aarch64.windows not in java.base
org.testng.TestNGException:
An error occurred while instantiating class VaListTest: null
        at org.testng.internal.InstanceCreator.createInstanceUsingObjectFactory(InstanceCreator.java:123)
        at org.testng.internal.InstanceCreator.createInstance(InstanceCreator.java:79)
...

I expected Bernhard’s code to be the one introducing Windows AArch64 ABI clean-up code. So why are there failures about the aarch64.windows foreign abi package missing? This requirement is from VaListTest.java and was introduced by the Foreign Function & Memory API (Preview) PR (it added the java.base/jdk.internal.foreign.abi.aarch64.windows module to the failing test).

Porting the Changes

I worked on porting Bernhard’s code on a Windows x64 machine.

# Switch the the OpenJDK repo directory
cd jdk

# This was the tip of the upstream master branch
# git checkout 18cd16d2eae2ee624827eb86621f3a4ffd98fe8c

git switch -c WinAArch64ABI
git remote add lewurm https://github.com/lewurm/openjdk
git fetch lewurm
git switch foreign-windows-aarch64
git rebase WinAArch64ABI

The files he modified have been deleted in the current repo:

Find when a file was deleted in Git – Stack Overflow has the command to view when these files were deleted. Turns out to be the same Foreign Function & Memory API (Preview) PR that added the aarch64.windows foreign abi package to VaListTest.java.

$ git log --full-history -2 -- src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java
commit 2c5d136260fa717afa374db8b923b7c886d069b7

Author: Maurizio Cimadamore <mcimadamore@openjdk.org>
Date:   Thu May 12 16:17:45 2022 +0000

    8282191: Implementation of Foreign Function & Memory API (Preview)

    Reviewed-by: erikj, jvernee, psandoz, dholmes, mchung

The deleted files moved to src/java.base/share/classes/jdk/internal/foreign. Bernhard’s changes are small enough that I manually port them (copy/paste) into the files in the new locations in the tree. It’s interesting seeing the newer Java language features in use, e.g. the permits keyword. Now build the changes using the build-aarch64.sh script:

bash configure --openjdk-target=aarch64-unknown-cygwin --with-debug-level=slowdebug --with-boot-jdk=/cygdrive/d/dev/repos/java/infra/binaries/jdk-18.0.2

/cygdrive/d/dev/repos/scratchpad/scripts/java/cygwin/build-aarch64.sh

The newly added files are packed as .class files.

$ find build/windows-aarch64-server-slowdebug/jdk/ -name "WindowsAArch64CallArranger*"
...
build/windows-aarch64-server-slowdebug/jdk/modules/java.base/jdk/internal/foreign/abi/aarch64/windows/WindowsAArch64CallArranger.class

# Verify last modification time

$ ls -l build/windows-aarch64-server-slowdebug/jdk/./modules/java.base/jdk/internal/foreign/abi/aarch64/windows/WindowsAArch64CallArranger.class

Need to create a WindowsAArch64CallArranger to match the current structure of the foreign ABI. With these changes, VaListTest.java now passes. However, StdLibTest.java and TestVarArgs.java fail.

TEST: java/foreign/StdLibTest.java
TEST JDK: C:\dev\java\abi\devbranch\jdk

ACTION: build -- Passed. All files up to date
REASON: Named class compiled on demand
TIME:   0.039 seconds
messages:
command: build StdLibTest
reason: Named class compiled on demand
elapsed time (seconds): 0.039

ACTION: testng -- Failed. Unexpected exit from test [exit code: -1073741819]
REASON: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED StdLibTest
TIME:   15.02 seconds
messages:
command: testng --enable-native-access=ALL-UNNAMED StdLibTest
reason: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED StdLibTest
Mode: othervm [/othervm specified]
elapsed time (seconds): 15.02
configuration:
STDOUT:
test StdLibTest.test_printf([STRING]): failure
java.lang.AssertionError: expected [11] but found [14]
        at org.testng.Assert.fail(Assert.java:99)
        ...
        at org.testng.Assert.assertEquals(Assert.java:917)
        at StdLibTest.test_printf(StdLibTest.java:135)
        ...
        at org.testng.TestNG.run(TestNG.java:1037)
        ...
        at java.base/java.lang.Thread.run(Thread.java:1589)
test StdLibTest.test_printf(java.util.ArrayList@5499b7af): success
test StdLibTest.test_printf([DOUBLE, DOUBLE, CHAR]): success

TEST: java/foreign/TestVarArgs.java
TEST JDK: C:\dev\java\abi\devbranch\jdk

ACTION: build -- Passed. All files up to date
REASON: Named class compiled on demand
TIME:   0.031 seconds
messages:
command: build TestVarArgs
reason: Named class compiled on demand
elapsed time (seconds): 0.031

ACTION: testng -- Failed. Unexpected exit from test [exit code: 1]
REASON: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED -Dgenerator.sample.factor=17 TestVarArgs
TIME:   17.52 seconds
messages:
command: testng --enable-native-access=ALL-UNNAMED -Dgenerator.sample.factor=17 TestVarArgs
reason: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED -Dgenerator.sample.factor=17 TestVarArgs
Mode: othervm [/othervm specified]
elapsed time (seconds): 17.52
configuration:
STDOUT:
test TestVarArgs.testVarArgs(0, "f0_V__", VOID, [], []): success
STDERR:
java.lang.RuntimeException: java.lang.IllegalStateException: java.lang.AssertionError: expected [24.0] but found [8.135772792034E-312]
        at TestVarArgs.check(TestVarArgs.java:134)
        ...
        at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:758)
        at TestVarArgs.testVarArgs(TestVarArgs.java:104)
        ...
        at org.testng.TestNG.runSuites(TestNG.java:1069)
        at org.testng.TestNG.run(TestNG.java:1037)
        ...

The data for these tests is supplied by a testng dataProvider that returns an array of arrays of objects. As per the dataProvider docs, the first dimension’s size is the number of times the test method will be invoked and the second dimension size contains an array of objects that must be compatible with the parameter types of the test method.

Java Concepts in the Tests

1. As per the article Enum Types, enums implicitly extend java.lang.Enum and cannot extend anything else because Java does not support multiple inheritance. The Enum class docs also point out that all the constants of an enum class can be obtained by calling the implicit public static T[] values() method of that class and that more information about enums, including descriptions of the implicitly declared methods synthesized by the compiler, can be found in section 8.9 of The Java Language Specification. Section 8.9 explains that an enum constant may be followed by arguments, which are passed to the constructor of the enum when the constant is created during class initialization as described later in this section. The constructor to be invoked is chosen using the normal rules of overload resolution (§15.12.2). If the arguments are omitted, an empty argument list is assumed. This is helpful for understanding all the code I’m seeing in the PrintfArg enum!
2. The printfArgs dataProvider permutes the values of the PrintfArg enum. The implementation uses streams, which are new to me since I last wrote Java before JDK 8 was released. The overview of streams on Oracle’s technical resources website is helpful in coming up to speed with streams. TODO: the implementation of the permutation is mysterious to me, need to study it closely. It uses List.of(), Set.of(), and Collections.shuffle().
3. Try blocks without catch or finally blocks is a try-with-resources statement. This helps prevent leaks of native resources.
4. StdLibTest.java uses functionality from JEP 424: Foreign Function & Memory API (Preview). This JEP provides a good overview of why we need a supported API for accessing off-heap data (i.e. foreign memory) designed from the ground up to be safe and with JIT optimizations in mind.

JEP 424 Concepts via vprintf

The StdLibTest passes when run with the test_printf test commented out. This implies that test_vprintf works as expected, making it a good candidate for reviewing JEP 424: Foreign Function & Memory API (Preview). This test

1. Creates a confined closeable MemorySession on line 311. Confined memory sessions, support strong thread-confinement guarantees as per the MemorySession docs.
2. Creates a memory segment on line 312 using the allocateUtf8String method of the MemorySession‘s SegmentAllocator base interface. This method “converts a Java string into a UTF-8 encoded, null-terminated C string, storing the result into a memory segment.”
3. Create a variable argument list using the VaList.make() method. This invokes SharedUtils.newVaList, which we modified to support Windows on AArch64.
4. Invoke the native vprintf function via its method handle: final static MethodHandle vprintf = abi.downcallHandle(abi.defaultLookup().lookup("vprintf").get(), FunctionDescriptor.of(C_INT, C_POINTER, C_POINTER));.

The value of the abi variable is computed by the SharedUtils.getSystemLinker method, hence the need for creating a WindowsAArch64Linker here. As explained at JEP 424: Foreign Function & Memory API (Preview), abi.defaultLookup() “creates a default lookup, which locates all the symbols in libraries that are commonly used on the OS and processor combination associated with the Linker instance.” defaultLookup() returns a SymbolLookup on which the lookup(“vprintf”) method is invoked. Note that Optional<T>.get() will throw a NoSuchElementException if no value is present. Otherwise, it will return the zero-length MemorySegment whose base address indicates the address of the vprintf function.

As per JEP 424, the Linker interface enables both downcalls (calls from Java code to native code) and upcalls (calls from native code back to Java code). The MemorySegment associated with the address of the vprintf function and a FunctionDescriptor (created by the static FunctionDescriptor.of method) are passed to Linker.downcallHandle to create a MethodHandle which can be used to call vprintf. The arguments to FunctionDescriptor.of are the MemoryLayouts representing the return type (int), the format string, and the format arguments. MethodHandle.invoke() is the how the native vprintf gets, well, invoked, with the format string and the variable argument list. Here’s the Java vprint method.

int vprintf(String format, List<PrintfArg> args) throws Throwable {
    try (MemorySession session = MemorySession.openConfined()) {
        MemorySegment formatStr = session.allocateUtf8String(format);
        VaList vaList = VaList.make(b -> args.forEach(
            a -> a.accept(b, session)), session);
        return (int)vprintf.invoke(formatStr, vaList);
    }
}

Reviewing test_printf

Inlining the code invoked by test_printf here for easy reference. See the docs for the printf function and the printf format specification for additional information about printf. Line 20 of specializedPrintf creates a MethodType for a method returning an int and taking a single pointer (MemoryAddress). appendParameterTypes is used to add all the other printf parameter types to the MethodType. The MemoryLayouts of the arguments are also accumulated into a list. It doesn’t look like we do anything with the method type (mt) though! Looks like dead code from this PR.

final static FunctionDescriptor printfBase = FunctionDescriptor.of(C_INT, C_POINTER);

...

int printf(String format, List<PrintfArg> args) throws Throwable {
    try (MemorySession session = MemorySession.openConfined()) {
        MemorySegment formatStr = session.allocateUtf8String(format);
        return (int)specializedPrintf(args).invoke(formatStr,
                args.stream().map(a -> a.nativeValue(session)).toArray());
    }
}

private MethodHandle specializedPrintf(List<PrintfArg> args) {
    //method type
    MethodType mt = MethodType.methodType(int.class, MemoryAddress.class);
    FunctionDescriptor fd = printfBase;
    List<MemoryLayout> variadicLayouts = new ArrayList<>(args.size());
    for (PrintfArg arg : args) {
        mt = mt.appendParameterTypes(arg.carrier);
        variadicLayouts.add(arg.layout);
    }
    MethodHandle mh = abi.downcallHandle(printfAddr,
            fd.asVariadic(variadicLayouts.toArray(new MemoryLayout[args.size()])));
    return mh.asSpreader(1, Object[].class, args.size());
}

That PR also changed from invokeExact to invoke. Why?

As an aside, notice that the test_time test (and every other test) passed when we disabled test_printf. test_time calls gmtime, which returns a tm struct so that side of things is working fine.

The question is what is all this spreading about? The asSpreader docs explain it as follow

Makes an array-spreading method handle, which accepts an array argument at a given position and spreads its elements as positional arguments in place of the array. The new method handle adapts, as its target, the current method handle. The type of the adapter will be the same as the type of the target, except that the arrayLength parameters of the target’s type, starting at the zero-based position spreadArgPos, are replaced by a single array parameter of type arrayType.

MethodHandle.asSpreader

Therefore, the test is essentially converting all the printf arguments into positional arguments.

Question: how is the translation from all this to native code actually done? PR 8282191: Implementation of Foreign Function & Memory API (Preview) · openjdk/jdk@2c5d136 (github.com) changes some of the hotspot code, which might make it easier to explore the related code.

Looking at the ABIDescriptor in the AArch64 CallArranger, there is a shadow space entry with the value of 0. windows – What is the ‘shadow space’ in x64 assembly? – Stack Overflow explains what shadow space is.

CallArranger.getBindings seems like an interesting place – it uses the abstract method varArgsOnStack() on line 145 and calls SharedUtils.isVarargsIndex(). Notice that the FunctionDescriptor has a firstVariadicArgumentIndex() method that returns -1. This is why specializedPrintf calls FunctionDescriptor.asVariadic(). VariadicFunction sets the firstVariadicIndex to the size of the argumentLayouts of the FunctionDescriptor.

CallArranger.classifyLayout() will return either INTEGER, FLOAT, or POINTER for the case I’m interested in. These cases in UnboxBindingCalculator.getBindings call storageCalculator.nextStorage. DIving into that implementation reveals that we don’t want adjustForVarArgs() to be called! Hmm, after looking at the optimized code in my post on “Building & Disassembling ARM64 Code using Visual C++”, I notice FMOV being used to load general purpose registers x1-x3 with the IEEE double! This looks idfferent from the getBindings implementation, which gets the next storage for FLOATs from the vector registers! et voila! The contradiction I’ve been waiting for: now the addendum on variadic functions at Overview of ARM64 ABI conventions makes sense.

Tests still fail with my change.

Creating a Narrow Test Case

Get a Windows x64 JDK 19 nightly build from Adoptium. Create a Java Project in Eclipse and change the JRE System Library to jdk-19+34. See MinimizedStdLibTest.java. We will use hsdis to explore this testcase. See Blog Theme – Details (oracle.com) and the post on the hsdis LLVM backend for Windows ARM64 for more info. Here is the updated configure command.

bash configure --openjdk-target=aarch64-unknown-cygwin --with-debug-level=slowdebug --with-boot-jdk=/cygdrive/d/dev/repos/java/infra/binaries/jdk-18.0.2  --with-hsdis=llvm --with-llvm=/cygdrive/d/dev/software/llvm-aarch64/

After running the build-aarch64.sh script, we can now disassemble the code on the host:

C:\dev\java\abi\devbranch4\jdk\bin\javac.exe -g --enable-preview --release 20 MinimizedStdLibTest.java

C:\dev\java\abi\devbranch4\jdk\bin\java.exe --enable-preview -XX:+PrintAssembly MinimizedStdLibTest > MinimizedStdLibTest.asm

Inspecting Disassembly using JitWatch

Found this blog post while looking up hsdis: Developers disassemble! Use Java and hsdis to see it all. (oracle.com)

Clone the JitWatch repo. Download the mvn binaries. Set JAVA_HOME to the path of our custom JDK (with hsdis) then start JitWatch. Errors running it though.

No Windows AArch64 binaries at Adoptium or Oracle though.

Let’s just try on x64. Might gain some insight:

cd /d/dev/repos/java/AdoptOpenJDK/jitwatch
/d/dev/repos/java/infra/binaries/jdk-19+34/bin/java --enable-preview -jar ./ui/target/jitwatch-ui-shaded.jar

Looking at these options, I wonder if manually setting the Compile Threshold could show more disassembly:

Update JitWatch to support preview features then change JAVA_HOME. This doesn’t make mvn clean package use my latest JDK…

$ echo $JAVA_HOME
C:\Program Files\Microsoft\jdk-17.0.1.12-hotspot\

$ JAVA_HOME=/d/dev/repos/java/infra/binaries/jdk-19+34/

I can get the JIT to assemble for the main method. Why doesn’t this work on Windows for ARM64? Perhaps I should try a non-debug configuration by configuring as follows before running the build-aarch64.sh script:

bash configure --openjdk-target=aarch64-unknown-cygwin --with-boot-jdk=/cygdrive/d/dev/repos/java/infra/binaries/jdk-18.0.2  --with-hsdis=llvm --with-llvm=/cygdrive/d/dev/software/llvm-aarch64/

I get the same results with the release build – no native code for my printf function! I wonder about downloading something heavier and seeing if anything interesting gets compiled to native code. How about Eclipse? Interestingly, there is no Eclipse build for Windows on ARM64!

Reexamining the Source Code

Desperation leads me to force java native code compilation at DuckDuckGo and java – Can I force the JVM to natively compile a given method? – Stack Overflow. At this point, a review of the java command options leads me to -XX:-Inline and –XX:CompileOnly=MinimizedStdLibTest.printf. This at least reduces the volume of the hsdis output from hundreds of thousands of lines to just under 5500 lines.

C:\...\devbranch-rel\jdk\bin\javac.exe -g --enable-preview --release 20 MinimizedStdLibTest.java

C:\...\devbranch-rel\jdk\bin\java.exe --enable-preview -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -XX:-Inline -XX:CompileOnly=MinimizedStdLibTest.printf MinimizedStdLibTest > MinimizedStdLibTestOnlyPrintf.asm

Examining this reduced output now helps me realize that the double keyword is what I should have been looking for all along! Look at this snippet with arguments that look similar to my modified test case (where I call with a char, a double, and an integer).

[Verified Entry Point]
  # {method} {0x000001dd8f866158} 'linkToStatic' '(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;IDILjava/lang/invoke/MemberName;)I' in 'java/lang/invoke/MethodHandle'
  # parm0:    c_rarg1:c_rarg1 
                        = 'java/lang/Object'
  # parm1:    c_rarg2:c_rarg2 
                        = 'java/lang/Object'
  # parm2:    c_rarg3:c_rarg3 
                        = 'java/lang/Object'
  # parm3:    c_rarg4   = int
  # parm4:    v0:v0     = double
  # parm5:    c_rarg5   = int
  # parm6:    c_rarg6:c_rarg6 
                        = 'java/lang/invoke/MemberName'
  #           [sp+0x0]  (sp of caller)
  0x000001dd87ae6080:   	nop
  0x000001dd87ae6084:   	ldr	w12, [x6, #0x24]
  0x000001dd87ae6088:   	lsl	x12, x12, #3
  0x000001dd87ae608c:   	ldr	x12, [x12, #0x10]
  0x000001dd87ae6090:   	cbz	x12, #0xc
  0x000001dd87ae6094:   	ldr	x8, [x12, #0x40]
  0x000001dd87ae6098:   	br	x8
  0x000001dd87ae609c:   	b	#-0x56729c          ;   {runtime_call AbstractMethodError throw_exception}

I’m still unsure what the parm fields mean but I’m assuming that the double is still being passed in a vector register! Sure enough, I changed the BoxBindingCalculator instead of the UnboxBindingCalculator. Fixed that then reran the test:

C:\dev\java\abi\devbranch-rel2\jdk\bin\java.exe --enable-preview -jar C:\dev\java\jtreg\lib\jtreg.jar -agentvm -timeoutFactor:4 -concurrency:4 -verbose:fail,error,summary -nativepath:C:\dev\java\abi\devbranch-rel2\support\test\jdk\jtreg\native\lib test/jdk/java/foreign/StdLibTest.java

The test fails but this time there is a fatal error! Feels like progress.

Note: C:\dev\repos\java\forks\jdk\test\jdk\java\foreign\StdLibTest.java uses preview features of Java SE 20.
Note: Recompile with -Xlint:preview for details.

ACTION: testng -- Failed. Unexpected exit from test [exit code: 1]
REASON: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED StdLibTest
TIME:   4.783 seconds
messages:
command: testng --enable-native-access=ALL-UNNAMED StdLibTest
reason: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED StdLibTest
Mode: othervm [/othervm specified]
elapsed time (seconds): 4.783
configuration:
STDOUT:
test StdLibTest.test_printf([INTEGRAL, STRING, CHAR, CHAR]): success
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (assembler_aarch64.hpp:253), pid=11060, tid=5996
#  guarantee(val < (1ULL << nbits)) failed: Field too big for insn
#
# JRE version: OpenJDK Runtime Environment (20.0) (build 20-internal-adhoc.sawesong.jdk)
# Java VM: OpenJDK 64-Bit Server VM (20-internal-adhoc.sawesong.jdk, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, windows-aarch64)
# No core dump will be written. Minidumps are not enabled by default on client versions of Windows
#
# An error report file with more information is saved as:
# C:\dev\repos\java\forks\jdk\JTwork\scratch\0\hs_err_pid11060.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#
hello(42,str,h,h)

Searching for the string “C1-compiled” (which shows up in the hsdis output) reveals its source: nmethod.cpp. The compilation summary is generated by nmethod::print. For an explanation of how to interpret hsdis output, see PrintAssembly output explained! | It’s All Relative (jpbempel.github.io)

Inspecting the Core Dump

Since the fatal error in the JRE states that Minidumps are not enabled by default on client versions of Windows, I enabled collection of dump files using the enable-crash-dumps.bat script. Now we see a minidump written to disk:

C:\dev\java\abi\devbranch5\jdk\bin\java.exe --enable-preview MinimizedStdLibTest
WARNING: A restricted method in java.lang.foreign.Linker has been called
WARNING: java.lang.foreign.Linker::nativeLinker has been called by the unnamed module
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module

# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=\vmreg_aarch64.hpp:48
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (c:\dev\repos\java\forks\jdk\src\hotspot\cpu\aarch64\vmreg_aarch64.hpp:48), pid=14728, tid=11380
#  assert(is_FloatRegister() && is_even(value())) failed: must be
#
# JRE version: OpenJDK Runtime Environment (20.0) (slowdebug build 20-internal-adhoc.sawesong.jdk)
# Java VM: OpenJDK 64-Bit Server VM (slowdebug 20-internal-adhoc.sawesong.jdk, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, windows-aarch64)
# Core dump will be written. Default location: C:\dev\java\abi\tests\hs_err_pid14728.mdmp
#
# An error report file with more information is saved as:
# C:\dev\java\abi\tests\hs_err_pid14728.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#

We can now open the dump file using WinDbg.

0:000> k
 # Child-SP          RetAddr               Call Site
00 00000096`df4ff310 00007ffe`72a05408     ntdll!NtWaitForSingleObject+0x4
01 00000096`df4ff310 00007ffe`6aa90c84     KERNELBASE!WaitForSingleObjectEx+0x88
02 00000096`df4ff3a0 00007ffe`6aa8ae18     jli!CallJavaMainInNewThread+0xac [c:\dev\repos\java\forks\jdk\src\java.base\windows\native\libjli\java_md.c @ 809] 
03 00000096`df4ff3d0 00007ffe`6aa90dc8     jli!ContinueInNewThread+0xd0 [c:\dev\repos\java\forks\jdk\src\java.base\share\native\libjli\java.c @ 2278] 
04 00000096`df4ff4d0 00007ffe`6aa89c18     jli!JVMInit+0x48 [c:\dev\repos\java\forks\jdk\src\java.base\windows\native\libjli\java_md.c @ 974] 
05 00000096`df4ff510 00007ff6`50751408     jli!JLI_Launch+0x360 [c:\dev\repos\java\forks\jdk\src\java.base\share\native\libjli\java.c @ 340] 
06 00000096`df4ff8d0 00007ff6`507517c4     java_exe!main+0x408 [c:\dev\repos\java\forks\jdk\src\java.base\share\native\launcher\main.c @ 166] 
07 (Inline Function) --------`--------     java_exe!invoke_main+0x24
08 00000096`df4ff980 00007ff6`50751850     java_exe!__scrt_common_main_seh+0x124
09 (Inline Function) --------`--------     java_exe!__scrt_common_main+0x8
0a 00000096`df4ff9c0 00007ffe`740b84a8     java_exe!mainCRTStartup+0x10
0b 00000096`df4ff9d0 00007ffe`76fc3108     kernel32!BaseThreadInitThunk+0x38
0c 00000096`df4ffa10 00000000`00000000     ntdll!RtlUserThreadStart+0x48

Running in WinDbg

Decide to run java under the debugger and see what happens.

1. Launch WinDbg and go to File > Open Executable…
2. Browse to the java.exe path.
3. Specify the starting directory containing the compiled MinimizedStdLibTest file.
4. Specify these arguments: --enable-preview MinimizedStdLibTest then click Open.
5. Press F5 to start the program.

After a few breaks due to unhandled exceptions, I decide to look up the warnings in the text on-screen when a foreign function API is invoked. These messages are from Reflection.ensureNativeAccess and are called by …

WARNING: A restricted method in java.lang.foreign.Linker has been called
WARNING: java.lang.foreign.Linker::nativeLinker has been called by the unnamed module
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module

Debugging in Visual Studio 2019

Create a C++ Console Application then open its Configuration Properties. On the Debug page, change the command, command arguments, and working directory to that of the newly built java.exe. Here are some interesting methods based on exploring after setting breakpoints in methodHandles.cpp:

1. InterpreterRuntime::resolve_from_cache
2. MethodHandles::resolve_MemberName
3. JavaCallArguments (from InstanceKlass.cpp:1163)
4. InterpreterRuntime::prepare_native_call
5. NativeLookup::lookup reveals to me the -verbose:jni flag.

C:\dev\repos\java\forks\dups\jdk\build\windows-x86_64-server-slowdebug\jdk\bin\javac.exe -g --enable-preview --release 20 MinimizedStdLibTest.java

C:\dev\repos\java\forks\dups\jdk\build\windows-x86_64-server-slowdebug\jdk\bin\java.exe --enable-preview -XX:+UnlockDiagnosticVMOptions -XX:+LogCompilation -XX:LogFile=jit_compiler.log -verbose:jni MinimizedStdLibTest
...
[2.232s][debug][jni,resolve] [Dynamic-linking native method sun.nio.ch.FileDispatcherImpl.size0 ... JNI]
[2.592s][debug][jni,resolve] [Dynamic-linking native method jdk.internal.foreign.abi.NativeEntryPoint.registerNatives ... JNI]
[2.592s][debug][jni,resolve] [Registering JNI native method jdk.internal.foreign.abi.NativeEntryPoint.makeDowncallStub]
[2.592s][debug][jni,resolve] [Registering JNI native method jdk.internal.foreign.abi.NativeEntryPoint.freeDowncallStub0]
hello(h,1.2345,42)

There is a NativeEntryPoint.java and NativeEntryPoint.cpp. Other interesting methods:

1. DowncallLinker::make_downcall_stub creates a CodeBuffer on line 98, which is initialized by CodeBuffer::initialize.

There are threads with native code (such as the methods above) but no method info. I think those are Java methods. I end up stepping through the code on x64 to gain a better understanding of how the native code stubs are generated. VZEROUPPER motivates a quick detour into AVX-512 just to get a better feel of what it’s about. The instruction set reference (from Intel® 64 and IA-32 Architectures Software Developer Manuals) explains that in 64-bit mode, VZEROUPPER zeroes the bits in positions 128 and higher in YMM0-YMM15 and ZMM0-ZMM15.

Reexamining the Assembly

I decide to find a way to compile everything to assembly. java – Can I force the JVM to natively compile a given method? – Stack Overflow suggests the -Xcomp flag, which works wonders!

javac.exe -g --enable-preview --release 20 MinimizedStdLibTest.java

java.exe --enable-preview -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -XX:-Inline -XX:CompileOnly=MinimizedStdLibTest.printf -Xcomp MinimizedStdLibTest > MinimizedStdLibTestAsmForPrintfOnly.asm

I end up updating the test to have a single MethodHandle.invoke() call on its own line to simplify narrowing down the call in the disassembly. To simplify debugging even further, I create another test (MinimizedStdLibTest20Args) with 20 arguments (most of them doubles) that need to be formatted. This should make it easier to identify the code I am interested in and how these arguments are passed. I have a better grasp of x86-64 architecture so that seems like a better place to start examining to better understanding how this native call is handled.

amd64 Disassembly

There are several verified entry points with these many parameters. Why? Here’s the last one on my Intel(R) Xeon(R) W-2133 CPU.

[Verified Entry Point]
  # {method} {0x000002876ccd2e30} 'linkToSpecial' '(Ljava/lang/Object;JJIDDIDDDDDDDDDDDDDDDDDLjava/lang/invoke/MemberName;)I' in 'java/lang/invoke/MethodHandle'
  # parm0:    rdx:rdx   = 'java/lang/Object'
  # parm1:    r8:r8     = long
  # parm2:    r9:r9     = long
  # parm3:    rdi       = int
  # parm4:    xmm0:xmm0   = double
  # parm5:    xmm1:xmm1   = double
  # parm6:    rsi       = int
  # parm7:    xmm2:xmm2   = double
  # parm8:    xmm3:xmm3   = double
  # parm9:    xmm4:xmm4   = double
  # parm10:   xmm5:xmm5   = double
  # parm11:   xmm6:xmm6   = double
  # parm12:   xmm7:xmm7   = double
  # parm13:   [sp+0x0]   = double  (sp of caller)
  # parm14:   [sp+0x8]   = double
  # parm15:   [sp+0x10]   = double
  # parm16:   [sp+0x18]   = double
  # parm17:   [sp+0x20]   = double
  # parm18:   [sp+0x28]   = double
  # parm19:   [sp+0x30]   = double
  # parm20:   [sp+0x38]   = double
  # parm21:   [sp+0x40]   = double
  # parm22:   [sp+0x48]   = double
  # parm23:   [sp+0x50]   = double
  # parm24:   rcx:rcx   = 'java/lang/invoke/MemberName'
 ;; verify_klass {
  0x000002875655e580:   	testq	%rcx, %rcx
  0x000002875655e583:   	je	0x40
  0x000002875655e589:   	pushq	%rdi
  0x000002875655e58a:   	pushq	%r10
  0x000002875655e58c:   	movl	0x8(%rcx), %edi
  0x000002875655e58f:   	movabsq	$0x800000000, %r10
  0x000002875655e599:   	addq	%r10, %rdi
  0x000002875655e59c:   	movabsq	$0x7ffc8959c6a0, %r10;   {external_word}
  0x000002875655e5a6:   	cmpq	(%r10), %rdi
  0x000002875655e5a9:   	je	0x36
  0x000002875655e5af:   	movq	0x40(%rdi), %rdi
  0x000002875655e5b3:   	movabsq	$0x7ffc8959c6a0, %r10;   {external_word}
  0x000002875655e5bd:   	cmpq	(%r10), %rdi
  0x000002875655e5c0:   	je	0x1f
  0x000002875655e5c6:   	popq	%r10
  0x000002875655e5c8:   	popq	%rdi
 ;; MemberName required for invokeVirtual etc.
  0x000002875655e5c9:   	movabsq	$0x7ffc88f3a110, %rcx;   {external_word}
  0x000002875655e5d3:   	andq	$-0x10, %rsp
  0x000002875655e5d7:   	movabsq	$0x7ffc88127ef0, %r10;   {runtime_call MacroAssembler::debug64}
  0x000002875655e5e1:   	callq	*%r10
  0x000002875655e5e4:   	hlt
 ;; L_ok:
  0x000002875655e5e5:   	popq	%r10
  0x000002875655e5e7:   	popq	%rdi
 ;; } verify_klass
.
.
.

The string “MemberName required for invokeVirtual etc” looks like a unique string and is therefore a reasonable one to use to find the code that set up the entry point. It comes from the generate_method_handle_dispatch method. Placing a breakpoint here reveals an interesting stack:

jvm.dll!MethodHandles::generate_method_handle_dispatch(MacroAssembler * _masm, vmIntrinsicID iid, RegisterImpl * receiver_reg, RegisterImpl * member_reg, bool for_compiler_entry) Line 364	C++
 	jvm.dll!gen_special_dispatch(MacroAssembler * masm, const methodHandle & method, const BasicType * sig_bt, const VMRegPair * regs) Line 1508	C++
 	jvm.dll!SharedRuntime::generate_native_wrapper(MacroAssembler * masm, const methodHandle & method, int compile_id, BasicType * in_sig_bt, VMRegPair * in_regs, BasicType ret_type) Line 1572	C++
 	jvm.dll!AdapterHandlerLibrary::create_native_wrapper(const methodHandle & method) Line 3159	C++
 	jvm.dll!SystemDictionary::find_method_handle_intrinsic(vmIntrinsicID iid, Symbol * signature, JavaThread * __the_thread__) Line 2017	C++
 	jvm.dll!LinkResolver::lookup_polymorphic_method(const LinkInfo & link_info, Handle * appendix_result_or_null, JavaThread * __the_thread__) Line 446	C++
 	jvm.dll!LinkResolver::resolve_method(const LinkInfo & link_info, Bytecodes::Code code, JavaThread * __the_thread__) Line 756	C++
 	jvm.dll!LinkResolver::linktime_resolve_static_method(const LinkInfo & link_info, JavaThread * __the_thread__) Line 1106	C++
 	jvm.dll!LinkResolver::resolve_static_call(CallInfo & result, const LinkInfo & link_info, bool initialize_class, JavaThread * __the_thread__) Line 1072	C++
 	jvm.dll!MethodHandles::resolve_MemberName(Handle mname, Klass * caller, int lookup_mode, bool speculative_resolve, JavaThread * __the_thread__) Line 777	C++
 	jvm.dll!MHN_resolve_Mem(JNIEnv_ * env, _jobject * igcls, _jobject * mname_jh, _jclass * caller_jh, long lookup_mode, unsigned char speculative_resolve) Line 1252	C++
 	0000020a0a26fb92()	Unknown
 	0000020a0058eb00()	Unknown
 	0000005f992fd040()	Unknown
 	0000005f992fd010()	Unknown

This is essentially all the interesting action I have been searching for! Especially AdapterHandlerLibrary::create_native_wrapper, which calls SharedRuntime::java_calling_convention and SharedRuntime::generate_native_wrapper. The latter are exactly what I’ve been seeking!

What does the new_native_nmethod implementation actually do? It ends up calling this nmethod constructor that reveals the existence of the PrintNativeNMethods flag.

javac.exe -g --enable-preview --release 20 MinimizedStdLibTest20Args.java

java.exe --enable-preview -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -XX:-Inline -XX:CompileOnly=MinimizedStdLibTest20Args.printf -Xcomp MinimizedStdLibTest20Args > MinimizedStdLibTest20ArgsAsmForPrintfOnly.asm

Some questions from inspecting the verify_klass method:

1. You can have spaces after the -> operator. See the expansion of __
2. You can have #defines inside the class itself since they are processed before the compiler is invoked. See c++ – Is it possible to use #define inside a function? – Stack Overflow.

The VerifyOops flag is off by default so the verify_oop doesn’t generate any code. The testptr is therefore the first MacroAssembler code to be generated. Notice that the code jumps to the MemberName required for invokeVirtual etc label if rcx is zero – that must be error-handling code. The jz mnemonic would be preferrable to je (see assembly – Difference between JE/JNE and JZ/JNZ – Stack Overflow) but they are identical opcodes. Here is the listing with links to the methods that generated them.

...
  # parm24:   rcx:rcx   = 'java/lang/invoke/MemberName'
 ;; verify_klass {
  0x000002875655e580:   	testq	%rcx, %rcx
  0x000002875655e583:   	je	0x40
  0x000002875655e589:   	pushq	%rdi
  0x000002875655e58a:   	pushq	%r10
  0x000002875655e58c:   	movl	0x8(%rcx), %edi
  0x000002875655e58f:   	movabsq	$0x800000000, %r10
  0x000002875655e599:   	addq	%r10, %rdi
  0x000002875655e59c:   	movabsq	$0x7ffc8959c6a0, %r10;   {external_word}
  0x000002875655e5a6:   	cmpq	(%r10), %rdi
  0x000002875655e5a9:   	je	0x36 // L_ok
  0x000002875655e5af:   	movq	0x40(%rdi), %rdi
  0x000002875655e5b3:   	movabsq	$0x7ffc8959c6a0, %r10;   {external_word}
  0x000002875655e5bd:   	cmpq	(%r10), %rdi
  0x000002875655e5c0:   	je	0x1f // L_ok
  0x000002875655e5c6:   	popq	%r10
  0x000002875655e5c8:   	popq	%rdi
 ;; MemberName required for invokeVirtual etc.
  0x000002875655e5c9:   	movabsq	$0x7ffc88f3a110, %rcx;   {external_word}
  0x000002875655e5d3:   	andq	$-0x10, %rsp
  0x000002875655e5d7:   	movabsq	$0x7ffc88127ef0, %r10;   {runtime_call MacroAssembler::debug64}
  0x000002875655e5e1:   	callq	*%r10
  0x000002875655e5e4:   	hlt
 ;; L_ok:
  0x000002875655e5e5:   	popq	%r10
  0x000002875655e5e7:   	popq	%rdi
 ;; } verify_klass
.
.
.

The movl is a 32-bit mov of the klass* into edi – see gcc – The difference between mov and movl instruction in X86? – Stack Overflow. The offset of 8 is the klass offset in bytes. This klass offset is computed using the offsetof macro. From the beginning of the oopDesc class definition below, the klass offset is 8 to accomodate the markWord.

class oopDesc {
  friend class VMStructs;
  friend class JVMCIVMStructs;
 private:
  volatile markWord _mark;
  union _metadata {
    Klass*      _klass;
    narrowKlass _compressed_klass;
  } _metadata;

The first movabsq instruction loads (int64_t)CompressedKlassPointers::base() into the temporary register r10. As per NarrowPtrStruct._base, this is the base address for oop-within-java-object materialization. Not yet exactly sure whether that means an offset to add to the klass* to get the virtual address of the object since this base is added to the klass* in rdi. That addition ends the MacroAssembler::load_klass call.

The 2nd movabsq instruction loads the external klass address of the klass with vmClassID java_lang_invoke_MemberName. This value is then compared with the computed klass address in r10. If these 2 values are equal, then all is well and the CPU will branch to L_ok. If this branch is not taken, then the super_check_offset of the MemberName Klass is computed by Klass::super_check_offset. This offset indicates where to look to observe a supertype. So for my purposes, everything in the ;; verify_klass {... ;; } verify_klass section can be ignored since it is MemberName validation.

Without looking at the rest of the assembly code, the key thing to notice is that rcx was assumed to have a MemberName, meaning that by the time all these instructions execute, all the arguments I passed to printf are already in registers/on the stack. A quick detour into the method header is in order though. Here’s the first instance of that signature.

-------------------------- Assembly (native nmethod) ---------------------------

Compiled method (n/a)   16155  119     n 0       java.lang.invoke.MethodHandle::linkToNative(JJIDDIDDDDDDDDDDDDDDDDDL)I (native)
 total in heap  [0x0000021b87aea310,0x0000021b87aea488] = 376
 main code      [0x0000021b87aea480,0x0000021b87aea487] = 7
 stub code      [0x0000021b87aea487,0x0000021b87aea488] = 1

[Disassembly]
--------------------------------------------------------------------------------
[Constant Pool (empty)]

--------------------------------------------------------------------------------

[Verified Entry Point]
  # {method} {0x0000021b978bb868} 'linkToNative' '(JJIDDIDDDDDDDDDDDDDDDDDLjava/lang/Object;)I' in 'java/lang/invoke/MethodHandle'
  # parm0:    rdx:rdx   = long
  # parm1:    r8:r8     = long
  # parm2:    r9        = int
  # parm3:    xmm0:xmm0   = double
  # parm4:    xmm1:xmm1   = double
  # parm5:    rdi       = int
  # parm6:    xmm2:xmm2   = double
  # parm7:    xmm3:xmm3   = double
  # parm8:    xmm4:xmm4   = double
  # parm9:    xmm5:xmm5   = double
  # parm10:   xmm6:xmm6   = double
  # parm11:   xmm7:xmm7   = double
  # parm12:   [sp+0x0]   = double  (sp of caller)
  # parm13:   [sp+0x8]   = double
  # parm14:   [sp+0x10]   = double
  # parm15:   [sp+0x18]   = double
  # parm16:   [sp+0x20]   = double
  # parm17:   [sp+0x28]   = double
  # parm18:   [sp+0x30]   = double
  # parm19:   [sp+0x38]   = double
  # parm20:   [sp+0x40]   = double
  # parm21:   [sp+0x48]   = double
  # parm22:   [sp+0x50]   = double
  # parm23:   rsi:rsi   = 'java/lang/Object'
 ;; jump_to_native_invoker {
  0x0000021b87aea480:   	movq	0x10(%rsi), %r10
  0x0000021b87aea484:   	jmpq	*%r10
[Stub Code]
 ;; } jump_to_native_invoker
  0x0000021b87aea487:   	hlt
--------------------------------------------------------------------------------
[/Disassembly]

What output the parm\d+ strings after the method header? These are from nmethod::print_nmethod_labels. This method also calls Method::print_value_on, which outputs the JJIDDIDDDDDDDDDDDDDDDDDL stuff in the method header. That is the method signature. Some digging around on SO, e.g. Compute a Java function’s signature – Stack Overflow and L, Z and V in Java method signature – Stack Overflow leads me to Java Native Interface Specification: 3 – JNI Types and Data Structures (oracle.com), which explains the types represented by each letter. Inspecting these signatures actually leads me to discover that there are double entries for the ‘linkToNative’ native methods. The difference is the Compiled method (n/a) line.

The string ;; jump_to_native_invoker { comes from MethodHandles::jump_to_native_invoker. I’m pleasantly surprised to see only 2 instances in the disassembly since that will simplify breaking in that code. jump_to_native_invoker mentions NEP, which takes me back to NativeEntryPoint.java and the fact that JVM_RegisterNativeEntryPointMethods get called after the program starts. Is this because NativeEntryPoint’s static constructor calls the native method registerNatives? This prompts a review of how the Java code gets into all this native code.

Java Code Going Native

The test’s printf function calls Linker.downcallHandle on line 119. The implementation of Linker.downcallHandle in my first port goes to AbstractLinker::downcallHandle. That implementation calls the abstract method arrangeDowncall. The AbstractLinker subclass I created (WindowsAArch64Linker) is similar to LinuxAArch64Linker and MacOsAArch64Linker in that it delegates arrangeDowncall to CallArranger.arrangeDowncall. This method in turn creates a new DowncallLinker and calls its getBoundMethodHandle method.

getBoundMethodHandle calls NativeEntryPoint.make. I suspect that this is what causes NativeEntryPoint’s static constructor to be executed (and JVM_RegisterNativeEntryPointMethods and NEP_makeDowncallStub in turn). Also observe that once a NativeEntryPoint has been created, a method handle is created by JLIA.nativeMethodHandle. I think the actual implementation of this is in MethodHandleImpl, which defers to NativeMethodHandle. The makePreparedLambdaForm method has a reference to the ‘linkToNative‘ method I’ve been seeing in the hsdis output.

Here is a particularly interesting callstack showing how NEP_makeDowncallStub ends up calling the DowncallStubGenerator.

>	jvm.dll!DowncallStubGenerator::generate() Line 142	C++
 	jvm.dll!DowncallLinker::make_downcall_stub(BasicType * signature, int num_args, BasicType ret_bt, const ABIDescriptor & abi, const GrowableArray<VMRegImpl *> & input_registers, const GrowableArray<VMRegImpl *> & output_registers, bool needs_return_buffer) Line 101	C++
 	jvm.dll!NEP_makeDowncallStub(JNIEnv_ * env, _jclass * _unused, _jobject * method_type, _jobject * jabi, _jobjectArray * arg_moves, _jobjectArray * ret_moves, unsigned char needs_return_buffer) Line 77	C++
 	0000017244641db1()	Unknown
...

What is interesting about this? The DowncallStubGenerator is not only generating assembly instructions that are most likely what I have been searching for, it also has logging code that is being skipped. That looks like unified logging code! Therefore, using +PrintAssembly was not sufficient to generate the code I wanted to see! Here’s an updated command line after which downcall.txt will contain the results of argument shuffling.

javac.exe -g --enable-preview --release 20 MinimizedStdLibTest20Args.java

java.exe --enable-preview -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -XX:-Inline -XX:CompileOnly=MinimizedStdLibTest20Args.printf -Xcomp -Xlog:foreign+downcall=trace:file=downcall.txt::filecount=0 MinimizedStdLibTest20Args > MinimizedStdLibTest20ArgsAsmForPrintfOnly.asm

Here is a stack revealing a bit more detail about how the arguments are set up.

jvm.dll!SharedRuntime::java_calling_convention(const BasicType * sig_bt, VMRegPair * regs, int total_args_passed) Line 505	C++
jvm.dll!JavaCallingConvention::calling_convention(BasicType * sig_bt, VMRegPair * regs, int num_args) Line 66	C++
jvm.dll!ArgumentShuffle::ArgumentShuffle(BasicType * in_sig_bt, int num_in_args, BasicType * out_sig_bt, int num_out_args, const CallingConventionClosure * input_conv, const CallingConventionClosure * output_conv, VMRegImpl * shuffle_temp) Line 328	C++
jvm.dll!DowncallStubGenerator::generate() Line 141	C++
jvm.dll!DowncallLinker::make_downcall_stub(BasicType * signature, int num_args, BasicType ret_bt, const ABIDescriptor & abi, const GrowableArray<VMRegImpl *> & input_registers, const GrowableArray<VMRegImpl *> & output_registers, bool needs_return_buffer) Line 101	C++
jvm.dll!NEP_makeDowncallStub(JNIEnv_ * env, _jclass * _unused, _jobject * method_type, _jobject * jabi, _jobjectArray * arg_moves, _jobjectArray * ret_moves, unsigned char needs_return_buffer) Line 77	C++
0000017244641db1()	Unknown

More questions about how all this works:

1. What happens after all the hsdis code is executed? Is the final jump to the native code?
2. Where is rbx loaded (since that’s what we’re jumping to)?

AArch64 Disassembly

Having now understood that I can log the downcall stubs using the unified logging flags, this is the stub I get on the Surface Pro X (generated by DowncallStubGenerator::generate)

Argument shuffle {
Move a double from ([-1137525940],[-1137525936]) to ([-1137525916],[-1137525912])
Move a double from ([-1137525948],[-1137525944]) to ([-1137525924],[-1137525920])
Move a double from ([-1137525956],[-1137525952]) to ([-1137525932],[-1137525928])
Move a double from ([-1137525964],[-1137525960]) to ([-1137525940],[-1137525936])
Move a double from ([-1137525972],[-1137525968]) to ([-1137525948],[-1137525944])
Move a double from ([-1137525980],[-1137525976]) to ([-1137525956],[-1137525952])
Move a double from ([-1137525988],[-1137525984]) to ([-1137525964],[-1137525960])
Move a double from ([-1137525996],[-1137525992]) to ([-1137525972],[-1137525968])
Move a double from ([-1137526004],[-1137526000]) to ([-1137525980],[-1137525976])
Move a double from ([-1137526012],[-1137526008]) to ([-1137525988],[-1137525984])
Move a double from (v7,v7) to ([-1137525996],[-1137525992])
Move a double from (v6,v6) to ([-1137526004],[-1137526000])
Move a double from (v5,v5) to ([-1137526012],[-1137526008])
Move a double from (v4,v4) to (c_rarg7,c_rarg7)
Move a double from (v3,v3) to (c_rarg6,c_rarg6)
Move a double from (v2,v2) to (c_rarg5,c_rarg5)
Move a long from (c_rarg1,c_rarg1) to (rscratch2,rscratch2)
Move a byte from (c_rarg3,BAD!) to (c_rarg1,BAD!)
Move a int from (c_rarg4,BAD!) to (c_rarg3,BAD!)
Move a double from (v1,v1) to (c_rarg4,c_rarg4)
Move a long from (c_rarg2,c_rarg2) to (c_rarg0,c_rarg0)
Move a double from (v0,v0) to (c_rarg2,c_rarg2)
Stack argument slots: 26
}

It is immediately evident that there are BAD! registers. Why isn’t there more output as one would expect from looking at the additional logging in DowncallStubGenerator::generate? Well, the JVM crash might have something to do with it…

# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=\vmreg_aarch64.hpp:48
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (c:\dev\repos\java\forks\jdk\src\hotspot\cpu\aarch64\vmreg_aarch64.hpp:48), pid=11888, tid=18884
#  assert(is_FloatRegister() && is_even(value())) failed: must be
#
# JRE version: OpenJDK Runtime Environment (20.0) (slowdebug build 20-internal-adhoc.sawesong.jdk)
# Java VM: OpenJDK 64-Bit Server VM (slowdebug 20-internal-adhoc.sawesong.jdk, compiled mode, tiered, compressed oops, compressed class ptrs, g1 gc, windows-aarch64)
# Core dump will be written. Default location: C:\dev\repos\scratchpad\compilers\tests\aarch64\abi\printf\java\hs_err_pid11888.mdmp
#
# An error report file with more information is saved as:
# C:\dev\repos\scratchpad\compilers\tests\aarch64\abi\printf\java\hs_err_pid11888.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#

The most likely culprit here is arg_shuffle.generate. It ends up in ArgumentShuffle::pd_generate which uses the MacroAssembler::double_move and float_move methods. However, addressing the BAD! registers is really the next step before dealing with the assertion failure.

NEP_makeDowncallStub calls ForeignGlobals::parse_vmstorage, which in turn defers to the architecture-specific ForeignGlobals::vmstorage_to_vmreg implementation. This code returns the BAD register if the VMStorage type and does not match the register type! This must be the culprit! How do I log the asString output?

Rexamining the x64 foreign downcall log below, I notice the BAD registers there too! Perhaps this is not an oddity after all. Could it be NativeCallingConvention::calling_convention marking half slots as bad? Actually, notice that in both x64 and AArch64 logs, only the byte and int have these BAD! entries. This must be the other 32-bit slot for the arguments! This means that the AArch64 log is actually fine!

Argument shuffle {
Move a double from ([79203860],[79203864]) to ([79203908],[79203912])
Move a double from ([79203852],[79203856]) to ([79203900],[79203904])
Move a double from ([79203844],[79203848]) to ([79203892],[79203896])
Move a double from ([79203836],[79203840]) to ([79203884],[79203888])
Move a double from ([79203828],[79203832]) to ([79203876],[79203880])
Move a double from ([79203820],[79203824]) to ([79203868],[79203872])
Move a double from ([79203812],[79203816]) to ([79203860],[79203864])
Move a double from ([79203804],[79203808]) to ([79203852],[79203856])
Move a double from ([79203796],[79203800]) to ([79203844],[79203848])
Move a double from ([79203788],[79203792]) to ([79203836],[79203840])
Move a double from ([79203780],[79203784]) to ([79203828],[79203832])
Move a double from (xmm7,xmm7) to ([79203820],[79203824])
Move a double from (xmm6,xmm6) to ([79203812],[79203816])
Move a double from (xmm5,xmm5) to ([79203804],[79203808])
Move a double from (xmm4,xmm4) to ([79203796],[79203800])
Move a double from (xmm3,xmm3) to ([79203788],[79203792])
Move a double from (xmm2,xmm2) to ([79203780],[79203784])
Move a long from (rdx,rdx) to (r10,r10)
Move a byte from (r9,BAD!) to (rdx,BAD!)
Move a int from (rdi,BAD!) to (r9,BAD!)
Move a double from (xmm1,xmm1) to (xmm2,xmm2)
Move a long from (r8,r8) to (rcx,rcx)
Move a double from (xmm0,xmm0) to (r8,r8)
Stack argument slots: 34
}

Back to the MacroAssembler’s and float_move methods… I think the fmovd instruction I seek is this one with a general purpose register operand. After changing double_move to support fmovd between general purpose and floating point registers, rerunning the test on AArch64 does not give any additional output in the downcall log file. Very strange since I don’t see an assertion failure preventing the logging code from running…

I realize though that instead of trying to mess with WinDbg, I can simply write to the unified logging stream (to which output is already successfully being written). Making the LogStream creation unconditional enables me to verify that the code is indeed being executed. __ flush looks like AbstractAssembler::flush. It is only now that I realize that this is not flushing the output stream of the assembler – it is instead invalidating the CPU’s instruction cache! This is done by calling FlushInstructionCache on Windows.

So how do block comments get written to disk? AbstractAssembler::block_comment ends up passing the comments to an AsmRemarks. The inserted comments will be output by AsmRemarks::print. Turns out flags like PrintAssembly or UnlockDiagnosticVMOptions are required to output these comments. Once the downcall stub has been generated, this output should get written to the log file in DowncallLinker::make_downcall_stub.

After fixing the assertion failure by now checking the register types for fmovd, I get an OOM. Lots of output in the hotspot.log as well. paste it here. The hsdis output ends with this:

...
  0x000001c9479b721c:   	add	x8, x8, #0xd40
  0x000001c9479b7220:   	br	x8
[Stub Code]
  0x000001c9479b7224:   	udf	#0x0
--------------------------------------------------------------------------------
[/Disassembly]
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 18446743994480037248 bytes for Chunk::new
# An error report file with more information is saved as:
# C:\dev\repos\scratchpad\compilers\tests\aarch64\abi\printf\java\hs_err_pid11288.log

The Chunk::new string is from Chunk::operator new. Before debugging this, I try adding a delay to the NEP.make call to see if the logs I want will be written to disk before the process dies but I still get the OOM without additional logging output.

Next idea, terminate the program with an assertion failure to see if the output will be written to disk at termination. _wassert – Search (bing.com) -> c – Why is `_wassert` wrapped in `(..,0)`? – Stack Overflow. The hotspot asserts appear to be defines for the CRT _assert function. The latter calls abort, which on Windows, lets a custom abort signal handler function to run (enabling cleanup of resources or log information). Does the JVM use this?

I sprinkle DowncallLinker::generate with this logging code: ls.print_cr("Returning stub after %d", __LINE__); The output shows that the generate method completes executing successfully. However, I don’t get any output from logging calls one level below it in the callstack – in DowncallLinker::make_downcall_stub. Commenting out the creation of the new RuntimeStub (by using the aforemention logging call then returning nullptr on the previous line) shows that execution makes it to that point successfully. That has got to be the culprint since logging messages after that stub do not appear in the logs. And now looking at the RuntimeStub class, it is evident that it has an operator new implementation!

Let’s take a look at happens in WinDbg. The bp, bu, bm (Set Breakpoint) and x (Examine Symbols) are quite useful. x * shows the local variables and their values. I didn’t have the matching sources on the Surface Pro when trying to step into DowncallLinker::make_downcall_stub so I cleaned up all the custom logging, committed my changes, and rebuilt the JDK.

bp jvm!NEP_makeDowncallStub
g
x *

Surprisingly, the newly built JDK successfully passes the StdLibTest.java. Unfortunately, it regresses VaListTest.java and still fails TestVarArgs.java. The error from VaListTest is surprising since that was passing before I began but it looks like a compiler error:

--------------------------------------------------
TEST: java/foreign/valist/VaListTest.java
TEST JDK: C:\dev\java\abi\devbranch5\jdk

ACTION: build -- Failed. Compilation failed: Compilation failed
REASON: Named class compiled on demand
TIME:   32.591 seconds
messages:
command: build VaListTest
reason: Named class compiled on demand
Test directory:
  compile: VaListTest
elapsed time (seconds): 32.591

ACTION: compile -- Failed. Compilation failed: Compilation failed
REASON: .class file out of date or does not exist
TIME:   32.384 seconds
messages:
command: compile C:\dev\repos\java\forks\jdk\test\jdk\java\foreign\valist\VaListTest.java
reason: .class file out of date or does not exist
...
direct:
C:\dev\repos\java\forks\jdk\test\jdk\java\foreign\valist\VaListTest.java:153: error: cannot find symbol
            = (builder, scope) -> WindowsAArch64Linker.newVaList(builder, scope.scope());
                                                                               ^
  symbol:   method scope()
  location: variable scope of type MemorySession
Note: C:\dev\repos\java\forks\jdk\test\jdk\java\foreign\valist\VaListTest.java uses preview features of Java SE 20.
Note: Recompile with -Xlint:preview for details.
1 error
...

The rvalue in the failing assignment needs to match the other lines (simply replace with WindowsAArch64Linker.newVaList). Then get this:

test VaListTest.testCopy(VaListTest$$Lambda$125/0x000000080013cb10@1156402a, i32): success
test VaListTest.testCopy(): failure
org.testng.internal.reflect.MethodMatcherException:
[public void VaListTest.testCopy(java.util.function.BiFunction,java.lang.foreign.ValueLayout$OfInt)] has no parameters defined but was found to be using a data provider (either explicitly specified or inherited from class level annotation).
Data provider mismatch
Method: testCopy([Parameter{index=0, type=java.util.function.BiFunction, declaredAnnotations=[]}, Parameter{index=1, type=java.lang.foreign.ValueLayout$OfInt, declaredAnnotations=[]}])
Arguments: [(VaListTest$$Lambda$120/0x000000080013c000) VaListTest$$Lambda$120/0x000000080013c000@6a8ce624,(java.lang.foreign.ValueLayout$OfInt) i32]
        at org.testng.internal.reflect.DataProviderMethodMatcher.getConformingArguments(DataProviderMethodMatcher.java:43)
        at org.testng.internal.Parameters.injectParameters(Parameters.java:905)
        at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:34)
        at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
        at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
        at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
        at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at org.testng.TestRunner.privateRun(TestRunner.java:764)
        at org.testng.TestRunner.run(TestRunner.java:585)
        at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
        at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
        at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
        at org.testng.SuiteRunner.run(SuiteRunner.java:286)
        at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
        at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
        at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
        at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
        at org.testng.TestNG.runSuites(TestNG.java:1069)
        at org.testng.TestNG.run(TestNG.java:1037)
        at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:93)
        at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:53)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:578)
        at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:125)
        at java.base/java.lang.Thread.run(Thread.java:1589)

Turns out to be a porting bug in which copy() used winAArch64VaListFactory instead of winAArch64VaListScopedFactory. Thankfully the test passes after this fix. Unfortunately, TestVaArgs.java still fails:

STDOUT:
test TestVarArgs.testVarArgs(0, "f0_V__", VOID, [], []): success
test TestVarArgs.testVarArgs(17, "f0_V_S_DI", VOID, [STRUCT], [DOUBLE, INT]): success
test TestVarArgs.testVarArgs(34, "f0_V_S_IDF", VOID, [STRUCT], [INT, DOUBLE, FLOAT]): success
test TestVarArgs.testVarArgs(51, "f0_V_S_FDD", VOID, [STRUCT], [FLOAT, DOUBLE, DOUBLE]): success
test TestVarArgs.testVarArgs(68, "f0_V_S_DDP", VOID, [STRUCT], [DOUBLE, DOUBLE, POINTER]): success
test TestVarArgs.testVarArgs(85, "f0_V_S_PPI", VOID, [STRUCT], [POINTER, POINTER, INT]): success
test TestVarArgs.testVarArgs(102, "f0_V_IS_FF", VOID, [INT, STRUCT], [FLOAT, FLOAT]): failure
java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
        at java.base/jdk.internal.foreign.abi.aarch64.windows.WindowsAArch64CallArranger$StorageCalculator.regAlloc(WindowsAArch64CallArranger.java:230)
        at java.base/jdk.internal.foreign.abi.aarch64.windows.WindowsAArch64CallArranger$UnboxBindingCalculator.getBindings(WindowsAArch64CallArranger.java:369)
        at java.base/jdk.internal.foreign.abi.aarch64.windows.WindowsAArch64CallArranger.getBindings(WindowsAArch64CallArranger.java:150)
        at java.base/jdk.internal.foreign.abi.aarch64.windows.WindowsAArch64CallArranger.arrangeDowncall(WindowsAArch64CallArranger.java:157)
        at java.base/jdk.internal.foreign.abi.aarch64.windows.WindowsAArch64Linker.arrangeDowncall(WindowsAArch64Linker.java:85)
        at java.base/jdk.internal.foreign.abi.AbstractLinker.lambda$downcallHandle$0(AbstractLinker.java:53)
        at java.base/jdk.internal.foreign.abi.SoftReferenceCache$Node.get(SoftReferenceCache.java:52)
        at java.base/jdk.internal.foreign.abi.SoftReferenceCache.get(SoftReferenceCache.java:38)
        at java.base/jdk.internal.foreign.abi.AbstractLinker.downcallHandle(AbstractLinker.java:51)
        at java.base/java.lang.foreign.Linker.downcallHandle(Linker.java:221)
        at TestVarArgs.testVarArgs(TestVarArgs.java:97)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at ...
        at java.base/java.lang.Thread.run(Thread.java:1589)

test TestVarArgs.testVarArgs(119, "f0_V_IS_IFD", VOID, [INT, STRUCT], [INT, FLOAT, DOUBLE]): success
test TestVarArgs.testVarArgs(136, "f0_V_IS_FFP", VOID, [INT, STRUCT], [FLOAT, FLOAT, POINTER]): success
test TestVarArgs.testVarArgs(153, "f0_V_IS_DDI", VOID, [INT, STRUCT], [DOUBLE, DOUBLE, INT]): success
test TestVarArgs.testVarArgs(170, "f0_V_IS_PDF", VOID, [INT, STRUCT], [POINTER, DOUBLE, FLOAT]): success
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=\code/vmreg.hpp:147
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (c:\dev\repos\java\forks\jdk\src\hotspot\share\code/vmreg.hpp:147), pid=10580, tid=10896
#  assert(is_stack()) failed: Not a stack-based register
#
# JRE version: OpenJDK Runtime Environment (20.0) (slowdebug build 20-internal-adhoc.sawesong.jdk)
# Java VM: OpenJDK 64-Bit Server VM (slowdebug 20-internal-adhoc.sawesong.jdk, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, windows-aarch64)
# Core dump will be written. Default location: C:\dev\repos\java\forks\jdk\JTwork\scratch\0\hs_err_pid10580.mdmp
#
# An error report file with more information is saved as:
# C:\dev\repos\java\forks\jdk\JTwork\scratch\0\hs_err_pid10580.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#

The problem turns out to be the fact that I had removed the vector registers from the list of input registers but the HFA code expects these to exist. The Windows AArch64 ABI also expected these vector registers to be used in this scenario. Restoring them addresses this bug, getting us back to the original failure (before I made any changes):

--------------------------------------------------
TEST: java/foreign/TestVarArgs.java
TEST JDK: C:\dev\java\abi\devbranch6\jdk

ACTION: build -- Passed. All files up to date
REASON: Named class compiled on demand
TIME:   0.015 seconds
messages:
command: build TestVarArgs
reason: Named class compiled on demand
elapsed time (seconds): 0.015

ACTION: testng -- Failed. Unexpected exit from test [exit code: 1]
REASON: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED -Dgenerator.sample.factor=17 TestVarArgs
TIME:   18.911 seconds
messages:
command: testng --enable-native-access=ALL-UNNAMED -Dgenerator.sample.factor=17 TestVarArgs
reason: User specified action: run testng/othervm --enable-native-access=ALL-UNNAMED -Dgenerator.sample.factor=17 TestVarArgs
Mode: othervm [/othervm specified]
elapsed time (seconds): 18.911
configuration:
STDOUT:
test TestVarArgs.testVarArgs(0, "f0_V__", VOID, [], []): success
test TestVarArgs.testVarArgs(17, "f0_V_S_DI", VOID, [STRUCT], [DOUBLE, INT]): success
test TestVarArgs.testVarArgs(34, "f0_V_S_IDF", VOID, [STRUCT], [INT, DOUBLE, FLOAT]): success
test TestVarArgs.testVarArgs(51, "f0_V_S_FDD", VOID, [STRUCT], [FLOAT, DOUBLE, DOUBLE]): success
test TestVarArgs.testVarArgs(68, "f0_V_S_DDP", VOID, [STRUCT], [DOUBLE, DOUBLE, POINTER]): success
test TestVarArgs.testVarArgs(85, "f0_V_S_PPI", VOID, [STRUCT], [POINTER, POINTER, INT]): success
STDERR:
java.lang.RuntimeException: java.lang.IllegalStateException: java.lang.AssertionError: expected [12.0] but found [2.8E-45]
        at TestVarArgs.check(TestVarArgs.java:134)
        at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
        at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:758)
        at TestVarArgs.testVarArgs(TestVarArgs.java:104)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:578)
        at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
        at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:599)
        at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
        at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
        at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
        at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
        at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
        at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at org.testng.TestRunner.privateRun(TestRunner.java:764)
        at org.testng.TestRunner.run(TestRunner.java:585)
        at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
        at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
        at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
        at org.testng.SuiteRunner.run(SuiteRunner.java:286)
        at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
        at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
        at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
        at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
        at org.testng.TestNG.runSuites(TestNG.java:1069)
        at org.testng.TestNG.run(TestNG.java:1037)
        at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:93)
        at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:53)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:578)
        at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:125)
        at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: java.lang.IllegalStateException: java.lang.AssertionError: expected [12.0] but found [2.8E-45]
        at CallGeneratorHelper.lambda$initStruct$10(CallGeneratorHelper.java:443)
        at TestVarArgs.lambda$check$4(TestVarArgs.java:132)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at TestVarArgs.check(TestVarArgs.java:132)
        ... 32 more
Caused by: java.lang.AssertionError: expected [12.0] but found [2.8E-45]
        at org.testng.Assert.fail(Assert.java:99)
        at org.testng.Assert.failNotEquals(Assert.java:1037)
        at org.testng.Assert.assertEqualsImpl(Assert.java:140)
        at org.testng.Assert.assertEquals(Assert.java:122)
        at org.testng.Assert.assertEquals(Assert.java:617)
        at CallGeneratorHelper.lambda$makeArg$8(CallGeneratorHelper.java:413)
        at CallGeneratorHelper.lambda$initStruct$10(CallGeneratorHelper.java:441)
        ... 35 more

Examining the test source shows that upcalls can also be traced using -XX:+TraceOptimizedUpcallStubs. I wonder how many other tests are failing though since I didn’t expect this failure. Rerunning them all results in these failures:

1. TestIntrinsics.java
2. TestUpcallHighArity.java
3. TestVarArgs.java

TestIntrinsics.java appears to be an easier test to minimize. Perhaps sorting out the failure there will mean less work in the more complex test.

1. Show assertion failure here
2. Discuss how to get to the assertion failure using WinDbg
3. Show command line with Xlog to get the downcall log
4. Show how the other data types are shuffled in the downcall log
5. Show how to step into float_move
6. first() -> Token-pasting operator (##) | Microsoft Docs
7. Viewing and Editing Memory in WinDbg – Windows drivers | Microsoft Docs
8. d, da, db, dc, dd, dD, df, dp, dq, du, dw (Display Memory) – Windows drivers | Microsoft Docs

The bug is that reg2offset_out is called on a single physical register on line 5894! This happens because the src.is_single_phys_reg returns false. I break out the local variables to get an explicit breakdown in the debugger:

// A float arg may have to do float reg int reg conversion
void MacroAssembler::float_move(VMRegPair src, VMRegPair dst, Register tmp) {
 VMReg src_first = src.first();
 VMReg dst_first = dst.first();
 if (src_first->is_stack()) {
    if (dst_first->is_stack()) {
      ldrw(tmp, Address(rfp, reg2offset_in(src.first())));
      strw(tmp, Address(sp, reg2offset_out(dst_first)));
    } else {
      ldrs(dst.first()->as_FloatRegister(), Address(rfp, reg2offset_in(src_first)));
    }
  } else if (src_first != dst_first) {
    bool src_is_single_phys_reg = src.is_single_phys_reg();
    bool dst_is_single_phys_reg = dst.is_single_phys_reg();

    bool src_is_float_reg = src_first->is_FloatRegister();
    bool src_is_reg = src_first->is_Register();

    bool dst_is_float_reg = dst_first->is_FloatRegister();
    bool dst_is_reg = dst_first->is_Register();

    if (src_is_single_phys_reg && dst_is_single_phys_reg)
      fmovs(dst_first->as_FloatRegister(), src_first->as_FloatRegister());
    else
      strs(src_first->as_FloatRegister(), Address(sp, reg2offset_out(dst_first)));
  }
}

Interestingly, the src register is a floating point register but the name is c_arg0. It is confusing to me that the regName field in both the source’s _first and _second fields point to the same location as the destination’s _first and _second VMRegImpl::regName pointers. Looking at the source, this makes sense because the regName pointer is a static field (missed this in WinDbg) and is set by the static set_regName method.

Notice that ArgumentShuffle::ArgumentShuffle calls NativeCallingConvention::calling_convention, which in turn calls out_regs[i].set1(reg). The set1 method explicitly sets _second to BAD (which is first() – 1). set2() on the other hand sets _second to first() + 1. The solution is then to simply check whether the dst is a register since it will not be a single physical register in this scenario. This fix addresses the assertion failure. We should now be able to get downcall logging.

java --enable-preview -Xlog:foreign+downcall=trace:file=downcall12.txt::filecount=0 MinimizedTestIntrinsics

MinimizedTestIntrinsics.java still fails with these errors:

java.lang.Exception: Expected 2 but found 4621819117588971520
java.lang.Exception: Expected 0 but found 2
java.lang.Exception: Expected 13 but found 0
java.lang.Exception: Expected a but found

4621819117588971520 is 0x4024000000000000, nothing revealing about that value. The native functions that were invoked must be invoke_high_arity2, invoke_high_arity4, invoke_high_arity5 , and invoke_high_arity6 since they are the only ones that match those expected return values. I remove the loop to run invoke_high_arity2 only. Here’s a snippet of the downcall log:

Argument shuffle {
Move a int from (c_rarg2,BAD!) to (c_rarg0,BAD!)
Move a long from (c_rarg3,c_rarg3) to (c_rarg2,c_rarg2)
Move a float from (v1,BAD!) to (c_rarg3,BAD!)
Move a long from (c_rarg1,c_rarg1) to (rscratch2,rscratch2)
Move a double from (v0,v0) to (c_rarg1,c_rarg1)
Stack argument slots: 0
}
[CodeBlob (0x00000259e688df90)]
Framesize: 4
Runtime Stub (0x00000259e688df90): nep_invoker_blob
--------------------------------------------------------------------------------
Decoding CodeBlob, name: nep_invoker_blob, at  [0x00000259e688e040, 0x00000259e688e118]  216 bytes
  0x00000259e688e040:   	stp	x29, x30, [sp, #-0x10]!
  0x00000259e688e044:   	mov	x29, sp
  0x00000259e688e048:   	sub	sp, x29, #0x10
  0x00000259e688e04c:   	adr	x9, #0x0
  0x00000259e688e050:   	str	x9, [x28, #0x318]
  0x00000259e688e054:   	mov	x9, sp
  0x00000259e688e058:   	str	x9, [x28, #0x310]
  0x00000259e688e05c:   	str	x29, [x28, #0x320]
 ;; 0x4
  0x00000259e688e060:   	orr	x9, xzr, #0x4
  0x00000259e688e064:   	add	x10, x28, #0x3c4
  0x00000259e688e068:   	stlr	w9, [x10]
 ;; { argument shuffle
 ;; bt=int
  0x00000259e688e06c:   	sxtw	x0, w2
 ;; bt=long
  0x00000259e688e070:   	mov	x2, x3
 ;; bt=float
  0x00000259e688e074:   	fmov	w3, s1
 ;; bt=long
  0x00000259e688e078:   	mov	x9, x1
 ;; bt=double
  0x00000259e688e07c:   	fmov	x1, d0
 ;; } argument shuffle
  0x00000259e688e080:   	blr	x9

Notice that the instructions correctly load the registers x0-x3. The question now is where the return value is used after this function. Here are the rest of the instructions:

 ;; 0x5
  0x00000259e688e084:   	mov	x9, #0x5
  0x00000259e688e088:   	str	w9, [x28, #0x3c4]
  0x00000259e688e08c:   	dmb	ish
  0x00000259e688e090:   	add	x9, x28, #0x3c8
  0x00000259e688e094:   	ldar	x9, [x9]
  0x00000259e688e098:   	cmp	x29, x9
  0x00000259e688e09c:   	b.hi	#0x3c
  0x00000259e688e0a0:   	ldr	w9, [x28, #0x3c0]
  0x00000259e688e0a4:   	cbnz	w9, #0x34
 ;; 0x8
  0x00000259e688e0a8:   	orr	x9, xzr, #0x8
  0x00000259e688e0ac:   	add	x10, x28, #0x3c4
  0x00000259e688e0b0:   	stlr	w9, [x10]
 ;; reguard stack check
  0x00000259e688e0b4:   	ldrb	w9, [x28, #0x450]
  0x00000259e688e0b8:   	cmp	w9, #0x2
  0x00000259e688e0bc:   	b.eq	#0x3c
  0x00000259e688e0c0:   	str	xzr, [x28, #0x310]
  0x00000259e688e0c4:   	str	xzr, [x28, #0x320]
  0x00000259e688e0c8:   	str	xzr, [x28, #0x318]
  0x00000259e688e0cc:   	mov	sp, x29
  0x00000259e688e0d0:   	ldp	x29, x30, [sp], #0x10
  0x00000259e688e0d4:   	ret
 ;; { L_safepoint_poll_slow_path
  0x00000259e688e0d8:   	str	x0, [sp]
  0x00000259e688e0dc:   	mov	x0, x28
 ;; 0x7FFF1FB2A870
  0x00000259e688e0e0:   	mov	x9, #0xa870
  0x00000259e688e0e4:   	movk	x9, #0x1fb2, lsl #16
  0x00000259e688e0e8:   	movk	x9, #0x7fff, lsl #32
  0x00000259e688e0ec:   	blr	x9
  0x00000259e688e0f0:   	ldr	x0, [sp]
  0x00000259e688e0f4:   	b	#-0x4c
 ;; } L_safepoint_poll_slow_path
 ;; { L_reguard
  0x00000259e688e0f8:   	str	x0, [sp]
 ;; 0x7FFF1FFFAAD0
  0x00000259e688e0fc:   	mov	x9, #0xaad0
  0x00000259e688e100:   	movk	x9, #0x1fff, lsl #16
  0x00000259e688e104:   	movk	x9, #0x7fff, lsl #32
  0x00000259e688e108:   	blr	x9
  0x00000259e688e10c:   	ldr	x0, [sp]
  0x00000259e688e110:   	b	#-0x50
 ;; } L_reguard
  0x00000259e688e114:   	udf	#0x0

I needed to search for B.cond in the ARM Architecture Reference Manual for A-profile architecture PDF. The HI mnemonic in b.hi means unsigned higher and is equivalent to the condition flags C==1 && Z == 0. This branch is to the safepoint poll slow path, which is the label immediately following the L_safepoint_poll_slow_path comment. I found it strange that 0x00000259e688e0a0 + #0x3c = 0x259E688E0DC, which is the 2nd instruction after the L_safepoint_poll_slow_path label. However, the B.cond documentation states that the program label to be conditionally branched to is given by an offset from the address of the branch instruction.

Looks like most of the above code is not relevant because it doesn’t touch x0. At this point, it seems like the problem could be in the native code we’re branching into. I set a breakpoint in invoke but the code doesn’t seem to make much sense:

bp intrinsics!invoke_high_arity2

Let us disassemble support/test/jdk/jtreg/native/lib/Intrinsics.dll and see what the compiler generated.

cd build\windows-aarch64-server-slowdebug\support\test\jdk\jtreg\native\support\libIntrinsics\
dumpbin /disasm /out:Intrinsics.asm libIntrinsics.obj
dumpbin /all /out:Intrinsics.txt libIntrinsics.obj

Here is the relevant code, which makes it apparent that libIntrinsics is not expecting floating point parameters in general purpose registers!

Dump of file libIntrinsics.obj

File Type: COFF OBJECT

empty:
  0000000000000000: D65F03C0  ret
  0000000000000004: 00000000
identity_bool:
  0000000000000008: D10043FF  sub         sp,sp,#0x10
  000000000000000C: 53001C08  uxtb        w8,w0
  0000000000000010: 390003E8  strb        w8,[sp]
  0000000000000014: 394003E0  ldrb        w0,[sp]
  0000000000000018: 910043FF  add         sp,sp,#0x10
  000000000000001C: D65F03C0  ret
identity_char:
  0000000000000020: D10043FF  sub         sp,sp,#0x10
  0000000000000024: 13001C08  sxtb        w8,w0
  0000000000000028: 390003E8  strb        w8,[sp]
  000000000000002C: 39C003E0  ldrsb       w0,[sp]
  0000000000000030: 910043FF  add         sp,sp,#0x10
  0000000000000034: D65F03C0  ret
...
identity_long:
  0000000000000068: D10043FF  sub         sp,sp,#0x10
  000000000000006C: F90003E0  str         x0,[sp]
  0000000000000070: F94003E0  ldr         x0,[sp]
  0000000000000074: 910043FF  add         sp,sp,#0x10
  0000000000000078: D65F03C0  ret
  000000000000007C: 00000000
identity_float:
  0000000000000080: D10043FF  sub         sp,sp,#0x10
  0000000000000084: BD0003E0  str         s0,[sp]
  0000000000000088: BD4003E0  ldr         s0,[sp]
  000000000000008C: 910043FF  add         sp,sp,#0x10
  0000000000000090: D65F03C0  ret
  0000000000000094: 00000000
identity_double:
  0000000000000098: D10043FF  sub         sp,sp,#0x10
  000000000000009C: FD0003E0  str         d0,[sp]
  00000000000000A0: FD4003E0  ldr         d0,[sp]
  00000000000000A4: 910043FF  add         sp,sp,#0x10
  00000000000000A8: D65F03C0  ret
  00000000000000AC: 00000000
...
invoke_high_arity2:
  0000000000000138: D10083FF  sub         sp,sp,#0x20
  000000000000013C: B9000BE0  str         w0,[sp,#8]
  0000000000000140: FD000FE0  str         d0,[sp,#0x18]
  0000000000000144: F9000BE1  str         x1,[sp,#0x10]
  0000000000000148: BD000FE1  str         s1,[sp,#0xC]
  000000000000014C: 13001C48  sxtb        w8,w2
  0000000000000150: 390003E8  strb        w8,[sp]
  0000000000000154: 13003C68  sxth        w8,w3
  0000000000000158: 790007E8  strh        w8,[sp,#2]
  000000000000015C: 13003C88  sxth        w8,w4
  0000000000000160: 79000BE8  strh        w8,[sp,#4]
  0000000000000164: F9400BE0  ldr         x0,[sp,#0x10]
  0000000000000168: 910083FF  add         sp,sp,#0x20
  000000000000016C: D65F03C0  ret

I update the WindowsAArch64CallArranger to specifically use general purpose registers for floating point data only for variadic FunctionDescriptors. This fixes both TestIntrinsics and TestUpcallHighArity but not TestVarArgs so I create a self contained test for it: MinimizedTestVarArgs.

TestVarArgs

This test depends on the native varargs.dll (built from libVarArgs.c). This DLL can be found in the build/windows-x86_64-server-slowdebug/support/test/jdk/jtreg/native/lib/ directory.

1. How does the test work?
2. It uses upcalls, how do they work?

Here’s how the native upcall linker is invoked to create an upcall stub:

1. Test calls Linker.upcallStub
2. AbstractLinker.upcallStub calls WindowsAArch64Linker.arrangeUpcall
3. CallArranger.arrangeUpcall calls
4. UpcallLinker.make, which calls the native
5. makeUpcallStub

bp varargs!varargs
bp UpcallLinker::make_upcall_stb

Finding Upcall Logs

I’m trying to see the logs for upcalls but realize that I only have the downcall logs! Here’s the updated command line:

java --enable-preview -Xlog:foreign+upcall=trace,foreign+downcall=trace:file=up-and-downcalls.txt::filecount=0 MinimizedTestIntrinsics

These logging options generate argument shuffling output only. I expected to see comments like on_entry.

[8.157s][trace][foreign,upcall] Argument shuffle {
[8.157s][trace][foreign,upcall] Move a long from (c_rarg1,c_rarg1) to (c_rarg3,c_rarg3)
[8.157s][trace][foreign,upcall] Move a int from (c_rarg0,BAD!) to (c_rarg2,BAD!)
[8.157s][trace][foreign,upcall] Stack argument slots: 0
[8.158s][trace][foreign,upcall] }
[8.860s][trace][foreign,downcall] Argument shuffle {
[8.860s][trace][foreign,downcall] Move a long from (c_rarg1,c_rarg1) to (rscratch2,rscratch2)
[8.860s][trace][foreign,downcall] Move a int from (c_rarg3,BAD!) to (c_rarg1,BAD!)
[8.860s][trace][foreign,downcall] Move a long from (c_rarg2,c_rarg2) to (c_rarg0,c_rarg0)
[8.862s][trace][foreign,downcall] Stack argument slots: 0
[8.862s][trace][foreign,downcall] }
[8.862s][trace][foreign,downcall] [CodeBlob (0x0000027b876f0810)]
[8.862s][trace][foreign,downcall] Framesize: 2
[8.862s][trace][foreign,downcall] Runtime Stub (0x0000027b876f0810): nep_invoker_blob
[8.862s][trace][foreign,downcall] --------------------------------------------------------------------------------
[8.862s][trace][foreign,downcall] Decoding CodeBlob, name: nep_invoker_blob, at  [0x0000027b876f08c0, 0x0000027b876f0980]  192 bytes
[8.879s][trace][foreign,downcall]   0x0000027b876f08c0:   	stp	x29, x30, [sp, #-0x10]!
[8.879s][trace][foreign,downcall]   0x0000027b876f08c4:   	mov	x29, sp
...

Turns out the upcallLinker requires the TraceOptimizedUpcallStubs flag to log this information. TODO: improve the consistency of this logging. The Xlog option I’m using is not available in the non-debug product though!

java --enable-preview -XX:+TraceOptimizedUpcallStubs -Xlog:foreign+upcall=trace,foreign+downcall=trace:file=up-and-downcalls.txt::filecount=0 MinimizedTestIntrinsics

That is not sufficient though. Simply outputs this to the command prompt:

[CodeBlob (0x0000025291ffe090)]
Framesize: 0
UpcallStub (0x0000025291ffe090) used for upcall_stub_(Ljava/lang/Object;IJ)V
[CodeBlob (0x0000025291ffe090)]
Framesize: 0
UpcallStub (0x0000025291ffe090) used for upcall_stub_(Ljava/lang/Object;IJ)V
...

The UpcallStub constructor turns out to have the UpcallStub tracing code (notice the stub name “UpcallStub”). It expects the PrintStubCode flag. This outputs the disassembly as I expected but does so for just about everything – 10MB of text. The stub name can be used to narrow down the calls we’re interested in.

java --enable-preview -XX:+PrintStubCode -Xlog:foreign+upcall=trace,foreign+downcall=trace:file=up-and-downcalls.txt::filecount=0 MinimizedTestIntrinsics > upcallStub.asm

To see the corresponding native code, run dumpbin to generate libVarArgs.asm and libVarArgs.txt:

cd build\windows-aarch64-server-slowdebug\support\test\jdk\jtreg\native\support\libVarArgs\
dumpbin /disasm /out:libVarArgs.asm libVarArgs.obj
dumpbin /all /out:libVarArgs.txt libVarArgs.obj

Setting aside all this learning and simply reviewing the Overview of ARM64 ABI conventions, the statement that floating-point values are returned in s0, d0, or v0, as appropriate should be enough to track down the bug. The change I made to the CallArranger switched the floating point storage to a general purpose register whenever floating point storage was requested for a variadic function. However, this doesn’t fix the test, thereby showing the value of understanding exactly how things are flowing through registers!

Understanding libVarArgs

The varargs function does not return a value. Here is an interpretation of the disassembly:

;$LN2:
;;
;; i++
;;
  0000000000000044: B9400BE8  ldr         w8,[sp,#8]
  0000000000000048: 11000508  add         w8,w8,#1
  000000000000004C: B9000BE8  str         w8,[sp,#8]
$LN4:
;;
;; i < num
;;
  0000000000000050: B9401FE9  ldr         w9,[sp,#0x1C]
  0000000000000054: B9400BE8  ldr         w8,[sp,#8]
  0000000000000058: 6B09011F  cmp         w8,w9
  000000000000005C: 5400F66A  bge         $LN3
;;
;; x8 = info
;;
  0000000000000060: F9401FE8  ldr         x8,[sp,#0x38]
;;
;; x10 = &info->argids
;;
  0000000000000064: 9100210A  add         x10,x8,#8
;;
;; x9 = i * 4
;;
  0000000000000068: B9400BE8  ldr         w8,[sp,#8]
  000000000000006C: 93407D09  sxtw        x9,w8
  0000000000000070: D2800088  mov         x8,#4
  0000000000000074: 9B087D29  mul         x9,x9,x8
;;
;; Get the pointer from the call_info
;;
  0000000000000078: F9400148  ldr         x8,[x10]
;;
;; computer the offset of element [i]
;;
  000000000000007C: 8B090108  add         x8,x8,x9
;;
;; w8 = info->argids[i];
;;
  0000000000000080: B9400108  ldr         w8,[x8]
  0000000000000084: B90023E8  str         w8,[sp,#0x20]
  0000000000000088: B94023E8  ldr         w8,[sp,#0x20]
  000000000000008C: B9001BE8  str         w8,[sp,#0x18]
  0000000000000090: B9401BE8  ldr         w8,[sp,#0x18]
;;
;; There are 88 (0x58) enums.
;;
  0000000000000094: 71015D1F  cmp         w8,#0x57
;;
;; Go to default case if not one of the defined enums
;;
  0000000000000098: 5400F3E8  bhi         $LN95
;;
;; w10 = info->argids[i];
;;
  000000000000009C: B9401BEA  ldr         w10,[sp,#0x18]
;;
;; x9 = PC-relative address of $LN100
;;
  00000000000000A0: 1000F509  adr         x9,$LN100
;;
;; uxtw: unsigned word extend
;; load a signed offset from the table at $LN100
;; x8 = sign-extend([x9 + w10 * 4])
;;
  00000000000000A4: B8AA5928  ldrsw       x8,[x9,w10 uxtw #2]
;;
;; x9 = PC-relative address of $LN51 (half-way point in the switch/45th label from here)
;;
  00000000000000A8: 10007969  adr         x9,$LN51
;;
;; x8 = address of the case statement to jump to
;; why the left shift though?
;;
  00000000000000AC: 8B080928  add         x8,x9,x8,lsl #2
  00000000000000B0: D61F0100  br          x8

...
$LN95:
  0000000000001F14: 12800000  mov         w0,#-1
  0000000000001F18: 90000008  adrp        x8,__imp_exit
  0000000000001F1C: F9400108  ldr         x8,[x8,__imp_exit]
  0000000000001F20: D63F0100  blr         x8
$LN188:
  0000000000001F24: 17FFF848  b           $LN2

;; va_end(a_list);
;; This expands to ((void)(a_list = (va_list)0))
;;
$LN3:
  0000000000001F28: D2800008  mov         x8,#0
  0000000000001F2C: F90003E8  str         x8,[sp]

;;
;; cleanup before returning
;;
  0000000000001F30: 9132C3FF  add         sp,sp,#0xCB0
  0000000000001F34: 94000000  bl          __security_pop_cookie
  0000000000001F38: A8C47BFD  ldp         fp,lr,[sp],#0x40
  0000000000001F3C: D65F03C0  ret
$LN100:
  0000000000001F40: FFFFFC38
$LN101:
  0000000000001F44: FFFFFC49

The unconditional branch to the address in x8 is to the upcall stub.Notice from the setup for the branch that the target is invoked by the blr.

Stepping through the code, I decide to look up the void* parameter that was passed into the upcall stub (just before the last instruction of preserve_callee_saved_regs – str d24, [sp, #0xd0]). Perhaps a more reasonable point would be at the end of the argument shuffle but the values will be the same ones below:

0:004> k
 # Child-SP          RetAddr               Call Site
00 0000009b`fe3fe1a0 00007fff`97a31234     0x000001d7`4c436790
01 0000009b`fe3fe1a0 00007fff`97a31234     VarArgs!varargs+0x17c
02 0000009b`fe3fe2f0 000001d7`4c430680     VarArgs!varargs+0x17c
03 0000009b`fe3fefc0 00000000`00000000     0x000001d7`4c430680
0:004> r
 x0=0000000000000000   x1=0000009bfe3fe390   x2=4038000000000000   x3=0000000000000001
 x4=0000000712645c90   x5=00000000fffffffe   x6=000001d7432ecb10   x7=000000000000000e
 x8=000001d74c436740   x9=0000000000000008  x10=0000000000000002  x11=000001d75e0add98
x12=000001d75ede1550  x13=0000000000000000  x14=a2e64eada2e64ead  x15=000001d763b8a2b8
x16=0000679de851517d  x17=ffff04a2bd67b2d3  x18=0000000000000000  x19=0000009bfe3fefd0
x20=0000009bfe3feff0  x21=00007fff16390f90  x22=000001d75ed662b6  x23=000001d751f7d000
x24=0000009bfe3ff0d8  x25=000001d751f7d000  x26=000001d75ed66410  x27=0000000000000000
x28=000001d7432ecb10   fp=0000009bfe3fe2b0   lr=00007fff97a31234   sp=0000009bfe3fe1a0
 pc=000001d74c436790  psr=80000040 N--- EL0
000001d7`4c436790 fd006bf8 str         d24,[sp,#0xD0]
0:004> dq 9bfe3fe390
0000009b`fe3fe390  40380000`00000000 000001d7`64267890
0000009b`fe3fe3a0  0000009b`fe3fe3c0 00007fff`149502a8
0000009b`fe3fe3b0  000001d7`64267890 00007fff`1494b754

The 64-bit value is 0x4038000000000000. The program below confirms this value to be 24.0. Therefore, everything has been correctly set up for the upcall.

#include <stdio.h>

int main()
{
    __int64 i = 0x4038000000000000;
    double* d = (double*)&i;
    printf("%f", *d);
}

1. Review earlier 0x4024 value.
2. Review set of volatile registers defined by the ABI since that’s what ends up in the upcall stub.

Let us take another look at upcallStub.asm. The hex value at the beginning of the receiver section is the immediate value being loaded into the register on the next line. It is generated by MacroAssembler::movptr and is the pointer to the reciever jobject. The movptr method explains that since the AArch64 mode VA space is 48 bits in size, only 3 instructions are sufficient to create a patchable instruction sequence that can reach anywhere. This helps me notice that the 3 mov instructions are recreating that immediate value in the comments.

1. Need to figure out how to set a breakpoint only when i == 2 in the varargs C function.

Now that I can break just before the branch into Java code, the question is where does the Java calling convention expect arguments to be? jvm – What’s the calling convention for the Java code in Linux platform? – Stack Overflow gives me the hint that I should be looking at assembler_aarch64.hpp for this info. At this point, I realize that I should have compiled the Java code as well. Back to the fuller command line:

javac -g --enable-preview --release 20 MinimizedTestVarArgs.java
java --enable-preview -Xlog:foreign+upcall=trace,foreign+downcall=trace:file=up-and-downcalls-TestVarArgs-16-05.txt::filecount=0 -XX:+PrintAssembly -XX:+PrintStubCode -XX:-Inline MinimizedTestVarArgs > TestVarArgs.asm

There is a level of indirection that works against this idea: the stub uses an offset into the receiver to retrieve the method to call. That is not directly output in the disassembly!

1. A good place to break is jvm!UpcallLinker::on_entry

Why don’t we review how these cases are handled in the native code? Here is the definition of va_arg from C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31823\include\vadefs.h:

#define __crt_va_arg(ap, t)                                                 \
   ((sizeof(t) > (2 * sizeof(__int64)))                                   \
       ? **(t**)((ap += sizeof(__int64)) - sizeof(__int64))               \
       : *(t*)((ap += _SLOTSIZEOF(t) + _APALIGN(t,ap)) - _SLOTSIZEOF(t)))

Below is the disassembly for the first case in libVarArgs.c. The 2nd definition of __crt_va_arg is used on ARM64. The _SLOTSIZEOF evaluates to 8 for both int and double. TODO: finish explaining this assembly.

$LN7:
  00000000000000B4: D2800009  mov         x9,#0
  00000000000000B8: F94003E8  ldr         x8,[sp]
  00000000000000BC: CB080128  sub         x8,x9,x8
  00000000000000C0: 92400508  and         x8,x8,#3
  00000000000000C4: 91002109  add         x9,x8,#8
  00000000000000C8: F94003E8  ldr         x8,[sp]
  00000000000000CC: 8B090108  add         x8,x8,x9
  00000000000000D0: F90003E8  str         x8,[sp]
  00000000000000D4: F94003E8  ldr         x8,[sp]
  00000000000000D8: D1002108  sub         x8,x8,#8
  00000000000000DC: B9400108  ldr         w8,[x8]
  00000000000000E0: B90027E8  str         w8,[sp,#0x24]
  00000000000000E4: 910093E1  add         x1,sp,#0x24
  00000000000000E8: B9400BE0  ldr         w0,[sp,#8]
  00000000000000EC: F9400BE8  ldr         x8,[sp,#0x10]
  00000000000000F0: D63F0100  blr         x8
  00000000000000F4: 1400078C  b           $LN188

So why does TestUpcallArity pass? It does not use variadic functions! I update MinimizedTestVarArgs to show the function signature codes when it fails. From the resulting log, a struct is being passed to the downcall.

f0_V_S_F java.lang.Exception: Expected 12.0 but found 7.95336E-11
f0_V_S_D java.lang.Exception: Expected 24.0 but found 9.022351855793E-312
f0_V_S_FF java.lang.Exception: Expected 12.0 but found 2.2120472E-11
f0_V_S_FF java.lang.Exception: Expected 12.0 but found 5.96E-43
f0_V_S_DD java.lang.Exception: Expected 24.0 but found 9.02227530708E-312
f0_V_S_DD java.lang.Exception: Expected 24.0 but found 4.9E-324
f0_V_S_FFF java.lang.Exception: Expected 12.0 but found 2.384152E-12
f0_V_S_FFF java.lang.Exception: Expected 12.0 but found 5.96E-43
f0_V_S_FFF java.lang.Exception: Expected 12.0 but found 1.4E-45
f0_V_S_DDD java.lang.Exception: Expected 24.0 but found 9.020261611475E-312
f0_V_S_DDD java.lang.Exception: Expected 24.0 but found 9.02168631996E-312
f0_V_S_DDD java.lang.Exception: Expected 24.0 but found 1.8075E-319
f0_V_IS_F java.lang.Exception: Expected 12.0 but found 2.8E-45
f0_V_IS_D java.lang.Exception: Expected 24.0 but found 9.9E-324
f0_V_IS_FF java.lang.Exception: Expected 12.0 but found 2.8E-45
f0_V_IS_FF java.lang.Exception: Expected 12.0 but found 0.0
f0_V_IS_DD java.lang.Exception: Expected 24.0 but found 9.9E-324
f0_V_IS_DD java.lang.Exception: Expected 24.0 but found 2.08E-322
f0_V_IS_FFF java.lang.Exception: Expected 12.0 but found 2.8E-45
f0_V_IS_FFF java.lang.Exception: Expected 12.0 but found 0.0
f0_V_IS_FFF java.lang.Exception: Expected 12.0 but found 5.9E-44

These signatures remind me of seeing 24.0 in d0 when debugging. I didn’t think about this as much as I should have. Breaking on the branch to the address from the table is the best way to examine the state of the registers and notice 24.0 in d0. Interestingly, only the general purpose registers are shown. See r (Registers) – Windows drivers | Microsoft Docs for details on how to view and modify additional registers.

bp VarArgs!varargs+0xb0
r
rF

The pattern in the above failing signatures implies that the UnboxBindingCalculator is using the STRUCT_HFA case to place them in floating point registers. Changing the code to use the STRUCT_REGISTER case for these causes some of the cases to pass (updated MinimizedTestVarArgs as well). The last case doesn’t work though..

Starting test 6 for f0_V_S_F ... Finished test 6 for f0_V_S_F
Starting test 7 for f0_V_S_D ... Finished test 7 for f0_V_S_D
Starting test 14 for f0_V_S_FF ... Finished test 14 for f0_V_S_FF
Starting test 19 for f0_V_S_DD ... Finished test 19 for f0_V_S_DD
Starting test 46 for f0_V_S_FFF ... Finished test 46 for f0_V_S_FFF
Starting test 67 for f0_V_S_DDD ...

My initial hypothesis is that there weren’t enough registers, but if that’s the case then why does the 3 floats case work? The above bp command in the debugger shows that $LN73 of VarArgs.dll is executed and that the integer registers contain the 4 floating point values (why 5 and not 3)? Turns out the reason the test failed to be complete is because there was an AccessViolation when loading the pair x8 and x9 from [x10].

Breakpoint 0 hit
VarArgs!varargs+0xb0:
00007fff`8f0f1168 d61f0100 br          x8 {VarArgs!varargs+0x1784 (00007fff`8f0f283c)}
0:005> r
 x0=0000018ccf9f3440   x1=0000000000000001   x2=4038000000000000   x3=4038000000000000
 x4=4038000000000000   x5=4038000000000000   x6=4038000000000000   x7=00000004e51301d8
 x8=00007fff8f0f283c   x9=00007fff8f0f208c  x10=0000000000000042  x11=0000018cc926be58
x12=0000018ccb9df990  x13=0000000000000000  x14=a2e64eada2e64ead  x15=0000018ccf798b7a
x16=0000b28569b6ec1d  x17=ffff9f321223209b  x18=0000000000000000  x19=0000000718bfed10
x20=0000000718bfed30  x21=00007fff16390f90  x22=0000018ccb95b2ba  x23=0000018cbf929000
x24=0000000718bfee58  x25=0000018cbf929000  x26=0000018ccb95b410  x27=0000000000000000
x28=0000018caf8c9b10   fp=0000000718bfecc0   lr=00007fff8f0f10d0   sp=0000000718bfe000
 pc=00007fff8f0f1168  psr=80000000 N--- EL0
VarArgs!varargs+0xb0:
00007fff`8f0f1168 d61f0100 br          x8 {VarArgs!varargs+0x1784 (00007fff`8f0f283c)}
0:005> rF

 d0=    2.47032822921e-323   d1=    5.92454341027e-270
 d2=    -3.98809525708e-16   d3=    -3.98809525708e-16
 d4=                     0   d5=                     0
 d6=                     0   d7=                     0
 d8=                     0   d9=                     0
d10=                     0  d11=                     0
d12=                     0  d13=                     0
d14=                     0  d15=                     0
d16=    6.46572227901e+170  d17=                     0
d18=     1.3906500245e-309  d19=     2.25252634258e-23
d20=                     0  d21=                     0
d22=     2.25252634258e-23  d23=                     0
d24=                     0  d25=                     0
d26=                     0  d27=                     0
d28=                     0  d29=                     0
d30=                     0  d31=                     0
VarArgs!varargs+0xb0:
00007fff`8f0f1168 d61f0100 br          x8 {VarArgs!varargs+0x1784 (00007fff`8f0f283c)}
0:005>

At this point, my curiosity about the correct solution for these registers leads me to create a self-contained varargs test SimpleVarArgs.c. The disassembly of call_S_DDD shows the struct being placed on the stack and a pointer to it being passed to varargs.

Other Notes

double and long each use 2 slots and void uses 0 as per the type2size array.

Note that the targetAddrStorage field is used by the downcall linker to branch to the native function. The retBufAddrStorage field is used to pass the address of the return buffer to the native function being invoked. See jdk/foreignGlobals_aarch64.cpp for how the Java ABIDescriptor is parsed in native code into an ABIDescriptor struct. The only usage of the _integer_additional_volatile_registers field seems to be the ABIDescriptor::is_volatile_reg method. Same for the _vector_additional_volatile_registers field. The only usage of is_volatile_reg is in the upcall linker, which saves and restores the callee saved registers. See the compute_reg_save_area_size, preserve_callee_saved_registers, and restore_callee_saved_registers methods. The strange thing is that the Overview of ARM ABI Conventions | Microsoft Docs document does not define what a volatile register is. Here is the definition from the x64 ABI page.

Volatile registers are scratch registers presumed by the caller to be destroyed across a call. Nonvolatile registers are required to retain their values across a function call and must be saved by the callee if used.

x64 ABI conventions | Microsoft Docs

Just when I think I’m done fixing up the CallArranger so that all the Windows AArch64 floating point ABI changes are in there, I realize when going through the other changes in the PR I would open that I don’t understand exactly what WindowsAArch64VaList is used for. I based it on the MacOsAArch64VaList class but perhaps WinVaList would be more appropriate.

While reviewing all this, I take a peek at the CallArranger tests. All but one of them use CallArranger.LINUX. This means I need to create a test for Windows. After replacing LINUX with WINDOWS, I run the test on the Surface Pro X and it passes, even though it should definitely fail! Oh boy, this turns out to be a copy/paste issue – I hadn’t updated the @run testng ClassName to the new class name so a different test was running!

Structure of CallArranger Tests

testStructHFA1 creates a struct with 2 floats for a downcall. One of the arrays it passes to checkArgumentBindings starts off with the dup() binding, which “duplicates the value on the top of the operand stack (without popping it!),
and pushes the duplicate onto the operand stack.“

Breaking Down WinVaList

As part of this port, I needed to implement VaList. Understanding the Windows x64 implementation (WinVaList) is helpful. The skip() method repeatedly calls MemorySegment.asSlice() to create a memory segment offset by VA_SLOT_SIZE_BYTES. WinVaList.Builder also uses VA_SLOT_SIZE_BYTES for each argument whereas MacOsAArch64VaList.Builder uses the sizeOf method to compute the slot sizes for the arguments. The definition of Utils.alignUp (shown below) is what I thought the builder was using but it is actually SharedUtils.alignUp.

// Utils.alignUp
public static long alignUp(long n, long alignment) {
    return (n + alignment - 1) & -alignment;
}

// SharedUtils.alignUp
public static long alignUp(long addr, long alignment) {
    return ((addr - 1) | (alignment - 1)) + 1;
}

// Compare these to _SLOTSIZEOF(t) in vadefs.h
#define _SLOTSIZEOF(t)  ((sizeof(t) + _VA_ALIGN - 1) & ~(_VA_ALIGN - 1))

This enables the AArch64 implementation to align up the size required for STRUCT_REGISTER and STRUCT_HFA layouts. This also matches the definition of Visual Studio’s __crt_va_arg in vadefs.h. The Builder.build() method uses MemorySegment.copyFrom().

Viewing Compilation Logs

Viewing sources in VS reveals that compilation logs can be saved. Java JIT compiler explained – Part 1 – The Bored Dev.

Applying Changes to Panama Repo

It’s only when I start preparing to engage the OpenJDK mailing lists about a PR that I discover that there’s a separate repo for the Foreign Function & Memory API development so I need to apply my changes onto my new fork of the panama-foreign repo.

git clone https://github.com/swesonga/panama-foreign
cd panama-foreign
git remote add myjdk https://github.com/swesonga/jdk
git fetch myjdk
git log -1 myjdk/WinAArch64ABI
git switch -c WinAArch64ABI
git cherry-pick 3f70c10369b297f15e53997f600a80680bfa698a

Interesting learning about the rev-parse command from How to find the hash of branch in Git? – Stack Overflow.

Changes from Panama

There were some conflicts to resolve after cherry-picking but nothing too bad. Looks like I didn’t have the commits starting from July when I was changing the TestAArch64CallArranger.

1. 8289285: Use records for binding classes · openjdk/panama-foreign@37b7935 (github.com) removed the Addressable and MemorySegment parameters to the unboxAddress method.
2. 8291473: Unify MemorySegment and MemoryAddress · openjdk/panama-foreign@8b1af9a (github.com) replaced the Addressable class with MemorySegment.
3. 8275644: Replace VMReg in shuffling code with something more fine gra… · openjdk/panama-foreign@123463f (github.com) changed the AArch64Architecture.stackStorage method to accept a size in addition to an offset. The cast to short is necessary to avoid the error “incompatible types: possible lossy conversion from int to short“
4. Convert classes into records · openjdk/panama-foreign@5b63be8 (github.com) converted bindings from a class to a record so isInMemoryReturn and callingSequence now need to be a method invocations to avoid the error “isInMemoryReturn has private access in Bindings“.
5. 8292047: Consider ways to add linkage parameters to downcall handles · openjdk/panama-foreign@60a47cb (github.com) removed the asVariadic function that my tests were using and added the LinkerOption for specifying the first variadic index.

Now the interesting behavior I observe is that 3 of the tests I worked on earlier now have assertion failures that terminate the JVM: StdLibTest, TestIntrinsics, and TestVarArgs. This assertion failure was added by 8275644: Replace VMReg in shuffling code with something more fine gra… · openjdk/panama-foreign@123463f (github.com)

# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=\foreignGlobals_aarch64.cpp:181
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (d:\dev\repos\java\forks\panama-foreign\src\hotspot\cpu\aarch64\foreignGlobals_aarch64.cpp:181), pid=18972, tid=18908
#  Error: ShouldNotReachHere()
#
# JRE version: OpenJDK Runtime Environment (20.0) (slowdebug build 20-internal-adhoc.sawesong.panama-foreign)
# Java VM: OpenJDK 64-Bit Server VM (slowdebug 20-internal-adhoc.sawesong.panama-foreign, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, windows-aarch64)
# Core dump will be written. Default location: C:\dev\repos\java\forks\panama-foreign\JTwork\scratch\0\hs_err_pid18972.mdmp
#
# An error report file with more information is saved as:
# C:\dev\repos\java\forks\panama-foreign\JTwork\scratch\0\hs_err_pid18972.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#

The minimized tests I created are now out of date as well, e.g. History for test/jdk/java/foreign/TestIntrinsics.java – openjdk/panama-foreign (github.com) has 2 commits showing the changes I need to make in addition to copying the DLL from support\test\jdk\jtreg\native\lib. Suprisingly, WinDbg cannot open the executable as it did earlier. I’m launching it from C:\Program Files (x86)\Windows Kits\10\Debuggers\arm64\windbg.exe.

Perhaps it’s the wrong one for the current Windows version? Search for “debugger” in the store and install the WinDbg Preview app.

Now we can set the breakpoint in foreignGlobals_aarch64.cpp:

bp jvm!move_v128
g
u jvm!move_v128

Here is the call stack when the breakpoint is hit:

0:004> k
 # Child-SP          RetAddr               Call Site
00 00000094`33efdee0 00007fff`370226a0     jvm!move_v128+0x20 [...src\hotspot\cpu\aarch64\foreignGlobals_aarch64.cpp @ 165] 
01 00000094`33efdfd0 00007fff`36fe846c     jvm!ArgumentShuffle::pd_generate+0x1a8 [...src\hotspot\cpu\aarch64\foreignGlobals_aarch64.cpp @ 200] 
02 00000094`33efe070 00007fff`36fe763c     jvm!ArgumentShuffle::generate+0x34 [...src\hotspot\share\prims\foreignGlobals.hpp @ 114] 
03 00000094`33efe0a0 00007fff`36fe7070     jvm!DowncallStubGenerator::generate+0x4e4 [...src\hotspot\cpu\aarch64\downcallLinker_aarch64.cpp @ 203] 
04 00000094`33efe920 00007fff`37566a04     jvm!DowncallLinker::make_downcall_stub+0x88 [...src\hotspot\cpu\aarch64\downcallLinker_aarch64.cpp @ 103] 
05 00000094`33efecc0 00000268`8fe255ec     jvm!NEP_makeDowncallStub+0x33c [...src\hotspot\share\prims\nativeEntryPoint.cpp @ 77] 
06 00000094`33efefd0 00000000`00000000     0x00000268`8fe255ec

The way the macro assembler is invoked to generate the vector-to-general purpose move was changed by 8275644: Replace VMReg in shuffling code with something more fine gra… · openjdk/panama-foreign@123463f (github.com).

1. Clean up & validate callarranger tests
2. clean up callarranger api
3. Create test showing broken VaList
4. Combine VaList implementations
5. Why isn’t using fmovd only failing for some test using a floating point argument?
6. Are my macroAssembler instructions really necessary?
7. Where is a test showing these instructions in use? MinimizedTestIntrinsics (run above)

Building on macOS

A newer boot JDK is required once again as explained by the error message when running bash configure. Download and install the macOS .pkg installer for JDK 19 from the adoptium site.

checking for java... /usr/bin/java
configure: Found potential Boot JDK using java(c) in PATH
configure: Potential Boot JDK found at /usr is incorrect JDK version (openjdk version "17.0.1" 2021-10-19 LTS OpenJDK Runtime Environment Microsoft-28056 (build 17.0.1+12-LTS) OpenJDK 64-Bit Server VM Microsoft-28056 (build 17.0.1+12-LTS, mixed mode)); ignoring
configure: (Your Boot JDK version must be one of: 19 20)

Testing 4-Float HFAs

I was reviewing the tests I added and realized that I wasn’t testing the variadic HFAs. Sure enough, I couldn’t get the tests for variadic HFA structs with 4 floats to pass. My code was assigning 2 64-bit general purpose registers to such a struct. Why isn’t this caught by one of the existing tests? TestVarArgs appears to simply pass the struct to the native code in the downcall and the native code passes it back in the upcall. Shouldn’t there be additional validation? testFloatStruct in VaListTest also looks like it should catch this. Is the problem that it only uses structs on the stack? Disassemble libVaList to find out:

cd build\windows-aarch64-server-slowdebug\support\test\jdk\jtreg\native\support\libVaList\
dumpbin /disasm /out:libVaList.asm libVaList.obj
dumpbin /all /out:libVaList.txt libVaList.obj

TODO: discuss sumDoubles.asm and sumFloats.asm.

I also tried taking a look at how this code runs using WinDbg. These are the arguments I provided to WinDbg on my system:

1. Executable: C:\dev\java\abi\devbranch30\jdk\bin\java.exe
2. Arguments: -jar C:\dev\java\jtreg\lib\jtreg.jar -agentvm -timeoutFactor:4 -concurrency:4 -verbose:fail,error,summary -nativepath:C:\dev\java\abi\devbranch30\support\test\jdk\jtreg\native\lib test/jdk/java/foreign/valist/VaListTest.java
3. Start directory: C:\dev\repos\java\forks\panama-foreign

When the debugger was done loading, I ran these commands to set a breakpoint in the native code invoked by VaListTest. Unfortunately, the breakpoint was not hit. Why this happens is still a mystery.

bp VaList!sumFloatStruct
g

Adding the HFA Field Values

The function descriptor for the downcall to the native sum_struct_hfa_floats function is created by calling FunctionDescriptor.of with C_FLOAT as the first argument. This allows the result of the invokeWithArguments method of the downcall’s MethodHandle to be cast to a float. Using C_INT, for example, results in this error: ClassCastException: java.lang.Integer cannot be cast to class java.lang.Float.

Validating the HFA Field Values

Although the existing varargs tests passed, they looked like they checked round-tripping of a single value. Adding the components of the HFA seemed like a better idea because it verified that all the values were delivered correctly. This caught a bug in my implementation – when there aren’t enough registers for a HFA being passed to a variadic function, the struct was partially loaded into the available registers and then the rest of the struct was spilled onto the stack. This behavior differs from the macOS & Linux environments and wasn’t caught by any of the existing tests.

In the process of testing these changes, I deployed the locally built JDK to the Surface Pro X and got this cryptic error message:

C:\dev\java\abi\devbranch35\jdk\bin\java.exe --enable-preview SumVariadicStructHfa
WARNING: A restricted method in java.lang.foreign.Linker has been called
WARNING: java.lang.foreign.Linker::nativeLinker has been called by the unnamed module
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module

Exception in thread "main" java.lang.UnsatisfiedLinkError: C:\dev\repos\scratchpad\compilers\tests\aarch64\abi\varargs\VarArgs.dll: Can't load ARM 64-bit .dll on a AMD 64-bit platform
        at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
        at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:331)
        at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:197)
        at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:139)
        at java.base/jdk.internal.loader.NativeLibraries.findFromPaths(NativeLibraries.java:259)
        at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:251)
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2437)
        at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:873)
        at java.base/java.lang.System.loadLibrary(System.java:2047)
        at SumVariadicStructHfa.<clinit>(SumVariadicStructHfa.java:61)

Turns out I deployed x64 binaries to the Surface Pro X and launched Java in a folder containing the prior ARM64 varargs test DLL. The solution was to delete that DLL and copy the DLL from the new build. The test passed successfully and it’s only then that I realized that x64 binaries run successfully on this ARM64 platform. Getting the correct ARM64 binaries in place without replacing the x64 varargs will give a similar error Exception in thread "main" java.lang.UnsatisfiedLinkError: C:\dev\repos\scratchpad\compilers\tests\aarch64\abi\varargs\VarArgs.dll: Can't load AMD 64-bit .dll on a ARM 64-bit platform.

Outstanding Questions

1. Why invoke and instead of invokeExact in the tests?
2. What happens if we return the method handle without the .asSpreader call?
3. Why do we need to shuffle the PrintfArgs?
4. Remove dead code
5. Show how to debug (VS/VS Code) into the native code (on Windows x64 first, then ARM64).
6. Generate logs showing the wrong downcall registers in use without my changes
7. Generate logs showing the wrong upcall registers in use without my changes
8. Make foreign+upcalls log the upcall stub details as is done for the downcall stubs.
9. Why does using r10 as the retBufAddrStorage field work on Windows? Is there not test for returning a struct?
10. Create test that returns a 16-byte result and verify that it is in x1:x0 (no tests failed with this change).
11. Create test that returns result in address stored in x8 – see Return Values: For types greater than 16 bytes, the caller shall reserve a block of memory of sufficient size and alignment to hold the result. The address of the memory block shall be passed as an additional argument to the function in x8. The callee may modify the result memory block at any point during the execution of the subroutine. The callee isn’t required to preserve the value stored in x8. How does this compare to the comments in assembler_aarch64.hpp, downcallLinker_aarch64.cpp, stubGenerator_aarch64.cpp?
12. Create test that uses r16-r17 and v24 and verify that they really are volatile.
13. Fix d24 not being a volatile register
14. Why doesn’t any test fail without the cursor update in MacOsAArch64VaList.Builder.read?