—Categories: Microfabrication

Semiconductor Substrates – Intro to Materials Science

An important concept when working with materials is how to represent their properties. Phase diagrams are often used for this. I have found watching lectures to be an easier way of getting into a field as new (to me) as microfabrication. This Intro to Phase Diagrams {Texas A&M: Intro to Materials} video, for example, was an easier introduction to the topic than the notes in the microfabrication book I was reading.

This video was also my first introduction to the types of issues studied in the materials science space. The next topic in the microfabrication book I was reading was crystallography. I wanted to get an overview of the area before delving into the microfabrication aspect of crystallography. A YouTube search led me to this Lecture – Intro to Crystallography from my alma mater (interestingly, from the materials science department again)!

After watching these videos, I did a quick search for material science in the Amazon books section, hoping to see the types of topics people study in this field. Materials Science and Engineering: An Introduction looks like a great candidate (cost aside)! Looks like this is an area folks in semiconductor manufacturing need to have a handle on… More to come on crystallography as it pertains to semiconductors.

—Categories: Graphics

OpenGL Programming Gotchas

It has been a while since I wrote graphics/rendering code. The bugs are very different from those I typically write/fix since so I thought I might as well share the types of issues I dealt with. Here are some of the pitfalls I encountered:

  1. Not checking all OpenGL API results. Continuing execution when vertex/fragment shader compilation failed, for example, wastes a ton of time debugging downstream failures.
  2. Assuming successful shader compilation means that you can assign values to every shader uniform you declared! If the uniform is not actually used to generate the shader’s output, the compiler (which I learned lives in the graphics driver) can eliminate the uniform, thereby causing attempts to set it to fail.
  3. Using glVertexAttribPointer instead of glVertexAttribIPointer to pass integer IDs needed by a fragment shader. Wasted so much time on this because I was feeling schedule pressure and didn’t carefully read the documentation. Since I set the normalized parameter to GL_FALSE, the IDs were being converted into floats directly without normalization. TODO: study this behavior now to see exactly which floats ended up in the shader.
  4. Passing a count of 0 to glDrawArrays. The count argument specifies the number of indices to be rendered. Took me a while to figure out why nothing was showing up after some refactoring that I did. Turns out the number of vertices in the class I created was 0. An assertion here would have saved a ton of time.
  5. Mismatched vertex attribute formats. Spiky rendered output instead of a shape like a cube/sphere makes this one rather easy to detect. In my case, I was using 2 structs and one had an extra ID field that ended up being vertex data when the other type of struct was passed to the rendering code.
  6. Passing GL_TEXTURE0 to a sampler2D shader instead of 0! This was a typo that I didn’t catch in course slides.
  7. Mixing up sizedInternalFormat and internalFormat values in calls to glTextureStorage2D and glTextureSubImage2D.

Buffer Texture Issues

Shader Compilation Errors

When storing vertex shader data into a buffer texture, the OpenGL APIs were used to create and bind the buffer: glCreateBuffers, glNamedBufferStorage, glCreateTextures, glTextureBuffer, glBindImageTexture. A snippet of the vertex shader code to write into the buffer is shown below. Unfortunately, the shader compilation failed with this error: 0(69) : error C1115: unable to find compatible overloaded function “imageStore(struct image1D4x32_bindless, int, struct DebugData)”.

layout(binding=0, rgba32f) uniform image1D bufferTexture;
...
struct DebugData
{
    int vertexId;
    vec4 vPosition;
    vec4 modelViewVertexPosition;
    vec4 glPosition;
    vec3 vNormal;
    vec3 transformedNormal;
};
...
DebugData debugData;
debugData.vertexId = faceId;
debugData.vPosition = position;
debugData.modelViewVertexPosition = vEyeSpacePosition;
debugData.glPosition = gl_Position;
debugData.vNormal = vNormal;
debugData.transformedNormal = vEyeSpaceNormal;

imageStore(bufferTexture, gl_VertexID, debugData);

I needed up having to define a const int numVec4sPerStruct = 6; and call imageStore for each member of the struct, e.g. imageStore(bufferTexture, gl_VertexID * numVec4sPerStruct + 1, debugData.vPosition);. See related discussions here and here.

Invalid Shader Data in Buffers

There were all 0s in the output written into the texture/shader storage buffers. I tried using imageBuffer instead of image1D to avoid getting all zeros when reading the texture image buffer. The code looked correct but I couldn’t explain why zeros were being read back despite the rendered output looking correct. To figure out why this could be happening, I initialized the texture memory to a known value (integer value -1, which turns out to be a NaN when interpreted as a float). This made it easier to explain the random crap that was being displayed (hint from stack overflow) since it made it obvious that many memory locations were not being written to. Here is a snippet of the fragment shader:

struct FragmentData
{
    uint uniqueFragmentId;
    vec2 texCoords;
    vec4 glFragCoord;
    uint faceId;
};

layout(std430, binding=1) buffer DebugFragmentData
{
    FragmentData fragmentData[];
} outputFragmentData;

The fragment shader’s main method wrote to the shader storage like this:

outputFragmentData.fragmentData[uniqueFragmentId].glFragCoord = gl_FragCoord;

Initializing the storage to all 0xFF bytes made it possible to conclude that the wrong data was being written to the host, more specifically that the wrong locations were being written to! Who knew structs and alignment were a thing (TODO: link to alignment discussion in GLSL spec)! The host needed to interpret the shader storage using this struct:

struct FragmentData
{
    unsigned int uniqueFragmentId;
    unsigned int fourBytesForAlignment1;
    TextureCoord2f texCoords;
    VertexCoord4f glFragCoord;
    unsigned int faceId;
    unsigned int fourBytesForAlignment2;
    unsigned int fourBytesForAlignment3;
    unsigned int fourBytesForAlignment4;
};

Also see discussion in GLSL spec about std430 vs std140 for shader block storage!

C++ Bugs

Some of the bugs I introduced were also plain old C++ bugs (not OpenGL specific), e.g.

  1. Uninitialized variables (the fovy float had a NaN value).
  2. Copy/pasting code and missing a key fact that both Gouraud and Phong shading code paths were calling the Gouraud shader (even though the scene state output in the console showed the state had been correctly updated to Phong). That’s what you get for copy pasting and not having tests…
  3. Wrong array indexing logic. In the bug below (commit 3cfe07aea6914a91), I was multiplying the indices by elementSize but that is wrong because lines 2-5 from the bottom already have a built in multiplication by the element size. Noticed this from the disassembly.
int VolumeDataset3D::GetIndexFromSliceAndCol(uint32_t slice, uint32_t column)
{
    const int sizeOf1Row = width * sizeof(uint8_t);

    int index = slice * sizeOf1Row + column;

    return index;
}

const int elementSize = sizeof(VertexDataPositionedByte);

    for (uint32_t slice = 0; slice < depth - 1; slice++)
    {
        for (uint32_t col = 0; col < width - 1; col++)
        {
            int index = elementSize * GetIndexFromSliceAndCol(slice, col);
            int rightIndex = elementSize * GetIndexFromSliceAndCol(slice, col + 1);
            int diagIndex = elementSize * GetIndexFromSliceAndCol(slice + 1, col + 1);
            int bottomIndex = elementSize * GetIndexFromSliceAndCol(slice + 1, col);

            VertexDataPositionedByte cellData[4] =
            {
                ((VertexDataPositionedByte*)vertexData2D.data)[bottomIndex],
                ((VertexDataPositionedByte*)vertexData2D.data)[diagIndex],
                ((VertexDataPositionedByte*)vertexData2D.data)[rightIndex],
                ((VertexDataPositionedByte*)vertexData2D.data)[index],
            };

Here is the corresponding disassembly:

            int bottomIndex = elementSize * GetIndexFromSliceAndCol(slice + 1, col);
00007FF7A79AE5FC  mov         eax,dword ptr [rbp+0C4h]  
00007FF7A79AE602  inc         eax  
00007FF7A79AE604  mov         r8d,dword ptr [rbp+0E4h]  
00007FF7A79AE60B  mov         edx,eax  
00007FF7A79AE60D  mov         rcx,qword ptr [this]  
00007FF7A79AE614  call        VolumeDataset3D::GetIndexFromSliceAndCol (07FF7A7990046h)  
00007FF7A79AE619  imul        eax,eax,10h  
00007FF7A79AE61C  mov         dword ptr [rbp+164h],eax  

            VertexDataPositionedByte cellData[4] =
            {
                ((VertexDataPositionedByte*)vertexData2D.data)[bottomIndex],
00007FF7A79AE622  movsxd      rax,dword ptr [rbp+164h]  
00007FF7A79AE629  imul        rax,rax,10h  
00007FF7A79AE62D  mov         rcx,qword ptr [this]  
00007FF7A79AE634  mov         rcx,qword ptr [rcx+0E8h]  
00007FF7A79AE63B  lea         rdx,[rbp+190h]  
00007FF7A79AE642  mov         rdi,rdx  
00007FF7A79AE645  lea         rsi,[rcx+rax]  
00007FF7A79AE649  mov         ecx,10h  
00007FF7A79AE64E  rep movs    byte ptr [rdi],byte ptr [rsi]  
                ((VertexDataPositionedByte*)vertexData2D.data)[diagIndex],
00007FF7A79AE650  movsxd      rax,dword ptr [rbp+144h]  
00007FF7A79AE657  imul        rax,rax,10h  
00007FF7A79AE65B  mov         rcx,qword ptr [this]  
00007FF7A79AE662  mov         rcx,qword ptr [rcx+0E8h]  
00007FF7A79AE669  lea         rdx,[rbp+1A0h]  
00007FF7A79AE670  mov         rdi,rdx  
00007FF7A79AE673  lea         rsi,[rcx+rax]  
00007FF7A79AE677  mov         ecx,10h  
00007FF7A79AE67C  rep movs    byte ptr [rdi],byte ptr [rsi]

Crashes

I ran into (currently unexplained) crashes in both the Intel and nVidia OpenGL drivers (I used Windows only). There were also crashes (on a specific commit) from a nullref/access violation on my HP desktop but not on my Surface Pro. Found out later that the desktop was actually right to crash but the difference in behavior was certainly troubling.

Debugging Resources

—Categories: OpenJDK

Backporting Async Logging to JDK11

Background

Longer than expected pauses were observed during GC in JDK 7 as explained on the Buffered Logging hotspot-dev mailing list:

Some folks noticed much longer than expected
pauses that seemed to coincide with GC logging in the midst of a GC
safepoint. In that setup, the GC logs were going to a disk file (these were
often useful for post-mortem analyses) rather than to a RAM-based tmpfs
which had been the original design center assumption. The vicissitudes of
the dirty page flushing policy in Linux when
IO load on the machine (not necessarily the JVM process doing the logging)
could affect the length and duration of these inline logging stalls.

A buffered logging scheme was then implemented by us (and independently by
others) which we have used successfully to date to avoid these pauses in
high i/o
multi-tenant environments.

[JDK-8229517] Support for optional asynchronous/buffered logging was filed for introducing that implementation to the public upstream OpenJDK. The release notes for the asynchronous logging feature describe it as a way to avoid undesirable delays in a thread using unified logging.

Note that Unified JVM Logging was introduced in JDK 9 whereas asynchronous logging was introduced in JDK17 in PR 3135. As per the Java docs, “logging messages are output synchronously” by default whereas in “asynchronous logging mode, log sites enqueue all logging messages to an intermediate buffer and a standalone thread is responsible for flushing them to the corresponding outputs.” The AWS Developer Tools Blog has an excellent writeup about how and why they implemented this feature as well as an overview of unified logging (e.g. run java -Xlog:'gc*=info:stdout' to see logging output from log_info_p, which in my case includes output from the G1InitLogger).

Starting the Backport

This is a relatively straightforward backport. Clone the jdk11u-dev repo (or your fork as appropriate). The repo was at commit 86d39a69 when I started the backport.

git clone https://github.com/openjdk/jdk11u-dev
cd jdk11u-dev/

To see the exact same outcomes, switch to that commit (if desired).

git checkout 86d39a69

To backport this feature to JDK11, cherry-pick the commit from PR 3135 onto a new branch. We need to add the upstream as a remote to enable cherry-picking PR commits. 

git checkout -b AsyncLogging
git remote add upstream-jdk https://github.com/openjdk/jdk
git fetch upstream-jdk
git cherry-pick 41185d38f21e448370433f7e4f1633777cab6170

Conflict Resolution

I used Visual Studio for conflict resolution with this strategy:

  1. Take Incoming (Source)
  2. Inspect the diff using Compare with Unmodified… to ensure that the changes being pulled are sensible.

The rest of this section can be skipped. I am including the details of the validation of the conflict resolution strategy (i.e. ensuring nothing undesirable is getting pulled in). The advantage of the strategy outlined above is that changes that are required by the code we want to backport are most likely going to be present after conflict resolution.

Conflict Resolution: logTagSet.cpp

As an example, the upstream PR introduced 1 new method and 1 extern size_t to logTagSet.hpp. After conflict resolution, the updated logTagSet.hpp contains improvements to the logging code such as

None of these changes would be present if only the changes from the PR 3135 commit were used. These lists are generated from the blame view are therefore likely omit any delete-only diffs.

Conflict Resolution: logConfiguration.cpp

This is the list of unrelated changes (i.e. changes not in commit from PR 3135) after taking the incoming changes to logConfiguration.cpp includes (potentially partial) changes from:

Conflict Resolution: logDecorators.hpp

Conflict Resolution: logFileOutput.hpp

Only the Copyright year conflicts. Other changes brought in include:

Conflict Resolution: logOutputList.hpp

Conflict Resolution: globals.hpp

Comparing the current and incoming globals.hpp reveals a significant rewriting of this file between the jdk and jdk11u-dev repos. To resolve the conflict, copy only the change from the PR 3135 commit to the target (local) globals.hpp by selecting the checkmark next to the conflict in the Visual Studio merge editor then manually fix up the last line.

Conflict Resolution: init.hpp

jdk and jdk11u-dev also have non-trivial changes to init.hpp so the Merge… command is necessary here.

Conflict Resolution: thread.cpp

The Merge… command is again necessary here due to the significant number of changes between the source and target versions. Take the single line from the source and accept the merge:

Conflict Resolution: hashtable.hpp

Use the Merge… command once more to resolve the changes between the source and target versions. Take the single line from the source and accept the merge:

Addressing Build Errors

Now that all conflicts have been resolved, build the code before committing anything. Here are additional issues that need to be resolved.

Missing ‘runtime/nonJavaThread.hpp’

D:\dev\repos\java\forks\jdk11u-dev\src\hotspot\share\logging/logAsyncWriter.hpp(31): fatal error C1083: Cannot open include file: 'runtime/nonJavaThread.hpp': No such file or directory

nonJavaThread.hpp is a file now in the upstream JDK repo. Blame shows that PR 2390 moved it out of thread.hpp. Fix:

-#include "runtime/nonJavaThread.hpp"
+#include "runtime/thread.hpp"

Missing ‘;’ before ‘<‘

D:\dev\repos\java\forks\jdk11u-dev\src\hotspot\share\logging/logAsyncWriter.hpp(111): error C2143: syntax error: missing ';' before '<'
D:\dev\repos\java\forks\jdk11u-dev\src\hotspot\share\logging/logAsyncWriter.hpp(111): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
D:\dev\repos\java\forks\jdk11u-dev\src\hotspot\share\logging/logAsyncWriter.hpp(144): error C3646: '_stats': unknown override specifier
D:\dev\repos\java\forks\jdk11u-dev\src\hotspot\share\logging/logAsyncWriter.hpp(144): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int

Line 111 contains:

typedef KVHashtable<LogFileOutput*, uint32_t, mtLogging> AsyncLogMap;

Line 144 contains:

AsyncLogMap _stats; // statistics for dropped messages

Turns out KVHashtable was removed after async logging support was added so the latest sources aren’t the place to go for details about this class. Instead, see the KVHashtable implementation in the parent commit before it was removed. KVHashtable “is a subclass of BasicHashtable that allows you to do a simple K -> V mapping without using tons of boilerplate code.” The blame view of hashtable.hpp in the async logging support commit reveals that KVHashtable was added in commit 6d269930fdd3. For our purposes, we need to use the KVHashtable implementation that was in use when async logging was added.

Fix: insert lines 223-310 of hashtable.cpp into the local jdk11u-dev hashtable.hpp.

Missing pre_run Method

D:\dev\repos\java\forks\jdk11u-dev\src\hotspot\share\logging/logAsyncWriter.hpp(155): error C3668: 'AsyncLogWriter::pre_run': method with override specifier 'override' did not override any base class methods
D:\dev\repos\java\forks\jdk11u-dev\src\hotspot\share\logging/logAsyncWriter.hpp(156): error C2039: 'pre_run': is not a member of 'NonJavaThread'

Notice that nonJavaThread.hpp in the upstream JDK repo has a pre_run method, unlike the NonJavaThread class in jdk11u-dev. The blame view of PR 2390’s parent commit reveals that these methods were added in commit 526f854c.

Fix: Remove the pre_run method from logAsyncWritter.hpp.

Stream Errors

./src/hotspot/share/logging/logAsyncWriter.cpp(108): error C2660: 'stringStream::as_string': function does not take 1 arguments
D:\dev\repos\java\jdk11u-dev\src\hotspot\share\utilities/ostream.hpp(220): note: see declaration of 'stringStream::as_string'
./src/hotspot/share/logging/logAsyncWriter.cpp(108): error C2661: 'AsyncLogMessage::AsyncLogMessage': no overloaded function takes 2 arguments

The as_string method only has a boolean parameter in the jdk repo (added in JDK15).

Fix: Remove the parameter to as_string.

Conversion loses qualifiers

./src/hotspot/share/logging/logAsyncWriter.cpp(143): error C2440: 'initializing': cannot convert from 'const E *' to 'AsyncLogMessage *'
        with
        [
            E=AsyncLogMessage
        ]
./src/hotspot/share/logging/logAsyncWriter.cpp(143): note: Conversion loses qualifiers

Line 143 contains:

AsyncLogMessage* e = it.next();

This works in the original async logging implementation because jdk/src/hotspot/share/utilities/linkedlist.hpp was updated by 8239066: make LinkedList<T> more generic (a next() method that returns an E* was added).

Fix: git cherry-pick b08595d8443bbfb141685dc5eda7c58a34738048 and resolve the conflict (year on copyright line) using Take Incoming (Source).

Unknown class AutoModifyRestore

./test/hotspot/gtest/logging/test_asynclog.cpp(205): error C2065: 'AutoModifyRestore': undeclared identifier
./test/hotspot/gtest/logging/test_asynclog.cpp(205): error C2275: 'size_t': illegal use of this type as an expression
./build/windows-x86_64-normal-server-release/hotspot/variant-server/libjvm/gtest/objs/BUILD_GTEST_LIBJVM_pch.cpp: note: see declaration of 'size_t'
./test/hotspot/gtest/logging/test_asynclog.cpp(205): error C3861: 'saver': identifier not found

Line 205 contains:

AutoModifyRestore<size_t> saver(AsyncLogBufferSize, sz * 1024 /*in byte*/);

AutoModifyRestore was introduced to fix JDK-8245226.

Fix:

cd src/hotspot/share/utilities/
curl -Lo autoRestore.hpp https://raw.githubusercontent.com/openjdk/jdk/195c45a0e11207e15c277e7671b2a82b8077c5fb/src/hotspot/share/utilities/autoRestore.hpp
# Now include autoRestore.hpp in test_asynclog.cpp

Atomic Errors

./src/hotspot/share/logging/logAsyncWriter.cpp(172): error C2039: 'release_store_fence': is not a member of 'Atomic'
D:\dev\repos\java\jdk11u-dev\src\hotspot\share\runtime/atomic.hpp(51): note: see declaration of 'Atomic'
./src/hotspot/share/logging/logAsyncWriter.cpp(172): error C3861: 'release_store_fence': identifier not found

This method was added to atomic.hpp by OrderAccess. Notice that it appears to have been moved from orderAccess.hpp.

Fix:

-Atomic::release_store_fence(&AsyncLogWriter::_instance, self);
+OrderAccess::release_store_fence(&AsyncLogWriter::_instance, self);

‘disable_outputs’ Identifier not Found

./src/hotspot/share/logging/logConfiguration.cpp(114): error C3861: 'disable_outputs': identifier not found
./src/hotspot/share/logging/logConfiguration.cpp(278): error C2039: 'disable_outputs': is not a member of 'LogConfiguration'
D:\dev\repos\java\forks\jdk11u-dev\src\hotspot\share\logging/logConfiguration.hpp(39): note: see declaration of 'LogConfiguration'
./src/hotspot/share/logging/logConfiguration.cpp(279): error C2065: '_n_outputs': undeclared identifier
./src/hotspot/share/logging/logConfiguration.cpp(293): error C2065: '_outputs': undeclared identifier
./src/hotspot/share/logging/logConfiguration.cpp(296): error C3861: 'delete_output': identifier not found
./src/hotspot/share/logging/logConfiguration.cpp(298): error C2248: 'LogOutput::set_config_string': cannot access protected member declared in class 'LogOutput'
D:\dev\repos\java\forks\jdk11u-dev\src\hotspot\share\logging/logOutput.hpp(63): note: see declaration of 'LogOutput::set_config_string'
D:\dev\repos\java\forks\jdk11u-dev\src\hotspot\share\logging/logConfiguration.hpp(31): note: see declaration of 'LogOutput'

Line 114 is simple the method call disable_outputs(); Since that method body is present in the file, it must be missing in the header file. The correct logConfiguration.hpp shows that 8255756: Disabling logging does unnecessary work is necessary. (This error might have been visible earlier in the process!)

Fix:

git cherry-pick e66fd6f0aa43356ab4b4361d6d332e5e3bcabeb6

# Resolve straightforward conflicts.

git cherry-pick --continue

Undeclared Identifier

./src/hotspot/share/runtime/thread.cpp(4694): error C2065: 'cl': undeclared identifier

Line 4706 contains:

cl.do_thread(AsyncLogWriter::instance());

The declaration of cl is missing. Blame says it was introduced by commit 06e47d05 of [JDK-8246622] Remove CollectedHeap::print_gc_threads_on() – Java Bug System. Simply paste that PrintClosure class definition into thread.cpp (after line 4654) and the cl declaration PrintOnClosure cl(st); on (now) line 4714.

Building on macOS

Once the build succeeds on Windows, validate the changes by building on macOS.

Undeclared identifier ‘primitive_hash’ 

/Users/saint/repos/java/forks/jdk11u-dev/src/hotspot/share/utilities/hashtable.hpp:326:36: error: use of undeclared identifier 'primitive_hash'
    unsigned (*HASH)  (K const&) = primitive_hash<K>,
                                   ^
/Users/saint/repos/java/forks/jdk11u-dev/src/hotspot/share/utilities/hashtable.hpp:327:46: error: use of undeclared identifier 'primitive_equals'
    bool     (*EQUALS)(K const&, K const&) = primitive_equals<K>

Fix:

diff --git a/src/hotspot/share/utilities/hashtable.hpp b/src/hotspot/share/utilities/hashtable.hpp
index 30483b2f36..5e4c414490 100644
--- a/src/hotspot/share/utilities/hashtable.hpp
+++ b/src/hotspot/share/utilities/hashtable.hpp
@@ -30,6 +30,7 @@
 #include "oops/oop.hpp"
 #include "oops/symbol.hpp"
 #include "runtime/handles.hpp"
+#include "utilities/resourceHash.hpp"
 
 // This is a generic hashtable, designed to be used for the symbol
 // and string tables.

Default Member Initializer is a C++11 Extension

/Users/saint/repos/java/forks/jdk11u-dev/src/hotspot/share/logging/logAsyncWriter.hpp:149:33: error: default member initializer for non-static data member is a C++11 extension [-Werror,-Wc++11-extensions]
  const size_t _buffer_max_size = {AsyncLogBufferSize / (sizeof(AsyncLogMessage) + vwrite_buffer_size)};
                                ^

Fix:

diff --git a/src/hotspot/share/logging/logAsyncWriter.cpp b/src/hotspot/share/logging/logAsyncWriter.cpp
index 0231be78a9..d9f9ddda5b 100644
--- a/src/hotspot/share/logging/logAsyncWriter.cpp
+++ b/src/hotspot/share/logging/logAsyncWriter.cpp
@@ -82,7 +82,8 @@ void AsyncLogWriter::enqueue(LogFileOutput& output, LogMessageBuffer::Iterator m
 
 AsyncLogWriter::AsyncLogWriter()
   : _initialized(false),
-    _stats(17 /*table_size*/) {
+    _stats(17 /*table_size*/),
+    _buffer_max_size(AsyncLogBufferSize / (sizeof(AsyncLogMessage) + vwrite_buffer_size)) {
   if (os::create_thread(this, os::asynclog_thread)) {
     _initialized = true;
   } else {
diff --git a/src/hotspot/share/logging/logAsyncWriter.hpp b/src/hotspot/share/logging/logAsyncWriter.hpp
index 313dd6de06..c4e28e5676 100644
--- a/src/hotspot/share/logging/logAsyncWriter.hpp
+++ b/src/hotspot/share/logging/logAsyncWriter.hpp
@@ -146,7 +146,7 @@ class AsyncLogWriter : public NonJavaThread {
 
   // The memory use of each AsyncLogMessage (payload) consists of itself and a variable-length c-str message.
   // A regular logging message is smaller than vwrite_buffer_size, which is defined in logtagset.cpp
-  const size_t _buffer_max_size = {AsyncLogBufferSize / (sizeof(AsyncLogMessage) + vwrite_buffer_size)};
+  const size_t _buffer_max_size;
 
   AsyncLogWriter();
   void enqueue_locked(const AsyncLogMessage& msg);

‘override’ keyword is a C++11 extension

/Users/saint/repos/java/forks/jdk11u-dev/src/hotspot/share/logging/logAsyncWriter.hpp:154:14: error: 'override' keyword is a C++11 extension [-Werror,-Wc++11-extensions]
  void run() override;
             ^
...

Fix: Remove the override keywords

diff --git a/src/hotspot/share/logging/logAsyncWriter.hpp b/src/hotspot/share/logging/logAsyncWriter.hpp
index 313dd6de06..e6ac8aab4a 100644
--- a/src/hotspot/share/logging/logAsyncWriter.hpp
+++ b/src/hotspot/share/logging/logAsyncWriter.hpp
@@ -151,10 +151,10 @@ class AsyncLogWriter : public NonJavaThread {
   AsyncLogWriter();
   void enqueue_locked(const AsyncLogMessage& msg);
   void write();
-  void run() override;
-  char* name() const override { return (char*)"AsyncLog Thread"; }
-  bool is_Named_thread() const override { return true; }
-  void print_on(outputStream* st) const override {
+  void run();
+  char* name() const { return (char*)"AsyncLog Thread"; }
+  bool is_Named_thread() const { return true; }
+  void print_on(outputStream* st) const {
     st->print("\"%s\" ", name());
     Thread::print_on(st);
     st->cr();

Building on Linux

Depending on the GCC version, logAsyncWriter.cpp, logFileOutput.cpp, and test_asynclog.cpp might need to define nullptr to successfully compile:

#ifdef __linux__
#define nullptr 0
#endif

Testing the Build

Windows

To test the async logging code, run this command (HelloWorld doesn’t even need to exist for a really basic test):

./build/windows-x86_64-normal-server-release/jdk/bin/java.exe -Xlog:async -Xlog:all=trace:file=all.log::filecount=0 HelloWorld

Fixing Runtime Bugs

Corrupted Output

After running the simple test above, it becomes evident from the output lgos that something is wrong:

[0.039s][info ][logging          ] The maximum entries of AsyncLogBuffer: 2319, estimated memory use: 2097152 bytes
[@ùŸôÊ ][debug][@ùŸôÊ            ] Async logging thread started.
[      ][info ][ôŸôÊ            ] TemplateTable initialization, 0.0000106 secs

Search for %.*.3.+ to find where the log decorations are done based on this output in the log file. Looks like the big difference is from 8266503: [UL] Make Decorations safely copy-able and reduce their size.

Fix:

git cherry-pick 94c6177f246fc569b416f85f1411f7fe031f7aaf
git cherry-pick 74fecc070a6462e6a2d061525b53a63de15339f9

Wrong Parameter Order

Notice that the order of the parameters passed to Atomic::cmpxchg was also changed so we need to ensure that the arguments are swapped (since they were written when the new Atomic::cmpxchg was already in place). Move the first argument into the last spot.

Resources

—Categories: Learning

Learning Resources

This post is a compilation of various resources I’ve encountered in the course of work/school/life, whether from coworkers or online searches. Might as well have a list instead of hundreds of open browser tabs.

Compiler/JVM

Globalization

Performance & Benchmarking

General Programming

Misc Resources

ARM64

Courses

Robotics

—Categories: hsdis, LLVM

hsdis LLVM backend for Windows ARM64

8253757: Add LLVM-based backend for hsdis by magicus · Pull Request #7531 makes it possible to easily use LLVM as the hsdis backend. An LLVM installation is required for this. The official LLVM builds for the Windows platform do not work for building hsdis because they do not have all the prerequisite LLVM include files. See Building LLVM for Windows ARM64 – Saint’s Log (swesonga.org) for instructions on how to build LLVM for ARM64 Windows (on an x64 Windows host). To configure OpenJDK for LLVM as an hsdis backend on Windows ARM64, use this command:

bash configure --openjdk-target=aarch64-unknown-cygwin \
 --with-hsdis=llvm \
 --with-llvm=/cygdrive/d/dev/software/llvm-aarch64/

The JDK and hsdis can then be built as usual with these commands:

make images
make build-hsdis
make install-hsdis
cp /cygdrive/d/dev/software/llvm-aarch64/bin/LLVM-C.dll build/windows-aarch64-server-slowdebug/jdk/bin/

The generated JDK can then be deployed to an ARM64 machine like the Surface Pro X. To test LLVM’s disassembly, use the -XX:CompileCommand flag on the ARM64 machine:

/java -XX:CompileCommand="print java.lang.String::checkIndex" -version

Behind the Scenes

Missing Include File that Exists?

The path given to --with-llvm needs to be a Cygwin path if building in Cygwin. Otherwise, the build-hsdis target will fail with this error: c:\...\jdk\src\utils\hsdis\llvm\hsdis-llvm.cpp(58): fatal error C1083: Cannot open include file: 'llvm-c/Disassembler.h': No such file or directory. I caught this by inspecting build\windows-aarch64-server-release\make-support\failure-logs\support_hsdis_hsdis-llvm.obj.cmdline after the build failed. This was the only include that didn’t have Cygwin paths: -IC:/dev/repos/llvm-project/build_llvm_AArch64/install_local/include

Investigating Missing Disassembly

My first disassembly attempt did not work – only abstract disassembly was displayed:

...
  # {method} {0x000002ca9940f2e8} 'checkIndex' '(II)V' in 'java/lang/String'
  # parm0:    c_rarg1   = int
  # parm1:    c_rarg2   = int
  #           [sp+0x30]  (sp of caller)
  0x000002ca87ad3940: 1f20 03d5 | e953 40d1 | 3f01 00f9 | ffc3 00d1 | fd7b 02a9 | a201 f837 | 3f00 026b | e200 0054
  0x000002ca87ad3960: fd7b 42a9 | ffc3 0091
...

I verified that hsdis-aarch64.dll was present in the JDK’s bin folder. That was the only issue I had seen before that caused this behavior so I dug around to find the code that loads the hsdis DLL. A search for the “hsdis-” DLL prefix in the sources reveals the hsdis_library_name string used in the Disassembler::dll_load method. Notice that there is a Verbose flag that can display what is happening when loading the hsdis DLL!

void* Disassembler::dll_load(char* buf, int buflen, int offset, char* ebuf, int ebuflen, outputStream* st) {
  int sz = buflen - offset;
  int written = jio_snprintf(&buf[offset], sz, "%s%s", hsdis_library_name, os::dll_file_extension());
  if (written < sz) { // written successfully, not truncated.
    if (Verbose) st->print_cr("Trying to load: %s", buf);
    return os::dll_load(buf, ebuf, ebuflen);
  } else if (Verbose) {
    st->print_cr("Try to load hsdis library failed: the length of path is beyond the OS limit");
  }
  return NULL;
}

This turns out to be a JVM flag! I try passing it to java.exe but -Verbose doesn’t do anything. Learn from HotSpot Command-Line Flags Overhaul – Design Doc – OpenJDK Wiki (java.net) that it’s a -XX: flag. Trying to use it causes JVM to complain that it is a develop-only flag.

Error: VM option 'Verbose' is develop and is available only in debug version of VM.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

The –enable-debug flag documented at Building OpenJDK (java.net) is required to generate a debug VM.

bash configure --openjdk-target=aarch64-unknown-cygwin \
 --with-hsdis=llvm \
 --with-llvm=/cygdrive/d/dev/software/llvm-aarch64/ \
 --enable-debug

Running the debug JVM with the verbose flag now displays some diagnostic information:

.\java.exe -XX:CompileCommand="print java.lang.String::checkIndex" -XX:+Verbose -XX:+PrintMiscellaneous
CompileCommand: print java/lang/String.checkIndex bool print = true


============================= C1-compiled nmethod ==============================
----------------------------------- Assembly -----------------------------------
Trying to load: C:\dev\software\jdk-aarch64\jdk\bin\server\hsdis-aarch64.dll
Trying to load: C:\dev\software\jdk-aarch64\jdk\bin\server\hsdis-aarch64.dll
Trying to load: C:\dev\software\jdk-aarch64\jdk\bin\hsdis-aarch64.dll
Trying to load: hsdis-aarch64.dll
Could not load hsdis-aarch64.dll; Can't find dependent libraries; PrintAssembly defaults to abstract disassembly.

...

The error message substring “find dependent libraries” appears only once in the hotspot source code – in os::dll_load (which is called by Disassembler::dll_load). This error is displayed because LoadLibrary returns ERROR_MOD_NOT_FOUND.

The case of the DLL that refuses to load – The Old New Thing (microsoft.com) mentions loader snaps. The loader snaps are an option for the gflags tool found in the Windows Kits folder. The docs explain that GFlags is included in the Debugging Tools for Windows 10 (WinDbg) so a search for “Debugging Tools for Windows arm64” leads to Debugging ARM64 – Windows drivers. This says to install the Windows SDK, after which I now have the gflags binary (in the x86 folder)!

C:\Program Files (x86)\Windows Kits\10\Debuggers\arm64\gflags.exe

I still wasn’t sure how to see the snaps output. Show Loader Snaps in GFlags.exe, fails to capture any output in WinDbg – Stack Overflow implies that I should be able to use WinDbg to see what is failing to load.

Turns out the loader snaps aren’t really necessary. There is some critical info in the WinDbg diagnostic output:

2698:2e5c @ 03908953 - LdrpResolveDllName - ENTER: DLL name: .\LLVM-C.dll
2698:2e5c @ 03908953 - LdrpResolveDllName - RETURN: Status: 0xc0000135
...
2698:2e5c @ 03908968 - LdrpResolveDllName - ENTER: DLL name: C:\WINDOWS\LLVM-C.dll
2698:2e5c @ 03908968 - LdrpResolveDllName - RETURN: Status: 0xc0000135
...
2698:2e5c @ 03908968 - LdrpSearchPath - RETURN: Status: 0xc0000135
2698:2e5c @ 03908968 - LdrpProcessWork - ERROR: Unable to load DLL: "LLVM-C.dll", Parent Module: "C:\dev\software\jdk-aarch64\jdk\bin\hsdis-aarch64.dll", Status: 0xc0000135

hsdis-aarch64.dll is not being loaded because LLVM-C.dll cannot be found! Still learning the need for reading the full instructions to avoid unnecessary pain.

—Categories: Hardware, Processors

Processors & Microfabrication

Having used the Macbook Pro M1 for a couple of months now, I have been thoroughly impressed with its performance. Combining this with all the news about the chip shortage, my curiosity about microchips and chip manufacturing has been piqued. Here is an interesting video I found about the M1 chip that seemed like a decent tour of the processor landscape, from how the ARM project was started, the basics of RISC vs CISC, Intel passing on the opportunity to fabricate iPhone chips, Apple becoming the biggest ARM licensee so that they can design their own chips, and so on. Highly recommended video!

What has TSMC, who is manufacturing these advanced chips, up to these days? CNBC reviews the massive investments they are making to address the chip shortage and stay competitive.

Naturally, the next question is what Intel is doing to catch up with this type of chip performance. Intel’s Process Roadmap to 2025: with 4nm, 3nm, 20A and 18A?! (anandtech.com) addresses this. CNBC also reviews the challenges facing Intel and their goals:

Intel is definitely putting their money where their mouth is!

Other Resources

I went in search of information about how chips are made. This video by Infineon is one of the more informative I’ve seen. The breakdown of the process (especially how a transistor works) is easy to understand. Great overview!

—Categories: Compilers, LLVM

Building LLVM for Windows ARM64

I was trying to test using LLVM as a backend for hsdis on the Windows ARM64 platform as implemented in PR 5920. I downloaded LLVM 13 and tried to use it in the build. Unfortunately, it didn’t have all the prerequisite include files and so building your own LLVM installation was the approach suggested for Windows. This post explicitly outlines the instructions needed to build LLVM for the Windows ARM64 platform on a Windows x64 host machine.

The first requirement is an LLVM build with native llvm-nm.exe and llvm-tblgen.exe binaries. These can be downloaded (I think) or generated by building LLVM for the native x64 platform as specified in the instructions from Jorn.

git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build_llvm
cd build_llvm
cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=X86" -D"CMAKE_BUILD_TYPE:STRING=Release" -D"CMAKE_INSTALL_PREFIX=install_local" -A x64 -T host=x64
cmake --build . --config Release --target install

Once that build successfully completes, we can then build LLVM for the Windows ARM64 platform with the commands below. Notice that we specify paths to the native llvm-nm and llvm-tblgen binaries to prevent the build from trying to use their ARM64 equivalents (which won’t run on the host).

cd llvm-project
mkdir build_llvm_AArch64
cd build_llvm_Aarch64

cmake ../llvm -DLLVM_TARGETS_TO_BUILD:STRING=AArch64 \
 -DCMAKE_BUILD_TYPE:STRING=Release \
 -DCMAKE_INSTALL_PREFIX=install_local \
 -DCMAKE_CROSSCOMPILING=True \
 -DLLVM_TARGET_ARCH=AArch64 \
 -DLLVM_NM=C:/repos/llvm-project/build_llvm/install_local/bin/llvm-nm.exe \
 -DLLVM_TABLEGEN=C:/repos/llvm-project/build_llvm/install_local/bin/llvm-tblgen.exe \
 -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-win32-msvc \
 -A ARM64 \
 -T host=x64

date; time \
 cmake --build . --config Release --target install ; \
 date

Once the build completes, the LLVM ARM64 files will be in the build_llvm_AArch64/install_local folder in the llvm-project repo. That build should have all the necessary header files and static libraries required for LLVM projects targeting Windows on ARM64. See the general cmake options and the LLVM-specific cmake options for details on the various flags and variables.

Behind the Scenes: Cross-Compiling LLVM

I naively started out by adding AArch64 to the list of LLVM_TARGETS_TO_BUILD, then using only AArch64 in the list. Trying to use the generated build would still fail with errors about mismatched platforms so I knew some cross compilation specific flags would be needed. How do I cross-compile LLVM/Clang for AArch64 on x64 host? – Stack Overflow and How To Cross-Compile Clang/LLVM using Clang/LLVM — LLVM 15.0.0git documentation were handy references. They didn’t have anything windows specific but got me walking down the right path (e.g. the importance of the native LLVM_TABLEGEN). I tried something along these lines:

cd llvm-project
mkdir build_llvm_AArch64
cd build_llvm_AArch64

cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=AArch64" \
 -D"CMAKE_BUILD_TYPE:STRING=Release" \
 -D"CMAKE_INSTALL_PREFIX=install_local_AArch64" \
 -D"CMAKE_CROSSCOMPILING=True" \
 -D"LLVM_TARGET_ARCH=AArch64" \
 -A x64 \
 -T host=x64

cmake --build . --config Release --target install

This still results in errors about conflicting machine types:

c:\...\llvm-project\build_llvm_aarch64\install_local_aarch64\\lib\llvmaarch64disassembler.lib : warning LNK4272: library machine type 'x64' conflicts with target machine type 'ARM64'

That’s when I tried adding the LLVM_TABLE_GEN from a Windows x64 LLVM build I had generated earlier. I accidentally omitted the options prefixed with a # below because I didn’t include the trailing slash after adding the llvm-tblgen.exe option.

cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=AArch64" \
 -D"CMAKE_BUILD_TYPE:STRING=Release" \
 -D"CMAKE_INSTALL_PREFIX=install_local_AArch64_2" \
 -D"CMAKE_CROSSCOMPILING=True" \
 -D"LLVM_TARGET_ARCH=AArch64" \
 -D"LLVM_TABLEGEN=C:\dev\repos\llvm-project\build_llvm\install_local\bin\llvm-tblgen.exe" \
 #-D"LLVM_DEFAULT_TARGET_TRIPLE=aarch64-win32-msvc" \
 #-A x64 \
 #-T host=x64

date; time \
 cmake --build . --config Release --target install; \
 date

The build still succeeded and generated AArch64 .lib files in the LLVM installation! Interestingly, they still had the x64 machine type in the header.

$ dumpbin /headers build_llvm_AArch64_2\install_local_AArch64_2\lib\LLVMAArch64AsmParser.lib
Microsoft (R) COFF/PE Dumper Version 14.29.30133.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file build_llvm_AArch64_2\install_local_AArch64_2\lib\LLVMAArch64AsmParser.lib

File Type: LIBRARY

FILE HEADER VALUES
            8664 machine (x64)
...

I had no choice but to reexamine my understanding of what the -A flag does. It is used to specify the platform name but it’s only after digging into the CMAKE_GENERATOR_PLATFORM docs that I noticed that this was the target platform! This also made me realize that I hadn’t noticed that the x64 C++ compiler was being used all along!

-- The C compiler identification is MSVC 19.29.30133.0
-- The CXX compiler identification is MSVC 19.29.30133.0
-- The ASM compiler identification is MSVC
-- Found assembler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - works

Some references to LLVM triples led back to the clang cross-compilation docs and the llvm::Triple source code so I tried again with the triple set and with -A now set to AArch64.

cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=AArch64" \
 -D"CMAKE_BUILD_TYPE:STRING=Release" \
 -D"CMAKE_INSTALL_PREFIX=install_local_AArch64_3" \
 -D"CMAKE_CROSSCOMPILING=True" \
 -D"LLVM_TARGET_ARCH=AArch64" \
 -D"LLVM_TABLEGEN=C:\dev\repos\llvm-project\build_llvm\install_local\bin\llvm-tblgen.exe" \
 -D"LLVM_DEFAULT_TARGET_TRIPLE=aarch64-win32-msvc" \
 -A AArch64 \
 -T host=x64

Setting -A to AArch64 causes MSBuild to fail with an error about an unknown platform. So -A just might be the argument I need to get ARM64 libraries built.

"C:\dev\repos\llvm-project\build_llvm_AArch64_3\CMakeFiles\3.17.3\VCTargetsPath.vcxproj" (default target) (1) ->
    (_CheckForInvalidConfigurationAndPlatform target) ->
      C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(820,5): error : The BaseOutputPath/OutputPath property is not set for project 'VCTargetsPath.vcxproj'.  Please check to make sure that you have specified a valid combination of Configuration and Platform for this project.  Configuration='Debug'  Platform='AArch64'.  You may be seeing this message because you are trying to build a project without a solution file, and have specified a non-default Configuration or Platform that doesn't exist for this project. [C:\dev\repos\llvm-project\build_llvm_AArch64_3\CMakeFiles\3.17.3\VCTargetsPath.vcxproj]

So I tried using -A ARM64 instead. I noticed that we now have the ARM64 C++ compiler selected! This is something I should have been paying attention to from the beginning, crucial for cross-compilation.

-- The C compiler identification is MSVC 19.29.30133.0
-- The CXX compiler identification is MSVC 19.29.30133.0
-- The ASM compiler identification is MSVC
-- Found assembler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/arm64/cl.exe - works

Unfortunately, the build still failed with an error from gen-msvc-exports.py. Taking a look at gen-msvc-exports.py, it looks like it is trying to run llvm-nm.exe (for the target platform).

  Generating export list for LLVM-C
  Traceback (most recent call last):
    File "C:/dev/repos/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 116, in <module>
      main()
    File "C:/dev/repos/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 112, in main
      gen_llvm_c_export(ns.output, ns.underscore, libs, ns.nm)
    File "C:/dev/repos/llvm-project/llvm/tools/llvm-shlib/gen-msvc-exports.py", line 72, in gen_llvm_c_export
      check_call([nm, '-g', lib], stdout=dumpout_f)
    File "C:\dev\tools\Anaconda3\lib\subprocess.py", line 359, in check_call
      retcode = call(*popenargs, **kwargs)
    File "C:\dev\tools\Anaconda3\lib\subprocess.py", line 340, in call
      with Popen(*popenargs, **kwargs) as p:
    File "C:\dev\tools\Anaconda3\lib\subprocess.py", line 854, in __init__
      self._execute_child(args, executable, preexec_fn, close_fds,
    File "C:\dev\tools\Anaconda3\lib\subprocess.py", line 1307, in _execute_child
      hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
  OSError: [WinError 216] This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(241,5): error MSB8066: Custom build for 'C:\dev\repos\llvm-project\build_llvm_AArch64_3\CMakeFi
les\02a88fa656bb9cf8b9ffd0e0debe57ae\libllvm-c.exports.rule;C:\dev\repos\llvm-project\build_llvm_AArch64_3\CMakeFiles\8ebc0efbf04134b25d0f37561fba0d55\LLVM-C.def.rule;C:\dev\repos\llvm-project\build_llvm_AArch64_
3\CMakeFiles\509fcb3f8bb132e9c560e15e8d25cb45\LLVM-C_exports.rule;C:\dev\repos\llvm-project\llvm\tools\llvm-shlib\CMakeLists.txt' exited with code 1. [C:\dev\repos\llvm-project\build_llvm_AArch64_3\tools\llvm-shl
ib\LLVM-C_exports.vcxproj]

A quick search for the general message (Generating export list for LLVM-C) reveals that it is from llvm-shlib/CMakeLists.txt. Looks like we just need to set LLVM_NM as per llvm-shlib/CMakeLists.txt.

date; time cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=AArch64" \
 -D"CMAKE_BUILD_TYPE:STRING=Release" \
 -D"CMAKE_INSTALL_PREFIX=install_local" \
 -D"CMAKE_CROSSCOMPILING=True" \
 -D"LLVM_TARGET_ARCH=AArch64" \
 -D"LLVM_NM=C:\dev\repos\llvm-project\build_llvm\install_local\bin\llvm-nm.exe" \
 -D"LLVM_TABLEGEN=C:\dev\repos\llvm-project\build_llvm\install_local\bin\llvm-tblgen.exe" \
 -D"LLVM_DEFAULT_TARGET_TRIPLE=aarch64-win32-msvc" \
 -A ARM64 \
 -T host=x64

date; time \
 cmake --build . --config Release --target install; \
 date

These build commands work! Dumpbin shows that the generated .lib files have ARM64 headers!

$ dumpbin /headers C:\dev\repos\llvm-project\build_llvm_AArch64_3\Release\lib\LLVMAArch64Disassembler.lib
Microsoft (R) COFF/PE Dumper Version 14.29.30133.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file C:\dev\repos\llvm-project\build_llvm_AArch64_3\Release\lib\LLVMAArch64Disassembler.lib

File Type: LIBRARY

FILE HEADER VALUES
            AA64 machine (ARM64)

Unfortunately, the JDK project that got me started down this path still doesn’t build. Cygwin shows defines like -DLLVM_DEFAULT_TRIPLET='”aarch64-pc-windows-msvc”‘ being passed to the compiler, which then complains:

C:/.../src/utils/hsdis/llvm/hsdis-llvm.cpp(217): error C2015: too many characters in constant

The quotes in the commands therefore needed to be dropped. This caused build failures since the paths used back-slashes!

Building Opts.inc...
  'C:devreposllvm-projectbuild_llvminstall_localbinllvm-tblgen.exe' is not recognized as an internal or external command,
  operable program or batch file

This is now the part where I find a nice document on the LLVM site with the 3-liner for this task 😀

—Categories: hsdis, OpenJDK

Troubleshooting hsdis LLVM backend MSVC Linker Errors

The post about Exploring the hsdis LLVM Support PR mentioned link errors when building hsdis using an LLVM backend on Windows (x86-64 host building JDK for the x86-64 platform). Before we look at why linking fails, we can get a simple repro for the error from the Cygwin logs. To get the command line used to invoke the linker, run make LOG=debug build-hsdis. Search the output for link.exe to find the failing command or open build\windows-x86_64-server-release\support\hsdis\BUILD_HSDIS_link.cmdline. Change the path from Cygwin to Windows style so that the command can be run in the x64 Native Tools Command Prompt.

cd C:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\

c:\progra~2\micros~3\2019\enterp~1\vc\tools\msvc\1429~1.301\bin\hostx86\x64\link.exe -nologo -libpath:c:\dev\repos\llvm-project\build_llvm\install_local\\lib -dll -debug "-pdb:c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.pdb" "-map:c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.map" "-implib:c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.lib" -libpath:c:\progra~2\micros~3\2019\enterp~1\vc\tools\msvc\1429~1.301\atlmfc\lib\x64 -libpath:c:\progra~2\micros~3\2019\enterp~1\vc\tools\msvc\1429~1.301\lib\x64 -libpath:c:\progra~2\wi3cf2~1\netfxsdk\4.8\lib\um\x64 -libpath:c:\progra~2\wi3cf2~1\10\lib\100190~1.0\ucrt\x64 -libpath:c:\progra~2\wi3cf2~1\10\lib\100190~1.0\um\x64 -out:c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.dll c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis-llvm.obj c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.dll.res

These are the resulting link errors mentioned in Exploring the hsdis LLVM Support PR.

   Creating library c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.lib and object c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.exp
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMCreateDisasm referenced in function "public: __cdecl hsdis_backend::hsdis_backend(unsigned __int64,unsigned __int64,unsigned char *,unsigned __int64,void * (__cdecl*)(void *,char const *,void *),void *,int (__cdecl*)(void *,char const *,...),void *,char const *,int)" (??0hsdis_backend@@QEAA@_K0PEAE0P6APEAXPEAXPEBD2@Z2P6AH23ZZ23H@Z)
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMSetDisasmOptions referenced in function "public: __cdecl hsdis_backend::hsdis_backend(unsigned __int64,unsigned __int64,unsigned char *,unsigned __int64,void * (__cdecl*)(void *,char const *,void *),void *,int (__cdecl*)(void *,char const *,...),void *,char const *,int)" (??0hsdis_backend@@QEAA@_K0PEAE0P6APEAXPEAXPEBD2@Z2P6AH23ZZ23H@Z)
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMDisasmDispose referenced in function "public: __cdecl hsdis_backend::~hsdis_backend(void)" (??1hsdis_backend@@QEAA@XZ)
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMDisasmInstruction referenced in function "protected: virtual unsigned __int64 __cdecl hsdis_backend::decode_instruction(unsigned __int64,unsigned __int64,unsigned __int64)" (?decode_instruction@hsdis_backend@@MEAA_K_K00@Z)
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86TargetInfo referenced in function LLVMInitializeNativeTarget
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86Target referenced in function LLVMInitializeNativeTarget
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86TargetMC referenced in function LLVMInitializeNativeTarget
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86AsmPrinter referenced in function LLVMInitializeNativeAsmPrinter
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86Disassembler referenced in function LLVMInitializeNativeDisassembler
c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.dll : fatal error LNK1120: 9 unresolved externals

The hsdis_backend class uses functions in the LLVM libraries that cannot be resolved:

The X86 specific symbols are referenced by the calls to LLVMInitializeNativeTarget, LLVMInitializeNativeAsmPrinter, and LLVMInitializeNativeDisassembler.

Tracking Down the Linker Issues

We can use the DUMPBIN tool to inspect the LLVM libraries.

cd c:\dev\repos\llvm-project\build_llvm\install_local\lib
dumpbin LLVMX86Disassembler.lib
dumpbin /symbols /out:LLVMX86Disassembler.txt LLVMX86Disassembler.lib

The forfiles command is useful for dumping the symbols from all the libraries (forfiles was suggested at How to do something to each file in a directory with a batch script). I thought forfiles would work without the “cmd /c” prefix but that only resulted in dumpbin /summary output!

cd c:\dev\repos\llvm-project\build_llvm\install_local\lib
forfiles /m *.lib /c "cmd /c dumpbin /symbols /out:@fname.txt @file"

Now we can easily search for the symbols of interest, e.g.

> findstr /sipnc:"LLVMInitializeX86Disassembler" *.txt
LLVMX86Disassembler.txt:151:090 00000000 SECT2C notype ()    External     | LLVMInitializeX86Disassembler
LLVMX86Disassembler.txt:926:397 00000000 SECT6B notype       Static       | $unwind$LLVMInitializeX86Disassembler
LLVMX86Disassembler.txt:929:39A 00000000 SECT6C notype       Static       | $pdata$LLVMInitializeX86Disassembler

So there really is no such symbol in this lib folder! I’m guessing I need to add another lib folder to the path. A quick search for LLVMInitializeX86Disassembler leads to this post on Using the LLVM MC Disassembly API. It mentions using llvm-config to set the linker flags. Shouldn’t running the bash configure command take care of this? Let’s see what’s in the configure output:

...
checking what hsdis backend to use... 'llvm'
checking for LLVM_CONFIG... C:/dev/repos/llvm-project/build_llvm/install_local/bin [user supplied]
/cygdrive/c/dev/repos/java/forks/jdk/build/.configure-support/generated-configure.sh: line 135451: C:/dev/repos/llvm-project/build_llvm/install_local/bin: Is a directory
/cygdrive/c/dev/repos/java/forks/jdk/build/.configure-support/generated-configure.sh: line 135452: C:/dev/repos/llvm-project/build_llvm/install_local/bin: Is a directory
/cygdrive/c/dev/repos/java/forks/jdk/build/.configure-support/generated-configure.sh: line 135453: C:/dev/repos/llvm-project/build_llvm/install_local/bin: Is a directory
...

Well, that could be the problem! I think I need to fix the llvm-config path in Cygwin by appending /llvm-config to LLVM_CONFIG.

bash configure --with-hsdis=llvm LLVM_CONFIG=C:/dev/repos/llvm-project/build_llvm/install_local/bin/llvm-config --with-llvm=C:/dev/repos/llvm-project/build_llvm/install_local/

Sure enough, that was the problem! The bash configure output (below) now looks good and make build-hsdis now works. The fix for this would be to ensure bash configure fails if LLVM_CONFIG is set to the directory instead of the executable!

checking what hsdis backend to use... 'llvm'
checking for LLVM_CONFIG... C:/dev/repos/llvm-project/build_llvm/install_local/bin/llvm-config [user supplied]
checking for number of cores... 8
...

$ make build-hsdis
Building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Creating support/hsdis/hsdis.dll from 1 file(s)
Finished building target 'build-hsdis' in configuration 'windows-x86_64-server-release'

Notice from the new build command line in build\windows-x86_64-server-release\support\hsdis\BUILD_HSDIS_link.cmdline that there are now many .lib files supplied to the linker! These are the lib files that I was inspecting with dumpbin so my earlier hypothesis was wrong (there were no additional .lib files required, the ones I was looking at were simply not being passed to the linker).

/cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/fixpath exec
 /cygdrive/c/progra~2/micros~3/2019/enterp~1/vc/tools/msvc/1429~1.301/bin/hostx86/x64/link.exe
 -nologo
 -libpath:/cygdrive/c/dev/repos/llvm-project/build_llvm/install_local//lib
 -dll
 -debug
 "-pdb:/cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.pdb"
 "-map:/cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.map"
 "-implib:/cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.lib"
 -libpath:/cygdrive/c/progra~2/micros~3/2019/enterp~1/vc/tools/msvc/1429~1.301/atlmfc/lib/x64
 -libpath:/cygdrive/c/progra~2/micros~3/2019/enterp~1/vc/tools/msvc/1429~1.301/lib/x64
 -libpath:/cygdrive/c/progra~2/wi3cf2~1/netfxsdk/4.8/lib/um/x64
 -libpath:/cygdrive/c/progra~2/wi3cf2~1/10/lib/100190~1.0/ucrt/x64
 -libpath:/cygdrive/c/progra~2/wi3cf2~1/10/lib/100190~1.0/um/x64
 -out:/cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.dll 
 /cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis-llvm.obj
 /cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.dll.res
 llvmx86targetmca.lib llvmmca.lib llvmx86disassembler.lib llvmx86asmparser.lib llvmx86codegen.lib llvmcfguard.lib llvmglobalisel.lib llvmx86desc.lib llvmx86info.lib llvmmcdisassembler.lib llvmselectiondag.lib llvminstrumentation.lib llvmasmprinter.lib llvmdebuginfomsf.lib llvmcodegen.lib llvmtarget.lib llvmscalaropts.lib llvminstcombine.lib llvmaggressiveinstcombine.lib llvmtransformutils.lib llvmbitwriter.lib llvmanalysis.lib llvmprofiledata.lib llvmdebuginfodwarf.lib llvmobject.lib llvmtextapi.lib llvmmcparser.lib llvmmc.lib llvmdebuginfocodeview.lib llvmbitreader.lib llvmcore.lib llvmremarks.lib llvmbitstreamreader.lib llvmbinaryformat.lib llvmsupport.lib llvmdemangle.lib

Now running make install-hsdis copies hsdis-amd64.dll into /build/windows-x86_64-server-release/jdk/bin. The LLVM hsdis backend can now be used to disassemble instructions:

$ ./java -XX:CompileCommand="print java.lang.String::checkIndex" -version
CompileCommand: print java/lang/String.checkIndex bool print = true

============================= C2-compiled nmethod ==============================
----------------------------------- Assembly -----------------------------------

Compiled method (c2)    5912   60       4       java.lang.String::checkIndex (10 bytes)
 total in heap  [0x00000162f39e3090,0x00000162f39e3308] = 632
 relocation     [0x00000162f39e31e8,0x00000162f39e3200] = 24
 main code      [0x00000162f39e3200,0x00000162f39e3280] = 128
 stub code      [0x00000162f39e3280,0x00000162f39e3298] = 24
 oops           [0x00000162f39e3298,0x00000162f39e32a0] = 8
 metadata       [0x00000162f39e32a0,0x00000162f39e32a8] = 8
 scopes data    [0x00000162f39e32a8,0x00000162f39e32c0] = 24
 scopes pcs     [0x00000162f39e32c0,0x00000162f39e3300] = 64
 dependencies   [0x00000162f39e3300,0x00000162f39e3308] = 8

[Disassembly]
--------------------------------------------------------------------------------
[Constant Pool (empty)]

--------------------------------------------------------------------------------

[Verified Entry Point]
  # {method} {0x000001628800f2f0} 'checkIndex' '(II)V' in 'java/lang/String'
  # parm0:    rdx       = int
  # parm1:    r8        = int
  #           [sp+0x30]  (sp of caller)
  0x00000162f39e3200:           movl    %eax, -0x7000(%rsp)
  0x00000162f39e3207:           pushq   %rbp
  0x00000162f39e3208:           subq    $0x20, %rsp
  0x00000162f39e320c:           testl   %r8d, %r8d
  0x00000162f39e320f:           jl      0x2f
  0x00000162f39e3211:           cmpl    %r8d, %edx
  0x00000162f39e3214:           jae     0x16
  0x00000162f39e3216:           vzeroupper
  0x00000162f39e3219:           addq    $0x20, %rsp
  0x00000162f39e321d:           popq    %rbp
  0x00000162f39e321e:           cmpq    0x338(%r15), %rsp   ;   {poll_return}
  0x00000162f39e3225:           ja      0x29
  0x00000162f39e322b:           retq
  0x00000162f39e322c:           movl    %edx, %ebp
  0x00000162f39e322e:           movl    %r8d, (%rsp)
  0x00000162f39e3232:           movl    $0xffffffe4, %edx
  0x00000162f39e3237:           nop
  0x00000162f39e3238:           vzeroupper
  0x00000162f39e323b:           callq   -0x7a80f40          ; ImmutableOopMap {}
                                                            ;*invokestatic checkIndex {reexecute=0 rethrow=0 return_oop=0}
                                                            ; - java.lang.String::checkIndex@5 (line 4554)
                                                            ;   {runtime_call UncommonTrapBlob}
  0x00000162f39e3240:           movl    %edx, %ebp
  0x00000162f39e3242:           movl    %r8d, (%rsp)
  0x00000162f39e3246:           movl    $0xffffffcc, %edx
...

References

Here are some of the bugs/questions I looked at when investigating these failures. Stack overflow taught me about dumpbin and C++ decorated names/ the undname tool.

—Categories: Assembly, hsdis, OpenJDK

hsdis+binutils on macOS/Linux

A previous post explored how to use LLVM as the backend disassembler for hsdis. The instructions for how to use GNU binutils (the currently supported option) are straightforward. Listing them here for completeness (assuming you have cloned the OpenJDK repo into your ~/repos/java/jdk folder). Note that they depend on more recent changes. See the docs on the Java command for more info about the -XX:CompileCommand option.

# Download and extract GNU binutils 2.37
cd ~
curl -Lo binutils-2.37.tar.gz https://ftp.gnu.org/gnu/binutils/binutils-2.37.tar.gz
tar xvf binutils-2.37.tar.gz

# Configure the OpenJDK repo for hsdis
cd ~/repos/java/jdk
bash configure --with-hsdis=binutils --with-binutils-src=~/binutils-2.37

# Build hsdis
make build-hsdis

To deploy the built hsdis library on macOS:

cd build/macosx-aarch64-server-release

# Copy the hsdis library into the JDK bin folder
cp support/hsdis/libhsdis.dylib jdk/bin/hsdis-aarch64.dylib

To deploy the built hsdis library on Ubuntu Linux (open question: is this step even necessary?):

cd build/linux-x86_64-server-release

# Copy the hsdis library into the JDK bin folder
cp support/hsdis/libhsdis.so jdk/bin/

Update 2024-03-13: use the make install-hsdis command to copy the hsdis binaries into the new OpenJDK build. This will ensure that the hsdis binary is copied to lib/hsdis-adm64.so (this file name should be used in place of any others that listed by find . -name *hsdis*).

Now we can disassemble some code, e.g. the String.checkIndex method mentioned in PR 5920.

# Disassemble some code
jdk/bin/java -XX:CompileCommand="print java.lang.String::checkIndex" -version

To see how to disassemble the code for a class, we can use the basic substitution cipher class from the post on Building HSDIS in Cygwin as an example. Download, compile and disassemble it using the commands below. Note that these commands save the .java file to a temp folder to make cleanup much easier. Also note the redirection to a file since the output can be voluminous.

cd jdk/bin
mkdir -p temp
cd temp

curl -Lo BasicSubstitutionCipher.java https://raw.githubusercontent.com/swesonga/scratchpad/main/apps/crypto/substitution-cipher/BasicSubstitutionCipher.java

../javac BasicSubstitutionCipher.java

../java -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -XX:+LogCompilation BasicSubstitutionCipher > disassembled.txt

open disassembled.txt

—Categories: hsdis, OpenJDK

Exploring the hsdis LLVM Support PR

The previous post described how LLVM can be configured as the disassembly backend for hsdis. Here, I explain the process it took for me to figure out the details of the change adding support for LLVM. One of the first things to do when learning these details of this change is to build it. Since I’m using my own fork of the OpenJDK repo, I need to add the upstream repo to my remotes. This makes it possible to fetch commits from PRs submitted to the upstream repo.

cd ~/repos/forks/jdk
git remote add upstream https://github.com/openjdk/jdk
git fetch upstream

The LLVM-backend PR has only 1 commit (as of this writing). Create a new branch then cherry-pick that commit (I was on commit 77757ba9 when I wrote this.

git checkout -b hsdis-backend-llvm
git cherry-pick effac9b87ecb3cdc8d3d149b9dcd72ee1ea88fec

Some conflicts need to be resolved:

Performing inexact rename detection: 100% (88356/88356), done.
Auto-merging make/autoconf/spec.gmk.in
Auto-merging make/autoconf/jdk-options.m4
CONFLICT (content): Merge conflict in make/autoconf/jdk-options.m4
Auto-merging make/Hsdis.gmk
error: could not apply effac9b87ec... Create hsdis backend using LLVM

The files view of PR 5920 shows that the change to make/autoconf/jdk-options.m4 is mostly adding another branch to the if-else statements checking the hsdis backend. Lines 841-854 of PR 5920 can therefore be added just before the else on line 890 to resolve the conflict. The diff from my branch can be seen here.

Building the Changes on macOS ARM64

Install LLVM using homebrew (if it is not already installed). 

brew install llvm

Set the the LDFLAGS and CPPFLAGS environment variables then run printenv | grep -i flags to verify that the flags have been set correctly. Exporting CC and CXX is crucial since that is how to let bash configure know that we need a custom compiler for the build!

# export LDFLAGS="-L/opt/homebrew/opt/llvm/lib"
# export CFLAGS="-I/opt/homebrew/opt/llvm/include"
export CC=/opt/homebrew/opt/llvm/bin/clang
export CXX=$(CC)++
bash configure --with-hsdis=llvm LLVM_CONFIG=/opt/homebrew/opt/llvm/bin

Run make build-hsdis in the root folder of the jdk repo.

If the proper flags have not been set, make will fail with the error below. Run make --debug=v for additional information on what make is doing.

saint@Saints-MBP-2021 jdk % make build-hsdis
Building target 'build-hsdis' in configuration 'macosx-aarch64-server-release'
/Users/saint/repos/java/forks/jdk/src/utils/hsdis/llvm/hsdis-llvm.cpp:58:10: fatal error: 'llvm-c/Disassembler.h' file not found
#include <llvm-c/Disassembler.h>
         ^~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[3]: *** [/Users/saint/repos/java/forks/jdk/build/macosx-aarch64-server-release/support/hsdis/hsdis-llvm.o] Error 1
make[2]: *** [build-hsdis] Error 2

ERROR: Build failed for target 'build-hsdis' in configuration 'macosx-aarch64-server-release' (exit code 2)

After all that fidgeting around, the fix is as simple as updating your path to include LLVM <insert facepalm / clown>. This is what installing LLVM using brew ends with:

...
==> llvm
To use the bundled libc++ please add the following LDFLAGS:
  LDFLAGS="-L/opt/homebrew/opt/llvm/lib -Wl,-rpath,/opt/homebrew/opt/llvm/lib"

llvm is keg-only, which means it was not symlinked into /opt/homebrew,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have llvm first in your PATH, run:
  echo 'export PATH="/opt/homebrew/opt/llvm/bin:$PATH"' >> ~/.zshrc

For compilers to find llvm you may need to set:
  export LDFLAGS="-L/opt/homebrew/opt/llvm/lib"
  export CPPFLAGS="-I/opt/homebrew/opt/llvm/include"

My MacBook didn’t even have a ~/.zshrc file. Setting the PATH using the suggestion above fixed the build errors!

echo 'export PATH="/opt/homebrew/opt/llvm/bin:$PATH"' >> ~/.zshrc

Now open a new terminal and configure the repo (no need for LLVM_CONFIG).

% bash configure --with-hsdis=llvm
% make build-hsdis

Interestingly, running make images does not work on subsequent attempts?! After further investigation, it turns out that the clang compiler installed by brew cannot successfully compile the OpenJDK sources. Why does it issue warnings that Apple’s clang compiler does not?

In file included from /Users/saint/repos/java/forks/jdk/src/hotspot/cpu/aarch64/abstractInterpreter_aarch64.cpp:31:
In file included from /Users/saint/repos/java/forks/jdk/src/hotspot/share/runtime/frame.inline.hpp:42:
In file included from /Users/saint/repos/java/forks/jdk/src/hotspot/cpu/aarch64/frame_aarch64.inline.hpp:31:
In file included from /Users/saint/repos/java/forks/jdk/src/hotspot/cpu/aarch64/pauth_aarch64.hpp:28:
/Users/saint/repos/java/forks/jdk/src/hotspot/os_cpu/bsd_aarch64/pauth_bsd_aarch64.inline.hpp:29:10: fatal error: 'ptrauth.h' file not found
#include <ptrauth.h>
         ^~~~~~~~~~~
1 error generated.
make[3]: *** [/Users/saint/repos/java/forks/jdk/build/macosx-aarch64-server-release/hotspot/variant-server/libjvm/objs/abstractInterpreter_aarch64.o] Error 1
m

To work around this, first build the JDK using Apple’s clang. Next, add brew’s LLVM installation to the PATH, then configure for hsdis. Finally, build hsdis:

# Warning: ensure /opt/homebrew/opt/llvm/bin is not in the PATH
cd ~/repos/java/forks/jdk
bash configure
make images

# Now add brew's LLVM to the PATH before running bash configure
export OLDPATH=$PATH
export PATH="/opt/homebrew/opt/llvm/bin:$PATH"

bash configure --with-hsdis=llvm
make build-hsdis
make install-hsdis
export PATH=$OLDPATH

# Why doesn't install-hsdis do this?
cd build/macosx-aarch64-server-release
cp support/hsdis/libhsdis.dylib jdk/bin/

The JVM did not appear to be generating the disassembly even with this approach. A quick search for hsdis not printing assembly macOS leads to this post mentioning the error Could not load hsdis-amd64.dylib; library not loadable; PrintAssembly is disabled. This reminds me that theRealAph had pointed out that the library seems to be built with the wrong name, so the runtime doesn’t find it. So I just needed to specify that file name when copying the hsdis dylib in the last step!

cp support/hsdis/libhsdis.dylib jdk/bin/hsdis-aarch64.dylib

Building the Changes on Windows x86-64

Install the 64-bit Windows LLVM. Configure the OpenJDK repo using both the --with-hsdis and LLVM_CONFIG options as shown. I needed to use the 8.3 path name (obtained using the command suggested on StackOverflow) for value of the LLVM_CONFIG parameter.

bash configure --with-hsdis=llvm LLVM_CONFIG=C:/PROGRA~1/LLVM/bin

Unfortunately, this is not sufficient to enable building on Windows as detailed by this error:

$ make build-hsdis
Building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Creating support/hsdis/hsdis.dll from 1 file(s)
/usr/bin/bash: x86_64-w64-mingw32-g++: command not found
make[3]: *** [Hsdis.gmk:135: /..../build/windows-x86_64-server-release/support/hsdis/hsdis-llvm.obj] Error 127
make[2]: *** [make/Main.gmk:530: build-hsdis] Error 2

ERROR: Build failed for target 'build-hsdis' in configuration 'windows-x86_64-server-release' (exit code 2)

Jorn fixed this so we can add Jorn’s upstream JDK, fetch its commits, then cherry pick the commit with the fix.

git remote add jorn https://github.com/JornVernee/jdk/
git fetch jorn
git cherry-pick 8de8b763c9159f84bcc044c04ee2fac9f2390774

Some conflicts in make/Hsdis.gmk need to be resolved. This is straightforward since Jorn’s change splits the existing binutils Windows code into the first branch of an if-statement then adds support for the LLVM backend in the else case. The resolved conflicts are in my fork in the branch. The repo should now be configured with the additional --with-llvm option added by Jorn.

bash configure --with-hsdis=llvm LLVM_CONFIG=C:/PROGRA~1/LLVM/bin --with-llvm=C:/PROGRA~1/LLVM

Running make build-hsdis results in errors about missing LLVM includes.

$ make build-hsdis
Building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Creating support/hsdis/hsdis.dll from 1 file(s)
d:\.....\jdk\src\utils\hsdis\llvm\hsdis-llvm.cpp(58): fatal error C1083: Cannot open include file: 'llvm-c/Disassembler.h': No such file or directory
make[3]: *** [Hsdis.gmk:142: /cygdrive/d/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis-llvm.obj] Error 1
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [make/Main.gmk:530: build-hsdis] Error 2

Let’s try setting CC and CXX then rerunning the above configure command.

export CC=C:/PROGRA~1/LLVM/bin/clang.exe
export CXX=C:/PROGRA~1/LLVM/bin/clang++.exe

Turns out a Microsoft compiler is required!

configure: Will use user supplied compiler CC=C:/PROGRA~1/LLVM/bin/clang.exe
checking resolved symbolic links for CC... no symlink
configure: The C compiler (located as C:/PROGRA~1/LLVM/bin/clang.exe) does not seem to be the required microsoft compiler.
configure: The result from running it was: "clang: error: no input files"
configure: error: A microsoft compiler is required. Try setting --with-tools-dir.
configure exiting with result code 1

But let’s see what happens if we change the toolchain type to clang:

# This command does not work
bash configure --with-hsdis=llvm LLVM_CONFIG=C:/PROGRA~1/LLVM/bin --with-llvm=C:/PROGRA~1/LLVM --with-toolchain-type=clang

I guess they were serious about that since clang is not valid on this platform.

configure: Toolchain type clang is not valid on this platform.
configure: Valid toolchains: microsoft.
configure: error: Cannot continue.
configure exiting with result code 1

Indeed, clang is not a valid toolchain for Windows as declared in make/autoconf/toolchain.m4. Open question: how is the VALID_TOOLCHAIN_windows actually checked? So we can now unset the environment variables.

unset CC
unset CXX

This brought me back to the first thing I should have done when I saw the “No such file or directory” error – verifying that the file existed on disk! This is all there is there:

$ ls C:/PROGRA~1/LLVM/include/llvm-c
Remarks.h  lto.h

Well, turns out this is the issue that led Jorn to build LLVM manually. I now know what the needed header files being referred to are. So let’s build LLVM using Jorn’s steps.

git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build_llvm
cd build_llvm
cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=X86" -D"CMAKE_BUILD_TYPE:STRING=Release" -D"CMAKE_INSTALL_PREFIX=install_local" -A x64 -T host=x64
cmake --build . --config Release --target install

The last command fails with the error below!??? Why can’t anything just simply work?

  Building Opts.inc...
  '..\..\RelWithDebInfo\bin\llvm-tblgen.exe' is not recognized as an internal or external command,
  operable program or batch file.
C:\Program Files\Microsoft Visual Studio\2022\Preview\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(243,5): error MSB8066: Custom build for 'D:\dev\repos\llvm-project\build_llvm\CMakeFiles\dd1f7b42098
1667d7f617e96802947d3\Opts.inc.rule;D:\dev\repos\llvm-project\build_llvm\CMakeFiles\9fbf2dc5caba7f0c75934f43d12abdf5\RcOptsTableGen.rule;D:\dev\repos\llvm-project\llvm\tools\llvm-rc\CMakeLists.txt' exited wit
h code 9009. [D:\dev\repos\llvm-project\build_llvm\tools\llvm-rc\RcOptsTableGen.vcxproj]

Switch to my Surface Book 2 and LLVM builds just fine!

bash configure --with-hsdis=llvm LLVM_CONFIG=C:/dev/repos/llvm-project/build_llvm/install_local/bin --with-llvm=C:/dev/repos/llvm-project/build_llvm/install_local/

Interestingly, this fails with the same errors I saw on macOS:

$ make build-hsdis
Building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Creating support/hsdis/hsdis.dll from 1 file(s)
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMCreateDisasm referenced in function "public: __cdecl hsdis_backend::hsdis_backend(unsigned __int64,unsi...,char const *,int)" (??0hsdis_backend@@QEAA@_K0PEAE0P6APEAXPEAXPEBD2@Z2P6AH23ZZ23H@Z)
...
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86Disassembler referenced in function LLVMInitializeNativeDisassembler
c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.dll : fatal error LNK1120: 9 unresolved externals
make[3]: *** [Hsdis.gmk:142: /cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.dll] Error 1

The PATH environment variable probably needs to be adjusted to work around this.

Update 2022-02-08: the problem above is that bash configure is invoked with the wrong LLVM_CONFIG option – the actual llvm-config executable name is missing. See Troubleshooting hsdis LLVM backend MSVC Linker Errors for details.