The post about Exploring the hsdis LLVM Support PR mentioned link errors when building hsdis using an LLVM backend on Windows (x86-64 host building JDK for the x86-64 platform). Before we look at why linking fails, we can get a simple repro for the error from the Cygwin logs. To get the command line used to invoke the linker, run make LOG=debug build-hsdis. Search the output for link.exe to find the failing command or open build\windows-x86_64-server-release\support\hsdis\BUILD_HSDIS_link.cmdline. Change the path from Cygwin to Windows style so that the command can be run in the x64 Native Tools Command Prompt.
So there really is no such symbol in this lib folder! I’m guessing I need to add another lib folder to the path. A quick search for LLVMInitializeX86Disassembler leads to this post on Using the LLVM MC Disassembly API. It mentions using llvm-config to set the linker flags. Shouldn’t running the bash configure command take care of this? Let’s see what’s in the configure output:
...
checking what hsdis backend to use... 'llvm'
checking for LLVM_CONFIG... C:/dev/repos/llvm-project/build_llvm/install_local/bin [user supplied]
/cygdrive/c/dev/repos/java/forks/jdk/build/.configure-support/generated-configure.sh: line 135451: C:/dev/repos/llvm-project/build_llvm/install_local/bin: Is a directory
/cygdrive/c/dev/repos/java/forks/jdk/build/.configure-support/generated-configure.sh: line 135452: C:/dev/repos/llvm-project/build_llvm/install_local/bin: Is a directory
/cygdrive/c/dev/repos/java/forks/jdk/build/.configure-support/generated-configure.sh: line 135453: C:/dev/repos/llvm-project/build_llvm/install_local/bin: Is a directory
...
Well, that could be the problem! I think I need to fix the llvm-config path in Cygwin by appending /llvm-config to LLVM_CONFIG.
Sure enough, that was the problem! The bash configure output (below) now looks good and make build-hsdis now works. The fix for this would be to ensure bash configure fails if LLVM_CONFIG is set to the directory instead of the executable!
checking what hsdis backend to use... 'llvm'
checking for LLVM_CONFIG... C:/dev/repos/llvm-project/build_llvm/install_local/bin/llvm-config [user supplied]
checking for number of cores... 8
...
$ make build-hsdis
Building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Creating support/hsdis/hsdis.dll from 1 file(s)
Finished building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Notice from the new build command line in build\windows-x86_64-server-release\support\hsdis\BUILD_HSDIS_link.cmdline that there are now many .lib files supplied to the linker! These are the lib files that I was inspecting with dumpbin so my earlier hypothesis was wrong (there were no additional .lib files required, the ones I was looking at were simply not being passed to the linker).
Now running make install-hsdis copies hsdis-amd64.dll into /build/windows-x86_64-server-release/jdk/bin. The LLVM hsdis backend can now be used to disassemble instructions:
Here are some of the bugs/questions I looked at when investigating these failures. Stack overflow taught me about dumpbin and C++ decorated names/ the undname tool.
A previous post explored how to use LLVM as the backend disassembler for hsdis. The instructions for how to use GNU binutils (the currently supported option) are straightforward. Listing them here for completeness (assuming you have cloned the OpenJDK repo into your ~/repos/java/jdk folder). Note that they depend on more recent changes. See the docs on the Java command for more info about the -XX:CompileCommand option.
# Download and extract GNU binutils 2.37
cd ~
curl -Lo binutils-2.37.tar.gz https://ftp.gnu.org/gnu/binutils/binutils-2.37.tar.gz
tar xvf binutils-2.37.tar.gz
# Configure the OpenJDK repo for hsdis
cd ~/repos/java/jdk
bash configure --with-hsdis=binutils --with-binutils-src=~/binutils-2.37
# Build hsdis
make build-hsdis
To deploy the built hsdis library on macOS:
cd build/macosx-aarch64-server-release
# Copy the hsdis library into the JDK bin folder
cp support/hsdis/libhsdis.dylib jdk/bin/hsdis-aarch64.dylib
To deploy the built hsdis library on Ubuntu Linux (open question: is this step even necessary?):
cd build/linux-x86_64-server-release
# Copy the hsdis library into the JDK bin folder
cp support/hsdis/libhsdis.so jdk/bin/
Update 2024-03-13: use the make install-hsdis command to copy the hsdis binaries into the new OpenJDK build. This will ensure that the hsdis binary is copied to lib/hsdis-adm64.so (this file name should be used in place of any others that listed by find . -name *hsdis*).
Now we can disassemble some code, e.g. the String.checkIndex method mentioned in PR 5920.
# Disassemble some code
jdk/bin/java -XX:CompileCommand="print java.lang.String::checkIndex" -version
To see how to disassemble the code for a class, we can use the basic substitution cipher class from the post on Building HSDIS in Cygwin as an example. Download, compile and disassemble it using the commands below. Note that these commands save the .java file to a temp folder to make cleanup much easier. Also note the redirection to a file since the output can be voluminous.
cd jdk/bin
mkdir -p temp
cd temp
curl -Lo BasicSubstitutionCipher.java https://raw.githubusercontent.com/swesonga/scratchpad/main/apps/crypto/substitution-cipher/BasicSubstitutionCipher.java
../javac BasicSubstitutionCipher.java
../java -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -XX:+LogCompilation BasicSubstitutionCipher > disassembled.txt
open disassembled.txt
The previous post described how LLVM can be configured as the disassembly backend for hsdis. Here, I explain the process it took for me to figure out the details of the change adding support for LLVM. One of the first things to do when learning these details of this change is to build it. Since I’m using my own fork of the OpenJDK repo, I need to add the upstream repo to my remotes. This makes it possible to fetch commits from PRs submitted to the upstream repo.
cd ~/repos/forks/jdk
git remote add upstream https://github.com/openjdk/jdk
git fetch upstream
The LLVM-backend PR has only 1 commit (as of this writing). Create a new branch then cherry-pick that commit (I was on commit 77757ba9 when I wrote this.
Install LLVM using homebrew (if it is not already installed).
brew install llvm
Set the the LDFLAGS and CPPFLAGS environment variables then run printenv | grep -i flags to verify that the flags have been set correctly. Exporting CC and CXX is crucial since that is how to let bash configure know that we need a custom compiler for the build!
Run make build-hsdis in the root folder of the jdk repo.
If the proper flags have not been set, make will fail with the error below. Run make --debug=v for additional information on what make is doing.
saint@Saints-MBP-2021 jdk % make build-hsdis
Building target 'build-hsdis' in configuration 'macosx-aarch64-server-release'
/Users/saint/repos/java/forks/jdk/src/utils/hsdis/llvm/hsdis-llvm.cpp:58:10: fatal error: 'llvm-c/Disassembler.h' file not found
#include <llvm-c/Disassembler.h>
^~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[3]: *** [/Users/saint/repos/java/forks/jdk/build/macosx-aarch64-server-release/support/hsdis/hsdis-llvm.o] Error 1
make[2]: *** [build-hsdis] Error 2
ERROR: Build failed for target 'build-hsdis' in configuration 'macosx-aarch64-server-release' (exit code 2)
After all that fidgeting around, the fix is as simple as updating your path to include LLVM <insert facepalm / clown>. This is what installing LLVM using brew ends with:
...
==> llvm
To use the bundled libc++ please add the following LDFLAGS:
LDFLAGS="-L/opt/homebrew/opt/llvm/lib -Wl,-rpath,/opt/homebrew/opt/llvm/lib"
llvm is keg-only, which means it was not symlinked into /opt/homebrew,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.
If you need to have llvm first in your PATH, run:
echo 'export PATH="/opt/homebrew/opt/llvm/bin:$PATH"' >> ~/.zshrc
For compilers to find llvm you may need to set:
export LDFLAGS="-L/opt/homebrew/opt/llvm/lib"
export CPPFLAGS="-I/opt/homebrew/opt/llvm/include"
My MacBook didn’t even have a ~/.zshrc file. Setting the PATH using the suggestion above fixed the build errors!
Now open a new terminal and configure the repo (no need for LLVM_CONFIG).
% bash configure --with-hsdis=llvm
% make build-hsdis
Interestingly, running make images does not work on subsequent attempts?! After further investigation, it turns out that the clang compiler installed by brew cannot successfully compile the OpenJDK sources. Why does it issue warnings that Apple’s clang compiler does not?
In file included from /Users/saint/repos/java/forks/jdk/src/hotspot/cpu/aarch64/abstractInterpreter_aarch64.cpp:31:
In file included from /Users/saint/repos/java/forks/jdk/src/hotspot/share/runtime/frame.inline.hpp:42:
In file included from /Users/saint/repos/java/forks/jdk/src/hotspot/cpu/aarch64/frame_aarch64.inline.hpp:31:
In file included from /Users/saint/repos/java/forks/jdk/src/hotspot/cpu/aarch64/pauth_aarch64.hpp:28:
/Users/saint/repos/java/forks/jdk/src/hotspot/os_cpu/bsd_aarch64/pauth_bsd_aarch64.inline.hpp:29:10: fatal error: 'ptrauth.h' file not found
#include <ptrauth.h>
^~~~~~~~~~~
1 error generated.
make[3]: *** [/Users/saint/repos/java/forks/jdk/build/macosx-aarch64-server-release/hotspot/variant-server/libjvm/objs/abstractInterpreter_aarch64.o] Error 1
m
To work around this, first build the JDK using Apple’s clang. Next, add brew’s LLVM installation to the PATH, then configure for hsdis. Finally, build hsdis:
# Warning: ensure /opt/homebrew/opt/llvm/bin is not in the PATH
cd ~/repos/java/forks/jdk
bash configure
make images
# Now add brew's LLVM to the PATH before running bash configure
export OLDPATH=$PATH
export PATH="/opt/homebrew/opt/llvm/bin:$PATH"
bash configure --with-hsdis=llvm
make build-hsdis
make install-hsdis
export PATH=$OLDPATH
# Why doesn't install-hsdis do this?
cd build/macosx-aarch64-server-release
cp support/hsdis/libhsdis.dylib jdk/bin/
Install the 64-bit WindowsLLVM. Configure the OpenJDK repo using both the --with-hsdis and LLVM_CONFIG options as shown. I needed to use the 8.3 path name (obtained using the command suggested on StackOverflow) for value of the LLVM_CONFIG parameter.
Unfortunately, this is not sufficient to enable building on Windows as detailed by this error:
$ make build-hsdis
Building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Creating support/hsdis/hsdis.dll from 1 file(s)
/usr/bin/bash: x86_64-w64-mingw32-g++: command not found
make[3]: *** [Hsdis.gmk:135: /..../build/windows-x86_64-server-release/support/hsdis/hsdis-llvm.obj] Error 127
make[2]: *** [make/Main.gmk:530: build-hsdis] Error 2
ERROR: Build failed for target 'build-hsdis' in configuration 'windows-x86_64-server-release' (exit code 2)
Jorn fixed this so we can add Jorn’s upstream JDK, fetch its commits, then cherry pick the commit with the fix.
git remote add jorn https://github.com/JornVernee/jdk/
git fetch jorn
git cherry-pick 8de8b763c9159f84bcc044c04ee2fac9f2390774
Some conflicts in make/Hsdis.gmk need to be resolved. This is straightforward since Jorn’s change splits the existing binutils Windows code into the first branch of an if-statement then adds support for the LLVM backend in the else case. The resolved conflicts are in my fork in the branch. The repo should now be configured with the additional --with-llvm option added by Jorn.
Running make build-hsdis results in errors about missing LLVM includes.
$ make build-hsdis
Building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Creating support/hsdis/hsdis.dll from 1 file(s)
d:\.....\jdk\src\utils\hsdis\llvm\hsdis-llvm.cpp(58): fatal error C1083: Cannot open include file: 'llvm-c/Disassembler.h': No such file or directory
make[3]: *** [Hsdis.gmk:142: /cygdrive/d/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis-llvm.obj] Error 1
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [make/Main.gmk:530: build-hsdis] Error 2
Let’s try setting CC and CXX then rerunning the above configure command.
configure: Will use user supplied compiler CC=C:/PROGRA~1/LLVM/bin/clang.exe
checking resolved symbolic links for CC... no symlink
configure: The C compiler (located as C:/PROGRA~1/LLVM/bin/clang.exe) does not seem to be the required microsoft compiler.
configure: The result from running it was: "clang: error: no input files"
configure: error: A microsoft compiler is required. Try setting --with-tools-dir.
configure exiting with result code 1
But let’s see what happens if we change the toolchain type to clang:
# This command does not work
bash configure --with-hsdis=llvm LLVM_CONFIG=C:/PROGRA~1/LLVM/bin --with-llvm=C:/PROGRA~1/LLVM --with-toolchain-type=clang
configure: Toolchain type clang is not valid on this platform.
configure: Valid toolchains: microsoft.
configure: error: Cannot continue.
configure exiting with result code 1
Indeed, clang is not a valid toolchain for Windows as declared in make/autoconf/toolchain.m4. Open question: how is the VALID_TOOLCHAIN_windows actually checked? So we can now unset the environment variables.
unset CC
unset CXX
This brought me back to the first thing I should have done when I saw the “No such file or directory” error – verifying that the file existed on disk! This is all there is there:
$ ls C:/PROGRA~1/LLVM/include/llvm-c
Remarks.h lto.h
Well, turns out this is the issue that led Jorn to build LLVM manually. I now know what the needed header files being referred to are. So let’s build LLVM using Jorn’s steps.
git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build_llvm
cd build_llvm
cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=X86" -D"CMAKE_BUILD_TYPE:STRING=Release" -D"CMAKE_INSTALL_PREFIX=install_local" -A x64 -T host=x64
cmake --build . --config Release --target install
The last command fails with the error below!??? Why can’t anything just simply work?
Building Opts.inc...
'..\..\RelWithDebInfo\bin\llvm-tblgen.exe' is not recognized as an internal or external command,
operable program or batch file.
C:\Program Files\Microsoft Visual Studio\2022\Preview\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(243,5): error MSB8066: Custom build for 'D:\dev\repos\llvm-project\build_llvm\CMakeFiles\dd1f7b42098
1667d7f617e96802947d3\Opts.inc.rule;D:\dev\repos\llvm-project\build_llvm\CMakeFiles\9fbf2dc5caba7f0c75934f43d12abdf5\RcOptsTableGen.rule;D:\dev\repos\llvm-project\llvm\tools\llvm-rc\CMakeLists.txt' exited wit
h code 9009. [D:\dev\repos\llvm-project\build_llvm\tools\llvm-rc\RcOptsTableGen.vcxproj]
Switch to my Surface Book 2 and LLVM builds just fine!
Interestingly, this fails with the same errors I saw on macOS:
$ make build-hsdis
Building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Creating support/hsdis/hsdis.dll from 1 file(s)
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMCreateDisasm referenced in function "public: __cdecl hsdis_backend::hsdis_backend(unsigned __int64,unsi...,char const *,int)" (??0hsdis_backend@@QEAA@_K0PEAE0P6APEAXPEAXPEBD2@Z2P6AH23ZZ23H@Z)
...
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86Disassembler referenced in function LLVMInitializeNativeDisassembler
c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.dll : fatal error LNK1120: 9 unresolved externals
make[3]: *** [Hsdis.gmk:142: /cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.dll] Error 1
The PATH environment variable probably needs to be adjusted to work around this.
Update 2022-02-08: the problem above is that bash configure is invoked with the wrong LLVM_CONFIG option – the actual llvm-config executable name is missing. See Troubleshooting hsdis LLVM backend MSVC Linker Errors for details.
To specify a backend for hsdis, the OpenJDK repo needs to be configured with the --with-hsdis option. As of commit 77757ba9, LLVM is not yet supported as an hsdis disassembly backend. Therefore, this error from make/autoconf/jdk-options.m4 is displayed. Here’s an example on the Windows platform:
$ bash configure --with-hsdis=llvm
...
checking what hsdis backend to use... invalid
configure: error: Incorrect hsdis backend "llvm"
configure exiting with result code 1
To test the LLVM backend for hsdis on macOS, install LLVM using brew (Apple’s LLVM does not have the llvm-c include files):
# install LLVM
brew install llvm
Now build the OpenJDK. This should use Apple’s compiler since we have not made any configuration changes.
cd ~/repos/java/jdk
bash configure
make images
Now add brew’s LLVM bin directory to the PATH and run bash configure again passing the --with-hsdis=llvm option as shown below. The configuration process will detect the clang++ compiler installed by brew and set it up for use when the build-hsdis target is executed.
# Now add brew's LLVM to the PATH before running bash configure
export OLDPATH=$PATH
export PATH="/opt/homebrew/opt/llvm/bin:$PATH"
bash configure --with-hsdis=llvm
make build-hsdis
make install-hsdis
export PATH=$OLDPATH
The install-hsdis target does not appear to be copying the hsdis library to the jdk/bin folder so these commands are required:
cd build/macosx-aarch64-server-release
cp support/hsdis/libhsdis.dylib jdk/bin/hsdis-aarch64.dylib
git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build_llvm
cd build_llvm
cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=X86" -D"CMAKE_BUILD_TYPE:STRING=Release" -D"CMAKE_INSTALL_PREFIX=install_local" -A x64 -T host=x64
cmake --build . --config Release --target install
Now we can configure the OpenJDK repo for hsdis, and build both the JDK and hsdis.
bash configure --with-hsdis=llvm \
LLVM_CONFIG=C:/dev/repos/llvm-project/build_llvm/install_local/bin \
--with-llvm=C:/dev/repos/llvm-project/build_llvm/install_local/
make build-hsdis
make images
hsdis LLVM backend on Windows ARM64
Open question: is this supported?
Testing the hsdis LLVM backend
The String.checkIndex method of PR 5920 is a good candidate for testing the hsdis LLVM backend. The -XX:CompileCommand option can be used to print the generated assembler code after compilation of the specified method.
To view the command lines being executed by make as well as the value of the variables in use: make LOG=debug build-hsdis
Autoconf macros are defined using the AC_DEFUN macro. The JDKOPT_SETUP_HSDIS macro (modified by PR 5920) is defined using AC_DEFUN_ONCE, which is for macros that should only be called once.
To address warnings like ld: warning: dylib (/opt/homebrew/opt/llvm/lib/libunwind.dylib) was built for newer macOS version (12.0) than being linked (11.0) update MACOSX_VERSION_MIN in make/autoconf/flags.m4.
Last month I took on the task of writing a script to run the JDK JMH microbenchmarks. One of my coworkers already had a bash script for running a different set of benchmarks (GC), so I used it as a starting point. Having written mostly C# code in that past few years, I found the bash scripting environment unintuitive. Thankfully, there is a bash manual to refer to. I thought it might be worthwhile documenting a few of my lessons and surprises from that undertaking.
Bash Ignores +x File Mode
As a mostly-Windows user, I am accustomed to batch files (.bat) and PowerShell scripts (.ps1). These do not need any ceremony to declare them executable. I was using Windows 11 and testing my scripts in a bash environment in Windows Terminal. However, the scripts didn’t work when cloned onto my Linux and MacOS systems. That was when I observed that such scripts are not shown as executable by ls -l. However, bash runs them just fine in the Windows Terminal as shown below (note that this is the version of bash distributed with git-scm (and therefore based on MSYS)!
$ echo "uname -a" > test.sh; ls -l; ./test.sh
total 1
-rw-r--r-- 1 saint 1049089 9 Jan 8 15:43 test.sh
MSYS_NT-10.0-22000 SAINT 3.1.7-340.x86_64 2020-10-23 13:08 UTC x86_64 Msys
Interestingly, chmod cannot set the execute mode in the Windows Terminal.
$ chmod +x ./test.sh; ls -l
total 1
-rw-r--r-- 1 saint 1049089 9 Jan 8 15:43 test.sh
However, prefixing the script with #!/bin/bash makes ls -l show it as executable without any other changes.
$ echo -e '#!/bin/bash\nuname -a' > test.sh; ls -l
total 1
-rwxr-xr-x 1 saint 1049089 21 Jan 8 16:16 test.sh*
On MacOS (and Linux), the script cannot be executed without running chmod +x ./test.sh
% echo "uname -a" > test.sh; ls -l; ./test.sh
total 8
-rw-r--r-- 1 saint staff 9 Jan 8 15:49 test.sh
zsh: permission denied: ./test.sh
Also note that the #!/bin/bash prefix is required for command completion in the shell.
Echoing #!/bin/bash to a File Needs Single Quotes
Most of the script writing I did used double quotes (since they support variable interpolation, which I needed). I was testing the impact of not using the #!/bin/bash prefix in my scripts for the scenarios above when I ran into this strange error:
$ echo "#!/bin/bash" > temp.sh
bash: !/bin/bash: event not found
I had forgotten about the shell’s support for history expansion as pointed out in the solution here: the way around this is to use single quotes as in echo -e '#!/bin/bash\nuname -a'
… Bash Supports History Expansion
I played around with this history expansion feature to get a feel for it. I first renamed my .bash_history file then opened a new bash tab to start a new session with clean history.
uname -a
ls ~/.bash*
git --version
history
grep --version
which bash
!-2 # Shows grep version
!3 # Shows git version
!-5 # Shows history
!! # Also shows history
Echoing Newlines Needs a Flag
The -e flag option is required to write newline characters to the output (instead of the literal \ and n characters. As per “Simple Commands” in the GNU bash manual, if the -e option is given, interpretation of the following backslash-escaped characters is enabled… \a, \b, \c, \e, \E, \f, \n, \r, \t, \v, \\, etc.
Git Supports File Modes
After pulling some changes to my MacBook and making some edits, I saw the diff below. I have never really had to pay attention to file modes, so this was intriguing.
diff --git a/java/jmh/setup_jmh_jdk_micros.sh b/java/jmh/setup_jmh_jdk_micros.sh
old mode 100644
new mode 100755
The difference is the execute bit is now on for all 3 user types. Apparently, 644 and 755 are the only file modes supported by git and the core.fileMode option, which tells Git if the executable bit of files in the working tree is to be honored, should be set to false if the filesystem a file is checked out on loses the executable bit on checkout. Open questions: does NTFS lose executable bits, i.e. does this property apply to NTFS? Is there a way to see the filemode in GitHub?
[ is an Executable!
Not even sure what to say about this one :D.
$ which [
/usr/bin/[
In Cygwin, we can get the path to the executable!
$ cygpath -w `which [`
C:\dev\cygwin64\bin\[.exe
Open questions: is this program being invoked for if-statements?
Miscellaneous Facts
These are observations are documented the Bash Manual but listed here since they’re things I learned along the way.
To view the full path of files in the current directory, use ls -d $PWD/*
To get rid of ^M (carriage-return characters) in a text file from Windows, open the file in vim then these commands: :e ++ff=dos then :set ff=unix then finally exit vim with :wq
Functions cannot return any value, they return codes. Specifically, (as per the GNU bash manual’s Shell Functions section), if a numeric argument is given to return, that is the function’s return status; otherwise the function’s return status is the exit status of the last command executed before the return.
~= is a binary operator used for regular expression matching. The string to the right of the operator is considered a POSIX extended regular expression. The return value is 0 if the string matches the pattern, and 1 otherwise. If the regular expression is syntactically incorrect, the conditional expression’s return value is 2. See the Conditional Constructs section of the GNU bash manual.
See Windows Terminal tips and tricks for various useful customizations of the Windows Terminal. Changing tab colors is something I wouldn’t have thought to do but now makes it very easy to know which tabs are relevant for the task at hand.
The previous post on Building HSDIS in Cygwin required running this command to actually build the hsdis DLL.
make OS=Linux MINGW=x86_64-w64-mingw32 BINUTILS=~/binutils-2.37
As it turns out, this make command fails because of a bug in the GNU binutils source code. This is the error I got:
...
x86_64-w64-mingw32-gcc -c -DHAVE_CONFIG_H -O -I. -I/home/User/binutils-2.37/libiberty/../include -W -Wall -Wwrite-strings -Wc++-compat -Wstrict-prototypes -Wshadow=local -pedantic -D_GNU_SOURCE /home/User/binutils-2.37/libiberty/rust-demangle.c -o rust-demangle.o
/home/User/binutils-2.37/libiberty/rust-demangle.c:78:3: error: unknown type name ‘uint’
78 | uint recursion;
| ^~~~
/home/User/binutils-2.37/libiberty/rust-demangle.c: In function ‘demangle_path’:
/home/User/binutils-2.37/libiberty/rust-demangle.c:81:37: error: ‘uint’ undeclared (first use in this function); did you mean ‘int’?
81 | #define RUST_NO_RECURSION_LIMIT ((uint) -1)
| ^~~~
...
make[2]: *** [Makefile:1230: rust-demangle.o] Error 1
...
At this point, I wasn’t sure which version I used to build successfully. Searching for that error (and binutils to narrow things down) led to this bug in the sourceware.org Bugzilla that appears to be the exact bug I ran into: 28207 – error: unknown type name ‘uint’ (78 | uint recursion;) avr-gcc mingw32 Windows Build (sourceware.org). Fortunately, one Alan helpfully points out that this bug fixed on the binutils-2.37 branch with commit 999566402e3.
I’m used to GitHub where looking at the repo structure implies that you’re at a URL you can copy and trim to clone. In this other web view, the URL to clone is listed above the shortlog:
At this point, it makes sense to verify that the 2.37 sources I downloaded actually contain the bug. Observe that:
the tags section contains a binutils-2_37 tag described as “Official GNU Binutils 2.37 Release” and committed on Sun, 18 Jul 2021 16:46:54 +0000 (17:46 +0100).
the fix for the build error shows a fix committed by Alan on Mon, 19 Jul 2021 11:32:21 +0000 (21:02 +0930)
Therefore, using binutils older than 2.37 should work just fine. However, it may still be necessary to run “rm -fr build” in the hsdis folder to enable 2.36 to be picked up when you run make (otherwise 2.37 is still baked into some of configure’s output).
Hsdis is an externally loadable disassembler plugin. It lets you see which assembly instructions the JVM generates for your Java code. On 64-bit Windows, it is a binary called hsdis-amd64.dll (and hsdis-i386.dll on 32-bit platforms). This binary needs to be in the same directory as jvm.dll. Some good resources out there on building the hsdis binary for the OpenJDK include:
For Cygwin, the latter resource (from 2012?) is all we need. I like that Gunnar’s blog post covered how to use hsdis after building it so this writeup aims to combine both blogs into a simple Cygwin install-build-disassemble set of instructions.
Building hsdis for 64-bit JVMs
Install Cygwin with the gcc-core, make, and mingw64-x86_64-gcc-core packages by launching the setup executable using this command (no need to bother selecting packages in the UI since you have already specified them on the command line)
setup-x86_64.exe -P gcc-core -P mingw64-x86_64-gcc-core -P make
mkdir ~/repos
cd ~/repos
git clone https://github.com/openjdk/jdk
Run these commands to download GNU binutils and build hsdis (Update 2022-01-07: version downgraded to 2.36 to avoid build failures investigated in Fixing Hsdis Compile Failure in GNU binutils).
cd ~
curl -Lo binutils-2.36.tar.gz https://ftp.gnu.org/gnu/binutils/binutils-2.36.tar.gz
tar xvf binutils-2.36.tar.gz
cd ~/repos/jdk/src/utils/hsdis
make OS=Linux MINGW=x86_64-w64-mingw32 BINUTILS=~/binutils-2.36
Copy the hsdis binary to the locally built java bin folder
I have created a basic substitution cipher, which we can compile and disassemble using the commands below. Note that these commands save the .java file to a temp folder to make cleanup much easier. Also note the redirection to a file since the output can be voluminous.
cd build/windows-x86_64-server-release/jdk/bin
mkdir -p temp
cd temp
curl -Lo BasicSubstitutionCipher.java https://raw.githubusercontent.com/swesonga/scratchpad/main/apps/crypto/substitution-cipher/BasicSubstitutionCipher.java
../javac BasicSubstitutionCipher.java
../java -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -XX:+LogCompilation BasicSubstitutionCipher > BasicSubstitutionCipher.disassembled.txt
Once the disassembly completes, we can view the instructions generated in the BasicSubstitutionCipher.disassembled.txt file.
One open question in this setup is why the installed GNU binutils cannot be used to build hsdis. Seems strange to have to build them from source when the binutils Cygwin package was also installed in step 1 above.
$ zip -usd ziphelp.zip *.txt
sd: Zipfile name 'ziphelp.zip'
sd: Command line read
sd: Reading archive
zip warning: expected 3 entries but found 0
zip error: Zip file structure invalid (ziphelp.zip)
Add debug = 1 to the top of win32\makefile.w32 then run these commands to generate PDBs along with the other binaries.
set LOCAL_ZIP=-DDEBUG
nmake -f win32\makefile.w32
Investigating the Failure
A quick way to set up Visual Studio to debug this issue is to create a new C++ console application then change the command, command arguments, and working directory of the project as shown below.
Setting Visual C++ Project Properties for Debugging Zip.exe
Pressing F10 should now start debugging the zip binary. The highlights are:
The question now is why scanzipf_regnew does not find any entries since there are 3 PK12 and 3 PK34 signatures in the file. Let us inspect the file offsets right after the code seeks to in_cd_start_offset (from step 5 above) and immediately before looking for the next signature to process.
This shows a value of 0x0000000000002168 which is not the proper offset in the zip file to seek to. Recall that in_cd_start_offset was 0x218e, which is the second-last byte of line 537 in the zipfile’s hexdump output. Could this be an error in the standard library fseek and ftell functions? TODO: why does the scan fail from an earlier starting point?
Pressing F12 to see the definitions of zftello and zfseeko went to the wrong place! These are not the standard library functions being used. Visual Studio was opening their definitions in tailor.h instead of the actual implementations being called. Turns out zftello and zfseeko are functions implemented in win32/win32i64.c. These comments above zftello raise some huge red flags.
/* 64-bit buffered ftello
*
* Win32 does not provide a 64-bit buffered
* ftell (in the published api anyway) so below provides
* hopefully close version.
* We have not gotten _telli64 to work with buffered
* streams. Below cheats by using fgetpos improperly and
* may not work on other ports.
*/
/* 64-bit buffered fseeko
*
* Win32 does not provide a 64-bit buffered
* fseeko so use _lseeki64 and fflush. Note
* that SEEK_CUR can lose track of location
* if fflush is done between the last buffered
* io and this call.
*/
Looks like this custom seek/tell code is responsible for the incorrect offsets into the zip file! We can work around this by simply removing these custom implementations.
Windows Specific Bug?
So why didn’t Cygwin’s zip.exe have this issue? Running zip.exe -v shows this compiler/OS description:
Compiled with gcc 4.8.3 for Unix (Cygwin) on Jun 23 2014.
The Cygwin OS name in parenthesis is defined in unix/unix.c only if __CYGWIN__ is defined. However, under this condition, the custom zftello and zfseeko implementations will not be included in the zip sources being compiled! Therefore, the issue does not occur in Cygwin’s distributed zip binary.
After figuring out how to build the Info-ZIP sources, I had a few commands to test the zip file by creating a few text files to zip.
zip -h > help.txt
zip -h2 > help2.txt
zip -L > license.txt
Unfortunately, the zip -qru ./files.zip -i *.txtcommand from the OpenJDK is not what we need. To actually create a zip file, use only the -u flag
zip -u files.zip *.txt
To test that the files were zipped successfully, unzip the files and compare them to the original files. Here’s the whole script for this:
echo "---Creating temp directory---"
mkdir temp; cd temp
echo "---Creating text files---"
zip -h > help.txt
zip -h2 > help2.txt
zip -L > license.txt
echo "---Adding text files to a new repo---"
git init
git add *.txt
git commit -m "Add original text files"
echo "---Zipping text files---"
zip -u files.zip *.txt
echo "---Removing text files---"
rm *.txt
echo "---Unzipping text files---"
unzip files.zip
echo "---Checking unzipped files---"
git diff
When using the zip binary for Windows, something strange happens when running the zip command a 2nd time:
$ zip -u files.zip *.txt
zip warning: files.zip not found or empty
adding: help.txt (176 bytes security) (deflated 49%)
adding: help2.txt (176 bytes security) (deflated 62%)
adding: license.txt (176 bytes security) (deflated 54%)
$ zip -u files.zip *.txt
zip warning: expected 3 entries but found 0
zip error: Zip file structure invalid (files.zip)
Info-ZIP supports a -sd flag that shows diagnostic information while it runs. It reveals that something is going wrong when reading the archive.
$ zip -usd files.zip *.txt
sd: Zipfile name 'files.zip'
sd: Command line read
sd: Reading archive
zip warning: expected 3 entries but found 0
zip error: Zip file structure invalid (files.zip)
In the last post, I described how to build the Info-ZIP sources. When using the resulting zip binaries in Cygwin, some important path handling issues come up. The paths passed to the zip binary when building the OpenJDK in Cygwin use forward slashes. The Cygwin User’s Guide has a section on File Access that outlines the support for POSIX and Win32-style file paths.
The Windows file system APIs support forward slashes in file paths. The zip source code uses the fopen CRT function, which eventually ends up calling CreateFileW. The CreateFileW docs state that you may use either forward slashes (/) or backslashes (\) in the lpFileName parameter. The translation of paths from Win32 to NT happens in a function called RtlDosPathNameToRelativeNtPathName_U as discussed in the Definitive Guide on Win32 to NT Path Conversion. Since this is a built-in Windows function, it does not support the /cygdrive/ style prefixes. Running the simple test program argtofile in Cygwin easily demonstrates this.
The /cygdrive/ prefixes will therefore not work for programs compiled for Windows (such as the zip binary directly compiled using Visual C++). Therefore, the cygpath command is necessary to translate these paths to Win32-style file paths. To peek into how cygpath works, we can take advantage of the fact that the source code for the cygpath utility is available online. I found it easier to browse the sources after cloning the repo:
The scenario of interest is what happens when cygpath -u ~ is invoked. In this case, we want to see how the “/cygdrive/” string is prefixed to the computed path.
normalizes the path by calling normalize_win32_path
before finally iterating through the mount items to find the path’s prefix in the mount table.
Also searching for the cygpath \s*( regex leads to the vcygpath function in winsup/utils/path.cc. That appears to be more directly related to the cygpath command (how?). Searching for the \"cygdrive\" regex also reveals that this is a magic string used in many places in the codebase.
All this shows that there is indeed some complexity behind maintaining the POSIX/Win32-style file path mapping in Cygwin but it should be possible to add some basic logic to the Windows Info-ZIP build to handle /cygdrive/ prefixes in its file arguments. The question I have at this point is how does compiling the zip binaries for the Cygwin environment (the shipping configuration) result in proper handling of POSIX-style filenames?
