The previous post described how LLVM can be configured as the disassembly backend for hsdis. Here, I explain the process it took for me to figure out the details of the change adding support for LLVM. One of the first things to do when learning these details of this change is to build it. Since I’m using my own fork of the OpenJDK repo, I need to add the upstream repo to my remotes. This makes it possible to fetch commits from PRs submitted to the upstream repo.
cd ~/repos/forks/jdk
git remote add upstream https://github.com/openjdk/jdk
git fetch upstream
The LLVM-backend PR has only 1 commit (as of this writing). Create a new branch then cherry-pick that commit (I was on commit 77757ba9 when I wrote this.
Install LLVM using homebrew (if it is not already installed).
brew install llvm
Set the the LDFLAGS and CPPFLAGS environment variables then run printenv | grep -i flags to verify that the flags have been set correctly. Exporting CC and CXX is crucial since that is how to let bash configure know that we need a custom compiler for the build!
Run make build-hsdis in the root folder of the jdk repo.
If the proper flags have not been set, make will fail with the error below. Run make --debug=v for additional information on what make is doing.
saint@Saints-MBP-2021 jdk % make build-hsdis
Building target 'build-hsdis' in configuration 'macosx-aarch64-server-release'
/Users/saint/repos/java/forks/jdk/src/utils/hsdis/llvm/hsdis-llvm.cpp:58:10: fatal error: 'llvm-c/Disassembler.h' file not found
#include <llvm-c/Disassembler.h>
^~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[3]: *** [/Users/saint/repos/java/forks/jdk/build/macosx-aarch64-server-release/support/hsdis/hsdis-llvm.o] Error 1
make[2]: *** [build-hsdis] Error 2
ERROR: Build failed for target 'build-hsdis' in configuration 'macosx-aarch64-server-release' (exit code 2)
After all that fidgeting around, the fix is as simple as updating your path to include LLVM <insert facepalm / clown>. This is what installing LLVM using brew ends with:
...
==> llvm
To use the bundled libc++ please add the following LDFLAGS:
LDFLAGS="-L/opt/homebrew/opt/llvm/lib -Wl,-rpath,/opt/homebrew/opt/llvm/lib"
llvm is keg-only, which means it was not symlinked into /opt/homebrew,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.
If you need to have llvm first in your PATH, run:
echo 'export PATH="/opt/homebrew/opt/llvm/bin:$PATH"' >> ~/.zshrc
For compilers to find llvm you may need to set:
export LDFLAGS="-L/opt/homebrew/opt/llvm/lib"
export CPPFLAGS="-I/opt/homebrew/opt/llvm/include"
My MacBook didn’t even have a ~/.zshrc file. Setting the PATH using the suggestion above fixed the build errors!
Now open a new terminal and configure the repo (no need for LLVM_CONFIG).
% bash configure --with-hsdis=llvm
% make build-hsdis
Interestingly, running make images does not work on subsequent attempts?! After further investigation, it turns out that the clang compiler installed by brew cannot successfully compile the OpenJDK sources. Why does it issue warnings that Apple’s clang compiler does not?
In file included from /Users/saint/repos/java/forks/jdk/src/hotspot/cpu/aarch64/abstractInterpreter_aarch64.cpp:31:
In file included from /Users/saint/repos/java/forks/jdk/src/hotspot/share/runtime/frame.inline.hpp:42:
In file included from /Users/saint/repos/java/forks/jdk/src/hotspot/cpu/aarch64/frame_aarch64.inline.hpp:31:
In file included from /Users/saint/repos/java/forks/jdk/src/hotspot/cpu/aarch64/pauth_aarch64.hpp:28:
/Users/saint/repos/java/forks/jdk/src/hotspot/os_cpu/bsd_aarch64/pauth_bsd_aarch64.inline.hpp:29:10: fatal error: 'ptrauth.h' file not found
#include <ptrauth.h>
^~~~~~~~~~~
1 error generated.
make[3]: *** [/Users/saint/repos/java/forks/jdk/build/macosx-aarch64-server-release/hotspot/variant-server/libjvm/objs/abstractInterpreter_aarch64.o] Error 1
m
To work around this, first build the JDK using Apple’s clang. Next, add brew’s LLVM installation to the PATH, then configure for hsdis. Finally, build hsdis:
# Warning: ensure /opt/homebrew/opt/llvm/bin is not in the PATH
cd ~/repos/java/forks/jdk
bash configure
make images
# Now add brew's LLVM to the PATH before running bash configure
export OLDPATH=$PATH
export PATH="/opt/homebrew/opt/llvm/bin:$PATH"
bash configure --with-hsdis=llvm
make build-hsdis
make install-hsdis
export PATH=$OLDPATH
# Why doesn't install-hsdis do this?
cd build/macosx-aarch64-server-release
cp support/hsdis/libhsdis.dylib jdk/bin/
Install the 64-bit WindowsLLVM. Configure the OpenJDK repo using both the --with-hsdis and LLVM_CONFIG options as shown. I needed to use the 8.3 path name (obtained using the command suggested on StackOverflow) for value of the LLVM_CONFIG parameter.
Unfortunately, this is not sufficient to enable building on Windows as detailed by this error:
$ make build-hsdis
Building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Creating support/hsdis/hsdis.dll from 1 file(s)
/usr/bin/bash: x86_64-w64-mingw32-g++: command not found
make[3]: *** [Hsdis.gmk:135: /..../build/windows-x86_64-server-release/support/hsdis/hsdis-llvm.obj] Error 127
make[2]: *** [make/Main.gmk:530: build-hsdis] Error 2
ERROR: Build failed for target 'build-hsdis' in configuration 'windows-x86_64-server-release' (exit code 2)
Jorn fixed this so we can add Jorn’s upstream JDK, fetch its commits, then cherry pick the commit with the fix.
git remote add jorn https://github.com/JornVernee/jdk/
git fetch jorn
git cherry-pick 8de8b763c9159f84bcc044c04ee2fac9f2390774
Some conflicts in make/Hsdis.gmk need to be resolved. This is straightforward since Jorn’s change splits the existing binutils Windows code into the first branch of an if-statement then adds support for the LLVM backend in the else case. The resolved conflicts are in my fork in the branch. The repo should now be configured with the additional --with-llvm option added by Jorn.
Running make build-hsdis results in errors about missing LLVM includes.
$ make build-hsdis
Building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Creating support/hsdis/hsdis.dll from 1 file(s)
d:\.....\jdk\src\utils\hsdis\llvm\hsdis-llvm.cpp(58): fatal error C1083: Cannot open include file: 'llvm-c/Disassembler.h': No such file or directory
make[3]: *** [Hsdis.gmk:142: /cygdrive/d/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis-llvm.obj] Error 1
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [make/Main.gmk:530: build-hsdis] Error 2
Let’s try setting CC and CXX then rerunning the above configure command.
configure: Will use user supplied compiler CC=C:/PROGRA~1/LLVM/bin/clang.exe
checking resolved symbolic links for CC... no symlink
configure: The C compiler (located as C:/PROGRA~1/LLVM/bin/clang.exe) does not seem to be the required microsoft compiler.
configure: The result from running it was: "clang: error: no input files"
configure: error: A microsoft compiler is required. Try setting --with-tools-dir.
configure exiting with result code 1
But let’s see what happens if we change the toolchain type to clang:
# This command does not work
bash configure --with-hsdis=llvm LLVM_CONFIG=C:/PROGRA~1/LLVM/bin --with-llvm=C:/PROGRA~1/LLVM --with-toolchain-type=clang
configure: Toolchain type clang is not valid on this platform.
configure: Valid toolchains: microsoft.
configure: error: Cannot continue.
configure exiting with result code 1
Indeed, clang is not a valid toolchain for Windows as declared in make/autoconf/toolchain.m4. Open question: how is the VALID_TOOLCHAIN_windows actually checked? So we can now unset the environment variables.
unset CC
unset CXX
This brought me back to the first thing I should have done when I saw the “No such file or directory” error – verifying that the file existed on disk! This is all there is there:
$ ls C:/PROGRA~1/LLVM/include/llvm-c
Remarks.h lto.h
Well, turns out this is the issue that led Jorn to build LLVM manually. I now know what the needed header files being referred to are. So let’s build LLVM using Jorn’s steps.
git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build_llvm
cd build_llvm
cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=X86" -D"CMAKE_BUILD_TYPE:STRING=Release" -D"CMAKE_INSTALL_PREFIX=install_local" -A x64 -T host=x64
cmake --build . --config Release --target install
The last command fails with the error below!??? Why can’t anything just simply work?
Building Opts.inc...
'..\..\RelWithDebInfo\bin\llvm-tblgen.exe' is not recognized as an internal or external command,
operable program or batch file.
C:\Program Files\Microsoft Visual Studio\2022\Preview\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(243,5): error MSB8066: Custom build for 'D:\dev\repos\llvm-project\build_llvm\CMakeFiles\dd1f7b42098
1667d7f617e96802947d3\Opts.inc.rule;D:\dev\repos\llvm-project\build_llvm\CMakeFiles\9fbf2dc5caba7f0c75934f43d12abdf5\RcOptsTableGen.rule;D:\dev\repos\llvm-project\llvm\tools\llvm-rc\CMakeLists.txt' exited wit
h code 9009. [D:\dev\repos\llvm-project\build_llvm\tools\llvm-rc\RcOptsTableGen.vcxproj]
Switch to my Surface Book 2 and LLVM builds just fine!
Interestingly, this fails with the same errors I saw on macOS:
$ make build-hsdis
Building target 'build-hsdis' in configuration 'windows-x86_64-server-release'
Creating support/hsdis/hsdis.dll from 1 file(s)
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMCreateDisasm referenced in function "public: __cdecl hsdis_backend::hsdis_backend(unsigned __int64,unsi...,char const *,int)" (??0hsdis_backend@@QEAA@_K0PEAE0P6APEAXPEAXPEBD2@Z2P6AH23ZZ23H@Z)
...
hsdis-llvm.obj : error LNK2019: unresolved external symbol LLVMInitializeX86Disassembler referenced in function LLVMInitializeNativeDisassembler
c:\dev\repos\java\forks\jdk\build\windows-x86_64-server-release\support\hsdis\hsdis.dll : fatal error LNK1120: 9 unresolved externals
make[3]: *** [Hsdis.gmk:142: /cygdrive/c/dev/repos/java/forks/jdk/build/windows-x86_64-server-release/support/hsdis/hsdis.dll] Error 1
The PATH environment variable probably needs to be adjusted to work around this.
Update 2022-02-08: the problem above is that bash configure is invoked with the wrong LLVM_CONFIG option – the actual llvm-config executable name is missing. See Troubleshooting hsdis LLVM backend MSVC Linker Errors for details.
To specify a backend for hsdis, the OpenJDK repo needs to be configured with the --with-hsdis option. As of commit 77757ba9, LLVM is not yet supported as an hsdis disassembly backend. Therefore, this error from make/autoconf/jdk-options.m4 is displayed. Here’s an example on the Windows platform:
$ bash configure --with-hsdis=llvm
...
checking what hsdis backend to use... invalid
configure: error: Incorrect hsdis backend "llvm"
configure exiting with result code 1
To test the LLVM backend for hsdis on macOS, install LLVM using brew (Apple’s LLVM does not have the llvm-c include files):
# install LLVM
brew install llvm
Now build the OpenJDK. This should use Apple’s compiler since we have not made any configuration changes.
cd ~/repos/java/jdk
bash configure
make images
Now add brew’s LLVM bin directory to the PATH and run bash configure again passing the --with-hsdis=llvm option as shown below. The configuration process will detect the clang++ compiler installed by brew and set it up for use when the build-hsdis target is executed.
# Now add brew's LLVM to the PATH before running bash configure
export OLDPATH=$PATH
export PATH="/opt/homebrew/opt/llvm/bin:$PATH"
bash configure --with-hsdis=llvm
make build-hsdis
make install-hsdis
export PATH=$OLDPATH
The install-hsdis target does not appear to be copying the hsdis library to the jdk/bin folder so these commands are required:
cd build/macosx-aarch64-server-release
cp support/hsdis/libhsdis.dylib jdk/bin/hsdis-aarch64.dylib
git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build_llvm
cd build_llvm
cmake ../llvm -D"LLVM_TARGETS_TO_BUILD:STRING=X86" -D"CMAKE_BUILD_TYPE:STRING=Release" -D"CMAKE_INSTALL_PREFIX=install_local" -A x64 -T host=x64
cmake --build . --config Release --target install
Now we can configure the OpenJDK repo for hsdis, and build both the JDK and hsdis.
bash configure --with-hsdis=llvm \
LLVM_CONFIG=C:/dev/repos/llvm-project/build_llvm/install_local/bin \
--with-llvm=C:/dev/repos/llvm-project/build_llvm/install_local/
make build-hsdis
make images
hsdis LLVM backend on Windows ARM64
Open question: is this supported?
Testing the hsdis LLVM backend
The String.checkIndex method of PR 5920 is a good candidate for testing the hsdis LLVM backend. The -XX:CompileCommand option can be used to print the generated assembler code after compilation of the specified method.
To view the command lines being executed by make as well as the value of the variables in use: make LOG=debug build-hsdis
Autoconf macros are defined using the AC_DEFUN macro. The JDKOPT_SETUP_HSDIS macro (modified by PR 5920) is defined using AC_DEFUN_ONCE, which is for macros that should only be called once.
To address warnings like ld: warning: dylib (/opt/homebrew/opt/llvm/lib/libunwind.dylib) was built for newer macOS version (12.0) than being linked (11.0) update MACOSX_VERSION_MIN in make/autoconf/flags.m4.
Last month I took on the task of writing a script to run the JDK JMH microbenchmarks. One of my coworkers already had a bash script for running a different set of benchmarks (GC), so I used it as a starting point. Having written mostly C# code in that past few years, I found the bash scripting environment unintuitive. Thankfully, there is a bash manual to refer to. I thought it might be worthwhile documenting a few of my lessons and surprises from that undertaking.
Bash Ignores +x File Mode
As a mostly-Windows user, I am accustomed to batch files (.bat) and PowerShell scripts (.ps1). These do not need any ceremony to declare them executable. I was using Windows 11 and testing my scripts in a bash environment in Windows Terminal. However, the scripts didn’t work when cloned onto my Linux and MacOS systems. That was when I observed that such scripts are not shown as executable by ls -l. However, bash runs them just fine in the Windows Terminal as shown below (note that this is the version of bash distributed with git-scm (and therefore based on MSYS)!
$ echo "uname -a" > test.sh; ls -l; ./test.sh
total 1
-rw-r--r-- 1 saint 1049089 9 Jan 8 15:43 test.sh
MSYS_NT-10.0-22000 SAINT 3.1.7-340.x86_64 2020-10-23 13:08 UTC x86_64 Msys
Interestingly, chmod cannot set the execute mode in the Windows Terminal.
$ chmod +x ./test.sh; ls -l
total 1
-rw-r--r-- 1 saint 1049089 9 Jan 8 15:43 test.sh
However, prefixing the script with #!/bin/bash makes ls -l show it as executable without any other changes.
$ echo -e '#!/bin/bash\nuname -a' > test.sh; ls -l
total 1
-rwxr-xr-x 1 saint 1049089 21 Jan 8 16:16 test.sh*
On MacOS (and Linux), the script cannot be executed without running chmod +x ./test.sh
% echo "uname -a" > test.sh; ls -l; ./test.sh
total 8
-rw-r--r-- 1 saint staff 9 Jan 8 15:49 test.sh
zsh: permission denied: ./test.sh
Also note that the #!/bin/bash prefix is required for command completion in the shell.
Echoing #!/bin/bash to a File Needs Single Quotes
Most of the script writing I did used double quotes (since they support variable interpolation, which I needed). I was testing the impact of not using the #!/bin/bash prefix in my scripts for the scenarios above when I ran into this strange error:
$ echo "#!/bin/bash" > temp.sh
bash: !/bin/bash: event not found
I had forgotten about the shell’s support for history expansion as pointed out in the solution here: the way around this is to use single quotes as in echo -e '#!/bin/bash\nuname -a'
… Bash Supports History Expansion
I played around with this history expansion feature to get a feel for it. I first renamed my .bash_history file then opened a new bash tab to start a new session with clean history.
uname -a
ls ~/.bash*
git --version
history
grep --version
which bash
!-2 # Shows grep version
!3 # Shows git version
!-5 # Shows history
!! # Also shows history
Echoing Newlines Needs a Flag
The -e flag option is required to write newline characters to the output (instead of the literal \ and n characters. As per “Simple Commands” in the GNU bash manual, if the -e option is given, interpretation of the following backslash-escaped characters is enabled… \a, \b, \c, \e, \E, \f, \n, \r, \t, \v, \\, etc.
Git Supports File Modes
After pulling some changes to my MacBook and making some edits, I saw the diff below. I have never really had to pay attention to file modes, so this was intriguing.
diff --git a/java/jmh/setup_jmh_jdk_micros.sh b/java/jmh/setup_jmh_jdk_micros.sh
old mode 100644
new mode 100755
The difference is the execute bit is now on for all 3 user types. Apparently, 644 and 755 are the only file modes supported by git and the core.fileMode option, which tells Git if the executable bit of files in the working tree is to be honored, should be set to false if the filesystem a file is checked out on loses the executable bit on checkout. Open questions: does NTFS lose executable bits, i.e. does this property apply to NTFS? Is there a way to see the filemode in GitHub?
[ is an Executable!
Not even sure what to say about this one :D.
$ which [
/usr/bin/[
In Cygwin, we can get the path to the executable!
$ cygpath -w `which [`
C:\dev\cygwin64\bin\[.exe
Open questions: is this program being invoked for if-statements?
Miscellaneous Facts
These are observations are documented the Bash Manual but listed here since they’re things I learned along the way.
To view the full path of files in the current directory, use ls -d $PWD/*
To get rid of ^M (carriage-return characters) in a text file from Windows, open the file in vim then these commands: :e ++ff=dos then :set ff=unix then finally exit vim with :wq
Functions cannot return any value, they return codes. Specifically, (as per the GNU bash manual’s Shell Functions section), if a numeric argument is given to return, that is the function’s return status; otherwise the function’s return status is the exit status of the last command executed before the return.
~= is a binary operator used for regular expression matching. The string to the right of the operator is considered a POSIX extended regular expression. The return value is 0 if the string matches the pattern, and 1 otherwise. If the regular expression is syntactically incorrect, the conditional expression’s return value is 2. See the Conditional Constructs section of the GNU bash manual.
See Windows Terminal tips and tricks for various useful customizations of the Windows Terminal. Changing tab colors is something I wouldn’t have thought to do but now makes it very easy to know which tabs are relevant for the task at hand.
The previous post on Building HSDIS in Cygwin required running this command to actually build the hsdis DLL.
make OS=Linux MINGW=x86_64-w64-mingw32 BINUTILS=~/binutils-2.37
As it turns out, this make command fails because of a bug in the GNU binutils source code. This is the error I got:
...
x86_64-w64-mingw32-gcc -c -DHAVE_CONFIG_H -O -I. -I/home/User/binutils-2.37/libiberty/../include -W -Wall -Wwrite-strings -Wc++-compat -Wstrict-prototypes -Wshadow=local -pedantic -D_GNU_SOURCE /home/User/binutils-2.37/libiberty/rust-demangle.c -o rust-demangle.o
/home/User/binutils-2.37/libiberty/rust-demangle.c:78:3: error: unknown type name ‘uint’
78 | uint recursion;
| ^~~~
/home/User/binutils-2.37/libiberty/rust-demangle.c: In function ‘demangle_path’:
/home/User/binutils-2.37/libiberty/rust-demangle.c:81:37: error: ‘uint’ undeclared (first use in this function); did you mean ‘int’?
81 | #define RUST_NO_RECURSION_LIMIT ((uint) -1)
| ^~~~
...
make[2]: *** [Makefile:1230: rust-demangle.o] Error 1
...
At this point, I wasn’t sure which version I used to build successfully. Searching for that error (and binutils to narrow things down) led to this bug in the sourceware.org Bugzilla that appears to be the exact bug I ran into: 28207 – error: unknown type name ‘uint’ (78 | uint recursion;) avr-gcc mingw32 Windows Build (sourceware.org). Fortunately, one Alan helpfully points out that this bug fixed on the binutils-2.37 branch with commit 999566402e3.
I’m used to GitHub where looking at the repo structure implies that you’re at a URL you can copy and trim to clone. In this other web view, the URL to clone is listed above the shortlog:
At this point, it makes sense to verify that the 2.37 sources I downloaded actually contain the bug. Observe that:
the tags section contains a binutils-2_37 tag described as “Official GNU Binutils 2.37 Release” and committed on Sun, 18 Jul 2021 16:46:54 +0000 (17:46 +0100).
the fix for the build error shows a fix committed by Alan on Mon, 19 Jul 2021 11:32:21 +0000 (21:02 +0930)
Therefore, using binutils older than 2.37 should work just fine. However, it may still be necessary to run “rm -fr build” in the hsdis folder to enable 2.36 to be picked up when you run make (otherwise 2.37 is still baked into some of configure’s output).
Hsdis is an externally loadable disassembler plugin. It lets you see which assembly instructions the JVM generates for your Java code. On 64-bit Windows, it is a binary called hsdis-amd64.dll (and hsdis-i386.dll on 32-bit platforms). This binary needs to be in the same directory as jvm.dll. Some good resources out there on building the hsdis binary for the OpenJDK include:
For Cygwin, the latter resource (from 2012?) is all we need. I like that Gunnar’s blog post covered how to use hsdis after building it so this writeup aims to combine both blogs into a simple Cygwin install-build-disassemble set of instructions.
Building hsdis for 64-bit JVMs
Install Cygwin with the gcc-core, make, and mingw64-x86_64-gcc-core packages by launching the setup executable using this command (no need to bother selecting packages in the UI since you have already specified them on the command line)
setup-x86_64.exe -P gcc-core -P mingw64-x86_64-gcc-core -P make
mkdir ~/repos
cd ~/repos
git clone https://github.com/openjdk/jdk
Run these commands to download GNU binutils and build hsdis (Update 2022-01-07: version downgraded to 2.36 to avoid build failures investigated in Fixing Hsdis Compile Failure in GNU binutils).
cd ~
curl -Lo binutils-2.36.tar.gz https://ftp.gnu.org/gnu/binutils/binutils-2.36.tar.gz
tar xvf binutils-2.36.tar.gz
cd ~/repos/jdk/src/utils/hsdis
make OS=Linux MINGW=x86_64-w64-mingw32 BINUTILS=~/binutils-2.36
Copy the hsdis binary to the locally built java bin folder
I have created a basic substitution cipher, which we can compile and disassemble using the commands below. Note that these commands save the .java file to a temp folder to make cleanup much easier. Also note the redirection to a file since the output can be voluminous.
cd build/windows-x86_64-server-release/jdk/bin
mkdir -p temp
cd temp
curl -Lo BasicSubstitutionCipher.java https://raw.githubusercontent.com/swesonga/scratchpad/main/apps/crypto/substitution-cipher/BasicSubstitutionCipher.java
../javac BasicSubstitutionCipher.java
../java -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -XX:+LogCompilation BasicSubstitutionCipher > BasicSubstitutionCipher.disassembled.txt
Once the disassembly completes, we can view the instructions generated in the BasicSubstitutionCipher.disassembled.txt file.
One open question in this setup is why the installed GNU binutils cannot be used to build hsdis. Seems strange to have to build them from source when the binutils Cygwin package was also installed in step 1 above.
$ zip -usd ziphelp.zip *.txt
sd: Zipfile name 'ziphelp.zip'
sd: Command line read
sd: Reading archive
zip warning: expected 3 entries but found 0
zip error: Zip file structure invalid (ziphelp.zip)
Add debug = 1 to the top of win32\makefile.w32 then run these commands to generate PDBs along with the other binaries.
set LOCAL_ZIP=-DDEBUG
nmake -f win32\makefile.w32
Investigating the Failure
A quick way to set up Visual Studio to debug this issue is to create a new C++ console application then change the command, command arguments, and working directory of the project as shown below.
Setting Visual C++ Project Properties for Debugging Zip.exe
Pressing F10 should now start debugging the zip binary. The highlights are:
The question now is why scanzipf_regnew does not find any entries since there are 3 PK12 and 3 PK34 signatures in the file. Let us inspect the file offsets right after the code seeks to in_cd_start_offset (from step 5 above) and immediately before looking for the next signature to process.
This shows a value of 0x0000000000002168 which is not the proper offset in the zip file to seek to. Recall that in_cd_start_offset was 0x218e, which is the second-last byte of line 537 in the zipfile’s hexdump output. Could this be an error in the standard library fseek and ftell functions? TODO: why does the scan fail from an earlier starting point?
Pressing F12 to see the definitions of zftello and zfseeko went to the wrong place! These are not the standard library functions being used. Visual Studio was opening their definitions in tailor.h instead of the actual implementations being called. Turns out zftello and zfseeko are functions implemented in win32/win32i64.c. These comments above zftello raise some huge red flags.
/* 64-bit buffered ftello
*
* Win32 does not provide a 64-bit buffered
* ftell (in the published api anyway) so below provides
* hopefully close version.
* We have not gotten _telli64 to work with buffered
* streams. Below cheats by using fgetpos improperly and
* may not work on other ports.
*/
/* 64-bit buffered fseeko
*
* Win32 does not provide a 64-bit buffered
* fseeko so use _lseeki64 and fflush. Note
* that SEEK_CUR can lose track of location
* if fflush is done between the last buffered
* io and this call.
*/
Looks like this custom seek/tell code is responsible for the incorrect offsets into the zip file! We can work around this by simply removing these custom implementations.
Windows Specific Bug?
So why didn’t Cygwin’s zip.exe have this issue? Running zip.exe -v shows this compiler/OS description:
Compiled with gcc 4.8.3 for Unix (Cygwin) on Jun 23 2014.
The Cygwin OS name in parenthesis is defined in unix/unix.c only if __CYGWIN__ is defined. However, under this condition, the custom zftello and zfseeko implementations will not be included in the zip sources being compiled! Therefore, the issue does not occur in Cygwin’s distributed zip binary.
After figuring out how to build the Info-ZIP sources, I had a few commands to test the zip file by creating a few text files to zip.
zip -h > help.txt
zip -h2 > help2.txt
zip -L > license.txt
Unfortunately, the zip -qru ./files.zip -i *.txtcommand from the OpenJDK is not what we need. To actually create a zip file, use only the -u flag
zip -u files.zip *.txt
To test that the files were zipped successfully, unzip the files and compare them to the original files. Here’s the whole script for this:
echo "---Creating temp directory---"
mkdir temp; cd temp
echo "---Creating text files---"
zip -h > help.txt
zip -h2 > help2.txt
zip -L > license.txt
echo "---Adding text files to a new repo---"
git init
git add *.txt
git commit -m "Add original text files"
echo "---Zipping text files---"
zip -u files.zip *.txt
echo "---Removing text files---"
rm *.txt
echo "---Unzipping text files---"
unzip files.zip
echo "---Checking unzipped files---"
git diff
When using the zip binary for Windows, something strange happens when running the zip command a 2nd time:
$ zip -u files.zip *.txt
zip warning: files.zip not found or empty
adding: help.txt (176 bytes security) (deflated 49%)
adding: help2.txt (176 bytes security) (deflated 62%)
adding: license.txt (176 bytes security) (deflated 54%)
$ zip -u files.zip *.txt
zip warning: expected 3 entries but found 0
zip error: Zip file structure invalid (files.zip)
Info-ZIP supports a -sd flag that shows diagnostic information while it runs. It reveals that something is going wrong when reading the archive.
$ zip -usd files.zip *.txt
sd: Zipfile name 'files.zip'
sd: Command line read
sd: Reading archive
zip warning: expected 3 entries but found 0
zip error: Zip file structure invalid (files.zip)
In the last post, I described how to build the Info-ZIP sources. When using the resulting zip binaries in Cygwin, some important path handling issues come up. The paths passed to the zip binary when building the OpenJDK in Cygwin use forward slashes. The Cygwin User’s Guide has a section on File Access that outlines the support for POSIX and Win32-style file paths.
The Windows file system APIs support forward slashes in file paths. The zip source code uses the fopen CRT function, which eventually ends up calling CreateFileW. The CreateFileW docs state that you may use either forward slashes (/) or backslashes (\) in the lpFileName parameter. The translation of paths from Win32 to NT happens in a function called RtlDosPathNameToRelativeNtPathName_U as discussed in the Definitive Guide on Win32 to NT Path Conversion. Since this is a built-in Windows function, it does not support the /cygdrive/ style prefixes. Running the simple test program argtofile in Cygwin easily demonstrates this.
The /cygdrive/ prefixes will therefore not work for programs compiled for Windows (such as the zip binary directly compiled using Visual C++). Therefore, the cygpath command is necessary to translate these paths to Win32-style file paths. To peek into how cygpath works, we can take advantage of the fact that the source code for the cygpath utility is available online. I found it easier to browse the sources after cloning the repo:
The scenario of interest is what happens when cygpath -u ~ is invoked. In this case, we want to see how the “/cygdrive/” string is prefixed to the computed path.
normalizes the path by calling normalize_win32_path
before finally iterating through the mount items to find the path’s prefix in the mount table.
Also searching for the cygpath \s*( regex leads to the vcygpath function in winsup/utils/path.cc. That appears to be more directly related to the cygpath command (how?). Searching for the \"cygdrive\" regex also reveals that this is a magic string used in many places in the codebase.
All this shows that there is indeed some complexity behind maintaining the POSIX/Win32-style file path mapping in Cygwin but it should be possible to add some basic logic to the Windows Info-ZIP build to handle /cygdrive/ prefixes in its file arguments. The question I have at this point is how does compiling the zip binaries for the Cygwin environment (the shipping configuration) result in proper handling of POSIX-style filenames?
curl -Lo zip30.tar.gz https://sourceforge.net/projects/infozip/files/Zip%203.x%20%28latest%29/3.0/zip30.tar.gz/download
tar xvf zip30.tar.gz
cd ./zip30
git init; git add *; git commit -m "Commit original Info-ZIP sources"
Now that we have the sources, let’s see how to build them. The scenario I’m working on is Windows specific so we need Visual Studio 2019 with the Desktop Development with C++ workload installed. I’ll be building a 32-bit zip executable. Launch the x86 Native Tools Command Prompt for VS 2019 and change to the zip30 source directory to start building. Some digging around reveals a makefile with build instructions (that seem one directory off). Here’s the command to build a 32-bit executable from the sources (note that building fails due to various errors that need to be addressed):
nmake -f win32\makefile.w32
Carriage Return (CR) Name Collisions
The first error is this rather cryptic mess of syntax errors:
Microsoft (R) Program Maintenance Utility Version 14.29.30133.0
Copyright (C) Microsoft Corporation. All rights reserved.
cl -nologo -c -W3 -O2 -DWIN32 -DASM_CRC -ML zip.c
cl : Command line warning D9002 : ignoring unknown option '-ML'
zip.c
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18822): error C2143: syntax error: missing ':' before 'constant'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18822): error C2143: syntax error: missing ';' before ':'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18822): error C2059: syntax error: ':'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18823): error C2143: syntax error: missing '{' before ':'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18823): error C2059: syntax error: ':'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18824): error C2059: syntax error: '}'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18825): error C2059: syntax error: '}'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18826): error C2059: syntax error: '}'
zip.c(5746): warning C4267: '=': conversion from 'size_t' to 'ush', possible loss of data
zip.c(5838): warning C4267: '=': conversion from 'size_t' to 'ush', possible loss of data
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86\cl.EXE"' : return code '0x2'
Stop.
Turns out this option was removed in Visual Studio 2010 as per the Microsoft C/C++ change history since the linker no longer supports optimizing for Windows 98. This is clearly a safe flag to remove from the linker flags in win32\makefile.w32.
Update the Branding
Change the VERSION string from “3.0” to “3.0-ioHardenedZIP”
Update the REVDATE from “July 5th 2008” to the current date (“December 18th 2021” in my case)
Update the about text to indicate that it is a custom build.
Testing the Zip Build
The sources should now build successfully in the x86 Native Tools Command Prompt for VS 2019. The OpenJDK build uses the -qru flags for creating zip files so we can easily test the zip executable by creating a zip of the Info-ZIP help and license text.
zip -h > help.txt
zip -h2 > help2.txt
zip -L > license.txt
zip -qru ./files.zip -i *.txt
We need to verify whether the zip was correctly created. Saving this for another day.
I documented how to set up an OpenJDK build environment for macOS and for Ubuntu Linux. Here is how to do the same for a Windows x86-64 environment on a Windows x86-64 machine. To create a JDK build for a Windows ARM64 environment, see the last section of this post.
setup-x86_64.exe -q -P autoconf -P make -P unzip -P zip
Launch Cygwin, clone the OpenJDK repo then run bash configure. This should output an error if there are any missing dependencies. Once that completes successfully, make images will build the OpenJDK code.
mkdir ~/repos
cd ~/repos
git clone https://github.com/openjdk/jdk
cd ~/repos/jdk
bash configure
make images
To try out the local JDK build, run java.exe in the build folder, e.g.
cd ~/repos/jdk/build/windows-x86_64-server-slowdebug
cd jdk/bin
./java.exe -version
To create a JDK build for a Windows ARM64 machine (as of this posting), you still need to set up the Windows x86-64 environment as described above with the additional changes below.
Launch the Visual Studio Installer then install the “MSVC v142 – VS 2019 C++ ARM64 build tools (Latest)” item.
In the Windows command line we get this message in the terminal and the subsequent dialog box:
C:\dev\repos\jdk\build\windows-aarch64-server-release\jdk\bin>.\java.exe
This version of C:\dev\repos\jdk\build\windows-aarch64-server-release\jdk\bin\java.exe is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher.
Machine Type Mismatch
Launching it from Windows Explorer fails with this error:
Platform mismatch launching ARM64 Java on Windows x86-64
