Categories: Bash

Bash Scripting Newbie Notes

Last month I took on the task of writing a script to run the JDK JMH microbenchmarks. One of my coworkers already had a bash script for running a different set of benchmarks (GC), so I used it as a starting point. Having written mostly C# code in that past few years, I found the bash scripting environment unintuitive. Thankfully, there is a bash manual to refer to. I thought it might be worthwhile documenting a few of my lessons and surprises from that undertaking.

Bash Ignores +x File Mode

As a mostly-Windows user, I am accustomed to batch files (.bat) and PowerShell scripts (.ps1). These do not need any ceremony to declare them executable. I was using Windows 11 and testing my scripts in a bash environment in Windows Terminal. However, the scripts didn’t work when cloned onto my Linux and MacOS systems. That was when I observed that such scripts are not shown as executable by ls -l. However, bash runs them just fine in the Windows Terminal as shown below (note that this is the version of bash distributed with git-scm (and therefore based on MSYS)!

$ echo "uname -a" > test.sh; ls -l; ./test.sh
total 1
-rw-r--r-- 1 saint 1049089 9 Jan  8 15:43 test.sh
MSYS_NT-10.0-22000 SAINT 3.1.7-340.x86_64 2020-10-23 13:08 UTC x86_64 Msys

Interestingly, chmod cannot set the execute mode in the Windows Terminal.

$ chmod +x ./test.sh; ls -l
total 1
-rw-r--r-- 1 saint 1049089 9 Jan  8 15:43 test.sh

However, prefixing the script with #!/bin/bash makes ls -l show it as executable without any other changes.

$ echo -e '#!/bin/bash\nuname -a' > test.sh; ls -l
total 1
-rwxr-xr-x 1 saint 1049089 21 Jan  8 16:16 test.sh*

On MacOS (and Linux), the script cannot be executed without running chmod +x ./test.sh

% echo "uname -a" > test.sh; ls -l; ./test.sh
total 8
-rw-r--r--  1 saint  staff  9 Jan  8 15:49 test.sh
zsh: permission denied: ./test.sh

Also note that the #!/bin/bash prefix is required for command completion in the shell.

Echoing #!/bin/bash to a File Needs Single Quotes

Most of the script writing I did used double quotes (since they support variable interpolation, which I needed). I was testing the impact of not using the #!/bin/bash prefix in my scripts for the scenarios above when I ran into this strange error:

$ echo "#!/bin/bash" > temp.sh
bash: !/bin/bash: event not found

I had forgotten about the shell’s support for history expansion as pointed out in the solution here: the way around this is to use single quotes as in echo -e '#!/bin/bash\nuname -a'

… Bash Supports History Expansion

I played around with this history expansion feature to get a feel for it. I first renamed my .bash_history file then opened a new bash tab to start a new session with clean history.

uname -a
ls ~/.bash*
git --version
history
grep --version
which bash
!-2 # Shows grep version
!3  # Shows git version
!-5 # Shows history
!!  # Also shows history

Echoing Newlines Needs a Flag

The -e flag option is required to write newline characters to the output (instead of the literal \ and n characters. As per “Simple Commands” in the GNU bash manual, if the -e option is given, interpretation of the following backslash-escaped characters is enabled… \a, \b, \c, \e, \E, \f, \n, \r, \t, \v, \\, etc.

Git Supports File Modes

After pulling some changes to my MacBook and making some edits, I saw the diff below. I have never really had to pay attention to file modes, so this was intriguing.

diff --git a/java/jmh/setup_jmh_jdk_micros.sh b/java/jmh/setup_jmh_jdk_micros.sh
old mode 100644
new mode 100755

The difference is the execute bit is now on for all 3 user types. Apparently, 644 and 755 are the only file modes supported by git and the core.fileMode option, which tells Git if the executable bit of files in the working tree is to be honored, should be set to false if the filesystem a file is checked out on loses the executable bit on checkout. Open questions: does NTFS lose executable bits, i.e. does this property apply to NTFS? Is there a way to see the filemode in GitHub?

[ is an Executable!

Not even sure what to say about this one :D.

$ which [
/usr/bin/[

In Cygwin, we can get the path to the executable!

$ cygpath -w `which [`
C:\dev\cygwin64\bin\[.exe

Open questions: is this program being invoked for if-statements?

Miscellaneous Facts

These are observations are documented the Bash Manual but listed here since they’re things I learned along the way.

  1. To view the full path of files in the current directory, use ls -d $PWD/*
  2. To get rid of ^M (carriage-return characters) in a text file from Windows, open the file in vim then these commands: :e ++ff=dos then :set ff=unix then finally exit vim with :wq
  3. $_ expands to the number of positional parameters. See Shell Parameters and Special Parameters.
  4. $@ expands to the positional parameters, starting from one. It can be used to create an array of the parameters.
  5. Argument numbering (aka positional parameters) is local to functions. In other words, when a function is executed, the arguments to the function become the positional parameters during its execution.
  6. Functions cannot return any value, they return codes. Specifically, (as per the GNU bash manual’s Shell Functions section), if a numeric argument is given to return, that is the function’s return status; otherwise the function’s return status is the exit status of the last command executed before the return.
  7. ~= is a binary operator used for regular expression matching. The string to the right of the operator is considered a POSIX extended regular expression. The return value is 0 if the string matches the pattern, and 1 otherwise. If the regular expression is syntactically incorrect, the conditional expression’s return value is 2. See the Conditional Constructs section of the GNU bash manual.

Open questions: how should the pattern in the JMH script be assigned to a variable?

Hints

See Windows Terminal tips and tricks for various useful customizations of the Windows Terminal. Changing tab colors is something I wouldn’t have thought to do but now makes it very easy to know which tabs are relevant for the task at hand.


Categories: Assembly, Compilers

Fixing Hsdis Compile Failure in GNU binutils

The previous post on Building HSDIS in Cygwin required running this command to actually build the hsdis DLL.

make OS=Linux MINGW=x86_64-w64-mingw32 BINUTILS=~/binutils-2.37

As it turns out, this make command fails because of a bug in the GNU binutils source code. This is the error I got:

...
x86_64-w64-mingw32-gcc -c -DHAVE_CONFIG_H -O    -I. -I/home/User/binutils-2.37/libiberty/../include  -W -Wall -Wwrite-strings -Wc++-compat -Wstrict-prototypes -Wshadow=local -pedantic  -D_GNU_SOURCE  /home/User/binutils-2.37/libiberty/rust-demangle.c -o rust-demangle.o
/home/User/binutils-2.37/libiberty/rust-demangle.c:78:3: error: unknown type name ‘uint’
   78 |   uint recursion;
      |   ^~~~
/home/User/binutils-2.37/libiberty/rust-demangle.c: In function ‘demangle_path’:
/home/User/binutils-2.37/libiberty/rust-demangle.c:81:37: error: ‘uint’ undeclared (first use in this function); did you mean ‘int’?
   81 | #define RUST_NO_RECURSION_LIMIT   ((uint) -1)
      |                                     ^~~~
...
make[2]: *** [Makefile:1230: rust-demangle.o] Error 1
...

At this point, I wasn’t sure which version I used to build successfully. Searching for that error (and binutils to narrow things down) led to this bug in the sourceware.org Bugzilla that appears to be the exact bug I ran into: 28207 – error: unknown type name ‘uint’ (78 | uint recursion;) avr-gcc mingw32 Windows Build (sourceware.org). Fortunately, one Alan helpfully points out that this bug fixed on the binutils-2.37 branch with commit 999566402e3.

To figure out where the binutils git repo is, I click on the Browse button in Bugzilla then navigate to the binutils product category, which has a link to the list of bugs for the binutils component. A re-opened bug seems likely to have a link to some commits. I select 26865 – windres: –preprocessor option won’t respect space in file path (sourceware.org) and sure enough, there is a link to a commit on the binutils repo. We can now view the history of rust-demangle.c. To find the commit in question, click on any commitdiff to get the URL format then replace the hash in the URL with 999566402e3 to reveal the aforementioned fix for the unknown type name uint error.

Cloning binutils Repo

I’m used to GitHub where looking at the repo structure implies that you’re at a URL you can copy and trim to clone. In this other web view, the URL to clone is listed above the shortlog:

git clone https://sourceware.org/git/binutils-gdb.git

Tracing the Bug

At this point, it makes sense to verify that the 2.37 sources I downloaded actually contain the bug. Observe that:

  1. the tags section contains a binutils-2_37 tag described as “Official GNU Binutils 2.37 Release” and committed on Sun, 18 Jul 2021 16:46:54 +0000 (17:46 +0100).
  2. the fix for the build error shows a fix committed by Alan on Mon, 19 Jul 2021 11:32:21 +0000 (21:02 +0930)
  3. the bug fix that introduced the error was committed on Thu, 15 Jul 2021 15:51:56 +0000 (16:51 +0100)

Therefore, using binutils older than 2.37 should work just fine. However, it may still be necessary to run “rm -fr build” in the hsdis folder to enable 2.36 to be picked up when you run make (otherwise 2.37 is still baked into some of configure’s output).


Categories: Assembly, Cygwin

Building HSDIS in Cygwin

Hsdis is an externally loadable disassembler plugin. It lets you see which assembly instructions the JVM generates for your Java code. On 64-bit Windows, it is a binary called hsdis-amd64.dll (and hsdis-i386.dll on 32-bit platforms). This binary needs to be in the same directory as jvm.dll. Some good resources out there on building the hsdis binary for the OpenJDK include:

For Cygwin, the latter resource (from 2012?) is all we need. I like that Gunnar’s blog post covered how to use hsdis after building it so this writeup aims to combine both blogs into a simple Cygwin install-build-disassemble set of instructions.

Building hsdis for 64-bit JVMs

  1. Install Cygwin with the gcc-core, make, and mingw64-x86_64-gcc-core packages by launching the setup executable using this command (no need to bother selecting packages in the UI since you have already specified them on the command line)
setup-x86_64.exe -P gcc-core -P mingw64-x86_64-gcc-core -P make
  1. Launch the Cygwin64 terminal
  2. Clone the OpenJDK repo to get the hsdis sources (if you have not yet set up a Windows OpenJDK Development Environment).
mkdir ~/repos
cd ~/repos
git clone https://github.com/openjdk/jdk
  1. Run these commands to download GNU binutils and build hsdis (Update 2022-01-07: version downgraded to 2.36 to avoid build failures investigated in Fixing Hsdis Compile Failure in GNU binutils).
cd ~
curl -Lo binutils-2.36.tar.gz https://ftp.gnu.org/gnu/binutils/binutils-2.36.tar.gz
tar xvf binutils-2.36.tar.gz

cd ~/repos/jdk/src/utils/hsdis
make OS=Linux MINGW=x86_64-w64-mingw32 BINUTILS=~/binutils-2.36
  1. Copy the hsdis binary to the locally built java bin folder
cp src/utils/hsdis/build/Linux-amd64/hsdis-amd64.dll build/windows-x86_64-server-release/jdk/bin/

Testing hsdis

I have created a basic substitution cipher, which we can compile and disassemble using the commands below. Note that these commands save the .java file to a temp folder to make cleanup much easier. Also note the redirection to a file since the output can be voluminous.

cd build/windows-x86_64-server-release/jdk/bin
mkdir -p temp
cd temp

curl -Lo BasicSubstitutionCipher.java https://raw.githubusercontent.com/swesonga/scratchpad/main/apps/crypto/substitution-cipher/BasicSubstitutionCipher.java

../javac BasicSubstitutionCipher.java

../java -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -XX:+LogCompilation BasicSubstitutionCipher > BasicSubstitutionCipher.disassembled.txt

Once the disassembly completes, we can view the instructions generated in the BasicSubstitutionCipher.disassembled.txt file.

One open question in this setup is why the installed GNU binutils cannot be used to build hsdis. Seems strange to have to build them from source when the binutils Cygwin package was also installed in step 1 above.


Categories: C, ZIP

Debugging Info-ZIP

We have seen that the Windows Info-ZIP build fails when invoked twice. I have created a sample zip file for debugging this issue along with a hexdump of it (I was glad to see that Cygwin includes hexdump). We observed that using the -sd flag showed a failure when reading the archive.

$ zip -usd ziphelp.zip *.txt
sd: Zipfile name 'ziphelp.zip'
sd: Command line read
sd: Reading archive
        zip warning: expected 3 entries but found 0

zip error: Zip file structure invalid (ziphelp.zip)

The sd: Readin archive line is output by the main function in zip.c. Since there is no other output prefixed by sd, execution must be continuing past the subsequent if-statement and into the readzipfile() function on line 4009.

Generate a Debug Zip Binary

Add debug = 1 to the top of win32\makefile.w32 then run these commands to generate PDBs along with the other binaries.

set LOCAL_ZIP=-DDEBUG
nmake -f win32\makefile.w32

Investigating the Failure

A quick way to set up Visual Studio to debug this issue is to create a new C++ console application then change the command, command arguments, and working directory of the project as shown below.

Setting Visual C++ Project Properties for Debugging Zip.exe

Pressing F10 should now start debugging the zip binary. The highlights are:

  1. The main function in zip.c calls readzipfile
  2. readzipfile calls scanzipf_regnew which computes an offset to start looking for the End of Central Directory Record (EOCD) in the zip file by trying to seek 128KiB from the end of the file. In our case, the zip file is less than 128 KiB so scanzipf_regnew seeks to the beginning of the file.
  3. scanzipf_regnew scans for the EOCD record signature (PK56 or hex bytes 50 4b 05 06). There is only 1 EOCD signature in the zip file and it is at offset 0x227C. It then (correctly) computes the offset of the start of the data after the EOCDR signature (0x2280) and seeks to that offset in the file.
  4. 18 bytes are now read from the EOCDR.
  5. The 32-bit offset of the start of the central directory, relative to start of archive is then read from the record and stored in the in_cd_start_offset variable. This is correctly read as 0x218e (the last 4 bytes on line 553 of the zip’s hexdump output).
  6. The number of entries is also read from the EOCDR (which is how zip knows to expect 3 entries).
  7. scanzipf_regnew now seeks to the first CD entry
  8. then looks for the next signature to process. Interestingly, it finds the EOCDR signature (PK56) and breaks out of the loops (signature scan and zipfile disks). Since it did not find any entries, it then displays the error about the expected number of entries vs the number found and returns an error code to the main function in zip.c, which in turn calls ziperr to print an error message and terminate the process.

The question now is why scanzipf_regnew does not find any entries since there are 3 PK12 and 3 PK34 signatures in the file. Let us inspect the file offsets right after the code seeks to in_cd_start_offset (from step 5 above) and immediately before looking for the next signature to process.

 uzoff_t zipoffset_before_finding_signatures =  zftello(in_file);

This shows a value of 0x0000000000002168 which is not the proper offset in the zip file to seek to. Recall that in_cd_start_offset was 0x218e, which is the second-last byte of line 537 in the zipfile’s hexdump output. Could this be an error in the standard library fseek and ftell functions? TODO: why does the scan fail from an earlier starting point?

Pressing F12 to see the definitions of zftello and zfseeko went to the wrong place! These are not the standard library functions being used. Visual Studio was opening their definitions in tailor.h instead of the actual implementations being called. Turns out zftello and zfseeko are functions implemented in win32/win32i64.c. These comments above zftello raise some huge red flags.

/* 64-bit buffered ftello
 *
 * Win32 does not provide a 64-bit buffered
 * ftell (in the published api anyway) so below provides
 * hopefully close version.
 * We have not gotten _telli64 to work with buffered
 * streams.  Below cheats by using fgetpos improperly and
 * may not work on other ports.
 */

zftello’s comments do not sound any more reassuring:

/* 64-bit buffered fseeko
 *
 * Win32 does not provide a 64-bit buffered
 * fseeko so use _lseeki64 and fflush.  Note
 * that SEEK_CUR can lose track of location
 * if fflush is done between the last buffered
 * io and this call.
 */

Looks like this custom seek/tell code is responsible for the incorrect offsets into the zip file! We can work around this by simply removing these custom implementations.

Windows Specific Bug?

So why didn’t Cygwin’s zip.exe have this issue? Running zip.exe -v shows this compiler/OS description:

Compiled with gcc 4.8.3 for Unix (Cygwin) on Jun 23 2014.

The Cygwin OS name in parenthesis is defined in unix/unix.c only if __CYGWIN__ is defined. However, under this condition, the custom zftello and zfseeko implementations will not be included in the zip sources being compiled! Therefore, the issue does not occur in Cygwin’s distributed zip binary.


Categories: C, Cygwin

Testing Info-ZIP

After figuring out how to build the Info-ZIP sources, I had a few commands to test the zip file by creating a few text files to zip.

zip -h > help.txt
zip -h2 > help2.txt
zip -L > license.txt

Unfortunately, the zip -qru ./files.zip -i *.txt command from the OpenJDK is not what we need. To actually create a zip file, use only the -u flag

zip -u files.zip *.txt

To test that the files were zipped successfully, unzip the files and compare them to the original files. Here’s the whole script for this:

echo "---Creating temp directory---"
mkdir temp; cd temp

echo "---Creating text files---"
zip -h > help.txt
zip -h2 > help2.txt
zip -L > license.txt

echo "---Adding text files to a new repo---"
git init
git add *.txt
git commit -m "Add original text files"

echo "---Zipping text files---"
zip -u files.zip *.txt

echo "---Removing text files---"
rm *.txt

echo "---Unzipping text files---"
unzip files.zip

echo "---Checking unzipped files---"
git diff

When using the zip binary for Windows, something strange happens when running the zip command a 2nd time:

$ zip -u files.zip *.txt
        zip warning: files.zip not found or empty
  adding: help.txt (176 bytes security) (deflated 49%)
  adding: help2.txt (176 bytes security) (deflated 62%)
  adding: license.txt (176 bytes security) (deflated 54%)

$ zip -u files.zip *.txt
        zip warning: expected 3 entries but found 0

zip error: Zip file structure invalid (files.zip)

Info-ZIP supports a -sd flag that shows diagnostic information while it runs. It reveals that something is going wrong when reading the archive.

$ zip -usd files.zip *.txt
sd: Zipfile name 'files.zip'
sd: Command line read
sd: Reading archive
        zip warning: expected 3 entries but found 0

zip error: Zip file structure invalid (files.zip)

This is filed as Running zip twice fails with invalid file structure error · Issue #35 · swesonga/Info-ZIP (github.com). Observe that this doesn’t happen when using the original zip binary that shipped with Cygwin:

$ zip.original -usd files.zip *.txt
sd: Zipfile name 'files.zip'
sd: Command line read
sd: Reading archive
sd: Scanning files
sd: Applying filters
sd: Checking dups
sd: Scanning files to update
sd: fcount = 0

Why is this bug only in the Windows build?


Categories: C, Cygwin, Windows

Windows vs Cygwin File Paths

In the last post, I described how to build the Info-ZIP sources. When using the resulting zip binaries in Cygwin, some important path handling issues come up. The paths passed to the zip binary when building the OpenJDK in Cygwin use forward slashes. The Cygwin User’s Guide has a section on File Access that outlines the support for POSIX and Win32-style file paths.

The Windows file system APIs support forward slashes in file paths. The zip source code uses the fopen CRT function, which eventually ends up calling CreateFileW. The CreateFileW docs state that you may use either forward slashes (/) or backslashes (\) in the lpFileName parameter. The translation of paths from Win32 to NT happens in a function called RtlDosPathNameToRelativeNtPathName_U as discussed in the Definitive Guide on Win32 to NT Path Conversion. Since this is a built-in Windows function, it does not support the /cygdrive/ style prefixes. Running the simple test program argtofile in Cygwin easily demonstrates this.

The /cygdrive/ prefixes will therefore not work for programs compiled for Windows (such as the zip binary directly compiled using Visual C++). Therefore, the cygpath command is necessary to translate these paths to Win32-style file paths. To peek into how cygpath works, we can take advantage of the fact that the source code for the cygpath utility is available online. I found it easier to browse the sources after cloning the repo:

git clone https://cygwin.com/git/newlib-cygwin.git

The scenario of interest is what happens when cygpath -u ~ is invoked. In this case, we want to see how the “/cygdrive/” string is prefixed to the computed path.

  1. Execution flows from main > action > do_pathconv
  2. do_pathconv calls cygwin_conv_path which
  3. calls the conv_to_posix_path method of the mount_table which then
  4. normalizes the path by calling normalize_win32_path
  5. before finally iterating through the mount items to find the path’s prefix in the mount table.

Also searching for the cygpath \s*( regex leads to the vcygpath function in winsup/utils/path.cc. That appears to be more directly related to the cygpath command (how?). Searching for the \"cygdrive\" regex also reveals that this is a magic string used in many places in the codebase.

All this shows that there is indeed some complexity behind maintaining the POSIX/Win32-style file path mapping in Cygwin but it should be possible to add some basic logic to the Windows Info-ZIP build to handle /cygdrive/ prefixes in its file arguments. The question I have at this point is how does compiling the zip binaries for the Cygwin environment (the shipping configuration) result in proper handling of POSIX-style filenames?


Categories: C, Java

Building Info-ZIP Source Code

I have been working on building the OpenJDK for the Windows ARM64 platform. The make images command has been failing in Cygwin with errors such as:

Creating java.se.jmod
zip I/O error: Device or resource busy
zip error: Could not create output file (/cygdrive/d/dev/repos/jdk/build/windows-aarch64-server-release/support/src.zip)
make[4]: *** [ZipSource.gmk:79: /cygdrive/d/dev/repos/jdk/build/windows-aarch64-server-release/support/src.zip] Error 1
make[4]: *** Deleting file '/cygdrive/d/dev/repos/jdk/build/windows-aarch64-server-release/support/src.zip'
make[3]: *** [ZipSource.gmk:93: zip] Error 2
make[2]: *** [make/Main.gmk:389: zip-source] Error 2
make[2]: *** Waiting for unfinished jobs....

ERROR: Build failed for target 'images' in configuration 'windows-aarch64-server-release' (exit code 2)

One of the troubleshooting approaches I tried was to build an instrumented zip executable to replace Cygwin’s zip.exe. I started by searching for the Cygwin zip package. Someone was kind enough to have a link to the Cygwin zip package’s official page which in turn linked to the Cygwin zip packaging repository. This repo’s tree view shows a single file with source URIs for zip and that’s how I learned that this zip utility was Info-ZIP. The Zip 3.0 page has a link to Info-ZIP’s SourceForge site, from which the zip sources can be downloaded. I used curl in the Windows Terminal as follows:

curl -Lo zip30.tar.gz https://sourceforge.net/projects/infozip/files/Zip%203.x%20%28latest%29/3.0/zip30.tar.gz/download
tar xvf zip30.tar.gz
cd ./zip30
git init; git add *; git commit -m "Commit original Info-ZIP sources"

Now that we have the sources, let’s see how to build them. The scenario I’m working on is Windows specific so we need Visual Studio 2019 with the Desktop Development with C++ workload installed. I’ll be building a 32-bit zip executable. Launch the x86 Native Tools Command Prompt for VS 2019 and change to the zip30 source directory to start building. Some digging around reveals a makefile with build instructions (that seem one directory off). Here’s the command to build a 32-bit executable from the sources (note that building fails due to various errors that need to be addressed):

nmake -f win32\makefile.w32

Carriage Return (CR) Name Collisions

The first error is this rather cryptic mess of syntax errors:

Microsoft (R) Program Maintenance Utility Version 14.29.30133.0
Copyright (C) Microsoft Corporation.  All rights reserved.

        cl -nologo -c -W3 -O2 -DWIN32 -DASM_CRC -ML  zip.c
cl : Command line warning D9002 : ignoring unknown option '-ML'
zip.c
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18822): error C2143: syntax error: missing ':' before 'constant'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18822): error C2143: syntax error: missing ';' before ':'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18822): error C2059: syntax error: ':'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18823): error C2143: syntax error: missing '{' before ':'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18823): error C2059: syntax error: ':'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18824): error C2059: syntax error: '}'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18825): error C2059: syntax error: '}'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18826): error C2059: syntax error: '}'
zip.c(5746): warning C4267: '=': conversion from 'size_t' to 'ush', possible loss of data
zip.c(5838): warning C4267: '=': conversion from 'size_t' to 'ush', possible loss of data
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86\cl.EXE"' : return code '0x2'
Stop.

Some head banging (and not looking at the precompiler output from cl -c -W3 -O2 -DWIN32 -DASM_CRC /P zip.c carefully) leads to c – Compiler errors in WINNT.H after retargeting solution to latest SDK (10.0.18362) – Stack Overflow where the solution is rather simple: do not use the name CR to define the carriage return since it maps to a bitfield in one of the structs in winnt.h. See Rename CR identifier to avoid collisions with ARM64 structs in winnt.h by swesonga · Pull Request #2 · swesonga/Info-ZIP (github.com)

Outdated Linker Flags

The next error is a complaint about the /OPT:NOWIN98 flag.


        link -nologo user32.lib advapi32.lib /OPT:NOWIN98 /INCREMENTAL:NO /PDB:zip.pdb  /RELEASE zip.obj crypt.obj ttyio.obj zipfile.obj zipup.obj fileio.obj util.obj  crc32.obj crci386c.obj globals.obj deflate.obj trees.obj match32.obj win32.obj win32zip.obj nt.obj win32i64.obj zip.res
LINK : fatal error LNK1117: syntax error in option 'OPT:NOWIN98'
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86\link.EXE"' : return code '0x45d'
Stop.

Turns out this option was removed in Visual Studio 2010 as per the Microsoft C/C++ change history since the linker no longer supports optimizing for Windows 98. This is clearly a safe flag to remove from the linker flags in win32\makefile.w32.

Update the Branding

  1. Change the VERSION string from “3.0” to “3.0-ioHardenedZIP”
  2. Update the REVDATE from “July 5th 2008” to the current date (“December 18th 2021” in my case)
  3. Update the about text to indicate that it is a custom build.

Testing the Zip Build

The sources should now build successfully in the x86 Native Tools Command Prompt for VS 2019. The OpenJDK build uses the -qru flags for creating zip files so we can easily test the zip executable by creating a zip of the Info-ZIP help and license text.

zip -h > help.txt
zip -h2 > help2.txt
zip -L > license.txt
zip -qru ./files.zip -i *.txt

We need to verify whether the zip was correctly created. Saving this for another day.

Building on Linux/macOS

[Update 2021-12-22] To build Info-ZIP on macOS, the way the memset function is detected needs to be fixed. Info-ZIP for either Linux or macOS can then be built using this command:

make -f unix/Makefile generic

Categories: DevOps, Java

Windows OpenJDK Development Environment Setup

I documented how to set up an OpenJDK build environment for macOS and for Ubuntu Linux. Here is how to do the same for a Windows x86-64 environment on a Windows x86-64 machine. To create a JDK build for a Windows ARM64 environment, see the last section of this post.

  1. Install Visual Studio Code.
  2. Install Git and set Visual Studio Code as Git’s default editor.
  3. Install Visual Studio 2019. Ensure the Desktop Development with C++ workload is selected so that the C++ compiler is available.
The Visual Studio Desktop development with C++ workload
  1. Install a boot JDK. I used winget, the Windows package manager, to do this as outlined in the OpenJDK Windows installation instructions.
winget search Microsoft.OpenJDK
winget install Microsoft.OpenJDK.17
Microsoft Build of OpenJDK with Hotspot 17.0.1+12 (x64) 
Please watt while Windows configures Microsoft Build of OpenJDK with 
Hotspot 170 1+12(x64) 
Time remaining 7 seconds
  1. Install Cygwin using the command below.
setup-x86_64.exe -q -P autoconf -P make -P unzip -P zip
  1. Launch Cygwin, clone the OpenJDK repo then run bash configure. This should output an error if there are any missing dependencies. Once that completes successfully, make images will build the OpenJDK code.
mkdir ~/repos
cd ~/repos
git clone https://github.com/openjdk/jdk
cd ~/repos/jdk
bash configure
make images

To try out the local JDK build, run java.exe in the build folder, e.g.

cd ~/repos/jdk/build/windows-x86_64-server-slowdebug
cd jdk/bin
./java.exe -version

Troubleshooting

Keep in mind that make images can fail due to Cygwin-specific issues. The only workaround may be to downgrade Cygwin to a version known to work well, as I had to do when I ran into segfaults when running make.

Windows ARM64 JDK

To create a JDK build for a Windows ARM64 machine (as of this posting), you still need to set up the Windows x86-64 environment as described above with the additional changes below.

  1. Launch the Visual Studio Installer then install the “MSVC v142 – VS 2019 C++ ARM64 build tools (Latest)” item.
  2. Run bash configure with the additional –openjdk-target argument as outlined at https://github.com/microsoft/openjdk-aarch64
bash configure --openjdk-target=aarch64-unknown-cygwin

Running the ARM64 JDK on x86-64?

I was curious to see what happens if you try to run the ARM64 Java build in Cygwin on x86-64:

User@Machine /cygdrive/c/dev/repos/jdk/build/windows-aarch64-server-release/jdk/bin
$ ./java -version
-bash: ./java: cannot execute binary file: Exec format error

In the Windows command line we get this message in the terminal and the subsequent dialog box:

C:\dev\repos\jdk\build\windows-aarch64-server-release\jdk\bin>.\java.exe
This version of C:\dev\repos\jdk\build\windows-aarch64-server-release\jdk\bin\java.exe is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher.
Machine Type Mismatch

Launching it from Windows Explorer fails with this error:

Platform mismatch launching ARM64 Java on Windows x86-64

Categories: DevOps, Java

Set up macOS for OpenJDK Development

Last week I bought a new MacBook Pro with the Apple M1 Chip and 16GB of RAM. These are the steps I used to set it up for building the OpenJDK codebase.

General macOS Configuration

  1. Set the OS appearance (theme). I tend to prefer dark theme.
  2. Set your Mac’s name (it bothers me when the terminal has some random host name). You might need to restart your terminal for this change to take effect.

Install Development Tools

Should any of the commands below fail, see the troubleshooting section at the end for possible workarounds.

  1. Install homebrew by running the recommended command (and see the troubleshooting section if there are any git errors):
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
  1. Clone the OpenJDK repo:
mkdir ~/repos
cd ~/repos
git clone https://github.com/openjdk/jdk
  1. Install Xcode from the App Store. Xcode 13.1 is the most recent as of this post.
  2. Download and install the Xcode command line tools.
  3. Select the Xcode command line tools: launch Xcode then go to Preferences > Locations. Select Xcode 13.1 in the Command Line Tools dropdown as shown below.
Selecting the Command Line Tools in Xcode
  1. Install autoconf: brew install autoconf
  2. Install Visual Studio Code (optional)

Installing the Boot JDK

This was surprisingly straightforward. The instructions on how to install the Microsoft Build of OpenJDK on macOS comprise a single homebrew command:

brew install --cask microsoft-openjdk

Installing the Microsoft OpenJDK on macOS

Building the OpenJDK

Running bash configure in the JDK folder should display any missing dependencies. Any errors from bash configure will need to be resolved before running make images.

cd ~/repos/jdk
bash configure
make images

To browse through the contents of the build folder in finder:

open ~/repos/jdk/build/

To try out your new build, switch to the bin folder and check the Java version:

cd ~/repos/jdk/build/macosx-aarch64-server-release/jdk/bin
./java -version

Here is the output I get:

saint@Saints-MBP-2021 bin % ./java -version
openjdk version "18-internal" 2022-03-22
OpenJDK Runtime Environment (build 18-internal+0-adhoc.saint.jdk)
OpenJDK 64-Bit Server VM (build 18-internal+0-adhoc.saint.jdk, mixed mode)
saint@Saints-MBP-2021 bin % 

Troubleshooting

Homebrew Installation Failure

Installing homebrew appeared to be successful (last message output below) but there was a git error in the output!

==> Downloading and installing Homebrew..
remote: Enumerating objects: 345, done.
remote: Counting objects: 100% (297/297), done.
remote: Compressing objects: 100% (107/107), done.
remote: Total 345 (delta 227), reused 238 (delta 184), pack-reused 48
Receiving objects: 100% (345/345), 171.73 KiB | 971.00 KiB/s, done.
Resolving deltas: 100% (227/227),
completed with 54 local objects.
From https://github.com/Homebrew/brew
* [new branch]
dependabot/bundler/Librarv/Homebrew/sorbet-0.5.9396
-> origin/dependabot/bundler/Librarv/Homebrew/sorbet-0.5.9396
778de69b0..be908f679 master -> origin/master
* [new tag] 3.3.6 -> 3.3.6
HEAD is now at be908f679 Merge pull request #12502 from carlocab/bug-template
error: Not a valid ref: refs/remotes/origin/master
fatal: ambiguous argument
'refs/remotes/origin/master': unknown revision or path not in the working tree.
Use
"_-' to separate paths from revisions, like this:
'git <command> [<revision>...] - I<files...11
fatal: Could not resolve HEAD to a revision
Warning: /opt/homebrew/bin is not in your PATH.
Instructions on how to configure vour shell for Homebrew
can be found in the 'Next steps' section below.
==> Installation successful!

Here is the relevant error, which I was able to copy/paste (with some typos) from the PNG!!!

error: Not a valid ref: refs/remotes/origin/master
fatal: ambiguous argument
'refs/remotes/origin/master': unknown revision or path not in the working tree.

I thought I could keep chugging along merrily but other steps will fail if this is not addressed. Thankfully, https://stackoverflow.com/questions/65605282/trying-to-install-hugo-via-homebrew-could-not-resolve-head-to-a-revision already addressed how to fix this. I had to look up the -C flag in the git docs (it executes the command in the context of the specified directory).

git -C $(brew --repository homebrew/core) checkout master
Fixing the homebrew/core git repo

Autoconf Installation Failures

No available formula with the name “autoconf”

I ran into errors of the form No available formula with the name "autoconf" when attempting to install autoconf. However, this happen with the unresolved brew installation git issue described above. Once that was resolved, https://stackoverflow.com/questions/11552171/cant-install-software-using-brew-on-my-mac helpfully pointed out that autoconf is part of the command line tools package (hence step 4 in the instructions above).

[saint@Saints-MBP-2021 jk % brew install autoconf
fatal: Could not resolve HEAD to a revision
Running 'brew update --preinstall'
==> Homebrew is run entirelv by unpaid volunteers. Please consider donating:
https://github.com/Homebrew/brew#donations
==› Auto-updated Homebrew!
Updated 1 tap (homebrew/cask).
==> Updated Casks
dated 1 cask.
Warning: No available formula with the name
"autoconf"
==› Searching for similarlv named formulae.
Error: No similarly named formulae found.
==> Searching for a previously deleted formula (in the last month).
Error: No previously deleted formula found.
==> Searching taps on GitHub.
Error: No formulae found in taps.

No Such File or Directory @ dir_chdir

Setting up a build environment on my Intel MacBook Pro led to errors like this:

==> Installing autoconf dependency: m4
Error: No such file or directory @ dir_chdir - /usr/local/Cellar

The workaround from https://programmerah.com/brew-install-node-error-no-such-file-or-directory-dir_chdir-bottle-installation-failed-5943/ is to simply reinstall brew.

Missing Xcodebuild tool

One of my bash configure runs failed with this error:

checking for sdk name..
configure: error: No xcodebuild tool and no system framework headers found, use --with-sysroot or --with-sdk-name to provide a path to a valid SDK
/Users/saint/repos/idk/build/.configure-support/generated-configure.sh: line 84: 5: Bad file descriptor
configure exiting with result code 1

This was because the command line tools had not been selected in Xcode as show in step 6. This resolution came from https://stackoverflow.com/questions/17980759/xcode-select-active-developer-directory-error.


Categories: DevOps

Downgrading Cygwin

Last week I was trying to set up an OpenJDK build environment on my Windows 11 machine. Building the repo would fail due to segfaults when running make. The workaround is to downgrade Cygwin from 3.3 to 3.2:

  1. Launch the Cygwin setup program (setup-x86_64.exe).
  2. Change the View dropdown to Full.
  3. Enter cygwin into the search box then select the cygwin package..
  4. Open the cygwin package’s dropdown in the New column.
  5. Change the selection from Keep to the desired version as shown in the image below.
Cygwin Setup - Select Packages 
Select packages to install 
Full 
Pack age 
base-cyg',vin 
cygnin-debuginfo 
cygwin-devel 
cygwin-doc 
cygwin-x-doc 
cygwin32 
cygwin32-binutiIs 
cygwin32-binutiIs-debuginfo 
cygwin32-defaufMıantfest 
cygwi n 32-gcc-core 
cygwi n 32-gcc-debuginfo 
cyg n 32-gcc*oftran 
cygwin32-gcc-g++ 
cygwin32-gettext 
cygwin324ibbfd 
cygwin324ibiconv 
cygwin324ibtooI 
cygwin32%N32apiheadem 
cygwi n -ıuntime 
cygwin32-zIib 
Korg -serv Sh repon er 
Korg -serv er-cygwin shrepoft 
Side obsolete packages 
Src7 Categofies 
Uninstall 
Skip 
31.7-1 
3.1 .ı-ı 
3.12-1 
3.1.5-ı 
3.1.6-ı 
3.1.7-ı 
Size 
O 
kep Best 
Descfiption 
O 
Jest 
332-1 
321-1 
331-0,1.9814cfd8f693 (Test) 
331-0.2.6c1f49f83fde (Test) 
331- ı 
33.1-1 
Keep 
Reinstall 
L] Lhils 
Debug 
2,931k 
Intial base installation helper scıipt 
LIIBtion engine 
or cyg',vın 
pment files 
%fıc documentation including man pages 
yectfic document ation 
t toolchain 
ygwin 3%tt toolchain 
or cygwin32-binutiIs 
ication manifest for Cygwin32 toolchain 
Win 32btt toolchain (C, OpenMP) 
or cygwin32-gcc 
Win 32btt toolchain (Fortran) 
Win 32bt toolchain 
t for Cygwin 32bt toolchain 
Nin 32btt toolchain 
gwin 3%tt toolchain 
gwin 3%tt toolchain 
,gwin 3%tt toolchain 
veadem for Cygwin 3hit toolchain 
libs for Cygwin 3hit toolchain 
in 32btt toolchain 
A breakpad crashrepofter for Cygwin 
Debug info for xorg-servercygwincrashrepofter
Cygwin version selection
  1. Click on the Next button then verify that the downgrade is to the desired version.
Cygwin Setup - Review and confirm changes 
Review coriirm 
Uninstall cygwin 3 3 2-1 (automatically added) 
Install cygwin 3 2 0-1 
c
Reviewing cygwin downgrade version

Finish the setup process then launch cygwin and verify that the version is correct by running:

cygcheck -c Cygwin
Verifying cygwin version after downgrading