In the last post, I described how to build the Info-ZIP sources. When using the resulting zip binaries in Cygwin, some important path handling issues come up. The paths passed to the zip binary when building the OpenJDK in Cygwin use forward slashes. The Cygwin User’s Guide has a section on File Access that outlines the support for POSIX and Win32-style file paths.
The Windows file system APIs support forward slashes in file paths. The zip source code uses the fopen CRT function, which eventually ends up calling CreateFileW. The CreateFileW docs state that you may use either forward slashes (/) or backslashes (\) in the lpFileName parameter. The translation of paths from Win32 to NT happens in a function called RtlDosPathNameToRelativeNtPathName_U as discussed in the Definitive Guide on Win32 to NT Path Conversion. Since this is a built-in Windows function, it does not support the /cygdrive/ style prefixes. Running the simple test program argtofile in Cygwin easily demonstrates this.
The /cygdrive/ prefixes will therefore not work for programs compiled for Windows (such as the zip binary directly compiled using Visual C++). Therefore, the cygpath command is necessary to translate these paths to Win32-style file paths. To peek into how cygpath works, we can take advantage of the fact that the source code for the cygpath utility is available online. I found it easier to browse the sources after cloning the repo:
The scenario of interest is what happens when cygpath -u ~ is invoked. In this case, we want to see how the “/cygdrive/” string is prefixed to the computed path.
normalizes the path by calling normalize_win32_path
before finally iterating through the mount items to find the path’s prefix in the mount table.
Also searching for the cygpath \s*( regex leads to the vcygpath function in winsup/utils/path.cc. That appears to be more directly related to the cygpath command (how?). Searching for the \"cygdrive\" regex also reveals that this is a magic string used in many places in the codebase.
All this shows that there is indeed some complexity behind maintaining the POSIX/Win32-style file path mapping in Cygwin but it should be possible to add some basic logic to the Windows Info-ZIP build to handle /cygdrive/ prefixes in its file arguments. The question I have at this point is how does compiling the zip binaries for the Cygwin environment (the shipping configuration) result in proper handling of POSIX-style filenames?
curl -Lo zip30.tar.gz https://sourceforge.net/projects/infozip/files/Zip%203.x%20%28latest%29/3.0/zip30.tar.gz/download
tar xvf zip30.tar.gz
cd ./zip30
git init; git add *; git commit -m "Commit original Info-ZIP sources"
Now that we have the sources, let’s see how to build them. The scenario I’m working on is Windows specific so we need Visual Studio 2019 with the Desktop Development with C++ workload installed. I’ll be building a 32-bit zip executable. Launch the x86 Native Tools Command Prompt for VS 2019 and change to the zip30 source directory to start building. Some digging around reveals a makefile with build instructions (that seem one directory off). Here’s the command to build a 32-bit executable from the sources (note that building fails due to various errors that need to be addressed):
nmake -f win32\makefile.w32
Carriage Return (CR) Name Collisions
The first error is this rather cryptic mess of syntax errors:
Microsoft (R) Program Maintenance Utility Version 14.29.30133.0
Copyright (C) Microsoft Corporation. All rights reserved.
cl -nologo -c -W3 -O2 -DWIN32 -DASM_CRC -ML zip.c
cl : Command line warning D9002 : ignoring unknown option '-ML'
zip.c
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18822): error C2143: syntax error: missing ':' before 'constant'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18822): error C2143: syntax error: missing ';' before ':'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18822): error C2059: syntax error: ':'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18823): error C2143: syntax error: missing '{' before ':'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18823): error C2059: syntax error: ':'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18824): error C2059: syntax error: '}'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18825): error C2059: syntax error: '}'
C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winnt.h(18826): error C2059: syntax error: '}'
zip.c(5746): warning C4267: '=': conversion from 'size_t' to 'ush', possible loss of data
zip.c(5838): warning C4267: '=': conversion from 'size_t' to 'ush', possible loss of data
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86\cl.EXE"' : return code '0x2'
Stop.
Turns out this option was removed in Visual Studio 2010 as per the Microsoft C/C++ change history since the linker no longer supports optimizing for Windows 98. This is clearly a safe flag to remove from the linker flags in win32\makefile.w32.
Update the Branding
Change the VERSION string from “3.0” to “3.0-ioHardenedZIP”
Update the REVDATE from “July 5th 2008” to the current date (“December 18th 2021” in my case)
Update the about text to indicate that it is a custom build.
Testing the Zip Build
The sources should now build successfully in the x86 Native Tools Command Prompt for VS 2019. The OpenJDK build uses the -qru flags for creating zip files so we can easily test the zip executable by creating a zip of the Info-ZIP help and license text.
zip -h > help.txt
zip -h2 > help2.txt
zip -L > license.txt
zip -qru ./files.zip -i *.txt
We need to verify whether the zip was correctly created. Saving this for another day.
I documented how to set up an OpenJDK build environment for macOS and for Ubuntu Linux. Here is how to do the same for a Windows x86-64 environment on a Windows x86-64 machine. To create a JDK build for a Windows ARM64 environment, see the last section of this post.
setup-x86_64.exe -q -P autoconf -P make -P unzip -P zip
Launch Cygwin, clone the OpenJDK repo then run bash configure. This should output an error if there are any missing dependencies. Once that completes successfully, make images will build the OpenJDK code.
mkdir ~/repos
cd ~/repos
git clone https://github.com/openjdk/jdk
cd ~/repos/jdk
bash configure
make images
To try out the local JDK build, run java.exe in the build folder, e.g.
cd ~/repos/jdk/build/windows-x86_64-server-slowdebug
cd jdk/bin
./java.exe -version
To create a JDK build for a Windows ARM64 machine (as of this posting), you still need to set up the Windows x86-64 environment as described above with the additional changes below.
Launch the Visual Studio Installer then install the “MSVC v142 – VS 2019 C++ ARM64 build tools (Latest)” item.
In the Windows command line we get this message in the terminal and the subsequent dialog box:
C:\dev\repos\jdk\build\windows-aarch64-server-release\jdk\bin>.\java.exe
This version of C:\dev\repos\jdk\build\windows-aarch64-server-release\jdk\bin\java.exe is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher.
Machine Type Mismatch
Launching it from Windows Explorer fails with this error:
Platform mismatch launching ARM64 Java on Windows x86-64
Last week I bought a new MacBook Pro with the Apple M1 Chip and 16GB of RAM. These are the steps I used to set it up for building the OpenJDK codebase.
Set your Mac’s name (it bothers me when the terminal has some random host name). You might need to restart your terminal for this change to take effect.
Install Development Tools
Should any of the commands below fail, see the troubleshooting section at the end for possible workarounds.
Install homebrew by running the recommended command (and see the troubleshooting section if there are any git errors):
Select the Xcode command line tools: launch Xcode then go to Preferences > Locations. Select Xcode 13.1 in the Command Line Tools dropdown as shown below.
Running bash configure in the JDK folder should display any missing dependencies. Any errors from bash configure will need to be resolved before running make images.
cd ~/repos/jdk
bash configure
make images
To browse through the contents of the build folder in finder:
open ~/repos/jdk/build/
To try out your new build, switch to the bin folder and check the Java version:
cd ~/repos/jdk/build/macosx-aarch64-server-release/jdk/bin
./java -version
Here is the output I get:
saint@Saints-MBP-2021 bin % ./java -version
openjdk version "18-internal" 2022-03-22
OpenJDK Runtime Environment (build 18-internal+0-adhoc.saint.jdk)
OpenJDK 64-Bit Server VM (build 18-internal+0-adhoc.saint.jdk, mixed mode)
saint@Saints-MBP-2021 bin %
Troubleshooting
Homebrew Installation Failure
Installing homebrew appeared to be successful (last message output below) but there was a git error in the output!
==> Downloading and installing Homebrew..
remote: Enumerating objects: 345, done.
remote: Counting objects: 100% (297/297), done.
remote: Compressing objects: 100% (107/107), done.
remote: Total 345 (delta 227), reused 238 (delta 184), pack-reused 48
Receiving objects: 100% (345/345), 171.73 KiB | 971.00 KiB/s, done.
Resolving deltas: 100% (227/227),
completed with 54 local objects.
From https://github.com/Homebrew/brew
* [new branch]
dependabot/bundler/Librarv/Homebrew/sorbet-0.5.9396
-> origin/dependabot/bundler/Librarv/Homebrew/sorbet-0.5.9396
778de69b0..be908f679 master -> origin/master
* [new tag] 3.3.6 -> 3.3.6
HEAD is now at be908f679 Merge pull request #12502 from carlocab/bug-template
error: Not a valid ref: refs/remotes/origin/master
fatal: ambiguous argument
'refs/remotes/origin/master': unknown revision or path not in the working tree.
Use
"_-' to separate paths from revisions, like this:
'git <command> [<revision>...] - I<files...11
fatal: Could not resolve HEAD to a revision
Warning: /opt/homebrew/bin is not in your PATH.
Instructions on how to configure vour shell for Homebrew
can be found in the 'Next steps' section below.
==> Installation successful!
Here is the relevant error, which I was able to copy/paste (with some typos) from the PNG!!!
error: Not a valid ref: refs/remotes/origin/master
fatal: ambiguous argument
'refs/remotes/origin/master': unknown revision or path not in the working tree.
I ran into errors of the form No available formula with the name "autoconf" when attempting to install autoconf. However, this happen with the unresolved brew installation git issue described above. Once that was resolved, https://stackoverflow.com/questions/11552171/cant-install-software-using-brew-on-my-mac helpfully pointed out that autoconf is part of the command line tools package (hence step 4 in the instructions above).
[saint@Saints-MBP-2021 jk % brew install autoconf
fatal: Could not resolve HEAD to a revision
Running 'brew update --preinstall'
==> Homebrew is run entirelv by unpaid volunteers. Please consider donating:
https://github.com/Homebrew/brew#donations
==› Auto-updated Homebrew!
Updated 1 tap (homebrew/cask).
==> Updated Casks
dated 1 cask.
Warning: No available formula with the name
"autoconf"
==› Searching for similarlv named formulae.
Error: No similarly named formulae found.
==> Searching for a previously deleted formula (in the last month).
Error: No previously deleted formula found.
==> Searching taps on GitHub.
Error: No formulae found in taps.
No Such File or Directory @ dir_chdir
Setting up a build environment on my Intel MacBook Pro led to errors like this:
==> Installing autoconf dependency: m4
Error: No such file or directory @ dir_chdir - /usr/local/Cellar
One of my bash configure runs failed with this error:
checking for sdk name..
configure: error: No xcodebuild tool and no system framework headers found, use --with-sysroot or --with-sdk-name to provide a path to a valid SDK
/Users/saint/repos/idk/build/.configure-support/generated-configure.sh: line 84: 5: Bad file descriptor
configure exiting with result code 1
I’m using a Windows 10 physical machine for my OpenJDK 17 development. Unfortunately, I ran into some issues getting the environment set up to build the JDK on Windows. To work around this, I created a Linux virtual machine. Although the instructions for building on Linux are on the OpenJDK site, I would like to have all the instructions in one spot, hence this post.
# Valid values are only '18.04' and '20.04'
# For other versions of Ubuntu, please use the tar.gz package
ubuntu_release=`lsb_release -rs`
cd ~/Downloads/
wget https://packages.microsoft.com/config/ubuntu/${ubuntu_release}/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo apt-get install apt-transport-https
Verify that everything is working by running “java -version”
Clone and Build the JDK
Clone the JDK. Note that cloning a fork might be much slower than cloning the upstream Github repo! I was averaging about 60KiB/s on my rork whereas cloning the upstream OpenJDK was averaging 6 MiB/s when receiving objects!
mkdir ~/repos
cd ~/repos
git clone https://github.com/openjdk/jdk
The JDK repo can now be configured and built
cd jdk
bash configure
make images
The configure command should display any missing dependencies that it needs and a suggestion for how to install them.
To try out your new build, switch to the bin folder and check the Java version:
cd ~/repos/jdk/build/linux-x86_64-server-release/jdk/bin
./java -version
Last week was my first week in the Java engineering group. It has been about 11 years since I took a compiler course (while in the CS MS program at BYU). A quick review of the history of Java was in order. Turns out I last used Java in 2012 in grad school. That must have been Java SE 7 from 2011 and Java SE 6 before that. Since I have not been in the compiler space since then, I have a steep learning curve ahead. That is the exciting thing about technology though – there is always more to learn!
I am currently a programmer in the developer division at Microsoft so it was helpful going through some of the Java development with Microsoft documentation for a high level overview of all our offerings. Also informative given my long absence from Java-land were the docs on how to Transition from Java 7 to Java 8 and from Java 8 to Java 11. It hadn’t yet dawned on me by the time I read through these that the reason references to 8, 11, and 17 keep coming up is because they are LTS releases.
As a newbie to the Java development world, I started by watching this 2019 OpenJDK Development talk on how to become an OpenJDK contributor. It is a great overview of concepts like project roles (author, committer, reviewer, etc), the contributor agreement, and (perhaps most importantly to me), how to find an issue to work on and build the OpenJDK. The breakdown of commonly used terminology and abbreviations was great to have as well.
For an introduction to the hotspot compiler, I started going through “A Simple Graph-Based Intermediate Representation“. I ended up watching Cliff Click’s talk on The Sea of Nodes and the HotSpot JIT before I got that far along in the paper. It was fascinating seeing details such as the CPU L1/L2 cache size playing into the design! Some of the concepts that I need to review after that talk include:
The sea of nodes talk also revealed to me how little I know about companies in the Java space. I don’t think I had heard of Azul before, for example. In fact, it’s not just companies but also technologies! I was going through some build documentation when I ran into mentions of AdoptOpenJDK and Adoptium, both of which were foreign to me. I was glad though to see my old friend Eclipse doing well.
One of the most enjoyable things about being a programmer is working with very skilled people, especially watching them in action! I always learn a lot! My colleagues David and Mat were kind enough to pull me into their triage and reporting of [JDK-8277299] STACK_OVERFLOW in Java_sun_awt_shell_Win32ShellFolder2_getIconBits – Java Bug System so I could get my feet wet with how things are done in OpenJDK development.
The OpenJDK process is certainly different from the other open source communities I’ve been a part of (.NET and Mozilla Firefox). My manager and I poked around the bug DB to see what compiler starter bugs are out there. I picked bug [JDK-7077093] labelOper::label() should return Label& but since I must start out as an author, issues cannot be assigned to me. Unusual to me but the logic appears sound. Here is the query for C2 starter bugs.
Other highlights of the week were setting up my dev box to build the OpenJDK source code (unsuccessfully), discovering that compiler explorer is a thing (and an open source one at that), learning from my teammates how to investigate a failure of a fairly complex test on MacOS (they were using LLDB). I hope to write follow-up entries on these at some point.
The first programming assignment in the Operating Systems course can be a challenge for students that haven’t written C++ code in a while. While working with the std::queue data structure in C++, it’s easy to make certain types of mistakes (especially if C++ isn’t your native tongue):
Not “using namespace std” when using standard library containers. This can result in some ugly error messages in Visual Studio, e.g. error/warning codes C2143, C4430, and C2238 for the class member array below (is there a better way for students/developers to find out what is happening when they make such a trivial mistake)?
Not understanding the assignment operator semantics on a container like a queue. If we write queue<type> myqueue = array[i]; we get a copy of the queue array[i] (we might have simply wanted a reference/alias). For such a mistake, the code obviously compiles but doesn’t function as intended.
Declaring a fixed-sized data structure to hold all values from a variable-sized container! Runtime errors take care of informing students about this bug (if they’re not lucky enough to have almost empty variable-size containers). The correct declaration of dynamic arrays of templated items is also not usually obvious: T* all_elements = new T[dynamic_integer_size];
For my Computer Aided Geometric Design course, a simple program called CPLOT is used for some of the project work. Its documentation is on the course website. After my initial draft of my project 1 implementation crashed, I was inspired to try out Valgrind on CPLOT (even though I was able to find the bugs using my debugger). All that’s needed to install Valgrind on Ubuntu is sudo apt-get install valgrind. The CPLOT code supplied needed a minor change in order to compile, so I changed
The eg1.dat file I used is the one on page 3 of the CPLOT documentation. Below is the output from valgrind:
==1594== Memcheck, a memory error detector
==1594== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==1594== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
==1594== Command: ./cplot.out eg1.dat eg1.eps
==1594==
In initps: eg1.eps
File processed successfully!
==1594==
==1594== HEAP SUMMARY:
==1594== in use at exit: 120 bytes in 6 blocks
==1594== total heap usage: 8 allocs, 2 frees, 824 bytes allocated
==1594==
==1594== 120 (8 direct, 112 indirect) bytes in 1 blocks are definitely lost in loss record 3 of 3
==1594== at 0x402641D: operator new(unsigned int) (vg_replace_malloc.c:255)
==1594== by 0x8049224: readCurve() (cplot.cpp:182)
==1594== by 0x8048D89: main (cplot.cpp:127)
==1594==
==1594== LEAK SUMMARY:
==1594== definitely lost: 8 bytes in 1 blocks
==1594== indirectly lost: 112 bytes in 5 blocks
==1594== possibly lost: 0 bytes in 0 blocks
==1594== still reachable: 0 bytes in 0 blocks
==1594== suppressed: 0 bytes in 0 blocks
==1594==
==1594== For counts of detected and suppressed errors, rerun with: -v
==1594== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 17 from 6)
This program can be improved by changing the return statements in the body of the main for(;;) loop into break statements. By so doing, all the cleanup code can be placed after that loop, just before the program exits. This also prevents the duplication of the cleanup code. The code after the for(;;) loop then becomes:
// Deallocate all allocated memory
for (int i=0; i < NCURVES; i++) {
delete curves[i];
}
// Close the file resources
fclose(in);
fclose(ps);
return 0;
==6539== Memcheck, a memory error detector
==6539== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==6539== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
==6539== Command: ./cplot.out eg1.dat eg1.eps
==6539==
In initps: eg1.eps
File processed successfully!
==6539==
==6539== HEAP SUMMARY:
==6539== in use at exit: 0 bytes in 0 blocks
==6539== total heap usage: 8 allocs, 8 frees, 824 bytes allocated
==6539==
==6539== All heap blocks were freed -- no leaks are possible
==6539==
==6539== For counts of detected and suppressed errors, rerun with: -v
==6539== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 17 from 6)
There was another bug in the program since it could attempt to plot control polygons for uninitialized curves because of a missing else. I pushed the trivial fix to that. The program also crashed if the input file specified did not exist because fscanf would try to use a NULL parameter. The fix is a simple NULL check.
A sample implementation of the dining philosophers problem is provided in the JPF core source code. The solution I gave to this problem was to use the following locking scheme: if a thread has an even id i, then it should lock fork i first and then fork (i + 1) % N. If its id i is odd, then it should lock fork (i + 1) % N first and then lock fork i. Replacing
