If curl fails with error 23 in Cygwin, it is likely that the curl command shipped with Windows is running in the Cygwin terminal instead of the curl binary distributed with Cygwin. To install the Cywin curl command, run the setup executable with these flags:
setup-x86_64.exe -q -P curl
Background
One of the issues I looked into this week was running an AQAvit™ Verification test on Windows and therefore in Cygwin. Here are the environment variables I set and the commands I executed as outlined on the AQAvit™ Verification page.
The last command above (compilation) failed on my Surface Pro X with this error:
...
Buildfile: C:\java\aqa\aqa-tests\TKG\scripts\build_tools.xml
build:
clean:
[delete] Deleting directory C:\java\aqa\aqa-tests\TKG\bin
init:
[mkdir] Created dir: C:\java\aqa\aqa-tests\TKG\bin
getDependentLibs:
[exec] --------------------------------------------
[exec] path is set to /cygdrive/c/java/aqa/aqa-tests/TKG/../TKG/lib
[exec] task is set to default
[exec] dependencyList is set to json_simple
[exec] --------------------------------------------
[exec] Starting download third party dependent jars
[exec] --------------------------------------------
[exec] downloading dependent third party jars to /cygdrive/c/java/aqa/aqa-tests/TKG/../TKG/lib
[exec] downloading https://repo1.maven.org/maven2/com/googlecode/json-simple/json-simple/1.1.1/json-simple-1.1.1.jar
[exec] % Total % Received % Xferd Average Speed Time Time Time Current
[exec] Dload Upload Total Spent Left Speed
[exec]
[exec] 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Failed to open the file Warning: /cygdrive/c/java/aqa/aqa-tests/TKG/../TKG/lib/json-simple.jar: No
[exec] Warning: such file or directory
[exec]
[exec] 5 23931 5 1371 0 0 4483 0 0:00:05 --:--:-- 0:00:05 4509curl: (23) Failure writing output to destination
[exec] ERROR: downloading https://repo1.maven.org/maven2/com/googlecode/json-simple/json-simple/1.1.1/json-simple-1.1.1.jar failed, return code: 5888
BUILD FAILED
C:\java\aqa\aqa-tests\TKG\scripts\build_tools.xml:58: The following error occurred while executing this line:
C:\java\aqa\aqa-tests\TKG\scripts\getDependencies.xml:27: exec returned: 2
I had core.autocrlf set to true when I initially checked out the aqa-tests repo so I suspected the culprit to be a file that didn’t get converted to LF when I ran git add --renormalize as suggested by How do I re-checkout all files in Git to convert from CRLF to LF? – Stack Overflow. I would get errors like “/cygdrive/c/java/aqa/aqa-tests/TKG/scripts/getTestenvProperties.sh: line 14: $’\r’: command not found” and ended up manually changing the line endings using VS Code since there were only 3 files that needed to be changed but I digress…
$ curl -k -o /c/java/aqa/aqa-tests/TKG/lib/file.jar https://repo1.maven.org/maven2/com/googlecode/json-simple/json-simple/1.1.1/json-simple-1.1.1.jar
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Failed to open the file /c/java/aqa/aqa-tests/TKG/lib/file.jar: No
Warning: such file or directory
5 23931 5 1371 0 0 4660 0 0:00:05 --:--:-- 0:00:05 4679
curl: (23) Failure writing output to destination
$ curl -k -o ../file.jar https://repo1.maven.org/maven2/com/googlecode/json-simple/json-simple/1.1.1/json-simple-1.1.1.jar
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 23931 100 23931 0 0 84394 0 --:--:-- --:--:-- --:--:-- 84861
Which curl is running? Turns out it’s the native Windows curl command!
I realize there must be a native Cygwin curl package so I install it using this command: \software\setup-x86_64.exe -q -P curl
Installing curl
I end up needing to open a new shell to get it to be picked up when building. That fixes the problem (this version successfully writes the downloaded file to disk)! Here’s the version information for the curl utility installed by Cygwin:
Hsdis is an externally loadable disassembler plugin. It lets you see which assembly instructions the JVM generates for your Java code. On 64-bit Windows, it is a binary called hsdis-amd64.dll (and hsdis-i386.dll on 32-bit platforms). This binary needs to be in the same directory as jvm.dll. Some good resources out there on building the hsdis binary for the OpenJDK include:
For Cygwin, the latter resource (from 2012?) is all we need. I like that Gunnar’s blog post covered how to use hsdis after building it so this writeup aims to combine both blogs into a simple Cygwin install-build-disassemble set of instructions.
Building hsdis for 64-bit JVMs
Install Cygwin with the gcc-core, make, and mingw64-x86_64-gcc-core packages by launching the setup executable using this command (no need to bother selecting packages in the UI since you have already specified them on the command line)
setup-x86_64.exe -P gcc-core -P mingw64-x86_64-gcc-core -P make
mkdir ~/repos
cd ~/repos
git clone https://github.com/openjdk/jdk
Run these commands to download GNU binutils and build hsdis (Update 2022-01-07: version downgraded to 2.36 to avoid build failures investigated in Fixing Hsdis Compile Failure in GNU binutils).
cd ~
curl -Lo binutils-2.36.tar.gz https://ftp.gnu.org/gnu/binutils/binutils-2.36.tar.gz
tar xvf binutils-2.36.tar.gz
cd ~/repos/jdk/src/utils/hsdis
make OS=Linux MINGW=x86_64-w64-mingw32 BINUTILS=~/binutils-2.36
Copy the hsdis binary to the locally built java bin folder
I have created a basic substitution cipher, which we can compile and disassemble using the commands below. Note that these commands save the .java file to a temp folder to make cleanup much easier. Also note the redirection to a file since the output can be voluminous.
cd build/windows-x86_64-server-release/jdk/bin
mkdir -p temp
cd temp
curl -Lo BasicSubstitutionCipher.java https://raw.githubusercontent.com/swesonga/scratchpad/main/apps/crypto/substitution-cipher/BasicSubstitutionCipher.java
../javac BasicSubstitutionCipher.java
../java -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -XX:+LogCompilation BasicSubstitutionCipher > BasicSubstitutionCipher.disassembled.txt
Once the disassembly completes, we can view the instructions generated in the BasicSubstitutionCipher.disassembled.txt file.
One open question in this setup is why the installed GNU binutils cannot be used to build hsdis. Seems strange to have to build them from source when the binutils Cygwin package was also installed in step 1 above.
After figuring out how to build the Info-ZIP sources, I had a few commands to test the zip file by creating a few text files to zip.
zip -h > help.txt
zip -h2 > help2.txt
zip -L > license.txt
Unfortunately, the zip -qru ./files.zip -i *.txtcommand from the OpenJDK is not what we need. To actually create a zip file, use only the -u flag
zip -u files.zip *.txt
To test that the files were zipped successfully, unzip the files and compare them to the original files. Here’s the whole script for this:
echo "---Creating temp directory---"
mkdir temp; cd temp
echo "---Creating text files---"
zip -h > help.txt
zip -h2 > help2.txt
zip -L > license.txt
echo "---Adding text files to a new repo---"
git init
git add *.txt
git commit -m "Add original text files"
echo "---Zipping text files---"
zip -u files.zip *.txt
echo "---Removing text files---"
rm *.txt
echo "---Unzipping text files---"
unzip files.zip
echo "---Checking unzipped files---"
git diff
When using the zip binary for Windows, something strange happens when running the zip command a 2nd time:
$ zip -u files.zip *.txt
zip warning: files.zip not found or empty
adding: help.txt (176 bytes security) (deflated 49%)
adding: help2.txt (176 bytes security) (deflated 62%)
adding: license.txt (176 bytes security) (deflated 54%)
$ zip -u files.zip *.txt
zip warning: expected 3 entries but found 0
zip error: Zip file structure invalid (files.zip)
Info-ZIP supports a -sd flag that shows diagnostic information while it runs. It reveals that something is going wrong when reading the archive.
$ zip -usd files.zip *.txt
sd: Zipfile name 'files.zip'
sd: Command line read
sd: Reading archive
zip warning: expected 3 entries but found 0
zip error: Zip file structure invalid (files.zip)
In the last post, I described how to build the Info-ZIP sources. When using the resulting zip binaries in Cygwin, some important path handling issues come up. The paths passed to the zip binary when building the OpenJDK in Cygwin use forward slashes. The Cygwin User’s Guide has a section on File Access that outlines the support for POSIX and Win32-style file paths.
The Windows file system APIs support forward slashes in file paths. The zip source code uses the fopen CRT function, which eventually ends up calling CreateFileW. The CreateFileW docs state that you may use either forward slashes (/) or backslashes (\) in the lpFileName parameter. The translation of paths from Win32 to NT happens in a function called RtlDosPathNameToRelativeNtPathName_U as discussed in the Definitive Guide on Win32 to NT Path Conversion. Since this is a built-in Windows function, it does not support the /cygdrive/ style prefixes. Running the simple test program argtofile in Cygwin easily demonstrates this.
The /cygdrive/ prefixes will therefore not work for programs compiled for Windows (such as the zip binary directly compiled using Visual C++). Therefore, the cygpath command is necessary to translate these paths to Win32-style file paths. To peek into how cygpath works, we can take advantage of the fact that the source code for the cygpath utility is available online. I found it easier to browse the sources after cloning the repo:
The scenario of interest is what happens when cygpath -u ~ is invoked. In this case, we want to see how the “/cygdrive/” string is prefixed to the computed path.
normalizes the path by calling normalize_win32_path
before finally iterating through the mount items to find the path’s prefix in the mount table.
Also searching for the cygpath \s*( regex leads to the vcygpath function in winsup/utils/path.cc. That appears to be more directly related to the cygpath command (how?). Searching for the \"cygdrive\" regex also reveals that this is a magic string used in many places in the codebase.
All this shows that there is indeed some complexity behind maintaining the POSIX/Win32-style file path mapping in Cygwin but it should be possible to add some basic logic to the Windows Info-ZIP build to handle /cygdrive/ prefixes in its file arguments. The question I have at this point is how does compiling the zip binaries for the Cygwin environment (the shipping configuration) result in proper handling of POSIX-style filenames?
