Saint's Log

curl Failures in Cygwin

If curl fails with error 23 in Cygwin, it is likely that the curl command shipped with Windows is running in the Cygwin terminal instead of the curl binary distributed with Cygwin. To install the Cywin curl command, run the setup executable with these flags:

setup-x86_64.exe -q -P curl

Background

One of the issues I looked into this week was running an AQAvit™ Verification test on Windows and therefore in Cygwin. Here are the environment variables I set and the commands I executed as outlined on the AQAvit™ Verification page.

export TEST_JDK_HOME=/cygdrive/c/java/binaries/jdk/aarch64/jdk-17.0.6+10
export USE_TESTENV_PROPERTIES=true
export JDK_VERSION=17
export BUILD_LIST=openjdk

git clone --depth 1 --branch v0.9.1-release https://github.com/adoptium/aqa-tests.git

cd aqa-tests
./get.sh
./compile.sh

The last command above (compilation) failed on my Surface Pro X with this error:

...
Buildfile: C:\java\aqa\aqa-tests\TKG\scripts\build_tools.xml

build:

clean:
   [delete] Deleting directory C:\java\aqa\aqa-tests\TKG\bin

init:
    [mkdir] Created dir: C:\java\aqa\aqa-tests\TKG\bin

getDependentLibs:
     [exec] --------------------------------------------
     [exec] path is set to /cygdrive/c/java/aqa/aqa-tests/TKG/../TKG/lib
     [exec] task is set to default
     [exec] dependencyList is set to json_simple
     [exec] --------------------------------------------
     [exec] Starting download third party dependent jars
     [exec] --------------------------------------------
     [exec] downloading dependent third party jars to /cygdrive/c/java/aqa/aqa-tests/TKG/../TKG/lib
     [exec] downloading https://repo1.maven.org/maven2/com/googlecode/json-simple/json-simple/1.1.1/json-simple-1.1.1.jar
     [exec]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
     [exec]                                  Dload  Upload   Total   Spent    Left  Speed
     [exec]
     [exec]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Failed to open the file Warning: /cygdrive/c/java/aqa/aqa-tests/TKG/../TKG/lib/json-simple.jar: No
     [exec] Warning: such file or directory
     [exec]
     [exec]   5 23931    5  1371    0     0   4483      0  0:00:05 --:--:--  0:00:05  4509curl: (23) Failure writing output to destination
     [exec] ERROR: downloading https://repo1.maven.org/maven2/com/googlecode/json-simple/json-simple/1.1.1/json-simple-1.1.1.jar failed, return code: 5888

BUILD FAILED
C:\java\aqa\aqa-tests\TKG\scripts\build_tools.xml:58: The following error occurred while executing this line:
C:\java\aqa\aqa-tests\TKG\scripts\getDependencies.xml:27: exec returned: 2

I had core.autocrlf set to true when I initially checked out the aqa-tests repo so I suspected the culprit to be a file that didn’t get converted to LF when I ran git add --renormalize as suggested by How do I re-checkout all files in Git to convert from CRLF to LF? – Stack Overflow. I would get errors like “/cygdrive/c/java/aqa/aqa-tests/TKG/scripts/getTestenvProperties.sh: line 14: $’\r’: command not found” and ended up manually changing the line endings using VS Code since there were only 3 files that needed to be changed but I digress…

Looks like curl is being invoked by TKG/getDependencies.pl · adoptium/TKG · GitHub. A search for perl return code 5888 leads to an issue about a Strange error while downloading perl: 5888 · Issue #748 · gugod/App-perlbrew · GitHub. This is helpful because it links to a PR Implement a standard set of error checking of system(). · gugod/App-perlbrew@42415bc · GitHub ensuring that the actual error from curl is displayed to the user. This error is obtaining by shifting the error code right by 8 bits. We get that 5888 >> 8 = 23. The description of error 23 at libcurl – Error Codes is not particularly insightful. However, it does lead me to manually test the curl command and see whether the issue is with curl or with all the build scripts. Sure enough, the curl command fails when given a fully qualified Cygwin path!

$ curl -k -o  /c/java/aqa/aqa-tests/TKG/lib/file.jar https://repo1.maven.org/maven2/com/googlecode/json-simple/json-simple/1.1.1/json-simple-1.1.1.jar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Failed to open the file /c/java/aqa/aqa-tests/TKG/lib/file.jar: No
Warning: such file or directory
  5 23931    5  1371    0     0   4660      0  0:00:05 --:--:--  0:00:05  4679
curl: (23) Failure writing output to destination

$ curl -k -o  ../file.jar https://repo1.maven.org/maven2/com/googlecode/json-simple/json-simple/1.1.1/json-simple-1.1.1.jar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 23931  100 23931    0     0  84394      0 --:--:-- --:--:-- --:--:-- 84861

Which curl is running? Turns out it’s the native Windows curl command!

$ which curl
/cygdrive/c/WINDOWS/system32/curl

$ curl --version
curl 7.83.1 (Windows) libcurl/7.83.1 Schannel
Release-Date: 2022-05-13
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS HSTS IPv6 Kerberos Largefile NTLM SPNEGO SSL SSPI UnixSockets

I realize there must be a native Cygwin curl package so I install it using this command: \software\setup-x86_64.exe -q -P curl

I end up needing to open a new shell to get it to be picked up when building. That fixes the problem (this version successfully writes the downloaded file to disk)! Here’s the version information for the curl utility installed by Cygwin:

$ which curl
/usr/bin/curl

$ /usr/bin/curl --version
curl 7.87.0 (x86_64-pc-cygwin) libcurl/7.87.0 OpenSSL/1.1.1s zlib/1.2.13 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.4 libpsl/0.21.2 (+libidn2/2.3.4) libssh2/1.10.0 nghttp2/1.51.0 libgsasl/2.2.0
Release-Date: 2022-12-21
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli Debug gsasl GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL threadsafe TLS-SRP TrackMemory UnixSockets zstd