—Categories: ZIP

Building zlib for Windows

Ensure these components are installed before building zlib:

  1. The MSVC v140 – VS 2015 C++ build tools (v14.00) individual component in the Visual Studio Installer
  2. The Windows 8.1 SDK from the Windows SDK and emulator archive

Once these are installed, run the commands below in a Visual Studio 2022 Developer Command Prompt to build zlib for Windows. The x64 directory will contain zlibwapi.dll, which can be renamed to zlib.dll according to zlib/readme.txt (from the latest commit as of this post).

git clone https://github.com/madler/zlib
cd zlib/contrib/vstudio/vc14
git switch --detach v1.2.11
msbuild zlibvc.vcxproj /p:Configuration=Release /p:Platform=x64
msbuild zlibvc.vcxproj /p:Configuration=Release /p:Platform=Win32

mkdir C:\dev\software\zlib\win32
copy x86\ZlibDllRelease\zlibwapi.dll C:\dev\software\zlib\win32\zlib.dll
copy x86\ZlibDllRelease\zlibwapi.lib C:\dev\software\zlib\win32\zdll.lib

Why Build zlib?

As part of Tracking Down Missing Headers in LLVM for Windows, I ran into NSIS compiler errors and decide to create a debug build of NSIS to debug them myself since there was no definitive solution online. Turns out, zlib is one of the prereqs for NSIS as per Code / [r7368] /NSIS/branches/WIN64/INSTALL (sourceforge.net).

Unfortunately (or maybe fortunately?), I didn’t see any binaries at zlib. There is a link to the zlib GitHub repo though and zlib/DLL_FAQ.txt at master · madler/zlib (github.com) says to review the zlib site for an alternative download location. Sure enough, it does have a link to zlib for Windows 9x/NT/2000/XP/2003 (DLL version, plus related utilities). That doesn’t inspire much confidence in the binaries though… Might as well build them myself.

Investigating How to Build zlib

Open a Visual Studio Developer Command Prompt then build the project I saw in the docs:

git clone https://github.com/madler/zlib
cd zlib/contrib/vstudio/vc14
msbuild zlibvc.vcxproj

There are other prereqs, apparently:

MSBuild version 17.3.1+2badb37d1 for .NET Framework
Build started 9/29/2022 11:01:01 AM.
Project "D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj" on node 1 (default targets).
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(460,5): error MSB8020: The build tools for Visual Studio 2015 (Platform Toolset = 'v140') cannot b
e found. To build using the v140 build tools, please install Visual Studio 2015 build tools.  Alternatively, you may upgrade to the current Visual Studio tools by selecting the Project menu or right-click the
 solution, and then selecting "Retarget solution". [D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj]
Done Building Project "D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj" (default targets) -- FAILED.


Build FAILED.

This component is 3.63 GB but that’s not the only prereq! This toolset requires the Windows 8.1 SDK! As per visual studio 2019 – VS2019 without Windows 8.1 SDK – Stack Overflow, it needs to be downloaded from the Windows SDK and emulator archive

D:\dev\repos\zlib\contrib\vstudio\vc14> msbuild zlibvc.vcxproj
MSBuild version 17.3.1+2badb37d1 for .NET Framework
Build started 9/29/2022 11:23:13 AM.
Project "D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj" on node 1 (default targets).
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Platforms\Win32\PlatformToolsets\v140\Toolset.targets(34,5): error MSB8036: The Windows SDK version 8.1 was not found. Install the required version of Wi
ndows SDK or change the SDK version in the project property pages or by right-clicking the solution and selecting "Retarget solution". [D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj]
Done Building Project "D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj" (default targets) -- FAILED.


Build FAILED.

After all that effort, the reward is the error below. Searching for “bat” is helpful: Issues · madler/zlib (github.com)

D:\dev\repos\zlib\contrib\vstudio\vc14> msbuild zlibvc.vcxproj
MSBuild version 17.3.1+2badb37d1 for .NET Framework
Build started 9/29/2022 11:51:10 AM.
Project "D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj" on node 1 (default targets).
PrepareForBuild:
  Creating directory "x86\ZlibDllDebug\Tmp\".
  Creating directory "x86\ZlibDllDebug\Tmp\zlibvc.tlog\".
InitializeBuildStatus:
  Creating "x86\ZlibDllDebug\Tmp\zlibvc.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
PreBuildEvent:
  cd ..\..\masmx86
  bld_ml32.bat
  :VCEnd
  The system cannot find the path specified.
  'bld_ml32.bat' is not recognized as an internal or external command,
  operable program or batch file.
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(123,5): error MSB3073: The command "cd ..\..\masmx86 [D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj]
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(123,5): error MSB3073: bld_ml32.bat [D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj]
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(123,5): error MSB3073: :VCEnd" exited with code 9009. [D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj]
Done Building Project "D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj" (default targets) -- FAILED.


Build FAILED.

To find out which commit removed this batch file, run this from the root of the repo:

git log --full-history -1 -- contrib/masmx86/bld_ml32.bat

This looks like really bad development on the zlib repo. Removing scripts without removing outdated documentation, much less documenting the new way to build. This is Please fix 1.2.12 compile · Issue #631 · madler/zlib (github.com). Instead of using the workarounds there, just build 1.2.11 and let the zlib folks deal with that mess.

git switch --detach v1.2.11
cd zlib/contrib/vstudio/vc14
msbuild zlibvc.vcxproj

Linking fails with error LNK2026:

match686.obj : error LNK2026: module unsafe for SAFESEH image. [D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj]
inffas32.obj : error LNK2026: module unsafe for SAFESEH image. [D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj]
     Creating library x86\ZlibDllDebug\zlibwapi.lib and object x86\ZlibDllDebug\zlibwapi.exp
x86\ZlibDllDebug\zlibwapi.dll : fatal error LNK1281: Unable to generate SAFESEH image. [D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj]
Done Building Project "D:\dev\repos\zlib\contrib\vstudio\vc14\zlibvc.vcxproj" (default targets) -- FAILED.

This option is on by default as per /SAFESEH (Image has Safe Exception Handlers) | Microsoft Learn. However, it only affects the x86 platform. I’m only interested in x64 so this command suffices:

msbuild zlibvc.vcxproj /p:Configuration=Release /p:Platform=x64

The compiler generates the zlibwapi DLL in the x64 directory. Unfortunately, NSIS requires a Win32 build so run this as well:

msbuild zlibvc.vcxproj /p:Configuration=Release /p:Platform=Win32

Deploy the binaries to the desired location, e.g.

mkdir C:\dev\software\zlib\win32
copy x86\ZlibDllRelease\zlibwapi.dll C:\dev\software\zlib\win32\zlib.dll
copy x86\ZlibDllRelease\zlibwapi.dll C:\dev\software\zlib\x64\zlib.dll
—Categories: C, ZIP

Debugging Info-ZIP

We have seen that the Windows Info-ZIP build fails when invoked twice. I have created a sample zip file for debugging this issue along with a hexdump of it (I was glad to see that Cygwin includes hexdump). We observed that using the -sd flag showed a failure when reading the archive.

$ zip -usd ziphelp.zip *.txt
sd: Zipfile name 'ziphelp.zip'
sd: Command line read
sd: Reading archive
        zip warning: expected 3 entries but found 0

zip error: Zip file structure invalid (ziphelp.zip)

The sd: Readin archive line is output by the main function in zip.c. Since there is no other output prefixed by sd, execution must be continuing past the subsequent if-statement and into the readzipfile() function on line 4009.

Generate a Debug Zip Binary

Add debug = 1 to the top of win32\makefile.w32 then run these commands to generate PDBs along with the other binaries.

set LOCAL_ZIP=-DDEBUG
nmake -f win32\makefile.w32

Investigating the Failure

A quick way to set up Visual Studio to debug this issue is to create a new C++ console application then change the command, command arguments, and working directory of the project as shown below.

Setting Visual C++ Project Properties for Debugging Zip.exe

Pressing F10 should now start debugging the zip binary. The highlights are:

  1. The main function in zip.c calls readzipfile
  2. readzipfile calls scanzipf_regnew which computes an offset to start looking for the End of Central Directory Record (EOCD) in the zip file by trying to seek 128KiB from the end of the file. In our case, the zip file is less than 128 KiB so scanzipf_regnew seeks to the beginning of the file.
  3. scanzipf_regnew scans for the EOCD record signature (PK56 or hex bytes 50 4b 05 06). There is only 1 EOCD signature in the zip file and it is at offset 0x227C. It then (correctly) computes the offset of the start of the data after the EOCDR signature (0x2280) and seeks to that offset in the file.
  4. 18 bytes are now read from the EOCDR.
  5. The 32-bit offset of the start of the central directory, relative to start of archive is then read from the record and stored in the in_cd_start_offset variable. This is correctly read as 0x218e (the last 4 bytes on line 553 of the zip’s hexdump output).
  6. The number of entries is also read from the EOCDR (which is how zip knows to expect 3 entries).
  7. scanzipf_regnew now seeks to the first CD entry…
  8. then looks for the next signature to process. Interestingly, it finds the EOCDR signature (PK56) and breaks out of the loops (signature scan and zipfile disks). Since it did not find any entries, it then displays the error about the expected number of entries vs the number found and returns an error code to the main function in zip.c, which in turn calls ziperr to print an error message and terminate the process.

The question now is why scanzipf_regnew does not find any entries since there are 3 PK12 and 3 PK34 signatures in the file. Let us inspect the file offsets right after the code seeks to in_cd_start_offset (from step 5 above) and immediately before looking for the next signature to process.

 uzoff_t zipoffset_before_finding_signatures =  zftello(in_file);

This shows a value of 0x0000000000002168 which is not the proper offset in the zip file to seek to. Recall that in_cd_start_offset was 0x218e, which is the second-last byte of line 537 in the zipfile’s hexdump output. Could this be an error in the standard library fseek and ftell functions? TODO: why does the scan fail from an earlier starting point?

Pressing F12 to see the definitions of zftello and zfseeko went to the wrong place! These are not the standard library functions being used. Visual Studio was opening their definitions in tailor.h instead of the actual implementations being called. Turns out zftello and zfseeko are functions implemented in win32/win32i64.c. These comments above zftello raise some huge red flags.

/* 64-bit buffered ftello
 *
 * Win32 does not provide a 64-bit buffered
 * ftell (in the published api anyway) so below provides
 * hopefully close version.
 * We have not gotten _telli64 to work with buffered
 * streams.  Below cheats by using fgetpos improperly and
 * may not work on other ports.
 */

zftello’s comments do not sound any more reassuring:

/* 64-bit buffered fseeko
 *
 * Win32 does not provide a 64-bit buffered
 * fseeko so use _lseeki64 and fflush.  Note
 * that SEEK_CUR can lose track of location
 * if fflush is done between the last buffered
 * io and this call.
 */

Looks like this custom seek/tell code is responsible for the incorrect offsets into the zip file! We can work around this by simply removing these custom implementations.

Windows Specific Bug?

So why didn’t Cygwin’s zip.exe have this issue? Running zip.exe -v shows this compiler/OS description:

Compiled with gcc 4.8.3 for Unix (Cygwin) on Jun 23 2014.

The Cygwin OS name in parenthesis is defined in unix/unix.c only if __CYGWIN__ is defined. However, under this condition, the custom zftello and zfseeko implementations will not be included in the zip sources being compiled! Therefore, the issue does not occur in Cygwin’s distributed zip binary.