Saint's Log

Debugging Info-ZIP

We have seen that the Windows Info-ZIP build fails when invoked twice. I have created a sample zip file for debugging this issue along with a hexdump of it (I was glad to see that Cygwin includes hexdump). We observed that using the -sd flag showed a failure when reading the archive.

$ zip -usd ziphelp.zip *.txt
sd: Zipfile name 'ziphelp.zip'
sd: Command line read
sd: Reading archive
        zip warning: expected 3 entries but found 0

zip error: Zip file structure invalid (ziphelp.zip)

The sd: Readin archive line is output by the main function in zip.c. Since there is no other output prefixed by sd, execution must be continuing past the subsequent if-statement and into the readzipfile() function on line 4009.

Generate a Debug Zip Binary

Add debug = 1 to the top of win32\makefile.w32 then run these commands to generate PDBs along with the other binaries.

set LOCAL_ZIP=-DDEBUG
nmake -f win32\makefile.w32

Investigating the Failure

A quick way to set up Visual Studio to debug this issue is to create a new C++ console application then change the command, command arguments, and working directory of the project as shown below.

Pressing F10 should now start debugging the zip binary. The highlights are:

1. The main function in zip.c calls readzipfile
2. readzipfile calls scanzipf_regnew which computes an offset to start looking for the End of Central Directory Record (EOCD) in the zip file by trying to seek 128KiB from the end of the file. In our case, the zip file is less than 128 KiB so scanzipf_regnew seeks to the beginning of the file.
3. scanzipf_regnew scans for the EOCD record signature (PK56 or hex bytes 50 4b 05 06). There is only 1 EOCD signature in the zip file and it is at offset 0x227C. It then (correctly) computes the offset of the start of the data after the EOCDR signature (0x2280) and seeks to that offset in the file.
4. 18 bytes are now read from the EOCDR.
5. The 32-bit offset of the start of the central directory, relative to start of archive is then read from the record and stored in the in_cd_start_offset variable. This is correctly read as 0x218e (the last 4 bytes on line 553 of the zip’s hexdump output).
6. The number of entries is also read from the EOCDR (which is how zip knows to expect 3 entries).
7. scanzipf_regnew now seeks to the first CD entry…
8. then looks for the next signature to process. Interestingly, it finds the EOCDR signature (PK56) and breaks out of the loops (signature scan and zipfile disks). Since it did not find any entries, it then displays the error about the expected number of entries vs the number found and returns an error code to the main function in zip.c, which in turn calls ziperr to print an error message and terminate the process.

The question now is why scanzipf_regnew does not find any entries since there are 3 PK12 and 3 PK34 signatures in the file. Let us inspect the file offsets right after the code seeks to in_cd_start_offset (from step 5 above) and immediately before looking for the next signature to process.

 uzoff_t zipoffset_before_finding_signatures =  zftello(in_file);

This shows a value of 0x0000000000002168 which is not the proper offset in the zip file to seek to. Recall that in_cd_start_offset was 0x218e, which is the second-last byte of line 537 in the zipfile’s hexdump output. Could this be an error in the standard library fseek and ftell functions? TODO: why does the scan fail from an earlier starting point?

Pressing F12 to see the definitions of zftello and zfseeko went to the wrong place! These are not the standard library functions being used. Visual Studio was opening their definitions in tailor.h instead of the actual implementations being called. Turns out zftello and zfseeko are functions implemented in win32/win32i64.c. These comments above zftello raise some huge red flags.

/* 64-bit buffered ftello
 *
 * Win32 does not provide a 64-bit buffered
 * ftell (in the published api anyway) so below provides
 * hopefully close version.
 * We have not gotten _telli64 to work with buffered
 * streams.  Below cheats by using fgetpos improperly and
 * may not work on other ports.
 */

zftello’s comments do not sound any more reassuring:

/* 64-bit buffered fseeko
 *
 * Win32 does not provide a 64-bit buffered
 * fseeko so use _lseeki64 and fflush.  Note
 * that SEEK_CUR can lose track of location
 * if fflush is done between the last buffered
 * io and this call.
 */

Looks like this custom seek/tell code is responsible for the incorrect offsets into the zip file! We can work around this by simply removing these custom implementations.

Windows Specific Bug?

So why didn’t Cygwin’s zip.exe have this issue? Running zip.exe -v shows this compiler/OS description:

Compiled with gcc 4.8.3 for Unix (Cygwin) on Jun 23 2014.

The Cygwin OS name in parenthesis is defined in unix/unix.c only if __CYGWIN__ is defined. However, under this condition, the custom zftello and zfseeko implementations will not be included in the zip sources being compiled! Therefore, the issue does not occur in Cygwin’s distributed zip binary.