—Categories: Installers, Windows

Building NSIS

One of the show stoppers in Tracking Down Missing Headers in LLVM for Windows was the NSIS Internal compiler error #12345: error mmapping datablock to 33555089. This issue is more common than I would expect for an internal compiler error, judging from I get “Internal compiler error #12345” when compiling large installers – NSIS Forums (nsis-dev.github.io). Before engaging some other folks about this, I decide to first build a debuggable NSIS to get a sense of what is happening. This can be done by downloading the NSIS 3.08 source code and using tar with the -j flag to filter the archive through bzip2.

tar xjf nsis-3.08-src.tar.bz2

Checking Out Sources from the Repo

Alternatively, subversion can be used to check out the source code. Been a while since I touched svn. Thankfully, we can use git-svn instead.

git svn clone https://svn.code.sf.net/p/nsis/code/ --stdlayout --prefix svn/

This command fails after about an hour, and git svn clone https://svn.code.sf.net/p/nsis/code/ times out after getting r960. Not sure why these folks aren’t on GitHub.

Building the Sources

NSIS: [r7368] /NSIS/branches/WIN64/INSTALL (sourceforge.net) lists SCons are a requirement. Never heard of it before so I’m relieved to discover that it is on GitHub SCons/scons: SCons – a software construction tool (github.com) and is easy to install. Unfortunately, I did not actually want the --user option on my machine.

D:\dev\repos> python -m pip install --user scons
Collecting scons
  Downloading SCons-4.4.0-py3-none-any.whl (4.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.2/4.2 MB 11.7 MB/s eta 0:00:00
Requirement already satisfied: setuptools in c:\python310\lib\site-packages (from scons) (58.1.0)
Installing collected packages: scons
  WARNING: The scripts scons-configure-cache.exe, scons.exe and sconsign.exe are installed in '%APPDATA%\Python\Python310\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed scons-4.4.0

[notice] A new release of pip available: 22.2 -> 22.2.2
[notice] To update, run: python.exe -m pip install --upgrade pip

D:\dev\repos> python -m pip uninstall scons
Found existing installation: SCons 4.4.0
Uninstalling SCons-4.4.0:
  Would remove:
    %APPDATA%\python\python310\scripts\scons-configure-cache.exe
    %APPDATA%\python\python310\scripts\scons.exe
    %APPDATA%\python\python310\scripts\sconsign.exe
    %APPDATA%\python\python310\site-packages\scons-4.4.0.dist-info\*
    %APPDATA%\python\python310\site-packages\scons\*
    %APPDATA%\python\scons-time.1
    %APPDATA%\python\scons.1
    %APPDATA%\python\sconsign.1
Proceed (Y/n)? y
  Successfully uninstalled SCons-4.4.0

D:\dev\repos> python -m pip install scons
Collecting scons
  Using cached SCons-4.4.0-py3-none-any.whl (4.2 MB)
Requirement already satisfied: setuptools in c:\python310\lib\site-packages (from scons) (58.1.0)
Installing collected packages: scons
Successfully installed scons-4.4.0

D:\dev\repos> where scons
C:\Python310\Scripts\scons.exe

The next prerequisite is zlib. Instead of downloading binaries from the unsecured site linked to, I decided to build the zlib sources myself. I only built the 64-bit version but turns out they are serious about setting ZLIB_W32:

D:\dev\repos\nsis\nsis-3.08-src> scons UNICODE=yes
scons: Reading SConscript files ...
Mkdir("build\urelease\config")
WARNING: VER_PACKED not set, defaulting to 0x03007666!
Delete("nsis-29-Sep-2022.cvs")
Delete(".instdist")
Delete(".test")
Using Microsoft tools configuration (14.3)
Checking for memset requirement... yes
Checking for memcpy requirement... no
Checking for C library gdi32... yes
Checking for C library user32... yes
Checking for C library pthread... no
Checking for C library iconv... no
Checking for C library shlwapi... yes
Checking for C library oleaut32... yes
Checking for C library version... yes
Checking for C library shell32... yes
Checking for C library version... yes
Please specify folder of zlib for Win32 via ZLIB_W32

Copying the DLL is not sufficient. To see why the error below occurs, consult config.log.

...
Checking for C library zdll... no
Checking for C library z... no
zlib (win32) is missing!

For example, config.log ends with C:\dev\software\zlib\win32\zlib.h(34): fatal error C1083: Cannot open include file: 'zconf.h': No such file or directory because I copied only zlib.h. I notice in config.log that it’s trying to also link using zdll.lib. Fix this by running:

cd /d D:\dev\repos\zlib
copy zlib.h C:\dev\software\zlib\win32\
copy zconf.h C:\dev\software\zlib\win32\
copy contrib\vstudio\vc14\x86\ZlibDllRelease\zlibwapi.lib C:\dev\software\zlib\win32\zdll.lib
set ZLIB_W32=C:\dev\software\zlib\win32\

Compilation now fails due to unresolved external symbols:

link /nologo /nocoffgrpinfo /map /subsystem:console,5.01 /STACK:2097152 /OUT:build\urelease\makensis\makensis.exe /LIBPATH:C:\dev\software\zlib\win32 gdi32.lib user32.lib shlwapi.lib oleaut32.lib version.lib shell32.lib version.lib zdll.lib build\urelease\makensis\build.obj build\urelease\makensis\clzma.obj build\urelease\makensis\crc32.obj build\urelease\makensis\DialogTemplate.obj build\urelease\makensis\dirreader.obj build\urelease\makensis\fileform.obj build\urelease\makensis\growbuf.obj build\urelease\makensis\icon.obj build\urelease\makensis\lang.obj build\urelease\makensis\lineparse.obj build\urelease\makensis\makenssi.obj build\urelease\makensis\manifest.obj build\urelease\makensis\mmap.obj build\urelease\makensis\Plugins.obj build\urelease\makensis\ResourceEditor.obj build\urelease\makensis\ResourceVersionInfo.obj build\urelease\makensis\BinInterop.obj build\urelease\makensis\script.obj build\urelease\makensis\scriptpp.obj build\urelease\makensis\ShConstants.obj build\urelease\makensis\strlist.obj build\urelease\makensis\tokens.obj build\urelease\makensis\tstring.obj build\urelease\makensis\utf.obj build\urelease\makensis\util.obj build\urelease\makensis\winchar.obj build\urelease\makensis\writer.obj build\urelease\makensis\bzip2\blocksort.obj build\urelease\makensis\bzip2\bzlib.obj build\urelease\makensis\bzip2\compress.obj build\urelease\makensis\bzip2\huffman.obj build\urelease\makensis\7zip\7zGuids.obj build\urelease\makensis\7zip\7zip\Common\OutBuffer.obj build\urelease\makensis\7zip\7zip\Common\StreamUtils.obj build\urelease\makensis\7zip\7zip\Compress\LZ\LZInWindow.obj build\urelease\makensis\7zip\7zip\Compress\LZMA\LZMAEncoder.obj build\urelease\makensis\7zip\7zip\Compress\RangeCoder\RangeCoderBit.obj build\urelease\makensis\7zip\Common\Alloc.obj build\urelease\makensis\7zip\Common\CRC.obj
build.obj : error LNK2019: unresolved external symbol _deflate referenced in function "public: virtual int __thiscall CZlib::Compress(bool)" (?Compress@CZlib@@UAEH_N@Z)
build.obj : error LNK2019: unresolved external symbol _deflateEnd referenced in function "public: virtual int __thiscall CZlib::End(void)" (?End@CZlib@@UAEHXZ)
build.obj : error LNK2019: unresolved external symbol _deflateInit2_ referenced in function "public: virtual int __thiscall CZlib::Init(int,unsigned int)" (?Init@CZlib@@UAEHHI@Z)
build\urelease\makensis\makensis.exe : fatal error LNK1120: 3 unresolved externals
scons: *** [build\urelease\makensis\makensis.exe] Error 1120
scons: building terminated because of errors.

Run dumpbin /headers zlibwapi.lib to examine the symbols in the lib file. Each of these does appear in a slightly different decorated form. For the declaration ZEXTERN int ZEXPORT deflateEnd OF((z_streamp strm)); in zlib.h we see the Name mangling below. This looks like __stdcall, coming from the expansion of ZEXPORT in zconf.h.

  Version      : 0
  Machine      : 14C (x86)
  TimeDateStamp: 6336126F Thu Sep 29 15:47:27 2022
  SizeOfData   : 0000001B
  DLL name     : zlibwapi.dll
  Symbol name  : _deflateEnd@4
  Type         : code
  Name type    : ordinal
  Ordinal      : 6

Just by chance, I CTRL+click on the deflateEnd method on the line int ret = deflateEnd(stream); in nsis-3.08-src/Source/czlib.h and it opens ZLIB.H in nsis-3.08-src/Source/zlib/. This file has been here the whole time, with the other header file I manually copied (and others that I might have needed to)! This header is directly included by Source\exehead\fileform.c, for example, so the build will fail if this folder is removed. (is this a bug though?)

In the NSIS sources, ZEXPORT is defined without a value. The link error is therefore caused by the use of _cdecl in the NSIS sources and __stdcall in the zlib source code I built. I end up changing the latter and rebuilding since the change in the former doesn’t seem to fix the build error and I don’t have time to investigate that. More specifically, I change line 355 of zconf.h to define ZEXPORT _cdecl. Now the build succeeds and this command create an installation:

scons UNICODE=yes
scons PREFIX="D:\dev\repos\nsis\local-install" install

I can run D:\dev\repos\nsis\local-install\makensisw.exe once but it is then blocked by Windows Defender. I guess I’ll have to review Troubleshoot problems with attack surface reduction rules. To create a debug build, use this command line:

scons UNICODE=yes DEBUG=yes
scons DEBUG=yes PREFIX="D:\dev\repos\nsis\local-install-debug" install

According to Enable attack surface reduction (ASR) rules, this command in an admin powershell should do the trick:

Add-MpPreference -AttackSurfaceReductionOnlyExclusions "D:\dev\repos\nsis\local-install\makensisw.exe"

Add-MpPreference -AttackSurfaceReductionOnlyExclusions "D:\dev\repos\nsis\local-install\makensis.exe"

Article info

Categories

Tags

Leave a Reply

Your email address will not be published. Required fields are marked *