—Categories: CMake, Elmer

Building the Elmer Install Folder

My post on How to Build Elmer on Windows has a succinct list of instructions but when I first built the Elmer source code (on Windows), I was unsure how to run the binaries. Running ElmerGUI, for example, failed because qt5.dll could not be found. This post details the process I used to figure out how to create a usable Elmer installation.

I started by going through the generated Makefile. It contained targets like install, install/local, and install/strip. I started with install/local, fearing that perhaps these targets would attempt to install the binaries into program files.

Installing only the local directory...
-- Install configuration: "RelWithDebInfo"
-- Installing: C:/dev/repos/fem/install/bin/libopenblas.dll.a
CMake Error at cmake_install.cmake:45 (file):
  file INSTALL cannot find "C:/dev/repos/fem/elmerfem/../bundle_msys2/bin":
  No error.


make: *** [Makefile:142: install/local] Error 1

The error at cmake_install.cmake:45 is from ElmerCPack.cmake. I did not even know about the existence of the CPack tool. Git blame points to commit b9914082 as the source of the failing command. The strange thing about that commit is that there is no mention of what is supposed to create the bundle folders (despite the callout that the change “Relies on selected bundled files selected prior to installation”).

A workaround I tried was to use this define when running cmake: CPACK_BUNDLE_EXTRA_WINDOWS_DLLS:BOOL=FALSE. That avoided the build error but did not result in a usable installation. ElmerGUI’s windows_bundle.cmake needs this to be set to TRUE to package the Qt5 binaries into the install folder. Strangely enough, ElmerGUILogger’s windows_bundle.cmake does not have this logic for Qt5. This is likely because ElmerGUI would already have installed the right files into the bin folder but this looks like a bug.

Another work-around I tried was to manually create the folders expected by the build. The install target then succeeded but the necessary binaries were not copied to the install folder.

mkdir -p bundle_msys2/bin
mkdir -p bundle_qt5/bin
mkdir -p platforms

This is when I dig around and find that only ElmerGUILogger and ElmerClips include windows_bundle.cmake. Hmm… the latter looks promising but doesn’t look up to date either since it requires Qt4. More exploring of the Makefile – the package target looks interesting but fails because NSIS is not installed.

...
[ 97%] Built target ElmerGUI_autogen
[100%] Built target ElmerGUI
Run CPack packaging tool...
CPack Error: Cannot find NSIS compiler makensis: likely it is not installed, or not in your PATH
CPack Error: Could not read NSIS registry value. This is usually caused by NSIS not being installed. Please install NSIS from http://nsis.sourceforge.net
CPack Error: Cannot initialize the generator NSIS
make: *** [Makefile:71: package] Error 1

I wonder if we couldn’t just use NSIS for the MSYS environment:

$ pacman -Ss nsis
...
mingw64/mingw-w64-x86_64-nsis 3.06.1-1
    Windows installer development tool (mingw-w64)
mingw64/mingw-w64-x86_64-nsis-nsisunz 1.0-2
    NSIS plugin which allows you to extract files from ZIP archives (mingw-w64)
...

$ pacman -S mingw64/mingw-w64-x86_64-nsis

Now we can create an Elmer installer by running make package. Unfortunately, that turns out to be insufficient. My next idea is to compare the binaries from the installed. This turned out to be easier when using ls -R1 to output only the file names and in 1 column only. Some obvious differences are that the build in Program Files has a bin folder containing the Qt and vtk binaries (as well as a stripped gfortran).

Qt5Core.dll
Qt5Gui.dll
Qt5OpenGL.dll
Qt5PrintSupport.dll
Qt5Script.dll
Qt5Svg.dll
Qt5Widgets.dll
Qt5Xml.dll
...
libvtkChartsCore-8.2.dll
libvtkChartsCorePython38D-8.2.dll
libvtkCommonColor-8.2.dll
libvtkCommonColorPython38D-8.2.dll
libvtkCommonComputationalGeometry-8.2.dll
libvtkCommonComputationalGeometryPython38D-8.2.dll
libvtkCommonCore-8.2.dll
libvtkCommonCorePython38D-8.2.dll
libvtkCommonDataModel-8.2.dll

The Qt binaries certainly look like the output of windows_bundle.cmake (found this time by a search for “vtk”) but it’s still not clear how this file would be included in the build. I’m using VSCode to search for “windows_bundle” and only 2 of the 3 references in the codebase were showing up (on my desktop). Looking for “ElmerGUILogger” then revealed yet another reference. Such a waste of time! Not cool VSCode, not cool. It’s included in ElmerGUI/CMakeLists.txt. So I probably only need to define WIN32. But why does the WIN32 code run if I add some statements to that IF block?

Some searching (TODO: put bing searches here) leads to indications that there might be an error in CMake where WIN32 is not defined. Seeing signs that MSYS can be in Cywgin? Trying to get to the bottom of why WIN32 is not respected by CMake, I review ElmerGUI/CMakeLists.txt again. It adds the netgen subdirectory. Interestingly, ElmerGUI/netgen/README points out that install\lib\ElmerGUI\ngcore\libng.a is the unix library and that the win32 extension should be .lib. Biggest sign I’ve seen so far that something is really off. At this point, I remember seeing a .a file in the install folder – and that seemed strange for a lib folder since I expected a DLL. This hypothesis fails though because the working Elmer installation also has the file “C:\Program Files\Elmer 9.0-Release\lib\ElmerGUI\ngcore\libng.a“.

The sure way to verify that the WIN32 include is working is to introduce an error into windows_bundle.cmake, e.g. by change the first IF into the unknown IF2. Even better, notice that the “Qt5 Windows packaging” message is correctly displayed. Perhaps the FIND_FILE command should have a REQUIRED option now that it is supported. Adding that doesn’t fail so reexamine the INSTALL command. Since windows_bundle.cmake uses a relative path for the DESTINATION, it is interpreted relative to the value of the CMAKE_INSTALL_PREFIX variable.

At this point, I take a step back and search for cmake install component. My suspision is that there is something about the elmergui COMPONENT specified in the INSTALL command. I discover from Understanding the CMake `COMPONENT` keyword in the `install` command – Code – CMake Discourse that cmake --install is a thing. It seems to do the same things. Unfortunately, it doesn’t copy the files I need either.

How about zooming into the windows_bundle.cmake file and outputting the list of files installed after the install command. The new message says the files were installed. However, they aren’t in the install folder! I need an explanation for this behavior. So let’s open Process Explorer and see which paths are actually used. These requests show up using the Qt5 path filter:

These files exist on disk (despite the Result column in process explorer having the value INVALID DEVICE REQUEST)! For a better understanding of what is happening, I change the bin path used by the install command to dbgcmake since there are multiple bin folders (under install and also in the MinGW installation). So such path shows up in Process Explorer when using a path filter for dbgcmake. This means I shouldn’t be expecting this files to be written by cmake at this point. In fact, running grep -Rin dbgcmake shows that build/ElmerGUI/cmake_install.cmake now contains this snippet (thereby verifying that windows_bundle.cmake is used to generate cmake_install.cmake).

if("x${CMAKE_INSTALL_COMPONENT}x" STREQUAL "xelmerguix" OR NOT CMAKE_INSTALL_COMPONENT)
  file(INSTALL DESTINATION "${CMAKE_INSTALL_PREFIX}/dbgcmake" TYPE FILE FILES
    "D:/dev/Software/msys64/mingw64/bin/Qt5Core.dll"
    "D:/dev/Software/msys64/mingw64/bin/Qt5Gui.dll"
    "D:/dev/Software/msys64/mingw64/bin/Qt5OpenGL.dll"
    "D:/dev/Software/msys64/mingw64/bin/Qt5Script.dll"
    "D:/dev/Software/msys64/mingw64/bin/Qt5Xml.dll"
    "D:/dev/Software/msys64/mingw64/bin/Qt5Svg.dll"
    "D:/dev/Software/msys64/mingw64/bin/Qt5Widgets.dll"
    "D:/dev/Software/msys64/mingw64/bin/Qt5PrintSupport.dll"
    )
endif()

So back to the question of why windows_bundle.cmake is not invoked by default – let us dig into the CMake sources for clues. Renaming it outputs an error for the include statement meaning the file is being included! Sigh… I end up poking around some CMake sources to see where WIN32 is set anyway.:

At this point, I’m down to simply finding out how to print CMake call stacks to confirm that this file is indeed included. The StackOverflow question about how to trace cmakelists has the big reveal: use the cmake –trace option!

date; time cmake -G "MSYS Makefiles" -DWITH_ELMERGUI:BOOL=TRUE  -DWITH_MPI:BOOL=FALSE -DCMAKE_INSTALL_PREFIX=../install  -DCMAKE_Fortran_COMPILER=d:/dev/Software/msys64/mingw64/bin/gfortran.exe  -DQWT_INCLUDE_DIR=d:/dev/Software/msys64/mingw64/include/qwt-qt5/  -DWIN32:BOOL=TRUE --trace-expand ../elmerfem > build.txt 2>&1; date;

I immediately notice that the install command of interest is skipped because CPACK_BUNDLE_EXTRA_WINDOWS_DLLS is not set to TRUE! Setting it to FALSE earlier then seeing a build failure gave the impression that it was set to TRUE everywhere (especially given that ElmerCPack.cmake sets it to TRUE) but that happens after windows_bundle.cmake has been evaluated! Here’s the final required command line:

date; time \
cmake -G "MSYS Makefiles" \
 -DWITH_ELMERGUI:BOOL=TRUE \
 -DWITH_MPI:BOOL=FALSE \
 -DCMAKE_INSTALL_PREFIX=../install \
 -DCMAKE_Fortran_COMPILER=d:/dev/Software/msys64/mingw64/bin/gfortran.exe \
 -DQWT_INCLUDE_DIR=d:/dev/Software/msys64/mingw64/include/qwt-qt5/ \
 -DWIN32:BOOL=TRUE \
 -DCPACK_BUNDLE_EXTRA_WINDOWS_DLLS:BOOL=TRUE \
 ../elmerfem

There are lessons there about making assumptions but the biggest takeaway for me is the need for (and existence) of tracing capabilities in CMake. Enabling tracing made it so easy to figure out exactly what was broken – the variable did not have the value I expected and I simply needed to define it! Running make install now results in new errors haha!

The code execution cannot proceed because libgcc_s_seh-1.dll was not found. Reinstalling the program may fix this problem. This error dialog is then followed by others for qwt-qt5.dll, libstdc++-6.dll, and libdouble-conversion.dll respectively. To see which other binaries are required, manually copy these 4 binaries from d:\dev\Software\msys64\mingw64\bin to install/bin. These are the other missing binaries that show up in error dialogs:

libwinpthread-1.dll
libicuin69.dll
libicuuc69.dll
libpcre2-16-0.dll
libharfbuzz-0.dll
libmd4c.dll
libmd4c.dll
libpng16-16.dll
zlib1.dll
libzstd.dll
libfreetype-6.dll
libgraphite2.dll
libintl-8.dll
libglib-2.0-0.dll
libicudt69.dll
libiconv-2.dll
libbz2-1.dll
libbrotlidec.dll
libbrotlicommon.dll
libpcre-1.dll

There are so many missing binaries that I wonder if building the other components might be necessary. I first manually copy them from the bin folder to see if ElmerGUI can load but that now fails with the error This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

The working installation from the downloaded public Elmer installer has a qwindows.dll in the bin/platforms directory. This file exists on my system as D:\dev\Software\msys64\mingw64\share\qt5\plugins\platforms\qwindows.dll. At this point, I’m really wondering how on earth they are gathering all these files. Hmm, maybe they build using their docker image? Didn’t think of seeing how their docker image is set up. One final try to set up the qwindows.dll first and see what happens.

cp /mingw64/share/qt5/plugins/platforms/qwindows.dll ../install/bin/platforms/

Lo and behold! ElmerGUI loads successfully! Whether it actually works… well, I will revisit that after I learn how to use Elmer/ElmerGUI :). Actually, turns out ElmerSolver.exe doesn’t start. Here is the fix for the 3 binaries that cause error dialogs:

# binaries required by ElmerSolver
cp /mingw64/bin/libgfortran-5.dll ../install/bin/
cp /mingw64/bin/libopenblas.dll ../install/bin/
cp /mingw64/bin/libquadmath-0.dll ../install/bin/

The final set of commands to deploy these binaries is therefore:

# binaries required by ElmerGUI
cp /mingw64/bin/libgcc_s_seh-1.dll ../install/bin/
cp /mingw64/bin/libdouble-conversion.dll ../install/bin/
cp /mingw64/bin/qwt-qt5.dll ../install/bin/
cp /mingw64/bin/libstdc++-6.dll ../install/bin/
cp /mingw64/bin/libwinpthread-1.dll ../install/bin/
cp /mingw64/bin/libicuin69.dll ../install/bin/
cp /mingw64/bin/libicuuc69.dll ../install/bin/
cp /mingw64/bin/libpcre2-16-0.dll ../install/bin/
cp /mingw64/bin/libharfbuzz-0.dll ../install/bin/
cp /mingw64/bin/libmd4c.dll ../install/bin/
cp /mingw64/bin/libpng16-16.dll ../install/bin/
cp /mingw64/bin/zlib1.dll ../install/bin/
cp /mingw64/bin/libzstd.dll ../install/bin/
cp /mingw64/bin/libfreetype-6.dll ../install/bin/
cp /mingw64/bin/libgraphite2.dll ../install/bin/
cp /mingw64/bin/libintl-8.dll ../install/bin/
cp /mingw64/bin/libglib-2.0-0.dll ../install/bin/
cp /mingw64/bin/libicudt69.dll ../install/bin/
cp /mingw64/bin/libiconv-2.dll ../install/bin/
cp /mingw64/bin/libbz2-1.dll ../install/bin/
cp /mingw64/bin/libbrotlidec.dll ../install/bin/
cp /mingw64/bin/libbrotlicommon.dll ../install/bin/
cp /mingw64/bin/libpcre-1.dll ../install/bin/
cp /mingw64/share/qt5/plugins/platforms/qwindows.dll ../install/bin/platforms/

# binaries required by ElmerSolver
cp /mingw64/bin/libgfortran-5.dll ../install/bin/
cp /mingw64/bin/libopenblas.dll ../install/bin/
cp /mingw64/bin/libquadmath-0.dll ../install/bin/

At this point, ElmerSolver starts up successfully but outputs an error about missing ELMERSOLVER_STARTINFO. This is an error from ElmerSolver.F90 (the Fortran 90 source code). From that code, it looks like the error is because I didn’t specify an input file name and the default file does not exist. This is the same behavior as ElmerSolver.exe from the official Elmer installation.

ELMER SOLVER (v 9.0) STARTED AT: 2022/06/26 21:55:22
ParCommInit:  Initialize #PEs:            1
MAIN:
MAIN: =============================================================
MAIN: ElmerSolver finite element software, Welcome!
MAIN: This program is free software licensed under (L)GPL
MAIN: Copyright 1st April 1995 - , CSC - IT Center for Science Ltd.
MAIN: Webpage http://www.csc.fi/elmer, Email elmeradm@csc.fi
MAIN: Version: 9.0 (Rev: af959fd0, Compiled: 2022-06-26)
MAIN:  Running one task without MPI parallelization.
MAIN:  Running with just one thread per task.
MAIN: =============================================================
ERROR:: ElmerSolver: Unable to find ELMERSOLVER_STARTINFO, can not execute.
STOP 1

Reviewing Docker Files

I was wondering after all this if there was a Windows docker file that had all these steps already baked in. However, the docker directory has only Ubuntu dockerfiles. Perhaps I could create a Windows docker file if I could figure out these dependencies.

Binaries Required by ElmerSolver

The above deployment instructions starts with the binaries required by ElmerGUI but ElmerSolver is the crucial component (since Elmer can be used with the GUI). Here are the binaries grouped such that the ElmerGUI binaries can be excluded if desired.

# binaries required by ElmerSolver
cp /mingw64/bin/libgfortran-5.dll ../install/bin/
cp /mingw64/bin/libgcc_s_seh-1.dll ../install/bin/
cp /mingw64/bin/libopenblas.dll ../install/bin/
cp /mingw64/bin/libquadmath-0.dll ../install/bin/
cp /mingw64/bin/libwinpthread-1.dll ../install/bin/

# binaries required by Mesh2D
cp /mingw64/bin/libstdc++-6.dll ../install/bin/

# binaries required by ElmerGUI
cp /mingw64/bin/qwt-qt5.dll ../install/bin/
cp /mingw64/bin/libdouble-conversion.dll ../install/bin/
cp /mingw64/bin/libicuin69.dll ../install/bin/
cp /mingw64/bin/libicuuc69.dll ../install/bin/
cp /mingw64/bin/libpcre2-16-0.dll ../install/bin/
cp /mingw64/bin/libharfbuzz-0.dll ../install/bin/
cp /mingw64/bin/libmd4c.dll ../install/bin/
cp /mingw64/bin/libpng16-16.dll ../install/bin/
cp /mingw64/bin/zlib1.dll ../install/bin/
cp /mingw64/bin/libzstd.dll ../install/bin/
cp /mingw64/bin/libicudt69.dll ../install/bin/
cp /mingw64/bin/libfreetype-6.dll ../install/bin/
cp /mingw64/bin/libglib-2.0-0.dll ../install/bin/
cp /mingw64/bin/libgraphite2.dll ../install/bin/
cp /mingw64/bin/libintl-8.dll ../install/bin/
cp /mingw64/bin/libbz2-1.dll ../install/bin/
cp /mingw64/bin/libbrotlidec.dll ../install/bin/
cp /mingw64/bin/libpcre-1.dll ../install/bin/
cp /mingw64/bin/libiconv-2.dll ../install/bin/
cp /mingw64/bin/libbrotlicommon.dll ../install/bin/

cp /mingw64/share/qt5/plugins/platforms/qwindows.dll ../install/bin/platforms/

Outstanding Issues

Deploying the above binaries manually gives a locally runnable build. Unfortunately, it does not fix the build created by CPack when you run make install – that build will not have any of these binaries/dependencies.

Article info

Categories

Leave a Reply

Your email address will not be published. Required fields are marked *