Below is a video from the Elmer folks demonstrating how to use parallelization to improve the performance of Elmer. I followed along on Windows with the publicly downloadable Elmer installation. I’m listing the commands in this post for easy reference.
We first clone the demo repository then open the geo file that serves as the basis for the demo in gmsh. I ran these commands in a Windows command prompt and used full paths to the gmsh and elmer executables.
git clone https://github.com/tzwinger/elmer_parallel_demo
cd elmer_parallel_demo
# Open the geo file in gmsh as a background job
gmsh.exe elmer_flow.geo &
# Create elmer_flow.msh
gmsh.exe -3 elmer_flow.geo
ElmerGrid 14 2 elmer_flow.msh -autoclean
The ElmerGrid command is used to create partitions from the mesh input file. Running it without any arguments displays the available options. The options used in the example are highlighted below.
This program can create simple 2D structured meshes consisting of
linear, quadratic or cubic rectangles or triangles. The meshes may
also be extruded and revolved to create 3D forms. In addition many
mesh formats may be imported into Elmer software. Some options have
not been properly tested. Contact the author if you face problems.
The program has two operation modes
A) Command file mode which has the command file as the only argument
'ElmerGrid commandfile.eg'
B) Inline mode which expects at least three input parameters
'ElmerGrid 1 3 test'
The first parameter defines the input file format:
1) .grd : ElmerGrid file format
2) .mesh.* : Elmer input format
3) .ep : Elmer output format
4) .ansys : Ansys input format
5) .inp : Abaqus input format by Ideas
6) .fil : Abaqus output format
7) .FDNEUT : Gambit (Fidap) neutral file
8) .unv : Universal mesh file format
9) .mphtxt : Comsol Multiphysics mesh format
10) .dat : Fieldview format
11) .node,.ele: Triangle 2D mesh format
12) .mesh : Medit mesh format
13) .msh : GID mesh format
14) .msh : Gmsh mesh format
15) .ep.i : Partitioned ElmerPost format
16) .2dm : 2D triangular FVCOM format
The second parameter defines the output file format:
1) .grd : ElmerGrid file format
2) .mesh.* : ElmerSolver format (also partitioned .part format)
3) .ep : ElmerPost format
4) .msh : Gmsh mesh format
5) .vtu : VTK ascii XML format
The third parameter is the name of the input file.
If the file does not exist, an example with the same name is created.
The default output file name is the same with a different suffix.
There are several additional in-line parameters that are
taken into account only when applicable to the given format.
-out str : name of the output file
-in str : name of a secondary input file
-decimals : number of decimals in the saved mesh (eg. 8)
...
-removeintbcs : remove internal boundaries if they are not needed
-removelowdim : remove boundaries that are two ranks lower than highest dim
-removeunused : remove nodes that are not used in any element
-bulkorder : renumber materials types from 1 so that every number is used
-boundorder : renumber boundary types from 1 so that every number is used
-autoclean : this performs the united action of the four above
...
Keywords are related to mesh partitioning for parallel ElmerSolver runs:
...
-metiskway int : mesh will be partitioned with Metis using graph Kway routine
-metisrec int : mesh will be partitioned with Metis using graph Recursive routine
-metiscontig : enforce that the metis partitions are contiguous
-metisseed int : random number generator seed for Metis algorithms
-partdual : use the dual graph in partition method (when available)
...
Now the solver can be invoked. The demo first does a serial run using the ElmerSolver command.
ElmerSolver elmer_flow_gcr.sif
This took just over 6 minutes on my machine. The wall clock time is the 2nd time (highlighted below) according to the demo video.
...
MAIN: Version: 9.0 (Rev: Release, Compiled: 2020-11-10)
MAIN: Running one task without MPI parallelization.
MAIN: Running with just one thread per task.
MAIN: Lua interpreted linked in.
...
ResultOutputSolver: Saving in unstructured VTK XML (.vtu) format
ResultOutputSolver: -------------------------------------
ElmerSolver: *** Elmer Solver: ALL DONE ***
ElmerSolver: The end
SOLVER TOTAL TIME(CPU,REAL): 369.31 369.31
ELMER SOLVER FINISHED AT: 2022/07/27 01:23:45
To increase parallelism, the demo continues with the serial mesh but uses OpenMP in multithreading mode by setting the OMP_NUM_THREADS environment variable. This does not appear to be sufficient to increase the number of threads it uses in my Windows setup so I need to get to the bottom of why OMP_NUM_THREADS is not being respected.
set OMP_NUM_THREADS=4
# setx /M OMP_NUM_THREADS 4
ElmerSolver elmer_flow_gcr.sif
Using 4 processes on my box takes 150s and increasing the processes to 6 drops the time to 121s.
MAIN: Version: 9.0 (Rev: Release, Compiled: 2020-11-10)
MAIN: Running in parallel using 4 tasks.
MAIN: Running with just one thread per task.
...
ResultOutputSolver: Saving in unstructured VTK XML (.vtu) format
ResultOutputSolver: -------------------------------------
ElmerSolver: *** Elmer Solver: ALL DONE ***
ElmerSolver: The end
SOLVER TOTAL TIME(CPU,REAL): 148.65 148.65
Visualization
The demo does not explicitly discuss how to visualize the results. However, this was covered in the Parallel Computing with Elmer talk.
Parallel Computing with Elmer
ParaView needs to be installed. Looks like they need to fix their installation dialog to not cut off text – or better yet, replace that long string with a user-friendly one. To visualize the ElmerSolver results, launch ParaView then open the parallel vtu file (*.pvtu).
Opening the Parallel VTU File
When the file opens, click on the green Apply button in the Properties panel. Notice that the pressure variable is preselected in the dropdown on the toolbar.
Visualizing Pressure in ParaView
Switching to the velocity variable shows all all blue rendering due to the 0 velocity. To see the animated visualization:
Click on the slice button (4th from left in screenshot).
Click on “Z Normal” in the Plane Parameters section on the Properties pane.
Click on Apply.
Click on the play button on the toolbar to see the animation of the flow.
The last part of the demo shows how to view the partitions in the mesh:
Deselect the eye on Slice1 in the Pipeline Browser.
Select the eye on “flow_gcr_t00” in the Pipeline Browser.
Go to the Filter menu > Alphabetic > Connectivity.
Click on the Apply button.
Change the selection in the combobox in the representation toolbar from from Surface to Surface with Edges to see how the partition seems to be based on the number of elements involved.
After completing the mesh generation in Crystal Growth Simulation – Part 1, I decided to investigate how to run the actual crystal growth simulation using Elmer. On Windows, Elmer needs to either be installed into “Program Files“ or be present in the path to avoid this error:
$ python run.py
crucible
melt
crystal
inductor
...
using material steel-1.4541 from self.materials_dict
using material vacuum from self.materials_dict
Wrote sif-file.
Starting simulation ./simdata/2022-07-25_13-17_ss_test-cz-induction_vacuum ...
Traceback (most recent call last):
File "D:\dev\repos\fem\research\test-cz-induction\run.py", line 32, in <module>
sim.execute()
File "d:\dev\repos\fem\research\opencgs\opencgs\sim.py", line 182, in execute
run_elmer_grid(self.sim_dir, "case.msh")
File "C:\Python310\lib\site-packages\pyelmer\execute.py", line 28, in run_elmer_grid
subprocess.run(args, cwd=sim_dir, stdout=f, stderr=f)
File "C:\Python310\lib\subprocess.py", line 501, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Python310\lib\subprocess.py", line 966, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Python310\lib\subprocess.py", line 1435, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
I tried installing a locally built Elmer – see How to Build Elmer on Windows. Unfortunately, the layout created by setup defaulted to “C:\Program Files\Elmer 9.0-1b8e4f7ec” (ending with the hash of the commit I built, instead of “Release”). Worse still, it didn’t launch because of missing DLLs. I need to take a closer look at how NSIS is creating the setup executable, and also figure out how to set up the publisher so that a certificate is displayed with the UAC prompt. I ended up installing the publicly downloadable Elmer then retrying the python script.
using material steel-1.4541 from self.materials_dict
using material vacuum from self.materials_dict
Wrote sif-file.
Starting simulation ./simdata/2022-07-25_17-23_ss_test-cz-induction_vacuum_1 ...
['ERROR:: systemc: Command exit status was 1'] [] {}
Finished simulation ./simdata/2022-07-25_17-23_ss_test-cz-induction_vacuum_1 .
Post processing...
Traceback (most recent call last):
File "D:\dev\repos\fem\research\test-cz-induction\run.py", line 32, in <module>
sim.execute()
File "d:\dev\repos\fem\research\opencgs\opencgs\sim.py", line 189, in execute
self._postprocessing_probes()
File "d:\dev\repos\fem\research\opencgs\opencgs\sim.py", line 139, in _postprocessing_probes
names_data = self._read_names_file(self.sim_dir + "/results/probes.dat.names")
File "d:\dev\repos\fem\research\opencgs\opencgs\sim.py", line 129, in _read_names_file
with open(names_file) as f:
FileNotFoundError: [Errno 2] No such file or directory: './simdata/2022-07-25_17-23_ss_test-cz-induction_vacuum_1/02_simulation/results/probes.dat.names'
A dialog popped up asking whether I wanted to grant some elmer process access to the domain network or private, etc, and I’m not sure if I took too long to answer it but the script continued with the above failure. This looks like a bug in opencgs – why continue post processing when the simulation has failed – I filed this issue to let the author know: Do not start post-processing when simulation fails · Issue #2 · nemocrys/opencgs (github.com).
I had no idea why the simulation is failing so I used Process Explorer to see which command line was being used to invoke ElmerGrid and ElmerSolver. I was able to then manually invoke ElmerSolver and notice a segmentation fault!
ELMER SOLVER (v 9.0) STARTED AT: 2022/07/25 17:23:13
ParCommInit: Initialize #PEs: 1
MAIN:
MAIN: =============================================================
MAIN: ElmerSolver finite element software, Welcome!
...
MAIN: Reading Model: case.sif
Caught LUA error:[string "loadfile("C:/Program Files (x86)/Elmer/shar..."]:1: attempt to call a nil value
Caught LUA error:[string "loadstring(readsif("case.sif"))()"]:1: attempt to call global 'readsif' (a nil value)
LoadInputFile: Scanning input file: case.sif
LoadInputFile: Scanning only size info
...
OptimizeBandwidth: Half bandwidth after optimization: 1447
OptimizeBandwidth: ---------------------------------------------------------
'ViewFactors' is not recognized as an internal or external command,
operable program or batch file.
ERROR:: systemc: Command exit status was 1
RadiationFactors: Message
RadiationFactors: All done time (s) 4.9000E-02
...
60 0.2041E-08
61 0.3710E-09
ComputeChange: NS (ITER=11) (NRM,RELC): ( 0.17032920E-04 0.41334346E-05 ) :: statmagsolver
StatMagSolve: Convergence after 11 iterations
StatMagSolve: Joule Heating (W): 4.1375E+01
StatMagSolve: All done, exiting
StatMagSolve: ------------------------------------------------
ComputeChange: SS (ITER=1) (NRM,RELC): ( 0.17032920E-04 2.0000000 ) :: statmagsolver
HeatSolve: Found Control Point at distance: 0.0000000000000000
HeatSolve: Control Point Index: 0
HeatSolve: Using Steady-state Heater Control
HeatSolve:
HeatSolve:
HeatSolve: -------------------------------------
HeatSolve: TEMPERATURE ITERATION 1
HeatSolve: -------------------------------------
HeatSolve:
HeatSolve: Starting Assembly...
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
#0 0xffffffff
#1 0xffffffff
#2 0xffffffff
...
#19 0xffffffff
#20 0xffffffff
Looking at the simulation folders such as “simdata\2022-07-25_22-52_ss_test-cz-induction_vacuum_1\02_simulation” revealed that there are ElmerGrid and ElmerSolver log files present with this info, so I didn’t even need Process Explorer! At this point, I have to save the SIGSEGV investigation for another day.
My post on How to Build Elmer on Windows has a succinct list of instructions but when I first built the Elmer source code (on Windows), I was unsure how to run the binaries. Running ElmerGUI, for example, failed because qt5.dll could not be found. This post details the process I used to figure out how to create a usable Elmer installation.
I started by going through the generated Makefile. It contained targets like install, install/local, and install/strip. I started with install/local, fearing that perhaps these targets would attempt to install the binaries into program files.
Installing only the local directory...
-- Install configuration: "RelWithDebInfo"
-- Installing: C:/dev/repos/fem/install/bin/libopenblas.dll.a
CMake Error at cmake_install.cmake:45 (file):
file INSTALL cannot find "C:/dev/repos/fem/elmerfem/../bundle_msys2/bin":
No error.
make: *** [Makefile:142: install/local] Error 1
The error at cmake_install.cmake:45 is from ElmerCPack.cmake. I did not even know about the existence of the CPack tool. Git blame points to commit b9914082 as the source of the failing command. The strange thing about that commit is that there is no mention of what is supposed to create the bundle folders (despite the callout that the change “Relies on selected bundled files selected prior to installation”).
A workaround I tried was to use this define when running cmake: CPACK_BUNDLE_EXTRA_WINDOWS_DLLS:BOOL=FALSE. That avoided the build error but did not result in a usable installation. ElmerGUI’s windows_bundle.cmake needs this to be set to TRUE to package the Qt5 binaries into the install folder. Strangely enough, ElmerGUILogger’s windows_bundle.cmake does not have this logic for Qt5. This is likely because ElmerGUI would already have installed the right files into the bin folder but this looks like a bug.
Another work-around I tried was to manually create the folders expected by the build. The install target then succeeded but the necessary binaries were not copied to the install folder.
This is when I dig around and find that only ElmerGUILogger and ElmerClips include windows_bundle.cmake. Hmm… the latter looks promising but doesn’t look up to date either since it requires Qt4. More exploring of the Makefile – the package target looks interesting but fails because NSIS is not installed.
...
[ 97%] Built target ElmerGUI_autogen
[100%] Built target ElmerGUI
Run CPack packaging tool...
CPack Error: Cannot find NSIS compiler makensis: likely it is not installed, or not in your PATH
CPack Error: Could not read NSIS registry value. This is usually caused by NSIS not being installed. Please install NSIS from http://nsis.sourceforge.net
CPack Error: Cannot initialize the generator NSIS
make: *** [Makefile:71: package] Error 1
I wonder if we couldn’t just use NSIS for the MSYS environment:
$ pacman -Ss nsis
...
mingw64/mingw-w64-x86_64-nsis 3.06.1-1
Windows installer development tool (mingw-w64)
mingw64/mingw-w64-x86_64-nsis-nsisunz 1.0-2
NSIS plugin which allows you to extract files from ZIP archives (mingw-w64)
...
$ pacman -S mingw64/mingw-w64-x86_64-nsis
Now we can create an Elmer installer by running make package. Unfortunately, that turns out to be insufficient. My next idea is to compare the binaries from the installed. This turned out to be easier when using ls -R1 to output only the file names and in 1 column only. Some obvious differences are that the build in Program Files has a bin folder containing the Qt and vtk binaries (as well as a stripped gfortran).
The Qt binaries certainly look like the output of windows_bundle.cmake (found this time by a search for “vtk”) but it’s still not clear how this file would be includedin the build. I’m using VSCode to search for “windows_bundle” and only 2 of the 3 references in the codebase were showing up (on my desktop). Looking for “ElmerGUILogger” then revealed yet another reference. Such a waste of time! Not cool VSCode, not cool. It’s included in ElmerGUI/CMakeLists.txt. So I probably only need to define WIN32. But why does the WIN32 code run if I add some statements to that IF block?
Some searching (TODO: put bing searches here) leads to indications that there might be an error in CMake where WIN32 is not defined. Seeing signs that MSYS can be in Cywgin? Trying to get to the bottom of why WIN32 is not respected by CMake, I review ElmerGUI/CMakeLists.txt again. It adds the netgen subdirectory. Interestingly, ElmerGUI/netgen/README points out that install\lib\ElmerGUI\ngcore\libng.a is the unix library and that the win32 extension should be .lib. Biggest sign I’ve seen so far that something is really off. At this point, I remember seeing a .a file in the install folder – and that seemed strange for a lib folder since I expected a DLL. This hypothesis fails though because the working Elmer installation also has the file “C:\Program Files\Elmer 9.0-Release\lib\ElmerGUI\ngcore\libng.a“.
The sure way to verify that the WIN32 include is working is to introduce an error into windows_bundle.cmake, e.g. by change the first IF into the unknown IF2. Even better, notice that the “Qt5 Windows packaging” message is correctly displayed. Perhaps the FIND_FILE command should have a REQUIRED option now that it is supported. Adding that doesn’t fail so reexamine the INSTALL command. Since windows_bundle.cmake uses a relative path for the DESTINATION, it is interpreted relative to the value of the CMAKE_INSTALL_PREFIX variable.
How about zooming into the windows_bundle.cmake file and outputting the list of files installed afterthe install command. The new message says the files were installed. However, they aren’t in the install folder! I need an explanation for this behavior. So let’s open Process Explorer and see which paths are actually used. These requests show up using the Qt5 path filter:
These files exist on disk (despite the Result column in process explorer having the value INVALID DEVICE REQUEST)! For a better understanding of what is happening, I change the bin path used by the install command to dbgcmake since there are multiple bin folders (under install and also in the MinGW installation). So such path shows up in Process Explorer when using a path filter for dbgcmake. This means I shouldn’t be expecting this files to be written by cmake at this point. In fact, running grep -Rin dbgcmake shows that build/ElmerGUI/cmake_install.cmake now contains this snippet (thereby verifying that windows_bundle.cmake is used to generate cmake_install.cmake).
if("x${CMAKE_INSTALL_COMPONENT}x" STREQUAL "xelmerguix" OR NOT CMAKE_INSTALL_COMPONENT)
file(INSTALL DESTINATION "${CMAKE_INSTALL_PREFIX}/dbgcmake" TYPE FILE FILES
"D:/dev/Software/msys64/mingw64/bin/Qt5Core.dll"
"D:/dev/Software/msys64/mingw64/bin/Qt5Gui.dll"
"D:/dev/Software/msys64/mingw64/bin/Qt5OpenGL.dll"
"D:/dev/Software/msys64/mingw64/bin/Qt5Script.dll"
"D:/dev/Software/msys64/mingw64/bin/Qt5Xml.dll"
"D:/dev/Software/msys64/mingw64/bin/Qt5Svg.dll"
"D:/dev/Software/msys64/mingw64/bin/Qt5Widgets.dll"
"D:/dev/Software/msys64/mingw64/bin/Qt5PrintSupport.dll"
)
endif()
So back to the question of why windows_bundle.cmake is not invoked by default – let us dig into the CMake sources for clues. Renaming it outputs an error for the include statement meaning the file is being included! Sigh… I end up poking around some CMake sources to see where WIN32 is set anyway.:
I immediately notice that the install command of interest is skipped because CPACK_BUNDLE_EXTRA_WINDOWS_DLLS is not set to TRUE! Setting it to FALSE earlier then seeing a build failure gave the impression that it was set to TRUE everywhere (especially given that ElmerCPack.cmake sets it to TRUE) but that happens afterwindows_bundle.cmake has been evaluated! Here’s the final required command line:
There are lessons there about making assumptions but the biggest takeaway for me is the need for (and existence) of tracing capabilities in CMake. Enabling tracing made it so easy to figure out exactly what was broken – the variable did not have the value I expected and I simply needed to define it! Running make install now results in new errors haha!
The code execution cannot proceed because libgcc_s_seh-1.dll was not found. Reinstalling the program may fix this problem. This error dialog is then followed by others for qwt-qt5.dll, libstdc++-6.dll, and libdouble-conversion.dll respectively. To see which other binaries are required, manually copy these 4 binaries from d:\dev\Software\msys64\mingw64\bin to install/bin. These are the other missing binaries that show up in error dialogs:
There are so many missing binaries that I wonder if building the other components might be necessary. I first manually copy them from the bin folder to see if ElmerGUI can load but that now fails with the error This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.
The working installation from the downloaded public Elmer installer has a qwindows.dll in the bin/platforms directory. This file exists on my system as D:\dev\Software\msys64\mingw64\share\qt5\plugins\platforms\qwindows.dll. At this point, I’m really wondering how on earth they are gathering all these files. Hmm, maybe they build using their docker image? Didn’t think of seeing how their docker image is set up. One final try to set up the qwindows.dll first and see what happens.
Lo and behold! ElmerGUI loads successfully! Whether it actually works… well, I will revisit that after I learn how to use Elmer/ElmerGUI :). Actually, turns out ElmerSolver.exe doesn’t start. Here is the fix for the 3 binaries that cause error dialogs:
At this point, ElmerSolver starts up successfully but outputs an error about missing ELMERSOLVER_STARTINFO. This is an error from ElmerSolver.F90 (the Fortran 90 source code). From that code, it looks like the error is because I didn’t specify an input file name and the default file does not exist. This is the same behavior as ElmerSolver.exe from the official Elmer installation.
ELMER SOLVER (v 9.0) STARTED AT: 2022/06/26 21:55:22
ParCommInit: Initialize #PEs: 1
MAIN:
MAIN: =============================================================
MAIN: ElmerSolver finite element software, Welcome!
MAIN: This program is free software licensed under (L)GPL
MAIN: Copyright 1st April 1995 - , CSC - IT Center for Science Ltd.
MAIN: Webpage http://www.csc.fi/elmer, Email elmeradm@csc.fi
MAIN: Version: 9.0 (Rev: af959fd0, Compiled: 2022-06-26)
MAIN: Running one task without MPI parallelization.
MAIN: Running with just one thread per task.
MAIN: =============================================================
ERROR:: ElmerSolver: Unable to find ELMERSOLVER_STARTINFO, can not execute.
STOP 1
Reviewing Docker Files
I was wondering after all this if there was a Windows docker file that had all these steps already baked in. However, the docker directory has only Ubuntu dockerfiles. Perhaps I could create a Windows docker file if I could figure out these dependencies.
Binaries Required by ElmerSolver
The above deployment instructions starts with the binaries required by ElmerGUI but ElmerSolver is the crucial component (since Elmer can be used with the GUI). Here are the binaries grouped such that the ElmerGUI binaries can be excluded if desired.
Deploying the above binaries manually gives a locally runnable build. Unfortunately, it does not fix the build created by CPack when you run make install – that build will not have any of these binaries/dependencies.
As of this post, neither the build instructions on Elmer’s webpage nor the Elmer GitHub repo include instructions for how to build Elmer on Windows. Here is a succinct set of instructions (discovered in the previous post, which detailed the stumbling around that led me a solution).
Clone the Elmer source code.
cd \repos
mkdir fem
cd fem
git clone git://www.github.com/ElmerCSC/elmerfem
Switch to the parent directory of the elmer repo using cd /repos/fem. Note that the elmer repo directory will have siblings such as build and install, so the separate parent directory (fem in this example) simplifies the directory organization.
Create the required directories and switch to the build directory.
# create folders required for building a local install
mkdir -p bundle_msys2/bin
mkdir -p bundle_qt5/bin
mkdir -p platforms
mkdir build
cd build
Deploy the prerequisite binaries for Elmer (Solver, Mesh2D, and GUI):
Build Elmer. When cmake completes, a message will be displayed confirming that generation is done and that build files have been written to the build/ directory. Run make to start compiling the source code or make install to compile the sources then create a local installation in the install/ directory. Note that there are some folders (created using mkdir -p below) that don’t appear to be used in a local build but the build still expects them to exist. You can also specify a debug build by adding the -DCMAKE_BUILD_TYPE=Debug define.
