Saint's Log

Elmer Parallel Demo

Below is a video from the Elmer folks demonstrating how to use parallelization to improve the performance of Elmer. I followed along on Windows with the publicly downloadable Elmer installation. I’m listing the commands in this post for easy reference.

We first clone the demo repository then open the geo file that serves as the basis for the demo in gmsh. I ran these commands in a Windows command prompt and used full paths to the gmsh and elmer executables.

git clone https://github.com/tzwinger/elmer_parallel_demo
cd elmer_parallel_demo

# Open the geo file in gmsh as a background job
gmsh.exe elmer_flow.geo &

# Create elmer_flow.msh
gmsh.exe -3 elmer_flow.geo

ElmerGrid 14 2 elmer_flow.msh -autoclean

The ElmerGrid command is used to create partitions from the mesh input file. Running it without any arguments displays the available options. The options used in the example are highlighted below.

This program can create simple 2D structured meshes consisting of
linear, quadratic or cubic rectangles or triangles. The meshes may
also be extruded and revolved to create 3D forms. In addition many
mesh formats may be imported into Elmer software. Some options have
not been properly tested. Contact the author if you face problems.

The program has two operation modes
A) Command file mode which has the command file as the only argument
   'ElmerGrid commandfile.eg'

B) Inline mode which expects at least three input parameters
   'ElmerGrid 1 3 test'

The first parameter defines the input file format:
1)  .grd      : ElmerGrid file format
2)  .mesh.*   : Elmer input format
3)  .ep       : Elmer output format
4)  .ansys    : Ansys input format
5)  .inp      : Abaqus input format by Ideas
6)  .fil      : Abaqus output format
7)  .FDNEUT   : Gambit (Fidap) neutral file
8)  .unv      : Universal mesh file format
9)  .mphtxt   : Comsol Multiphysics mesh format
10) .dat      : Fieldview format
11) .node,.ele: Triangle 2D mesh format
12) .mesh     : Medit mesh format
13) .msh      : GID mesh format
14) .msh      : Gmsh mesh format
15) .ep.i     : Partitioned ElmerPost format
16) .2dm      : 2D triangular FVCOM format

The second parameter defines the output file format:
1)  .grd      : ElmerGrid file format
2)  .mesh.*   : ElmerSolver format (also partitioned .part format)
3)  .ep       : ElmerPost format
4)  .msh      : Gmsh mesh format
5)  .vtu      : VTK ascii XML format

The third parameter is the name of the input file.
If the file does not exist, an example with the same name is created.
The default output file name is the same with a different suffix.

There are several additional in-line parameters that are
taken into account only when applicable to the given format.
-out str             : name of the output file
-in str              : name of a secondary input file
-decimals            : number of decimals in the saved mesh (eg. 8)
...
-removeintbcs        : remove internal boundaries if they are not needed
-removelowdim        : remove boundaries that are two ranks lower than highest dim
-removeunused        : remove nodes that are not used in any element
-bulkorder           : renumber materials types from 1 so that every number is used
-boundorder          : renumber boundary types from 1 so that every number is used
-autoclean           : this performs the united action of the four above
...

Keywords are related to mesh partitioning for parallel ElmerSolver runs:
...
-metiskway int       : mesh will be partitioned with Metis using graph Kway routine

-metisrec int        : mesh will be partitioned with Metis using graph Recursive routine
-metiscontig         : enforce that the metis partitions are contiguous
-metisseed int       : random number generator seed for Metis algorithms
-partdual            : use the dual graph in partition method (when available)
...

Now the solver can be invoked. The demo first does a serial run using the ElmerSolver command.

ElmerSolver elmer_flow_gcr.sif

This took just over 6 minutes on my machine. The wall clock time is the 2nd time (highlighted below) according to the demo video.

...
MAIN: Version: 9.0 (Rev: Release, Compiled: 2020-11-10)
MAIN:  Running one task without MPI parallelization.
MAIN:  Running with just one thread per task.
MAIN:  Lua interpreted linked in.
...
ResultOutputSolver: Saving in unstructured VTK XML (.vtu) format
ResultOutputSolver: -------------------------------------
ElmerSolver: *** Elmer Solver: ALL DONE ***
ElmerSolver: The end
SOLVER TOTAL TIME(CPU,REAL):       369.31      369.31
ELMER SOLVER FINISHED AT: 2022/07/27 01:23:45

To increase parallelism, the demo continues with the serial mesh but uses OpenMP in multithreading mode by setting the OMP_NUM_THREADS environment variable. This does not appear to be sufficient to increase the number of threads it uses in my Windows setup so I need to get to the bottom of why OMP_NUM_THREADS is not being respected.

set OMP_NUM_THREADS=4
# setx /M OMP_NUM_THREADS 4
ElmerSolver elmer_flow_gcr.sif

Next, the demo uses ElmerGrid to partition the input, this time using the arguments 2 2 instead of 14 2 since we already have an existing serial Elmer mesh. It was interesting learning about partitioning algorithms like METIS – KarypisLab/METIS: METIS – Serial Graph Partitioning and Fill-reducing Matrix Ordering (github.com).

ElmerGrid 2 2 elmer_flow -partdual -metiskway 4

# notice the partitioning.4 directory in elmer_flow
dir elmer_flow

The demo uses mpirun but the Windows equivalent is “C:\Program Files\Microsoft MPI\Bin\mpiexec.exe“.

"C:\Program Files\Microsoft MPI\Bin\mpiexec.exe" -np 4 "C:\Program Files\Elmer 9.0-Release\bin\ElmerSolver.exe" elmer_flow_gcr.sif

Using 4 processes on my box takes 150s and increasing the processes to 6 drops the time to 121s.

MAIN: Version: 9.0 (Rev: Release, Compiled: 2020-11-10)
MAIN:  Running in parallel using 4 tasks.
MAIN:  Running with just one thread per task.
...
ResultOutputSolver: Saving in unstructured VTK XML (.vtu) format
ResultOutputSolver: -------------------------------------
ElmerSolver: *** Elmer Solver: ALL DONE ***
ElmerSolver: The end
SOLVER TOTAL TIME(CPU,REAL):       148.65      148.65

Visualization

The demo does not explicitly discuss how to visualize the results. However, this was covered in the Parallel Computing with Elmer talk.

ParaView needs to be installed. Looks like they need to fix their installation dialog to not cut off text – or better yet, replace that long string with a user-friendly one. To visualize the ElmerSolver results, launch ParaView then open the parallel vtu file (*.pvtu).

When the file opens, click on the green Apply button in the Properties panel. Notice that the pressure variable is preselected in the dropdown on the toolbar.

Switching to the velocity variable shows all all blue rendering due to the 0 velocity. To see the animated visualization:

1. Click on the slice button (4th from left in screenshot).
2. Click on “Z Normal” in the Plane Parameters section on the Properties pane.
3. Click on Apply.
4. Click on the play button on the toolbar to see the animation of the flow.

The last part of the demo shows how to view the partitions in the mesh:

1. Deselect the eye on Slice1 in the Pipeline Browser.
2. Select the eye on “flow_gcr_t00” in the Pipeline Browser.
3. Go to the Filter menu > Alphabetic > Connectivity.
4. Click on the Apply button.

Change the selection in the combobox in the representation toolbar from from Surface to Surface with Edges to see how the partition seems to be based on the number of elements involved.

ParaView’s source code is available on GitLab so naturally, I’ll try to build it one of these fine days.