Categories: Fluid Flow, Simulation

Building the VM2D Source Code

One of the challenges I have run into when reading through numerical methods/simulation papers is the dearth of source code. I was therefore pleasantly surprised to find the Vortex method for 2D flow simulation program. Of course, my luck is that the comments and docs are in Russian, which I do not understand. However, availability of the source code more than makes up for that. The publication of The VM2D Open Source Code for Incompressible Flow Simulation by Using Meshless Lagrangian Vortex Methods on CPU and GPU | IEEE Conference Publication | IEEE Xplore turns out to be the documenation. To build the source code, open a developer command prompt.

git clone https://github.com/vortexmethods/VM2D
cd VM2D
cmake .

This fails with an error about MPI missing:

-- -------------------------2D CODE-------------------------
-- Checking for module 'mpi-c'
--   Can't find mpi-c.pc in any of C:/software/strawberry/c/lib/pkgconfig
use the PKG_CONFIG_PATH environment variable, or
specify extra search paths via 'search_paths'
-- Could NOT find MPI_C (missing: MPI_C_LIB_NAMES MPI_C_HEADER_DIR MPI_C_WORKS)
-- Checking for module 'mpi-cxx'
--   Can't find mpi-cxx.pc in any of C:/software/strawberry/c/lib/pkgconfig
use the PKG_CONFIG_PATH environment variable, or
specify extra search paths via 'search_paths'
-- Could NOT find MPI_CXX (missing: MPI_CXX_LIB_NAMES MPI_CXX_HEADER_DIR MPI_CXX_WORKS)
CMake Error at C:/Program Files/CMake/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find MPI (missing: MPI_C_FOUND MPI_CXX_FOUND)
Call Stack (most recent call first):
  C:/Program Files/CMake/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  C:/Program Files/CMake/share/cmake-3.25/Modules/FindMPI.cmake:1837 (find_package_handle_standard_args)
  src/VMlib/CMakeLists.txt:31 (find_package)

StackOverflow articles e.g. Installing MPI for Windows – Stack Overflow suggest installing Microsoft MPI. I Download Microsoft MPI v10.0 from Official Microsoft Download Center but why on earth does it have an IE warning when I’m already using Edge?

Microsoft MPI v10.0 Download Page

How do I know which one I want? I’ll start with the SDK MSI.

Choose the download you want
MPI SDK Setup
asdf

The publisher certificate expired in December 2021. Shouldn’t there be a warning about that? I guess the publisher is well known and not revoked? Oh well, plough ahead and install it. Reopen the developer command prompt and run cmake . again. This time, there are no errors (and I ignore all the warnings since I have things to do). A Visual Studio solution is generated. Open VM.sln in Visual Studio 2022. Building fails with these errors:

...
3>C:\repos\edu\VM2D\src\VMcuda>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc.exe"  --use-local-env ... -Wno-deprecated-gpu-targets -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_53,code=sm_53 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_62,code=sm_62 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_72,code=sm_72 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -std=c++14 ... -o VMcuda.dir\Debug\cuLib2D.obj "C:\repos\edu\VM2D\src\VMcuda\Cuda\cuLib2D.cu"
3>nvcc fatal   : Unsupported gpu architecture 'compute_35'
...
3>Done building project "VMcuda.vcxproj" -- FAILED.
...
4>LINK : fatal error LNK1104: cannot open file 'VMcuda.lib'
4>Done building project "VM2D.vcxproj" -- FAILED.

The solution is to remove the segment -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 from VM2D/CMakeLists.txt · vortexmethods/VM2D (github.com). Launching the application (VM2D.exe) fails with this message: The code execution cannot proceed because msmpi.dll was not found. Reinstalling the program may fix this problem.

VM2D.exe Failing to Launch

The SDK doesn’t have that DLL, so I guess that’s what the other setup executable is for.

MPI 10.0.12498 Setup
MPI Setup

I don’t see any DLLs in that installation directory. However, mpi – msmpi.dll error message in Visual Studio C++ – Stack Overflow says reinstalling MPI is the solution. Before doing so, I run the application in Visual Studio once again and this time it launches successfully. This message is displayed: queue ERROR: file problems is not found. These files are in the VM2D/run folder. Other files will not be found though if using the run folder as the current directory.

I guess I need to translate VM2D/03_starting.rst. This is the first time I have needed to translate a web page. Looks like the problems file expects to be in the tutorials directory. Copying the file in the VM2D/run folder to the VM2D/tutorials directory allows the program to find the expect files and it appears to run now. Task manager shows 100% CPU usage on my AMD Ryzen 7 5800X 8-Core Processor. The program runs for an hour and as per VM2D/05_run.rst, creates a csv file and a snapshots directory containing vtk files. To view these files, Download ParaView and install it. Launch ParaView then open the snapshots directory (it should recognize all the vtk files as a group).

Opening snapshots

Click on the Apply button on the Properties pane (see image below).

Properties of the Snapshots

Finally, click on the Play button on the toolbar to see the animation of the snapshots. The next step will be figuring out how to use the GPU to generate these snapshots in (hopefully) much less than an hour.

Snapshots in Paraview

Categories: Crystal Growth

Crystal Growth Simulation on Linux

Crystal Growth Simulation – Part 1 walked through setting up a mesh for running crystal simulation in Elmer on Windows. Crystal Growth Simulation Failure described the process of trying to run the actual simulation on Windows and the resulting segmentation fault. What happens if we try this same experiment on Linux? First install Elmer using the instructions on the Elmer FEM blog.

sudo apt-add-repository ppa:elmer-csc-ubuntu/elmer-csc-ppa
sudo apt-get update
sudo apt-get install elmerfem-csc

I’m not sure if it’s because I’ve built other repos on my Ubuntu VM, but python3 is already installed. I still need to install pip though.

sudo apt-get install python3
sudo apt install python3-pip

pip install pyelmer
pip install objectgmsh

We need to clone 2 repos to execute the crystal growth experiment. The opencgs repo is a dependency of the test-cz-induction repo.

cd ~/repos/fem/elmer/research
git clone https://github.com/nemocrys/opencgs
git clone https://github.com/nemocrys/test-cz-induction

Next, install the opencgs module using pip then set up the crystal growth simulation mesh.

cd opencgs
pip install -e .

This installation output contains an error about an incompatible numpy version but ends with a message about all the components being successfully installed.

Requirement already satisfied: kiwisolver>=1.0.1 in /home/saint/.local/lib/python3.8/site-packages (from matplotlib->pyelmer->opencgs==0.3.1) (1.4.4)
Requirement already satisfied: fonttools>=4.22.0 in /home/saint/.local/lib/python3.8/site-packages (from matplotlib->pyelmer->opencgs==0.3.1) (4.34.4)
ERROR: pandas 1.4.3 has requirement numpy>=1.18.5; platform_machine != "aarch64" and platform_machine != "arm64" and python_version < "3.10", but you'll have numpy 1.17.4 which is incompatible.
...
Successfully installed commonmark-0.9.1 meshio-5.3.4 opencgs pandas-1.4.3 pygments-2.12.0 python-dateutil-2.8.2 pytz-2022.1 rich-12.5.1 typing-extensions-4.3.0

I ignore the error and forge ahead with running the crystal simulation setup:

cd ../test-cz-induction
python3 setup.py

This fails with an error related to numpy.

python3 setup.py 
Traceback (most recent call last):
  File "setup.py", line 19, in <module>
    import opencgs.control as ctrl
  File "/home/saint/repos/fem/elmer/research/opencgs/opencgs/__init__.py", line 5, in <module>
    import opencgs.post
  File "/home/saint/repos/fem/elmer/research/opencgs/opencgs/post.py", line 2, in <module>
    import meshio
  File "/home/saint/.local/lib/python3.8/site-packages/meshio/__init__.py", line 1, in <module>
    from . import (
  File "/home/saint/.local/lib/python3.8/site-packages/meshio/_cli/__init__.py", line 1, in <module>
    from ._main import main
  File "/home/saint/.local/lib/python3.8/site-packages/meshio/_cli/_main.py", line 5, in <module>
    from . import _ascii, _binary, _compress, _convert, _decompress, _info
  File "/home/saint/.local/lib/python3.8/site-packages/meshio/_cli/_ascii.py", line 4, in <module>
    from .. import ansys, flac3d, gmsh, mdpa, ply, stl, vtk, vtu, xdmf
  File "/home/saint/.local/lib/python3.8/site-packages/meshio/ansys/__init__.py", line 1, in <module>
    from ._ansys import read, write
  File "/home/saint/.local/lib/python3.8/site-packages/meshio/ansys/_ansys.py", line 14, in <module>
    from .._helpers import register_format
  File "/home/saint/.local/lib/python3.8/site-packages/meshio/_helpers.py", line 7, in <module>
    from numpy.typing import ArrayLike

I guess I need to heed that error from pip. python – How can I upgrade NumPy? – Stack Overflow says to use the –upgrade flag.

$ pip install numpy --upgrade
Collecting numpy
  Downloading numpy-1.23.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
     |████████████████████████████████| 17.1 MB 3.6 MB/s 
Installing collected packages: numpy
  WARNING: The scripts f2py, f2py3 and f2py3.8 are installed in '/home/saint/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed numpy-1.23.1

That’s all that’s needed to get things to run. For example, setting up the mesh now works:

python3 setup.py

The crystal growth simulation can now be executed as well:

python3 run.py

Interestingly, the simulation completes successfully!

crucible
melt
crystal
inductor
seed
insulation
crucible_adapter
axis_bt
...
using material graphite-CZ3R6300 from self.materials_dict
using material insulation from self.materials_dict
using material steel-1.4541 from self.materials_dict
using material vacuum from self.materials_dict
Wrote sif-file.
Starting simulation  ./simdata/2022-07-31_13-35_ss_test-cz-induction_vacuum  ...
[] [] {'CPU-time': 55.87, 'real-time': 57.49}
Finished simulation  ./simdata/2022-07-31_13-35_ss_test-cz-induction_vacuum  .
Post processing...
evaluating heat fluxes

Finished post processing.

This implies that the segmentation fault I ran into on Windows was specific to the Windows Elmer build.

Visualization

Use ParaView to visualize the results of the simulation.

sudo apt install paraview

There are different variables that can be visualized. I selected temperature for the screenshot below. Other options included joule field, joule heating, potential im, newy, etc.

Visualizing the Crystal Growth Simulation

Categories: Elmer, Simulation

Elmer Parallel Demo

Below is a video from the Elmer folks demonstrating how to use parallelization to improve the performance of Elmer. I followed along on Windows with the publicly downloadable Elmer installation. I’m listing the commands in this post for easy reference.

We first clone the demo repository then open the geo file that serves as the basis for the demo in gmsh. I ran these commands in a Windows command prompt and used full paths to the gmsh and elmer executables.

git clone https://github.com/tzwinger/elmer_parallel_demo
cd elmer_parallel_demo

# Open the geo file in gmsh as a background job
gmsh.exe elmer_flow.geo &

# Create elmer_flow.msh
gmsh.exe -3 elmer_flow.geo

ElmerGrid 14 2 elmer_flow.msh -autoclean

The ElmerGrid command is used to create partitions from the mesh input file. Running it without any arguments displays the available options. The options used in the example are highlighted below.

This program can create simple 2D structured meshes consisting of
linear, quadratic or cubic rectangles or triangles. The meshes may
also be extruded and revolved to create 3D forms. In addition many
mesh formats may be imported into Elmer software. Some options have
not been properly tested. Contact the author if you face problems.

The program has two operation modes
A) Command file mode which has the command file as the only argument
   'ElmerGrid commandfile.eg'

B) Inline mode which expects at least three input parameters
   'ElmerGrid 1 3 test'

The first parameter defines the input file format:
1)  .grd      : ElmerGrid file format
2)  .mesh.*   : Elmer input format
3)  .ep       : Elmer output format
4)  .ansys    : Ansys input format
5)  .inp      : Abaqus input format by Ideas
6)  .fil      : Abaqus output format
7)  .FDNEUT   : Gambit (Fidap) neutral file
8)  .unv      : Universal mesh file format
9)  .mphtxt   : Comsol Multiphysics mesh format
10) .dat      : Fieldview format
11) .node,.ele: Triangle 2D mesh format
12) .mesh     : Medit mesh format
13) .msh      : GID mesh format
14) .msh      : Gmsh mesh format
15) .ep.i     : Partitioned ElmerPost format
16) .2dm      : 2D triangular FVCOM format

The second parameter defines the output file format:
1)  .grd      : ElmerGrid file format
2)  .mesh.*   : ElmerSolver format (also partitioned .part format)
3)  .ep       : ElmerPost format
4)  .msh      : Gmsh mesh format
5)  .vtu      : VTK ascii XML format

The third parameter is the name of the input file.
If the file does not exist, an example with the same name is created.
The default output file name is the same with a different suffix.

There are several additional in-line parameters that are
taken into account only when applicable to the given format.
-out str             : name of the output file
-in str              : name of a secondary input file
-decimals            : number of decimals in the saved mesh (eg. 8)
...
-removeintbcs        : remove internal boundaries if they are not needed
-removelowdim        : remove boundaries that are two ranks lower than highest dim
-removeunused        : remove nodes that are not used in any element
-bulkorder           : renumber materials types from 1 so that every number is used
-boundorder          : renumber boundary types from 1 so that every number is used
-autoclean           : this performs the united action of the four above
...

Keywords are related to mesh partitioning for parallel ElmerSolver runs:
...
-metiskway int       : mesh will be partitioned with Metis using graph Kway routine

-metisrec int        : mesh will be partitioned with Metis using graph Recursive routine
-metiscontig         : enforce that the metis partitions are contiguous
-metisseed int       : random number generator seed for Metis algorithms
-partdual            : use the dual graph in partition method (when available)
...

Now the solver can be invoked. The demo first does a serial run using the ElmerSolver command.

ElmerSolver elmer_flow_gcr.sif

This took just over 6 minutes on my machine. The wall clock time is the 2nd time (highlighted below) according to the demo video.

...
MAIN: Version: 9.0 (Rev: Release, Compiled: 2020-11-10)
MAIN:  Running one task without MPI parallelization.
MAIN:  Running with just one thread per task.
MAIN:  Lua interpreted linked in.
...
ResultOutputSolver: Saving in unstructured VTK XML (.vtu) format
ResultOutputSolver: -------------------------------------
ElmerSolver: *** Elmer Solver: ALL DONE ***
ElmerSolver: The end
SOLVER TOTAL TIME(CPU,REAL):       369.31      369.31
ELMER SOLVER FINISHED AT: 2022/07/27 01:23:45

To increase parallelism, the demo continues with the serial mesh but uses OpenMP in multithreading mode by setting the OMP_NUM_THREADS environment variable. This does not appear to be sufficient to increase the number of threads it uses in my Windows setup so I need to get to the bottom of why OMP_NUM_THREADS is not being respected.

set OMP_NUM_THREADS=4
# setx /M OMP_NUM_THREADS 4
ElmerSolver elmer_flow_gcr.sif

Next, the demo uses ElmerGrid to partition the input, this time using the arguments 2 2 instead of 14 2 since we already have an existing serial Elmer mesh. It was interesting learning about partitioning algorithms like METIS – KarypisLab/METIS: METIS – Serial Graph Partitioning and Fill-reducing Matrix Ordering (github.com).

ElmerGrid 2 2 elmer_flow -partdual -metiskway 4

# notice the partitioning.4 directory in elmer_flow
dir elmer_flow

The demo uses mpirun but the Windows equivalent is “C:\Program Files\Microsoft MPI\Bin\mpiexec.exe“.

"C:\Program Files\Microsoft MPI\Bin\mpiexec.exe" -np 4 "C:\Program Files\Elmer 9.0-Release\bin\ElmerSolver.exe" elmer_flow_gcr.sif

Using 4 processes on my box takes 150s and increasing the processes to 6 drops the time to 121s.

MAIN: Version: 9.0 (Rev: Release, Compiled: 2020-11-10)
MAIN:  Running in parallel using 4 tasks.
MAIN:  Running with just one thread per task.
...
ResultOutputSolver: Saving in unstructured VTK XML (.vtu) format
ResultOutputSolver: -------------------------------------
ElmerSolver: *** Elmer Solver: ALL DONE ***
ElmerSolver: The end
SOLVER TOTAL TIME(CPU,REAL):       148.65      148.65

Visualization

The demo does not explicitly discuss how to visualize the results. However, this was covered in the Parallel Computing with Elmer talk.

Parallel Computing with Elmer

ParaView needs to be installed. Looks like they need to fix their installation dialog to not cut off text – or better yet, replace that long string with a user-friendly one. To visualize the ElmerSolver results, launch ParaView then open the parallel vtu file (*.pvtu).

Opening the Parallel VTU File

When the file opens, click on the green Apply button in the Properties panel. Notice that the pressure variable is preselected in the dropdown on the toolbar.

Visualizing Pressure in ParaView

Switching to the velocity variable shows all all blue rendering due to the 0 velocity. To see the animated visualization:

  1. Click on the slice button (4th from left in screenshot).
  2. Click on “Z Normal” in the Plane Parameters section on the Properties pane.
  3. Click on Apply.
  4. Click on the play button on the toolbar to see the animation of the flow.

The last part of the demo shows how to view the partitions in the mesh:

  1. Deselect the eye on Slice1 in the Pipeline Browser.
  2. Select the eye on “flow_gcr_t00” in the Pipeline Browser.
  3. Go to the Filter menu > Alphabetic > Connectivity.
  4. Click on the Apply button.

Change the selection in the combobox in the representation toolbar from from Surface to Surface with Edges to see how the partition seems to be based on the number of elements involved.

View of Surface with Edges

ParaView’s source code is available on GitLab so naturally, I’ll try to build it one of these fine days.


Categories: Crystal Growth, Elmer

Crystal Growth Simulation Failure

After completing the mesh generation in Crystal Growth Simulation – Part 1, I decided to investigate how to run the actual crystal growth simulation using Elmer. On Windows, Elmer needs to either be installed into “Program Files or be present in the path to avoid this error:

$ python run.py
crucible
melt
crystal
inductor
...
using material steel-1.4541 from self.materials_dict
using material vacuum from self.materials_dict
Wrote sif-file.
Starting simulation  ./simdata/2022-07-25_13-17_ss_test-cz-induction_vacuum  ...
Traceback (most recent call last):
  File "D:\dev\repos\fem\research\test-cz-induction\run.py", line 32, in <module>
    sim.execute()
  File "d:\dev\repos\fem\research\opencgs\opencgs\sim.py", line 182, in execute
    run_elmer_grid(self.sim_dir, "case.msh")
  File "C:\Python310\lib\site-packages\pyelmer\execute.py", line 28, in run_elmer_grid
    subprocess.run(args, cwd=sim_dir, stdout=f, stderr=f)
  File "C:\Python310\lib\subprocess.py", line 501, in run
    with Popen(*popenargs, **kwargs) as process:
  File "C:\Python310\lib\subprocess.py", line 966, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Python310\lib\subprocess.py", line 1435, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

I tried installing a locally built Elmer – see How to Build Elmer on Windows. Unfortunately, the layout created by setup defaulted to “C:\Program Files\Elmer 9.0-1b8e4f7ec” (ending with the hash of the commit I built, instead of “Release”). Worse still, it didn’t launch because of missing DLLs. I need to take a closer look at how NSIS is creating the setup executable, and also figure out how to set up the publisher so that a certificate is displayed with the UAC prompt. I ended up installing the publicly downloadable Elmer then retrying the python script.

using material steel-1.4541 from self.materials_dict
using material vacuum from self.materials_dict
Wrote sif-file.
Starting simulation  ./simdata/2022-07-25_17-23_ss_test-cz-induction_vacuum_1  ...
['ERROR:: systemc: Command exit status was 1'] [] {}
Finished simulation  ./simdata/2022-07-25_17-23_ss_test-cz-induction_vacuum_1  .
Post processing...
Traceback (most recent call last):
  File "D:\dev\repos\fem\research\test-cz-induction\run.py", line 32, in <module>
    sim.execute()
  File "d:\dev\repos\fem\research\opencgs\opencgs\sim.py", line 189, in execute
    self._postprocessing_probes()
  File "d:\dev\repos\fem\research\opencgs\opencgs\sim.py", line 139, in _postprocessing_probes
    names_data = self._read_names_file(self.sim_dir + "/results/probes.dat.names")
  File "d:\dev\repos\fem\research\opencgs\opencgs\sim.py", line 129, in _read_names_file
    with open(names_file) as f:
FileNotFoundError: [Errno 2] No such file or directory: './simdata/2022-07-25_17-23_ss_test-cz-induction_vacuum_1/02_simulation/results/probes.dat.names'

A dialog popped up asking whether I wanted to grant some elmer process access to the domain network or private, etc, and I’m not sure if I took too long to answer it but the script continued with the above failure. This looks like a bug in opencgs – why continue post processing when the simulation has failed – I filed this issue to let the author know: Do not start post-processing when simulation fails · Issue #2 · nemocrys/opencgs (github.com).

I had no idea why the simulation is failing so I used Process Explorer to see which command line was being used to invoke ElmerGrid and ElmerSolver. I was able to then manually invoke ElmerSolver and notice a segmentation fault!

ELMER SOLVER (v 9.0) STARTED AT: 2022/07/25 17:23:13
ParCommInit:  Initialize #PEs:            1
MAIN: 
MAIN: =============================================================
MAIN: ElmerSolver finite element software, Welcome!
...
MAIN: Reading Model: case.sif
 Caught LUA error:[string "loadfile("C:/Program Files (x86)/Elmer/shar..."]:1: attempt to call a nil value
 Caught LUA error:[string "loadstring(readsif("case.sif"))()"]:1: attempt to call global 'readsif' (a nil value)
LoadInputFile: Scanning input file: case.sif
LoadInputFile: Scanning only size info
...
OptimizeBandwidth: Half bandwidth after optimization: 1447
OptimizeBandwidth: ---------------------------------------------------------
'ViewFactors' is not recognized as an internal or external command,
operable program or batch file.
ERROR:: systemc: Command exit status was 1
RadiationFactors: Message
RadiationFactors: All done time (s)                      4.9000E-02
...
      60 0.2041E-08
      61 0.3710E-09
ComputeChange: NS (ITER=11) (NRM,RELC): ( 0.17032920E-04 0.41334346E-05 ) :: statmagsolver
StatMagSolve: Convergence after 11 iterations
StatMagSolve: Joule Heating (W):      4.1375E+01
StatMagSolve: All done, exiting
StatMagSolve: ------------------------------------------------
ComputeChange: SS (ITER=1) (NRM,RELC): ( 0.17032920E-04  2.0000000     ) :: statmagsolver
HeatSolve:  Found Control Point at distance:   0.0000000000000000
HeatSolve:  Control Point Index:           0
HeatSolve: Using Steady-state Heater Control
HeatSolve: 
HeatSolve: 
HeatSolve: -------------------------------------
HeatSolve:  TEMPERATURE ITERATION           1
HeatSolve: -------------------------------------
HeatSolve: 
HeatSolve: Starting Assembly...

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0xffffffff
#1  0xffffffff
#2  0xffffffff
...
#19  0xffffffff
#20  0xffffffff

Looking at the simulation folders such as “simdata\2022-07-25_22-52_ss_test-cz-induction_vacuum_1\02_simulation” revealed that there are ElmerGrid and ElmerSolver log files present with this info, so I didn’t even need Process Explorer! At this point, I have to save the SIGSEGV investigation for another day.


Crystal Growth Simulation – Part 1

Ever since I read chapter 2 of Fabrication Engineering at the Micro- and Nanoscale, I have been interested in simulating crystal growth, primarily because of the multi-discplinary nature of this problem: finite element modeling software, modeling using high performance computing, the physics of the problem (governing laws and equations), and visualization techniques. I had run across Elmer when looking up crystal growth examples a while back. About two weeks ago, I was watching this Elmer demo in which they used gmsh to show the geometry for the simulation.

I wanted to learn more about Gmsh and how it is used, so I took a detour into this Gmsh introductory video. It was an excellent tour of Gmsh and its features.

Gmsh for Crystal Growth

At this point, I wanted to see how Gmsh could be used for crystal growth simulation so I searched for “crystal growth gmsh“. I was very pleased to find this paper: Development and validation of a thermal simulation for the Czochralski crystal growth process using model experiments – ScienceDirect. It appeared to have a corresponding set of slides and a talk as well. It especially piqued my interest that there is a script to set up a mesh for a Czochralski furnace.

Running the Code

Thankfully, the code in this paper is open source. I already had python installed so the first thing I tried was executing run.py in Git bash

$ git clone https://github.com/nemocrys/test-cz-induction
$ cd test-cz-induction
$ python run.py
Traceback (most recent call last):
  File "D:\dev\repos\fem\research\test-cz-induction\run.py", line 5, in <module>
    import opencgs.control as ctrl
ModuleNotFoundError: No module named 'opencgs'

Without using my brain, I decided to install pyelmer since I knew it was a dependency (from the talk).

$ pip install pyelmer
Collecting pyelmer
  Downloading pyelmer-1.0.0-py3-none-any.whl (27 kB)
Collecting gmsh
  Downloading gmsh-4.10.5-py2.py3-none-win_amd64.whl (38.4 MB)
     |████████████████████████████████| 38.4 MB 285 kB/s
Collecting pyyaml
  Downloading PyYAML-6.0-cp310-cp310-win_amd64.whl (151 kB)
     |████████████████████████████████| 151 kB ...
Collecting matplotlib
  Downloading matplotlib-3.5.2-cp310-cp310-win_amd64.whl (7.2 MB)
     |████████████████████████████████| 7.2 MB 6.4 MB/s
Collecting pillow>=6.2.0
  Downloading Pillow-9.2.0-cp310-cp310-win_amd64.whl (3.3 MB)
     |████████████████████████████████| 3.3 MB ...
Collecting cycler>=0.10
  Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting numpy>=1.17
  Downloading numpy-1.23.1-cp310-cp310-win_amd64.whl (14.6 MB)
     |████████████████████████████████| 14.6 MB 6.4 MB/s
Collecting packaging>=20.0
  Downloading packaging-21.3-py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 2.5 MB/s
Collecting python-dateutil>=2.7
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     |████████████████████████████████| 247 kB ...
Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.4.4-cp310-cp310-win_amd64.whl (55 kB)
     |████████████████████████████████| 55 kB 1.6 MB/s
Collecting pyparsing>=2.2.1
  Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
     |████████████████████████████████| 98 kB 6.8 MB/s
Collecting fonttools>=4.22.0
  Downloading fonttools-4.34.4-py3-none-any.whl (944 kB)
     |████████████████████████████████| 944 kB 6.4 MB/s
Collecting six>=1.5
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, pyparsing, python-dateutil, pillow, packaging, numpy, kiwisolver, fonttools, cycler, pyyaml, matplotlib, gmsh, pyelmer
  WARNING: Failed to write executable - trying to use .deleteme logic
ERROR: Could not install packages due to an OSError: [WinError 2] The system cannot find the file specified: 'C:\\Python310\\Scripts\\f2py.exe' -> 'C:\\Python310\\Scripts\\f2py.exe.deleteme'

WARNING: You are using pip version 21.2.4; however, version 22.2 is available.
You should consider upgrading via the 'C:\Python310\python.exe -m pip install --upgrade pip' command.

Looks like I need to run this as an administrator.

C:\dev>pip install pyelmer
Collecting pyelmer
  Using cached pyelmer-1.0.0-py3-none-any.whl (27 kB)
Collecting pyyaml
  Using cached PyYAML-6.0-cp310-cp310-win_amd64.whl (151 kB)
Collecting gmsh
  Using cached gmsh-4.10.5-py2.py3-none-win_amd64.whl (38.4 MB)
Collecting matplotlib
  Using cached matplotlib-3.5.2-cp310-cp310-win_amd64.whl (7.2 MB)
Requirement already satisfied: python-dateutil>=2.7 in c:\python310\lib\site-packages (from matplotlib->pyelmer) (2.8.2)
Requirement already satisfied: packaging>=20.0 in c:\python310\lib\site-packages (from matplotlib->pyelmer) (21.3)
Requirement already satisfied: fonttools>=4.22.0 in c:\python310\lib\site-packages (from matplotlib->pyelmer) (4.34.4)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\python310\lib\site-packages (from matplotlib->pyelmer) (1.4.4)
Requirement already satisfied: pillow>=6.2.0 in c:\python310\lib\site-packages (from matplotlib->pyelmer) (9.2.0)
Requirement already satisfied: pyparsing>=2.2.1 in c:\python310\lib\site-packages (from matplotlib->pyelmer) (3.0.9)
Requirement already satisfied: numpy>=1.17 in c:\python310\lib\site-packages (from matplotlib->pyelmer) (1.23.1)
Collecting cycler>=0.10
  Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Requirement already satisfied: six>=1.5 in c:\python310\lib\site-packages (from python-dateutil>=2.7->matplotlib->pyelmer) (1.16.0)
Installing collected packages: cycler, pyyaml, matplotlib, gmsh, pyelmer
Successfully installed cycler-0.11.0 gmsh-4.10.5 matplotlib-3.5.2 pyelmer-1.0.0 pyyaml-6.0
WARNING: You are using pip version 21.2.4; however, version 22.2 is available.
You should consider upgrading via the 'C:\Python310\python.exe -m pip install --upgrade pip' command.

Installing pyelmer is not sufficient to avoid the “No module named ‘opencgs’” error. The solution is to clone the opencgs repo and use pip to install (as administrator) from that repo.

git clone https://github.com/nemocrys/opencgs
cd opencgs
D:\...\research\opencgs>pip install -e .
Obtaining file:///D:/dev/repos/fem/research/opencgs
Collecting meshio
  Using cached meshio-5.3.4-py3-none-any.whl (167 kB)
Collecting pandas
  Using cached pandas-1.4.3-cp310-cp310-win_amd64.whl (10.5 MB)
Requirement already satisfied: pyyaml in c:\python310\lib\site-packages (from opencgs==0.3.1) (6.0)
Requirement already satisfied: pyelmer in c:\python310\lib\site-packages (from opencgs==0.3.1) (1.0.0)
Collecting rich
  Using cached rich-12.5.1-py3-none-any.whl (235 kB)
Requirement already satisfied: numpy in c:\python310\lib\site-packages (from meshio->opencgs==0.3.1) (1.23.1)
Requirement already satisfied: python-dateutil>=2.8.1 in c:\python310\lib\site-packages (from pandas->opencgs==0.3.1) (2.8.2)
Collecting pytz>=2020.1
  Using cached pytz-2022.1-py2.py3-none-any.whl (503 kB)
Requirement already satisfied: six>=1.5 in c:\python310\lib\site-packages (from python-dateutil>=2.8.1->pandas->opencgs==0.3.1) (1.16.0)
Requirement already satisfied: matplotlib in c:\python310\lib\site-packages (from pyelmer->opencgs==0.3.1) (3.5.2)
Requirement already satisfied: gmsh in c:\python310\lib\site-packages (from pyelmer->opencgs==0.3.1) (4.10.5)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\python310\lib\site-packages (from matplotlib->pyelmer->opencgs==0.3.1) (1.4.4)
Requirement already satisfied: pyparsing>=2.2.1 in c:\python310\lib\site-packages (from matplotlib->pyelmer->opencgs==0.3.1) (3.0.9)
Requirement already satisfied: cycler>=0.10 in c:\python310\lib\site-packages (from matplotlib->pyelmer->opencgs==0.3.1) (0.11.0)
Requirement already satisfied: pillow>=6.2.0 in c:\python310\lib\site-packages (from matplotlib->pyelmer->opencgs==0.3.1) (9.2.0)
Requirement already satisfied: fonttools>=4.22.0 in c:\python310\lib\site-packages (from matplotlib->pyelmer->opencgs==0.3.1) (4.34.4)
Requirement already satisfied: packaging>=20.0 in c:\python310\lib\site-packages (from matplotlib->pyelmer->opencgs==0.3.1) (21.3)
Collecting commonmark<0.10.0,>=0.9.0
  Using cached commonmark-0.9.1-py2.py3-none-any.whl (51 kB)
Requirement already satisfied: pygments<3.0.0,>=2.6.0 in c:\python310\lib\site-packages (from rich->meshio->opencgs==0.3.1) (2.12.0)
Installing collected packages: commonmark, rich, pytz, pandas, meshio, opencgs
  Running setup.py develop for opencgs
Successfully installed commonmark-0.9.1 meshio-5.3.4 opencgs-0.3.1 pandas-1.4.3 pytz-2022.1 rich-12.5.1
WARNING: You are using pip version 21.2.4; however, version 22.2 is available.
You should consider upgrading via the 'C:\Python310\python.exe -m pip install --upgrade pip' command.

Trying to run gives an error about a missing module named objectgmsh. My assumption was the installing pyelmer was supposed to bring in all such modules.

D:\dev\repos\fem\research\test-cz-induction>python run.py
Traceback (most recent call last):
  File "D:\dev\repos\fem\research\test-cz-induction\run.py", line 5, in <module>
    import opencgs.control as ctrl
  File "d:\dev\repos\fem\research\opencgs\opencgs\__init__.py", line 3, in <module>
    import opencgs.geo
  File "d:\dev\repos\fem\research\opencgs\opencgs\geo.py", line 1, in <module>
    from objectgmsh import *
ModuleNotFoundError: No module named 'objectgmsh'

I notice while reviewing the paper yet again that it mentions the setup being in a Dockerfile. Sure enough, objectgmsh is a separate package that needs to be installed using pip. I upgrade pip for good measure since those warnings are getting annoying.

\test-cz-induction>pip install objectgmsh
Collecting objectgmsh
  Downloading objectgmsh-0.9-py3-none-any.whl (25 kB)
Requirement already satisfied: gmsh in c:\python310\lib\site-packages (from objectgmsh) (4.10.5)
Requirement already satisfied: numpy in c:\python310\lib\site-packages (from objectgmsh) (1.23.1)
Installing collected packages: objectgmsh
Successfully installed objectgmsh-0.9

I realize I should probably be running setup.py instead of run.py. Here’s the output now!

\test-cz-induction>python setup.py
crucible
melt
crystal
inductor
seed
insulation
crucible_adapter
axis_bt
vessel
axis_top
crucible 0.001999999999999999
melt 0.003
crystal 0.0004
inductor 0.001
seed 0.00017999999999999998
insulation 0.005
crucible_adapter 0.0039
axis_bt 0.0025
vessel 0.0025
axis_top 0.0025
filling 0.0263
Warning: Mesh size = 0 for bnd_melt. Ignoring this shape...
Warning: Mesh size = 0 for bnd_seed. Ignoring this shape...
Warning: Mesh size = 0 for bnd_crystal_side. Ignoring this shape...
Warning: Mesh size = 0 for bnd_crystal_top. Ignoring this shape...
Warning: Mesh size = 0 for bnd_axtop_side. Ignoring this shape...
Warning: Mesh size = 0 for bnd_axtop_bt. Ignoring this shape...
Warning: Mesh size = 0 for bnd_crucible_bt. Ignoring this shape...
Warning: Mesh size = 0 for bnd_crucible_outside. Ignoring this shape...
Warning: Mesh size = 0 for bnd_ins. Ignoring this shape...
Warning: Mesh size = 0 for bnd_adp. Ignoring this shape...
Warning: Mesh size = 0 for bnd_axbt. Ignoring this shape...
Warning: Mesh size = 0 for bnd_vessel_inside. Ignoring this shape...
Warning: Mesh size = 0 for bnd_vessel_outside. Ignoring this shape...
Warning: Mesh size = 0 for bnd_inductor_outside. Ignoring this shape...
Warning: Mesh size = 0 for bnd_inductor_inside. Ignoring this shape...
Warning: Mesh size = 0 for bnd_symmetry_axis. Ignoring this shape...
Warning: Mesh size = 0 for if_crucible_melt. Ignoring this shape...
Warning: Mesh size = 0 for if_melt_crystal. Ignoring this shape...
Warning: Mesh size = 0 for if_crystal_seed. Ignoring this shape...
Warning: Mesh size = 0 for if_seed_axtop. Ignoring this shape...
Warning: Mesh size = 0 for if_axtop_vessel. Ignoring this shape...
Warning: Mesh size = 0 for if_crucible_ins. Ignoring this shape...
Warning: Mesh size = 0 for if_ins_adp. Ignoring this shape...
Warning: Mesh size = 0 for if_adp_axbt. Ignoring this shape...
Warning: Mesh size = 0 for if_axbt_vessel. Ignoring this shape...
Info    : Meshing 1D...
Info    : [  0%] Meshing curve 3 (Line)
Info    : [ 10%] Meshing curve 4 (Line)
Info    : [ 10%] Meshing curve 15 (Line)
Info    : [ 10%] Meshing curve 16 (Line)
Info    : [ 10%] Meshing curve 17 (Line)
Info    : [ 10%] Meshing curve 18 (BSpline)
Info    : [ 20%] Meshing curve 19 (Circle)
...
Info    : [100%] Meshing surface 12 order 2
Info    : [100%] Meshing surface 13 order 2
Info    : [100%] Meshing surface 14 order 2
Info    : Surface mesh: worst distortion = 0.526353 (0 elements in ]0, 0.2]); worst gamma = 0.252838
Info    : Done meshing order 2 (Wall 0.990998s, CPU 0.75s)
-------------------------------------------------------
Version       : 4.10.5
License       : GNU General Public License
Build OS      : Windows64-sdk
Build date    : 20220701
Build host    : gmsh.info
Build options : 64Bit ALGLIB[contrib] ANN[contrib] Bamg Blas[petsc] Blossom Cgns DIntegration DomHex Eigen[contrib] Fltk Gmm[contrib] Hxt Jpeg Kbipack Lapack[petsc] MathEx[contrib] Med Mesh Metis[contrib] Mmg Mpeg Netgen NoSocklenT ONELAB ONELABMetamodel OpenCASCADE OpenCASCADE-CAF OpenGL OpenMP OptHom PETSc Parser Plugins Png Post QuadMeshingTools QuadTri Solver TetGen/BR Voro++[contrib] WinslowUntangler Zlib
FLTK version  : 1.4.0
PETSc version : 3.15.0 (real arithmtic)
OCC version   : 7.6.1
MED version   : 4.1.0
Packaged by   : nt authority system
Web site      : https://gmsh.info
Issue tracker : https://gitlab.onelab.info/gmsh/gmsh/issues
-------------------------------------------------------

Gmsh launches (although not as a separate process) with the generated mesh:

Crystal Growth Gmsh Model

setup.py outputs the path to the mesh after closing Gmsh.

Info    : Writing './simdata/_test/case.msh'...
Info    : Done writing './simdata/_test/case.msh'
Gmsh model created with objectgmsh.
Shapes: ['crucible', 'melt', 'crystal', 'inductor', 'seed', 'insulation', 'crucible_adapter', 'axis_bt', 'vessel', 'axis_top', 'filling', 'bnd_melt', 'bnd_seed', 'bnd_crystal_side', 'bnd_crystal_top', 'bnd_axtop_side', 'bnd_axtop_bt', 'bnd_crucible_bt', 'bnd_crucible_outside', 'bnd_ins', 'bnd_adp', 'bnd_axbt', 'bnd_vessel_inside', 'bnd_vessel_outside', 'bnd_inductor_outside', 'bnd_inductor_inside', 'bnd_symmetry_axis', 'if_crucible_melt', 'if_melt_crystal', 'if_crystal_seed', 'if_seed_axtop', 'if_axtop_vessel', 'if_crucible_ins', 'if_ins_adp', 'if_adp_axbt', 'if_axbt_vessel']

The next step will be to figure out how to simulate crystal growth using pyelmer.