ExaCA is a cellular automata (CA) code for grain growth under additive manufacturing conditions, created by ExaAM within the Exascale Computing Project.
ExaCA-Kokkos uses Kokkos and MPI for parallelism and JSON for input files.
| Dependency | Version | Required | Details |
|---|---|---|---|
| CMake | 3.11+ | Yes | Build system |
| Kokkos | 4.0+ | Yes | Portable on-node parallelism |
| MPI | GPU Aware if CUDA/HIP Enabled | Yes | Multi-node parallelism |
| json | 3.10+ | Yes | Input parsing |
| GoogleTest | 1.10+ | No | Unit test framework |
| CUDA | 9+ | No | Programming model for NVIDIA GPUs |
| HIP | 3.5+ | No | Programming model for AMD GPUs |
CMake must be available to build ExaCA and Kokkos. The underlying parallel programming models and MPI are available on most systems and can generally be found automatically by CMake. Note these dependencies must all be installed first, if not available. Kokkos is also available on many systems; if not, obtain the desired version:
git clone https://github.com/kokkos/kokkos.git --branch 4.0.00
ExaCA is available on GitHub, by default starting from the current master branch:
git clone https://github.com/LLNL/ExaCA.git
Note that ExaCA runs with the default enabled Kokkos backend (see https://github.com/kokkos/kokkos/wiki/Initialization).
ExaCA has been tested with the Serial, OpenMP, Threads, CUDA, and HIP backends.
If Kokkos is not already installed on your system, configure, build, and install Kokkos:
# Change this path to desired Kokkos installation location
export KOKKOS_INSTALL_DIR=`pwd`/install/kokkos
# Change this path to Kokkos source
cd ./kokkos
# Configure Kokkos
cmake \
-B build \
-D CMAKE_BUILD_TYPE="Release" \
-D CMAKE_INSTALL_PREFIX=$KOKKOS_INSTALL_DIR \
-D Kokkos_ENABLE_OPENMP=ON
# Build Kokkos
cmake --build build
# Install Kokkos
cmake --install build
cd ../
Note that there are other host backends available. Kokkos architecture flags can also be set to improve performance and must match the hardware you run on (e.g. -DKokkos_ARCH_POWER9=ON); see Kokkos architecture flag details.
Similar to the CPU build above, Kokkos can instead be configured and built for NVIDIA GPUs:
# Change this path to desired Kokkos installation location
export KOKKOS_INSTALL_DIR=`pwd`/install/kokkos
# Change this path to Kokkos source
cd ./kokkos
# Check the GPU architecture flag matches the hardware
# Configure Kokkos
cmake \
-B build \
-D CMAKE_BUILD_TYPE="Release" \
-D CMAKE_INSTALL_PREFIX=$KOKKOS_INSTALL_DIR \
-D Kokkos_ENABLE_CUDA=ON \
-D Kokkos_ENABLE_CUDA_LAMBDA=ON \
-D Kokkos_ARCH_VOLTA70=ON
# Build Kokkos
cmake --build build
# Install Kokkos
cmake --install build
cd ../
Note the two flags needed for the Kokkos::Cuda backend. The Kokkos architecture flag must match the hardware you run on. Kokkos will automatically redirect the (default) host compiler to nvcc in the example above. By default, the host will use Kokkos::Serial; other parallel host backends can also be used, e.g. by adding -D Kokkos_ENABLE_OPENMP as was done above.
To build Kokkos for HIP the hipcc compiler must be explicitly passed, along with architecture and backend flags analogous to the previous examples:
cd ./kokkos
# Configure Kokkos
cmake \
-B build \
-D CMAKE_BUILD_TYPE="Release" \
-D CMAKE_CXX_COMPILER=hipcc \
-D CMAKE_INSTALL_PREFIX=install \
-D Kokkos_ENABLE_HIP=ON \
-D Kokkos_ARCH_VEGA908=ON
# Build Kokkos
cmake --build build
# Install Kokkos
cmake --install build
cd ../
Once Kokkos and MPI are installed, ExaCA can be built:
# Change this path to desired ExaCA installation location
export EXACA_INSTALL_DIR=`pwd`/install/exaca
# Change this path to Kokkos installation location
export KOKKOS_INSTALL_DIR=`pwd`/install/kokkos
# Change this path to ExaCA source
# Configure ExaCA
cd ./ExaCA
cmake \
-B build \
-D CMAKE_BUILD_TYPE="Release" \
-D CMAKE_PREFIX_PATH=$KOKKOS_INSTALL_DIR \
-D CMAKE_INSTALL_PREFIX=$EXACA_INSTALL_DIR
# Build ExaCA
cmake --build build
# Install ExaCA
cmake --install build
cd ../
Kokkos will forward the compilation flags and details to ExaCA automatically.
By default, ExaCA will download the JSON library dependency used for input files. This automatic download does not work on all systems; a separate build of this library can be done instead. As with the dependencies described above, first obtain the source:
git clone https://github.com/nlohmann/json
And then build the json library (header only):
# Change this path to desired JSON installation location
export JSON_INSTALL_DIR=`pwd`/install/json
cd json
# Configure json
cmake \
-B build \
-D CMAKE_BUILD_TYPE="Release" \
-D CMAKE_INSTALL_PREFIX=$JSON_INSTALL_DIR \
-D JSON_BuildTests=OFF
# Build json
cmake --build build
# Install json
cmake --install build
cd ../
Then add this install path to the ExaCA configuration (example above) together with the path to Kokkos -D CMAKE_PREFIX_PATH=$KOKKOS_INSTALL_DIR;$JSON_INSTALL_DIR and build ExaCA.
Unit tests can be run if the ExaCA_ENABLE_TESTING CMake option is enabled in the build and if the GoogleTest framework is available on the system or built locally with the install path passed to ExaCA (see the previous section describing the JSON build and pointing ExaCA to the installation).
After building, tests can be run with cmake --build build --target test from the source directory (presuming build/ is the relative location of the build folder). Tests are automatically generated for all enabled Kokkos backend.
ExaCA-Kokkos runs using an input file, passed on the command line. Example problems are provided in the examples/ directory - a separate README file located in the examples/ directory goes into more detail on the problem types, the optional and required arguments needed for each problem type, and additional files used by ExaCA. The example input files present in this repository are:
Inp_DirSolidification.json: simulates grain growth from a surface with a fixed thermal gradient and cooling rateInp_SmallDirSolidification.json: a smaller and simpler version of the previousInp_SpotMelt.json: simulates overlapping spot melts with fixed a fixed thermal gradient and cooling rateInp_SmallSpotMelt.json: a smaller and simpler version of the previous
Example problems only possible with external data (available via https://github.com/LLNL/ExaCA-Data):
Inp_SingleLine.json: simulates melting and solidification of a single line of melt pool dataInp_TwoLineTwoLayer.json: simulates two layers consisting of segments of two overlapping melt pools
Run by calling the created executable with an ExaCA input file:
mpiexec -n 1 ./build/install/bin/ExaCA-Kokkos examples/Inp_DirSolidification.json
Automated input file generation using Tasmanian (https://tasmanian.ornl.gov/)
Within the utilities directory, an example python script for the generation of an ensemble of input files is available. By running the example script TasmanianTest.py, 69 ExaCA input files are generated with a range of heterogenous nucleation density, mean nucleation undercooling, and mean substrate grain size values, based on the ranges in python code (N0Min-N0Max, dTNMin-dTNMax, and S0Min-S0Max), respectively. Running the python script from the ExaCA source directory, via the command
python utilities/TasmanianTest.py PathToTemperatureFile1 PathToTemperatureFile2 ...
the script will generate an ensemble of input files in the examples directory, for a series of simulations that will use the thermal history or histories described in PathToTemperatureFile1(s) being repeated for a certain number of layers (56 in this example). If a simulation repeating multiple thermal histories is desired (for example, and even layer and an odd layer scan pattern), both paths to/file names of the thermal history data should be given on the command line. Running this code will generate N = 1 to 69 input files named examples/Inp_TasmanianTest_[N].json. Other CA inputs, such as the time step or cell size, must be adjusted manually inside of the python script. Separate instances of ExaCA can be run with each ensemble member to probe microstructure dependency on nucleation and substrate.
If the "Print file of grain misorientations" option is turned on within an input file, ExaCA will output a scalar field "Angle_z" as a vtk file ending with "Misorientations.vtk". Angle_z corresponds to the orientation (in degrees) of a given grain relative to the positive Z direction in a simulation (the thermal gradient direction for directional solidification problems, the build/layer offset direction for other problems). Epitaxial grains (from the initial grain structure or powder layer) are assigned values between 0 and 62.7, while nucleated grains (not present in the initial grain structure) are assigned values between 100 and 162.7 (the offset of 100 is simply used to ensure the two types of grains are differentiated, but a nucleated grain with Angle_z = 135 actually has a misorientation of 35 degrees).
If the "Print Paraview vtk file" option is turned on within an input file, post-processing can be performed on the output data set. This functionality is a separate executable from ExaCA, located in the analysis/ directory and is linked to the ExaCA library for input utilities.
Specifying debug check options can be done to print various ExaCA data fields to files following simulation initialization. The "reduced" debug check will print "CritTimeStep" (the time step at which each cell goes below the liquidus for the final time), "LayerID" (the layer associated with each cell going below the liquidus for the final time, with layer 0 being the first layer, and -1 for all cells that did not undergo solidification) and "CellType" (integers corrsponding to cell types specified in src/CAtypes.hpp). The "extensive" debug check will, in addition to the "reduced" data fields, also print "UndercoolingChange" (the rate at which a cell cools per time step after reaching its "CritTimeStep" value), "UndercoolingCurrent" (the initial undercooling of each cell), and "Melted" (1 for cells that are part of the melt pool, 0 for cells that are not).
ExaCA can optionally print the system state at intermediate time values as part of a series of vtk files that can be read by Paraview to make animations, if the "Print intermediate output frames" option is turned on. "Increment to separate frames" is the separation between intermediate output files in microseconds - if there is a long time period between solidification events (such as two overlapping melt pools formed via line scan with a long dwell time between them), setting "Intermediate output even if system is unchanged from previous state" to off will skip printing of those files.
Running ExaCA for the test problem Inp_DirSolidification.txt yields the output files TestProblemDirS.vtk (containing LayerID, GrainID, and Melted data) and TestProblemDirS.json (containing information regarding the simulation parameters used, simulation dimensions, and some timing data). To analyze this data, run grain_analysis (installed in the same location as ExaCA-Kokkos), with two command line arguments: the first being the path to/name of the analysis input file, and the second being the path to and filename (excluding extensions) of the .vtk and .json files associated with the data set of interest.
./build/install/bin/grain_analysis analysis/examples/AnalyzeDirS.json TestProblemDirS
Within the analysis/examples directory, there are example analysis input files. Note that the microstructure data files TestProblemDirS.vtk and TestProblemDirS.json must both be in the location given on the command line.
The analysis executable, in addition to outputting grain statistics, can also output files that can be further post-processing in Matlab using the MTEX toolbox to generate pole figures, inverse pole figures, and inverse pole figure-colored cross-sections. More details on this are provided in analysis/README.md
If you use ExaCA in your work, please cite the following paper. In addition, cite the current release or version used from Zenodo.
We encourage you to contribute to ExaCA. Please check the contribution guidelines.
ExaCA is distributed under an MIT license.
LLNL-CODE-821827