API and Functionality

Summary of discussion on API and functionality for GHEX

Design dimensions

1. Regular vs. unstructured Grid
2. Message passing vs. remote direct memory access
3. Backends (GPU, MC, ...)
4. Threading model (work threads vs. computing task)

2x2x2x2=16

Neighbors

• Neighbors are found using a collective setup phase
• We rely on MPI for this (for now)
• Neighbor concepts must be designed for both structured (directions?) and unstructured grids
• Unstructured and structured domains must be able to communicate (for FVM)

Communication Protocol

• Communication libraries shall be abstracted by basic API for Message Passing and Remote Memory
• Inspiration: UCX, libfabric, MPI, ...
• Or we could use UCX directly (it is already a low-level abstraction) ...?
• MPI is still required for the setup phase (for now )

Runtime vs. Compile time

• Buffer space computed and allocated in "setup phase" (requires compile time knowledge of fields to be communicated)
• Buffer space computed and allocated in actual exchange (performance?)
• Buffer space computed the first time the data is collected (similar to std::vector, but this would be hard on GPUs) and then reused in the subsequent calls (and eventually adapted is more is needed)
• Should phases like packing, unpacking, send, recv etc be exposed publicly?

Threading and Programming model

• GHEX shall be implemented for threaded application
• Different threading models with different requirements on execution enginge and communication

Additional Notes

• We should not focus on the simple GCL test case
• FVM model should be our focus (mixed structured and unstructured)
• Look at papers for hybrid structured/unstructured codes/communication