Mpi

Use of MPI in HemeLB

I have created minimal wrapper classes for some MPI types that allow them to be used in a more C++/OO way.

As the code is improved, I would recommend that further types also be wrapped.

The largest omission at the moment is lack of support for asynchronous communications (i.e. anything that uses an MPI_Status object).

Calling MPI routines

I have created a simple macro HEMELB_MPI_CALL that should be used for making all unwrapped calls to the underlying MPI routines. The advantage is that the error codes are always checked and if not successful, it throws an exception with a good error message.

Most MPI routines need to be used as before, with the exception outlined above. So

HEMELB_MPI_CALL(MPI_Routine,
    (arg1, arg2, arg3, comms)
);

Note that the arguments are in parentheses (Varadic macros aren't allowed without C++11). Proper examples below.

Classes

• MpiEnvironment
• MpiCommunicator
• IOCommunicator - this is a subclass of MpiCommunicator that adds the concept of an "IORank" which is used a lot in the code.
• MpiGroup
• MpiFile

MpiDataType / MpiDataTypeTraits

There are templated functions for getting the MPI_Datatype corresponding to a type or variable below. Use as:

Foo bar;
MPI_Datatype tp = MpiDataType(bar);
// or alternatively
tp = MpiDataType<Foo>();

The generic version, for custom classes, is implemented using traits. You must specialize the template, e.g.

template<>
MPI_Datatype hemelb::net::MpiDataTypeTraits<Foo>::RegisterMpiDataType()
{
  // Create the type
  MPI_Type_commit(&type);
  return type;
}

Built-in MPI types have specializations defined in the implementation.

For creating the MPI_Datatype, there are four macros to remove a lot of boiler-plate:

1. HEMELB_MPI_TYPE_BEGIN(outType, Type, n) - Begin the definition block.

• outType: The name you want the MPI_Datatype variable to have. It will be declared in the current scope.
• Type: The class that this datatype is describing.
• n: The number of members to be added to the description.

1. HEMELB_MPI_TYPE_ADD_MEMBER_N(name, count) - Add one entry to the datatype definition

• name: The name of the member to add.
• n: The number of (consecutive) elements.

1. HEMELB_MPI_TYPE_ADD_MEMBER(name) - As above but with n=1.
2. HEMELB_MPI_TYPE_END(outType, Type) - End the definition block, arguments should be identical to HEMELB_MPI_TYPE_BEGIN

The effect of these will be to declare a local variable of type MPI_Datatype and name outType which describes how MPI should serialise variables of class Type. Note that this does not commit the datatype to MPI.

Examples

Create a communicator with only the even ranks:

net::MpiCommunicator comm = net::MpiCommunicator::World();
std::vector<int> evenRanks;

for (int i = 0; i < comm.Size(); ++i) {
  if (i % 2 == 0)
    evenRanks.push_back(i);
}

net::MpiGroup evenGroup = comm.Group().Include(evenRanks);
net::MpiCommunicator evenComms = comms.Create(evenGroup);

Broadcast something to all ranks:

net::MpiCommunicator comms = net::MpiCommunicator::World();

int importantNumber;
if (comms.Rank() == 0)
  importantNumber = ReadFromFile();

HEMELB_MPI_CALL(MPI_Bcast, (&importantNumber, 1, net::MpiDataType(importantNumber), 0, comms));

Note the use of MpiDataType instead of MPI_INT.

Describe a particle class to MPI:

HEMELB_MPI_TYPE_BEGIN(type, vis::streaklinedrawer::Particle, 3);
  HEMELB_MPI_TYPE_ADD_MEMBER(position);
  HEMELB_MPI_TYPE_ADD_MEMBER(vel);
  HEMELB_MPI_TYPE_ADD_MEMBER(inletID);
HEMELB_MPI_TYPE_END(type, vis::streaklinedrawer::Particle);