MultiCommsPerTimestep

h2. Functional design overview

There are multiple Phases per timestep. Each !Phase consists of multiple Steps. Each Concern class implements each Action it may want to carry out as a class method. A StepManager manages calling each Action, for each Phase, for each Step. No guarantee is made about the order of Actions for a Step.

Each !Phase consists of the following !Steps:

• RequestComms
• Receive
• PreSend
• Send
• PreReceive
• Wait
• PostReceive

In addition, there are BeginIteration and EndIteration steps before the first phase and after the last phase.

The management of the number of Phases, and the registration of Actions with steps, is intended eventually to be an automated process, based on the identification of prerequisite steps, and a graph analysis. However, in the first instance, this will be manual.

h2. Implementation details

• A base StepManager class will be derived from, and the choice of number of phases and the registration of the actions with the phases will occur within the derived constructor.
• The StepManager interface to execute a timestep will be called by SimulationMaster within its timestep method.
• Each !Concern which wishes to register actions with the StepManager will need derive from !Concern.
• Due to absence of an effective member-function-callback-registry in C++ (without using boost), the StepManager will dispatch calls to steps through an enum-based dispatch, via the method bool !CarryOutStep(unsigned int step_label)
• For simple classes which wish to register with only one timestep, IteratedActor will be rewritten as a !SinglePhaseConcern class, with a !Register[WithPhase] method to quickly register all its actions with the nth phase.
• The Net object itself becomes a concern, implementing the Send() Receive() and Wait() steps of each phase.

Examples:

Registration:

enum phase {one,two,three};
[[SomeConcernClass]] myConcern;
my[[StepManager]].Register(myConcern,one,[[StepManager]]::[[RequestComms]],myConcern::Advance);
my[[StepManager]].Register(myConcern,one,[[StepManager]]::[[PreReceive]],myConcern::Share);

Implementation of a concern:

class [[SomeConcernClass]]: public Concern{
public:
enum Step {Advance,Share}
bool [[CarryOutStep]](Step todo){
    switch todo {
        case Advance: return [[DoAdvance]]();
        case Share: return [[DoShare]]();
     }
}
private:
bool [[DoAdvance]]();
bool [[DoShare]]();
}

Note the annoying boilerplate here, could be obviated using boost. Using enums to name a phase gives fairly rapid dispatch, without complicated brittle bound-member-function-pointer code. Alternatively: we could use string names, and dispatch via a map<>

h2. Route to the dependencies analysis solution

Following implementation of the above, we intend to move toward a design where instead of specifying which step to use for each action, we specify prerequisite actions for each action.

This will be implemented by creating a subclass of the step manager, which uses the Request method of the current class, but implements a requisite based interface.

We would then move the specification of steps or dependencies into files, rather than in code. We could, if we wish, retain the specific-step specification option, as well as dependency specification, to allow for gradual migration.

h2. Older notes below

For various bits of the code it would be useful to do more than one inter-core communication per timestep.

h3. Method 1

Do it manually, effectively duplicate the communications steps we have now, so communicating bits of the code implement methods like

[[PreComms1]],
[[DoComms1]],
[[MidComms1]],
[[BetweenComms]],
[[DoComms2]],
[[MidComms2]],
[[PostComms2]]

This could be brittle, is quite an ugly solution (even if we name the methods better), and if we later need 3 steps for whatever reason it'll be a pain to add the functionality.

h3. Method 2

The things that we currently call IteratedActors, instead of having a series of predefined step names, can register a series of Tasks with some kind of TaskController. Tasks can be computational or communicative and have some kind of enumerated task-type along with a vector of preceding task ids and a functional (no arguments for a computational task, a single Net argument for a communicative task).

TaskController is initialised with the set of all tasks, and works out an execution plan. I have some ideas of how this would be done in a way that is able to coalesce communications and which prioritises tasks necessary before send tasks. I'd also implement a list of the communications in progress, to use MPI Test functions in order to progress dependant tasks soon after they finish (rather than at an arbitrary point in the timestep).

This is more complicated, more can go wrong. But potentially, could get a more optimal result because the communication points don't have to remain fixed at arbitrary positions in the cycle.