running cluster openMPI inside bv container

[HdeV999]

Hello BrainVISA team!
I've been trying to get brainvisa workflows running in parallel on my current cluster, and my current difficulty is that the cluster uses openMPI 4.0.4 but within my bv bash the openMPI version I have is 2.1.1.
How can I get my brainvisa container (installed on the cluster) to run the cluster's openMPI?
Currently investigating that from my side but maybe you have a quick and easy answer to that. Many thanks!

[denisri]

Hi,
Well, I think it depends on what you are doing with MPI. In Brainvisa we are almost not using it, except for the Soma-Workflow MPI runner, which allows to run a workflow in a single parallel cluster job. As MPI is used internally by soma-workflow here (I mean, to communicate with itself), I guess its version does not really matter and it will not conflict with other implementation on the host system of the cluster nodes. But I admit I'm not a specialist of it and I could ba wrong.
The "regular" distributed computing mode of Soma-Workflow (used by brainvisa) is not making use of MPI, but rund a job on the cluster for each process execution.
Denis

[HdeV999]

Hi Denis,
Yeah, it's precisely to run a workflow in a parallel cluster job that I need MPI. I believe it does create an issue as when I try to launch my cluster job using multiple tasks/nodes, I get an error message stating that soma workflow needs at least two workers (one master and one slave) to run. It's supposed to be the case so I believe the communication goes wrong, and I've seen on a thread that MPI version mismatch can really be problematic...

[denisri]

I assume you have followed instructions on https://brainvisa.info/soma-workflow/sphinx/MPI_workflow_runner.html and that you are using the MPI mode because the number of jobs is limited on your cluster.
I would have thought MPI is used only internally in the bv job. You have a workflow file, for instance ~/mywf.workflow, and want it to run in a job with 20 CPUs. You would create a job reserving 21 CPUs (one more for the MPI scheduler), which would run the following commandline:

bv  mpirun -n 21 python3 -m soma_workflow.MPI_workflow_runner local_MPI --workflow $HOME/mywf.workflow

If I'm right (I have not checked or tried it right now and I have not used for a while), mpirun should run inside the bv container here. The cluster job does not need to require any MPI facility I guess.

[HdeV999]

Thank you so much Denis, this command you just gave allowed me to tweak my own command a bit and my parallel workflow seems to be working wonders!
I've been stuck on this for a while, and this parallel computing method opens so many doors for my research, I'm really grateful!

[denisri]

Cool :)