disable mutlithreading for Import FreeSurfer grey/white segmentation to Morphologist

[HdeV999]

Dear BrainVISA team,

I've got my workflows set up to run the "Import FreeSurfer grey/white segmentation to Morphologist" on a high performance cluster for a big database. I get some errors when scaling up the number of subjects I run using this workflow, and it seems that it may be related to the processings that are internally parallelized, like the "cortical folds graph" and "sulci recognition using CNN" as mentioned in issue #70 Set CPU usage for BrainVISA on HPC.
It is not clear to me where in the workflow I should specify turning off multithreading. Here is what my workflow looks like for one subject:
"0": {
"class": "soma_workflow.client_types.Job",
"name": "1. Import FreeSurfer grey/white segmentation to Morphologist",
"join_stderrout": false,
"priority": 101,
"native_specification": null,
"parallel_job_info": null,
"disposal_timeout": 168,
"env": null,
"use_input_params_file": false,
"has_outputs": false,
"configuration": {},
"command": [
"python",
"-m",
"brainvisa.axon.runprocess",
"Import_FROM_FreeSurfer_TO_Morpho",
[and here a succession of paths and parameters directly related to the expected inputs and outputs of the "Import FreeSurfer grey/white segmentation to Morphologist", with no parameter to allow multithreading. I don't think they are relevant, except maybe for the option "use morphologist" where I chose "in batch" because the other options were "don't use it" or "graphically" which in not compatible with my cluster]
}
Is there any way of disabling internal multithreading using this function, or should I just disable the use of morphologist and create a new workflow to run afterwards and explicitly disable multithreading?

Thank you kindly!

[denisri]

Hi,
We have to add the option to limit threading for this process. We have run into similar threading problems on clusters, where threading leads to hangouts or incredibly slowed processing. The only program (I think) that is making use of heavy threading is the cortical graph attributes processing, which is used in "Cortical folds graph" and "sulci recognition using CNN" steps of the Morphologist pipeline. I will add the option and produce a patch when I find a minute for that.
Denis

[denisri]

I have added parameters in both Morphologist main pipeline and the Import_FROM_FreeSurfer_TO_Morpho process, in the master git branches. If I remember, you are using a developer container and can update / rebuild from sources ? If not you have to download the modified process, https://github.com/brainvisa/brainvisa_freesurfer/blob/master/brainvisa/toolboxes/freesurfer/processes/Import/Import_FROM_FreeSurfer_TO_Morpho.py and save it in your ~/.brainvisa/processes/ directory so that it will get used instead of the former one.
Denis

[HdeV999]

Thank you so much! Yes I have the developer version, I'll update.