Remove joblib in favor of ParallelProcessExecutor

[alejoe91]

No description provided.

[samuelgarcia]

This is a good news!

[zm711]

I actually had a question about multiprocessing in general. For n_jobs=-1 you do os.cpu_count(), but sometimes I read it is best to leave at least one core behind for the computer itself. I can understand that os.cpu_count()-1 could lead to 0 if someone is on a single core, but is there really a potential problem with n_jobs=-1 or do you think that's fine for users to default to using all cores? (I really don't know multiprocessing in the python context :( sorry). But I love the branch name!

[samuelgarcia]

In spikeinetrface, you can do n_jobs=-2to leave one core free.
You can also do n_jobs=0.8 you use 80% of cores.

[h-mayorquin]

In spikeinetrface, you can do n_jobs=-2to leave one core free. You can also do n_jobs=0.8 you use 80% of cores.

@CodyCBakerPhD You have some evolutionary convergence here in yours and Dandi's preference:
https://github.com/catalystneuro/neuroconv/blob/1a7ae7a62e502614fa6b52e511310736aed55552/src/neuroconv/tools/nwb_helpers/_models/_zarr_models.py#L156-L163

Sam went further and even made the datatype a float though.

[zm711]

Thanks Sam, I guess my question is more out of scope for this PR, but I meant more in the sense that some users have had issues with n_jobs=-1 and so I wonder if there's a more sensible default (like n_jobs=-2 to save one core). SI already does an excellent job of letting users tune their multiprocessing, but for the new users who just say "these guys are the experts I'm not going to change any of their defaults", I wonder if -1 is the best.

[alejoe91]

Thanks Sam, I guess my question is more out of scope for this PR, but I meant more in the sense that some users have had issues with n_jobs=-1 and so I wonder if there's a more sensible default (like n_jobs=-2 to save one core). SI already does an excellent job of letting users tune their multiprocessing, but for the new users who just say "these guys are the experts I'm not going to change any of their defaults", I wonder if -1 is the best.

Good point. Maybe default should be something like 0.9

[h-mayorquin]

@zm711 @alejoe91
Note that this calculates number of logical cpus, though, what you should have is one physical cpu available in cases as this which are mostly -we expect- computational expensive. Usually you have two logical per physical, we should discuss this in a different issue though.

[zm711]

Fair enough. Do we want to just open an overall job_kwargs issue/discussion to brainstorm on this? (the logical processors are an intel thing right (so not amd/Mac), so this might get super complicated, ie I agree a separate thread would be way better). I can open the issue later today!

[h-mayorquin]

Fine by me. Merging!