feat: refactor core/thread logic for mpibackend

[asoplata]

This takes George's old GUI-specific _available_cores() method, moves it, and greatly expands it to include updates to the logic about cores and hardware-threading which was previously inside MPIBackend.__init__(). This was necessary due to the number of common but different outcomes based on platform, architecture, hardware-threading support, and user choice. These changes do not involve very many lines of code, but a good amount of thought and testing has gone into them. Importantly, these MPIBackend API changes are backwards-compatible, and no changes to current usage code are needed. I suggest you read the long comments in
parallel_backends.py::_determine_cores_hwthreading() outlining how each variation is handled.

Previously, if the user did not provide the number of MPI Processes they wanted to use, MPIBackend assumed that the number of detected "logical" cores would suffice. As George previously showed, this does not work for HPC environments like on OSCAR, where the only true number of cores that we are allowed to use is found by
psutil.Process().cpu_affinity(), the "affinity" core number. There is a third type of number of cores besides "logical" and "affinity" which is important: "physical". However, there was an additional problem here that was still unaddressed: hardware-threading. Different platforms and situations report different numbers of logical, affinity, and physical CPU cores. One of the factors that affects this is if there is hardware-threading present on the machine, such as Intel Hyperthreading. In the case of an example Linux laptop having an Intel chip with Hyperthreading, the logical and physical core numbers will report different values with respect to each other: logical includes Hyperthreads
(e.g. psutil.cpu_count(logical=True) reports 8 cores), but physical does not
(e.g. psutil.cpu_count(logical=False) reports 4 cores). If we tell MPI to use 8 cores ("logical"), then we ALSO need to tell it to also enable the hardware-threading option. However, if the user does not want to enable hardware-threading, then we need to make this an option, tell MPI to use 4 cores
("physical"), and tell MPI to not use the hardware-threading option. The "affinity" core number makes things even more complicated, since in the Linux laptop example, it is equal to the logical core number. However, on OSCAR, it is very different than the logical core number, and on Macos, it is not present at all.

In _determine_cores_hwthreading(), if you read the lengthy comments, I have thought through each common scenario, and I believe resolved what to do for each, with respect to the number of cores to use and whether or not to use hardware-threading. These scenarios include: the user choosing to use hardware-threading (default) or not, across Macos variations with and without hardware-threading, Linux local computer variations with and without hardware-threading, and Linux HPC (e.g. OSCAR) variations which appear to never support hardware-threading. In the Windows case, due to both jonescompneurolab#589 and the currently-untested MPI integration on Windows, I always report the machine as not having hardware-threading.

Additionally, previously, if the user did provide a number for MPI Processes, MPIBackend used some "heuristics" to decide whether to use MPI oversubscription and/or hardware-threading, but the user could not override these heuristics. Now, when a user instantiates an MPIBackend with __init__() and uses the defaults, hardware-threading is detected more robustly and enabled by default, and oversubscription is enabled based on its own heuristics; this is the case when the new arguments hwthreading and oversubscribe are set to their default value of None. However, if the user knows what they're doing, they can also pass either True or False to either of these options to force them on or off. Furthermore, in the case of hwthreading, if the user indicates they do not want to use it, then
_determine_cores_hwthreading() correctly returns the number of NON-hardware-threaded cores for MPI's use, instead of the core number including hardware-threads.

I have also modified and expanded the appropriate testing to compensate for these changes.

Note that this does NOT change the default number of jobs to use for the GUI if MPI is detected. Such a change breaks the current test_gui.py testing: see jonescompneurolab#960
jonescompneurolab#960

[ntolley]

Just curious, what are the pros/cons of having this false by default?

[ntolley]

I'm guessing you're moving towards typing throughout the code

Are you thinking of doing it through small changes like this and just progressively update the code base?

[ntolley]

It honestly seems like the ability to reliably detect the number of cores could be a standalone python package