Is it possible to *not* release the GIL?

[paulmueller]

I am using mahotas in a pipeline that needs to be single-threaded, because

• there are other computations taking place that rely on e.g. openCV or numpy and that are by design single-threaded
• I want to limit the number of CPUs for my pipeline using multiprocessing.Pool

The problem that I am having is that, e.g. when computing the Haralick texture features, each process created via multiprocessing now creates cpu_count() threads and I end up with 100% CPU on each core. I also think this is not very efficient.

Is it possible to disable the release of the GIL in mahotas? It seems like I could do this by removing all gil_release statements and compiling mahotas myself.

Is anyone else interested in this? Would it make sense to make available a mahotas-gil package on PyPI that does not release the GIL?

Thanks for your thoughts!

[luispedro]

From your description, it should not matter whether mahotas releases the GIL or not. As long as the processes created by multiprocessing are not multithreaded, the GIL should not matter.

The easiest way to test would be to simply deactivate the gil_release object in mahotas/utils.hpp instead of replacing every single instance.

[paulmueller]

Thanks for your reply.

AFAIK it is not possible to confine a multiprocessing.Process to a specific number of CPUs. Mahotas will always use all of them. There is not much information on the topic in the mahotas docs, so I assume that it is not possible to tune the number CPUs mahotas will use?

To illustrate my point, take a look at the linux command line tool taskset, which can confine an entire program (no matter how many processes are spawned) to a fixed number of CPUs. I think something like this is not possible with multiprocessing.

Thanks for the hint about mahotas/utils.hpp.

[paulmueller]

I did some more research and I found that the major players when it comes to parallelization (e.g. during compile-time in numpy), all have a switch to control the number of threads, e.g. via environment variables:

export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
export NUMEXPR_NUM_THREADS=1

There is also a python package that interfaces with those libraries and can change the behavior during runtime: https://github.com/joblib/threadpoolctl

Is it possible to introduce a MAHOTAS_NUM_THREADS environment variable?

[paulmueller]

I just realized that mahotas apparently honors the OMP_NUM_THREADS environment variable.
When setting OMP_NUM_THREADS to a fixed number, mahotas will only spawn so many processes. This is exactly what I needed.

I assume this means that mahotas uses OpenMP (at least for me on Linux)?

I believe this information should somehow be part of the documentation. I don't know how OpenMP plays into this during compile time of mahotas. I don't know whether Intels MKL could also affect the performance of mahotas. And since I don't understand what is going on at all, it would be helpful to explain this a little in the docs.

Thanks for your time anyway. I will leave it up to you to close this issue. 👍