True thread parallelism with Python 3.12

[pyscripter]

For decades python developers have been trying to work around the infamous Global Interpreter Lock (aka GIL), which has been preventing true multi-threading. With modern CPUs having many CPU cores it is a terrible waste for python to be able to use only one at a time.

Well, the time has come and Python 3.12 brings a breakthrough in this area. Subinterpreters was an obscure and rarely used feature of python, that allowed threads to run in clean slate environment. But they still had to acquire the GIL before they could run code. This is no longer with Python 3.12. Each subinterpreter may have its own GIL, which means they can run in parallel. Truly!

This feature is not available in python code yet, but can be exploited using the C-API. It has also a number of limitations the most important of which are:

• You cannot share python objects between the subinterpreters. Also, there are no standard primitives for synchronization or communication between the interpreters. But this will change in future versions.
• Certain extension modules that have not been updated to use multi-phase inititalization (see PEP 489) may not be available.

But despite these limitations, the performance boost that comes from the exploitation of multiple cores makes this feature very desirable.

P4D has already added support for the new subinterpreters and has integrated that functionality within the TPythonThread. A new thread execution mode emNewInterpreterOwnGIL allows you to run python threads in parallel. There is a new demo (Demo 36) that shows off this new feature.

Here are the results of this demo on my machine demonstrating the performance boost you expect to get:

Classic Subinterpreter:
prime count 78498
prime count 78498
prime count 78498
prime count 78498
prime count 78498
Elapsed ms: 13695

Subinterpreter with own GIL:
prime count 78498
prime count 78498
prime count 78498
prime count 78498
prime count 78498
Elapsed ms: 3482