Skip to content

Removed Parallel Experiment Execution via Fork and Using pycommons 0.8.45

Compare
Choose a tag to compare
@thomasWeise thomasWeise released this 21 Sep 06:33
· 32 commits to main since this release

Python 3.12 uses multi-threading.
The parallel experiment execution via forking cannot be made to work properly, as it may lead to deadlocks and other issues in Python 3.12 due to the multi-threading.
We can also not use the spawn or forkserver contexts to create new processes, because they require that every shared data must be pickled.
Unfortunately, lambdas, which we basically use everywhere to create instance and algorithm objects in the experiment execution, cannot be pickled.
There is just no way to make this work.
Thus, I removed it entirely.
If we want parallel experiments, we just have to execute the experiment program multiple times in parallel.
This seems to be the safest method.

We also stepped to pycommons 0.8.45, which offers improved static testing and also uses the newest pylint version.
By applying this also to the tests folder, lots of additional issues were found and removed.