Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use multiprocessing.set_start_method('fork') to simplify use of global data #45

Open
cemaccam opened this issue Jan 11, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@cemaccam
Copy link
Collaborator

Currently, the initial state of each new process spawned in the process pool is the initial state of the controller process. This is why "global" variables are passed as arguments to any functions within the process pool, which is inefficient. It was a quirk of developing the software on a Mac.

Using multiprocessing.set_start_method('fork') for the process pool would eliminate the need to pass “global” variables between functions. It results in each spawned process inheriting the current state of the parent process. Because this only works on Unix-related systems and not Windows/Mac OS, the OS should be queried at initialisation to ensure this will work. Data integrity across multiple processes remains a problem, however, so it would be best if only “parameters” were global, and anything changeable were treated as “volatile”. Appropriate semaphores and temporary files may eliminate the latter altogether. Return values can then be restricted to logging, which can be collected by the controller process.

@cemaccam cemaccam added the enhancement New feature or request label Jan 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant