Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel is greedy #592

Open
iguinn opened this issue Nov 1, 2024 · 0 comments
Open

Parallel is greedy #592

iguinn opened this issue Nov 1, 2024 · 0 comments

Comments

@iguinn
Copy link
Collaborator

iguinn commented Nov 1, 2024

The parallelism in math enabled by PYGAMA_PARALLEL=True by default creates as many threads as possible. This has a couple of problems. First, it can slow things down in some situations due to overhead associated with moving things to threads. Second, NERSC doesn't want us doing this on login/jupyter nodes.

For now the solution will be to default it to False. However, in the future, we should add a way to control the number of threads used (perhaps replace PYGAMA_PARALLEL with PYGAMA_NTRHEADS). This can be controlled in numba using numba.set_num_threads() as discussed here: https://numba.readthedocs.io/en/stable/user/threading-layer.html. For default behavior we have a couple of options:

  1. Default to threading off. The disadvantage of this approach is that it means people will be testing their code single-threaded and may miss bugs. For this reason, we would want to make sure tests still ran multi-threaded in some instances
  2. Choose a smaller number of threads as the default.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant