Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow constraint of pool resources on multiprocess.pool() execution within parallel AutoAttack #2529

Open
lockwoodar opened this issue Nov 22, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@lockwoodar
Copy link
Collaborator

Is your feature request related to a problem? Please describe.

Problem

python=3.11

When leveraging attacks/evasion/auto_attack.py in parallel, the multiprocess.pool() call is unconstrained and will use os.cpu_count() as a default value (all cpus). This is problematic in scenarios when all resources should not be consumed by some mechanism leveraging AutoAttack.

multiprocess.pool documentation

processes is the number of worker processes to use.
If processes is None then the number returned by os.cpu_count() is used.

Note: The functionality for multprocess.pool is slightly different in python>=3.13. It will leverage os.process_cpu_count() instead, which is an abstraction of threads available.

Evidence
Screenshot_2024-08-20_102803

  • Jupyter notebook leveraging parallel AutoAttack
  • All CPUs are maxed by this process until completion

Describe the solution you'd like

Ask

Allow users to optionally specify the number of processes that parallel AutoAttack will use within its pool.

Possible Solution:

  1. Expose new param for parallel_pool_size
  2. Init pool size as 0/Unmodified: parallel_pool_size: int = 0,
  3. If pool size was unmodified, capture max available - 1 (to prevent 100% resource consumption)

import multiprocess

147 ++    pool_size = multiprocess.cpu_count() - 1 if multiprocess.cpu_count() > 1 else multiprocess.cpu_count()
148 ++    if self.parallel_pool_size > 0:
149 ++        pool_size = self.parallel_pool_size             
  1. Use pool_size as the multiprocess.pool() constraint

if self.parallel

291  ++    with multiprocess.get_context("spawn").Pool(processes=pool_size) as pool:

Describe alternatives you've considered

This is a feature to allow users the ability to control resource consumption with parallel execution of AutoAttack (with the addition of not consuming all CPUs in all cases if unmodified by a user). The above example could be reworked for cleaner code, but the premise of exposing a new param for users to control pool size would still be necessary for this feature.

Additional context

@beat-buesser My work here is in context with the internal heart-library research I conducted for IBM located in this Internal GitLab thread

@lockwoodar lockwoodar changed the title Allow constraint of pool resources on multiprocess.pool() execution on parallel AutoAttack Allow constraint of pool resources on multiprocess.pool() execution within parallel AutoAttack Nov 22, 2024
@lockwoodar lockwoodar added the enhancement New feature or request label Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant