Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shared objects on multithreading #77

Open
tgrandje opened this issue Dec 14, 2022 · 0 comments
Open

Shared objects on multithreading #77

tgrandje opened this issue Dec 14, 2022 · 0 comments

Comments

@tgrandje
Copy link

I was a bit mistified to see that timeout-decorator doesn't work well with multithreading. I know this will seem quite obvious (now that I have considered what happened under the hood).

A simple example to detail my case, where you use a shared list of arguments between threads and "pop" from it :

import pebble
from timeout_decorator import timeout
from multiprocess import Manager

class Object():

    THREADS = 4
    def _thread_func(self):
        self.L = Manager().list(range(10))
        with pebble.ThreadPool(self.THREADS) as pool:
            future = pool.map(self.func_to_thread, self.L)
            pool.close()
            for k in future.result():
                print(k)

    @timeout(seconds=1, use_signals=False)
    def func_to_thread(self, _dummy):
        # _dummy is only there to force multithreading
        return self.L.pop()


my_obj = Object()
my_obj._thread_func()
print(my_obj.L)

If you execute that code and (un)comment the @timeout(...) line, you will see that the behaviour is changed a lot (I suspect one or more deep copies of self.L are created by timeout). Note that it keeps happening with a manager instead of a list (which I used at first).

I'd say two elements need to be taken into consideration here:

  • first, the warning on the module's README doesn't seem sufficient. It clearly impacts multithreading whatever your functions return; I'm suspecting that the module should be flagged unsafe for multithreading.
  • secondly: I don't understand why this is still hapenning with a manager. Is this a bug and can/should it be resolved ? For instance, if I keep the manager and move from multithreading to multiprocessing (replacing pebble.ThreadPool by pebble.ProcessPool), everything is now fine.
@tgrandje tgrandje changed the title Shared ressources on multithreading Shared objects on multithreading Dec 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant