Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High memory usage on large tasks #1

Open
trifle opened this issue Jul 25, 2016 · 1 comment
Open

High memory usage on large tasks #1

trifle opened this issue Jul 25, 2016 · 1 comment
Assignees
Milestone

Comments

@trifle
Copy link

trifle commented Jul 25, 2016

Hi,
thanks for making tourbillon-celery!

I have an (admittedly rare) use case where some of my tasks are in the range of 2-10 MB large.
This means that tourbillon will consume a large amount of RAM (I suspect up to 100GB, ouch!).

A brief investigation leads me to suspect that the reason for this is a LRU cache in celery's events.State class. State has a tasks attribute that is defined as:

https://github.com/celery/celery/blob/82e478e07cb9beee490369b71b7008fcf0f08f89/celery/events/state.py#L377

    self.tasks = (LRUCache(max_tasks_in_memory)
                  if tasks is None else tasks)

with the default parameter in its __init__ method set to max_tasks_in_memory=10000.

So - it would be very nice if one could tune this parameter.

@trifle
Copy link
Author

trifle commented Jul 25, 2016

By the way, this is exactly what flower (celery monitoring web app) does:

https://github.com/mher/flower/blob/a86bf0478f23f5834b72c256ee0ce0afc4199fed/flower/app.py#L42

@ffaraone ffaraone self-assigned this Jul 25, 2016
@ffaraone ffaraone added this to the 0.5 milestone Jul 25, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants