Replies: 1 comment 2 replies
-
|
Not sure what problem you are facing but in Bull when using sandbox processors child processes are reused for every new job... |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
Right, but I assume each job spawns a new child process and kills it after the job is completed? |
Beta Was this translation helpful? Give feedback.
-
|
no, why do you assume that? |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I am using a bull queue to process jobs in concurrent mode, so that the jobs can be handled by child processes across multiple CPUs. As a result, each job spawns a child process (I believe) that re imports all dependencies and must re-initialize all required state, which is quite expensive, before actually processing the job.
Is it possible to have some sort of long-lived processes that can import dependencies and initialize state only once during startup, instead of on each incoming job? I cannot pass the dependencies as job data as they are not json-serializable.
This performance overhead is too significant to ignore. Would love any input or ideas!
I had considered something like spawning child processes manually at startup, and having each incoming job cycle through the child processes to pass commands to them, but this is very manual and I'm sure there are better solutions.
Beta Was this translation helpful? Give feedback.
All reactions