Question: How to setup Bull for a scalable environment

[selected-pixel-jameson]

Hello,

I'm wondering how we can setup the Bull Job server to work in a scalable environment?

We have a project that may at times require 10's of millions of jobs to be queued up at any given time and ideally we would like the job server be able to scale horizontally at this point.

Would it be plausible to just spin up another instance running Bull and have it connect to the same Redis cluster? Would the jobs be managed correctly across multiple instances?

Thank you!

[manast]

Bull is designed for solving this exact problem. You can run as many workers in parallel as you want. If you are new to Bull I recommend you to use BullMQ: https://bullmq.io/
If you have tens of millions of jobs waiting in the queues you will need to have a Redis instance with a large memory, please make the proper calculations so that you have it dimensioned so that it works for your load.

[derN3rd]

EDIT:
Moved my comment into a separate discussion: #2700