How do scheduled tasks work under the covers with Bull?

[amehta-plume]

Hello,

I'm fairly new to redis and Bull. As per the documentation https://github.com/OptimalBits/bull/blob/develop/README.md#bull-features bull has " Minimal CPU usage due to a polling-free design.".

I'm curious, for a use case like scheduled tasks/cron triggers using bull, how does this actually work under the covers? Is there some documentation/code that indicates how that works? If there's no/minimal polling from bull to redis, how does it trigger scheduled tasks accurately?

For example, when I add a repeatable/cron task to the queue,

<host>:6379> KEYS *
1) "bull:status checking:failed"
2) "bull:status checking:repeat"
3) "bull:status checking:repeat:4b4a412ce56c960a0a2c0b742ac6f556:1614337680000"
4) "bull:status checking:stalled-check"
5) "bull:status checking:delayed"
6) "bull:status checking:id"
<host>:6379> type "bull:status checking:repeat:4b4a412ce56c960a0a2c0b742ac6f556:1614337680000"
hash
<host>:6379> HGETALL "bull:status checking:repeat:4b4a412ce56c960a0a2c0b742ac6f556:1614337680000"
 1) "name"
 2) "statusChecking"
 3) "timestamp"
 4) "1614337636839"
 5) "data"
 6) "{\"data\":\"35\"}"
 7) "opts"
 8) "{\"repeat\":{\"count\":1,\"cron\":\"* * * * *\",\"limit\":2,\"jobId\":\"35\"},\"jobId\":\"repeat:4b4a412ce56c960a0a2c0b742ac6f556:1614337680000\",\"delay\":43161,\"timestamp\":1614337636839,\"prevMillis\":1614337680000,\"removeOnComplete\":true,\"attempts\":1}"
 9) "priority"
10) "0"
11) "delay"
12) "43161"

I notice that its using a hash to store the information about the repeatable task. I'm wondering how that makes things work though.

Thank you.

[manast]

Because repeatable jobs are just delayed jobs, when you add a delay job the workers will get notified on when the delayed job should run, so they wait until then (unless they are already waiting for a delayed job that should run earlier). However there are some watchdogs so it is not completely poll free but almost.

[amehta-plume]

@manast thanks for your response.

Just to clarify, when you say delayed jobs, are you referring to redis delayed tasks?

Likewise, the workers which you are referring to - are these workers in redis? And the watchdogs you mentioned, these are nodejs processes that bull has running internally?

My use case is that I will be having a cloud hosted redis service which I will be using to back Bull, and a different API entrypoint service to handle queueing repeatable tasks etc via Bull. If I have say, 5 scheduled jobs that are added to a queue, to be triggered every hour, is there a good way to estimate how much CPU usage that would result in the background to get things to run? Additionally, how much traffic that would result in between the API service and the external redis service - aside from the initial add calls to enqueue tasks?

[manast]

Just to clarify, when you say delayed jobs, are you referring to redis delayed tasks?

No, that has nothing to do with it. Bull is a queue system implemented on top of redis using lua scripts. Workers are instances that run nodejs functions that executes when a give job needs to be processed.

The CPU usage is dependent on the complexity of your process functions... 5 jobs per hour though is very little considering you can process thousands of jobs per second with simple queues.