-
Notifications
You must be signed in to change notification settings - Fork 410
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(job): add exclusive execution option #2465
base: master
Are you sure you want to change the base?
Changes from 9 commits
6a502c6
947c0e1
4054264
8ef2414
4354e5f
3c657bc
d108b32
aa3084d
857815a
bbd3bba
1eb415a
5e28ed6
92a8b62
5db5242
094ab70
65c908d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
# Preserve Order | ||
|
||
BullMQ supports preserving execution order in jobs. When _preserveOrder_ option is provided as *true*, jobs will be processed in the same order, indendently of retry strategies. If the current job fails and has a retry strategy, queue will be in rate limit state until the delay is accomplish. | ||
|
||
```typescript | ||
const worker = new Worker('queueName', async (job: Job) => { | ||
// do some work | ||
}, { | ||
preserveOrder: true | ||
}); | ||
``` | ||
|
||
{% hint style="warning" %} | ||
This feature is only allowed when using concurrency 1, any greater value will throw an error: | ||
{% endhint %} |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -222,13 +222,18 @@ export class Worker< | |
stalledInterval: 30000, | ||
autorun: true, | ||
runRetryDelay: 15000, | ||
preserveOrder: false, | ||
...this.opts, | ||
}; | ||
|
||
if (this.opts.stalledInterval <= 0) { | ||
throw new Error('stalledInterval must be greater than 0'); | ||
} | ||
|
||
if (this.opts.preserveOrder && this.opts.concurrency > 1) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It would be nice to refactor this check as we are doing exactly the same on "set concurrency". In fact, couldn't we just do There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Another thing, shouldn't we have this check on the python version as well? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I was think on adding it later |
||
throw new Error('concurrency must be 1 when preserveOrder is enabled'); | ||
} | ||
|
||
if (this.opts.drainDelay <= 0) { | ||
throw new Error('drainDelay must be greater than 0'); | ||
} | ||
|
@@ -385,6 +390,11 @@ export class Worker< | |
) { | ||
throw new Error('concurrency must be a finite number greater than 0'); | ||
} | ||
|
||
if (this.opts.preserveOrder && concurrency > 1) { | ||
throw new Error('concurrency must be 1 when preserveOrder is enabled'); | ||
} | ||
|
||
this.opts.concurrency = concurrency; | ||
} | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -21,6 +21,12 @@ export type JobsOptions = BaseJobOptions & { | |
* These fields are the ones stored in Redis with smaller keys for compactness. | ||
*/ | ||
export type RedisJobOptions = BaseJobOptions & { | ||
/** | ||
* If true, it will rate limit the queue when moving this job into delayed. | ||
* Will stop rate limiting the queue until this job is moved to completed or failed. | ||
*/ | ||
ee?: boolean; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. wouldn't this be better as "po" from Preserver Order? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. that option comes from the old version of this pr, I'm going to delete it |
||
|
||
/** | ||
* If true, moves parent to failed. | ||
*/ | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should add another hint, when using retries and backoffs, for instance, a failed job will keep the queue idle during the time the job is being backed off until it is picked again.