feat(job): add exclusive execution option #2465

roggervalf · 2024-03-08T04:12:52Z

No description provided.

manast · 2024-03-10T22:40:09Z

Ok, interesting use of rate-limiter for achieving this effect. A couple of things:

I am not sure the name of the option conveys the actual function, for me it is more like "sequential execution", although the queue already executes sequentially, except when we are doing retries. I think we need to think more about the name here.
This option only makes sense with concurrency is 1, as otherwise we have no guarantee that the execution is "sequential", so maybe if concurrency > 1 and this option is chosen we should rise an error.

manast · 2024-03-10T22:46:57Z

Another thing to think about, what happens if there are delayed jobs already in the queue, should we care about them?
Maybe not actually... as they are delayed it means they should be executed as soon as possible after they expire. Also if the user cares about order, then they probably will not have delayed jobs running side by side regular jobs...

roggervalf · 2024-03-18T16:40:57Z

src/classes/job.ts

@@ -38,6 +38,7 @@ import type { QueueEvents } from './queue-events';
 const logger = debuglog('bull');

 const optsDecodeMap = {
+  ee: 'exclusiveExecution',


preserveOrder must be used

manast · 2024-03-27T10:10:10Z

docs/gitbook/guide/workers/preserve-order.md

+const worker = new Worker('queueName', async (job: Job) => {
+    // do some work
+}, {
+    concurrency: 1


As concurrency is by default 1, I assume it is not required to specify any when using "preserveOrder".

yes, no necessary, I added it just to be clear, I can add a comment about the default value

manast · 2024-03-27T10:23:38Z

I think we need more tests for cover the scenario when you have several workers running in parallel. For instance, let's say you have 10 workers (assume all have concurrency 1 AND preserver order flag to true, if any has another set of options then it will not work as expected, we should also write this in the documentation). Now, I assume that as soon one job is fetched, rate limit kicks in and the other workers will just be idling. We need tests to verify that this is the case, also, what happens if the job stalls, it may go back to wait but we need to make sure that no other job waiting is taken by any of the other workers that are idling when this happens, so order is preserved. We also need to document that if you add delayed jobs, then the order cannot be guaranteed either...

manast · 2024-03-27T10:24:07Z

It is a lot, but if we want to guarantee order there are several edge cases to consider, probably others we have not thought about yet.

manast · 2024-03-27T10:15:19Z

src/classes/worker.ts

      ...this.opts,
    };

    if (this.opts.stalledInterval <= 0) {
      throw new Error('stalledInterval must be greater than 0');
    }

+    if (this.opts.preserveOrder && this.opts.concurrency > 1) {


It would be nice to refactor this check as we are doing exactly the same on "set concurrency". In fact, couldn't we just do this.concurrency = ... here and let the checks from set concurrency take care of it?

Another thing, shouldn't we have this check on the python version as well?

I was think on adding it later

manast · 2024-03-27T10:16:29Z

src/types/job-options.ts

+   * If true, it will rate limit the queue when moving this job into delayed.
+   * Will stop rate limiting the queue until this job is moved to completed or failed.
+   */
+  ee?: boolean;


wouldn't this be better as "po" from Preserver Order?

that option comes from the old version of this pr, I'm going to delete it

roggervalf · 2024-03-28T05:51:38Z

I think we need more tests for cover the scenario when you have several workers running in parallel. For instance, let's say you have 10 workers (assume all have concurrency 1 AND preserver order flag to true, if any has another set of options then it will not work as expected, we should also write this in the documentation). Now, I assume that as soon one job is fetched, rate limit kicks in and the other workers will just be idling. We need tests to verify that this is the case, also, what happens if the job stalls, it may go back to wait but we need to make sure that no other job waiting is taken by any of the other workers that are idling when this happens, so order is preserved. We also need to document that if you add delayed jobs, then the order cannot be guaranteed either...

there is no global concurrency, so having more than 1 worker will break preserve order behavior, also need to document it.

# [5.9.0](v5.8.7...v5.9.0) (2024-07-15) ### Features * **queue:** support global concurrency ([#2496](#2496)) ref [#2465](#2465) ([47ba055](47ba055))

manast · 2024-07-16T20:20:08Z

docs/gitbook/guide/workers/preserve-order.md

+```
+
+{% hint style="warning" %}
+This feature is only allowed when using concurrency 1, any greater value will throw an error:


We should add another hint, when using retries and backoffs, for instance, a failed job will keep the queue idle during the time the job is being backed off until it is picked again.

manast

I just had a couple of comments on the documentation.

manast · 2024-08-02T21:52:00Z

docs/gitbook/guide/workers/preserve-order.md

@@ -0,0 +1,19 @@
+# Preserve Order
+
+BullMQ supports preserving execution order in jobs. When _preserveOrder_ option is provided as *true*, jobs will be processed in the same order, independently of retry strategies. If the current job fails and has a retry strategy, queue will be in rate limit state until the delay is accomplish.


-> BullMQ supports preserving execution order in jobs. When the preserveOrder option is provided as true, jobs will be processed in the same order, independently of retry strategies. If the current job fails and has a retry strategy, the queue will be in rate limit state until the retry delay has expired.

"queue will be in..." -> "the queue will be in..."

manast · 2024-08-02T21:52:52Z

docs/gitbook/guide/workers/preserve-order.md

+{% endhint %}
+
+{% hint style="warning" %}
+This feature is only allowed when using concurrency 1, any greater value will throw an error. Make sure to also set a [global concurrency](https://docs.bullmq.io/guide/queues/global-concurrency)


-> This feature is only allowed when using concurrency 1, any greater value will throw an error. Make sure to also set a global concurrency if you want to preserve order having multiple running workers.

"until it is picked again." -> "until it is picked up again."

TomKaltz · 2024-12-02T03:01:36Z

Any reason this can't be merged?

manast

I think we also need tests to cover the case where we have multiple workers running in parallel.

manast · 2024-12-03T09:10:09Z

docs/gitbook/guide/workers/preserve-order.md

@@ -0,0 +1,19 @@
+# Preserve Order
+
+BullMQ supports preserving execution order in jobs. When _preserveOrder_ option is provided as *true*, jobs will be processed in the same order, independently of retry strategies. If the current job fails and has a retry strategy, queue will be in rate limit state until the delay is accomplish.


"queue will be in..." -> "the queue will be in..."

manast · 2024-12-03T09:10:38Z

docs/gitbook/guide/workers/preserve-order.md

+{% endhint %}
+
+{% hint style="warning" %}
+This feature is only allowed when using concurrency 1, any greater value will throw an error. Make sure to also set a [global concurrency](https://docs.bullmq.io/guide/queues/global-concurrency)


"until it is picked again." -> "until it is picked up again."

manast · 2024-12-03T09:14:05Z

@TomKaltz mostly because it is a difficult feature to get right, so it gets often sidelined in favor of other new features and fixes.

roggervalf added 2 commits March 7, 2024 23:11

feat(job): add exclusive execution option

6a502c6

refactor: move exclusive execution option into retry script

947c0e1

This was referenced Mar 11, 2024

How to ensure ordering of jobs in case of delayed retries. taskforcesh/bullmq-pro-support#68

Open

Global Concurrency Setting for Bull-MQ Workers #2461

Closed

roggervalf commented Mar 18, 2024

View reviewed changes

roggervalf added 4 commits March 25, 2024 23:20

refactor: use preserveOrder option

4054264

Merge branch 'master' into exclusive-execution

8ef2414

test(worker): add test case

4354e5f

chore: change command name

3c657bc

roggervalf marked this pull request as ready for review March 26, 2024 04:35

roggervalf added 2 commits March 25, 2024 23:46

chore(job): check preserveOrder

d108b32

docs(worker): add preserveOrder section

aa3084d

manast reviewed Mar 27, 2024

View reviewed changes

docs: :add concurrency warning

857815a

manast reviewed Mar 27, 2024

View reviewed changes

roggervalf added 2 commits March 28, 2024 00:57

chore: remove duplicated concurrency check

bbd3bba

chore: ignore fifo if preserveOrder is provided

1eb415a

roggervalf mentioned this pull request Mar 30, 2024

feat(queue): add global concurrency #2496

Merged

Merge branch 'master' into exclusive-execution

5e28ed6

roggervalf added a commit that referenced this pull request Jul 15, 2024

feat(queue): support global concurrency (#2496) ref #2465

47ba055

github-actions bot pushed a commit that referenced this pull request Jul 15, 2024

chore(release): 5.9.0 [skip ci]

8025f99

# [5.9.0](v5.8.7...v5.9.0) (2024-07-15) ### Features * **queue:** support global concurrency ([#2496](#2496)) ref [#2465](#2465) ([47ba055](47ba055))

manast reviewed Jul 16, 2024

View reviewed changes

roggervalf mentioned this pull request Jul 19, 2024

Sequentially queue processing OptimalBits/bull#1446

Closed

roggervalf added 2 commits July 28, 2024 18:32

chore: fix merge conflicts

92a8b62

docs: add hint

5db5242

roggervalf added 2 commits July 30, 2024 22:08

Merge branch 'master' into exclusive-execution

094ab70

Merge branch 'master' into exclusive-execution

65c908d

manast approved these changes Aug 2, 2024

View reviewed changes

roggervalf mentioned this pull request Aug 3, 2024

feat: support pending state #2687

Open

roggervalf mentioned this pull request Sep 26, 2024

Keep order of jobs in group upon retry #2774

Open

manast reviewed Dec 3, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(job): add exclusive execution option #2465

feat(job): add exclusive execution option #2465

roggervalf commented Mar 8, 2024

manast commented Mar 10, 2024

manast commented Mar 10, 2024

roggervalf Mar 18, 2024

manast Mar 27, 2024

roggervalf Mar 27, 2024

manast commented Mar 27, 2024 •

edited

Loading

manast commented Mar 27, 2024

manast Mar 27, 2024

manast Mar 27, 2024

roggervalf Mar 28, 2024

manast Mar 27, 2024

roggervalf Mar 28, 2024

roggervalf commented Mar 28, 2024

manast Jul 16, 2024

manast left a comment

manast Aug 2, 2024

manast Dec 3, 2024

manast Aug 2, 2024

manast Dec 3, 2024

TomKaltz commented Dec 2, 2024

manast left a comment

manast Dec 3, 2024

manast Dec 3, 2024

manast commented Dec 3, 2024

		@@ -0,0 +1,19 @@
		# Preserve Order

		BullMQ supports preserving execution order in jobs. When _preserveOrder_ option is provided as true, jobs will be processed in the same order, independently of retry strategies. If the current job fails and has a retry strategy, queue will be in rate limit state until the delay is accomplish.

feat(job): add exclusive execution option #2465

Are you sure you want to change the base?

feat(job): add exclusive execution option #2465

Conversation

roggervalf commented Mar 8, 2024

manast commented Mar 10, 2024

manast commented Mar 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

manast commented Mar 27, 2024 • edited Loading

manast commented Mar 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roggervalf commented Mar 28, 2024

Choose a reason for hiding this comment

manast left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomKaltz commented Dec 2, 2024

manast left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

manast commented Dec 3, 2024

manast commented Mar 27, 2024 •

edited

Loading