I have jobs that can take from seconds to days to finish.. Best practice?

[zxpectre]

I have jobs that can take from seconds to days to finish, but once they start on a process (deployment instance), they have to stick to that process on retries or server reboots, I cannot avoid this limitation as there is IO taking place locally. So I thought 3 ways on solving this

1. Limiting retries to run on same process (I guess more fault-tolerant)

2. Running the job and somehow never unlock it until finish, but If process crashes, a retry will break the main limitation I have as it can run on any other process.

3. create 2 type of queues, global inter-process ones and local ones, and make global queues add local jobs and wait until completition.

I really want to avoid checking if on same process inside job handler because that will be very inneficient and will consume max retries running jobs unnecesarely.

I will try (3), but I don't know how to implement 1 and 2 and if there are better and simpler solutions.
Can anyone help me? I feel kind of stuck :(

[manast]

As I see it the problem is that the requirement is a bit unrealistic, I mean it is impossible to guarantee that a process will run on the same instance for the simple reason that the instance itself could disappear for any external reason. So I would instead try to get rid from that requirement. If by IO you mean storage there are many ways to have storage to be shared by any instance. All the other 3 workarounds would most certainly bring other issues and edge cases that will cause more pain down the road.

[zxpectre]

I'ts not unrealistic, I'ts uncommon, but I'ts ok, I wish I could get rid of it too. If it fails critically on that instance, it fails entirely and maybe should start again from zero on other process/instance. There are no imposible tasks out there, I was asking suggestions about solving it as I'm so new to bull that I don't know best ways to implement it. What I know is that bull is a great solid product and I will not give up so soon :).

I'll be giving this a try. Thanks for the reply.