minion use case for a scheduled cron job?

[fluca1978]

I'm not sure if minions can help me here, but I have an application that harvest data from different sources and requires a long time to complete (25+ hours). In order to make sure the application completes, I've created a batch operational mode and I scheduled via cron the repeated execution, so that the application can check at which point it stopped last time and restart from there.

I'm thinking to convert this to a minion based approach, but I'm not sure if this is the right scenario. My idea is to:

• create a worker
• enqueue a batch processing
• the enqueued process will re-enqueue itself once it completes, so to ensure there will be a next run.

Is it worth? Any idea about how to use minions to run scheduled repeated tasks?

[Grinnz]

Generally no need to reinvent cron; a mojolicious command will have access to the application and minion and a cron task can easily run that. https://docs.mojolicious.org/Mojolicious/Guides/Cookbook#Adding-commands-to-Mojolicious

[kraih]

I usually add small Mojolicious commands that enqueue a Minion task for me (like this one). And then schedule the command to be executed with systemd timers.

[christopherraa]

Same approach here as @kraih really. Have commands that enqueue and systemd timers that trigger those. Super simple and effective. I also take care to split any job I can into smaller / multiple tasks with dependencies between them. So instead of one monster-job it would look like this:

                   |--> [ subtask 1 ] ---|
[ initial task ] --+--> [ subtask 2 ] ---+---> [ "collector" task ]
                   |         ...         | 
                   |--> [ subtask n ] ---|

This makes it a great deal easier to handle retries of failed actions and basically limit the effects of partial failures during batch procesing.

[fluca1978]

Thanks all, it becomes to appear clearer now.
Just one design suggestion: the minion database handling enqueued tasks should be within the application database or is it better/advisable to keep it separated (since it does not store any data related to the application)?

[christopherraa]

I guess the only response I can give is "it depends", but at least for us here the decision has been to keep minon-data together with application data. This is because for us the minion-data is specific to each application so it makes sense for the data to live together. It also makes manual lookups easier. An example of this is pulling some id's from minion_jobs.args and joining with og retrieving data from other relations based on that. There is also an argument to be made for reducing the number of active database connections (sockets on the client). You double socket usage if you need to maintain connections to two different databases VS a shared one for the application.