Complex case of running subprocesses

[tarkhil]

Hello

I have a task to process some 400 documents. Extract data, put into template, store result.

Template is processed by LibreOffice (I need precise control of layout and could not achieve it with HTML to PDF converters), running it takes about 20-30 seconds per document.

I want to run several instances in parallel, but no more than 4 maybe 6.

Please suggest me a code that

• put a request into a queue
• run it in subprocess when slot is available
• return result to calling code
• wait until a queue is empty

[rabbiveesh]

have you looked into Minion? you can do all of this and more:

• put a request into a queue - $minion->enqueue('YourTaskHere')
• run in a subprocess - ./your_app minion worker -j $amt_of_workers
• return result to calling code - set up a task which depends on all the workers and then collects all the results and then my $result; $minion->result_p($id_of_collector)->then(sub ($info) { $result = $info->{result} })->wait
• wait until a queue is empty - using the collector task above should accomplish what you want