Parallel Execution + Overlap Support

[checkphi]

... Continuing the issue about parallel execution.

I think the feature set Laravel offers is exactly what this bundle should be aiming for as well.
https://laravel.com/docs/8.x/scheduling
They offer background tasks as well as protection from overlapping tasks.

Task overlapping should be easily achievable by making use of Symfonys nice lock component.

Regarding background tasks I have two (independently usable) ideas:

1. Simply spawn background shell tasks (like using & in the console)
2. Apply a similar idea like https://github.com/corvus-ch/rabbitmq-cli-consumer

Some elaboration on point 2:
The main problem is the lack of multithreading support in PHP. If there was multithread support, we wouldn't have this discussion.
However, Golang has multithreading. So a solution could be to write a small program in Go that (1) calls a scheduler command which outputs all due tasks (commands) and then starts a thread for each. Just a few lines in Go and problem solved. I do understand that it is not a nice solution to build a PHP-only Symfony bundle and then create a Go program to solve a problem, but all that is necessary here is extend the scheduler:consume command to have some machine-readable output (like a comma-separated list or whatever) that would be the interface for the external Go program that doesn't need to be officially supported or mentioned at all.

What do you think?

[Guikingone]

Hi @checkphi 👋🏻

When it comes to Laravel vision of scheduling tasks, here's my thoughts:

• This bundle is inspired by Laravel, by this, I mean that background tasks are already supported for ShellTask (not for others but PR's are welcome 🙂 ).
• Same things applied to "overlapping", by default, every task executed by the worker is locked using Lock component.

When it comes to multi-thread support, I'm aware of the PHP limitations and the fact that this bundle is limited by cascade by PHP ... 🙁

I don't have a "battle-tested" solution but the "Go bridge" could be one, things like RoadRunner: https://github.com/spiral/roadrunner can define a new way to use PHP applications, again, it's not 100% "perfect" but it could be an interesting way to explore, if you have any idea to submit using PR's, feel free to, I'll be more than happy to explore it 🙂

[Guikingone]

Hi @checkphi 👋🏻

I'm currently working on a parallel implementation thanks to spatie/fork, if you want to take a look: #90