DAGs exit immediately when there are no products to process

[randomorder]

Would be nice to have DAGs runs exit immediately after the search / download tasks is there is no products to process.

Currently this does not happen and all tasks are executed anyway. Operators should handle this gracefully as per #243 but I think would be good to have the DAGs terminate their execution early when there is nothing to process.

One way to achieve this would be to branch after the search / download and have a Dummy "finish" task by changing existing DAGs:

One possibility is to branch after the search / download and have a Dummy "finish" task:

From

Into:

Airflow may also provide other mechanisms to do this

@torse What do you think

[torse]

Maybe we could branch the DAG before Download? Otherwise LGTM.

I like the approach to define a Finishing task. It could later on also send notification and/or update the elasticsearch database..

[maabdelghaffar]

@torse @randomorder how about using ShortCircuitOperator to allow the DAG to continue running only if our condition met ?

[randomorder]

I don't have a strong opinion on this either would work for me