Scaling the service horizontally

[anuveyatsu]

From @pwalsh

Reading all the above, can you confirm if my understanding is correct:

1. We have multiple data subscription services running (horizontal scalability).
2. These services **pull** data from CKAN (at an interval), and create tasks (data subscription tasks, i.e.: sending emails for changes to activity).
3. Because there were multiple data subscription services running, we had a duplication of tasks (multiple emails sent for same event in CKAN).

At a guess, what is missing is, for each "activity" that data subscriptions creates a task for, it needs to create a unique identifier for that task, and thereby, even if the same activity is submitted multiple times, then, data subscriptions will gracefully handle this and still only manage one task per activity.

Basically, the data subscriptions app needs to ensure it processes any activity **exactly once**.

From @anuveyatsu

yes, that's correct, in other words:

* We have **pull activities** job running at a given interval which pulls activities from CKAN.
* We have **dispatch notifications** job also running at a given interval which sends out emails to subscribers.
* We had multiple data subscription services running **using the same database credentials** which caused duplicate data in the db and therefore duplicate notifications.