Plans for 1.0

[bradjc]

Main changes: focus on individual streams and embrace a distributed structure.

1. Formatters and processors will maintain their distinction: formatters are stateless and can be replicated indiscriminately whereas processors are assumed to be stateful and cannot just be replicated.
2. Formatters will conceptually stay the same. But they will also have to provide what the source of packets is (rabbitmq, some other queue or something) and where that is located. This way some other computer can handle reception and put the packets in a queue and the formatter can then pull from that. The queue could also be in the local server/cloud.
3. The output of formatters will go into two queues: 1) headed to the DB and 2) headed to any processors. Not sure exactly how to implement the second queue. Would love for the queue to maintain packets (at least for a while) in case the processor goes down.
4. Each formatter will have its own queues to take from and put into. This will separate streams.
5. Each profile will have its own archive DB and short-term streaming DB.
6. Processors will operate on the queue from its formatter or another processor.
7. Streamers will still work as DB queries.
8. Processors will push into the same DB queue as the formatter for that profile does.
9. Receivers will change too. Receivers will be responsible for separating streams and putting the data they receive into profile-specific queues.
10. Receivers will note the destination IP address of the incoming packet. There will be generic addresses where the 10 byte (or whatever) id will be used and there will be profile specific addresses that wont have the 10 byte id.
11. Getting streaming data from multiple profiles will probably need its own block.

[bradjc]

1. Allow for a list of multiple packets to be returned from a formatter