-
Notifications
You must be signed in to change notification settings - Fork 25
Polling system
The polling system is supposed to interact with external resources and fetch the data from outside world. Therefore, the actor model has been utilized in the current implementation. The whole polling system contains one polling supervisor, one polling monitor and several poller. The system is started when the engine starts, and when a new stream is created the associated poller will generated. The poller will periodically fetch the data, store the data into elasticsearch or push it to the pub/sub system.
The system has been implemented using Erlang with gen_server and supervisor frameworks.
- gen_server: gen_server framework could encapsulate some frequently used operations, i.e. asynchronized call and the handling of coming messages. More information about the gen_server could be found here
- supervisor: supervisor framework is used to build a hierarchical process structure called a supervision tree. When one child process is terminated by accident, the supervisor could restart it automatically. On the other hand, supervisor could provide some shorthand ways to overview or operate on the supervision tree. More information could be found here.
The whole polling system contains the following program files:
- polling_system.erl: the controller of the polling system, and provides the main interfaces for the developer to call. This process will be registered as "polling_supervisor".
- polling_monitor.erl: the code implementing supervisor framework, Li Hao fails to integrate supervisor and gen_server together, so he has to split into two code. Not much logic contained, but this code offers a way to overview all pollers. registered as "polling_monitor".
- poller.erl: the actor to interact with the external resources, will be supervised by monitor. Each poller has been implemented as a gen_server.
- parser.erl: contains the functions which are used to parse the coming data.
- poll_help.erl: contains some help functions.
polling_system:start_link().
This command will start the who system, and extract the streams` information from the elasticsearch to generate pollers.
gen_server:cast(polling_supervisor, {create_poller, #pollerInfo{}}).
This command will notify the polling system to create a new poller. The record #pollerInfo is defined in the file poller.hrl. It is a asynchronized call. So when you use the command supervisor:which_children(polling_monitor). to check the children of polling system, you will find a new member!
gen_server:cast(polling_supervisor, {rebuild, Stream_id})
After you change the information of stream, you may need this command to refresh the poller, just provide the id of the stream. Is is asynchronized call. The poller will fetch the data from the elasicsearch and change the state of itself.
gen_server:cast(polling_supervisor, {terminate, Stream_id})
This asynchronized call will terminate the poller with the stream id, if the poller not found, the polling system will return the error message.