-
Notifications
You must be signed in to change notification settings - Fork 31
Polling system
Li Hao edited this page Nov 27, 2013
·
13 revisions
The polling system is supposed to interact with external resources and fetch the data from outside world. Therefore, the actor model has been utilized in the current implementation. The whole polling system contains one polling supervisor, one polling monitor and several poller. The system is started when the engine starts, and when a new stream is created the associated poller will generated. The poller will periodically fetch the data, store the data into elasticsearch or push it to the pub/sub system.
The system has been implemented using Erlang with gen_server and supervisor frameworks.
- gen_server: gen_server framework could encapsulate some frequently used operations, i.e. asynchronized call and the handling of coming messages. More information about the gen_server could be found here
- supervisor: supervisor framework is used to build a hierarchical process structure called a supervision tree. When one child process is terminated by accident, the supervisor could restart it automatically. On the other hand, supervisor could provide some shorthand ways to overview or operate on the supervision tree. More information could be found here.
The whole polling system contains the following program files:
- polling_system.erl: the controller of the polling system, and provides the main interfaces for the developer to call. This process will be registered as "polling_supervisor".
- polling_monitor.erl: the code implementing supervisor framework, Li Hao fails to integrate supervisor and gen_server together, so he has to split into two code. Not much logic contained, but this code offers a way to overview all pollers. registered as "polling_monitor".
- poller.erl: the actor to interact with the external resources, will be supervised by monitor. Each poller has been implemented as a gen_server.
- parser.erl: contains the functions which are used to parse the coming data.
- poll_help.erl: contains some help functions.
polling_system:start_link().