Statsd-aggregator does local aggregation of statsd metrics, lowering the network traffic and amount of data sent to statsd clusters or instances.
Statsd aggregator accepts udp traffic in statsd format. Counters are aggregated by summing values, all other metrics are aggregated by sending all values with same name prefix. Aggregated data is being fit into packets not exceeding MTU and is flushed to the downstream according to the flush interval. E.g. if statsd aggregator would get following data:
test.counter:1|c
test.timer:23|ms
test.counter:10|c
test.timer:51|ms
it would aggregate it into
test.counter:11|c
test.timer:23|ms:51|ms
Please ensure you have a development version of libev installed
- redhat/centos:
yum install libev-devel - debian/ubuntu:
apt-get install libev-dev
$ make install
You also can create a deb package (fpm is required):
$ make pkg
Sample configuration file can be found in /usr/share/statsd-aggregator/statsd-aggregator.conf.sample
Working configuration file location is /etc/statsd-aggregator.conf
- data_port - statsd-aggregator would listen on this port (e.g. data_port=8125)
- downstream_flush_interval - How often we flush data to the downstream (float value in seconds e.g. downstream_flush_interval=1.0)
- downstream - Downstream statsd address:data_port:health_port (e.g. downstream=127.0.0.1:8126:8126).
- log_level - How noisy are our logs (4 - error, 3 - warn, 2 - info, 1 - debug, 0 - trace, e.g. log_level=4)
- dns_refresh_interval - how often we check for dns updates (e.g. dns_refresh_interval=60)
- downstream_health_check_interval - how often we check downstream health (e.g. downstream_health_check_interval=1.0)
Downstream host name can have multiple A records. In this case Statsd-aggregator will send data in the round robin fashion to all healthy downstream hosts.
Statsd-aggregator can be controlled via /etc/init.d/statsd-aggregator
Testing framework is written in ruby and requires evenmachine gem, please install it using following command:
$ sudo gem install eventmachine
To run tests use:
$ make test
Tests are simulating metrics source and check that either correct data is
being send to the statsd downstream or correct message is being logged.
For more details please see test/statsd-aggregator-test-lib.rb