Docker Logstash Integration

Ship all your logs from Docker (including in container logs) to Logstash via logstash-forwarder (aka lumberjack).

This means:

• any Docker log files
• whatever log files you configure logstash-forwarder to ship within a container (just put a config at /etc/logstash-forwarder.conf, only the files section gets evaluated while network section is globally configured).

Why?

I wasn't too happy with existing possibilities and while I know that the Docker team is working on a solution, this scratches my itch right now.

Also I didn't see an obvious way to extend docker-gen to handle generic in container templates.

Besides that, how much reason do you need to play with Go & Docker? ;-)

How it works:

docker-logstash-forwarder listens to Docker events and continually restarts a logstash-forwarder instance, after refreshing its configuration, every laziness seconds after a new event was received (to avoid unnecessary restarts - configurable via -laziness flag - defaults to 5 seconds).

For every running container the docker log file is added and it is checked if a logstash-forwarder config exists within the container at /etc/logstash-forwarder.conf.

If an in container specific config exists, the path of all files will be expanded to be valid within the logstash-forwarder container before adding them to the global configuration.

This requires the following (in container defaults in brackets):

• read-only access to the directory containing your docker data (/var/lib/docker)
• connection to Docker (unix:///var/run/docker.sock)
• connection to Logstash (logstash:5043)

Read-only access to Docker data:

Mount the directory containing your Docker data into the containers /var/lib/docker - i.e. run the container with -v /var/lib/docker:/var/lib/docker:ro (assuming your Docker files are stored in /var/lib/docker on the host).

Connection with Docker:

For communication with Docker the following endpoints are evaluated:

1. whatever is passed via the -docker command line flag
2. the $DOCKER_HOST environment variable
3. unix:///var/run/docker.sock

It is suggested to use the later - as in run the container with -v /var/run/docker.sock:/var/run/docker.sock

Behind the screens fsouza/go-dockerclient is used for communication with Docker.

Connection with Logstash:

For communication with Logstash the following endpoints are evaluated:

1. whatever is passed via the -logstash command line flag
2. the $LOGSTASH_HOST environment variable
3. logstash:5043

This allows you to docker -link your Logstash instance to the containers logstash host.

Certificate Handling:

logstash-forwarder authentication can be managed in the following ways:

1. specify a custom config pointing to some imported volume containing the required cert & key via the -config flag (only the network section is evaluated)
2. make your keys available bellow /mnt/logstash-forwarder

TL;DR / Quickstart:

If you have my elasticsearch & logstash containers running just do

$ docker pull digitalwonderland/logstash-forwarder
$ docker run -d --name logstash-forwarder -v /var/lib/docker:/var/lib/docker:ro -v /var/run/docker.sock:/var/run/docker.sock --link logstash:logstash --volumes-from logstash digitalwonderland/logstash-forwarder

If you start from scratch / use Vagrant / are on a Mac: just clone this repository and run vagrant up. This gives you a VM based on CoreOS (which is awesome btw) running those 3 containers & Kibana listening to localhost:8888 (Docker listens to localhost:2375).

Known Issues:

1. the container is pretty big (>500MB). This is because everything is build from source since sharing your whole docker directory requires quite some trust. This might change once I found a way to reliable integrate some build service with docker hub so binaries get build on commit and only after that a docker build is triggered which includes the just built binaries.
2. docker-logstash-forwarder must be run as root until Docker provides configurable ownership of shared volumes, because /var/lib/docker is owned by root on the host and mounted read only, so a non root user can not read from it.

Last but not least it probably should be mentioned, that this is the first time I wrote any go code (a few days, after work), so any 'Duh' pointers are greatly appreciated.

Pull Requests welcome :)