Testing the `kafka` sink

This document describes how to a setup kafka sink locally using Docker.

Setup

The simplest kafka setup:

git clone https://github.com/confluentinc/kafka-images.git
cd kafka-images/examples/kafka-single-node
docker-compose up -d

For more details see here.

Config

Creates a source that produces events consistently and a kafka sink that consumes them.

[sources.demo_logs]
format = "json"
interval = 1.0
type = "demo_logs"

[sinks.demo_kafka]
type = "kafka"
inputs = [ "demo_logs" ]
bootstrap_servers = "0.0.0.0:9092"
topic = "test"
compression = "gzip"
encoding.codec = "json"
librdkafka_options."message.max.bytes" = "1000000"

Test sink

Open one terminal and run:

path/to/vector/target/debug/vector -v  --config path/to/config_provided_above

Open another terminal and run:

docker exec -it kafka-single-node-kafka-1 /bin/bash
/bin/kafka-console-consumer --topic=test --bootstrap-server=localhost:9092

The above terminal should output events.