-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
b69ff5d
commit fb40903
Showing
1 changed file
with
125 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,125 @@ | ||
# Kafka Toolbox Pod | ||
|
||
This is a Docker image containing usefull tools preconfigured for easy | ||
usage with Kafka. | ||
|
||
## Included Tools | ||
|
||
|
||
Currently following tools are implemented: | ||
|
||
### Kafka CLI tools | ||
|
||
* `kafka-acls` | ||
* `kafka-broker-api-versions` | ||
* `kafka-configs` | ||
* `kafka-console-producer` | ||
* `kafka-console-consumer` | ||
* `kafka-consumer-groups` | ||
* `kafka-consumer-perf-test` | ||
* `kafka-delegation-tokens` | ||
* `kafka-delete-records` | ||
* `kafka-log-dirs` | ||
* `kafka-preferred-replica-election` | ||
* `kafka-producer-perf-test` | ||
* `kafka-reassign-partitions` | ||
* `kafka-streams-application-reset` | ||
* `kafka-verifiable-producer` | ||
* `kafka-verifiable-consumer` | ||
* `kafka-topics` | ||
* `zookeeper-shell` | ||
|
||
If you have ideas for other useful tools please open an issue and | ||
preferably submit a pull request. | ||
|
||
## Features | ||
|
||
* No need to remember if the tool requires `--bootstrap-server`, | ||
`--bootstrap-servers`, `--broker-list`, `--zookeeper`, `-b` | ||
* Configuration via environment variables | ||
* Support for PLAINTEXT and mutual TLS communication with brokers | ||
|
||
## Usage | ||
|
||
Start the docker image with the environment variables you need. In | ||
general all environment variables are optional. But remember to set | ||
them explicitly when you log into the container. | ||
|
||
**Do not forget to first [Configure](#configuration) the environment variables** | ||
|
||
### Run it on Kubernetes | ||
|
||
1. Download your appropriate yaml file and configure as described below. | ||
2. `kubectl apply -f my.yaml` | ||
3. `kubectl run -it kafka-toolbox bash` | ||
4. Do your work using the tools provided | ||
5. Exit the pod | ||
6. Delete the pod: `kubectl delete -f my.yaml` | ||
|
||
### Use the tools | ||
|
||
All tools are in the `PATH` so you can call them by name | ||
(e.g. `kafka-console-consumer` or `kafkacat`) | ||
|
||
See [./bin/test-toolbox.sh](./bin/test-toolbox.sh) for example usages | ||
|
||
#### Kafka CLI Tools | ||
|
||
```sh | ||
kafka-acls --list | ||
kafka-broker-api-versions | ||
kafka-configs --entity-type topics --describe | ||
kafka-topics --create --topic kafka-toolbox-test --partitions 1 --replication-factor 1 | ||
echo -e "foo\nbar" | kafka-console-producer --topic kafka-toolbox-test | ||
kafka-console-consumer --topic kafka-toolbox-test --offset 0 --partition 0 --max-messages 2 | ||
kafka-consumer-groups --list | ||
kafka-consumer-perf-test --topic kafka-toolbox-test --messages 2 | ||
kafka-delegation-tokens | ||
kafka-delete-records | ||
kafka-log-dirs --describe --broker-list 0 | ||
kafka-preferred-replica-election | ||
kafka-producer-perf-test --topic kafka-toolbox-test --num-records 10 --record-size 10 --throughput 100 | ||
kafka-reassign-partitions | ||
kafka-streams-application-reset --application-id foo | ||
kafka-verifiable-producer --topic kafka-toolbox-test --max-messages 10 | ||
kafka-verifiable-consumer --topic kafka-toolbox-test --max-messages 10 --group-id verifiable-consumer | ||
kafka-topics --delete --topic kafka-toolbox-test | ||
zookeeper-shell | ||
``` | ||
|
||
## Configuration | ||
|
||
### Kafka | ||
|
||
Configure the `KAFKA_BOOTSTRAP_SERVERS` environment variable to point | ||
to your bootstrap servers (usually one of the Kafka brokers – if you | ||
are using Strimzi it is probably called `kafka-cluster-bootstrap`). | ||
|
||
#### No authentication, no transport encryption | ||
|
||
No additional configuration is required | ||
|
||
#### Mutual TLS | ||
|
||
You need to provide following environment variables: | ||
|
||
* `KAFKA_USER_KEY_LOCATION` | ||
* `KAFKA_USER_CERT_LOCATION` | ||
* `KAFKA_CA_CERT_LOCATION` | ||
|
||
#### Other Authentication methods | ||
|
||
currently not supported. If you need it, open a ticket or provide a | ||
pull request. This should be quite straight forward. | ||
|
||
### Zookeeper | ||
|
||
### Plaintext Zookeeper traffic | ||
|
||
Just provide the `KAFKA_ZOOKEEPER` environment variable. | ||
|
||
### Encrypted Zookeeper traffic (the Strimzi way) | ||
|
||
You need a TLS proxy sidecar that encrypts the Zookeeper traffic. Set | ||
the `KAFKA_ZOOKEEPER` environment variable to `localhost:2181` (or the | ||
port your sidecar requires) |