Skip to content

Latest commit

 

History

History

wurstmeister

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
Dockerfile
Dockerfile
 
 
README.md
README.md
 
 
docker-compose.yml
docker-compose.yml
 
 
produce_consume.py
produce_consume.py
 
 
requirements.txt
requirements.txt
 
 

README.md

Setting up Wurstmeister Kafka and kafka-python

We are going to use these two docker images:

Check out the docker-compose.yml file:

version: '3.8'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    networks:
      - kafka_network
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    networks:
      - kafka_network
    ports:
      - "9092:9092"
    expose:
      - "9093"
    environment:
      KAFKA_LISTENERS: DOCKER_NET://0.0.0.0:9093,HOST_NET://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: DOCKER_NET://kafka:9093,HOST_NET://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: DOCKER_NET:PLAINTEXT,HOST_NET:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: DOCKER_NET
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      # Create Kafka topics "NAME:PARTITION:REPLICAS,..."
      KAFKA_CREATE_TOPICS: "topic-container:1:1,topic-macos:1:1"
networks:
  kafka_network:
    name: kafka
    driver: bridge

Start Kafka and Zookeeper dockers:

docker-compose up -d

Check if the processes are running:

docker-compose ps

Check Zookeeper logs:

docker-compose logs zookeeper | grep -i binding

Check Kafka broker logs:

docker-compose logs kafka | grep -i started

Kafka Hello world from Python container

Check out the Dockerfile for the Python container:

FROM python:3.7-alpine

WORKDIR /workspace

ADD ./requirements.txt /workspace
ADD ./produce_consume.py /workspace

RUN pip install --no-cache-dir -r /workspace/requirements.txt

Build the Docker image:

docker build --rm --tag kafka-app-py .

Run the produce_consume.py script inside the Docker container:

docker run --rm -it --network kafka \
    kafka-app-py:latest \
    python produce_consume.py

Kafka Hello world from macOS

Check out my tutorial mac-dev-setup to install Python on your MacBook.

Create a Python virtual environment:

mkvirtualenv kafka

Install the requirements:

pip install kafka-python

Run the produce_consume.py script from your macOS:

python produce_consume.py

Clean up everything

Stop Kafka cluster:

docker-compose down --volumes

Remove Docker images:

docker image rmi \
    kafka-app-py:latest \
    python:3.7-alpine \
    wurstmeister/kafka:latest \
    wurstmeister/zookeeper:latest

Remove Python virtual environment:

rmvirtualenv kafka

References