Building a real-time analytics dashboard with Streamlit, Apache Pinot, and Apache Kafka

Clone repository

git clone [email protected]:mneedham/pinot-wiki.git && cd pinot-wiki

Spin up all components

docker-compose up

or on the Mac M1:

docker-compose -f docker-compose-m1.yml up

Setup Python

Ingest Wikipedia events

python -m venv .venv
source venv/bin/activate
pip install -r requirements.txt

Create Kafka topic

docker exec -it kafka-wiki kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --partitions 5 \
  --topic wiki-events \
  --create

Ingest Wikipedia events

python wiki_to_kafka.py

Check Wikipedia events are ingesting

docker exec -it kafka-wiki kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list localhost:9092 \
  --topic wiki-events

kafkacat -C -b localhost:9092 -t wiki-events

Add Pinot Table

docker exec -it pinot-controller-wiki bin/pinot-admin.sh AddTable \
  -tableConfigFile /config/table.json \
  -schemaFile /config/schema.json \
  -exec

Open the Pinot UI http://localhost:9000/

Run Streamlit app

streamlit run streamlit/app.py