Skip to content

Commit

Permalink
Update readme
Browse files Browse the repository at this point in the history
  • Loading branch information
thspinto committed Jun 27, 2018
1 parent 65e3edf commit 159e62f
Show file tree
Hide file tree
Showing 2 changed files with 37 additions and 5 deletions.
40 changes: 36 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,41 @@
[![Build Status](https://travis-ci.org/quintoandar/python-kafka-idempotence-client.svg?branch=master)](https://travis-ci.org/quintoandar/python-kafka-idempotence-client)
[![Build Status](https://travis-ci.org/quintoandar/python-kafka.svg?branch=master)](https://travis-ci.org/quintoandar/python-kafka)

# Idempotence Client for Kafka
# Python Kafka

Checks messages to avoid reprocessing events.
QuintoAndar's kafka-python lib wrapper with additional fuctionalities.

## Usage
## KafkaConsumer

A simple wrapper for kafka-python lib.

### Configuration

| Name | Description |
| ----------------- | -------------------------------------------- |
| group_id | The consumer group id |
| bootstrap_servers | The bootstrap servers |
| topic | The topic to consume from |
| processor | The function that processes the event |
| idempotenceClient | (optional) A client that checks for repeated events. Must implement the same interface as [IdempotenceClient](/clients/idempotence_client.py) |
| deserializer | (optional) The deserializer |

## IdempotenceClient for Kafka

A redis implementation to check messages and avoid reprocessing events.

### Usage

`markConsumedMessage(topic, message)` creates a hash from the message and concatenates it with the topic and the group id. The message is reigstered as consumed with a default ttl of 2 weeks.

`isUnique(message)` verifies if the message has already been processed.

### Configuration

| Name | Description |
| ------------- | ------------------------------- |
| host | The redis host |
| port | The redis port |
| groupId | The consumer group id |
| db | (optional) The redis db option |
| expire | (optional) The redis ttl |
| key_extractor | (optional) A funtion that extracts a key that uniquely identifies the event |
2 changes: 1 addition & 1 deletion clients/kafka_consumer.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
class KafkaConsumerClient:

def __init__(self, group_id, bootstrap_servers, topic,
processor, idempotenceClient=None, deserializer=None):
processor, deserializer=None, idempotenceClient=None):
self.log = logging.getLogger(__name__)
self.topic = topic
self.processor = processor
Expand Down

0 comments on commit 159e62f

Please sign in to comment.