https://github.com/quintoandar/kafka-python
QuintoAndar's kafka-python lib wrapper with additional fuctionalities.
https://github.com/quintoandar/kafka-python
idempotency kafka python
Last synced: 6 months ago
JSON representation
QuintoAndar's kafka-python lib wrapper with additional fuctionalities.
- Host: GitHub
- URL: https://github.com/quintoandar/kafka-python
- Owner: quintoandar
- License: mit
- Created: 2018-06-22T22:49:58.000Z (about 8 years ago)
- Default Branch: master
- Last Pushed: 2024-03-20T15:45:48.000Z (over 2 years ago)
- Last Synced: 2025-09-22T01:46:55.516Z (9 months ago)
- Topics: idempotency, kafka, python
- Language: Python
- Homepage:
- Size: 49.8 KB
- Stars: 1
- Watchers: 245
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://travis-ci.org/github/quintoandar/kafka-python)
# QuintoAndar Kafka Python Library
QuintoAndar's kafka-python lib wrapper with additional functionalities.
## KafkaIdempotentConsumer
A simple wrapper for kafka-python lib that uses redis to check duplicate events.
### Configuration
| Name | Description |
| ----------------- | -------------------------------------------- |
| group_id | The consumer group id |
| bootstrap_servers | The bootstrap servers |
| redis_host | The redis host |
| redis_port | The function that processes the event |
| idempotent_key | Function which extract an unique identifier from the event |
See [examples](/examples)
## Development
### Environment
At the bare minimum you'll need the following for your development
environment:
1. [Python 3.6.8](http://www.python.org/)
It is strongly recommended to also install and use [pyenv](https://github.com/pyenv/pyenv):
- [pyenv-installer](https://github.com/pyenv/pyenv-installer)
This tool eases the burden of dealing with virtualenvs and having to activate and
deactivate'em by hand. Once you run `pyenv local my-project-venv` the directory you're
in will be bound to the `my-project-venv` virtual environment and then you will have
never to bother again activating the correct venv.
### Getting Started
#### 1. Clone the project:
```bash
git clone git@github.com:quintoandar/kafka-python.git
cd kafka-python
```
#### 2. Setup the python environment for the project:
```bash
make environment
```
#### 3. Install dependencies
You can just the make recipe:
```bash
make install
```