Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/wintoncode/winton-kafka-streams
A Python implementation of Apache Kafka Streams
https://github.com/wintoncode/winton-kafka-streams
kafka-streams librdkafka python streaming-api
Last synced: about 2 months ago
JSON representation
A Python implementation of Apache Kafka Streams
- Host: GitHub
- URL: https://github.com/wintoncode/winton-kafka-streams
- Owner: wintoncode
- License: apache-2.0
- Created: 2017-07-12T08:58:48.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2019-01-23T12:34:10.000Z (over 5 years ago)
- Last Synced: 2024-03-25T21:34:25.156Z (3 months ago)
- Topics: kafka-streams, librdkafka, python, streaming-api
- Language: Python
- Size: 272 KB
- Stars: 313
- Watchers: 32
- Forks: 55
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Roadmap: ROADMAP.md
Lists
- awesome-kafka - winton-kafka-streams - A Python implementation of Apache Kafka Streams. (Libraries / Kafka Streams)
- awesome-kafka - Winton Kafka Streams
- my-awesome-stars - wintoncode/winton-kafka-streams - A Python implementation of Apache Kafka Streams (Python)
- awesome-ccamel - wintoncode/winton-kafka-streams - A Python implementation of Apache Kafka Streams (Python)
- awesome-kafka - winton-kafka-streams - A Python implementation of Apache Kafka Streams. (Libraries / Kafka Streams)
README
# Winton Kafka Streams
[](https://travis-ci.org/wintoncode/winton-kafka-streams)
Implementation of [Apache Kafka's Streams API](https://kafka.apache.org/documentation/streams/) in Python.
## What and why?
Apache Kafka is an open-source stream processing platform developed
by the Apache Software Foundation written in Scala and Java. Kafka
has Streams API added for building stream processing applications
using Apache Kafka. Applications built with Kafka's Streams API do not require any
setup beyond the provision of a Kafka cluster.
Winton Kafka Streams is a Python implementation of Apache Kafka's
Streams API. It builds on Confluent's librdkafka (a high
performance C library implementing the Kafka protocol) and the
Confluent Python Kafka library to achieve this.
The power and simplicity of both Python and Kafka's Streams API combined
opens the streaming model to many more people and applications.
## Getting started
### Dependencies
The minimum Python version is currently 3.6 and a working Kafka
cluster (a single replica is sufficient for testing) are required.
You will require [librdkafka](https://github.com/edenhill/librdkafka). On Mac OS, we recommend installing this via HomeBrew and setting `CFLAGS=-I/usr/local/include` and `LDFLAGS=-L/usr/local/lib` is
when installing Confluent Python Kafka (see below).
The librdkafka GitHub page lists packages available for Debian and Ubuntu, as well as RPMS.
For Arch Linux it is available via [AUR](https://aur.archlinux.org/packages/librdkafka-git/).
Confluent Python Kafka is also required and it should be installed
as a dependency by pip.
### Installing
Cloning the Winton Kafka Streams repository from GitHub is
recommended if you want to contribute to the project.
Then use
`pip install --editable [develop]`
to install as an editable workspace with additional dependencies
required for development.
You may need to do this using `sudo` on Linux.
If you want to install the code and get a feel for it as a user then
we recommend using `pip install git+https://github.com/wintoncode/winton-kafka-streams`.
### Running tests
Tests will run when py.test is called in the root of the repository.
### Running examples
To run examples, you must have cloned the code locally from GitHub.
The debug and wordcount examples will run without further additional
requirements.
The Jupyter notebook in the binning example requires some additional
packages. Install these with the command:
pip install [binning_example]
## Contributing
Please see the CONTRIBUTING.md document for more details on getting involved.
## Contact
- GitHub: https://github.com/wintoncode/
- Email: [email protected]
- Twitter: @wintoncapital