Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/gabrielmbmb/kaflow
Kafka streams processing in Python.
https://github.com/gabrielmbmb/kaflow
aiokafka async asyncio avro di json kafka processing protobuf pydantic python-types streams
Last synced: 27 days ago
JSON representation
Kafka streams processing in Python.
- Host: GitHub
- URL: https://github.com/gabrielmbmb/kaflow
- Owner: gabrielmbmb
- License: mit
- Created: 2023-03-18T10:32:44.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2023-07-12T19:27:54.000Z (over 1 year ago)
- Last Synced: 2024-04-23T00:10:25.075Z (9 months ago)
- Topics: aiokafka, async, asyncio, avro, di, json, kafka, processing, protobuf, pydantic, python-types, streams
- Language: Python
- Homepage: https://kaflow.gabrielmb.com
- Size: 154 KB
- Stars: 10
- Watchers: 2
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
kaflow
Kafka streams topic processing in Python.
---
`kaflow` is a simple framework that allows you to build Kafka streams processing aplications in Python with ease.
Some of the features offered by `kaflow`:
- Dependency Injection system inspired by [FastAPI](https://github.com/tiangolo/fastapi) and [xpresso](https://github.com/adriangb/xpresso), and backed by [di](https://github.com/adriangb/di).
- Automatic deserialization of incoming messages and serialization of outgoing messages. Supports popular formats like `JSON`, `Avro` or `Protobuf`.
- Message validation thanks to [pydantic](https://github.com/pydantic/pydantic).
## Requirements
Python 3.8+
## Installation
```shell
pip install kaflow
```
## Example
```python
from kaflow import (
FromHeader,
FromKey,
FromValue,
Json,
Kaflow,
Message,
MessageOffset,
MessagePartition,
MessageTimestamp,
String,
)
from pydantic import BaseModel
class UserClick(BaseModel):
user_id: int
url: str
timestamp: int
class Key(BaseModel):
environment: str
app = Kaflow(name="AwesomeKakfaApp", brokers="localhost:9092")
@app.consume(topic="user_clicks", sink_topics=["user_clicks_json"])
async def consume_user_clicks(
message: FromValue[Json[UserClick]],
key: FromKey[Json[Key]],
x_correlation_id: FromHeader[String[str]],
x_request_id: FromHeader[String[str]],
partition: MessagePartition,
offset: MessageOffset,
timestamp: MessageTimestamp,
) -> Message:
# Do something with the message
...
# Publish to another topic
return Message(value=b'{"user_clicked": "true"}')
app.run()
```