https://github.com/lostclus/aiosafeconsumer
Safely consume and process data.
https://github.com/lostclus/aiosafeconsumer
consumers data-synchronization elasticsearch kafka message-bus message-queue microservices mongodb redis
Last synced: 10 months ago
JSON representation
Safely consume and process data.
- Host: GitHub
- URL: https://github.com/lostclus/aiosafeconsumer
- Owner: lostclus
- License: bsd-2-clause
- Created: 2024-10-10T16:57:08.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-02-02T13:17:33.000Z (about 1 year ago)
- Last Synced: 2025-04-12T05:14:54.139Z (10 months ago)
- Topics: consumers, data-synchronization, elasticsearch, kafka, message-bus, message-queue, microservices, mongodb, redis
- Language: Python
- Homepage:
- Size: 122 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.rst
- License: LICENSE
Awesome Lists containing this project
README
aiosafeconsumer
===============
.. image:: https://github.com/lostclus/aiosafeconsumer/actions/workflows/tests.yml/badge.svg
:target: https://github.com/lostclus/aiosafeconsumer/actions
.. image:: https://img.shields.io/pypi/v/aiosafeconsumer.svg
:target: https://pypi.org/project/aiosafeconsumer/
:alt: Current version on PyPi
.. image:: https://img.shields.io/pypi/pyversions/aiosafeconsumer
:alt: PyPI - Python Version
aiosafeconsumer is a library that provides abstractions and some implementations
to consume data somewhere and process it.
Features:
* Based on AsyncIO
* Type annotated
* Use logging with contextual information
Abstractions:
* `DataSource` - waits for data and returns batch of records using Python generator
* `DataProcessor` - accepts batch of records and precess it
* `DataTransformer` - accepts batch of records and transform it and calls
another processor to precess it. Extends `DataProcessor`
* `Worker` - abstract worker. Do a long running task
* `ConsumerWorker` - connects `DataSource` and `DataProcessor`. Extends Worker
* `DataWriter` - base abstraction to perform data synchronization. Extends DataProcessor
Current implementations:
* `KafkaSource` - read data from Kafka
* `RedisWriter` - synchronize data in Redis
* `ElasticsearchWriter` - synchronize data in Elasticsearch
* `WorkerPool` - controller to setup and run workers in parallel. Can handle worker failures and restarts workers when it fails or exits.
Install::
pip install aiosafeconsumer
Install with Kafka::
pip install aiosafeconsumer[kafka]
Install with Redis::
pip install aiosafeconsumer[redis]
Install with Elasticsearch::
pip install aiosafeconsumer[elasticsearch]
Install with MongoDB::
pip install aiosafeconsumer[mongo]
Links:
* Producer library: https://github.com/lostclus/django-kafka-streamer
* Example application: https://github.com/lostclus/WeatherApp
TODO:
* PostgreSQL writer
* ClickHouse writer