{"id":18073320,"url":"https://github.com/lostclus/aiosafeconsumer","last_synced_at":"2025-04-12T05:15:07.732Z","repository":{"id":257831348,"uuid":"870784199","full_name":"lostclus/aiosafeconsumer","owner":"lostclus","description":"Safely consume and process data.","archived":false,"fork":false,"pushed_at":"2025-02-02T13:17:33.000Z","size":125,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-12T05:14:54.139Z","etag":null,"topics":["consumers","data-synchronization","elasticsearch","kafka","message-bus","message-queue","microservices","mongodb","redis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lostclus.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-10T16:57:08.000Z","updated_at":"2025-02-02T13:15:15.000Z","dependencies_parsed_at":"2024-10-25T16:56:33.063Z","dependency_job_id":"fa8df4b9-8bc0-45f9-9f96-1d1ca3c638cb","html_url":"https://github.com/lostclus/aiosafeconsumer","commit_stats":null,"previous_names":["lostclus/aiosafeconsumer"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lostclus%2Faiosafeconsumer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lostclus%2Faiosafeconsumer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lostclus%2Faiosafeconsumer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lostclus%2Faiosafeconsumer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lostclus","download_url":"https://codeload.github.com/lostclus/aiosafeconsumer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248519557,"owners_count":21117761,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consumers","data-synchronization","elasticsearch","kafka","message-bus","message-queue","microservices","mongodb","redis"],"created_at":"2024-10-31T10:06:07.187Z","updated_at":"2025-04-12T05:15:07.693Z","avatar_url":"https://github.com/lostclus.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"aiosafeconsumer\n===============\n\n.. image:: https://github.com/lostclus/aiosafeconsumer/actions/workflows/tests.yml/badge.svg\n    :target: https://github.com/lostclus/aiosafeconsumer/actions\n\n.. image:: https://img.shields.io/pypi/v/aiosafeconsumer.svg\n    :target: https://pypi.org/project/aiosafeconsumer/\n    :alt: Current version on PyPi\n\n.. image:: https://img.shields.io/pypi/pyversions/aiosafeconsumer\n    :alt: PyPI - Python Version\n\naiosafeconsumer is a library that provides abstractions and some implementations\nto consume data somewhere and process it.\n\nFeatures:\n\n* Based on AsyncIO\n* Type annotated\n* Use logging with contextual information\n\nAbstractions:\n\n* `DataSource` - waits for data and returns batch of records using Python generator\n* `DataProcessor` - accepts batch of records and precess it\n* `DataTransformer` - accepts batch of records and transform it and calls\n  another processor to precess it. Extends `DataProcessor`\n* `Worker` - abstract worker. Do a long running task\n* `ConsumerWorker` - connects `DataSource` and `DataProcessor`. Extends Worker\n* `DataWriter` - base abstraction to perform data synchronization. Extends DataProcessor\n\nCurrent implementations:\n\n* `KafkaSource` - read data from Kafka\n* `RedisWriter` - synchronize data in Redis\n* `ElasticsearchWriter` - synchronize data in Elasticsearch\n* `WorkerPool` - controller to setup and run workers in parallel. Can handle worker failures and restarts workers when it fails or exits.\n\nInstall::\n\n    pip install aiosafeconsumer\n\nInstall with Kafka::\n\n    pip install aiosafeconsumer[kafka]\n\nInstall with Redis::\n\n    pip install aiosafeconsumer[redis]\n\nInstall with Elasticsearch::\n\n    pip install aiosafeconsumer[elasticsearch]\n\nInstall with MongoDB::\n\n    pip install aiosafeconsumer[mongo]\n\nLinks:\n\n* Producer library: https://github.com/lostclus/django-kafka-streamer\n* Example application: https://github.com/lostclus/WeatherApp\n\nTODO:\n\n* PostgreSQL writer\n* ClickHouse writer\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flostclus%2Faiosafeconsumer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flostclus%2Faiosafeconsumer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flostclus%2Faiosafeconsumer/lists"}