{"id":13501852,"url":"https://github.com/joowani/kq","last_synced_at":"2025-04-12T18:50:09.820Z","repository":{"id":49498581,"uuid":"71026498","full_name":"joowani/kq","owner":"joowani","description":"Kafka-based Job Queue for Python","archived":false,"fork":false,"pushed_at":"2022-02-04T12:38:12.000Z","size":111,"stargazers_count":572,"open_issues_count":0,"forks_count":23,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-04-03T23:11:18.179Z","etag":null,"topics":["apache-kafka","job-queue","kafka","kafka-client","kafka-consumer","kafka-producer","producer-consumer","python","python3","queueing","worker-queue"],"latest_commit_sha":null,"homepage":"http://kq.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joowani.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-10-16T02:49:53.000Z","updated_at":"2025-03-19T13:28:53.000Z","dependencies_parsed_at":"2022-08-28T02:41:49.604Z","dependency_job_id":null,"html_url":"https://github.com/joowani/kq","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joowani%2Fkq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joowani%2Fkq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joowani%2Fkq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joowani%2Fkq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joowani","download_url":"https://codeload.github.com/joowani/kq/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248618219,"owners_count":21134199,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","job-queue","kafka","kafka-client","kafka-consumer","kafka-producer","producer-consumer","python","python3","queueing","worker-queue"],"created_at":"2024-07-31T22:01:53.132Z","updated_at":"2025-04-12T18:50:09.800Z","avatar_url":"https://github.com/joowani.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# KQ: Kafka Job Queue for Python\n\n![Build](https://github.com/joowani/kq/workflows/Build/badge.svg)\n![CodeQL](https://github.com/joowani/kq/workflows/CodeQL/badge.svg)\n[![codecov](https://codecov.io/gh/joowani/kq/branch/main/graph/badge.svg?token=ha9Xg7eULv)](https://codecov.io/gh/joowani/kq)\n[![PyPI version](https://badge.fury.io/py/kq.svg)](https://badge.fury.io/py/kq)\n[![GitHub license](https://img.shields.io/github/license/joowani/kq?color=brightgreen)](https://github.com/joowani/kq/blob/main/LICENSE)\n![Python version](https://img.shields.io/badge/python-3.6%2B-blue)\n\n**KQ (Kafka Queue)** is a lightweight Python library which lets you enqueue and\nexecute jobs asynchronously using [Apache Kafka](https://kafka.apache.org/). It uses\n[kafka-python](https://github.com/dpkp/kafka-python) under the hood.\n\n## Announcements\n\n* Support for Python 3.5 will be dropped from KQ version 3.0.0.\n* See [releases](https://github.com/joowani/kq/releases) for latest updates.\n\n## Requirements\n\n* [Apache Kafka](https://kafka.apache.org) 0.9+\n* Python 3.6+\n\n## Installation\n\nInstall using [pip](https://pip.pypa.io):\n\n```shell\npip install kq\n```\n\n## Getting Started\n\nStart your Kafka instance. \nExample using [Docker](https://github.com/lensesio/fast-data-dev):\n\n```shell\ndocker run -p 9092:9092 -e ADV_HOST=127.0.0.1 lensesio/fast-data-dev\n```\n\nDefine your KQ ``worker.py`` module:\n\n```python\nimport logging\n\nfrom kafka import KafkaConsumer\nfrom kq import Worker\n\n# Set up logging.\nformatter = logging.Formatter(\"[%(levelname)s] %(message)s\")\nstream_handler = logging.StreamHandler()\nstream_handler.setFormatter(formatter)\nlogger = logging.getLogger(\"kq.worker\")\nlogger.setLevel(logging.DEBUG)\nlogger.addHandler(stream_handler)\n\n# Set up a Kafka consumer.\nconsumer = KafkaConsumer(\n    bootstrap_servers=\"127.0.0.1:9092\",\n    group_id=\"group\",\n    auto_offset_reset=\"latest\"\n)\n\n# Set up a worker.\nworker = Worker(topic=\"topic\", consumer=consumer)\nworker.start()\n```\n\nStart your worker:\n\n```shell\npython my_worker.py\n[INFO] Starting Worker(hosts=127.0.0.1:9092 topic=topic, group=group) ...\n```\n\nEnqueue a function call:\n\n```python\nimport requests\n\nfrom kafka import KafkaProducer\nfrom kq import Queue\n\n# Set up a Kafka producer.\nproducer = KafkaProducer(bootstrap_servers=\"127.0.0.1:9092\")\n\n# Set up a queue.\nqueue = Queue(topic=\"topic\", producer=producer)\n\n# Enqueue a function call.\njob = queue.enqueue(requests.get, \"https://google.com\")\n\n# You can also specify the job timeout, Kafka message key and partition.\njob = queue.using(timeout=5, key=b\"foo\", partition=0).enqueue(requests.get, \"https://google.com\")\n```\n\nThe worker executes the job in the background:\n\n```shell\npython my_worker.py\n[INFO] Starting Worker(hosts=127.0.0.1:9092, topic=topic, group=group) ...\n[INFO] Processing Message(topic=topic, partition=0, offset=0) ...\n[INFO] Executing job c7bf2359: requests.api.get(\"https://www.google.com\")\n[INFO] Job c7bf2359 returned: \u003cResponse [200]\u003e\n```\n\nSee the [documentation](https://kq.readthedocs.io) for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoowani%2Fkq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoowani%2Fkq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoowani%2Fkq/lists"}