{"id":18076577,"url":"https://github.com/strongbugman/oxalis","last_synced_at":"2025-04-12T08:23:39.231Z","repository":{"id":45787757,"uuid":"500822032","full_name":"strongbugman/oxalis","owner":"strongbugman","description":"Distributed async task/job queue, like Celery for asyncio world","archived":false,"fork":false,"pushed_at":"2023-07-13T09:52:19.000Z","size":130,"stargazers_count":85,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-26T03:33:12.731Z","etag":null,"topics":["asyncio","celery","python","queue","rabbitmq","redis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/strongbugman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-07T12:01:09.000Z","updated_at":"2024-05-15T16:52:58.000Z","dependencies_parsed_at":"2024-10-31T11:10:34.110Z","dependency_job_id":"50bba271-4da5-4133-b1e4-ee8ff2a8bf14","html_url":"https://github.com/strongbugman/oxalis","commit_stats":null,"previous_names":[],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/strongbugman%2Foxalis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/strongbugman%2Foxalis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/strongbugman%2Foxalis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/strongbugman%2Foxalis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/strongbugman","download_url":"https://codeload.github.com/strongbugman/oxalis/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248538055,"owners_count":21120935,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asyncio","celery","python","queue","rabbitmq","redis"],"created_at":"2024-10-31T11:10:24.085Z","updated_at":"2025-04-12T08:23:39.211Z","avatar_url":"https://github.com/strongbugman.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp\u003e\n\u003ca href=\"https://pypi.org/project/oxalis/\"\u003e\n    \u003cimg src=\"https://badge.fury.io/py/oxalis.svg\" alt=\"Package version\"\u003e\n\u003c/a\u003e\n\u003c/p\u003e\n\n# Oxalis  \n\nDistributed async task/job queue, like Celery for `asyncio` world\n\n## Feature\n\n* Redis and AMQP(RabbitMQ etc.) support\n* Task timeout and concurrency limit support\n* Delayed task(Both Redis and RabbitMQ) support\n* Cron task/job beater\n* Built-in coroutine pool with concurrency and time limit\n\n## Install\n\n```pip install oxalis```\n\n## Example with Redis backend\n\nDefine task:\n```python\nfrom redis.asyncio.client import Redis\nfrom oxalis.redis import Oxalis\n\n\noxalis = Oxalis(Redis(host=os.getenv(\"REDIS_HOST\", \"redis\")))\n\n@oxalis.register()\nasync def hello_task():\n    print(\"Hello oxalis\")\n```\n\nRun worker(consumer):\n```python\noxalis.run_worker_master()\n```\n\n```shell\npython ex.py\nINFO:oxalis:Registered Task: \u003cTask(hello_task)\u003e\nINFO:oxalis:Run worker: \u003cOxalis(pid-101547)\u003e...\nINFO:oxalis:Run worker: \u003cOxalis(pid-101548)\u003e...\nINFO:oxalis:Run worker: \u003cOxalis(pid-101549)\u003e...\nINFO:oxalis:Run worker: \u003cOxalis(pid-101550)\u003e...\nINFO:oxalis:Run worker: \u003cOxalis(pid-101551)\u003e...\nINFO:oxalis:Run worker: \u003cOxalis(pid-101552)\u003e...\nINFO:oxalis:Run worker: \u003cOxalis(pid-101554)\u003e...\n```\n\nRun client(producer):\n```python\nimport asyncio\n\nasyncio.get_event_loop().run_until_complete(oxalis.connect())\nfor i in range(10):\n    asyncio.get_event_loop().run_until_complete(hello_task.delay())\n    asyncio.get_event_loop().run_until_complete(hello_task.delay(_delay=1))  # delay execution after 1s\n```\n\nRun cron beater:\n```python\nfrom oxalis.beater import Beater\n\nbeater = Beater(oxalis)\n\nbeater.register(\"*/1 * * * *\", hello_task)\nbeater.run()\n```\n```shell\npython exb.py \nINFO:oxalis:Beat task: \u003cTask(hello_task)\u003e at \u003c*/1 * * * *\u003e ...\n```\n\n## TaskCodec\n\nThe `TaskCodec` will encode/decode task args, default codec will use `json`\n\nCustom task codec:\n```python\nfrom oxalis.base import TaskCodec\n\nclass MyTaskCodec(TaskCodec):\n    @classmethod\n    def encode(\n        cls,\n        task: Task,\n        task_args: tp.Sequence[tp.Any],\n        task_kwargs: tp.Dict[str, tp.Any],\n    ) -\u003e bytes:\n        ...\n\n    @classmethod\n    def decode(cls, content: bytes) -\u003e TaskCodec.MESSAGE_TYPE:\n        ...\n\n\n\noxalis = Oxalis(Redis(host=os.getenv(\"REDIS_HOST\", \"redis\")), task_codec=MyTaskCodec())\n...\n```\n\n\n## Task pool\n\nOxalis use one coroutine pool with concurrency limit and timeout limit to run all task\n\nCustom pool:\n\n```python\nfrom redis.asyncio.client import Redis\nfrom oxalis.redis import Oxalis\nfrom oxalis.pool import Pool\n\noxalis = Oxalis(Redis(host=os.getenv(\"REDIS_HOST\", \"redis\")), pool=Pool(concurrency=10, timeout=60))\n```\n\n* For Redis task, the `queue` will be blocked util `pool` is not fully loaded\n* For AMQP task, oxalis use AMQP's QOS to limit worker concurrency(`pool`'s concurrency will be -1 which means the pool's concurrency will not be limited)\n* `asyncio.TimeoutError` will be raised if one task is timeout\n* Every worker process has owned limited pool\n\n\nSpecified one task timeout limit:\n```python\n@oxalis.register(queue=custom_queue, timeout=10)\ndef custom_task():\n    print(\"Hello oxalis\")\n```\n\n## Custom hook\n\nOxalis defined some hook API for inherited subclass:\n```python\nclass MyOxalis(Oxalis):\n    def on_worker_init():\n        # will be called before worker started\n        pass\n\n    def on_worker_close():\n        # will be called after worker started\n        pass\n```\n\nSome API can be rewritten or inherited for custom usage, eg:\n```python\nimport sentry_sdk\n\nclass MyOxalis(Oxalis):\n    async def exec_task(self, task: Task, *task_args, **task_kwargs):\n        \"\"\"\n        capture exception to sentry\n        \"\"\"\n        try:\n            await super().exec_task(task, *task_args, **task_kwargs)\n        except Exception as e:\n            sentry_sdk.capture_exception(e)\n```\n\n\n## Redis Backend Detail\n\nOxalis use redis's `list` and `pubsub` structure as a message queue\n\n### Queue\n\nCustom queue:\n```python\nfrom oxalis.redis import Queue, PubsubQueue\n\ncustom_queue = Queue(\"custom\")\nbus_queue = PubsubQueue(\"bus\")\n```\n\nRegister task:\n```python\n@oxalis.register(queue=custom_queue)\ndef custom_task():\n    print(\"Hello oxalis\")\n\n@oxalis.register(queue=bus_queue)\ndef bus_task():\n    print(\"Hello oxalis\")\n```\n\n* For task producer, the task will send to specified queue when call `task.delay()`\n* For task consumer, oxalis will listen those queues and receive task from them\n\n### Concurrency limit\n\nOxalis using coroutine pool's concurrency limit way, we can set different concurrency limit with specified pool for one task:\n\n```python\n@oxalis.register(pool=Pool(concurrency=1))\ndef custom_task():\n    print(\"Hello oxalis\")\n```\n\n### Delayed task\n\nSupport by redis [zset](https://redis.com/ebook/part-2-core-concepts/chapter-6-application-components-in-redis/6-4-task-queues/6-4-2-delayed-tasks/)\n\n##  AMQP Backend Detail\n\n\n### Custom Queue and Exchange\n\nOxalis using AMQP's way to define Exchange, Queue and their bindings\n\n```python\nimport asyncio\nimport logging\nimport time\n\nfrom aio_pika import RobustConnection\nfrom oxalis.amqp import Exchange, ExchangeType, Oxalis, Pool, Queue\n\ne = Exchange(\"test\")\nq = Queue(\"test\", durable=False)\ne2 = Exchange(\"testfanout\", type=ExchangeType.FANOUT)\nq2 = Queue(\"testfanout\", durable=False)\n\n\noxalis = Oxalis(RobustConnection(\"amqp://root:letmein@rabbitmq:5672/\"))\noxalis.register_binding(q, e, \"test\")\noxalis.register_binding(q2, e2, \"\")\noxalis.register_queues([q, q2])\n\n\n@oxalis.register(exchange=e, routing_key=\"test\")\nasync def task1():\n    await asyncio.sleep(1)\n    print(\"hello oxalis\")\n\n\n@oxalis.register(exchange=e2)\nasync def task2():\n    await asyncio.sleep(10)\n    print(\"hello oxalis\")\n\n```\n\n* For producer, task `oxalis.register`  defined one task message will send to which exchange(by routing key)\n* For consumer, `register_queues` defined which queues oxalis will listened\n* Task routing defined by bindings\n\n### Concurrency limit\n\nOxalis use AMQP's QOS to limit worker concurrency(Task's `ack_later` should be true), so coroutine pool's concurrency should not be limited.\n\nCustom queue QOS:\n```python\noxalis = Oxalis(RobustConnection(\"amqp://root:letmein@rabbitmq:5672/\"), default_queue=Queue(\"custom\",consumer_prefetch_count=10))\n...\nfanout_queue = Queue(\"testfanout\", durable=False, consumer_prefetch_count=3)\noxalis.register_queues([fanout_queue])\n...\n```\n\n### Custom task behavior\n\nDefine task how to perform `ack` and `reject` \n\n```python\n# always ack even task failed(raise exception)\n@oxalis.register(ack_always=True, reject=False)\nasync def task2():\n    await asyncio.sleep(10)\n    print(\"hello oxalis\")\n\n#  reject with requeue when task failed\n@oxalis.register(reject_requeue=True)\nasync def task2():\n    await asyncio.sleep(10)\n    print(\"hello oxalis\")\n```\n\n### Delayed task\n\nSupport by RabbitMq's [plugin](https://blog.rabbitmq.com/posts/2015/04/scheduling-messages-with-rabbitmq)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstrongbugman%2Foxalis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstrongbugman%2Foxalis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstrongbugman%2Foxalis/lists"}