{"id":19606350,"url":"https://github.com/taskiq-python/taskiq-redis","last_synced_at":"2025-04-08T03:19:56.717Z","repository":{"id":50598750,"uuid":"518152326","full_name":"taskiq-python/taskiq-redis","owner":"taskiq-python","description":"Broker and result backend for taskiq","archived":false,"fork":false,"pushed_at":"2024-09-28T21:42:31.000Z","size":375,"stargazers_count":40,"open_issues_count":5,"forks_count":18,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-10-29T14:22:15.227Z","etag":null,"topics":["asyncio","python","redis","taskiq","taskiq-broker","taskiq-result-backend"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/taskiq-python.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-26T17:16:20.000Z","updated_at":"2024-10-18T20:34:36.000Z","dependencies_parsed_at":"2022-08-12T21:50:30.522Z","dependency_job_id":"bf2937a0-32f7-49ea-9f0a-f1d446e60e2a","html_url":"https://github.com/taskiq-python/taskiq-redis","commit_stats":{"total_commits":105,"total_committers":13,"mean_commits":8.076923076923077,"dds":"0.47619047619047616","last_synced_commit":"e30ed080f4df584f8de03535bffd3a0c1c090586"},"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taskiq-python%2Ftaskiq-redis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taskiq-python%2Ftaskiq-redis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taskiq-python%2Ftaskiq-redis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taskiq-python%2Ftaskiq-redis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/taskiq-python","download_url":"https://codeload.github.com/taskiq-python/taskiq-redis/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246709985,"owners_count":20821314,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asyncio","python","redis","taskiq","taskiq-broker","taskiq-result-backend"],"created_at":"2024-11-11T10:04:56.293Z","updated_at":"2025-04-08T03:19:56.701Z","avatar_url":"https://github.com/taskiq-python.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TaskIQ-Redis\n\nTaskiq-redis is a plugin for taskiq that adds a new broker and result backend based on redis.\n\n# Installation\n\nTo use this project you must have installed core taskiq library:\n```bash\npip install taskiq\n```\nThis project can be installed using pip:\n```bash\npip install taskiq-redis\n```\n\n# Usage\n\nLet's see the example with the redis broker and redis async result:\n\n```python\n# broker.py\nimport asyncio\n\nfrom taskiq_redis import RedisAsyncResultBackend, RedisStreamBroker\n\nresult_backend = RedisAsyncResultBackend(\n    redis_url=\"redis://localhost:6379\",\n)\n\n# Or you can use PubSubBroker if you need broadcasting\n# Or ListQueueBroker if you don't want acknowledges\nbroker = RedisStreamBroker(\n    url=\"redis://localhost:6379\",\n).with_result_backend(result_backend)\n\n\n@broker.task\nasync def best_task_ever() -\u003e None:\n    \"\"\"Solve all problems in the world.\"\"\"\n    await asyncio.sleep(5.5)\n    print(\"All problems are solved!\")\n\n\nasync def main():\n    task = await best_task_ever.kiq()\n    print(await task.wait_result())\n\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\nLaunch the workers:\n`taskiq worker broker:broker`\nThen run the main code:\n`python3 broker.py`\n\n\n## Brokers\n\nThis package contains 6 broker implementations.\n3 broker types:\n* PubSub broker\n* ListQueue broker\n* Stream broker\n\nEach of type is implemented for each redis architecture:\n* Single node\n* Cluster\n* Sentinel\n\nHere's a small breakdown of how they differ from eachother.\n\n\n### PubSub\n\nBy default on old redis versions PUBSUB was the way of making redis into a queue.\nBut using PUBSUB means that all messages delivered to all subscribed consumers.\n\n\u003e [!WARNING]\n\u003e This broker doesn't support acknowledgements. If during message processing\n\u003e Worker was suddenly killed the message is going to be lost.\n\n### ListQueue\n\nThis broker creates a list of messages at some key. Adding new tasks will be done\nby appending them from the left side using `lpush`, and taking them from the right side using `brpop`.\n\n\u003e [!WARNING]\n\u003e This broker doesn't support acknowledgements. If during message processing\n\u003e Worker was suddenly killed the message is going to be lost.\n\n### Stream\n\nStream brokers use redis [stream type](https://redis.io/docs/latest/develop/data-types/streams/) to store and fetch messages.\n\n\u003e [!TIP]\n\u003e This broker **supports** acknowledgements and therefore is fine to use in cases when data durability is\n\u003e required.\n\n## RedisAsyncResultBackend configuration\n\nRedisAsyncResultBackend parameters:\n* `redis_url` - url to redis.\n* `keep_results` - flag to not remove results from Redis after reading.\n* `result_ex_time` - expire time in seconds (by default - not specified)\n* `result_px_time` - expire time in milliseconds (by default - not specified)\n* Any other keyword arguments are passed to `redis.asyncio.BlockingConnectionPool`.\n  Notably, you can use `timeout` to set custom timeout in seconds for reconnects\n  (or set it to `None` to try reconnects indefinitely).\n\n\u003e [!WARNING]\n\u003e **It is highly recommended to use expire time in RedisAsyncResultBackend**\n\u003e If you want to add expiration, either `result_ex_time` or `result_px_time` must be set.\n\u003e ```python\n\u003e # First variant\n\u003e redis_async_result = RedisAsyncResultBackend(\n\u003e     redis_url=\"redis://localhost:6379\",\n\u003e     result_ex_time=1000,\n\u003e )\n\u003e\n\u003e # Second variant\n\u003e redis_async_result = RedisAsyncResultBackend(\n\u003e     redis_url=\"redis://localhost:6379\",\n\u003e     result_px_time=1000000,\n\u003e )\n\u003e ```\n\n\n## Schedule sources\n\n\nYou can use this package to add dynamic schedule sources. They are used to store\nschedules for taskiq scheduler.\n\nThe advantage of using schedule sources from this package over default `LabelBased` source is that you can\ndynamically add schedules in it.\n\nWe have two types of schedules:\n\n* `RedisScheduleSource`\n* `ListRedisScheduleSource`\n\n\n### RedisScheduleSource\n\nThis source is super simple. It stores all schedules by key `{prefix}:{schedule_id}`. When scheduler requests\nschedules, it retrieves all values from redis that start with a given `prefix`.\n\nThis is very ineficent and should not be used for high-volume schedules. Because if you have `1000` schedules, this scheduler will make at least `20` requests to retrieve them (we use `scan` and `mget` to minimize number of calls).\n\n### ListRedisScheduleSource\n\nThis source holds values in lists.\n\n* For cron tasks it uses key `{prefix}:cron`.\n* For timed schedules it uses key `{prefix}:time:{time}` where `{time}` is actually time where schedules should run.\n\nThe main advantage of this approach is that we only fetch tasks we need to run at a given time and do not perform any excesive calls to redis.\n\n\n### Migration from one source to another\n\nTo migrate from `RedisScheduleSource` to `ListRedisScheduleSource` you can define the latter as this:\n\n```python\n# broker.py\nimport asyncio\nimport datetime\n\nfrom taskiq import TaskiqScheduler\n\nfrom taskiq_redis import ListRedisScheduleSource, RedisStreamBroker\nfrom taskiq_redis.schedule_source import RedisScheduleSource\n\nbroker = RedisStreamBroker(url=\"redis://localhost:6379\")\n\nold_source = RedisScheduleSource(\"redis://localhost/1\", prefix=\"prefix1\")\narray_source = ListRedisScheduleSource(\n    \"redis://localhost/1\",\n    prefix=\"prefix2\",\n    # To migrate schedules from an old source.\n).with_migrate_from(\n    old_source,\n    # To delete schedules from an old source.\n    delete_schedules=True,\n)\nscheduler = TaskiqScheduler(broker, [array_source])\n```\n\nDuring startup the scheduler will try to migrate schedules from an old source to a new one. Please be sure to specify different prefixe just to avoid any kind of collision between these two.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftaskiq-python%2Ftaskiq-redis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftaskiq-python%2Ftaskiq-redis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftaskiq-python%2Ftaskiq-redis/lists"}