{"id":15968858,"url":"https://github.com/vonsteer/taskiq-aio-sqs","last_synced_at":"2025-03-18T00:30:41.962Z","repository":{"id":257414408,"uuid":"858199700","full_name":"vonsteer/taskiq-aio-sqs","owner":"vonsteer","description":"SQS/S3 Broker for TaskIQ using Aiobotocore","archived":false,"fork":false,"pushed_at":"2024-09-17T23:26:30.000Z","size":49,"stargazers_count":7,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-14T05:03:10.377Z","etag":null,"topics":["aws","queue","s3","sqs","taskiq","taskiq-broker"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vonsteer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-16T13:39:07.000Z","updated_at":"2025-03-06T11:01:09.000Z","dependencies_parsed_at":null,"dependency_job_id":"6034cd9d-a06b-47de-9f99-8601fbb90d85","html_url":"https://github.com/vonsteer/taskiq-aio-sqs","commit_stats":null,"previous_names":["vonsteer/taskiq-aio-sqs"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vonsteer%2Ftaskiq-aio-sqs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vonsteer%2Ftaskiq-aio-sqs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vonsteer%2Ftaskiq-aio-sqs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vonsteer%2Ftaskiq-aio-sqs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vonsteer","download_url":"https://codeload.github.com/vonsteer/taskiq-aio-sqs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243526945,"owners_count":20305114,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","queue","s3","sqs","taskiq","taskiq-broker"],"created_at":"2024-10-07T19:04:19.137Z","updated_at":"2025-03-18T00:30:41.949Z","avatar_url":"https://github.com/vonsteer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TaskIQ SQS/S3 aiobotocore\n\n[![PyPI](https://img.shields.io/pypi/v/taskiq-aio-sqs)](https://pypi.org/project/taskiq-aio-sqs/)\n[![Python Versions](https://img.shields.io/pypi/pyversions/taskiq-aio-sqs)](https://pypi.org/project/taskiq-aio-sqs/)\n[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)\n[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit\u0026logoColor=white)](https://github.com/pre-commit/pre-commit)\n[![Coverage Status](./coverage-badge.svg?dummy=8484744)](./coverage.xml)\n\nThis library provides you with a fully asynchronous SQS broker and S3 backend for TaskIQ using aiobotocore.\nInspired by the [taskiq-sqs](https://github.com/ApeWorX/taskiq-sqs) broker.\n\nBesides the SQS broker, this library also provides an S3 backend for the results, this is useful when the results are too large for SQS.\nAddidionally, the broker itself can be configured to use S3 + SQS for messages that are too large for SQS,\nreplicating the behaviour of the [Amazon Extended Client Library](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-managing-large-messages.html).\n\n## Installation\n\n```bash\npip install taskiq-aio-sqs\n```\n\n## General Usage:\nHere is an example of how to use the SQS broker with the S3 backend:\n\n```python\n# broker.py\nimport asyncio\nfrom taskiq_aio_sqs import SQSBroker, S3Backend\n\ns3_result_backend = S3Backend(\n    endpoint_url=\"http://localhost:4566\",\n    bucket_name=\"response-bucket\",  # bucket must exist\n)\n\nbroker = SQSBroker(\n    endpoint_url=\"http://localhost:4566\",\n    result_backend=s3_result_backend,\n    sqs_queue_name=\"my-queue\",\n)\n\n\n@broker.task\nasync def i_love_aws() -\u003e None:\n    \"\"\"I hope my cloud bill doesn't get too high!\"\"\"\n    await asyncio.sleep(5.5)\n    print(\"Hello there!\")\n\n\nasync def main():\n    task = await i_love_aws.kiq()\n    print(await task.wait_result())\n\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n\n```\n### Delayed Tasks:\n\nDelayed tasks can be created in 3 ways:\n - by using the `delay` parameter in the task decorator\n - by using the kicker with the `delay` label\n - by setting the `delay_seconds` parameter in the broker, which will apply to all tasks processed by the broker.\n\nHere's an example of how to use delayed tasks:\n\n```python\nbroker = SQSBroker(\n    endpoint_url=\"http://localhost:4566\",\n    delay_seconds=3,\n    sqs_queue_name=\"my-queue\",\n)\n\n@broker.task()\nasync def general_task() -\u003e int:\n    return 1\n\n@broker.task(delay=7)\nasync def delayed_task() -\u003e int:\n    return 1\n\nasync def main():\n    await broker.startup()\n    # This message will be received by workers after 3 seconds\n    # delay using the delay_seconds parameter in the broker init.\n    await general_task.kiq()\n\n    # This message will be received by workers after 7 seconds delay.\n    await delayed_task.kiq()\n\n    # This message is going to be received after the delay in 4 seconds.\n    # Since we overriden the `delay` label using kicker.\n    await delayed_task.kicker().with_labels(delay=4).kiq()\n\n```\n\n### Extended Messages with S3:\n\nYou can also use S3 to store messages that are too large for SQS. To do this, you need to set the `s3_extended_bucket_name` parameter in the broker configuration.\n\nHere's an example of this behaviour:\n```python\npub_broker = SQSBroker(\n    endpoint_url=\"http://localhost:4566\",\n    sqs_queue_name=\"my-queue\",\n    s3_extended_bucket_name=\"response-bucket\",\n)\n\nsub_broker = SQSBroker(\n    endpoint_url=\"http://localhost:4566\",\n    sqs_queue_name=\"my-queue\",\n)\n\nLARGE_MESSAGE = b\"x\" * (256 * 1024 + 1)  # 256 KB is the limit for SQS\n\n@pub_broker.task()\nasync def large_task() -\u003e bytes:\n    return LARGE_MESSAGE\n\n\nasync def main():\n    await pub_broker.startup()\n    await sub_broker.startup()\n    # This message will store data in S3 and send a reference to SQS\n    # This reference will include the S3 bucket and key.\n    await large_task.kiq()\n\n    async for msg in sub_broker.listen():\n        message = msg\n        break  # Stop after receiving one message\n\n    # The message will be automatically retrieved from S3\n    # and the full data will be available in the message.\n    assert message.data == LARGE_MESSAGE\n\n\n```\n\n## Configuration:\n\nSQS Broker parameters:\n* `endpoint_url` - url to access sqs, this is particularly useful if running on ECS.\n* `sqs_queue_name` - name of the sqs queue.\n* `region_name` - region name, defaults to `us-east-1`.\n* `aws_access_key_id` - aws access key id (Optional).\n* `aws_secret_access_key` - aws secret access key (Optional).\n* `use_task_id_for_deduplication` - use task_id for deduplication, this is useful when using a Fifo queue without content based deduplication, defaults to False.\n* `wait_time_seconds` - wait time in seconds for long polling, defaults to 0.\n* `max_number_of_messages` - maximum number of messages to receive, defaults to 1 (max 10).\n* `s3_extended_bucket_name` - extended bucket name for the s3 objects,\n  adding this will allow the broker to kick messages that are too large for SQS by using S3 as well,\n  by default the listen function handles this behaviour, defaults to None.\n* `task_id_generator` - custom task_id generator (Optional).\n* `result_backend` - custom result backend (Optional).\n\n\nS3 Result Backend parameters:\n* `bucket_name` - name of the s3 bucket.\n* `base_path` - base path for the s3 objects, defaults to \"\".\n* `endpoint_url` - url to access s3, this is particularly useful if running on ECS.\n* `region_name` - region name, defaults to `us-east-1`.\n* `aws_access_key_id` - aws access key id (Optional).\n* `aws_secret_access_key` - aws secret access key (Optional).\n* `serializer` - custom serializer, defaults to `OrjsonSerializer`.\n\n# Local Development:\nWe use make to handle the commands for the project, you can see the available commands by running this in the root directory:\n```bash\nmake\n```\n\n## Setup\nTo setup the project, you can run the following commands:\n```bash\nmake install\n```\nThis will install the required dependencies for the project just using pip.\n\n## Linting\nWe use pre-commit to do linting locally, this will be included in the dev dependencies.\nWe use ruff for linting and formatting, and pyright for static type checking.\nTo install the pre-commit hooks, you can run the following command:\n```bash\npre-commit install\n```\nIf you for some reason hate pre-commit, you can run the following command to lint the code:\n```bash\nmake check\n```\n\n## Testing\nTo run tests, you can use the following command:\n```bash\nmake test\n```\nIn the background this will setup localstack to replicate the AWS services, and run the tests.\nIt will also generate the coverage report and the badge.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvonsteer%2Ftaskiq-aio-sqs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvonsteer%2Ftaskiq-aio-sqs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvonsteer%2Ftaskiq-aio-sqs/lists"}