{"id":20059114,"url":"https://github.com/beam-pyio/sqs_pyio","last_synced_at":"2026-01-05T10:03:19.113Z","repository":{"id":251165640,"uuid":"832490081","full_name":"beam-pyio/sqs_pyio","owner":"beam-pyio","description":"Apache Beam Python I/O connector for Amazon SQS","archived":false,"fork":false,"pushed_at":"2024-09-20T00:06:35.000Z","size":2966,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-11-01T14:18:16.970Z","etag":null,"topics":["apache-beam","aws","data-engineering","data-streaming","python","sqs"],"latest_commit_sha":null,"homepage":"https://beam-pyio.github.io/sqs_pyio/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/beam-pyio.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-23T06:13:26.000Z","updated_at":"2024-09-21T09:48:27.000Z","dependencies_parsed_at":"2024-09-17T10:23:21.809Z","dependency_job_id":null,"html_url":"https://github.com/beam-pyio/sqs_pyio","commit_stats":null,"previous_names":["beam-pyio/sqs_pyio"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beam-pyio%2Fsqs_pyio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beam-pyio%2Fsqs_pyio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beam-pyio%2Fsqs_pyio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beam-pyio%2Fsqs_pyio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/beam-pyio","download_url":"https://codeload.github.com/beam-pyio/sqs_pyio/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224452831,"owners_count":17313668,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-beam","aws","data-engineering","data-streaming","python","sqs"],"created_at":"2024-11-13T13:06:05.349Z","updated_at":"2026-01-05T10:03:14.080Z","avatar_url":"https://github.com/beam-pyio.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sqs_pyio\n\n![doc](https://github.com/beam-pyio/sqs_pyio/workflows/doc/badge.svg)\n![test](https://github.com/beam-pyio/sqs_pyio/workflows/test/badge.svg)\n[![release](https://img.shields.io/github/release/beam-pyio/sqs_pyio.svg)](https://github.com/beam-pyio/sqs_pyio/releases)\n![pypi](https://img.shields.io/pypi/v/sqs_pyio)\n![python](https://img.shields.io/pypi/pyversions/sqs_pyio)\n\n[Amazon Simple Queue Service (Amazon SQS)](https://aws.amazon.com/sqs/) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. The Apache Beam Python I/O connector for Amazon SQS (`sqs_pyio`) aims to integrate with the queue service by supporting a source and sink connectors. Currently, a sink connector is available.\n\n## Installation\n\nThe connector can be installed from PyPI.\n\n```bash\npip install sqs_pyio\n```\n\n## Usage\n\n### Sink Connector\n\nIt has the main composite transform ([`WriteToSqs`](https://beam-pyio.github.io/sqs_pyio/autoapi/sqs_pyio/io/index.html#sqs_pyio.io.WriteToSqs)), and it expects a list or tuple _PCollection_ element. If the element is a tuple, the tuple's first element is taken. If the element is not of the accepted types, you can apply the [`GroupIntoBatches`](https://beam.apache.org/documentation/transforms/python/aggregation/groupintobatches/) or [`BatchElements`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.BatchElements) transform beforehand. Then, the records of the element are sent into a SQS queue using the [`send_message_batch`](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs/client/send_message_batch.html) method of the boto3 package. Note that the above batch transforms can also be useful to overcome the API limitation listed below.\n\n- Each `SendMessageBatch` request supports up to 10 messages. The maximum allowed individual message size and the maximum total payload size (the sum of the individual lengths of all the batched messages) are both 256 KiB (262,144 bytes).\n\nThe transform also has options that handle failed records as listed below.\n\n- _max_trials_ - The maximum number of trials when there is one or more failed records - it defaults to 3. Note that failed records after all trials are returned by a tagged output, which allows users to determine how to handle them subsequently.\n- _append_error_ - Whether to append error details to failed records. Defaults to True.\n\nAs mentioned earlier, failed elements are returned by a tagged output where it is named as `write-to-sqs-failed-output` by default. You can change the name by specifying a different name using the `failed_output` argument.\n\n#### Sink Connector Example\n\nIf a _PCollection_ element is key-value pair (i.e. keyed stream), it can be batched in group using the `GroupIntoBatches` transform before it is connected into the main transform.\n\n```python\nimport apache_beam as beam\nfrom apache_beam import GroupIntoBatches\nfrom sqs_pyio.io import WriteToSqs\n\nrecords = [(i % 2, {\"Id\": str(i), \"MessageBody\": str(i)}) for i in range(3)]\n\nwith beam.Pipeline() as p:\n    (\n        p\n        | beam.Create(records)\n        | GroupIntoBatches(batch_size=2)\n        | WriteToSqs(queue_name=self.queue_name)\n    )\n```\n\nFor a list element (i.e. unkeyed stream), we can apply the `BatchElements` transform instead.\n\n```python\nimport apache_beam as beam\nfrom apache_beam.transforms.util import BatchElements\nfrom sqs_pyio.io import WriteToSqs\n\nrecords = [{\"Id\": str(i), \"MessageBody\": str(i)} for i in range(3)]\n\nwith beam.Pipeline() as p:\n    (\n        p\n        | beam.Create(records)\n        | BatchElements(min_batch_size=2, max_batch_size=2)\n        | WriteToSqs(queue_name=self.queue_name)\n    )\n```\n\nSee [Introduction to SQS PyIO Sink Connector](/blog/2024/sqs-pyio-intro/) for more examples.\n\n## Contributing\n\nInterested in contributing? Check out the contributing guidelines. Please note that this project is released with a Code of Conduct. By contributing to this project, you agree to abide by its terms.\n\n## License\n\n`sqs_pyio` was created as part of the [Apache Beam Python I/O Connectors](https://github.com/beam-pyio) project. It is licensed under the terms of the Apache License 2.0 license.\n\n## Credits\n\n`sqs_pyio` was created with [`cookiecutter`](https://cookiecutter.readthedocs.io/en/latest/) and the `pyio-cookiecutter` [template](https://github.com/beam-pyio/pyio-cookiecutter).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbeam-pyio%2Fsqs_pyio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbeam-pyio%2Fsqs_pyio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbeam-pyio%2Fsqs_pyio/lists"}