{"id":27992833,"url":"https://github.com/fabiogouw/spark-aws-messaging","last_synced_at":"2025-05-08T18:43:11.456Z","repository":{"id":47714550,"uuid":"354409420","full_name":"fabiogouw/spark-aws-messaging","owner":"fabiogouw","description":"A custom sink provider for Apache Spark that sends the content of a dataframe to an AWS SQS","archived":false,"fork":false,"pushed_at":"2024-06-15T13:42:01.000Z","size":902,"stargazers_count":19,"open_issues_count":0,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-11-15T12:13:21.244Z","etag":null,"topics":["aws-sqs","spark","spark-sql"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fabiogouw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-03T22:45:23.000Z","updated_at":"2024-10-21T18:12:38.000Z","dependencies_parsed_at":"2024-05-23T02:38:49.745Z","dependency_job_id":"74f91853-d921-40a4-bb68-56cb29a80896","html_url":"https://github.com/fabiogouw/spark-aws-messaging","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabiogouw%2Fspark-aws-messaging","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabiogouw%2Fspark-aws-messaging/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabiogouw%2Fspark-aws-messaging/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabiogouw%2Fspark-aws-messaging/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fabiogouw","download_url":"https://codeload.github.com/fabiogouw/spark-aws-messaging/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253131792,"owners_count":21859071,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-sqs","spark","spark-sql"],"created_at":"2025-05-08T18:43:10.982Z","updated_at":"2025-05-08T18:43:11.448Z","avatar_url":"https://github.com/fabiogouw.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark - AWS SQS Sink\nA custom sink provider for Apache Spark that sends the contents of a dataframe to AWS SQS.\n\nIt grabs the content of the first column of the dataframe and sends it to an AWS SQS queue. It needs the following parameters:\n- **region** of the queue (us-east-2, for instance)\n- **name** of the queue\n- **batch size** so we can group N messages in one call\n- **queueOwnerAWSAccountId** you might have an architecture where the Spark job and the SQS are in different AWS accounts. In that case, you can specify one extra option to make the writer aware of which account to use. This is an optional argument.\n\n```java\ndf.write()\n        .format(\"sqs\")\n        .mode(SaveMode.Append)\n        .option(\"region\", \"us-east-2\")\n        .option(\"queueName\", \"my-test-queue\")\n        .option(\"batchSize\", \"10\")\n        .option(\"queueOwnerAWSAccountId\", \"123456789012\") // optional\n        .save();\n```\n\nThe dataframe:\n- must have a column called **value** (string), because this column will be used as the body of each message.\n- may have a column called **msg_attributes** (map of [string, string]). In this case, the library will add each key/value as a [metadata attribute](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html) to the SQS message.\n- may have a column called **group_id** (string). In this case, the library will add the group id used by [FIFO queues](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagegroupid-property.html).\n\nThe folder [/spark-aws-messaging/src/test/resources](/spark-aws-messaging/src/test/resources) contains some PySpark simple examples used in the integration tests (the *endpoint* option is not required).\n\nDon't forget you'll need to configure the default credentials in your machine before running the example. See\n[Configuration and credential file settings](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) for more information.\n\nIt also needs the *com.amazonaws:aws-java-sdk-sqs* package to run, so you can provide it through the *packages* parameter of spark-submit.\n\nThe following command can be used to run the sample of how to use this library.\n\n``` bash\nspark-submit \\\n--packages com.fabiogouw:spark-aws-messaging:1.1.0,com.amazonaws:aws-java-sdk-sqs:1.12.13 \\\ntest.py sample.txt\n```\n\nAnd this is the test.py file content.\n\n``` python\nimport sys \nfrom operator import add\n\nfrom pyspark.sql import SparkSession\n\nif __name__ == \"__main__\":\n    print(\"File: \" + sys.argv[1])\n\n    spark = SparkSession\\\n        .builder\\\n        .appName(\"SQS Write\")\\\n        .getOrCreate()\n\n    df = spark.read.text(sys.argv[1])\n    df.show()\n    df.printSchema()\n\n    df.write.format(\"sqs\").mode(\"append\")\\\n        .option(\"queueName\", \"test\")\\\n        .option(\"batchSize\", \"10\") \\\n        .option(\"region\", \"us-east-2\") \\\n        .save()\n\n    spark.stop()\n```\nThis library is available at Maven Central repository, so you can reference it in your project with the following snippet.\n\n``` xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.fabiogouw\u003c/groupId\u003e\n    \u003cartifactId\u003espark-aws-messaging\u003c/artifactId\u003e\n    \u003cversion\u003e1.1.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nThe IAM permissions needed for this library to write on a SQS queue are *sqs:GetQueueUrl* and *sqs:SendMessage*.\n\n## Messaging delivery semantics and error handling\n\nThe sink is at least once  so some messages might be duplicated. If something wrong happens when the data is being written by a worker node, Spark will retry the task in another node. Messages that have already been sent could be sent again.\n\n## Architecture\n\nIt's easy to get lost while understanding all the classes are needed, so we can create a custom sink for Spark. Here's a class diagram to make it a little easy to find yourself. Start at SQSSinkProvider, it's the class that we configure in Spark code as a *format* method's value.\n\n![Class diagram showing all the classes needed to implement a custom sink](/doc/assets/Class%20Diagram-Page-1.png \"Class diagram showing all the classes needed to implement a custom sink\")\n\n## How to\n\n- [Use this library with AWS Glue](doc/aws-glue.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabiogouw%2Fspark-aws-messaging","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffabiogouw%2Fspark-aws-messaging","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabiogouw%2Fspark-aws-messaging/lists"}