https://github.com/qubole/s3-sqs-connector

A library for reading data from Amzon S3 with optimised listing using Amazon SQS using Spark SQL Streaming ( or Structured streaming).
https://github.com/qubole/s3-sqs-connector

s3 scala spark spark-streaming sqs streaming structured-streaming

Last synced: about 1 month ago
JSON representation

A library for reading data from Amzon S3 with optimised listing using Amazon SQS using Spark SQL Streaming ( or Structured streaming).

• Host: GitHub
• URL: https://github.com/qubole/s3-sqs-connector
• Owner: qubole
• License: apache-2.0
• Created: 2019-06-12T06:18:07.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2021-04-29T13:43:27.000Z (over 3 years ago)
• Last Synced: 2024-04-17T22:49:43.905Z (9 months ago)
• Topics: s3, scala, spark, spark-streaming, sqs, streaming, structured-streaming
• Language: Scala
• Homepage: http://www.qubole.com
• Size: 41 KB
• Stars: 16
• Watchers: 5
• Forks: 11
• Open Issues: 8
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# S3-SQS Connector

[![Build Status](https://travis-ci.org/qubole/s3-sqs-connector.svg?branch=master)](https://travis-ci.org/qubole/s3-sqs-connector)

A library for reading data from Amzon S3 with optimised listing using Amazon SQS using Spark SQL Streaming ( or Structured streaming.). 

## Linking

Using SBT:

    libraryDependencies += "com.qubole" %% "spark-sql-streaming-sqs_{{site.SCALA_BINARY_VERSION}}" % "{{site.PROJECT_VERSION}}"

Using Maven:

    

        com.qubole

        spark-sql-streaming-sqs_{{site.SCALA_BINARY_VERSION}}

        {{site.PROJECT_VERSION}}

    

This library can also be added to Spark jobs launched through `spark-shell` or `spark-submit` by using the `--packages` command line option.

For example, to include it when starting the spark shell:

    $ bin/spark-shell --packages com.qubole:spark-sql-streaming-sqs_{{site.SCALA_BINARY_VERSION}}:{{site.PROJECT_VERSION}}

Unlike using `--jars`, using `--packages` ensures that this library and its dependencies will be added to the classpath.

The `--packages` argument can also be used with `bin/spark-submit`.

This library is compiled for Scala 2.11 only, and intends to support Spark 2.4.0 onwards.

## Building S3-SQS Connector

S3-SQS Connector is built using Apache Maven](http://maven.apache.org/).

To build S3-SQS connector, clone this repository and run:

```

mvn -DskipTests clean package

```

This will create `target/spark-sql-streaming-sqs_2.11-0.5.1.jar` file which contains s3-sqs connector code and associated dependencies. Make sure the Scala and Java versions correspond to those required by your Spark cluster. We have tested it with Java 7/8, Scala 2.11 and Spark version 2.4.0.

## Configuration options

The configuration is obtained from parameters.

Name |Default | Meaning

--- |:---:| ---

sqsUrl|required, no default value|sqs queue url, like 'https://sqs.us-east-1.amazonaws.com/330183209093/TestQueue'

region|required, no default value|AWS region where queue is created

fileFormat|required, no default value|file format for the s3 files stored on Amazon S3

schema|required, no default value|schema of the data being read 

sqsFetchIntervalSeconds|10|time interval (in seconds) after which to fetch messages from Amazon SQS queue

sqsLongPollingWaitTimeSeconds|20|wait time (in seconds) for long polling on Amazon SQS queue 

sqsMaxConnections|1|number of parallel threads to connect to Amazon SQS queue

sqsMaxRetries|10|Maximum number of consecutive retries in case of a connection failure to SQS before giving up

ignoreFileDeletion|false|whether to ignore any File deleted message in SQS queue

fileNameOnly|false|Whether to check new files based on only the filename instead of on the full path

shouldSortFiles|true|whether to sort files based on timestamp while listing them from SQS

useInstanceProfileCredentials|false|Whether to use EC2 instance profile credentials for connecting to Amazon SQS

maxFilesPerTrigger|no default value|maximum number of files to process in a microbatch

maxFileAge|7d|Maximum age of a file that can be found in this directory

## Example

An example to create a SQL stream which uses Amazon SQS to list files on S3,

        val inputDf = sparkSession

                          .readStream

                          .format("s3-sqs")

                          .schema(schema)

                          .option("sqsUrl", queueUrl)

                          .option("region", awsRegion)

                          .option("fileFormat", "json")

                          .option("sqsFetchIntervalSeconds", "2")

                          .option("useInstanceProfileCredentials", "true")

                          .option("sqsLongPollingWaitTimeSeconds", "5")

                          .load()