https://github.com/jhole89/aws-glue-sbt-quickstart
Example of how to set SBT up for local development of AWS Glue Scripts
https://github.com/jhole89/aws-glue-sbt-quickstart
apache-spark aws-glue quickstart sbt
Last synced: 4 months ago
JSON representation
Example of how to set SBT up for local development of AWS Glue Scripts
- Host: GitHub
- URL: https://github.com/jhole89/aws-glue-sbt-quickstart
- Owner: jhole89
- License: gpl-3.0
- Created: 2020-01-14T22:23:05.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2021-01-04T17:05:40.000Z (over 4 years ago)
- Last Synced: 2024-08-13T07:15:47.063Z (8 months ago)
- Topics: apache-spark, aws-glue, quickstart, sbt
- Language: Scala
- Size: 30.3 KB
- Stars: 16
- Watchers: 2
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- jimsghstars - jhole89/aws-glue-sbt-quickstart - Example of how to set SBT up for local development of AWS Glue Scripts (Scala)
README
# AWS Glue SBT Quickstart
This repo provides a quickstart for new AWS Glue projects using Scala. Using SBT and the AWS Glue SDK,
this repo enables local development and unit testing of AWS Glue scripts.
## Monorepo Pattern
AWS Glue restricts users to submitting a single file to execute a job. These files are then compiled by
AWS during the initial stage of job execution (to ensure that they are compiled using the correct Spark
version as the Spark executors it will be run on). This means that any generic or shared logic used by
multiple scripts would have to be repeated in each file, which is against DRY principles. To correct
this we can abstract any shared logic into a separate module, package it ourselves, and provide it as
an internal dependency to our Glue scripts. To accomplish this we structure the quickstart as a
monorepo containing two modules:
* shared - this contains abstracted shared logic for interfacing with the underlying Glue & Spark
contexts, and any other functionality we may miss to shared cross multiple scripts (e.g. abstract base
classes, traits, dataclasses, cleansers, readers, metrics, reporting, etc). This is made available as a
compiled jar and provided to the Glue job via the `--extra-jars` parameter. The purpose of this jar is to
simplify each script by abstracting complex code patterns into base classes and simple reusable functions.
* scripts - this contains individual Glue scripts, using elements from the shared library. This will
be compiled by AWS Glue prior to execution.
## Getting Started
1. Fork or clone the repo `git clone [email protected]/jhole89/aws-glue-sbt-quickstart.git`
2. In `build.sbt` update the `projectName` and `organization`
3. Run a full sanity check `sbt sanity`
You should now have a running quickstart which you can further develop for your needs. A typical pattern
would be
1. Start writing your Glue scripts & tests
2. Abstract any shared functionality into the shared lib
3. Run a full sanity check `sbt sanity`
4. Upload your Glue scripts (located at `scripts/src/main/scala/scripts/Script.scala`) and shared
library jar (located at `shared/target/scala-*/*-assembly-*.jar`) to AWS S3 buckets
5. Create a AWS Glue Job which points to the script, set the main class using the parameter
`--class scripts.Script`, set the shared dependency using the parameter
`--extra-jars .jar`
## sbt sanity? Thats not a command!
`sbt sanity` is simply an alias used to bind multiple stages of building together. So instead of having
to run `sbt compile`, `sbt test`, `sbt scalafmtAll`, `sbt scalastyle`, and `sbt assembly` separately,
we can just run `sbt sanity` to call each of these sequentially. It's definition is in the `build.sbt`
via `addCommandAlias("sanity", ";clean ;compile ;test ;scalafmtAll ;scalastyle ;assembly")`.
## Extending
Maybe this simple structure doesn't suit all your needs. Maybe you need another module within the
monorepo for something else, or maybe you need multiple shared libs for different business logic? This
is fine and easily done within this framework. If you need other modules, simply declare them in the
`build.sbt` as was done for the shared, and copy over the relevant settings and options. Remember to
link your dependencies together using the `.dependsOn` syntax.