https://github.com/dmschauer/aws-glue-local-dev-and-test
Code for a blog article. Mainly copied from https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html#develop-local-python
https://github.com/dmschauer/aws-glue-local-dev-and-test
Last synced: about 2 months ago
JSON representation
Code for a blog article. Mainly copied from https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html#develop-local-python
- Host: GitHub
- URL: https://github.com/dmschauer/aws-glue-local-dev-and-test
- Owner: dmschauer
- Created: 2022-12-18T14:36:49.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2022-12-23T20:40:48.000Z (over 3 years ago)
- Last Synced: 2025-02-25T10:34:33.854Z (over 1 year ago)
- Language: Python
- Size: 5.86 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# What is this?
This repository contains example code to be used with an article I wrote.
Here you can find the article: https://medium.com/@dominikschauer/professional-aws-glue-pyspark-development-local-development-and-unit-tests-190be8ed75fb
# What does it do?
This repository helps you to setup a local development environment for AWS Glue PySpark jobs.
It works for both Windows and Ubuntu/Mac OS.
It contains:
- a template for an AWS Glue PySpark job
- a template for a corresponding pytest unit test
For more details, please read the article. It explains step by step what to do to start developing on your local machine (i.e. without AWS Glue charges!).
# Commands
- start tests from project root `python3 -m pytest ./tests/`
# Sources
- The code is mainly copied from the official AWS Glue documentation: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html#develop-local-python