https://github.com/jaehyeon-kim/emr-local-dev
Spark Local Development Environment Using Docker (and vscode)
https://github.com/jaehyeon-kim/emr-local-dev
aws docker emr spark vscode
Last synced: over 1 year ago
JSON representation
Spark Local Development Environment Using Docker (and vscode)
- Host: GitHub
- URL: https://github.com/jaehyeon-kim/emr-local-dev
- Owner: jaehyeon-kim
- Created: 2022-03-30T22:30:49.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2023-02-28T17:57:06.000Z (over 3 years ago)
- Last Synced: 2025-03-22T04:51:17.123Z (over 1 year ago)
- Topics: aws, docker, emr, spark, vscode
- Language: Python
- Homepage: https://cevo.com.au/post/develop-and-test-apache-spark-apps-for-emr-locally-using-docker/
- Size: 26.4 KB
- Stars: 7
- Watchers: 2
- Forks: 3
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## Spark Local Development Environment Using Docker
[Develop and Test Apache Spark Apps for EMR Locally Using Docker](https://cevo.com.au/post/develop-and-test-apache-spark-apps-for-emr-locally-using-docker/)
- [Amazon EMR](https://aws.amazon.com/emr/) is a managed service that simplifies running Apache Spark on AWS. It has multiple deployment options that cover EC2, [EKS](https://aws.amazon.com/emr/features/eks/), [Outposts](https://aws.amazon.com/emr/features/outposts/) and [Serverless](https://aws.amazon.com/emr/serverless/). For development and testing, [EMR Notebooks](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-notebooks.html) or [EMR Studio](https://aws.amazon.com/emr/features/studio/) can be an option. Both provide a Jupyter Notebook environment and the former is only available for EMR on EC2. There are cases, however, that development (and learning) is performed in a local environment more efficiently. The AWS Glue team understands this demand and they illustrate how to make use of a custom Docker image for Glue in a [recent blog post](https://aws.amazon.com/blogs/big-data/develop-and-test-aws-glue-version-3-0-jobs-locally-using-a-docker-container/). However we don’t hear similar news from the EMR team. In order to fill the gap, we’ll discuss how to create a Spark local development environment for EMR using Docker and/or VSCode. Typical Spark development examples will be demonstrated, which covers Spark Submit, pytest, PySpark shell, Jupyter Notebook and Spark Structured Streaming. For the Spark Submit and Jupyter Notebook examples, [Glue Catalog integration](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-glue.html) will be illustrated as well. And both the cases of utilising [Visual Studio Code Remote - Containers](https://code.visualstudio.com/docs/remote/containers) extension and running as an isolated container will be covered in some key examples.