{"id":18013858,"url":"https://github.com/jaehyeon-kim/emr-local-dev","last_synced_at":"2025-03-26T17:32:38.639Z","repository":{"id":101142520,"uuid":"476071453","full_name":"jaehyeon-kim/emr-local-dev","owner":"jaehyeon-kim","description":"Spark Local Development Environment Using Docker (and vscode)","archived":false,"fork":false,"pushed_at":"2023-02-28T17:57:06.000Z","size":27,"stargazers_count":7,"open_issues_count":1,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-22T04:51:17.123Z","etag":null,"topics":["aws","docker","emr","spark","vscode"],"latest_commit_sha":null,"homepage":"https://cevo.com.au/post/develop-and-test-apache-spark-apps-for-emr-locally-using-docker/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jaehyeon-kim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-30T22:30:49.000Z","updated_at":"2024-12-13T02:00:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"9e0ad9e6-58be-4e77-a5ef-9794bdb11630","html_url":"https://github.com/jaehyeon-kim/emr-local-dev","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaehyeon-kim%2Femr-local-dev","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaehyeon-kim%2Femr-local-dev/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaehyeon-kim%2Femr-local-dev/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaehyeon-kim%2Femr-local-dev/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jaehyeon-kim","download_url":"https://codeload.github.com/jaehyeon-kim/emr-local-dev/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245702608,"owners_count":20658646,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","docker","emr","spark","vscode"],"created_at":"2024-10-30T03:41:59.867Z","updated_at":"2025-03-26T17:32:38.634Z","avatar_url":"https://github.com/jaehyeon-kim.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Spark Local Development Environment Using Docker\n\n[Develop and Test Apache Spark Apps for EMR Locally Using Docker](https://cevo.com.au/post/develop-and-test-apache-spark-apps-for-emr-locally-using-docker/)\n\n- [Amazon EMR](https://aws.amazon.com/emr/) is a managed service that simplifies running Apache Spark on AWS. It has multiple deployment options that cover EC2, [EKS](https://aws.amazon.com/emr/features/eks/), [Outposts](https://aws.amazon.com/emr/features/outposts/) and [Serverless](https://aws.amazon.com/emr/serverless/). For development and testing, [EMR Notebooks](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-notebooks.html) or [EMR Studio](https://aws.amazon.com/emr/features/studio/) can be an option. Both provide a Jupyter Notebook environment and the former is only available for EMR on EC2. There are cases, however, that development (and learning) is performed in a local environment more efficiently. The AWS Glue team understands this demand and they illustrate how to make use of a custom Docker image for Glue in a [recent blog post](https://aws.amazon.com/blogs/big-data/develop-and-test-aws-glue-version-3-0-jobs-locally-using-a-docker-container/). However we don’t hear similar news from the EMR team. In order to fill the gap, we’ll discuss how to create a Spark local development environment for EMR using Docker and/or VSCode. Typical Spark development examples will be demonstrated, which covers Spark Submit, pytest, PySpark shell, Jupyter Notebook and Spark Structured Streaming. For the Spark Submit and Jupyter Notebook examples, [Glue Catalog integration](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-glue.html) will be illustrated as well. And both the cases of utilising [Visual Studio Code Remote - Containers](https://code.visualstudio.com/docs/remote/containers) extension and running as an isolated container will be covered in some key examples.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaehyeon-kim%2Femr-local-dev","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaehyeon-kim%2Femr-local-dev","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaehyeon-kim%2Femr-local-dev/lists"}