{"id":20018577,"url":"https://github.com/rupeshtr78/aws-emr","last_synced_at":"2026-04-18T15:40:36.455Z","repository":{"id":156959821,"uuid":"244028706","full_name":"rupeshtr78/aws-emr","owner":"rupeshtr78","description":"Spark Job on Amazon EMR cluster","archived":false,"fork":false,"pushed_at":"2020-02-29T19:56:48.000Z","size":1362,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-08T23:42:21.029Z","etag":null,"topics":["aws","cluster","emr-cluster","mapreduce","mapredue","scala","spark"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rupeshtr78.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-29T19:27:12.000Z","updated_at":"2022-02-09T23:48:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"a0e7d791-d68c-4cf0-b18e-60f6fc78add9","html_url":"https://github.com/rupeshtr78/aws-emr","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rupeshtr78/aws-emr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rupeshtr78%2Faws-emr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rupeshtr78%2Faws-emr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rupeshtr78%2Faws-emr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rupeshtr78%2Faws-emr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rupeshtr78","download_url":"https://codeload.github.com/rupeshtr78/aws-emr/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rupeshtr78%2Faws-emr/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31974952,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T00:39:45.007Z","status":"online","status_checked_at":"2026-04-18T02:00:07.018Z","response_time":103,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cluster","emr-cluster","mapreduce","mapredue","scala","spark"],"created_at":"2024-11-13T08:23:16.544Z","updated_at":"2026-04-18T15:40:36.426Z","avatar_url":"https://github.com/rupeshtr78.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS EMR Spark \r\n\r\n**Steps for Deploying Spark App on Amazon EMR**\r\n\r\n- **Step 1** \r\n\r\n  Test your application using Scala - ide using sample data.\r\n\r\n  ![](data/scala-ide01.png)\r\n\r\n  \r\n\r\n- **Step 2**\r\n\r\n  - Remove all local path and Spark Context master local reference from Scala file.\r\n\r\n  - Use SBT to package your application\r\n\r\n    - Create and empty directory **sbt**\r\n    - **sbt new scala/hello-world.g8.** \r\n    - add your scale files under sbt\\movies\\src\\main\\scala directory\r\n    - edit sbt\\movies\\built.sbt\r\n\r\n  - ```yaml\r\n    name := \"MostRatedMovies100k\"\r\n    \r\n    version := \"1.0\"\r\n    \r\n    organization := \"com.forsynet.sparkemr\"\r\n    \r\n    scalaVersion := \"2.11.12\"\r\n    \r\n    libraryDependencies ++= Seq(\r\n    \"org.apache.spark\" %% \"spark-core\" % \"2.4.5\" % \"provided\"\r\n    )\r\n    ```\r\n\r\n    \r\n\r\n  - at command prompt run command \"sbt assembly\"\r\n\r\n  - This will create jar files with all dependencies under sbt\\movies\\target\\scala-2.11\\MostRatedMovies100k-assembly-1.0.jar\r\n\r\n- **Step 3**\r\n\r\n  - upload the jar files and the data files to s3 bucket .Use UI or below cli commands\r\n\r\n  - ```powershell\r\n    aws configure\r\n    aws s3api create-bucket --bucket rupeshemr\r\n    aws s3 sync data/\r\n    \r\n    ```\r\n\r\n    - Verify the data is uploaded to s3 bucket\r\n    - ![](data/s3-upload-files.png)\r\n\r\n\r\n\r\n## Create an Amazon EMR cluster\r\n\r\n- **Step 4**\r\n\r\n- Use the aws cli or the UI to create the cluster.\r\n\r\n- ```shell\r\n  aws emr create-cluster \\\r\n      --instance-type m3.xlarge \\\r\n      --release-label emr-5.10.0 \\\r\n      --service-role EMR_DefaultRole \\\r\n      --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \\\r\n      --security-configuration mySecurityConfiguration \\\r\n      --kerberos-attributes file://kerberos_attributes.json\r\n  ```\r\n\r\n  - Verify Cluster Creation\r\n  - ![](data/aws-emr01.PNG)\r\n\r\n  - **Step5**\r\n\r\n  - Add SSH Inbound rule to security groups\r\n  - ![](data/aws-emr-securitygrp.PNG)\r\n\r\n- **Step 6**\r\n- ssh into emr master node\r\n- ![](data/putty-emr-login.PNG)\r\n\r\n- copy the jar file from s3 bucker \r\n\r\n- ```\r\n  aws s3 cp s3://rupeshemr/MostRatedMovies-1.0.jar ./\r\n  ```\r\n\r\n  ![](data/aws-s3-cp-jar.png)\r\n\r\n\r\n\r\n## Submit the Spark Job\r\n\r\n- **Step 7**\r\n\r\n  - ```sh\r\n    spark-submit MostRatedMovies-1.0.jar\r\n    ```\r\n\r\n- Verify Results of top rated movies\r\n  - ![](data/spark-submit-results.PNG)\r\n\r\n- Use the Spark History Server UI to see the Spark Job History for submitted job\r\n- ![](data/spark-historyui01.PNG)\r\n\r\n![](data/spark-historyui01-step1.PNG)\r\n\r\n- Verify Amazon s3 bucket for logs created for the job\r\n\r\n![](data/s3-emr-output.PNG)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frupeshtr78%2Faws-emr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frupeshtr78%2Faws-emr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frupeshtr78%2Faws-emr/lists"}