{"id":18414786,"url":"https://github.com/navdeep-g/sparkling-water-emr","last_synced_at":"2025-04-13T00:58:43.711Z","repository":{"id":145427620,"uuid":"59526345","full_name":"navdeep-G/sparkling-water-emr","owner":"navdeep-G","description":"Launch Sparkling Water on EMR","archived":false,"fork":false,"pushed_at":"2022-10-08T00:04:17.000Z","size":3,"stargazers_count":5,"open_issues_count":0,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-13T00:58:39.164Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/navdeep-G.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-05-23T23:54:21.000Z","updated_at":"2022-10-08T00:04:21.000Z","dependencies_parsed_at":"2023-09-13T05:16:28.987Z","dependency_job_id":null,"html_url":"https://github.com/navdeep-G/sparkling-water-emr","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/navdeep-G%2Fsparkling-water-emr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/navdeep-G%2Fsparkling-water-emr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/navdeep-G%2Fsparkling-water-emr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/navdeep-G%2Fsparkling-water-emr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/navdeep-G","download_url":"https://codeload.github.com/navdeep-G/sparkling-water-emr/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248650435,"owners_count":21139672,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T03:52:18.586Z","updated_at":"2025-04-13T00:58:43.692Z","avatar_url":"https://github.com/navdeep-G.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sparkling-Water-EMR \n\nBelow are steps to set up [AWS EMR](https://aws.amazon.com/elasticmapreduce/) with [Sparkling Water](https://github.com/h2oai/sparkling-water)\n\n# Setting up your EMR Cluster:\n\n## Step 1\n  - Set up AWS credentials\n    - Go to [https://aws.amazon.com/](https://aws.amazon.com/) and click the link `Create an AWS Account`\n\n## Step 2\n  - Sign into your AWS Console and navigate to `EMR` under `Analytics`\n\n## Step 3\n  - Configure your EMR instance as follows:\n    - Cluster name: 'MyCluster'\n    - Leave `Logging` checked\n    - Leave `S3 folder` as its default (can change if ou have a specific folder in mind)\n    - Launch Mode should be set to `Cluster`\n    - Vendor should be set to `Amazon`\n    - Release should be set to latest\n    - Applications should be `Spark: Spark 1.6.1 on Hadoop 2.7.2 YARN wih Ganglia 3.7.2`\n    - Choose `Instance Type` and `Number of Instances` as you see fit\n    - `Security and access` should be left as default, but choose an appropriate `EC2 key pair` (made when you set up an AWS account)\n    - Click `Create Cluster`\n\n## Now you have an AWS EMR cluster ready to go!\n\n# Launch Sparkling Water from your new EMR cluster:\n\n  - SSH into your new EMR cluster as follows:\n    -  Navigate to `Master public DNS` and click on `SSH`, which will give you instructions on getting into the EMR master node\n  - After you ssh into your cluster run the following to start up Sparkling Water (in Spark Shell):\n      ```\n      spark-shell --packages ai.h2o:sparkling-water-examples_2.10:1.6.3 --executor-memory=6g --driver-memory=6g\n      ```\n  - Now you should have a spark shell running. Next, run the following commands to set up Sparkling Water:\n      ```\n      import org.apache.spark.h2o._\n      val hc = H2OContext.getOrCreate(sc)\n      ```\n  - Now you should see two URL's that point to H2O's flow interface. Please point to those in your browser to get access.\n  \n## Now you have an AWS EMR cluster running Sparkling Water!\n\n  - Sparkling Water demos are also available here: [https://github.com/h2oai/sparkling-water/tree/master/examples](https://github.com/h2oai/sparkling-water/tree/master/examples)\n  - To run these demos you must copy them into your master EMR node.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnavdeep-g%2Fsparkling-water-emr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnavdeep-g%2Fsparkling-water-emr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnavdeep-g%2Fsparkling-water-emr/lists"}