{"id":20004825,"url":"https://github.com/us8945/aws_emr_pysparkling","last_synced_at":"2026-04-30T16:31:05.017Z","repository":{"id":91102613,"uuid":"187952687","full_name":"us8945/AWS_EMR_Pysparkling","owner":"us8945","description":"Set Up Python environment on AWS EMR cluster with H2O Sparkling Water (Pysparling)","archived":false,"fork":false,"pushed_at":"2019-05-23T00:55:30.000Z","size":10,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-30T00:04:15.331Z","etag":null,"topics":["aws","emr","h2o","jupyter-notebook","pyspark","pysparkling","spark","sparkling-water"],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/us8945.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-22T03:00:40.000Z","updated_at":"2022-06-05T16:45:24.000Z","dependencies_parsed_at":"2023-07-09T13:01:37.702Z","dependency_job_id":null,"html_url":"https://github.com/us8945/AWS_EMR_Pysparkling","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/us8945/AWS_EMR_Pysparkling","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/us8945%2FAWS_EMR_Pysparkling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/us8945%2FAWS_EMR_Pysparkling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/us8945%2FAWS_EMR_Pysparkling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/us8945%2FAWS_EMR_Pysparkling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/us8945","download_url":"https://codeload.github.com/us8945/AWS_EMR_Pysparkling/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/us8945%2FAWS_EMR_Pysparkling/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32470879,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"ssl_error","status_checked_at":"2026-04-30T13:12:06.837Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","emr","h2o","jupyter-notebook","pyspark","pysparkling","spark","sparkling-water"],"created_at":"2024-11-13T05:36:44.965Z","updated_at":"2026-04-30T16:31:05.002Z","avatar_url":"https://github.com/us8945.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Installing H2O Sparkling Water on EMR - Python (Jupyter) environment\r\n\r\n\r\n\r\n\r\nTable of Contents\r\n=================\r\n\r\n   * [Installing H2O Sparkling Water on EMR](#installing-h2o-sparkling-water-on-emr)\r\n      * [Prerequisites](#prerequisites)\r\n      * [Guide Purpose](#guide-purpose)\r\n      * [Create and login to EMR Cluster](#create-and-login-to-emr-cluster)\r\n      * [Installation Steps](#installation-steps)\r\n         * [Step 1 - Install Python Anaconda Distribution](#step-1---install-python-anaconda-distribution)\r\n         * [Step 2 - Install and configure Jupyter notebook/lab environment](#step-2---install-and-configure-jupyter-notebooklab-environment)\r\n         * [Step 3 - Download and install H2O Sparkling Water](#step-3---download-and-install-h2o-sparkling-water)\r\n         * [Step 4 - Jupyter notebook](#step-4---jupyter-notebook)\r\n         * [Step 5 (Optional) - Start H2O in Scala Shell](#step-5-optional---start-h2o-in-scala-shell)\r\n\r\nCreated by [gh-md-toc](https://github.com/ekalinin/github-markdown-toc)\r\n\r\n\r\n\r\n\r\n## Prerequisites\r\n\r\nThis guide assumes Spark 2.4 is already provisioned with the EMR cluster. All tests have been performed using following SW versions:\r\n\r\n1. EMR 5.23.0\r\n2. Spark 2.4\r\n3. H2O Sparkling Water 2.4.11\r\n4. Anaconda 5.2.0 (Python 3.6.5)\r\n\r\nYou should be able to use the scripts and procedure unmodified as long as your EMR cluster has Spark 2.4 . You will need to make changes to the deployment script is the SW versions above change.\r\n\r\n## Guide Purpose\r\n\r\nThis guide provides set of instructions to set up Python based environment to run H2O Sparling Water on AWS EMR (with Spark) cluster.\r\n\r\nThe Python run-time environment set up is automated and is based on Anaconda Python distribution. Once complete, the steps will provide following configuration:\r\n\r\n1. Latest Anaconda Python 3.6 distribution. \r\n2. Jupyter Notebook/Lab set up and running under separate OS user\r\n3. H2O Sparkling Water 2.4.11\r\n4. Example of starter Notebook to demonstrate creating:\r\n   1. Spark Session to connect to YARN cluser\r\n   2. H2O context running side by side with Spark session\r\n\r\n\r\n\r\n## Create and login to EMR Cluster\r\n\r\nThis is guide doesn't cover steps to provision EMR cluster. You could find many guides helping you to provision your first EMR cluster. When provisioning EMR cluster make sure you read \"Prerequisites section\" first. \r\n\r\n\r\n\r\n## Installation Steps \r\n\r\nAll steps listed below are fully automated, and are part of the [aws_python_tools_install.py](https://github.com/us8945/AWS_EMR_Pysparkling/blob/master/set_up_scripts/aws_python_tools_install.py \"Auto-install File\") installation file.\r\n\r\n\r\n\r\n### Step 1 - Install Python Anaconda Distribution\r\n\r\nIn this step we download and install Python Anaconda distribution. The distribution provides handy set of majority of the Python libraries. Many of the big organizations certify Anaconda distribution for use on its company servers. \r\n\r\n### Step 2 - Install and configure Jupyter notebook/lab environment \r\n\r\nThe goal of this step is to perform all necessary steps to create Jupyter notebook environment.  \r\n\r\n1. Create OS user `jupyterlab`\r\n\r\n2. The password for the user is created using random password generator. If you need to know the password, you will need to reset it using `sudo` access from `hadoop` OS user.\r\n\r\n3. Define enviornment variables and place them into `.bash_profile`:\r\n\r\n ```\r\n   # .bash_profile\r\n   # Get the aliases and functions\r\n   if [ -f ~/.bashrc ]; then\r\n           . ~/.bashrc\r\n   fi\r\n   # User specific environment and startup programs\r\n   PATH=$PATH:$HOME/.local/bin:$HOME/bin\r\n   export PATH\r\n   export PATH=/mnt/opt/anaconda520:/mnt/opt/anaconda520/bin:$PATH\r\n   export PYSPARK_PYTHON=/mnt/opt/anaconda520/bin/python\r\n   export SPARK_HOME=/usr/lib/spark\r\n   export HADOOP_CONF_DIR=/etc/hadoop/conf\r\n   export MASTER=\"yarn\"\r\n ```\r\n\r\n6. Initial token is generated when notebook starts for the first time. The token can be found by running following command: `sudo cat /home/jupyterlab/notebooks/jupyter_notebook.log | grep token`\r\n\r\n7. The notebook is running using port `28888` . You can modify the port after restart. Following command is used to start the notebook:\r\n``` \r\nsudo su - jupyterlab -c 'nohup jupyter notebook --no-browser --port 28888 \u003e\u003e /home/jupyterlab/notebooks/jupyter_notebook.log 2\u003e\u00261 \u0026'\r\n```\r\n\r\n\r\n\r\n### Step 3 - Download and install H2O Sparkling Water\r\n\r\nFollowing steps are automated and are part of the script.\r\n\r\n```\r\nmkdir /mnt/opt\r\ncd /mnt/opt\r\nwget https://s3.amazonaws.com/h2o-release/sparkling-water/rel-2.4/11/sparkling-water-2.4.11.zip\r\nunzip sparkling-water-2.4.11.zip\r\n```\r\n\r\n\r\n\r\n### Step 4 - Jupyter notebook\r\n\r\nThe following steps are performed to validate working environment and ability to create Spark Session and H2O context from Jupyter notebook.\r\n\r\n1. In this step we assume you have already opened port 28888 or using Browser pluging to route Web traffic to/from EMR\r\n\r\n2. Connect to Jupyter notebook using EMR host name and port 28888. Optionally you can specify token all in one command. To find the info:\r\n\r\n```\r\nsudo cat /home/jupyterlab/notebooks/jupyter_notebook.log | grep\r\n```\r\nExample URL: http://ip-172-31-46-145:28888/tree?token=eede425006d001d6bdeb9d8afb2c910e87f3a8137ab38516\r\n\r\n4. Upload provided notebook `H2O Sparkling Water EMR starter.ipynb` and repeat the steps. The notebook can be found under the `notebooks` folder of this Github repo.\r\n5. The Starter notebook can be found here [H2O Sparkling Water EMR starter.ipynb](https://github.com/us8945/AWS_EMR_Pysparkling/blob/master/notebooks/H2O%20Sparkling%20Water%20EMR%20starter.ipynb \"Starter Notebook\")\r\n\r\n\r\n### Step 5 (Optional) - Start H2O in Scala Shell\r\n\r\nYou can validate H2O Sparkling Water installation by running below commands and  `scala` shell.\r\n\r\n\r\n\r\n```\r\nexport SPARK_HOME=/usr/lib/spark\r\nexport HADOOP_CONF_DIR=/etc/hadoop/conf\r\nexport MASTER=\"yarn\"\r\n   \r\nbin/sparkling-shell \\\r\n--master yarn \\\r\n--conf spark.scheduler.maxRegisteredResourcesWaitingTime=1000000 \\\r\n--conf spark.ext.h2o.fail.on.unsupported.spark.param=false \\\r\n--conf spark.dynamicAllocation.enabled=false \\\r\n--conf spark.sql.autoBroadcastJoinThreshold=-1 \\\r\n--conf spark.locality.wait=30000 \\\r\n--conf spark.scheduler.minRegisteredResourcesRatio=1 \\\r\n--conf spark.executor.instances=4 \\\r\n--conf spark.executor.memory=2g \\\r\n--conf spark.driver.memory=2g \r\n```\r\n\r\n\r\n\r\nThe parameter `spark.ext.h2o.fail.on.unsupported.spark.param` is set because otherwise following error is raised on EMR: \r\n\r\n```\r\njava.lang.IllegalArgumentException: Unsupported argument: (spark.dynamicAllocation.enabled,true)\r\n```\r\n\r\n\r\n\r\nStart H2O sessions in Scala shell\r\n\r\n```\r\nimport org.apache.spark.h2o._\r\nval h2oContext = H2OContext.getOrCreate(spark)\r\nimport h2oContext._\r\n```\r\n\r\n\r\n\r\nIf successful following message is displayed:\r\n\r\n```\r\nscala\u003e h2oContext\r\nres0: org.apache.spark.h2o.H2OContext =\r\n\r\nSparkling Water Context:\r\n * H2O name: sparkling-water-hadoop_application_1558311857271_0004\r\n * cluster size: 3\r\n * list of used nodes:\r\n  (executorId, host, port)\r\n  ------------------------\r\n  (2,ip-172-31-12-99.ec2.internal,54321)\r\n  (1,ip-172-31-10-203.ec2.internal,54321)\r\n  (3,ip-172-31-10-203.ec2.internal,54323)\r\n  ------------------------\r\n\r\n  Open H2O Flow in browser: http://ip-172-31-14-136.ec2.internal:54321 (CMD + click in Mac OSX)\r\n\r\n\r\n * Yarn App ID of Spark application: application_1558311857271_0004\r\n```\r\n\r\n\r\n\r\nCreate sample DataFrame and validate from H2O UI:\r\n\r\n```\r\nval someDF = Seq(\r\n  (8, \"bat\"),\r\n  (64, \"mouse\"),\r\n  (-27, \"horse\")\r\n).toDF(\"number\", \"word\")\r\n\r\nsomeDF.show()\r\n+------+-----+\r\n|number| word|\r\n+------+-----+\r\n|     8|  bat|\r\n|    64|mouse|\r\n|   -27|horse|\r\n+------+-----+\r\n\r\n\r\nscala\u003e val hfNamed: H2OFrame = h2oContext.asH2OFrame(someDF, Some(\"h2oframe\"))\r\nhfNamed: org.apache.spark.h2o.H2OFrame =\r\nFrame key: h2oframe\r\n   cols: 2\r\n   rows: 3\r\n chunks: 3\r\n   size: 487\r\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fus8945%2Faws_emr_pysparkling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fus8945%2Faws_emr_pysparkling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fus8945%2Faws_emr_pysparkling/lists"}