{"id":22178351,"url":"https://github.com/newfront/odsc-east2019-warmup","last_synced_at":"2026-02-12T11:35:31.259Z","repository":{"id":138838419,"uuid":"171183554","full_name":"newfront/odsc-east2019-warmup","owner":"newfront","description":"Warmup Presentation for The 2019 Open Data Science Conference in Boston","archived":false,"fork":false,"pushed_at":"2019-07-24T18:17:35.000Z","size":50944,"stargazers_count":1,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-24T18:49:58.098Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/newfront.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-17T23:16:07.000Z","updated_at":"2019-08-03T17:32:20.000Z","dependencies_parsed_at":"2023-03-13T10:52:49.076Z","dependency_job_id":null,"html_url":"https://github.com/newfront/odsc-east2019-warmup","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/newfront/odsc-east2019-warmup","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/newfront%2Fodsc-east2019-warmup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/newfront%2Fodsc-east2019-warmup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/newfront%2Fodsc-east2019-warmup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/newfront%2Fodsc-east2019-warmup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/newfront","download_url":"https://codeload.github.com/newfront/odsc-east2019-warmup/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/newfront%2Fodsc-east2019-warmup/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272338627,"owners_count":24917003,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-27T02:00:09.397Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-02T08:46:16.236Z","updated_at":"2026-02-12T11:35:31.211Z","avatar_url":"https://github.com/newfront.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"### Real-ish Time Predictive Analytics Warm Up\nThis is the essential prerequist to the May 1st workshop in Boston at the ODSC East Conference.\n\n#### WarmUp DataSet\n[Wine Reviews](https://www.kaggle.com/zynicide/wine-reviews) - Thanks to zynicide and kaggle.com for the data set.\n\n#### Technologies\n1. [Spark 2.4.0](https://www.apache.org/dyn/closer.lua/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz)\n2. [Zeppelin 0.8.1](http://www.apache.org/dyn/closer.cgi/zeppelin/zeppelin-0.8.1/zeppelin-0.8.1-bin-netinst.tgz)\n3. [Hadoop 2.7.7](https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz)\n\n#### Spark Local Setup\nJust download, untar, and move spark. I usually just drop into usr/local\n\n`tar -xvzf /path/to/spark-2.4.0-bin-hadoop2.7.tgz \u0026\u0026 mv /path/to/spark-2.4.0-bin-hadoop2.7/ /usr/local/spark-2.4.0/`\n\n#### Zeppelin Setup\n[Spark Setup](https://zeppelin.apache.org/docs/0.8.1/interpreter/spark.html)\n\n#### Hadoop Single Node Setup\n[Setup Documentation](http://hadoop.apache.org/docs/r2.7.7/hadoop-project-dist/hadoop-common/SingleCluster.html)\n\n#### Local Aliases in .bashrc or .bash_profile\n~~~\nexport SPARK_HOME=/usr/local/spark-2.4.0\nexport ZEPPELIN_HOME=/usr/local/zeppelin-0.8.1\nexport HADOOP_HOME=/usr/local/hadoop-2.7.7\n\n# Zeppelin\nalias zeppelin_start=\"$ZEPPELIN_HOME/bin/zeppelin-daemon.sh --config $ZEPPELIN_HOME/conf/ start\"\nalias zeppelin_stop=\"$ZEPPELIN_HOME/bin/zeppelin-daemon.sh --config $ZEPPELIN_HOME/conf/ stop\"\n\n# Hadoop\nalias start_hdfs=\"$HADOOP_HOME/sbin/start-dfs.sh\"\nalias stop_hdfs=\"$HADOOP_HOME/sbin/stop-dfs.sh\"\nalias hdfs=\"$HADOOP_HOME/bin/hdfs\"\n~~~\n\n```\nsource ~/.bash_profile\n```\n\n#### Zeppelin Config (zeppelin-env.sh)\nYou will need to setup some basic options in the zeppelin-env.sh\n\n`vim /usr/local/zeppelin-0.8.1/conf/zeppelin-env.sh`\n~~~bash\n/usr/local/zeppelin-0.8.1/conf/zeppelin-env.sh\n#### Spark interpreter configuration ####\n\n## Use provided spark installation ##\n## defining SPARK_HOME makes Zeppelin run spark interpreter process using spark-submit\n##\nexport SPARK_HOME=/usr/local/spark-2.4.0        # (required) When it is defined, load it instead of Zeppelin embedded Spark libraries\n# export SPARK_SUBMIT_OPTIONS                   # (optional) extra options to pass to spark submit. eg) \"--driver-memory 512M --executor-memory 1G\".\n# export SPARK_APP_NAME                         # (optional) The name of spark application.\n\n## Spark interpreter options ##\n##\n# export ZEPPELIN_SPARK_USEHIVECONTEXT  # Use HiveContext instead of SQLContext if set true. true by default.\n# export ZEPPELIN_SPARK_CONCURRENTSQL   # Execute multiple SQL concurrently if set true. false by default.\n# export ZEPPELIN_SPARK_IMPORTIMPLICIT  # Import implicits, UDF collection, and sql if set true. true by default.\n# export ZEPPELIN_SPARK_MAXRESULT       # Max number of Spark SQL result to display. 1000 by default.\nexport ZEPPELIN_WEBSOCKET_MAX_TEXT_MESSAGE_SIZE=2048000 # Size in characters of the maximum text message to be received by websocket. Defaults to 1024000\nexport ZEPPELIN_INTERPRETER_OUTPUT_LIMIT=2048000\n~~~\n\n#### Zeppelin Interpreter Spark Settings\n1. in the terminal, if you have added the aliases to your bash, `zeppelin_start` - should emit green `[OK]` when running\n2. go to http://localhost:8080/#/interpreter\n3. under the Spark section, click the edit icon, and add `spark.executor.memory: 6g`, `zeppelin.spark.maxResult: 50000`. It is worth noting that with `spark.cores.max: 4` you will need `24g` of ram to run zeppelin. `spark.cores.max * spark.executor.memory = runtime ram dependency`\n4. click `save` and the interpreter will restart with your updated settings.\n\n[Zeppelin Spark Doc](https://zeppelin.apache.org/docs/0.8.1/interpreter/spark.html)\n\n#### Note to Self\nIt is possible for localhost alias to be broken on your computer due to firewall issues. If `etc/hosts` is pointing localhost to 127.0.0.1 you are golden. Sometimes VPN makes things not work - you can use `127.0.0.1:8080` for the zeppelin UI if localhost bindings are somehow hosed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnewfront%2Fodsc-east2019-warmup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnewfront%2Fodsc-east2019-warmup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnewfront%2Fodsc-east2019-warmup/lists"}