https://github.com/drorata/jupyter-spark-python-setup
https://github.com/drorata/jupyter-spark-python-setup
Last synced: over 1 year ago
JSON representation
- Host: GitHub
- URL: https://github.com/drorata/jupyter-spark-python-setup
- Owner: drorata
- Created: 2016-01-25T11:58:15.000Z (over 10 years ago)
- Default Branch: master
- Last Pushed: 2016-01-25T12:00:03.000Z (over 10 years ago)
- Last Synced: 2025-01-15T23:59:20.804Z (over 1 year ago)
- Language: Jupyter Notebook
- Size: 20.5 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
This repository holds my attempts to allow interaction with [Spark](https://spark.apache.org/) on a local machine as well as on a remote cluster using Python from within [Jupyter](http://jupyter.org/).
Main steps:
* Setup Spark on a local machine (DONE)
* Execute simple Spark job(s) on a local machine from Jupyter (DONE)
* Load CSV files into spark, using `spark_csv`
* Link *local* notebook to a cluster instance of Spark and run jobs remotely. Here, it is important to understand how to setup the notebook where to execute Spark. Should be simple to toggle between the local and remote instances.