https://github.com/drorata/jupyter-spark-python-setup

https://github.com/drorata/jupyter-spark-python-setup

Last synced: 27 days ago
JSON representation

• Host: GitHub
• URL: https://github.com/drorata/jupyter-spark-python-setup
• Owner: drorata
• Created: 2016-01-25T11:58:15.000Z (almost 9 years ago)
• Default Branch: master
• Last Pushed: 2016-01-25T12:00:03.000Z (almost 9 years ago)
• Last Synced: 2024-04-16T03:56:14.673Z (7 months ago)
• Language: Jupyter Notebook
• Size: 20.5 KB
• Stars: 2
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: readme.md

Awesome Lists containing this project

README

        
This repository holds my attempts to allow interaction with [Spark](https://spark.apache.org/) on a local machine as well as on a remote cluster using Python from within [Jupyter](http://jupyter.org/).

Main steps:

  * Setup Spark on a local machine (DONE)

  * Execute simple Spark job(s) on a local machine from Jupyter (DONE)

  * Load CSV files into spark, using `spark_csv`

  * Link *local* notebook to a cluster instance of Spark and run jobs remotely. Here, it is important to understand how to setup the notebook where to execute Spark. Should be simple to toggle between the local and remote instances.