https://github.com/sflender/pyspark-test
testing features in pyspark
https://github.com/sflender/pyspark-test
Last synced: 8 months ago
JSON representation
testing features in pyspark
- Host: GitHub
- URL: https://github.com/sflender/pyspark-test
- Owner: sflender
- Created: 2018-06-13T20:51:19.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2018-06-13T20:55:08.000Z (over 7 years ago)
- Last Synced: 2025-01-21T12:34:02.802Z (10 months ago)
- Language: Jupyter Notebook
- Size: 1.95 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Testing pyspark on Mac OSX
## Installation guide:
- install anaconda
- install Java 1.8 development toolkit (I had issues with Java 10 on my Mac)
- put anaconda, spark, and java 1.8 into your environment (.bashrc):
```
export PATH=/anaconda3/bin/:$PATH
export SPARK_HOME=/Users/flender/projects/spark-2.3.1-bin-hadoop2.7/
export PATH=$SPARK_HOME/bin:$PATH
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/
```
- for running the pyspark REPL, simply type pyspark
- for running a notebook, first source env.sh. Then type pyspark.