https://github.com/vrann/spark-k8s-template
Spark K8S Template
https://github.com/vrann/spark-k8s-template
Last synced: 3 months ago
JSON representation
Spark K8S Template
- Host: GitHub
- URL: https://github.com/vrann/spark-k8s-template
- Owner: vrann
- Created: 2019-08-01T00:58:01.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2019-08-01T00:58:25.000Z (almost 6 years ago)
- Last Synced: 2025-01-08T01:52:48.473Z (5 months ago)
- Language: Shell
- Size: 4.88 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Spark Standalone on Kubernetes
## Install
1. Download Spark and unpack
2. cd spark-2.4.3-bin-hadoop2.7
3. Download lapack 3.0.8 //Replace with step in Dockerfile
4. git clone vrann/spark-k8s-template
5. Build and copy application
6. Build image, run `docker build -t spark-hadoop:1.0.35 -f spark-k8s-template/Dockerfile .`
7. Start master, run `kubectl apply -f spark-k8s-template/spark-master-deployment.yaml`
8. Start master service, run `kubectl apply -f spark-k8s-template/spark-worker-service.yaml`
9. verify that master UI is working, go to http://localhost:8083
10. Start workers, run `kubectl apply -f spark-k8s-template/spark-worker-deployment.yaml`
11. Verify 6 containers are running, one for master and five for workers, run `kubectl get pods`
## Run Application
1. log into worker container, run `kubectl exec -it spark-worker-76574dc79-kpt5b /bin/bash`
2. `./bin/spark-submit --class "SparkALS" --master spark://spark-master.default.svc.cluster.local:7077 --conf spark.driver.bindAddress=$MY_POD_IP --conf spark.local.ip=$MY_POD_IP --conf spark.driver.host=$MY_POD_IP --deploy-mode client /opt/spark/data/spark-counter_2.11-0.4.jar`
## Description