https://github.com/felixcheung/spark-k8s
Apache Spark to run on Kubernetes
https://github.com/felixcheung/spark-k8s
apache-spark docker docker-image k8s kubernetes
Last synced: 6 months ago
JSON representation
Apache Spark to run on Kubernetes
- Host: GitHub
- URL: https://github.com/felixcheung/spark-k8s
- Owner: felixcheung
- License: apache-2.0
- Created: 2017-09-25T07:18:28.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2017-11-16T06:04:23.000Z (over 8 years ago)
- Last Synced: 2025-02-01T21:42:36.690Z (over 1 year ago)
- Topics: apache-spark, docker, docker-image, k8s, kubernetes
- Language: Shell
- Homepage:
- Size: 7.81 KB
- Stars: 2
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# To deploy
Replace `$IMAGE_NAME` with your image server, image name and tag
# To run job
```
# First, find the pod name from the deployment
SPARKPOD="$(echo -e "$(kubectl get pods -l component=sparkclient -n=spark -o=custom-columns=:.metadata.name)" | tr -d '[:space:]')"
# Run a shell to the pod
kubectl exec -ti "$SPARKPOD" -n=spark -- bash
# In the shell, run the spark-shell command, pointing to the master
bin/spark-shell --master spark://sparkmaster:7077
```
# To access Spark UI
Spark Master UI
http://localhost:8001/api/v1/proxy/namespaces/spark/services/sparkmaster:8080/
Spark Application UI
http://localhost:8001/api/v1/proxy/namespaces/spark/services/sparkclient:4040/jobs/
Alternatively k8s port forwarding can be used to access
https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/