https://github.com/staffbase/spark-submit-server
lightweight go server that exposes spark-submit
https://github.com/staffbase/spark-submit-server
Last synced: about 1 year ago
JSON representation
lightweight go server that exposes spark-submit
- Host: GitHub
- URL: https://github.com/staffbase/spark-submit-server
- Owner: Staffbase
- License: apache-2.0
- Created: 2023-06-04T12:16:30.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-03-13T22:42:29.000Z (over 2 years ago)
- Last Synced: 2025-01-23T05:45:07.213Z (over 1 year ago)
- Language: Go
- Homepage:
- Size: 130 KB
- Stars: 1
- Watchers: 12
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# spark-submit-server
lightweight go server that exposes [`spark-submit`](https://spark.apache.org/docs/latest/submitting-applications.html)
## Run with a kind cluster
1. create the cluster
```bash
kind create cluster
```
2. create spark namespace & serviceaccount
```bash
kubectl create -f example/spark.yaml
```
3. proxy kubernetes API to host
```bash
kubectl proxy
```
4. run the go server locally
```bash
go run . --spark-home=$SPARK_HOME_DIR --master=k8s://http://127.0.0.1:8001 --spark-conf-dir=example/sparkConf --debug
```
5. start the example
```
curl -XPOST http://localhost:7070?preset=pi
```
6. delete the kind cluster
```bash
kind cluster delete`
```