https://github.com/flix-tech/k8s-spark-example
https://github.com/flix-tech/k8s-spark-example
Last synced: over 1 year ago
JSON representation
- Host: GitHub
- URL: https://github.com/flix-tech/k8s-spark-example
- Owner: flix-tech
- Created: 2019-12-03T16:31:04.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2020-09-11T12:10:02.000Z (almost 6 years ago)
- Last Synced: 2025-02-14T04:51:12.883Z (over 1 year ago)
- Language: Python
- Size: 4.15 MB
- Stars: 1
- Watchers: 6
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Spark on Kubernetes - example app
This is an example project running a Spark app on Kubernetes. It runs a PySpark job as spark driver deployment on Kubernetes. More information [here](https://spark.apache.org/docs/latest/running-on-kubernetes.html).
This example contains two spark deployment mode:
* local mode(helm-values/k8s-spark-local-example
)
* client mode(helm-values/k8s-spark-client-example
)
## run app locally
Install local Kubernetes cluster first. Use [minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/).
Install `task` build tools [task](https://taskfile.dev/#/installation).
Run locally:
```shell script
task run.local
```
Undeploy locally:
```shell script
task spark.helm.undeploy
```
## deploy on EKS Kubernetes cluster
Edit `.gitlab-ci.yml` file to adapt it to deploy to your own k8s namespace, please read team plateform documentation for details about gitlab runner and Kubernetes.
Create a branch will automatically deploy it on ew1d2 cluster, data-flux-dev namespace. Merge branch will deploy code on data-flux-stg and then to ew1p3 data-flux namespace.