https://github.com/flix-tech/k8s-spark-cluster-mode-example
running spark app on k8s with cluster mode
https://github.com/flix-tech/k8s-spark-cluster-mode-example
Last synced: 11 months ago
JSON representation
running spark app on k8s with cluster mode
- Host: GitHub
- URL: https://github.com/flix-tech/k8s-spark-cluster-mode-example
- Owner: flix-tech
- Created: 2019-10-31T10:42:12.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2019-11-20T16:45:31.000Z (over 6 years ago)
- Last Synced: 2025-04-08T07:43:18.592Z (over 1 year ago)
- Language: Python
- Homepage:
- Size: 4.14 MB
- Stars: 0
- Watchers: 6
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Spark on Kubernetes - example app
This is an example project running a Spark app on Kubernetes. It runs a PySpark job on Kubernetes. More information [here](https://spark.apache.org/docs/latest/running-on-kubernetes.html).
The application base image relies on the spark image built on [here](https://git.flix.tech/data/flux/web-tracking/spark-on-kubernetes-docker-image).
## run app locally
Install local Kubernetes cluster first. Use [minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/).
Install `task` build tools [task](https://taskfile.dev/#/installation).
Run locally:
```shell script
task run.local
```
Undeploy locally:
```shell script
source ./local.env && task spark.undeploy
```
## deploy on EKS Kubernetes cluster
Edit `.gitlab-ci.yml` file to adapt it to deploy to your own k8s namespace, please read team plateform documentation for details about gitlab runner and Kubernetes.
Create a branch will automatically deploy it on ew1d2 cluster, data-flux-dev namespace. Merge branch will deploy code on data-flux-stg and then to ew1p3 data-flux namespace.