Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/radanalyticsio/spark-operator
Operator for managing the Spark clusters on Kubernetes and OpenShift.
https://github.com/radanalyticsio/spark-operator
apache-spark kubernetes kubernetes-operator openshift spark
Last synced: about 1 month ago
JSON representation
Operator for managing the Spark clusters on Kubernetes and OpenShift.
- Host: GitHub
- URL: https://github.com/radanalyticsio/spark-operator
- Owner: radanalyticsio
- License: apache-2.0
- Created: 2018-05-17T15:48:59.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2021-11-18T14:01:48.000Z (almost 3 years ago)
- Last Synced: 2024-08-08T09:14:21.119Z (3 months ago)
- Topics: apache-spark, kubernetes, kubernetes-operator, openshift, spark
- Language: Java
- Homepage:
- Size: 3.39 MB
- Stars: 156
- Watchers: 9
- Forks: 61
- Open Issues: 63
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: .github/CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# spark-operator
[![Build status](https://travis-ci.org/radanalyticsio/spark-operator.svg?branch=master)](https://travis-ci.org/radanalyticsio/spark-operator)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0)`{CRD|ConfigMap}`-based approach for managing the Spark clusters in Kubernetes and OpenShift.
[![Watch the full asciicast](https://github.com/radanalyticsio/spark-operator/raw/master/docs/ascii.gif)](https://asciinema.org/a/230927?&cols=123&rows=27&theme=monokai)
# How does it work
![UML diagram](https://github.com/radanalyticsio/spark-operator/raw/master/docs/standardized-UML-diagram.png "UML Diagram")# Quick Start
Run the `spark-operator` deployment: _Remember to change the `namespace` variable for the `ClusterRoleBinding` before doing this step_
```bash
kubectl apply -f manifest/operator.yaml
```Create new cluster from the prepared example:
```bash
kubectl apply -f examples/cluster.yaml
```After issuing the commands above, you should be able to see a new Spark cluster running in the current namespace.
```bash
kubectl get pods
NAME READY STATUS RESTARTS AGE
my-spark-cluster-m-5kjtj 1/1 Running 0 10s
my-spark-cluster-w-m8knz 1/1 Running 0 10s
my-spark-cluster-w-vg9k2 1/1 Running 0 10s
spark-operator-510388731-852b2 1/1 Running 0 27s
```Once you don't need the cluster anymore, you can delete it by deleting the custom resource by:
```bash
kubectl delete sparkcluster my-spark-cluster
```# Very Quick Start
```bash
# create operator
kubectl apply -f http://bit.ly/sparkop# create cluster
cat < OperatorHub` (make sure the namespace is set to `openshift-marketplace`).### Troubleshooting
Show the log:
```bash
# last 25 log entries
kubectl logs --tail 25 -l app.kubernetes.io/name=spark-operator
``````bash
# follow logs
kubectl logs -f `kubectl get pod -l app.kubernetes.io/name=spark-operator -o='jsonpath="{.items[0].metadata.name}"' | sed 's/"//g'`
```Run the operator from your host (also possible with the debugger/profiler):
```bash
java -jar target/spark-operator-*.jar
```