Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/radanalyticsio/spark-operator

Operator for managing the Spark clusters on Kubernetes and OpenShift.
https://github.com/radanalyticsio/spark-operator

apache-spark kubernetes kubernetes-operator openshift spark

Last synced: 28 days ago
JSON representation

Operator for managing the Spark clusters on Kubernetes and OpenShift.

Awesome Lists containing this project

README

        

# spark-operator

[![Build status](https://travis-ci.org/radanalyticsio/spark-operator.svg?branch=master)](https://travis-ci.org/radanalyticsio/spark-operator)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0)

`{CRD|ConfigMap}`-based approach for managing the Spark clusters in Kubernetes and OpenShift.

[![Watch the full asciicast](https://github.com/radanalyticsio/spark-operator/raw/master/docs/ascii.gif)](https://asciinema.org/a/230927?&cols=123&rows=27&theme=monokai)

# How does it work
![UML diagram](https://github.com/radanalyticsio/spark-operator/raw/master/docs/standardized-UML-diagram.png "UML Diagram")

# Quick Start

Run the `spark-operator` deployment: _Remember to change the `namespace` variable for the `ClusterRoleBinding` before doing this step_
```bash
kubectl apply -f manifest/operator.yaml
```

Create new cluster from the prepared example:

```bash
kubectl apply -f examples/cluster.yaml
```

After issuing the commands above, you should be able to see a new Spark cluster running in the current namespace.

```bash
kubectl get pods
NAME READY STATUS RESTARTS AGE
my-spark-cluster-m-5kjtj 1/1 Running 0 10s
my-spark-cluster-w-m8knz 1/1 Running 0 10s
my-spark-cluster-w-vg9k2 1/1 Running 0 10s
spark-operator-510388731-852b2 1/1 Running 0 27s
```

Once you don't need the cluster anymore, you can delete it by deleting the custom resource by:
```bash
kubectl delete sparkcluster my-spark-cluster
```

# Very Quick Start

```bash
# create operator
kubectl apply -f http://bit.ly/sparkop

# create cluster
cat < OperatorHub` (make sure the namespace is set to `openshift-marketplace`).

### Troubleshooting

Show the log:

```bash
# last 25 log entries
kubectl logs --tail 25 -l app.kubernetes.io/name=spark-operator
```

```bash
# follow logs
kubectl logs -f `kubectl get pod -l app.kubernetes.io/name=spark-operator -o='jsonpath="{.items[0].metadata.name}"' | sed 's/"//g'`
```

Run the operator from your host (also possible with the debugger/profiler):

```bash
java -jar target/spark-operator-*.jar
```