https://github.com/radanalyticsio/spark-operator

Operator for managing the Spark clusters on Kubernetes and OpenShift.
https://github.com/radanalyticsio/spark-operator

apache-spark kubernetes kubernetes-operator openshift spark

Last synced: 1 day ago
JSON representation

Operator for managing the Spark clusters on Kubernetes and OpenShift.

• Host: GitHub
• URL: https://github.com/radanalyticsio/spark-operator
• Owner: radanalyticsio
• License: apache-2.0
• Created: 2018-05-17T15:48:59.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2021-11-18T14:01:48.000Z (over 3 years ago)
• Last Synced: 2024-08-08T09:14:21.119Z (7 months ago)
• Topics: apache-spark, kubernetes, kubernetes-operator, openshift, spark
• Language: Java
• Homepage:
• Size: 3.39 MB
• Stars: 156
• Watchers: 9
• Forks: 61
• Open Issues: 63
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • Contributing: .github/CONTRIBUTING.md
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md
  • Codeowners: .github/CODEOWNERS

Awesome Lists containing this project

README

        
# spark-operator

[![Build status](https://travis-ci.org/radanalyticsio/spark-operator.svg?branch=master)](https://travis-ci.org/radanalyticsio/spark-operator)

[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0)

`{CRD|ConfigMap}`-based approach for managing the Spark clusters in Kubernetes and OpenShift.

[![Watch the full asciicast](https://github.com/radanalyticsio/spark-operator/raw/master/docs/ascii.gif)](https://asciinema.org/a/230927?&cols=123&rows=27&theme=monokai)

# How does it work

![UML diagram](https://github.com/radanalyticsio/spark-operator/raw/master/docs/standardized-UML-diagram.png "UML Diagram")

# Quick Start

Run the `spark-operator` deployment: _Remember to change the `namespace` variable for the `ClusterRoleBinding` before doing this step_

```bash

kubectl apply -f manifest/operator.yaml

```

Create new cluster from the prepared example:

```bash

kubectl apply -f examples/cluster.yaml

```

After issuing the commands above, you should be able to see a new Spark cluster running in the current namespace.

```bash

kubectl get pods

NAME                               READY     STATUS    RESTARTS   AGE

my-spark-cluster-m-5kjtj           1/1       Running   0          10s

my-spark-cluster-w-m8knz           1/1       Running   0          10s

my-spark-cluster-w-vg9k2           1/1       Running   0          10s

spark-operator-510388731-852b2     1/1       Running   0          27s

```

Once you don't need the cluster anymore, you can delete it by deleting the custom resource by:

```bash

kubectl delete sparkcluster my-spark-cluster

```

# Very Quick Start

```bash

# create operator

kubectl apply -f http://bit.ly/sparkop

# create cluster

cat < OperatorHub` (make sure the namespace is set to `openshift-marketplace`).

### Troubleshooting

Show the log:

```bash

# last 25 log entries

kubectl logs --tail 25 -l app.kubernetes.io/name=spark-operator

```

```bash

# follow logs

kubectl logs -f `kubectl get pod -l app.kubernetes.io/name=spark-operator -o='jsonpath="{.items[0].metadata.name}"' | sed 's/"//g'`

```

Run the operator from your host (also possible with the debugger/profiler):

```bash

java -jar target/spark-operator-*.jar

```