https://github.com/apache/dolphinscheduler-operator
Apache DolphinScheduler Kubernetes Operator.
https://github.com/apache/dolphinscheduler-operator
airflow azkaban cronjob-schedule data-schedule distributed-schedule-system etl-dependency job-schedule job-scheduler oozie schedule task-schedule task-scheduler workflow-platform workflow-schedule workflow-scheduling-system
Last synced: 3 months ago
JSON representation
Apache DolphinScheduler Kubernetes Operator.
- Host: GitHub
- URL: https://github.com/apache/dolphinscheduler-operator
- Owner: apache
- License: apache-2.0
- Created: 2022-07-04T03:28:10.000Z (almost 3 years ago)
- Default Branch: master
- Last Pushed: 2023-12-18T22:14:16.000Z (over 1 year ago)
- Last Synced: 2025-01-30T10:34:38.283Z (3 months ago)
- Topics: airflow, azkaban, cronjob-schedule, data-schedule, distributed-schedule-system, etl-dependency, job-schedule, job-scheduler, oozie, schedule, task-schedule, task-scheduler, workflow-platform, workflow-schedule, workflow-scheduling-system
- Language: Go
- Homepage: https://dolphinscheduler.apache.org/
- Size: 269 KB
- Stars: 27
- Watchers: 21
- Forks: 18
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# dolphinscheduler-operator
## Features
- Deploy and manage the master, worker, alert, api components.
- Scale the Pod numbers with one commond.
- Update the component's version (not include the database schema).
## Project Status
Project status: `alpha1`
Current API version: `v1alpha1`
## Get Started
- Create a namespace `ds`
```shell
kubectl create namespace ds
```
- Install PostgreSQL database (Optional)
If you don't have a running database, you can run
```shell
kubectl apply -f config/ds/postgreSQL
```
to create a demo database, note that this is only for demonstration, DO NOT use it in production environment.
You need to replace the `hostPath.path` in `postgres-pv.yaml` if you don't have a directory `/var/lib/data`.
Connect to PostgreSQL and initialize the database schema by executing
[`dolphinscheduler/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql`](https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql).
- Install zookeeper (Optional)
If you don't have a running zookeeper, the demo doployment file is in `config/ds/zookeeper`,
```shell
kubectl apply -f config/ds/zookeeper
```
- Create pv and pvc (Optional)
If you have pv and pvc, you can config it in `config/sameples`.
Or you can create it with `config/ds/ds-pv.yaml` and `config/ds/ds-pvc.yaml`.
Notice to replace the `hostPath.path` in `ds-pv.yaml`.
And you can mount the lib in dolphinscheduler `/opt/soft` in config/samples/ds_v1alpha1_dsworker.yaml with paramter named lib_pvc_name
Mount the logs in `/opt/dolphinscheduler/logs` with the pvcname named `log_pvc_name`.
## query the status of worker
```shell
kubectl get dsworkers.ds.apache.dolphinscheduler.dev -n ds
```
## api explain
```shell
kubectl explain dsmaster
```
## how to test
* Replace the database config and zookeeper config paramters in [`config/samples/`](./config/samples/).
* Replace the nodeport in [`config/samples/ds_v1alpha1_api.yaml`](./config/samples/ds_v1alpha1_dsapi.yaml)
* Install CRDs and controller
## how to deploy
```shell
export IMG=ghcr.io/apache/dolphinscheduler-operator:latest
make build && make manifests && make install && make deploy
```
* Deploy the sample
```shell
cd config/samples
kubectl apply -f ds_v1alpha1_dsalert.yaml
kubectl apply -f ds_v1alpha1_dsapi.yaml -f ds_v1alpha1_dsmaster.yaml -f ds_v1alpha1_dsworker.yaml
```