Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/dtstack/taier

Taier is a big data development platform for submission, scheduling, operation and maintenance, and indicator information display
https://github.com/dtstack/taier

azkaban chunjun cronjob-scheduler dag data-schedule distributed-schedule-system flink hadoop hive job-scheduler scheduler spark task-schedule workflow-scheduling-system

Last synced: about 15 hours ago
JSON representation

Taier is a big data development platform for submission, scheduling, operation and maintenance, and indicator information display

Host: GitHub
URL: https://github.com/dtstack/taier
Owner: DTStack
License: apache-2.0
Created: 2021-03-02T04:49:33.000Z (over 3 years ago)
Default Branch: master
Last Pushed: 2024-08-07T03:06:51.000Z (4 months ago)
Last Synced: 2024-11-19T21:06:53.139Z (about 15 hours ago)
Topics: azkaban, chunjun, cronjob-scheduler, dag, data-schedule, distributed-schedule-system, flink, hadoop, hive, job-scheduler, scheduler, spark, task-schedule, workflow-scheduling-system
Language: Java
Homepage: https://dtstack.github.io/Taier/
Size: 148 MB
Stars: 1,336
Watchers: 33
Forks: 336
Open Issues: 72
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

        


         

           

        

 Taier

 A distributed dispatching system





  

  

  

  

   

  

  


    Office Website |

    Document

  


[中文](./README_zh-CN.md) | [English](./README.md) 

## Introduction

> **Taier** , spelling in chinese is 太阿, is one of the ancient chinese celebrated swords.

**Taier** is distributed dispatching system that focus on different tasks submitted and scheduled. It's aimed at reducing the **ETL**'s cost, making the complex dependencies between tasks clearly and reducing the labor cost about submitting, scheduling and O&M.

It's unnecessary to concern about the complex dependencies between tasks and the underlying architecture about the big data platform at **Taier**, so that you can pay more attention into business.

**Taier** provide an one-stop big data platform for submitting tasks, scheduling tasks, O&M, presentation about indicators.

The core features for Taier are as follows:

- Easy to distributed extend

- Visualization config for DAG

- With an IDE development platform designed for big-data users

- Supports to develop your own plugins

- Multiply task modes including guide mode and script mode

- Supports to the dependencies between upstream/downstream tasks

- Supports batch and stream tasks

- Integrates various different versions of Hadoop

- Easy to integrate Flink Standalone

- Completely safe and non-intrusive to the cluster's environment

- Isolation by tenants and clusters

- Supports kerberos authentication

- Different versions for tasks

- Supports user-defined parameters for task

- Real-time monitoring about cluster's resources

- Real-time presented about data indicators

- Restrict the task's resource

## Architecture

![architecture](/website/static/img/readme/taier-architecture.png)

## Quick start

#### [In Docker](https://dtstack.github.io/Taier/docs/quickstart/deploy/docker#2-%E4%BD%BF%E7%94%A8docker-compose)

```shell

$ wget https://raw.githubusercontent.com/DTStack/Taier/master/docker-compose.yml

$ docker-compose up -d

```

#### [Online Preview](http://taier.dtstack.cn/)

#### [Develop Tasks](https://dtstack.github.io/Taier/docs/quickstart/start)

![main](/website/static/img/readme/main.png)

## Tasks

|       Tasks       | 
| :---------------: 
|     Work Flow     | 
|     Data Sync     | 
| Data Acquisition 
|       Flink 
|       Shell 
|      Python 
|     Spark SQL 
|     Hive SQL 
|     Flink SQL 
|   OceanBase SQL 
|  ClickHouse SQL 
|     Doris SQL 
|      TiDB SQL     | 
|      MySQL SQL    | 
|      Vertica SQL  | 
|      Postgre SQL  | 
|     SqlServer SQL | 
|   Greenplum SQL   | 
|   MaxCompute SQL  | 
|     GaussDB SQL   | 
|     DataX 
| User-defined Task |

Documentation                                     | | :-----------------------------------------------------------------------------------: | [Documentation](https://dtstack.github.io/Taier/docs/functions/task/workflow)   | [Documentation](https://dtstack.github.io/Taier/docs/functions/task/sync)       | | [Documentation](https://dtstack.github.io/Taier/docs/functions/task/data-acquisition) | |      [Documentation](https://dtstack.github.io/Taier/docs/functions/task/flink)       | |      [Documentation](https://dtstack.github.io/Taier/docs/functions/task/shell)       | |      [Documentation](https://dtstack.github.io/Taier/docs/functions/task/python)      | |    [Documentation](https://dtstack.github.io/Taier/docs/functions/task/spark-sql)     | |     [Documentation](https://dtstack.github.io/Taier/docs/functions/task/hive-sql)     | |    [Documentation](https://dtstack.github.io/Taier/docs/functions/task/flink-sql)     | |  [Documentation](https://dtstack.github.io/Taier/docs/functions/task/oceanbase-sql)   | |  [Documentation](https://dtstack.github.io/Taier/docs/functions/task/clickhouse-sql)  | |    [Documentation](https://dtstack.github.io/Taier/docs/functions/task/doris-sql)     | [Documentation](https://dtstack.github.io/Taier/docs/functions/task/tidb-sql)    | [Documentation](https://dtstack.github.io/Taier/docs/functions/task/mysql-sql)   | [Documentation](https://dtstack.github.io/Taier/docs/functions/task/vertica-sql) | [Documentation](https://dtstack.github.io/Taier/docs/functions/task/postgre-sql) | [Documentation](https://dtstack.github.io/Taier/docs/functions/task/sqlserver-sql)| [Documentation](https://dtstack.github.io/Taier/docs/functions/task/greenplum-sql)| [Documentation](https://dtstack.github.io/Taier/docs/functions/task/maxcompute-sql)| [Documentation](https://dtstack.github.io/Taier/docs/functions/task/guassdb-sql) | |      [Documentation](https://dtstack.github.io/Taier/docs/functions/task/datax) | [Documentation](https://dtstack.github.io/Taier/docs/expand/task)           |

## Questions

[FAQ Reference](https://dtstack.github.io/Taier/docs/quickstart/faq) For questions, bugs and supports please open an [issue](https://github.com/DTStack/Taier/issues/new/choose), we'll reply you in time.

## Stay in touch

- Slack https://join.slack.com/t/slack-p437975/shared_invite/zt-1iw5x1fw7-A6rVolqjP2z8V09~WwFUiA

## Contribution

Please make sure to read the [Contributing Guide](https://dtstack.github.io/Taier/docs/contributing) before making a pull request.

## Contributor



  



## License

**Taier** is under the Apache 2.0 license. See the [LICENSE](http://www.apache.org/licenses/LICENSE-2.0) file for

details.