{"id":13415188,"url":"https://github.com/datavane/tis","last_synced_at":"2025-12-25T09:32:45.866Z","repository":{"id":38364987,"uuid":"167190269","full_name":"datavane/tis","owner":"datavane","description":"Support agile DataOps Based on Flink, DataX and Flink-CDC, Chunjun with Web-UI","archived":false,"fork":false,"pushed_at":"2025-04-13T03:32:32.000Z","size":29847,"stargazers_count":1124,"open_issues_count":66,"forks_count":240,"subscribers_count":40,"default_branch":"master","last_synced_at":"2025-04-13T04:31:21.510Z","etag":null,"topics":["cdc","chunjun","dataops","datax","etl","flink","flink-streaming","java"],"latest_commit_sha":null,"homepage":"https://tis.pub","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datavane.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.txt","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-23T13:44:33.000Z","updated_at":"2025-04-13T03:32:35.000Z","dependencies_parsed_at":"2023-08-27T15:59:56.959Z","dependency_job_id":"b7e1e632-877b-4cb9-b2a8-5df32af068b7","html_url":"https://github.com/datavane/tis","commit_stats":null,"previous_names":["datavane/tis","qlangtech/tis"],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datavane%2Ftis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datavane%2Ftis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datavane%2Ftis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datavane%2Ftis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datavane","download_url":"https://codeload.github.com/datavane/tis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248749994,"owners_count":21155682,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdc","chunjun","dataops","datax","etl","flink","flink-streaming","java"],"created_at":"2024-07-30T21:00:44.767Z","updated_at":"2025-12-25T09:32:45.811Z","avatar_url":"https://github.com/datavane.png","language":"Java","readme":"## Stargazers over time\n\n[![Stargazers over time](https://starchart.cc/qlangtech/tis-solr.svg)](https://starchart.cc/qlangtech/tis-solr)\n![tis](docs/tis-logo2.jpg)\n\n![](https://tokei.rs/b1/github/datavane/tis)\n## TIS介绍\n\nTIS为您提供企业级数据集成服务，**基于批(DataX)，流(Flink-CDC、Chunjun)一体，提供简单易用的操作界面，降低用户实施端到端数据同步的实施门槛，缩短任务配置时间，避免配置过程中出错，使数据同步变得简单、有趣且容易上手** [详细介绍](https://tis.pub/docs/)\n\n如果您已经觉得利用DataX、Flink-CDC、SeaTunnel编写脚本来驱动执行端到端数据同步程序执行，是非常烦冗且容易出错的事儿。并因此感到些许厌倦，您可以尝试一下TIS，相信会给您带来不小的惊喜。\n\n![tis](docs/execute_model_batch_incr.jpg)\n\n## :arrow_forward: 安装说明\n\n快速安装（支持私有云、公有云等环境），方便、快捷 [:green_circle:单机](https://tis.pub/docs/install/tis/uber) [:purple_circle:Docker](https://tis.pub/docs/install/tis/docker) [:large_blue_circle:Docker Compose](https://tis.pub/docs/install/tis/docker-compose) [:brown_circle:K8S](https://tis.pub/docs/install/tis/kubernetes)\n\n## v4.1.0 发布（2025/2/27）：\nhttps://github.com/datavane/tis/releases/tag/v4.1.0\n\n**历史发布**： v4.0.1 发布（2024/10/19）：https://github.com/datavane/tis/releases/tag/V4.0.1\n\u003ca href=\"https://www.bilibili.com/video/BV1rXC1YjECW\"\u003e \u003cimg src=\"https://github.com/user-attachments/assets/59a10a10-4a08-45d8-b157-01df5f9dbf5a\" /\u003e \u003c/a\u003e\n\n\n## 架构\n\n![tis](docs/conceptual-diagram.png)\n\n![tis](docs/tis-synoptic.png)\n\n\n\n## 核心特性\n\n* :hammer:安装简单\n\n  TIS的安装还是和传统软件安装一样，只需要三个步骤：一、下载tar包，二、解压tar包，三、启动TIS。是的，就这么简单。另外还支持多种部署模式（Docker、Docker-Compose、Kubernetes）。\n\n* :rocket:强大的扩展性\n\n  TIS 继承了Jenkin 的设计思想，使用微前端技术，重新构建了一套前端框架，前端页面可以自动渲染。\n\n  TIS 提供了良好的扩展性和SPI机制，开发者可以很轻松地开发新的插件。\n\n* :repeat:支持各种Transformer算子\n\n  通过在TIS流程中设置Transformer功能逻辑，可实现在ETL数仓功能中，在E（数据抽取）阶段即可高效实现各种字段值转换功能，如：字段脱敏，字段拼接，取子字符串等功能。\n\n* :smile:易与大数据生态整合\n\n  支持与DolphinScheduler整合，用户只需在TIS中配置数据管道，验证无误后，一键推送同步表任务到DolphinScheduler平台，即可在其上完成数据同步任务调度。为以DolphinScheduler构建ETL数仓方案如虎添翼。\n\n* :ok_hand: 基于 DataOps 理念\n\n  借鉴了 DataOps、DataPipeline 理念，对各各执行流程建模。不需要了解底层模块的实现原理，基本实现傻瓜化方式操作。\n\n\n## 支持的读写组件 [详细](https://tis.pub/docs/plugin/source-sink/)\n|Reader|Writer|\n|--|--|\n|\u003cimg src=\"docs/logo/cassandra.svg\" width=\"40\" /\u003e\u003cimg src=\"docs/logo/ftp.svg\" width=\"40\" /\u003e  \u003cimg src=\"docs/logo/hdfs.svg\" width=\"40\" /\u003e \u003cimg src=\"docs/logo/mongodb.svg\" width=\"40\" /\u003e  \u003cimg src=\"docs/logo/mysql.svg\" width=\"40\" /\u003e \u003cimg src=\"docs/logo/oracle.svg\" width=\"40\" /\u003e  \u003cimg src=\"docs/logo/oss.svg\" width=\"40\" /\u003e  \u003cimg src=\"docs/logo/postgresql.svg\" width=\"40\" /\u003e \u003cimg src=\"docs/logo/sqlserver.svg\" width=\"40\" /\u003e \u003cimg src=\"docs/logo/tidb.svg\" width=\"40\" /\u003e | \u003cimg src=\"docs/logo/mysql.svg\" width=\"40\" /\u003e \u003cimg src=\"docs/logo/doris.svg\" width=\"40\" /\u003e \u003cimg src=\"docs/logo/spark.svg\" width=\"40\" /\u003e\u003cimg src=\"docs/logo/starrocks.svg\" width=\"40\" /\u003e\u003cimg src=\"docs/logo/cassandra.svg\" width=\"40\" /\u003e \u003cimg src=\"docs/logo/postgresql.svg\" width=\"40\" /\u003e\u003cimg src=\"docs/logo/hive.svg\" width=\"40\" /\u003e\u003cimg src=\"docs/logo/clickhouse.svg\" width=\"40\" /\u003e\u003cimg src=\"docs/logo/ftp.svg\" width=\"40\" /\u003e\u003cimg src=\"docs/logo/oracle.svg\" width=\"40\" /\u003e \u003cimg src=\"docs/logo/hdfs.svg\" width=\"40\" /\u003e\u003cimg src=\"docs/logo/es.svg\" width=\"40\" /\u003e |\n\n\n\n## 功能一瞥\n- 示例\n    * [基于TIS快速实现MySQL到StarRocks的实时数据同步方案](https://tis.pub/docs/example/mysql-sync-starrocks)\n    * [多源同步Doris方案](https://tis.pub/docs/example/mysql-sync-doris)\n    * [将数据变更同步到Kafka](https://tis.pub/docs/example/sink-2-kafka)\n    * [利用TIS实现T+1离线分析](https://tis.pub/docs/example/dataflow)\n- 视频示例\n    * [安装示例](https://www.bilibili.com/video/BV18q4y1p73B/)\n    * [启用分布式执行功能](https://www.bilibili.com/video/BV1Cq4y1D7z4?share_source=copy_web)\n    * [MySQL导入ElasticSearch](https://www.bilibili.com/video/BV1G64y1B7wm?share_source=copy_web)\n    * [MySQL导入Hive](https://www.bilibili.com/video/BV1Vb4y1z7DN?share_source=copy_web)\n    * [MySQL导入Clickhouse](https://www.bilibili.com/video/BV1x64y1B7V8/)\n    * [MySQL同步StarRocks](https://www.bilibili.com/video/BV19o4y1M7eq/)\n    * MySQL同步Doris [\u003e批量同步](https://www.bilibili.com/video/BV1eh4y1o7yQ) [\u003e实时同步](https://www.bilibili.com/video/BV1nX4y1h7SW)\n### 批量导入流程设置\n\n选择Reader/Writer插件类型\n![tis](docs/datax-add-step2.png)\n\n添加MySqlReader\n![tis](docs/add-mysql-reader.png)\n\n设置MySqlReader目标表、列  \n![tis](docs/select-tab-cols.png)\n\n设置MySqlReader目标列设置Transformer逻辑。例如，将表中JSON内容的列分拆成多个字段`json_splitter`\n![tis](docs/select-tab-cols-json-splitter.png)\n\n添加ElasticWriter,可视化设置ElasticSearch的Schema Mapping\n![tis](docs/add-elastic-writer.png)\n\n执行MySql-\u003eElasticSearch DataX实例，运行状态\n![tis](docs/datax-exec-status.png)\n\n### 开通Flink实时数据通道\n\n添加Flink-Cluster、设置重启策略、CheckPoint机制等\n![tis](docs/incr_step_1.png)\n\n设置Source/Sink组件属性\n![tis](docs/incr_step_2.png)\n\nTIS基于数据库元数据信息自动生成Flink-SQL脚本,您没看错全部脚本自动生！\n![tis](docs/incr_step_3.png)\n\n实时数据通道创建完成！构建一个实时数仓就这么简单！！！   \n![tis](docs/incr_step_4.png)\n\n## 依赖项目\n\n- WEB UI [https://github.com/qlangtech/ng-tis](https://github.com/qlangtech/ng-tis)\n- 发版信息生成器[https://github.com/qlangtech/tis-git-manager](https://github.com/qlangtech/tis-git-manager)\n- 基于Ansible的打包工具 [https://github.com/qlangtech/tis-ansible](https://github.com/qlangtech/tis-ansible)\n- TIS 插件 \n   1. [plugins](https://github.com/qlangtech/plugins) \n   2. [tis-plugins-commercial](https://github.com/qlangtech/tis-plugins-commercial)\n   3. SqlServer Connector [qlangtech/tis-sqlserver-plugin](https://github.com/qlangtech/tis-sqlserver-plugin)\n- TIS 插件元数据生成工具 [https://github.com/qlangtech/update-center2](https://github.com/qlangtech/update-center2)\n- DataX [https://github.com/qlangtech/DataX](https://github.com/qlangtech/DataX)\n- Flink Extend [https://github.com/qlangtech/flink](https://github.com/qlangtech/flink)\n- Dolphinscheduler [https://github.com/qlangtech/dolphinscheduler](https://github.com/qlangtech/dolphinscheduler)\n- TIS 插件功能脚手架 [https://github.com/qlangtech/tis-archetype-plugin](https://github.com/qlangtech/tis-archetype-plugin)\n- Chunjun [https://github.com/qlangtech/chunjun](https://github.com/qlangtech/chunjun)\n- TIS Docs Manager [https://github.com/qlangtech/tis-doc](https://github.com/qlangtech/tis-doc)\n- 扩展Debezium [qlangtech/debezium](https://github.com/qlangtech/debezium)\n- 扩展Flink-CDC [qlangtech/flink-cdc](https://github.com/qlangtech/flink-cdc)\n- 部分插件参数配置参考 Airbyte [https://github.com/airbytehq/airbyte](https://github.com/airbytehq/airbyte)\n- 架构思想参考 [https://github.com/jenkinsci/jenkins](https://github.com/jenkinsci/jenkins)\n\n## 如何开发\n\n[https://tis.pub/docs/develop/compile-running/](https://tis.pub/docs/develop/compile-running/)\n\n## 许可协议\n\nTIS is under the Apache2 License. See the [LICENSE](https://github.com/qlangtech/tis-solr/blob/master/LICENSE) file for details.\n\n## 反馈\n\n您在使用过程中对TIS有任何不满或者批评都请不惜斧正，您提出的宝贵意见是对我们最大的支持和鼓励，[我要提建议](https://github.com/qlangtech/tis/issues/new)\n","funding_links":[],"categories":["Java","大数据","新项目脚手架"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatavane%2Ftis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatavane%2Ftis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatavane%2Ftis/lists"}