{"id":13487453,"url":"https://github.com/ucarGroup/DataLink","last_synced_at":"2025-03-27T22:31:07.307Z","repository":{"id":37382200,"uuid":"149598464","full_name":"ucarGroup/DataLink","owner":"ucarGroup","description":"DataLink是一个满足各种异构数据源之间的实时增量同步、离线全量同步，分布式、可扩展的数据交换平台。","archived":false,"fork":false,"pushed_at":"2022-12-06T00:35:57.000Z","size":16624,"stargazers_count":1078,"open_issues_count":56,"forks_count":413,"subscribers_count":64,"default_branch":"master","last_synced_at":"2024-10-29T18:05:32.115Z","etag":null,"topics":["data-exchange","data-replication","datalink","full-replication","realtime-replication"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ucarGroup.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-09-20T11:26:16.000Z","updated_at":"2024-10-27T05:30:41.000Z","dependencies_parsed_at":"2023-01-24T07:46:07.717Z","dependency_job_id":null,"html_url":"https://github.com/ucarGroup/DataLink","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucarGroup%2FDataLink","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucarGroup%2FDataLink/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucarGroup%2FDataLink/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucarGroup%2FDataLink/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ucarGroup","download_url":"https://codeload.github.com/ucarGroup/DataLink/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222322034,"owners_count":16966433,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-exchange","data-replication","datalink","full-replication","realtime-replication"],"created_at":"2024-07-31T18:00:59.536Z","updated_at":"2024-10-30T22:31:04.454Z","avatar_url":"https://github.com/ucarGroup.png","language":"Java","funding_links":[],"categories":["Java","数据库中间件"],"sub_categories":[],"readme":"## 项目介绍\nDataLink是一个满足各种异构数据源之间的实时增量同步、离线全量同步，分布式、可扩展的数据交换平台。\n#### 项目背景\n着眼于未来，我们的目标是打造一个平台，满足各种异构数据源之间的实时增量同步和离线全量同步，支撑公司业务的快速发展。在充分调研的基础之上，我们发现，没有任何一款开源产品能轻易的满足我们的目标，每个产品都有其明显的短板和局限性，所以最终的选项只有\"自行设计\"。但自行设计并不是凭空设计，现有的数据交换平台、已有的经验、大大小小的开源产品都是我们的设计根基，与其说是自行设计，倒不如说是站在巨人的肩膀上做了一次飞跃。由此诞生了DataLink这样一个产品：\n* 满足各种异构数据源之间的实时增量同步和离线全量同步\n* 平台提供统一的基础设施（高可用、动态负载、同步任务管理、插件管理、监控报警、公用业务组件等等），让设计人员专注于同步插件开发，一次投入，长久受益\n* 吸收、整合业内经验，在架构模型、设计方法论、功能特性、可运维、易用性上进行全面的升级，在前瞻性和扩展性上下足功夫，满足未来5-10年内的各种同步需求\n\nDataLink开发时间从2016年12月开始，第一版于2017年5月份上线，在神州优车集团服役到现在，基本上满足了公司所有业务线的同步需求。此次外部开源版本为去除内部依赖后的版本。\n#### 内部使用情况\n* 支持了神州优车和瑞幸咖啡的核心业务运行\n* 涉及500多个数据库实例之间的6000+个同步映射\n* 100台Worker+2台Manager机器的集群规模\n* 日均数据同步量TB级\n#### 当前规划\n整合业内更多的产品经验，打造一个全新的平台，功能丰富程度、扩展性、标准化层面更上一层楼\n## 相关文章\n神州优车数据交换平台的架构、建设与痛点难点详解 \u003cbr\u003e\nhttps://mp.weixin.qq.com/s/BVuDbS-2Ra5pIJ7oV78FBA \u003cbr\u003e\n（神州优车）大数据平台建设经验分享 \u003cbr\u003e\nhttps://www.cnblogs.com/ucarinc/p/12091053.html\n## 总体架构\n![总体架构](https://github.com/ucarGroup/DataLink/blob/master/img/0-0.%E5%85%A8%E9%87%8F%26%E5%A2%9E%E9%87%8F%E6%80%BB%E4%BD%93%E6%9E%B6%E6%9E%84.png)\n## 工作原理(增量)\n![基础架构](https://github.com/ucarGroup/DataLink/blob/master/img/0-0.%E6%9E%B6%E6%9E%84%E5%8E%9F%E7%90%86.png)\n* 典型Master-Slave式系统架构，Manager(Web管理)+Worker(工作节点)\u003cbr\u003e\n    a. Manager负责worker的负载均衡、集群的配置管理和系统监控\u003cbr\u003e\n    b. Worker核心功能是管理Task的生命周期，并配合Manager进行Re-Balance\u003cbr\u003e\n* Zookeeper：Manager的高可用需要依赖于Zookeeper，另外，Task会将运行时信息注册到Zookeeper\n* Mysql：Datalink的运行需要依赖各种配置信息、以及在运行过程中会动态产生监控和统计数据，统一保存到Mysql中\n## QuickStart\nSee the page for quick start: [QuickStart](https://github.com/ucarGroup/DataLink/wiki/0.0_QuickStart)\n## 架构\u0026文档\nSee the page for introduction: [架构\u0026文档](https://github.com/ucarGroup/DataLink/wiki/1.0_DataLink%E6%80%BB%E4%BD%93%E6%9E%B6%E6%9E%84)\n## 常见问题\nSee the page for FAQ: [FAQ](https://github.com/ucarGroup/DataLink/wiki/FAQ)\n## 类似开源\ncanal：http://github.com/alibaba/canal\u003cbr\u003e\notter：https://github.com/alibaba/otter\u003cbr\u003e\nKafka-Connect：https://github.com/apache/kafka\u003cbr\u003e\nDataBus：https://github.com/linkedin/databus\n## 版本历史\n当前最新版本为1.0.2-beta，版本发布历史如下：\u003cbr\u003e\nhttps://github.com/ucarGroup/DataLink/releases/\n## 问题反馈\n目前有关DataLink的问题交流方式有如下几种，欢迎各位加入进行技术讨论。\u003cbr\u003e\nqq交流群： 758937055\u003cbr\u003e\n邮件交流： tech_plat_data@ucarinc.com\u003cbr\u003e\n报告issue：[issues](https://github.com/ucarGroup/DataLink/issues)\u003cbr\u003e\n![](https://github.com/ucarGroup/DataLink/blob/master/img/DataLink%E4%BA%A4%E6%B5%81%E7%BE%A4%E7%BE%A4%E4%BA%8C%E7%BB%B4%E7%A0%81.png)\n\u003cpre name=\"code\" class=\"java\"\u003e\n\u003c/pre\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FucarGroup%2FDataLink","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FucarGroup%2FDataLink","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FucarGroup%2FDataLink/lists"}