{"id":33179048,"url":"https://github.com/shaiwz/data-platform-open","last_synced_at":"2025-11-20T21:03:16.309Z","repository":{"id":290228365,"uuid":"918934944","full_name":"shaiwz/data-platform-open","owner":"shaiwz","description":"🔥🔥🔥可视化拖拽式大数据集成平台、大数据平台、大数据，包含数据流、数据源、数据对齐、查询模板、完善的监控等。像画流程图一样且无代码方式同步、清洗数据。","archived":false,"fork":false,"pushed_at":"2025-10-24T10:59:09.000Z","size":5182,"stargazers_count":1077,"open_issues_count":1,"forks_count":97,"subscribers_count":46,"default_branch":"master","last_synced_at":"2025-10-24T12:35:52.809Z","etag":null,"topics":["big-data","dataflow","doris","java","kafka","starrocks"],"latest_commit_sha":null,"homepage":"https://shaiwz.com/home","language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shaiwz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-01-19T09:13:21.000Z","updated_at":"2025-10-24T10:59:13.000Z","dependencies_parsed_at":"2025-04-27T18:20:52.539Z","dependency_job_id":"b44ac8e1-28f4-4389-b8e7-0772bddffc12","html_url":"https://github.com/shaiwz/data-platform-open","commit_stats":null,"previous_names":["shaiwz/data-platform-open"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/shaiwz/data-platform-open","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shaiwz%2Fdata-platform-open","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shaiwz%2Fdata-platform-open/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shaiwz%2Fdata-platform-open/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shaiwz%2Fdata-platform-open/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shaiwz","download_url":"https://codeload.github.com/shaiwz/data-platform-open/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shaiwz%2Fdata-platform-open/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285511775,"owners_count":27184237,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-20T02:00:05.334Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","dataflow","doris","java","kafka","starrocks"],"created_at":"2025-11-16T03:00:36.818Z","updated_at":"2025-11-20T21:03:16.302Z","avatar_url":"https://github.com/shaiwz.png","language":"Java","funding_links":[],"categories":["大数据"],"sub_categories":[],"readme":"# 📌可视化拖拽式大数据集成平台后端 📌\n\n# 我正在参加 Gitee 2025 最受欢迎的开源软件投票活动，快来给我投票吧！https://gitee.com/activity/2025opensource?ident=IFIWR2\n\n#### 以数据为舟，驭流而上，让大数据如流水般自由汇聚、智能流转，零代码构建您的数字未来！\n\n## 关注Star不迷路，我们的目标是在一个平台上完成对数据的所有操作。\n\n## 本项目已申请外观专利+发明专利，未经明确书面授权，任何单位或个人不得对本软件进行抄袭、复制、修改、分发、逆向工程、商业用途等任何形式的非法使用。违者将面临人民币 100 万元的法定罚款及可能的法律追责。举报侵权行为可获得实际罚款金额 40% 的现金奖励。\n\n官网：https://shaiwz.com/home \u003cbr\u003e\n前端代码地址：https://gitee.com/shaiwz/data-platform-front-open \u003cbr\u003e\n项目展示地址：http://dataplatform.cn/login\n\n##### 举报、咨询联系方式：\n\n- QQ：761945125\n- Email：761945125@qq.com\n\n## 页面预览\n\n### 首页\n\n综合看板，方便查看系统运行情况，数据一致性，数据源健康率，以及告警信息等。\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/oO9691.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n#### 数据流统计\n\n用来监控数据流的运行情况，数据流的运行情况，以及数据流服务器的压力。\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/mYlRE2.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n点击处理器、或者内存时展示服务器运行信息\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/7Kf1ko.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n同时可以在服务监控中看具体的服务器信息\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/SC1x8z.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n#### 查询模板统计\n\n用来查看查询模板的调用情况，查询服务器运行信息\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/CZHc2H.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n### 数据管理\n\n#### 数据流\n\n支持随意构建数据流向，拖拖拽拽完成数据同步，清洗，过滤等任何操作，全程不需要写任何代码。\n\n##### 数据流列表\n\n查看有哪些数据流任务\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/dpa9WI.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n##### 数据流画布\n\n支持在一个画布里面完成数据同步，清洗，过滤、报表数据生成等任何操作，全程不需要写任何代码。\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/qXGKgj.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n下游节点支持指定执行顺序。\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/HXT80M.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n支持接收FlinkCDC、Canal消息，基于策略模式可以随机接入其他平台的数据，无缝、快速对接，数据发送方不需要任何改动。\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/1D3nSp.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n发布服务器资源可控，自动容错，节点支持动态扩容，服务节点越多，性能越好。支持多版本，支持回滚。\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/G4Lydk.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n##### 数据流实时运行日志\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/4EtS6z.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n##### 数据流发布记录\n\n\u003cimg src=\"https://oss-xuxin.oss-cn-beijing.aliyuncs.com/dp/img/NMdFJ5.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n#### 数据源\n\n支持若干种数据源，插件方式，便于扩展其他数据源\n\n##### 数据源创建\n\n支持分表算法、健康检查、脱敏配置\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/QoEp4g.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n##### 数据源控制台\n\n不同数据源支持不同的控制台，例如MySQL、StarRocks、Doris、Oracle等控制台样式如下\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/CcV9Uf.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\nKafka控制台\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/cq7I6f.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/rN7kLO.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\nElasticSearch控制台\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/vGE5zJ.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n等等其他控制台\n\n#### 数据对齐\n\n通过页面简易配置即可能完成两个任意数据源、不同类型数据源数据表、索引等一致性校验，告警、并支持自动化修复！\n\n##### 数据对齐列表\n\n这里方便查看有哪些对齐任务\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/BsQfTK.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n##### 数据对齐创建\n\n配置何时触发，以及策略对齐策略：数量一致、内容一致、随机数量，对比时间范围等等配置。\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/Y66Y0S.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n##### 数据对齐日志\n\n开源快速方便查看任务的运行情况\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/di6jt7.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n##### 数据对齐日志详情\n\n针对某次对齐任务，不一致的数据快速查看，并提示出差异字段等。\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/eqhKvQ.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n#### 查询模板\n\n通过查询模板可以快速创建查询任务，支持多种数据源。不用后端开发再去编写查询报表、对外提供数据查询接口。\n\n##### 查询模板列表\n\n支持版本控制、密钥、权限配置等\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/xLF1a7.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n##### 查询模板创建\n\n支持动态条件\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/iQg63A.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n支持限流、记录日志、缓存等配置\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/smOzXG.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n预览页面有示例文档，以及快速测试，测试没问题可以点击发布，外部服务即可调用。\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/MDjMXp.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n##### 查询模板调用日志\n\n支持完善的日志记录，日志可跟踪。\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/PTHd2a.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n日志详情\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/S2exjI.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n#### 等等还有若干功能，欢迎联系作者进行体验\n\n## 性能测试\n\n| 模式     | 硬件规格  | 实例数量 | 速率（单任务） |\n|--------|-------|------|---------|\n| 监听-流处理 | 6c12g | 1    | 30058/s |\n| 批处理    | 6c12g | 1    | 60268/s |\n\n## 整体架构\n\n所有服务实例均支持动态缩、扩容\n\n\u003cimg src=\"https://oss-xuxin.oss-accelerate.aliyuncs.com/dp/img/EVYZif.png\" alt=\"none\" style=\"width: 60%;height: 60%;border-radius: 6px;\"\u003e\n\n## 业务价值\n\n| 同步方案          | 人员最低要求     | 单表预估工时（同步+过滤+转换） |\n|---------------|------------|------------------|\n| 硬编码方式查询-过滤-写入 | 中高级开发人员    | 1人天              |\n| FlinkCDC      | 专业大数据源开发人员 | 0.5人天            |\n| Canal         | 中高级开发人员    | 0.5人天            |\n| DataX         | 中高级开发人员    | 0.5人天            |\n| 数据流平台         | 初级开发或初级运维  | 10分钟（拖拖拽拽）       |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshaiwz%2Fdata-platform-open","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshaiwz%2Fdata-platform-open","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshaiwz%2Fdata-platform-open/lists"}