{"id":28493642,"url":"https://github.com/dromara/datacompare","last_synced_at":"2025-07-08T11:30:54.951Z","repository":{"id":50296650,"uuid":"518715119","full_name":"dromara/dataCompare","owner":"dromara","description":"big data comparison and data profiling platform: low code，data comparison and data profiling","archived":false,"fork":false,"pushed_at":"2024-04-16T11:20:18.000Z","size":11378,"stargazers_count":265,"open_issues_count":5,"forks_count":62,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-06-08T09:09:44.818Z","etag":null,"topics":["bigdata","doris","hive","low-code","spark"],"latest_commit_sha":null,"homepage":"https://www.reddit.com/r/bigdata/comments/zlm2mw/design_and_practice_of_open_source_big_data/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dromara.png","metadata":{"files":{"readme":"README-CN.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-07-28T05:40:05.000Z","updated_at":"2025-05-30T07:55:10.000Z","dependencies_parsed_at":"2024-04-16T12:44:52.849Z","dependency_job_id":null,"html_url":"https://github.com/dromara/dataCompare","commit_stats":null,"previous_names":["zhugezifang/datacompare"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/dromara/dataCompare","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dromara%2FdataCompare","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dromara%2FdataCompare/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dromara%2FdataCompare/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dromara%2FdataCompare/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dromara","download_url":"https://codeload.github.com/dromara/dataCompare/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dromara%2FdataCompare/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264259592,"owners_count":23580840,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigdata","doris","hive","low-code","spark"],"created_at":"2025-06-08T09:09:51.389Z","updated_at":"2025-07-08T11:30:54.946Z","avatar_url":"https://github.com/dromara.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dataCompare\n![](https://gitee.com/ZhuGeZiFang/data-compare/badge/star.svg)\n![](https://gitee.com/ZhuGeZiFang/data-compare/badge/fork.svg?theme=gvp)\n![](https://img.shields.io/github/stars/zhugezifang/dataCompare.svg?logo=GitHub)\n![](https://img.shields.io/github/forks/zhugezifang/dataCompare.svg?logo=GitHub)\n![](https://img.shields.io/github/watchers/zhugezifang/dataCompare.svg?logo=GitHub)\n![](https://img.shields.io/github/license/zhugezifang/dataCompare.svg)\n![](https://img.shields.io/github/v/release/zhugezifang/dataCompare?label=latest\u0026style=flat-square)\n\n[![EN doc](https://img.shields.io/badge/document-English-blue.svg)](README.md)\n[![CN doc](https://img.shields.io/badge/文档-中文版-blue.svg)](README-CN.md)\n\n#### 介绍\ndataCompare 是一个大数据数据比对和数据探测平台\n\n(1)支持hive表数据比对，mysql、Doris 数据比对，实现自动化配置进行数据比对，避免频繁写sql 进行处理\n\n(2)支持少量配置即可实现数据探测\n\n![image](https://user-images.githubusercontent.com/28300167/207563954-6e3dba02-84de-4881-9a23-371b88ed5b1e.png)\n\n![image](https://user-images.githubusercontent.com/28300167/226346775-f1c1ed2d-8370-45db-878c-8ab81d9e402e.png)\n\n#### 功能介绍\n\n数据对比：\n\n(1)界面级交互数据对比任务配置，低代码少量配置快速生成对比任务\n\n(2)量级对比、一致性对比、自动化差异case发现\n\n(3)目前已经支持MySQL、Apache Hive、Apache Doris 等JDBC 数据库\n\n(4)已经支持对比结果自动发送邮件告警报告\n\n数据探针:\n\n(1)低代码、少量配置即可完成数据探测\n\n(2)主键、枚举值、空值探测\n#### 软件架构\n\n![输入图片说明](image77.png)\n\n技术栈:\n\n后端：Spring boot + Mybatis\n\n数据库:MySQL\n\n解析引擎：Antrl\n\n数据存储引擎、计算引擎：Hive、Spark 等\n\n\n#### 系统流程图\n\n![输入图片说明](image1.png)\n![输入图片说明](image2.png)\n![输入图片说明](image3.png)\n![输入图片说明](image4.png)\n\n![img_1.png](img_1.png)\n\n![img.png](img.png)\n\n\n#### 系统功能演示\n系统主页\n![image](https://user-images.githubusercontent.com/28300167/207257662-273fc531-c21e-437a-9d20-f15a533b58bd.png)\n\n(1)数据对比功能展示\n\n数据库配置\n\nmysql配置\n\n![image](https://user-images.githubusercontent.com/28300167/207256310-8c6d0be4-90c2-4a71-a49b-c54d3537a7bf.png)\n\nhive配置\n\n![image](https://user-images.githubusercontent.com/28300167/207497891-8dc317f6-06f3-4d53-96d5-400586e0a488.png)\n\n\njob配置\n![image](https://user-images.githubusercontent.com/28300167/207256145-7ce5eaa2-7030-4c2c-91d9-3e566162e91e.png)\n\n对比结果展示\n![image](https://user-images.githubusercontent.com/28300167/208607718-0767ff93-223a-408d-a586-7d509f278197.png)\n\n![image](https://user-images.githubusercontent.com/28300167/207259977-fd2258a8-fce1-4a3b-85a3-2b6213e3b0a7.png)\n\n![3fd83de9c582347f7f88cc82f438db4](https://user-images.githubusercontent.com/28300167/208607767-94cffce0-30f3-45ec-a280-978964e153bb.png)\n\n(2)数据探针\n\njob配置\n\n![img_2.png](img_2.png)\n\n探针结果\n\n![img_3.png](img_3.png)\n\n![img_4.png](img_4.png)\n\n![image](https://user-images.githubusercontent.com/28300167/229501053-4e33b6fb-851a-4fb2-9b19-16308359f57e.png)\n\n#### 系统运行\n系统运行环境要求：\n\njava jdk8\n\nmysql 5.7.36\n\n运行配置\n\n(1)将sql目录中的sql文件进行运行在数据库，创建库和表\n\n(2)在根目录下面进行mvn clean package,使用target 目录下的dataCompare.jar\n\n(3)修改数据库配置信息 application.yml，将数据库连接信息改为步骤(1)的数据库连接信息\n![47d1145d147214348d6d0f2fc599ea7](https://user-images.githubusercontent.com/28300167/219598181-5731e845-a49e-43af-a96c-db6d6148b11a.png)\n\n(4)运行 java -jar -Dspring.config.location=application.yml dataCompare.jar (application.yml和jar在同一目录下)\n\n(5)访问 http://127.0.0.1/ 即可 (账号:admin 密码:admin123)\n\n环境安装配置\n\n(1)如果要实现Hive数据对比配置需要先安装Hive 环境(安装文档参考docker快速安装Hive环境：https://blog.csdn.net/ifenggege/article/details/107860477)\n\n(2)安装好之后新建数据源连接时选择Hive，地址为：jdbc:hive2://ip:10000 \n\n\n#### 技术交流\n![image](https://user-images.githubusercontent.com/28300167/207255900-152d6834-9602-4ada-91ca-ad9906d89bf8.png)\n\n\n#### 致谢\n感谢ruoyi 提供前端服务\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdromara%2Fdatacompare","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdromara%2Fdatacompare","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdromara%2Fdatacompare/lists"}