{"id":49815505,"url":"https://github.com/datavane/consilens","last_synced_at":"2026-05-13T10:01:34.961Z","repository":{"id":354120537,"uuid":"1209032087","full_name":"datavane/consilens","owner":"datavane","description":"跨数据源数据一致性校验工具","archived":false,"fork":false,"pushed_at":"2026-05-12T07:27:30.000Z","size":2330,"stargazers_count":26,"open_issues_count":2,"forks_count":7,"subscribers_count":1,"default_branch":"dev","last_synced_at":"2026-05-12T09:31:48.053Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datavane.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-13T03:00:55.000Z","updated_at":"2026-05-12T07:27:34.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/datavane/consilens","commit_stats":null,"previous_names":["noeticlens/consilens","datavane/consilens"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/datavane/consilens","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datavane%2Fconsilens","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datavane%2Fconsilens/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datavane%2Fconsilens/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datavane%2Fconsilens/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datavane","download_url":"https://codeload.github.com/datavane/consilens/tar.gz/refs/heads/dev","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datavane%2Fconsilens/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32977273,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-13T06:31:55.726Z","status":"ssl_error","status_checked_at":"2026-05-13T06:31:51.336Z","response_time":115,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-13T06:00:43.983Z","updated_at":"2026-05-13T10:01:34.953Z","avatar_url":"https://github.com/datavane.png","language":"Java","funding_links":[],"categories":["数据库中间件"],"sub_categories":["微服务框架"],"readme":"# Consilens\n\n\u003e 跨数据源数据一致性校验工具。\n\nConsilens 用于验证数据同步、迁移、ETL、回填后的结果是否一致，并输出行级、列级差异。\n\n适合这类场景：\n\n- MySQL → PostgreSQL / ClickHouse / StarRocks 迁移验收\n- 数据同步链路校验\n- ETL / 回填结果核对\n- 双写 / 灰度切换对账\n\n## 核心特性\n\n- 跨数据源比对，不要求两侧数据集在同一个实例\n- 基于 connector/dataset 抽象，架构可扩展至 ES、MongoDB、HDFS 等非关系型数据源\n- 大表优先走 checksum 收敛，差异可定位到主键和字段\n- 支持多格式 sink 输出：控制台、JSON 文件、CSV 文件、结果表\n- 基于 SPI 的连接器插件扩展\n\n## 内置连接器\n\n| 数据库 | 连接器模块 | 验证状态 |\n| --- | --- | --- |\n| MySQL | `consilens-connector-mysql` | 已验证 |\n| PostgreSQL | `consilens-connector-postgresql` | 已验证 |\n| SQL Server | `consilens-connector-sqlserver` | 内置（待验证） |\n| Oracle | `consilens-connector-oracle` | 内置（待验证） |\n| ClickHouse | `consilens-connector-clickhouse` | 内置（待验证） |\n| Doris | `consilens-connector-doris` | 内置（待验证） |\n| StarRocks | `consilens-connector-starrocks` | 已验证 |\n| Presto | `consilens-connector-presto` | 内置（待验证） |\n| Trino | `consilens-connector-trino` | 内置（待验证） |\n| TiDB | `consilens-connector-tidb` | 内置（待验证） |\n\n说明：`内置（待验证）` 表示仓库里已有对应连接器模块，但 README 目前只把已完成端到端验证的数据库作为对外首推能力。\n\n## 工作方式\n\n```text\n边界探测\n  ↓\n首轮分段\n  ↓\nchecksum 比对\n  ↓\n一致段跳过\n  ↓\n差异段继续收敛\n  ↓\n小段本地精确比较\n```\n\n## 快速开始\n\n### 构建\n\n```bash\nmvn clean package -DskipTests -Prelease\n```\n\n### 执行\n\n```bash\n./bin/consilens-cli.sh diff -c config.yaml\n```\n\n### 最小配置示例\n\n```yaml\nsource:\n  type: mysql\n  connection:\n    url: jdbc:mysql://localhost:3306/source_db\n    username: ${env.MYSQL_USER}\n    password: ${env.MYSQL_PASSWORD}\n  resource:\n    type: table\n    name: orders\n\ntarget:\n  type: postgresql\n  connection:\n    url: jdbc:postgresql://localhost:5432/target_db?currentSchema=public\n    username: ${env.PG_USER}\n    password: ${env.PG_PASSWORD}\n  resource:\n    type: table\n    name: orders\n\ncomparison:\n  keys:\n    source:\n      - order_id\n    target:\n      - order_id\n  fields:\n    source:\n      - order_id\n      - customer_id\n      - amount\n      - status\n      - created_at\n    target:\n      - order_id\n      - customer_id\n      - amount\n      - status\n      - created_at\n\nstrategy:\n  mode: checksum\n  algorithm: xor\n  bisectionFactor: 8\n  bisectionThreshold: 5000\n  batchSize: 1000\n  enableProfiling: false\n\nresult:\n  sinks:\n    - format: console\n      type: result\n    - format: json\n      type: diff-record\n      properties:\n        path: ./diff_results.json\n        pretty: true\n```\n\n请将上面的用户名和密码替换成真实值。配置支持环境变量占位符，语法为 `${env.VAR_NAME}` 或 `${env.VAR_NAME:默认值}`。\n\n## 策略\n\n| 策略 | 状态 | 说明 |\n| --- | --- | --- |\n| `checksum` | 已实现 | 推荐默认使用，支持跨数据源 |\n| `join` | 已实现 | 仅支持同一个 JDBC URL |\n\n## 文档\n\n- [快速开始](./docs/01-快速开始.md)\n- [配置详解](./docs/02-配置详解.md)\n- [连接器与数据源支持](./docs/03-插件与数据库支持.md)\n- [架构设计](./docs/04-架构设计.md)\n- [开发指南](./docs/05-开发指南.md)\n\n## 社交媒体\n\n- 微信公众号，扫描二维码关注\n\n![微信二维码](docs/img/wechat-qrcode.png)\n\n## 联系作者\n\n- 添加时备注：Consilens\n\n![wechat-author-qrcode](docs/img/wechat-author-qrcode.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatavane%2Fconsilens","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatavane%2Fconsilens","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatavane%2Fconsilens/lists"}