{"id":23467348,"url":"https://github.com/vipshop/drc","last_synced_at":"2025-08-26T23:33:09.086Z","repository":{"id":57507776,"uuid":"159437612","full_name":"vipshop/drc","owner":"vipshop","description":"MySQL active-active replication solution.","archived":true,"fork":false,"pushed_at":"2018-12-13T14:46:41.000Z","size":1574,"stargazers_count":127,"open_issues_count":2,"forks_count":35,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-14T12:27:52.299Z","etag":null,"topics":["high-availability","multi-master","mysql","replication"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vipshop.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-11-28T03:23:34.000Z","updated_at":"2025-04-07T23:07:02.000Z","dependencies_parsed_at":"2022-09-19T19:20:59.144Z","dependency_job_id":null,"html_url":"https://github.com/vipshop/drc","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vipshop/drc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vipshop%2Fdrc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vipshop%2Fdrc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vipshop%2Fdrc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vipshop%2Fdrc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vipshop","download_url":"https://codeload.github.com/vipshop/drc/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vipshop%2Fdrc/sbom","scorecard":{"id":923048,"data":{"date":"2025-08-11","repo":{"name":"github.com/vipshop/drc","commit":"66c35ad67223a70adadc4c68419d001dc83cd354"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"project is archived","details":["Warn: Repository is archived."],"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 0/7 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-25T06:30:15.214Z","repository_id":57507776,"created_at":"2025-08-25T06:30:15.214Z","updated_at":"2025-08-25T06:30:15.214Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272267230,"owners_count":24903642,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-26T02:00:07.904Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["high-availability","multi-master","mysql","replication"],"created_at":"2024-12-24T12:30:39.888Z","updated_at":"2025-08-26T23:33:08.343Z","avatar_url":"https://github.com/vipshop.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 关于DRC\nDRC(Data Replication Center)是唯品会自研的MySQL双向复制方案，主要应用场景是数据库双向复制、单向复制。\n\n目前DRC在唯品会还未正式使用，正在逐步验证，功能仍在完善中，目前可能存在一些未知问题，如果你需要使用DRC，请评估好相应风险。\n\nDRC在[实时数据管道(RDP)](https://github.com/vipshop/RDP)的基础上，实现了以下功能：\n* 支持MySQL实例的数据双向复制。\n* 规避双向复制导致的循环复制。\n* 检测异常情况下导致的数据冲突，并能自动处理, 保证两端的数据`最终一致`。\n* 支持多线程重放binlog数据(MTS算法), 保证复制的吞吐量。\n* 支持数据对账, 检验双向复制正确性。\n\n# 总体架构\n\n以双向复制的场景为例，MySQL复制拓扑可以抽象为下图所示：\n\n![active_active](/docs/picture/active_active.png)\n\n图中，MySQL分别部署在IDC_1和IDC_2，彼此相互同步变更事件（Change）。其中同步的工作由DRC来完成。\n\n\n\nDRC总体架构如下所示：\n\n![drc_architecture](/docs/picture/drc_architecture2.png)\n\n图中，S0和S1是DB数据分片。\n\n经过api_router流量分发后，写分片S0的流量由IDC1负责，写分片S1的流量由IDC2负责。\nIDC1与IDC2之间的数据通过两个DRC实例进行同步。\n\n\u003e 数据分片可以对应不同的子表，也可以是对应同一个表中的不同范围。\n\n\n\n单个DRC实例的架构如下所示：\n\n![drc_instance](/docs/picture/drc_instance.png)\n\nDRC实例包含三个组件：\n* RDP：实时拉取源DB的binlog，解析之后的事务数据写入到Kafka。\n* Kafka：存储解析之后的事务数据。\n* Applier：订阅Kafka中的事务数据，通过执行SQL语句的方式，重放到目标DB。\n\n# 关于Applier\nApplier是DRC中的主要组件，订阅来自RDP解析后的binlog数据，构造SQL语句后在目标MySQL执行。\n\nApplier的主要功能有：SQL构造, 并行执行，冲突检测与处理，过滤循环复制的数据，中断恢复，重复数据幂等处理。\n\n\n# 限制条件\n* MySQL版本必须在5.7.19及以上(因为MySQL 5.7.19之前的MTS算法存在BUG)。\n* MySQL需要开启GTID, 并设置binlog_format=ROW, binlog_row_image=FULL。\n* 业务数据库必须有主键字段(Applier处理数据冲突时依赖记录的主键)。\n\n\n# 快速开始\n## 1. 安装\n### 源码安装\n在源码根目录执行make，即可在build/mysql_applier/bin目录生成Applier可执行文件，同时产生实例配置文件：build/mysql-applier/etc/applier.ini.example。\n```\n[apps@localhost drc]$ make\n[apps@localhost drc]$ ls build/mysql-applier/bin/\nalarm.sh mysql-applier\n[apps@localhost drc]$ ls build/mysql-applier/etc/\napplier.ini.example\n\n```\n\n\n## 2. 修改配置\n执行以下命令，修改Applier依赖的配置文件。\n\n```\ncp build/mysql-applier/etc \u0026\u0026 cp applier.ini.example applier.ini\nvi applier.ini \n```\n\n修改Kafka连接信息, 包括：brokerlist、订阅topic、partition和version。\n```\n[kafka]\n#The kafka address of applier to fetch binlog event\nbrokerlist = 10.10.10.1:9092,10.10.10.2:9092,10.10.10.3:9092\ntopic = test_topic_1\npartition = 0\nversion = 1.1.0\n```\n\u003e 该Kafka topic中的数据是由RDP生成的，可以认为是源DB的binlog流。\n\n修改目标DB连接信息。\n```\n[mysql]\n#The mysql address of applier to write\nhost = localhost\nport = 3306\nuser = root\npasswd = 123456\n```\n\n\u003e 注意：目标DB需要是源DB的历史镜像，后续通过Applier重放源DB的增量binlog。\n\n\u003e 注意：目标DB的所需最低权限是：SELECT , INSERT, UPDATE, DELETE, CREATE, DROP, ALTER,  REFERENCES, INDEX \n\n修改Zookeeper连接信息,包括：Zookeeper地址、zk节点路径（多个Applier副本依赖zk进行选主）。\n```\n[zk]\nzk_addr_list = 10.10.10.1:2181,10.10.10.2:2181,10.10.10.3:2181\nzk_root = /drc/mysql_applier/1000x\n```\n\n修改冲突处理方式，这里选择overwrite，表示如果复制过程中出现冲突，直接覆盖目标DB的记录。\n\n当然，也可以选择time_overwrite, 表示基于时间戳来进行冲突处理，但是前提是表结构需要有特定的时间字段。\n\n```\n#The strategy to handle conflict, may be time_ignore|time_overwrite|ignore|overwrite\n#if handle conflict base on time,need set the column name of row update time\nhandle_conflict_strategy=overwrite\nupdate_time_column = update_time\n```\n\u003e 对于线上使用，需要审慎选择冲突处理的方式，如果是双向复制，必须两个方向的处理方式不同：比如一边是overwrite，另一边是ignore。\n\n## 3. 启动\n假设配置文件build/mysql-applier/etc/applier.ini已经修改完成，那么执行以下命令启动Applier：\n\n```\nmkdir -p build/mysql_applier/logs\nmkdir -p build/mysql_applier/metrics\ncd build/mysql_applier/bin\n./mysql-applier \n```\n可以通过查看build/mysql_applier/logs目录下的日志文件，检查是否有ERROR信息。\n如果有ERROR信息，请检查配置文件是否配置妥当。\n\n## 4. 测试\n在源DB执行DML或者DDL，检查是否在目标DB重放成功。\n\n\u003e 上述是单向复制搭建的步骤，双向复制搭建也是类似但更复杂些，建议用户根据自身需求，打造自动化搭建流程。\n\n# 性能\n关于DRC Applier的性能，请查看文档: [DRC性能概览](./docs/drc_perf.md)。\n\n# 监控特性\n关于DRC Applier的暴露的监控特性，请查看文档: [监控特性](./docs/applier_metrics.md)。\n\n# 异常告警\n关于DRC Applier运行过程中的异常告警，请查看文档：[异常告警](./docs/alarm_introduce.md)。\n\n# 管理API\n关于DRC Applier的提供的http管理API，请查看文档: [管理API](./docs/applier_api.md)。\n\n# 数据对账\nDRC提供了对账工具，用于比对两端数据是否一致，请查看文档: [数据对账](./docs/table_checksum.md)。\n\n# 开发团队\nDRC项目由唯品会基础架构部—数据中间件组开发和维护。开发成员如下：\n- [陈非](https://github.com/flike)\n- [赵百忠](https://github.com/firnsan)\n- [汤锦平](https://github.com/tom-tangjp)\n- [范力彪](https://github.com/libiaofan)\n\n# License\nDRC项目遵循Apache 2.0 license。\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvipshop%2Fdrc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvipshop%2Fdrc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvipshop%2Fdrc/lists"}