{"id":13458555,"url":"https://github.com/zhuchao941/canal2sql","last_synced_at":"2026-01-15T01:30:54.512Z","repository":{"id":212443331,"uuid":"731485128","full_name":"zhuchao941/canal2sql","owner":"zhuchao941","description":"同时支持在线和离线解析binlog，输出标准sql","archived":false,"fork":false,"pushed_at":"2025-02-06T08:50:00.000Z","size":91,"stargazers_count":142,"open_issues_count":3,"forks_count":38,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T15:42:50.586Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zhuchao941.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-14T07:31:41.000Z","updated_at":"2025-03-17T04:52:28.000Z","dependencies_parsed_at":"2024-10-29T03:31:15.003Z","dependency_job_id":"fe5bbdf8-54f6-4e00-a392-7fbeeb0b287c","html_url":"https://github.com/zhuchao941/canal2sql","commit_stats":null,"previous_names":["zhuchao941/canal2sql"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/zhuchao941/canal2sql","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhuchao941%2Fcanal2sql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhuchao941%2Fcanal2sql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhuchao941%2Fcanal2sql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhuchao941%2Fcanal2sql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zhuchao941","download_url":"https://codeload.github.com/zhuchao941/canal2sql/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhuchao941%2Fcanal2sql/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28441031,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-15T00:55:22.719Z","status":"ssl_error","status_checked_at":"2026-01-15T00:55:20.945Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T09:00:53.784Z","updated_at":"2026-01-15T01:30:54.497Z","avatar_url":"https://github.com/zhuchao941.png","language":"Java","funding_links":[],"categories":["Java","数据库中间件"],"sub_categories":[],"readme":"# canal2sql\n\n用Java写的 MySQL Binlog 解析工具，底层依赖了Canal\n\n## 设计初衷\n\n1. 为了方便的找到数据的更新轨迹\n2. 为了填补Java在解析离线binlog的空白\n\n### binlog读取模式\n\n暂时支持了三种读取模式\n\n- 在线模式（online）：指定远程数据库地址，实时从对应数据库获取binlog并解析\n- binlog文件模式（file）：离线模式，指定binlog文件url（支持http/file等），读取对应文件并解析\n- 阿里云rds模式（aliyun）：离线模式，指定对应的时间范围以及实例id（instanceId），读取符合条件的binlog文件列表并解析\n\n### 列补全方式\n\n我们知道binlog里row_event和table_map是没有字段名的，所以需要借助外部元数据来补全，canal2sql支持两种列补全方式\n\n- 本地DDL文件：指定本地DDL文件，来获取表结构数据，从而补全binlog里的列信息\n- 数据库实时DDL：指定数据库地址信息，直接用实时表结构来补全binlog里的列信息\n\n在线模式只支持使用数据库实时DDL的方式，而binlog文件模式和阿里云rds模式则可以根据实际情况选择合适的列补全方式\n\n### 限制和要求\n\n- MySQL必须开启binlog\n- binlog_format = row\n- binlog_row_image = full\n- 如果用在线解析binlog，对应用户要有SELECT, REPLICATION SLAVE, REPLICATION CLIENT权限\n\n## 可执行jar包下载\n\n最新版本v1.1.5 可在releases页面下载 \nhttps://github.com/zhuchao941/canal2sql/releases/tag/v1.1.5\n\n## 运行选项\n\n**binlog模式指定**\n\n- -mode online/file/aliyun，默认online\n\n**SQL显示模式**\n\n- -A, --append   追加模式，支持原SQL和回滚SQL同时显示，可选，默认true\n- -B, --rollback 显示回滚SQL（如在追加模式，则原SQL也会显示，作为注释添加在同一行），可选，默认false\n- -M, --minimal  最小化insert/delete语句，如原语句是batchInsert和batchDelete的，那么会尽量按照原批量语句输出，可选， 默认false（v1.1.0版本新增）\n- -C, --clean    纯净模式，只输出SQL语句和耗时，适用于直接捞出SQL回滚的场景（可结合`tac`命令或者`tail -r`倒序输出）（v1.1.5版本新增）\n\n**解析范围控制**\n\n- -start_position 起始解析位置。可选。\n  - 对于在线模式，需要指定起始文件+position，如'mysql-bin.007756|52416008'\n  - 对于离线模式，只有position\n\n- -end_position 终止解析位置。可选\n  - 对于在线模式，需要指定终止文件+position，如'mysql-bin.007756|52416008'\n  - 对于离线模式，只有position\n\n- -start_time 起始解析时间，格式'yyyy-MM-dd HH:mm:ss'。可选\n\n- -end_time 终止解析时间，格式'yyyy-MM-dd HH:mm:ss'。可选\n\n**对象过滤**\n\n- -filter 库表维度过滤（白名单），如：a库下所有表a.*，多个规则英文逗号(,)隔开\n\n- -black_filter 库表维度过滤（黑名单），语法和filter一致\n\n- -sql-type 只解析指定类型，支持 insert,update,delete,ddl。多个类型用逗号隔开，如--sql-type=insert,delete。可选。默认为insert,update,delete,ddl\n\n**离线解析模式**\n\n- -ddl 本地DDL文件，用来还原列\n- -file_url 离线binlog文件，支持各种协议，填入可访问url即可\n\n**mysql连接配置**\n\n在线解析必填，离线模式作为表结构来源用来做列还原\n\n```\n -h host\n -P port\n -u user\n -p password\n```\n\n**云rds配置**\n\n- -instanceId 云数据库实例instanceId，阿里云是主从库公用一个instanceId，所以还要指定对应的编号，用\"|\"隔开，如'rm-bp13bt768ubdz4eb2|26588881'\n- -ak AccessKey\n- -sk SecretKey\n- -external 公网模式，公网下载binlog要收费，而内网下载binlog不收钱且速度更快，默认为内网模式\n- -start_time 起始解析时间，格式'yyyy-MM-dd HH:mm:ss'。此模式下必传\n- -end_time 终止解析时间，格式'yyyy-MM-dd HH:mm:ss'。此模式下必传\n\n## 使用示例\n\n### 1. 在线模式\n```sh\njava -jar ./canal2sql-${version}.jar -mode online -uroot -proot -P3306 -hlocalhost\n```\n\n在线解析模式只需要指定远程数据库连接参数即可，表结构数据直接实时获取\n\n### 2. binlog文件模式\n\nbinlog文件模式需要指定binlog文件url，表结构数据支持从mysql实时获取也支持从本地ddl文件读取\n\n```sh\njava -jar ./canal2sql-${version}.jar -mode file -file_url 'file:/Users/abc/binlog/mysql-bin.000474' -uroot -proot -P3306 -hlocalhost\njava -jar ./canal2sql-${version}.jar -mode file -ddl '/Users/xxx/ddl.sql' -file_url 'http://localhost:8080/binlog/mysql-bin.000474'\n```\n\n### 3. 阿里云rds模式\n\n阿里云rds模式\n\n```\njava -jar ./canal2sql-${version}.jar -mode aliyun -uroot -proot -P3306 -hlocalhost -start_time \"2023-12-07 00:00:00\" -end_time \"2023-12-07 09:00:00\" -instanceId \"rm-12345678|1111111\" -ak ak -sk sk\n```\n\n## 输出格式解析\n\n类似如下输出，第一行为 binlog文件名称:事务开始的offset 时间戳\n后面就是具体的事务里的sql语句，同一个事务会被聚合到一起输出\n\n```sql\n#mysql-bin.007795:171260617 2023-12-14 16:27:00\nUPDATE `comment`.`comment_category` SET `upd_tm` = '2023-12-14 16:27:00' WHERE `id` = 1;\nUPDATE `comment`.`comment_category` SET `upd_tm` = '2023-12-14 16:27:00' WHERE `id` = 12;\n\n#mysql-bin.007795:171261166 2023-12-14 16:27:00\nUPDATE `comment`.`comment_category` SET `upd_tm` = '2023-12-14 16:27:00' WHERE `id` = 9;\n```\n\n## RoadMap\n\n1. ~~支持binlog的在线解析、离线解析~~（已支持）\n2. ~~列还原同时支持在线模式和离线模式~~（已支持）\n3. ~~支持生成回滚SQL~~（已支持）\n4. ~~支持解析范围控制~~（已支持）\n    1. ~~时间维度~~\n    2. ~~position维度~~\n5. ~~支持对象过滤~~（已支持）\n    1. ~~库表维度~~\n    2. ~~sql类型维度~~\n6. 支持多binlog文件顺序解析\n7. ~~支持云rds binlog文件列表解析~~（已支持）\n8. 支持分析统计功能\n    1. 耗时事务统计\n    2. 大事务统计\n    3. 表维度/库维度 更新数据量统计\n    4. qps统计\n\n暂时只剩下一些分析统计相关的功能未实现，后续有时间会考虑实现。\n\n另外对于命令行参数的一些校验也没做，这块后续也可以优化\n\n## 致谢\n    canal2sql借鉴和学习了很多业界知名的开源项目，在此表示感谢！\n- [canal](https://github.com/alibaba/canal)\n- [bingo2sql](https://github.com/hanchuanchuan/bingo2sql)\n- [binlog2sql](https://github.com/danfengcao/binlog2sql)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhuchao941%2Fcanal2sql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzhuchao941%2Fcanal2sql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhuchao941%2Fcanal2sql/lists"}