{"id":13694819,"url":"https://github.com/sasou/syncClient","last_synced_at":"2025-05-03T04:31:08.083Z","repository":{"id":37664778,"uuid":"91542854","full_name":"sasou/syncClient","owner":"sasou","description":"syncClient，数据实时同步中间件（同步mysql到kafka、redis、elasticsearch、httpmq）！","archived":false,"fork":false,"pushed_at":"2021-12-16T06:24:52.000Z","size":131592,"stargazers_count":273,"open_issues_count":1,"forks_count":108,"subscribers_count":22,"default_branch":"master","last_synced_at":"2024-11-12T21:39:23.038Z","etag":null,"topics":["binlog","canal","elasticsearch","httpmq","java","kafka","redis"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sasou.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-05-17T06:47:46.000Z","updated_at":"2024-08-23T15:50:46.000Z","dependencies_parsed_at":"2022-09-09T04:01:25.137Z","dependency_job_id":null,"html_url":"https://github.com/sasou/syncClient","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sasou%2FsyncClient","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sasou%2FsyncClient/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sasou%2FsyncClient/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sasou%2FsyncClient/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sasou","download_url":"https://codeload.github.com/sasou/syncClient/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252144549,"owners_count":21701429,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binlog","canal","elasticsearch","httpmq","java","kafka","redis"],"created_at":"2024-08-02T17:01:43.803Z","updated_at":"2025-05-03T04:31:03.074Z","avatar_url":"https://github.com/sasou.png","language":"Java","readme":"### **syncClient**\n\n\u003e   syncClient，数据实时同步中间件（同步mysql到kafka、redis、elasticsearch、httpmq）！\n\n 本项目使用canal，将mysql的表数据实时同步到kafka、redis、elasticsearch、httpmq；  \n \n 基本原理：    \n canal解析binlog的数据，由syncClient订阅，然后实时推送到kafka或者redis、elasticsearch、httpmq、ssdb；如果kafka、redis、es、httpmq服务异常，syncClient会回滚操作；canal、kafka、redis、es、httpmq的异常退出，都不会影响数据的传输；\n\n\n---\n\n**目录：**  \nbin：已编译二进制项目，可以直接使用；  \nsrc：源代码；  \n\n---\n\n**配置说明：**\n\n#common  \nsystem.debug=1          # 是否开始调试：1未开启，0为关闭（线上运行请关闭）  \n\n#canal server  \ncanal.ip=127.0.0.1      # canal 服务端 ip;  \ncanal.port=11111        # canal 服务端 端口：默认11111;  \ncanal.destination=one   # canal 服务端项目（destinations），多个用逗号分隔，如：redis,kafka;    \ncanal.username=         # canal 用户名：默认为空;     \ncanal.password=         # canal 密码：默认为空;  \n\n#redis plugin  \nredis.target_type=redis  # 同步插件类型 kafka or redis、elasticsearch、httpmq;   \nredis.target_ip=         # redis服务端 ip;   \nredis.target_port=       # redis端口：默认6379;   \nredis.target_deep=       # 同步到redis的队列名称规则;  \nredis.target_filter_api= # rest api地址，配置后会根据api返回的数据过滤同步数据;   \n\n#kafka plugin  \nkafka.target_type=kafka  # 同步插件类型 kafka;    \nkafka.target_ip=         # kafka服务端 ip;   \nkafka.target_port=       # kafka端口：默认9092;   \nkafka.target_deep=       # 同步到kafka的集合名称规则;  \nkafka.target_filter_api= # rest api地址，配置后会根据api返回的数据过滤同步数据;    \n\n#elasticsearch plugin  \nes.target_type=elasticsearch  # 同步插件类型elasticsearch;    \nes.target_ip=10.5.3.66        # es服务端 ip; \nes.target_port=               # es端口：默认9200; \nes.target_deep=               # 同步到es的index名称规则;  \nes.target_filter_api= # rest api地址，配置后会根据api返回的数据过滤同步数据;   \n\n#httpmq plugin  \nhttpmq.target_type=httpmq    # 同步插件类型 httpmq;    \nhttpmq.target_ip=10.5.3.66   # httpmq服务端 ip; \nhttpmq.target_port=1218      # httpmq端口：默认 1218  \nhttpmq.target_deep=          # 同步到httpmq的队列名称规则;  \nhttpmq.target_filter_api= # rest api地址，配置后会根据api返回的数据过滤同步数据;   \n\n#cache plugin  \ncache.target_type=cache         # 缓存同步插件\ncache.target_plugin=memcached   # 缓存同步类型：暂支持redis、memcached缓存服务器;  \ncache.target_ip=127.0.0.1       # 缓存服务器ip;   \ncache.target_port=11211         # 缓存服务器端口;   \ncache.target_filter_api=        # rest api地址，配置后会根据api返回的数据过滤同步数据;  \ncache.target_version_sign=      # 缓存key前缀  \n\n#target_deep参数影响topic规则，默认值1： \n1、sync_{项目名}_{db}_{table}; \n2、sync_{项目名}_{db};\n3、sync_{项目名}; \n4、sync_{db}_{table}; \n\n---\n\n**使用场景(基于日志增量订阅\u0026消费支持的业务)：**\n\n数据库镜像  \n数据库实时备份  \n多级索引 (分库索引)  \nsearch build  \n业务cache刷新  \n数据变化等重要业务消息  \n\n**Kafka：**\n\nTopic规则：对应配置项目target_deep指定的规则，比如：target_deep=4，数据库的每个表有单独的topic，如数据库admin的user表，对应的kafka主题名为：sync_admin_user  \nTopic数据字段：  \n\n\t插入数据同步格式：\n    {\n        \"head\": {\n            \"binlog_pos\": 53036,\n            \"type\": \"INSERT\",\n            \"binlog_file\": \"mysql-bin.000173\",\n            \"db\": \"sdsw\",\n            \"table\": \"sys_log\"\n        },\n        \"after\": [\n            {\n                \"log_id\": \"1\",\n            },\n            {\n                \"log_ip\": \"27.17.47.100\",\n            },\n            {\n                \"log_addtime\": \"1494204717\",\n            }\n        ]\n    }\n\t\n\t修改数据同步格式：\n    {\n        \"head\": {\n            \"binlog_pos\": 53036,\n            \"type\": \"UPDATE\",\n            \"binlog_file\": \"mysql-bin.000173\",\n            \"db\": \"sdsw\",\n            \"table\": \"sys_log\"\n        },\n        \"before\": [\n            {\n                \"log_id\": \"1\",\n            },\n            {\n                \"log_ip\": \"27.17.47.100\",\n            },\n            {\n                \"log_addtime\": \"1494204717\",\n            }\n        ],\n        \"after\": [\n            {\n                \"log_id\": \"1\",\n            },\n            {\n                \"log_ip\": \"27.17.47.1\",\n            },\n            {\n                \"log_addtime\": \"1494204717\",\n            }\n        ]\n    }\n\t\n\t删除数据同步格式：\n    {\n        \"head\": {\n            \"binlog_pos\": 53036,\n            \"type\": \"DELETE\",\n            \"binlog_file\": \"mysql-bin.000173\",\n            \"db\": \"sdsw\",\n            \"table\": \"sys_log\"\n        },\n        \"before\": [\n            {\n                \"log_id\": \"1\",\n            },\n            {\n                \"log_ip\": \"27.17.47.1\",\n            },\n            {\n                \"log_addtime\": \"1494204717\",\n            }\n        ]\n    }\n\nhead.type 类型：INSERT（插入）、UPDATE（修改）、DELETE（删除）； \n\nhead.db 数据库； \n\nhead.table 数据库表；\n\nhead.binlog_pos  日志位置； \n\nhead.binlog_file 日志文件；  \n\nbefore： UPDATE（修改前）、DELETE（删除前）的数据；  \n\nafter：  INSERT（插入后）、UPDATE（修改后）的数据；  \n\n\n**Redis：**\n\nList规则：对应配置项目target_deep指定的规则，比如：target_deep=4，数据库的每个表有单独的list，如数据库admin的user表，对应的redis list名为：sync_admin_user  \n\n**Elasticsearch**\n\n规则：数据库的每个表有单独的Elasticsearch index，如数据库admin的user表，对应的es index名为：sync_admin_user, index type 为default;\n\nElasticsearch同步数据的head中有id字段；  \n\nMysql 同步到 Elasticsearch注意事项：\n\n1、表需要有一个唯一id主键；  \n2、表时间字段datetime会转为es的时间字段，其他字段对应es的文本类型；  \n3、主键、时间字段禁止修改，其他字段尽量提前规划好；   \n\n**Httpmq：**\n\nList规则：对应配置项目target_deep指定的规则，比如：target_deep=4，数据库的每个表有单独的list，如数据库admin的user表，对应的redis list名为：sync_admin_user  \n\n\n**Cache：**\n\n缓存同步插件：原理是根据数据库变更同步更新表及字段的版本号，业务sdk根据版本号变化判断是否需要更新数据。同步开发了缓存配置管理中心、缓存版本调用sdk（未开源）；  \n\n\n\u003ca href=\"https://info.flagcounter.com/AEYx\"\u003e\u003cimg src=\"https://s11.flagcounter.com/count2/AEYx/bg_FFFFFF/txt_000000/border_CCCCCC/columns_2/maxflags_10/viewers_0/labels_1/pageviews_1/flags_0/percent_0/\" alt=\"Flag Counter\" border=\"0\"\u003e\u003c/a\u003e\n\n","funding_links":[],"categories":["Java"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsasou%2FsyncClient","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsasou%2FsyncClient","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsasou%2FsyncClient/lists"}