{"id":19993394,"url":"https://github.com/starcwang/canal_mysql_elasticsearch_sync","last_synced_at":"2025-05-04T12:31:26.859Z","repository":{"id":41562609,"uuid":"101387454","full_name":"starcwang/canal_mysql_elasticsearch_sync","owner":"starcwang","description":"基于canal的mysql和elasticsearch实时同步方案，支持增量同步和全量同步","archived":false,"fork":false,"pushed_at":"2021-08-09T20:42:47.000Z","size":82,"stargazers_count":496,"open_issues_count":2,"forks_count":153,"subscribers_count":33,"default_branch":"master","last_synced_at":"2024-11-13T04:56:29.890Z","etag":null,"topics":["binlog","canal","elasticsearch","mysql","synchronization"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/starcwang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-08-25T09:14:18.000Z","updated_at":"2024-09-03T03:30:01.000Z","dependencies_parsed_at":"2022-09-21T13:23:58.748Z","dependency_job_id":null,"html_url":"https://github.com/starcwang/canal_mysql_elasticsearch_sync","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/starcwang%2Fcanal_mysql_elasticsearch_sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/starcwang%2Fcanal_mysql_elasticsearch_sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/starcwang%2Fcanal_mysql_elasticsearch_sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/starcwang%2Fcanal_mysql_elasticsearch_sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/starcwang","download_url":"https://codeload.github.com/starcwang/canal_mysql_elasticsearch_sync/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252334385,"owners_count":21731395,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binlog","canal","elasticsearch","mysql","synchronization"],"created_at":"2024-11-13T04:52:37.868Z","updated_at":"2025-05-04T12:31:21.852Z","avatar_url":"https://github.com/starcwang.png","language":"Java","funding_links":[],"categories":["Java"],"sub_categories":[],"readme":"# canal_mysql_elasticsearch_sync 支持请star✨\n# 交流Q群：733688083\n\n**canal于v1.1.2版本后，已支持自动同步到Elasticsearch。赞canal！[canal传送门](https://github.com/alibaba/canal)**\n\n基于 *canal* 的 *Mysql* 与 *Elasticsearch* 实时同步的 *javaweb* 服务。      \ncanal是阿里巴巴mysql数据库binlog的增量订阅\u0026消费组件。\n\n## 工作原理\n### 全量\n暴露Http接口（接口定义见[wiki](https://github.com/starcwang/canal_mysql_elasticsearch_sync/wiki/HttpApi)），待调用后开启后台线程，通过主键分批同步指定数据库中数据到Elasticsearch\n\u003e 读取数据库会加**读锁**   \n\u003e 主键必须为数字类型\n#### 过程\n1. 首先会根据所给的数据库主键字段，拿到最大的主键数字max_id；\n2. 设*pk*=min_id（默认是数据库中的主键最小值）；\n2. 加读锁🔐，从数据库中取出*pk* —— *pk*+*stepSize* 大小的数据（默认500）的数据；\n3. 插入到Elasticsearch中；\n4. 释放读锁🔐，pk累加*stepSize*，循环3.操作，直到*pk*\u003e*max_id*\n\n### 增量\n循环监听canal通过binlog同步过来的event事件，区别增删改进行与之对应的Elasticsearch的操作。\n\u003e 目前只解析了 insert、update、delete，其它数据库操作会被忽略\n\n## 默认相关字段映射\n\u003ctable  class=\"bbcode\"\u003e \n\u003ctr\u003e  \n\u003ctd\u003eMysql字段类型\u003c/td\u003e\n\u003ctd\u003eElasticsearch类型\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e  \n\u003ctd\u003echar\u003c/td\u003e\n\u003ctd\u003e{\"type\": \"text\", \"fields\": {\"keyword\": {\"type\": \"keyword\", \"ignore_above\": 256}}\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e  \n\u003ctd\u003etext\u003c/td\u003e\n\u003ctd\u003e{\"type\": \"text\", \"fields\": {\"keyword\": {\"type\": \"keyword\", \"ignore_above\": 256}}\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e  \n\u003ctd\u003eblob\u003c/td\u003e\n\u003ctd\u003e{\"type\": \"text\", \"fields\": {\"keyword\": {\"type\": \"keyword\", \"ignore_above\": 256}}\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e  \n\u003ctd\u003eint\u003c/td\u003e\n\u003ctd\u003e{\"type\": \"long\"}\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e  \n\u003ctd\u003edate\u003c/td\u003e\n\u003ctd\u003e{\"type\": \"date\"}\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e  \n\u003ctd\u003etime\u003c/td\u003e\n\u003ctd\u003e{\"type\": \"date\"}\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e  \n\u003ctd\u003efloat\u003c/td\u003e\n\u003ctd\u003e{\"type\": \"float\"}\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e  \n\u003ctd\u003edouble\u003c/td\u003e\n\u003ctd\u003e{\"type\": \"float\"}\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e  \n\u003ctd\u003edecimal\u003c/td\u003e\n\u003ctd\u003e{\"type\": \"float\"}\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e  \n\u003ctd\u003e其它\u003c/td\u003e\n\u003ctd\u003e{\"type\": \"text\", \"fields\": {\"keyword\": {\"type\": \"keyword\", \"ignore_above\": 256}}\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e \n\n## 注意事项\n- Mysql的binlog格式必须为**ROW**\n- 因为有行锁，Mysql中table使用的存储引擎须为**InnoDB**\n- 由于使用binlog进行增量同步，和数据库主从类似，不可避免的会有一定的主从延迟，延迟时间取决于机房网络、机器负载、数据量大小等\n- Elasticsearch支持的版本为**5.x**\n- canal已测试版为**v1.0.24**，其他版本请自行测试\n- 增量同步只监听了 **INSERT、UPDATE、DELETE**，其它如建表、删表等尚未支持\n- 建议Elasticsearch的mapping手动来创建，因为默认的创建方式不能保证满足业务需求\n\n## 相关文档\n- [wiki](https://github.com/starcwang/canal_mysql_elasticsearch_sync/wiki)\n- [HttpApi](https://github.com/starcwang/canal_mysql_elasticsearch_sync/wiki/HttpApi)\n- [QuickStart](https://github.com/starcwang/canal_mysql_elasticsearch_sync/wiki/QuickStart)\n\n## 联系方式\n如果有不合理的地方，还请不吝赐教。\n- QQ群：733688083 \n \n**支持记得star✨**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstarcwang%2Fcanal_mysql_elasticsearch_sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstarcwang%2Fcanal_mysql_elasticsearch_sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstarcwang%2Fcanal_mysql_elasticsearch_sync/lists"}