{"id":18602818,"url":"https://github.com/obgnail/mysql-river","last_synced_at":"2026-04-27T11:31:34.504Z","repository":{"id":65732823,"uuid":"507995048","full_name":"obgnail/mysql-river","owner":"obgnail","description":"监控 binlog，执行自定义的 handler","archived":false,"fork":false,"pushed_at":"2023-02-09T12:42:59.000Z","size":1310,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-16T18:12:19.716Z","etag":null,"topics":["binlog","canal","golang","gtid","mysql"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/obgnail.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-27T17:05:41.000Z","updated_at":"2023-10-24T18:15:12.000Z","dependencies_parsed_at":"2023-02-19T07:45:46.040Z","dependency_job_id":null,"html_url":"https://github.com/obgnail/mysql-river","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/obgnail/mysql-river","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obgnail%2Fmysql-river","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obgnail%2Fmysql-river/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obgnail%2Fmysql-river/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obgnail%2Fmysql-river/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/obgnail","download_url":"https://codeload.github.com/obgnail/mysql-river/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obgnail%2Fmysql-river/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32335295,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-26T23:26:28.701Z","status":"online","status_checked_at":"2026-04-27T02:00:06.769Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binlog","canal","golang","gtid","mysql"],"created_at":"2024-11-07T02:12:37.930Z","updated_at":"2026-04-27T11:31:34.488Z","avatar_url":"https://github.com/obgnail.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mysql River\r\n\r\n\u003cdiv align=\"center\"\u003e\r\n  \u003cimg src=\"assets/mysql-river.png\" alt=\"worktop\" width=\"400\" /\u003e\r\n\u003c/div\u003e\r\n\r\n## introduction\r\n\r\n解析 mysql binlog，提供简单易用的同步方案。\r\n\r\n内置三个电池：\r\n\r\n- TraceLog：将 binlog 实时翻译为 sql 语句。\r\n- ElasticSearchSync：将 binlog 的数据同步到 es 中。\r\n- KafkaBroker：将 binlog 的数据同步到 Kafka 中，和 MySQL 彻底解耦。\r\n\r\n\r\n\r\n## feather\r\n\r\nmysql-river 内置 auto position saver 和 auto health checker 两个功能：\r\n\r\n- auto position saver：自动记录 river 的处理进展，将其保存为 master.info 文件。当 river 挂掉重启后依旧可以恢复进展，不必担心数据丢失。\r\n- auto health checker：提供健康检测接口。当 river 的进展和 mysql binlog 的进展差值超过阈值时，触发对应函数。\r\n\r\n\r\n\r\n### health check rule\r\n\r\n对比 `master.info`(file-pos) 和 `canal.GetMasterPos()`(db-pos) 的 position 信息，当触发规则时，调用对应函数。可以对接自动告警功能。\r\n\r\n- 当获取 db-pos 失败时， 健康状态为 red\r\n- 当 db-pos 跟 file-pos 相差在阈值内， 健康状态为 green\r\n- 当 db-pos 跟 file-pos 相关在阈值外时， 健康状态为 yellow\r\n- 当 db-pos 跟 上次记录的 db-pos 没有变化时，且file-pos 跟 上次记录的 file-pos 没有变化时，且 db-pos  跟  file-pos 相等时 健康状态为 green\r\n- 当 db-pos 跟 上次记录的 db-pos 没有变化时，且file-pos 跟 上次记录的 file-pos 没有变化时，且 db-pos  大于  file-pos 时 健康状态为 red\r\n- 当 db-pos 跟 上次记录的 db-pos 没有变化时，且file-pos 跟 上次记录的 file-pos 有变化时， 健康状态为 green\r\n- 当 db-pos 跟 上次记录的 db-pos 有变化时， 且 file-pos 跟 上次记录的 file-pos 没有变化时， 健康状态为 red\r\n- 当 db-pos 跟 上次记录的 db-pos 有变化时， 且 file-pos 跟 上次记录的 file-pos 有变化时， 健康状态为 green\r\n\r\n\r\n\r\n### Usage\r\n\r\n只需实现 Handler 接口：\r\n\r\n- OnEvent：核心函数。river 会自动解析 mysql binlog 文件，将 20+ 种 event 归纳为 insert、update、delete、ddl、gtid、xid、rotate、table_changed 几种。\r\n- OnAlert：auto health check 不通过时自动调用此函数，可以对接自动告警功能。\r\n- OnClose：river 发生不可恢复错误时，自动调用此函数，可以用此关闭 handler 或对接自动告警功能。\r\n\r\n```go\r\ntype Handler interface {\r\n\tString() string\r\n\tOnEvent(event *EventData) error\r\n\tOnAlert(msg *StatusMsg) error\r\n\tOnClose(river *River) // OnEvent、OnAlert抛出的error会触发OnClose\r\n}\r\n```\r\n\r\n```go\r\ntype EventData struct {\r\n\t// insert、update、delete、ddl、gtid、xid、rotate、table_changed\r\n\tEventType string                 `json:\"event_type\"`\r\n\tServerID  uint32                 `json:\"server_id\"`\r\n\tLogName   string                 `json:\"log_name\"`\r\n\tLogPos    uint32                 `json:\"log_pos\"`\r\n\tDb        string                 `json:\"db\"`\r\n\tTable     string                 `json:\"table\"`\r\n\tSQL       string                 `json:\"sql\"` // 仅当EventType为ddl有值\r\n\tGTIDSet   string                 `json:\"gtid_set\"`\r\n\tPrimary   []string               `json:\"primary\"`   // 主键字段；EventType为insert、update、delete时有值\r\n\tBefore    map[string]interface{} `json:\"before\"`    // 变更前数据, insert 类型的 before 为空\r\n\tAfter     map[string]interface{} `json:\"after\"`     // 变更后数据, delete 类型的 after 为空\r\n\tTimestamp uint32                 `json:\"timestamp\"` // 事件时间\r\n}\r\n```\r\n\r\n```go\r\ntype StatusMsg struct {\r\n\tStatus        HealthStatus\r\n\tLastStatus    HealthStatus\r\n\tReason        []string // 发生告警时的消息(可能有多条不通过)\r\n\tFilePos       *mysql.Position\r\n\tDBPos         *mysql.Position\r\n\tCheckInterval time.Duration\r\n\tPosThreshold  int\r\n}\r\n```\r\n\r\n\r\n\r\n## example\r\n\r\n```go\r\npackage main\r\n\r\nimport (\r\n\t\"fmt\"\r\n\t\"github.com/obgnail/mysql-river/river\"\r\n\t\"time\"\r\n)\r\n\r\nvar config = \u0026river.Config{\r\n\tMySQLConfig: \u0026river.MySQLConfig{\r\n\t\tHost:     \"127.0.0.1\",\r\n\t\tPort:     3306,\r\n\t\tUser:     \"root\",\r\n\t\tPassword: \"root\",\r\n\t},\r\n\tPosAutoSaverConfig: \u0026river.PosAutoSaverConfig{\r\n\t\tSaveDir:      \"./\",\r\n\t\tSaveInterval: 3 * time.Second,\r\n\t},\r\n\tHealthCheckerConfig: \u0026river.HealthCheckerConfig{\r\n\t\tCheckPosThreshold: 3000,\r\n\t\tCheckInterval:     5 * time.Second,\r\n\t},\r\n}\r\n\r\nfunc main() {\r\n\terr := river.New(config).\r\n\t\tSetHandler(river.NopCloserAlerter(func(event *river.EventData) error {\r\n\t\t\tfmt.Println(event.EventType, event.LogName, event.LogPos, event.Before, event.After)\r\n\t\t\treturn nil\r\n\t\t})).\r\n\t\tSync(river.FromFile) // 从 master.info 文件开始解析\r\n\tPanicIfError(err)\r\n}\r\n```\r\n\r\n\r\n\r\n## Built-in battery\r\n\r\n### trace log\r\n\r\n![image-20230205212745428](assets/image-20230205212745428.png)\r\n\r\n```go\r\nvar config = \u0026river.Config{\r\n\tMySQLConfig: \u0026river.MySQLConfig{\r\n\t\tHost:     \"127.0.0.1\",\r\n\t\tPort:     3306,\r\n\t\tUser:     \"root\",\r\n\t\tPassword: \"root\",\r\n\t},\r\n\tPosAutoSaverConfig: \u0026river.PosAutoSaverConfig{\r\n\t\tSaveDir:      \"./\",\r\n\t\tSaveInterval: 3 * time.Second,\r\n\t},\r\n\tHealthCheckerConfig: \u0026river.HealthCheckerConfig{\r\n\t\tCheckPosThreshold: 3000,\r\n\t\tCheckInterval:     5 * time.Second,\r\n\t},\r\n}\r\n\r\nfunc main() {\r\n\ttraceConfig := \u0026trace_log.Config{\r\n\t\tDBs:          []string{\"testdb01\"},\r\n\t\tEntireFields: false,\r\n\t\tShowTxMsg:    true,\r\n\t\tHighlight:    true,\r\n\t}\r\n\thandler := trace_log.New(traceConfig)\r\n\terr := river.New(config).SetHandler(handler).Sync(river.FromDB) // 从最新位置开始解析\r\n\tPanicIfError(err)\r\n}\r\n```\r\n\r\n\r\n\r\n### elastic search sync\r\n\r\n```go\r\nvar config = \u0026river.Config{\r\n\tMySQLConfig: \u0026river.MySQLConfig{\r\n\t\tHost:     \"127.0.0.1\",\r\n\t\tPort:     3306,\r\n\t\tUser:     \"root\",\r\n\t\tPassword: \"root\",\r\n\t},\r\n\tPosAutoSaverConfig: \u0026river.PosAutoSaverConfig{\r\n\t\tSaveDir:      \"./\",\r\n\t\tSaveInterval: 3 * time.Second,\r\n\t},\r\n\tHealthCheckerConfig: \u0026river.HealthCheckerConfig{\r\n\t\tCheckPosThreshold: 3000,\r\n\t\tCheckInterval:     5 * time.Second,\r\n\t},\r\n}\r\n\r\nfunc main() {\r\n\thandlerConfig := \u0026elasticsearch.EsHandlerConfig{\r\n\t\tHost:          \"127.0.0.1\",\r\n\t\tPort:          9200,\r\n\t\tUser:          \"\",\r\n\t\tPassword:      \"\",\r\n\t\tBulkSize:      128,\r\n\t\tFlushInterval: time.Second,\r\n\t\tSkipNoPkTable: true,\r\n\t\tRules: []*elasticsearch.Rule{\r\n\t\t\telasticsearch.NewDefaultRule(\"testdb01\", \"user\"),\r\n\t\t},\r\n\t}\r\n\thandler := elasticsearch.New(handlerConfig)\r\n\terr := river.New(config).SetHandler(handler).Sync(river.FromDB)\r\n\tPanicIfError(err)\r\n}\r\n```\r\n\r\n\r\n\r\n### kafka broker\r\n\r\n因为引入了 kafka 这个组件，谁也不能保证 kafka 不会挂掉，进而引入了[bbolt](https://github.com/etcd-io/bbolt)，系统会自动在指定位置生成 `kafka_offset.bolt`。该文件会自动记录所有 partition 的 offset，并且在下次启动 river 的时候自动加载在此文件并自动进行偏移处理。\r\n\r\n故，此机制是透明的。\r\n\r\n```go\r\nvar config = \u0026river.Config{\r\n\tMySQLConfig: \u0026river.MySQLConfig{\r\n\t\tHost:     \"127.0.0.1\",\r\n\t\tPort:     3306,\r\n\t\tUser:     \"root\",\r\n\t\tPassword: \"root\",\r\n\t},\r\n\tPosAutoSaverConfig: \u0026river.PosAutoSaverConfig{\r\n\t\tSaveDir:      \"./\",\r\n\t\tSaveInterval: 3 * time.Second,\r\n\t},\r\n\tHealthCheckerConfig: \u0026river.HealthCheckerConfig{\r\n\t\tCheckPosThreshold: 3000,\r\n\t\tCheckInterval:     5 * time.Second,\r\n\t},\r\n}\r\n\r\nfunc main() {\r\n\tkafkaConfig := \u0026kafka.Config{\r\n\t\tAddrs:           []string{\"127.0.0.1:9092\"},\r\n\t\tTopic:           \"binlog\",\r\n\t\tOffsetStoreDir:  \"./\",\r\n\t\tOffset:          nil,\r\n\t\tUseOldestOffset: false,\r\n\t}\r\n\thandler, err := kafka.New(kafkaConfig)\r\n\tPanicIfError(err)\r\n\tgo handler.Consume(func(msg *sarama.ConsumerMessage) error {\r\n\t\tfmt.Printf(\"Partition:%d, Offset:%d, key:%s, value:%s\\n\",\r\n\t\t\tmsg.Partition, msg.Offset, string(msg.Key), string(msg.Value))\r\n\t\treturn nil\r\n\t})\r\n\terr = river.New(config).SetHandler(handler).Sync(river.FromFile)\r\n\tPanicIfError(err)\r\n}\r\n```\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobgnail%2Fmysql-river","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fobgnail%2Fmysql-river","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobgnail%2Fmysql-river/lists"}