{"id":20001152,"url":"https://github.com/apecloud/mysql-binlog-connector-rust","last_synced_at":"2025-04-04T19:07:17.317Z","repository":{"id":216778720,"uuid":"575276904","full_name":"apecloud/mysql-binlog-connector-rust","owner":"apecloud","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-27T09:33:47.000Z","size":184,"stargazers_count":71,"open_issues_count":0,"forks_count":12,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-28T18:14:35.317Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apecloud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-07T06:26:44.000Z","updated_at":"2025-03-27T09:33:51.000Z","dependencies_parsed_at":"2024-08-13T05:23:53.326Z","dependency_job_id":"6d979bdb-d750-41b4-a315-23b43dce7bd2","html_url":"https://github.com/apecloud/mysql-binlog-connector-rust","commit_stats":null,"previous_names":["apecloud/mysql-binlog-connector-rust"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apecloud%2Fmysql-binlog-connector-rust","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apecloud%2Fmysql-binlog-connector-rust/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apecloud%2Fmysql-binlog-connector-rust/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apecloud%2Fmysql-binlog-connector-rust/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apecloud","download_url":"https://codeload.github.com/apecloud/mysql-binlog-connector-rust/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247234921,"owners_count":20905854,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T05:16:51.244Z","updated_at":"2025-04-04T19:07:17.282Z","avatar_url":"https://github.com/apecloud.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"English | [中文](README_ZH.md)\n\n# mysql-binlog-connector-rust\n\n## Overview\n- A simple but strong lib to dump mysql binlog (binlog_format=ROW) and parse Row Based Replication Events in RUST with async IO.\n\n### Supported mysql versions\n- mysql 5.6 (tested in mysql:5.6.51)\n- mysql 5.7 (tested in mysql:5.7.40)\n- mysql 8.0 (tested in mysql:8.0.31)\n\n### Supported replication type\n- binlog-file-position-based replication\n- gtid-based replication\n\n### Supported event types\n- FORMAT_DESCRIPTION_EVENT\n- ROTATE_EVENT\n- PREVIOUS_GTIDS_LOG_EVENT\n- GTID_LOG_EVENT\n- QUERY_EVENT\n- XID_EVENT\n- XA_PREPARE_LOG_EVENT\n- TRANSACTION_PAYLOAD_EVENT\n- ROWS_QUERY_LOG_EVENT\n- TABLE_MAP_EVENT\n- WRITE_ROWS_EVENT_V1\n- WRITE_ROWS_EVENT\n- UPDATE_ROWS_EVENT_V1\n- UPDATE_ROWS_EVENT\n- DELETE_ROWS_EVENT_V1\n- DELETE_ROWS_EVENT\n\n- for more details, refer to: [mysql doc](https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_replication_binlog_event.html)\n\n### Mapping between mysql columns and rust types\n| mysql column type | binlog column type(raw) | binlog column type(parsed from binlog column meta) | rust type       |\n| :---------------- | :---------------------- | :------------------------------------------------- | :---------      |\n| BIT | MYSQL_TYPE_BIT = 16 | ColumnType::Bit | ColumnValue::Bit(u64) |\n| TINYINT [UNSIGNED] | MYSQL_TYPE_TINY = 1 | ColumnType::Tiny | ColumnValue::Tiny(i8) |\n| SMALLINT [UNSIGNED] | MYSQL_TYPE_SHORT = 2 | ColumnType::Short | ColumnValue::Short(i16) |\n| MEDIUMINT [UNSIGNED] | MYSQL_TYPE_INT24 = 9 | ColumnType::Int24 | ColumnValue::Long(i32) |\n| INT [UNSIGNED] | MYSQL_TYPE_LONG = 3 | ColumnType::Long | ColumnValue::Long(i32) |\n| BIGINT [UNSIGNED] | MYSQL_TYPE_LONGLONG = 8 | ColumnType::LongLong | ColumnValue::LongLong(i64) |\n| FLOAT | MYSQL_TYPE_FLOAT = 4 | ColumnType::Float | ColumnValue::Float(f32) |\n| DOUBLE | MYSQL_TYPE_DOUBLE = 5 | ColumnType::Double | ColumnValue::Double(f64) |\n| DECIMAL | MYSQL_TYPE_NEWDECIMAL = 246 | ColumnType::NewDecimal | ColumnValue::Decimal(String) |\n| DATE | MYSQL_TYPE_DATE = 10 | ColumnType::Date | ColumnValue::Date(String) |\n| TIME | MYSQL_TYPE_TIME2 = 19 | ColumnType::Time2 | ColumnValue::Time(String) |\n| TIMESTAMP | MYSQL_TYPE_TIMESTAMP2 = 17 | ColumnType::TimeStamp2 | ColumnValue::Timestamp(i64) |\n| DATETIME | MYSQL_TYPE_DATETIME2 = 18 | ColumnType::DateTime2 | ColumnValue::DateTime(String) |\n| YEAR | MYSQL_TYPE_YEAR = 13 | ColumnType::Year | ColumnValue::Year(u16) |\n| CHAR | MYSQL_TYPE_STRING = 254 | ColumnType::String | ColumnValue::String(Vec\u0026lt;u8\u0026gt;) |\n| VARCHAR | MYSQL_TYPE_VARCHAR = 15 | ColumnType::VarChar | ColumnValue::String(Vec\u0026lt;u8\u0026gt;) |\n| BINARY | MYSQL_TYPE_STRING = 254 | ColumnType::String | ColumnValue::String(Vec\u0026lt;u8\u0026gt;) |\n| VARBINARY | MYSQL_TYPE_VARCHAR = 15 | ColumnType::VarChar | ColumnValue::String(Vec\u0026lt;u8\u0026gt;) |\n| ENUM | MYSQL_TYPE_STRING = 254 | ColumnType::Enum | ColumnValue::Enum(u32) |\n| SET | MYSQL_TYPE_STRING = 254 | ColumnType::Set | ColumnValue::Set(u64) |\n| TINYTEXT TEXT MEDIUMTEXT LONGTEXT TINYBLOB BLOB MEDIUMBLOB LONGBLOB | MYSQL_TYPE_BLOB = 252 | ColumnType::Blob | ColumnValue::Blob(Vec\u0026lt;u8\u0026gt;) |\n| GEOMETRY | MYSQL_TYPE_GEOMETRY = 255 | ColumnType::Geometry | ColumnValue::Blob(Vec\u0026lt;u8\u0026gt;) |\n| JSON | MYSQL_TYPE_JSON = 245 | ColumnType::Json | ColumnValue::Json(Vec\u0026lt;u8\u0026gt;) |\n\n- for CHAR / VARCHAR columns, since binlog contains no charset information, we just get raw bytes and store them in ColumnValue::String(Vec\u0026lt;u8\u0026gt;) objects, you may need to convert them into strings based on column metadatas for further usage.\n- for UNSIGNED numeric columns, since binlog contains no unsigned flags, we just parse them as signed numerics, you may need to convert them into unsigned values based on column metadatas for further usage.\n- for JSON columns, we get raw bytes and store them in ColumnValue::Json(Vec\u0026lt;u8\u0026gt;) objects, we also provide a default deserializer \"JsonBinary\" to parse them into strings, find example later in this doc.\n\n## Quick start\n### Run tests\n- start a mysql, enable binlog without binlog-transaction-compression\n:\n```\ndocker run -d --name mysql57 \\\n--platform linux/x86_64 \\\n-it --restart=always \\\n-p 3307:3306 \\\n-e MYSQL_ROOT_PASSWORD=\"123456\" \\\nmysql:5.7.40 \\\n--lower_case_table_names=1 \\\n--character-set-server=utf8 \\\n--collation-server=utf8_general_ci \\\n--datadir=/var/lib/mysql \\\n--user=mysql \\\n--server_id=1 \\\n--log_bin=/var/lib/mysql/mysql-bin.log \\\n--max_binlog_size=100M \\\n--gtid_mode=ON \\\n--enforce_gtid_consistency=ON \\\n--binlog_format=ROW \n```\n\n- start a mysql, enable binlog with binlog-transaction-compression\n```\ndocker run -d --name mysql80 \\\n--platform linux/x86_64 \\\n-it  --restart=always \\\n-p 3308:3306 -e MYSQL_ROOT_PASSWORD=\"123456\" \\\n mysql:8.0.31 \\\n --lower_case_table_names=1 \\\n --character-set-server=utf8 \\\n --collation-server=utf8_general_ci \\\n --datadir=/var/lib/mysql \\\n --user=mysql \\\n --server_id=1 \\\n --log_bin=/var/lib/mysql/mysql-bin.log \\\n --max_binlog_size=100M \\\n --gtid_mode=ON \\\n --enforce_gtid_consistency=ON \\\n --binlog_format=ROW \\\n --binlog-transaction-compression \\\n --binlog_rows_query_log_events=ON \\\n --default_authentication_plugin=mysql_native_password \\\n --default_time_zone=\"+08:00\"\n```\n\n- update configs in tests/.env\n```\ndb_url=mysql://root:123456@127.0.0.1:3307\nserver_id=200\ndefault_db=\"db_test\"\ndefault_tb=\"tb_test\"\nbinlog_parse_millis=100\n```\n\n- run tests\n```\ncargo test --package mysql-binlog-connector-rust --test integration_test\n```\n- each test will:\n- \u0026nbsp; execute sqls to create tables and generate binlogs\n- \u0026nbsp; dump and parse binlogs\n- \u0026nbsp; wait binlog_parse_millis for all binlogs to be parsed\n- you may increase binlog_parse_millis for big transactions\n\n## Examples\n```rust\nfn main() {\n    block_on(dump_and_parse())\n}\n\nasync fn dump_and_parse() {\n    let env_path = env::current_dir().unwrap().join(\"example/src/.env\");\n    dotenv::from_path(env_path).unwrap();\n    let url = env::var(\"db_url\").unwrap();\n    let server_id: u64 = env::var(\"server_id\").unwrap().parse().unwrap();\n    let binlog_filename = env::var(\"binlog_filename\").unwrap();\n    let binlog_position: u32 = env::var(\"binlog_position\").unwrap().parse().unwrap();\n    let gtid_enabled: bool = env::var(\"gtid_enabled\").unwrap().parse().unwrap();\n    let gtid_set = env::var(\"gtid_set\").unwrap();\n\n    let mut client = BinlogClient {\n        url,\n        binlog_filename,\n        binlog_position,\n        server_id,\n        gtid_enabled,\n        gtid_set,\n    };\n\n    let mut stream = client.connect().await.unwrap();\n\n    loop {\n        let (header, data) = stream.read().await.unwrap();\n        println!(\"header: {:?}\", header);\n        println!(\"data: {:?}\", data);\n        println!();\n    }\n}\n```\n\n### Example 1: parse binlogs with binlog-transaction-compression disabled\n- execute sqls\n```sql\nflush logs;\n\nSET autocommit=0; \nCREATE DATABASE test_db;\nUSE test_db;\nCREATE TABLE test_tb(id INT, value INT);\nINSERT INTO test_tb VALUES(1,1),(2,2),(3,3),(4,4);\nUPDATE test_tb SET value=3 WHERE id in(1,2);\nDELETE FROM test_tb WHERE id in (1,2);\nTRUNCATE TABLE test_tb;\nDROP TABLE test_tb;\ncommit;\n```\n\n- show binlog events\n```sql\nmysql\u003e show binary logs;\n+------------------+-----------+\n| Log_name         | File_size |\n+------------------+-----------+\n| mysql-bin.000050 |      1255 |\n\nmysql\u003e show binlog events in 'mysql-bin.000050';\n+------------------+------+----------------+-----------+-------------+------------------------------------------------------------------------+\n| Log_name         | Pos  | Event_type     | Server_id | End_log_pos | Info                                                                   |\n+------------------+------+----------------+-----------+-------------+------------------------------------------------------------------------+\n| mysql-bin.000050 |    4 | Format_desc    |         1 |         123 | Server ver: 5.7.40-log, Binlog ver: 4                                  |\n| mysql-bin.000050 |  123 | Previous_gtids |         1 |         194 | 50dc6874-13d3-11ee-a17a-0242ac110002:1-176027                          |\n| mysql-bin.000050 |  194 | Gtid           |         1 |         259 | SET @@SESSION.GTID_NEXT= '50dc6874-13d3-11ee-a17a-0242ac110002:176028' |\n| mysql-bin.000050 |  259 | Query          |         1 |         378 | use `test_db`; CREATE TABLE test_tb(id INT, value INT)                 |\n| mysql-bin.000050 |  378 | Gtid           |         1 |         443 | SET @@SESSION.GTID_NEXT= '50dc6874-13d3-11ee-a17a-0242ac110002:176029' |\n| mysql-bin.000050 |  443 | Query          |         1 |         518 | BEGIN                                                                  |\n| mysql-bin.000050 |  518 | Table_map      |         1 |         572 | table_id: 12832 (test_db.test_tb)                                      |\n| mysql-bin.000050 |  572 | Write_rows     |         1 |         643 | table_id: 12832 flags: STMT_END_F                                      |\n| mysql-bin.000050 |  643 | Table_map      |         1 |         697 | table_id: 12832 (test_db.test_tb)                                      |\n| mysql-bin.000050 |  697 | Update_rows    |         1 |         769 | table_id: 12832 flags: STMT_END_F                                      |\n| mysql-bin.000050 |  769 | Table_map      |         1 |         823 | table_id: 12832 (test_db.test_tb)                                      |\n| mysql-bin.000050 |  823 | Delete_rows    |         1 |         876 | table_id: 12832 flags: STMT_END_F                                      |\n| mysql-bin.000050 |  876 | Xid            |         1 |         907 | COMMIT /* xid=13739 */                                                 |\n| mysql-bin.000050 |  907 | Gtid           |         1 |         972 | SET @@SESSION.GTID_NEXT= '50dc6874-13d3-11ee-a17a-0242ac110002:176030' |\n| mysql-bin.000050 |  972 | Query          |         1 |        1064 | use `test_db`; TRUNCATE TABLE test_tb                                  |\n| mysql-bin.000050 | 1064 | Gtid           |         1 |        1129 | SET @@SESSION.GTID_NEXT= '50dc6874-13d3-11ee-a17a-0242ac110002:176031' |\n| mysql-bin.000050 | 1129 | Query          |         1 |        1255 | use `test_db`; DROP TABLE `test_tb` /* generated by server */          |\n+------------------+------+----------------+-----------+-------------+------------------------------------------------------------------------+\n```\n\n- binlogs parsed\n```\nheader: EventHeader { timestamp: 0, event_type: 4, server_id: 1, event_length: 47, next_event_position: 0, event_flags: 32 }\ndata: Rotate(RotateEvent { binlog_filename: \"mysql-bin.000050\", binlog_position: 194 })\n\nheader: EventHeader { timestamp: 1704443761, event_type: 15, server_id: 1, event_length: 119, next_event_position: 0, event_flags: 0 }\ndata: FormatDescription(FormatDescriptionEvent { binlog_version: 4, server_version: \"5.7.40-log\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\", create_timestamp: 0, header_length: 19, checksum_type: CRC32 })\n\nheader: EventHeader { timestamp: 1704443769, event_type: 33, server_id: 1, event_length: 65, next_event_position: 259, event_flags: 0 }\ndata: Gtid(GtidEvent { flags: 1, gtid: \"50dc6874-13d3-11ee-a17a-0242ac110002:176028\" })\n\nheader: EventHeader { timestamp: 1704443769, event_type: 2, server_id: 1, event_length: 119, next_event_position: 378, event_flags: 0 }\ndata: Query(QueryEvent { thread_id: 493, exec_time: 0, error_code: 0, schema: \"test_db\", query: \"CREATE TABLE test_tb(id INT, value INT)\" })\n\nheader: EventHeader { timestamp: 1704443769, event_type: 33, server_id: 1, event_length: 65, next_event_position: 443, event_flags: 0 }\ndata: Gtid(GtidEvent { flags: 0, gtid: \"50dc6874-13d3-11ee-a17a-0242ac110002:176029\" })\n\nheader: EventHeader { timestamp: 1704443769, event_type: 2, server_id: 1, event_length: 75, next_event_position: 518, event_flags: 8 }\ndata: Query(QueryEvent { thread_id: 493, exec_time: 0, error_code: 0, schema: \"test_db\", query: \"BEGIN\" })\n\nheader: EventHeader { timestamp: 1704443769, event_type: 19, server_id: 1, event_length: 54, next_event_position: 572, event_flags: 0 }\ndata: TableMap(TableMapEvent { table_id: 12832, database_name: \"test_db\", table_name: \"test_tb\", column_types: [3, 3], column_metas: [0, 0], null_bits: [true, true] })\n\nheader: EventHeader { timestamp: 1704443769, event_type: 30, server_id: 1, event_length: 71, next_event_position: 643, event_flags: 0 }\ndata: WriteRows(WriteRowsEvent { table_id: 12832, included_columns: [true, true], rows: [RowEvent { column_values: [Long(1), Long(1)] }, RowEvent { column_values: [Long(2), Long(2)] }, RowEvent { column_values: [Long(3), Long(3)] }, RowEvent { column_values: [Long(4), Long(4)] }] })\n\nheader: EventHeader { timestamp: 1704443769, event_type: 19, server_id: 1, event_length: 54, next_event_position: 697, event_flags: 0 }\ndata: TableMap(TableMapEvent { table_id: 12832, database_name: \"test_db\", table_name: \"test_tb\", column_types: [3, 3], column_metas: [0, 0], null_bits: [true, true] })\n\nheader: EventHeader { timestamp: 1704443769, event_type: 31, server_id: 1, event_length: 72, next_event_position: 769, event_flags: 0 }\ndata: UpdateRows(UpdateRowsEvent { table_id: 12832, included_columns_before: [true, true], included_columns_after: [true, true], rows: [(RowEvent { column_values: [Long(1), Long(1)] }, RowEvent { column_values: [Long(1), Long(3)] }), (RowEvent { column_values: [Long(2), Long(2)] }, RowEvent { column_values: [Long(2), Long(3)] })] })\n\nheader: EventHeader { timestamp: 1704443769, event_type: 19, server_id: 1, event_length: 54, next_event_position: 823, event_flags: 0 }\ndata: TableMap(TableMapEvent { table_id: 12832, database_name: \"test_db\", table_name: \"test_tb\", column_types: [3, 3], column_metas: [0, 0], null_bits: [true, true] })\n\nheader: EventHeader { timestamp: 1704443769, event_type: 32, server_id: 1, event_length: 53, next_event_position: 876, event_flags: 0 }\ndata: DeleteRows(DeleteRowsEvent { table_id: 12832, included_columns: [true, true], rows: [RowEvent { column_values: [Long(1), Long(3)] }, RowEvent { column_values: [Long(2), Long(3)] }] })\n\nheader: EventHeader { timestamp: 1704443769, event_type: 16, server_id: 1, event_length: 31, next_event_position: 907, event_flags: 0 }\ndata: Xid(XidEvent { xid: 13739 })\n\nheader: EventHeader { timestamp: 1704443769, event_type: 33, server_id: 1, event_length: 65, next_event_position: 972, event_flags: 0 }\ndata: Gtid(GtidEvent { flags: 1, gtid: \"50dc6874-13d3-11ee-a17a-0242ac110002:176030\" })\n\nheader: EventHeader { timestamp: 1704443769, event_type: 2, server_id: 1, event_length: 92, next_event_position: 1064, event_flags: 0 }\ndata: Query(QueryEvent { thread_id: 493, exec_time: 0, error_code: 0, schema: \"test_db\", query: \"TRUNCATE TABLE test_tb\" })\n\nheader: EventHeader { timestamp: 1704443769, event_type: 33, server_id: 1, event_length: 65, next_event_position: 1129, event_flags: 0 }\ndata: Gtid(GtidEvent { flags: 1, gtid: \"50dc6874-13d3-11ee-a17a-0242ac110002:176031\" })\n\nheader: EventHeader { timestamp: 1704443769, event_type: 2, server_id: 1, event_length: 126, next_event_position: 1255, event_flags: 4 }\ndata: Query(QueryEvent { thread_id: 493, exec_time: 0, error_code: 0, schema: \"test_db\", query: \"DROP TABLE `test_tb` /* generated by server */\" })\n```\n\n### Example 2: parse binlogs with binlog-transaction-compression enabled\n- execute sqls\n```sql\nflush logs;\n\nSET autocommit=0; \nCREATE DATABASE test_db;\nUSE test_db;\nCREATE TABLE test_tb(id INT, value INT);\nINSERT INTO test_tb VALUES(1,1),(2,2),(3,3),(4,4);\nUPDATE test_tb SET value=3 WHERE id in(1,2);\nDELETE FROM test_tb WHERE id in (1,2);\nTRUNCATE TABLE test_tb;\nDROP TABLE test_tb;\ncommit;\n```\n\n- show binlog events\n```sql\nmysql\u003e show binary logs;\n+------------------+-----------+-----------+\n| Log_name         | File_size | Encrypted |\n+------------------+-----------+-----------+\n| mysql-bin.000033 |      1429 | No        |\n\nmysql\u003e show binlog events in 'mysql-bin.000033';\n+------------------+------+---------------------+-----------+-------------+----------------------------------------------------------------------------+\n| Log_name         | Pos  | Event_type          | Server_id | End_log_pos | Info                                                                       |\n+------------------+------+---------------------+-----------+-------------+----------------------------------------------------------------------------+\n| mysql-bin.000033 |    4 | Format_desc         |         1 |         126 | Server ver: 8.0.31, Binlog ver: 4                                          |\n| mysql-bin.000033 |  126 | Previous_gtids      |         1 |         197 | 36682cf3-a048-11ed-b4b3-0242ac110004:1-125753                              |\n| mysql-bin.000033 |  197 | Gtid                |         1 |         274 | SET @@SESSION.GTID_NEXT= '36682cf3-a048-11ed-b4b3-0242ac110004:125754'     |\n| mysql-bin.000033 |  274 | Query               |         1 |         391 | CREATE DATABASE test_db /* xid=5 */                                        |\n| mysql-bin.000033 |  391 | Gtid                |         1 |         468 | SET @@SESSION.GTID_NEXT= '36682cf3-a048-11ed-b4b3-0242ac110004:125755'     |\n| mysql-bin.000033 |  468 | Query               |         1 |         601 | use `test_db`; CREATE TABLE test_tb(id INT, value INT) /* xid=10 */        |\n| mysql-bin.000033 |  601 | Gtid                |         1 |         680 | SET @@SESSION.GTID_NEXT= '36682cf3-a048-11ed-b4b3-0242ac110004:125756'     |\n| mysql-bin.000033 |  680 | Transaction_payload |         1 |        1033 | compression='ZSTD', decompressed_size=633 bytes                            |\n| mysql-bin.000033 | 1033 | Query               |         1 |        1033 | BEGIN                                                                      |\n| mysql-bin.000033 | 1033 | Rows_query          |         1 |        1033 | # INSERT INTO test_tb VALUES(1,1),(2,2),(3,3),(4,4)                        |\n| mysql-bin.000033 | 1033 | Table_map           |         1 |        1033 | table_id: 90 (test_db.test_tb)                                             |\n| mysql-bin.000033 | 1033 | Write_rows          |         1 |        1033 | table_id: 90 flags: STMT_END_F                                             |\n| mysql-bin.000033 | 1033 | Rows_query          |         1 |        1033 | # UPDATE test_tb SET value=3 WHERE id in(1,2)                              |\n| mysql-bin.000033 | 1033 | Table_map           |         1 |        1033 | table_id: 90 (test_db.test_tb)                                             |\n| mysql-bin.000033 | 1033 | Update_rows         |         1 |        1033 | table_id: 90 flags: STMT_END_F                                             |\n| mysql-bin.000033 | 1033 | Rows_query          |         1 |        1033 | # DELETE FROM test_tb WHERE id in (1,2)                                    |\n| mysql-bin.000033 | 1033 | Table_map           |         1 |        1033 | table_id: 90 (test_db.test_tb)                                             |\n| mysql-bin.000033 | 1033 | Delete_rows         |         1 |        1033 | table_id: 90 flags: STMT_END_F                                             |\n| mysql-bin.000033 | 1033 | Xid                 |         1 |        1033 | COMMIT /* xid=11 */                                                        |\n| mysql-bin.000033 | 1033 | Gtid                |         1 |        1110 | SET @@SESSION.GTID_NEXT= '36682cf3-a048-11ed-b4b3-0242ac110004:125757'     |\n| mysql-bin.000033 | 1110 | Query               |         1 |        1214 | use `test_db`; TRUNCATE TABLE test_tb /* xid=14 */                         |\n| mysql-bin.000033 | 1214 | Gtid                |         1 |        1291 | SET @@SESSION.GTID_NEXT= '36682cf3-a048-11ed-b4b3-0242ac110004:125758'     |\n| mysql-bin.000033 | 1291 | Query               |         1 |        1429 | use `test_db`; DROP TABLE `test_tb` /* generated by server */ /* xid=15 */ |\n+------------------+------+---------------------+-----------+-------------+----------------------------------------------------------------------------+\n```\n\n- binlogs parsed\n```\nheader: EventHeader { timestamp: 1704445709, event_type: 15, server_id: 1, event_length: 122, next_event_position: 126, event_flags: 0 }\ndata: FormatDescription(FormatDescriptionEvent { binlog_version: 4, server_version: \"8.0.31\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\", create_timestamp: 0, header_length: 19, checksum_type: CRC32 })\n\nheader: EventHeader { timestamp: 1704445709, event_type: 35, server_id: 1, event_length: 71, next_event_position: 197, event_flags: 128 }\ndata: PreviousGtids(PreviousGtidsEvent { gtid_set: \"36682cf3-a048-11ed-b4b3-0242ac110004:1-125753\" })\n\nheader: EventHeader { timestamp: 1704445716, event_type: 33, server_id: 1, event_length: 77, next_event_position: 274, event_flags: 0 }\ndata: Gtid(GtidEvent { flags: 1, gtid: \"36682cf3-a048-11ed-b4b3-0242ac110004:125754\" })\n\nheader: EventHeader { timestamp: 1704445716, event_type: 2, server_id: 1, event_length: 117, next_event_position: 391, event_flags: 8 }\ndata: Query(QueryEvent { thread_id: 8, exec_time: 0, error_code: 0, schema: \"test_db\", query: \"CREATE DATABASE test_db\" })\n\nheader: EventHeader { timestamp: 1704445716, event_type: 33, server_id: 1, event_length: 77, next_event_position: 468, event_flags: 0 }\ndata: Gtid(GtidEvent { flags: 1, gtid: \"36682cf3-a048-11ed-b4b3-0242ac110004:125755\" })\n\nheader: EventHeader { timestamp: 1704445716, event_type: 2, server_id: 1, event_length: 133, next_event_position: 601, event_flags: 0 }\ndata: Query(QueryEvent { thread_id: 8, exec_time: 0, error_code: 0, schema: \"test_db\", query: \"CREATE TABLE test_tb(id INT, value INT)\" })\n\nheader: EventHeader { timestamp: 1704445717, event_type: 33, server_id: 1, event_length: 79, next_event_position: 680, event_flags: 0 }\ndata: Gtid(GtidEvent { flags: 0, gtid: \"36682cf3-a048-11ed-b4b3-0242ac110004:125756\" })\n\nheader: EventHeader { timestamp: 1704445717, event_type: 40, server_id: 1, event_length: 353, next_event_position: 1033, event_flags: 0 }\ndata: TransactionPayload(TransactionPayloadEvent { uncompressed_size: 633, uncompressed_events: [(EventHeader { timestamp: 1704445716, event_type: 2, server_id: 1, event_length: 74, next_event_position: 0, event_flags: 8 }, Query(QueryEvent { thread_id: 8, exec_time: 1, error_code: 0, schema: \"test_db\", query: \"BEGIN\" })), (EventHeader { timestamp: 1704445716, event_type: 29, server_id: 1, event_length: 69, next_event_position: 0, event_flags: 128 }, RowsQuery(RowsQueryEvent { query: \"INSERT INTO test_tb VALUES(1,1),(2,2),(3,3),(4,4)\" })), (EventHeader { timestamp: 1704445716, event_type: 19, server_id: 1, event_length: 53, next_event_position: 0, event_flags: 0 }, TableMap(TableMapEvent { table_id: 90, database_name: \"test_db\", table_name: \"test_tb\", column_types: [3, 3], column_metas: [0, 0], null_bits: [true, true] })), (EventHeader { timestamp: 1704445716, event_type: 30, server_id: 1, event_length: 67, next_event_position: 0, event_flags: 0 }, WriteRows(WriteRowsEvent { table_id: 90, included_columns: [true, true], rows: [RowEvent { column_values: [Long(1), Long(1)] }, RowEvent { column_values: [Long(2), Long(2)] }, RowEvent { column_values: [Long(3), Long(3)] }, RowEvent { column_values: [Long(4), Long(4)] }] })), (EventHeader { timestamp: 1704445717, event_type: 29, server_id: 1, event_length: 63, next_event_position: 0, event_flags: 128 }, RowsQuery(RowsQueryEvent { query: \"UPDATE test_tb SET value=3 WHERE id in(1,2)\" })), (EventHeader { timestamp: 1704445717, event_type: 19, server_id: 1, event_length: 53, next_event_position: 0, event_flags: 0 }, TableMap(TableMapEvent { table_id: 90, database_name: \"test_db\", table_name: \"test_tb\", column_types: [3, 3], column_metas: [0, 0], null_bits: [true, true] })), (EventHeader { timestamp: 1704445717, event_type: 31, server_id: 1, event_length: 68, next_event_position: 0, event_flags: 0 }, UpdateRows(UpdateRowsEvent { table_id: 90, included_columns_before: [true, true], included_columns_after: [true, true], rows: [(RowEvent { column_values: [Long(1), Long(1)] }, RowEvent { column_values: [Long(1), Long(3)] }), (RowEvent { column_values: [Long(2), Long(2)] }, RowEvent { column_values: [Long(2), Long(3)] })] })), (EventHeader { timestamp: 1704445717, event_type: 29, server_id: 1, event_length: 57, next_event_position: 0, event_flags: 128 }, RowsQuery(RowsQueryEvent { query: \"DELETE FROM test_tb WHERE id in (1,2)\" })), (EventHeader { timestamp: 1704445717, event_type: 19, server_id: 1, event_length: 53, next_event_position: 0, event_flags: 0 }, TableMap(TableMapEvent { table_id: 90, database_name: \"test_db\", table_name: \"test_tb\", column_types: [3, 3], column_metas: [0, 0], null_bits: [true, true] })), (EventHeader { timestamp: 1704445717, event_type: 32, server_id: 1, event_length: 49, next_event_position: 0, event_flags: 0 }, DeleteRows(DeleteRowsEvent { table_id: 90, included_columns: [true, true], rows: [RowEvent { column_values: [Long(1), Long(3)] }, RowEvent { column_values: [Long(2), Long(3)] }] })), (EventHeader { timestamp: 1704445717, event_type: 16, server_id: 1, event_length: 27, next_event_position: 0, event_flags: 0 }, Xid(XidEvent { xid: 11 }))] })\n\nheader: EventHeader { timestamp: 1704445717, event_type: 33, server_id: 1, event_length: 77, next_event_position: 1110, event_flags: 0 }\ndata: Gtid(GtidEvent { flags: 1, gtid: \"36682cf3-a048-11ed-b4b3-0242ac110004:125757\" })\n\nheader: EventHeader { timestamp: 1704445717, event_type: 2, server_id: 1, event_length: 104, next_event_position: 1214, event_flags: 0 }\ndata: Query(QueryEvent { thread_id: 8, exec_time: 0, error_code: 0, schema: \"test_db\", query: \"TRUNCATE TABLE test_tb\" })\n\nheader: EventHeader { timestamp: 1704445717, event_type: 33, server_id: 1, event_length: 77, next_event_position: 1291, event_flags: 0 }\ndata: Gtid(GtidEvent { flags: 1, gtid: \"36682cf3-a048-11ed-b4b3-0242ac110004:125758\" })\n\nheader: EventHeader { timestamp: 1704445717, event_type: 2, server_id: 1, event_length: 138, next_event_position: 1429, event_flags: 4 }\ndata: Query(QueryEvent { thread_id: 8, exec_time: 0, error_code: 0, schema: \"test_db\", query: \"DROP TABLE `test_tb` /* generated by server */\" })\n```\n\n### Example 3: parse json column to string\n```rust\nfn parse_json_columns(data: EventData) {\n    let parse_row = |row: RowEvent| {\n        for column_value in row.column_values {\n            if let ColumnValue::Json(bytes) = column_value {\n                println!(\n                    \"json column: {}\",\n                    JsonBinary::parse_as_string(\u0026bytes).unwrap()\n                )\n            }\n        }\n    };\n\n    match data {\n        EventData::WriteRows(event) =\u003e {\n            for row in event.rows {\n                parse_row(row)\n            }\n        }\n        EventData::DeleteRows(event) =\u003e {\n            for row in event.rows {\n                parse_row(row)\n            }\n        }\n        EventData::UpdateRows(event) =\u003e {\n            for (before, after) in event.rows {\n                parse_row(before);\n                parse_row(after);\n            }\n        }\n        _ =\u003e {}\n    }\n}\n```\n\n- execute sqls\n```sql\n\nCREATE TABLE test_db_1.json_test(id INT AUTO_INCREMENT, json_col JSON, PRIMARY KEY(id));\n\nSET autocommit=0;\nINSERT INTO test_db_1.json_test VALUES (NULL, '{\"k.1\":1,\"k.0\":0,\"k.-1\":-1,\"k.true\":true,\"k.false\":false,\"k.null\":null,\"k.string\":\"string\",\"k.true_false\":[true,false],\"k.32767\":32767,\"k.32768\":32768,\"k.-32768\":-32768,\"k.-32769\":-32769,\"k.2147483647\":2147483647,\"k.2147483648\":2147483648,\"k.-2147483648\":-2147483648,\"k.-2147483649\":-2147483649,\"k.18446744073709551615\":18446744073709551615,\"k.18446744073709551616\":18446744073709551616,\"k.3.14\":3.14,\"k.{}\":{},\"k.[]\":[]}');\nINSERT INTO test_db_1.json_test VALUES (NULL, '{\"中文\":\"😀\"}');\ncommit;\n```\n\n- parsed json column values:\n```\njson column: {\"k.0\":0,\"k.1\":1,\"k.-1\":-1,\"k.[]\":[],\"k.{}\":{},\"k.3.14\":3.14,\"k.null\":null,\"k.true\":true,\"k.32767\":32767,\"k.32768\":32768,\"k.false\":false,\"k.-32768\":-32768,\"k.-32769\":-32769,\"k.string\":\"string\",\"k.2147483647\":2147483647,\"k.2147483648\":2147483648,\"k.true_false\":[true,false],\"k.-2147483648\":-2147483648,\"k.-2147483649\":-2147483649,\"k.18446744073709551615\":18446744073709551615,\"k.18446744073709551616\":18446744073709552000}\n\njson column: {\"中文\":\"😀\"}\n```\n\n### Example 4: parse binlog file\n```rust\nasync fn parse_file() {\n    let file_path = \"path-to-binlog-file\";\n    let mut file = File::open(file_path).unwrap();\n\n    let mut parser = BinlogParser {\n        checksum_length: 4,\n        table_map_event_by_table_id: HashMap::new(),\n    };\n\n    assert!(parser.check_magic(\u0026mut file).is_ok());\n    while let Ok((header, data)) = parser.next(\u0026mut file) {\n        println!(\"header: {:?}\", header);\n        println!(\"data: {:?}\", data);\n        println!(\"\");\n    }\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapecloud%2Fmysql-binlog-connector-rust","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapecloud%2Fmysql-binlog-connector-rust","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapecloud%2Fmysql-binlog-connector-rust/lists"}