{"id":13487491,"url":"https://github.com/shyiko/mysql-binlog-connector-java","last_synced_at":"2025-05-14T02:09:59.375Z","repository":{"id":9462562,"uuid":"11344961","full_name":"shyiko/mysql-binlog-connector-java","owner":"shyiko","description":"MySQL Binary Log connector","archived":false,"fork":false,"pushed_at":"2023-04-15T13:27:03.000Z","size":1405,"stargazers_count":2243,"open_issues_count":104,"forks_count":824,"subscribers_count":136,"default_branch":"master","last_synced_at":"2025-04-21T09:06:47.665Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shyiko.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2013-07-11T15:05:22.000Z","updated_at":"2025-04-14T06:32:01.000Z","dependencies_parsed_at":"2022-06-26T00:12:50.806Z","dependency_job_id":"22ca884a-8502-4fae-96f7-978cb3a5038f","html_url":"https://github.com/shyiko/mysql-binlog-connector-java","commit_stats":null,"previous_names":[],"tags_count":47,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shyiko%2Fmysql-binlog-connector-java","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shyiko%2Fmysql-binlog-connector-java/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shyiko%2Fmysql-binlog-connector-java/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shyiko%2Fmysql-binlog-connector-java/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shyiko","download_url":"https://codeload.github.com/shyiko/mysql-binlog-connector-java/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254053330,"owners_count":22006717,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T18:00:59.989Z","updated_at":"2025-05-14T02:09:54.362Z","avatar_url":"https://github.com/shyiko.png","language":"Java","funding_links":[],"categories":["Java","数据库中间件","Database Tools"],"sub_categories":[],"readme":"# mysql-binlog-connector-java [![Build Status](https://travis-ci.org/shyiko/mysql-binlog-connector-java.svg?branch=master)](https://travis-ci.org/shyiko/mysql-binlog-connector-java) [![Coverage Status](https://coveralls.io/repos/shyiko/mysql-binlog-connector-java/badge.svg?branch=master)](https://coveralls.io/r/shyiko/mysql-binlog-connector-java?branch=master) [![Maven Central](https://img.shields.io/maven-central/v/com.github.shyiko/mysql-binlog-connector-java.svg)](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.github.shyiko%22%20AND%20a%3A%22mysql-binlog-connector-java%22)\n\n## ATTENTION: This repository is no longer maintained. I recommend migrating to [osheroff/mysql-binlog-connector-java](https://github.com/osheroff/mysql-binlog-connector-java).  \n\nMySQL Binary Log connector.\n\nInitially project was started as a fork of [open-replicator](https://code.google.com/p/open-replicator), \nbut ended up as a complete rewrite. Key differences/features:\n\n- automatic binlog filename/position | GTID resolution\n- resumable disconnects\n- plugable failover strategies\n- binlog_checksum=CRC32 support (for MySQL 5.6.2+ users)\n- secure communication over the TLS\n- JMX-friendly\n- real-time stats\n- availability in Maven Central\n- no third-party dependencies\n- test suite over different versions of MySQL releases\n\n\u003e If you are looking for something similar in other languages - check out \n[siddontang/go-mysql](https://github.com/siddontang/go-mysql) (Go), \n[noplay/python-mysql-replication](https://github.com/noplay/python-mysql-replication) (Python).\n\n## Usage\n\nGet the latest JAR(s) from [here](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.github.shyiko%22%20AND%20a%3A%22mysql-binlog-connector-java%22). Alternatively you can include following Maven dependency (available through Maven Central):\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.shyiko\u003c/groupId\u003e\n    \u003cartifactId\u003emysql-binlog-connector-java\u003c/artifactId\u003e\n    \u003cversion\u003e0.21.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n#### Reading binary log file\n\n```java\nFile binlogFile = ...\nEventDeserializer eventDeserializer = new EventDeserializer();\neventDeserializer.setCompatibilityMode(\n    EventDeserializer.CompatibilityMode.DATE_AND_TIME_AS_LONG,\n    EventDeserializer.CompatibilityMode.CHAR_AND_BINARY_AS_BYTE_ARRAY\n);\nBinaryLogFileReader reader = new BinaryLogFileReader(binlogFile, eventDeserializer);\ntry {\n    for (Event event; (event = reader.readEvent()) != null; ) {\n        ...\n    }\n} finally {\n    reader.close();\n}\n```\n\n#### Tapping into MySQL replication stream\n\n\u003e PREREQUISITES: Whichever user you plan to use for the BinaryLogClient, he MUST have [REPLICATION SLAVE](http://dev.mysql.com/doc/refman/5.5/en/privileges-provided.html#priv_replication-slave) privilege. Unless you specify binlogFilename/binlogPosition yourself (in which case automatic resolution won't kick in), you'll need [REPLICATION CLIENT](http://dev.mysql.com/doc/refman/5.5/en/privileges-provided.html#priv_replication-client) granted as well.\n\n```java\nBinaryLogClient client = new BinaryLogClient(\"hostname\", 3306, \"username\", \"password\");\nEventDeserializer eventDeserializer = new EventDeserializer();\neventDeserializer.setCompatibilityMode(\n    EventDeserializer.CompatibilityMode.DATE_AND_TIME_AS_LONG,\n    EventDeserializer.CompatibilityMode.CHAR_AND_BINARY_AS_BYTE_ARRAY\n);\nclient.setEventDeserializer(eventDeserializer);\nclient.registerEventListener(new EventListener() {\n\n    @Override\n    public void onEvent(Event event) {\n        ...\n    }\n});\nclient.connect();\n```\n\n\u003e You can register a listener for `onConnect` / `onCommunicationFailure` / `onEventDeserializationFailure` / `onDisconnect` using `client.registerLifecycleListener(...)`.\n\n\u003e By default, BinaryLogClient starts from the current (at the time of connect) master binlog position. If you wish to\nkick off from a specific filename or position, use `client.setBinlogFilename(filename)` + `client.setBinlogPosition(position)`.\n\n\u003e `client.connect()` is blocking (meaning that client will listen for events in the current thread). \n`client.connect(timeout)`, on the other hand, spawns a separate thread.  \n\n#### Controlling event deserialization\n\n\u003e You might need it for several reasons: \nyou don't want to waste time deserializing events you won't need; \nthere is no EventDataDeserializer defined for the event type you are interested in (or there is but it contains a bug); \nyou want certain type of events to be deserialized in a different way (perhaps *RowsEventData should contain table \nname and not id?); etc.\n\n```java\nEventDeserializer eventDeserializer = new EventDeserializer();\n\n// do not deserialize EXT_DELETE_ROWS event data, return it as a byte array\neventDeserializer.setEventDataDeserializer(EventType.EXT_DELETE_ROWS, \n    new ByteArrayEventDataDeserializer()); \n\n// skip EXT_WRITE_ROWS event data altogether\neventDeserializer.setEventDataDeserializer(EventType.EXT_WRITE_ROWS, \n    new NullEventDataDeserializer());\n\n// use custom event data deserializer for EXT_DELETE_ROWS\neventDeserializer.setEventDataDeserializer(EventType.EXT_DELETE_ROWS, \n    new EventDataDeserializer() {\n        ...\n    });\n\nBinaryLogClient client = ...\nclient.setEventDeserializer(eventDeserializer);\n```\n\n#### Exposing BinaryLogClient through JMX\n\n```java\nMBeanServer mBeanServer = ManagementFactory.getPlatformMBeanServer();\n\nBinaryLogClient binaryLogClient = ...\nObjectName objectName = new ObjectName(\"mysql.binlog:type=BinaryLogClient\");\nmBeanServer.registerMBean(binaryLogClient, objectName);\n\n// following bean accumulates various BinaryLogClient stats \n// (e.g. number of disconnects, skipped events)\nBinaryLogClientStatistics stats = new BinaryLogClientStatistics(binaryLogClient);\nObjectName statsObjectName = new ObjectName(\"mysql.binlog:type=BinaryLogClientStatistics\");\nmBeanServer.registerMBean(stats, statsObjectName);\n```\n\n#### Using SSL\n\n\u003e Introduced in 0.4.0.\n\nTLSv1.1 \u0026 TLSv1.2 require [JDK 7](http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6916074)+.  \nPrior to MySQL 5.7.10, MySQL supported only TLSv1 \n(see [Secure Connection Protocols and Ciphers](http://dev.mysql.com/doc/refman/5.7/en/secure-connection-protocols-ciphers.html)). \n\n\u003e To check that MySQL server is [properly configured with SSL support](http://dev.mysql.com/doc/refman/5.7/en/using-secure-connections.html) -\n`mysql -h host -u root -ptypeyourpasswordmaybe -e \"show global variables like 'have_%ssl';\"` (\"Value\" \nshould be \"YES\"). State of the current session can be determined using `\\s` (\"SSL\" should not be blank).\n\n```java\nSystem.setProperty(\"javax.net.ssl.trustStore\", \"/path/to/truststore.jks\");\nSystem.setProperty(\"javax.net.ssl.trustStorePassword\",\"truststore.password\");\nSystem.setProperty(\"javax.net.ssl.keyStore\", \"/path/to/keystore.jks\");\nSystem.setProperty(\"javax.net.ssl.keyStorePassword\", \"keystore.password\");\n\nBinaryLogClient client = ...\nclient.setSSLMode(SSLMode.VERIFY_IDENTITY);\n```\n\n## Implementation notes\n\n- data of numeric types (tinyint, etc) always returned signed(!) regardless of whether column definition includes \"unsigned\" keyword or not.\n- data of var\\*/\\*text/\\*blob types always returned as a byte array (for var\\* this is true starting from 1.0.0). \n\n## Frequently Asked Questions\n\n**Q**. How does a typical transaction look like?\n \n**A**. GTID event (if gtid_mode=ON) -\u003e QUERY event with \"BEGIN\" as sql -\u003e ... -\u003e XID event | QUERY event with \"COMMIT\" or \"ROLLBACK\" as sql. \n\n**Q**. EventData for inserted/updated/deleted rows has no information about table (except for some weird id). \nHow do I make sense out of it?  \n\n**A**. Each [WriteRowsEventData](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/WriteRowsEventData.java)/[UpdateRowsEventData](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/UpdateRowsEventData.java)/[DeleteRowsEventData](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/DeleteRowsEventData.java) event is preceded by [TableMapEventData](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/TableMapEventData.java) which\ncontains schema \u0026 table name. If for some reason you need to know column names (types, etc). - the easiest way is to\n\n```sql\nselect TABLE_SCHEMA, TABLE_NAME, COLUMN_NAME, ORDINAL_POSITION, COLUMN_DEFAULT, IS_NULLABLE, \nDATA_TYPE, CHARACTER_MAXIMUM_LENGTH, CHARACTER_OCTET_LENGTH, NUMERIC_PRECISION, NUMERIC_SCALE, \nCHARACTER_SET_NAME, COLLATION_NAME from INFORMATION_SCHEMA.COLUMNS;\n# see https://dev.mysql.com/doc/refman/5.6/en/columns-table.html for more information\n```\n\n(yes, binary log DOES NOT include that piece of information).\n\nYou can find JDBC snippet [here](https://github.com/shyiko/mysql-binlog-connector-java/issues/24#issuecomment-43747417).\n\n## Documentation\n\n#### API overview\n\nThere are two entry points - [BinaryLogClient](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/BinaryLogClient.java) (which you can use to read binary logs from a MySQL server) and \n[BinaryLogFileReader](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/BinaryLogFileReader.java) (for offline log processing). Both of them rely on [EventDeserializer](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/EventDeserializer.java) to deserialize \nstream of events. Each [Event](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/Event.java) consists of [EventHeader](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/EventHeader.java) (containing among other things reference to [EventType](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/EventType.java)) and \n[EventData](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/EventData.java). The aforementioned EventDeserializer has one [EventHeaderDeserializer](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/EventHeaderDeserializer.java) ([EventHeaderV4Deserializer](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/EventHeaderV4Deserializer.java) by default) \nand [a collection of EventDataDeserializer|s](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/EventDeserializer.java#L82). If there is no EventDataDeserializer registered for\nsome particular type of Event - default EventDataDeserializer kicks in ([NullEventDataDeserializer](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/NullEventDataDeserializer.java)).\n\n#### MySQL Internals Manual\n\nFor the insight into the internals of MySQL look [here](https://dev.mysql.com/doc/internals/en/index.html). [MySQL Client/Server Protocol](http://dev.mysql.com/doc/internals/en/client-server-protocol.html) and [The Binary Log](http://dev.mysql.com/doc/internals/en/binary-log.html) sections are particularly useful as a reference documentation for the `**.binlog.network` and `**.binlog.event` packages.\n\n## Real-world applications\n\nSome of the OSS using / built on top of mysql-binlog-conector-java: \n* [apache/nifi](https://github.com/apache/nifi) An easy to use, powerful, and reliable system to process and distribute data.\n* [debezium](https://github.com/debezium/debezium) A low latency data streaming platform for change data capture (CDC).\n* [mavenlink/changestream](https://github.com/mavenlink/changestream) - A stream of changes for MySQL built on Akka.\n* [mardambey/mypipe](https://github.com/mardambey/mypipe) MySQL binary log consumer with the ability to act on changed rows and publish changes to different systems with emphasis on Apache Kafka.\n* [ngocdaothanh/mydit](https://github.com/ngocdaothanh/mydit) MySQL to MongoDB data replicator.\n* [sharetribe/dumpr](https://github.com/sharetribe/dumpr) A Clojure library for live replicating data from a MySQL database.\n* [shyiko/rook](https://github.com/shyiko/rook) Generic Change Data Capture (CDC) toolkit.\n* [streamsets/datacollector](https://github.com/streamsets/datacollector) Continuous big data ingestion infrastructure.\n* [twingly/ecco](https://github.com/twingly/ecco) MySQL replication binlog parser in JRuby.\n* [zendesk/maxwell](https://github.com/zendesk/maxwell) A MySQL-to-JSON Kafka producer.\n* [zzt93/syncer](https://github.com/zzt93/syncer) A tool sync \u0026 manipulate data from MySQL/MongoDB to ES/Kafka/MySQL, which make 'Eventual Consistency' promise.\n\nIt's also used [on a large scale](https://twitter.com/atwinmutt/status/626816601078300672) in MailChimp. You can read about it [here](http://devs.mailchimp.com/blog/powering-mailchimp-pro-reporting/).  \n\n## Development\n\n```sh\ngit clone https://github.com/shyiko/mysql-binlog-connector-java.git\ncd mysql-binlog-connector-java\nmvn # shows how to build, test, etc. project\n```\n\n## Contributing\n\nIn lieu of a formal styleguide, please take care to maintain the existing coding style.  \nExecuting `mvn checkstyle:check` within project directory should not produce any errors.  \nIf you are willing to install [vagrant](http://www.vagrantup.com/) (required by integration tests) it's highly recommended\nto check (with `mvn clean verify`) that there are no test failures before sending a pull request.  \nAdditional tests for any new or changed functionality are also very welcomed.\n\n## License\n\n[Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshyiko%2Fmysql-binlog-connector-java","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshyiko%2Fmysql-binlog-connector-java","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshyiko%2Fmysql-binlog-connector-java/lists"}