{"id":13694148,"url":"https://github.com/zalora/binlog-parser","last_synced_at":"2026-01-14T21:36:32.877Z","repository":{"id":47601690,"uuid":"88000627","full_name":"zalora/binlog-parser","owner":"zalora","description":"A parser for MySQL binlog files that creates JSON messages. Useful for creating a stream of database events that can be stored and analyzed.","archived":false,"fork":false,"pushed_at":"2024-03-21T08:06:36.000Z","size":336,"stargazers_count":142,"open_issues_count":5,"forks_count":45,"subscribers_count":41,"default_branch":"master","last_synced_at":"2025-05-03T01:46:11.498Z","etag":null,"topics":["audit-logs","binlog","golang","mysql","parsers"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zalora.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-12T02:26:46.000Z","updated_at":"2025-04-18T07:19:46.000Z","dependencies_parsed_at":"2024-11-12T20:32:54.395Z","dependency_job_id":null,"html_url":"https://github.com/zalora/binlog-parser","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/zalora/binlog-parser","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalora%2Fbinlog-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalora%2Fbinlog-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalora%2Fbinlog-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalora%2Fbinlog-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zalora","download_url":"https://codeload.github.com/zalora/binlog-parser/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalora%2Fbinlog-parser/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28435523,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T21:32:52.117Z","status":"ssl_error","status_checked_at":"2026-01-14T21:32:33.442Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["audit-logs","binlog","golang","mysql","parsers"],"created_at":"2024-08-02T17:01:25.455Z","updated_at":"2026-01-14T21:36:32.858Z","avatar_url":"https://github.com/zalora.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# binlog-parser\n\n[![Build Status](https://travis-ci.org/zalora/binlog-parser.svg?branch=master)](https://travis-ci.org/zalora/binlog-parser)\n\nA tool for parsing a MySQL binlog file to JSON. Reads a binlog input file, queries a database for field names, writes JSON to stdout. The output looks like this:\n\n    {\n        \"Header\": {\n            \"Schema\": \"test_db\",\n            \"Table\": \"buildings\",\n            \"BinlogMessageTime\": \"2017-04-13T06:34:30Z\",\n            \"BinlogPosition\": 397,\n            \"XId\": 9\n        },\n        \"Type\": \"Insert\",\n        \"Data\": {\n            \"Row\": {\n                \"address\": \"3950 North 1st Street CA 95134\",\n                \"building_name\": \"ACME Headquaters\",\n                \"building_no\": 1\n            },\n            \"MappingNotice\": \"\"\n        }\n    }\n    ...\n\n# Installation\n\nRequires Go version 1.7 or higher.\n\n    $ git clone https://github.com/zalora/binlog-parser.git\n    $ cd binlog-parser\n    $ git submodule update --init\n    $ make\n    $ ./bin/binlog-parser -h\n\n## Assumptions\n\n- It is assumed that MySQL row-based binlog format is used (or mixed, but be aware, that then only the row-formatted data in mixed binlogs can be extracted)\n- This tool is written with MySQL 5.6 in mind, although it should also work for MariaDB when GTIDs are not used\n\n# Usage\n\nRun `binlog-parser -h` to get the list of available options:\n\n    Usage:\tbinlog-parser [options ...] binlog\n\n    Options are:\n\n      -alsologtostderr\n        \tlog to standard error as well as files\n      -include_schemas string\n        \tcomma-separated list of schemas to include\n      -include_tables string\n        \tcomma-separated list of tables to include\n      -log_backtrace_at value\n        \twhen logging hits line file:N, emit a stack trace\n      -log_dir string\n        \tIf non-empty, write log files in this directory\n      -logtostderr\n        \tlog to standard error instead of files\n      -prettyprint\n        \tPretty print json\n      -stderrthreshold value\n        \tlogs at or above this threshold go to stderr\n      -v value\n        \tlog level for V logs\n      -vmodule value\n        \tcomma-separated list of pattern=N settings for file-filtered logging\n\n    Required environment variables:\n\n    DB_DSN\t Database connection string, needs read access to information_schema\n\n## Example usage\n\nUsing `dbuser` and no password, connecting to `information_schema` database on localhost, parsing the binlog file `/some/binlog.bin`:\n\n    DB_DSN=dbuser@/information_schema ./binlog-parser /some/binlog.bin\n\n## Matching field names and data\n\nThe mysql binlog format doesn't include the fieldnames for row events (INSERT/UPDATE/DELETE). As the goal of the parser is to output\nusable JSON, it connects to a running MySQL instance and queries the `information_schema` database for information on field names in the table.\n\nThe database connection is creatd by using the environment variable `DB_DSN`, which should contain the database credentials in the form of\n`user:password@/dbname` - the format that the [Go MySQL driver](https://godoc.org/github.com/go-sql-driver/mysql) uses.\n\n## Effect of schema changes\n\nAs this tool doesn't keep an internal representation of the database schema, it is very well possible that the database schema and the schema used in the\nqueries in the binlog file already have diverged (e. g. parsing a binlog file from a few days ago, but the schema on the main database already changed\nby dropping or adding columns).\n\nThe parser will NOT make an attempt to map data to fields in a table if the information schema retuns more or too less columns\ncompared to the format found in the binlog. The field names will be mapped as \"unknown\":\n\n    {\n        \"Header\": {\n            \"Schema\": \"test_db\",\n            \"Table\": \"employees\",\n            \"BinlogMessageTime\": \"2017-04-13T08:02:04Z\",\n            \"BinlogPosition\": 635,\n            \"XId\": 8\n        },\n        \"Type\": \"Insert\",\n        \"Data\": {\n            \"Row\": {\n                \"(unknown_0)\": 1,\n                \"(unknown_1)\": \"2017-04-13\",\n                \"(unknown_2)\": \"Max\",\n                \"(unknown_3)\": \"Mustermann\"\n            },\n            \"MappingNotice\": \"column names array is missing field(s), will map them as unknown_*\"\n        }\n    }\n\n### More complex case\n\nChanging the order of fields in a table can lead to unexpected parser results. Consider an example binlog file `A.bin`.\nA query like `INSERT INTO db.foo SET field_1 = 10, field_2 = 20` will look in the binlog like this:\n\n    ...\n    ### INSERT INTO `db`.`foo`\n    ### SET\n    ###   @1=20 /* ... */\n    ###   @2=20 /* ... */\n    ...\n\nThe parser queries `information_schema` for the field names of the `db.foo` table:\n\n    +-------------+-----+\n    | Field       | ... |\n    +-------------+-----+\n    | field_1     | ... |\n    | field_2     | ... |\n    +-------------+-----+\n\nThe fields will be mapped by the parser in the order as specified in the table and the JSON will look like this:\n\n    {\n        ...\n        \"Type\": \"Insert\",\n        \"Data\": {\n            \"Row\": {\n                \"field_1\": 10,\n                \"field_2\": 20\n            }\n        }\n    }\n    ...\n\nNow if a schema change happened after some time, `db.foo` fields might look now like this (the order of the fiels changed):\n\n    +-------------+-----+\n    | Field       | ... |\n    +-------------+-----+\n    | field_2     | ... |\n    | field_1     | ... |\n    +-------------+-----+\n\nIf you parse the same binlog file `A.bin` now again, but against the new schema of `db.foo` (in which the fields changed position), the resulting JSON\nwill look like that:\n\n\n    {\n        ...\n        \"Type\": \"Insert\",\n        \"Data\": {\n            \"Row\": {\n                \"field_2\": 10,\n                \"field_1\": 20\n            }\n        }\n    }\n    ...\n\nThis means you have to be very careful when parsing old binlog files, as the db schema can have evolved since the binlog was generated and the parser\nhas no way of knowing of these changes.\n\nIf this limitation is not acceptable, some tools like [Maxwell's Daemon by Zendesk](https://github.com/zendesk/maxwell) can work around that issue at the cost of greater complexity.\n\n# Releases\n\nHow to do a release:\n\n    git tag -a X.X.X -m \"... release note ...\"\n    git push --follow-tags\n\nTravis CI will attach a statically built binary to the release tag on GitHub.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzalora%2Fbinlog-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzalora%2Fbinlog-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzalora%2Fbinlog-parser/lists"}