{"id":16577237,"url":"https://github.com/allwefantasy/spark-binlog","last_synced_at":"2025-03-17T14:17:49.558Z","repository":{"id":36579882,"uuid":"192836823","full_name":"allwefantasy/spark-binlog","owner":"allwefantasy","description":"A library for querying Binlog with Apache Spark structure streaming,   for Spark SQL , DataFrames and [MLSQL](https://www.mlsql.tech).","archived":false,"fork":false,"pushed_at":"2023-04-21T20:42:49.000Z","size":160,"stargazers_count":153,"open_issues_count":12,"forks_count":54,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-02T13:08:31.240Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/allwefantasy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-20T02:45:52.000Z","updated_at":"2025-02-07T03:31:28.000Z","dependencies_parsed_at":"2024-10-26T20:29:01.929Z","dependency_job_id":"4c6d8082-7f19-4041-a7d9-a7db142df701","html_url":"https://github.com/allwefantasy/spark-binlog","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allwefantasy%2Fspark-binlog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allwefantasy%2Fspark-binlog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allwefantasy%2Fspark-binlog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allwefantasy%2Fspark-binlog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/allwefantasy","download_url":"https://codeload.github.com/allwefantasy/spark-binlog/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244047645,"owners_count":20389206,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T22:10:14.785Z","updated_at":"2025-03-17T14:17:49.525Z","avatar_url":"https://github.com/allwefantasy.png","language":"Scala","funding_links":[],"categories":["数据库中间件"],"sub_categories":["Spring Cloud框架"],"readme":"# Spark Binlog Library\n\nA library for querying Binlog with Apache Spark structure streaming, \nfor Spark SQL , DataFrames and [MLSQL](http://www.mlsql.tech).\n\n1. [jianshu: How spark-binlog works](https://www.jianshu.com/p/e7c3e84a0ea7)\n2. [medium: How spark-binlog works](https://medium.com/@williamsmith_74955/how-spark-binlog-works-323c16fb1498)\n  \n## Requirements\n\nThis library requires Spark 2.4+ (tested).\nSome older versions of Spark may work too but they are not officially supported.\n\n## Linking \n\nYou can link against this library in your program at the following coordinates:\n\n### Scala 2.11\n\nThis is the latest stable versions.\n\nMySQL Binlog:\n\n```      \ngroupId: tech.mlsql\nartifactId: mysql-binlog_2.11\nversion: 1.0.4\n```\n\nHBase WAL:\n\n```      \ngroupId: tech.mlsql\nartifactId: hbase-wal_2.11\nversion: 1.0.4\n```\n\n## Limitation\n\n1. mysql-binlog only support insert/update/delete events. The other events will ignore.\n2. hbase-wal only support Put/Delete events. The other events will ignore.\n\n## MySQL Binlog Usage\n\nThe example should work with [delta-plus](https://github.com/allwefantasy/delta-plus)\n\nMLSQL Code:\n\n```sql\nset streamName=\"binlog\";\n\nload binlog.`` where \nhost=\"127.0.0.1\"\nand port=\"3306\"\nand userName=\"xxxxx\"\nand password=\"xxxxx\"\nand databaseNamePattern=\"mlsql_console\"\nand tableNamePattern=\"script_file\"\nas table1;\n\nsave append table1  \nas rate.`mysql_{db}.{table}` \noptions mode=\"Append\"\nand idCols=\"id\"\nand duration=\"5\"\nand syncType=\"binlog\"\nand checkpointLocation=\"/tmp/cpl-binlog2\";\n```\n\nDataFrame Code:\n\n```scala\nval spark = SparkSession.builder()\n      .master(\"local[*]\")\n      .appName(\"Binlog2DeltaTest\")\n      .getOrCreate()\n\nval df = spark.readStream.\n  format(\"org.apache.spark.sql.mlsql.sources.MLSQLBinLogDataSource\").\n  option(\"host\",\"127.0.0.1\").\n  option(\"port\",\"3306\").\n  option(\"userName\",\"root\").\n  option(\"password\",\"123456\").\n  option(\"databaseNamePattern\",\"test\").\n  option(\"tableNamePattern\",\"mlsql_binlog\").\n  load()\n\nval query = df.writeStream.\n  format(\"org.apache.spark.sql.delta.sources.MLSQLDeltaDataSource\").\n  option(\"__path__\",\"/tmp/datahouse/{db}/{table}\").\n  option(\"path\",\"{db}/{table}\").\n  option(\"mode\",\"Append\").\n  option(\"idCols\",\"id\").\n  option(\"duration\",\"3\").\n  option(\"syncType\",\"binlog\").\n  option(\"checkpointLocation\", \"/tmp/cpl-binlog2\").\n  outputMode(\"append\")\n  .trigger(Trigger.ProcessingTime(\"3 seconds\"))\n  .start()\n\nquery.awaitTermination()\n\n```\n\n\nBefore you run the streaming application, make sure you have fully sync the table \n\nMLSQL Code:\n\n```sql\nconnect jdbc where\n url=\"jdbc:mysql://127.0.0.1:3306/mlsql_console?characterEncoding=utf8\u0026zeroDateTimeBehavior=convertToNull\u0026tinyInt1isBit=false\"\n and driver=\"com.mysql.jdbc.Driver\"\n and user=\"xxxxx\"\n and password=\"xxxx\"\n as db_cool;\n \nload jdbc.`db_cool.script_file`  as script_file;\n\nrun script_file as TableRepartition.`` where partitionNum=\"2\" and partitionType=\"range\" and partitionCols=\"id\"\nas rep_script_file;\n\nsave overwrite rep_script_file as delta.`mysql_mlsql_console.script_file` ;\n\nload delta.`mysql_mlsql_console.script_file`  as output;\n```\n\nDataFrame Code:\n\n```scala\nimport org.apache.spark.sql.SparkSession\nval spark = SparkSession.builder()\n  .master(\"local[*]\")\n  .appName(\"wow\")\n  .getOrCreate()\n\nval mysqlConf = Map(\n  \"url\" -\u003e \"jdbc:mysql://localhost:3306/mlsql_console?characterEncoding=utf8\u0026zeroDateTimeBehavior=convertToNull\u0026tinyInt1isBit=false\",\n  \"driver\" -\u003e \"com.mysql.jdbc.Driver\",\n  \"user\" -\u003e \"xxxxx\",\n  \"password\" -\u003e \"xxxx\",\n  \"dbtable\" -\u003e \"script_file\"\n)\n\nimport org.apache.spark.sql.functions.col\nvar df = spark.read.format(\"jdbc\").options(mysqlConf).load()\ndf = df.repartitionByRange(2, col(\"id\") )\ndf.write\n  .format(\"org.apache.spark.sql.delta.sources.MLSQLDeltaDataSource\").\n  mode(\"overwrite\").\n  save(\"/tmp/datahouse/mlsql_console/script_file\")\nspark.close()\n```\n\n## HBase WAL Usage\n\nDataFrame code:\n\n```scala\nval spark = SparkSession.builder()\n      .master(\"local[*]\")\n      .appName(\"HBase WAL Sync\")\n      .getOrCreate()\n\n    val df = spark.readStream.\n      format(\"org.apache.spark.sql.mlsql.sources.hbase.MLSQLHBaseWALDataSource\").\n      option(\"walLogPath\", \"/Users/allwefantasy/Softwares/hbase-2.1.8/WALs\").\n      option(\"oldWALLogPath\", \"/Users/allwefantasy/Softwares/hbase-2.1.8/oldWALs\").\n      option(\"startTime\", \"1\").\n      option(\"databaseNamePattern\", \"test\").\n      option(\"tableNamePattern\", \"mlsql_binlog\").\n      load()\n\n    val query = df.writeStream.\n      format(\"console\").\n      option(\"mode\", \"Append\").\n      option(\"truncate\", \"false\").\n      option(\"numRows\", \"100000\").\n      option(\"checkpointLocation\", \"/tmp/cpl-binlog25\").\n      outputMode(\"append\")\n      .trigger(Trigger.ProcessingTime(\"10 seconds\"))\n      .start()\n\n    query.awaitTermination()\n```\n\n## RoadMap\n\nWe hope we can support more DBs including traditional DB e.g Oracle and \nNoSQL e.g. HBase(WAL),ES,Cassandra in future.  \n\n\n## How to get the initial offset\n\nYou can mannually set binlog offset, For example:\n\n```\nbingLogNamePrefix=\"mysql-bin\"\nbinlogIndex=\"4\"\nbinlogFileOffset=\"4\"\n```\n\nTry using command like following to get the offset you want:\n\n```\nmysql\u003e show master status;\n+------------------+----------+--------------+------------------+-------------------+\n| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |\n+------------------+----------+--------------+------------------+-------------------+\n| mysql-bin.000014 | 34913156 |              |                  |                   |\n+------------------+----------+--------------+------------------+-------------------+\n1 row in set (0.04 sec)\n```  \n\nIn this example, we knows that:\n\n```       \nbingLogNamePrefix      binlogFileOffset   binlogFileOffset\nmysql-bin        .     000014             34913156\n```\n\nthis means you should configure parameters like this: \n\n```\nbingLogNamePrefix=\"mysql-bin\"\nbinlogIndex=\"14\"\nbinlogFileOffset=\"34913156\"\n```\n\n\nOr you can use `mysqlbinlog` command.\n```\nmysqlbinlog \\ \n--start-datetime=\"2019-06-19 01:00:00\" \\ \n--stop-datetime=\"2019-06-20 23:00:00\" \\ \n--base64-output=decode-rows \\\n-vv  master-bin.000004\n\n```\n\n## Questions\n\n## Q1\n\nPeople may meet some log like following:\n\n```\nTrying to restore lost connectioin to .....\nConnected to ....\n```\n\nPlease check the server_id is configured in my.cnf of your MySQL Server.\n\n## Q2\n\nWhen you have started your stream to consume the binlog, but it seem nothong happen or just print :\n\n```\nBatch: N\n-------------------------------------------\n+-----+\n|value|\n+-----+\n+-----+\n```\n\nPlease check spark log:\n\n```\n20/06/18 11:57:00 INFO MicroBatchExecution: Streaming query made progress: {\n  \"id\" : \"e999af90-8d0a-48e2-b9fc-fcf1e140f622\",\n  \"runId\" : \"547ce891-468a-43c5-bb62-614b38f60c39\",\n  \"name\" : null,\n  \"timestamp\" : \"2020-06-18T03:57:00.002Z\",\n  \"batchId\" : 1,\n  \"numInputRows\" : 1,\n  \"inputRowsPerSecond\" : 0.4458314757021846,\n  \"processedRowsPerSecond\" : 2.9673590504451037,\n  \"durationMs\" : {\n    \"addBatch\" : 207,\n    \"getBatch\" : 3,\n    \"getOffset\" : 15,\n    \"queryPlanning\" : 10,\n    \"triggerExecution\" : 337,\n    \"walCommit\" : 63\n  },\n  \"stateOperators\" : [ ],\n  \"sources\" : [ {\n    \"description\" : \"MLSQLBinLogSource(ExecutorBinlogServer(192.168.111.14,52612),....\",\n    \"startOffset\" : 160000000004104,\n    \"endOffset\" : 170000000000154,\n    \"numInputRows\" : 0,\n    \"inputRowsPerSecond\" : 0,\n    \"processedRowsPerSecond\" : 0\n  } ],\n  \"sink\" : {\n    \"description\" : \"org.apache.spark.sql.execution.streaming.ConsoleSinkProvider@4f82b82f\"\n  }\n}\n```\n\nAs we can see, the startOffset/f is changing but the  numInputRows is not chagned. Please try a table with a simple\nschema to make sure the binlog connection  works fine.\n\nIf the simple schema table works fine, this is may caused by some special sql type. Please address an issue and\npaste spark log and your target table schema.\n\nYou can use code like this to test in your local machine:\n\n```scala\npackage tech.mlsql.test.binlogserver\n\nimport java.sql.Timestamp\n\nimport org.apache.spark.sql.SparkSession\nimport org.apache.spark.sql.streaming.Trigger\nimport org.scalatest.FunSuite\n\n\nobject Main{\n  def main(args: Array[String]): Unit = {\n    val spark = SparkSession.builder()\n          .master(\"local[*]\")\n          .appName(\"MySQL B Sync\")\n          .getOrCreate()\n\n        val df = spark.readStream.\n          format(\"org.apache.spark.sql.mlsql.sources.MLSQLBinLogDataSource\").\n          option(\"host\", \"127.0.0.1\").\n          option(\"port\", \"3306\").\n          option(\"userName\", \"xxxx\").\n          option(\"password\", \"xxxx\").\n          option(\"databaseNamePattern\", \"wow\").\n          option(\"tableNamePattern\", \"users\").\n          option(\"bingLogNamePrefix\", \"mysql-bin\").\n          option(\"binlogIndex\", \"16\").\n          option(\"binlogFileOffset\", \"3869\").\n          option(\"binlog.field.decode.first_name\", \"UTF-8\").\n          load()\n\n        // print the binlog(json format)\n        val query = df.writeStream.\n              format(\"console\").\n              option(\"mode\", \"Append\").\n              option(\"truncate\", \"false\").\n              option(\"numRows\", \"100000\").\n              option(\"checkpointLocation\", \"/tmp/cpl-mysql6\").\n              outputMode(\"append\")\n              .trigger(Trigger.ProcessingTime(\"10 seconds\"))\n              .start()\n\n        query.awaitTermination()\n  }\n}\n\n```\n\n\n\n\n\n  \n\n \n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallwefantasy%2Fspark-binlog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fallwefantasy%2Fspark-binlog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallwefantasy%2Fspark-binlog/lists"}