{"id":20651572,"url":"https://github.com/rafaeljusto/gomaxscale","last_synced_at":"2025-04-17T14:16:28.744Z","repository":{"id":43704161,"uuid":"460857145","full_name":"rafaeljusto/gomaxscale","owner":"rafaeljusto","description":"CDC consumer for MaxScale","archived":false,"fork":false,"pushed_at":"2022-10-16T21:07:27.000Z","size":39,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-29T06:11:23.587Z","etag":null,"topics":["cdc","database","golang","mariadb","maxscale","mysql"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rafaeljusto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null}},"created_at":"2022-02-18T13:07:57.000Z","updated_at":"2023-08-24T14:09:15.000Z","dependencies_parsed_at":"2023-01-20T02:17:41.348Z","dependency_job_id":null,"html_url":"https://github.com/rafaeljusto/gomaxscale","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaeljusto%2Fgomaxscale","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaeljusto%2Fgomaxscale/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaeljusto%2Fgomaxscale/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaeljusto%2Fgomaxscale/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rafaeljusto","download_url":"https://codeload.github.com/rafaeljusto/gomaxscale/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249347354,"owners_count":21255148,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdc","database","golang","mariadb","maxscale","mysql"],"created_at":"2024-11-16T17:28:17.038Z","updated_at":"2025-04-17T14:16:28.716Z","avatar_url":"https://github.com/rafaeljusto.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# gomaxscale\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/rafaeljusto/gomaxscale/v2.svg)](https://pkg.go.dev/github.com/rafaeljusto/gomaxscale/v2)\n[![license](http://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/rafaeljusto/gomaxscale/master/LICENSE)\n\nGo library that allows consuming from [MaxScale](https://mariadb.com/kb/en/maxscale/)\nCDC listener. Useful for detecting database changes via binlog.\n\nThis consumer follows the connection protocol defined by MaxScale 6\n[here](https://mariadb.com/kb/en/mariadb-maxscale-6-change-data-capture-cdc-protocol/).\n\n## Testing with Docker\n\nFor this test environment the following file structure was used:\n\n* 📂 `mariadb-config`\n  - 📄 `mariadb.cnf`\n* 📂 `mariadb-init`\n  - 📄 `00_schema.sql`\n* 📂 `maxscale-config`\n  - 📄 `maxscale.cnf`\n* 📄 `docker-compose.yml`\n* 📄 `consumer.go`\n\n### mariadb.cnf\n\nWe need to enable replication in the MariaDB master database:\n```dosini\n[mysqld]\nserver_id=1\nbinlog_format=row\nbinlog_row_image=full\nlog-bin=/var/log/mysql/mariadb-bin\n```\n\n### 00_schema.sql\n\nA basic schema adding the MaxScale user, and some testing database to play with:\n\n```sql\nRESET MASTER;\n\n-- https://mariadb.com/kb/en/mariadb-maxscale-6-setting-up-mariadb-maxscale/#creating-a-user-account-for-maxscale\nCREATE USER 'maxuser'@'%' IDENTIFIED BY 'maxpwd';\nGRANT REPLICATION SLAVE ON *.* TO 'maxuser'@'%';\nGRANT REPLICATION CLIENT ON *.* TO 'maxuser'@'%';\nGRANT SELECT ON mysql.user TO 'maxuser'@'%';\nGRANT SELECT ON mysql.db TO 'maxuser'@'%';\nGRANT SELECT ON mysql.tables_priv TO 'maxuser'@'%';\nGRANT SELECT ON mysql.columns_priv TO 'maxuser'@'%';\nGRANT SELECT ON mysql.procs_priv TO 'maxuser'@'%';\nGRANT SELECT ON mysql.proxies_priv TO 'maxuser'@'%';\nGRANT SELECT ON mysql.roles_mapping TO 'maxuser'@'%';\nGRANT SHOW DATABASES ON *.* TO 'maxuser'@'%';\nFLUSH PRIVILEGES;\n\nDROP DATABASE IF EXISTS example;\nCREATE DATABASE IF NOT EXISTS example;\n\n-- Allows MaxScale user to run 'SHOW CREATE TABLE' for DDL events\n-- https://mariadb.com/kb/en/mariadb-maxscale-6-avrorouter/#avro-schema-generator\nGRANT SELECT ON example.* TO 'maxuser'@'%';\nFLUSH PRIVILEGES;\n\nUSE example;\n\nDROP TABLE IF EXISTS users;\nCREATE TABLE users (\n  `id` INT NOT NULL AUTO_INCREMENT,\n  `name` VARCHAR(255) NOT NULL,\n  `email` VARCHAR(255) NOT NULL,\n  PRIMARY KEY (id)\n);\n\nINSERT INTO `users` (`name`, `email`) VALUES ('John Doe', 'john@doe.com');\nINSERT INTO `users` (`name`, `email`) VALUES ('Jane Doe', 'jane@doe.com');\n```\n\n### maxscale.cnf\n\nMaxScale configuration to configure the [Avro router](https://mariadb.com/kb/en/mariadb-maxscale-6-avrorouter/)\n([direct replication mode](https://mariadb.com/kb/en/mariadb-maxscale-6-avrorouter/#direct-replication-mode))\nand expose a listener so `gomaxscale` can retrieve the information.\n\n```dosini\n[MaxScale]\nthreads=1\nadmin_secure_gui=false\nthreads=auto\nadmin_host=0.0.0.0\n\n[server1]\ntype=server\naddress=db\nport=3306\nprotocol=MariaDBBackend\n\n[cdc-service]\ntype=service\nrouter=avrorouter\nservers=server1\nserver_id=1\nuser=maxuser\npassword=maxpwd\n\n[cdc-listener]\ntype=listener\nservice=cdc-service\nprotocol=CDC\nport=4001\n\n[MariaDB-Monitor]\ntype=monitor\nmodule=mariadbmon\nservers=server1\nuser=maxuser\npassword=maxpwd\nmonitor_interval=5000\n```\n\n### docker-compose.yml\n\nTo setup a MariaDB database and a MaxScale server we will use docker-compose\nwith the following configuration:\n\n```yaml\nversion: '2.4'\nservices:\n  db:\n    container_name: \"lab-db\"\n    image: mariadb:10.3.8\n    volumes:\n      - ./mariadb-config:/etc/mysql/conf.d\n      - ./mariadb-init:/docker-entrypoint-initdb.d\n    environment:\n      MYSQL_ROOT_PASSWORD: abc123\n    healthcheck:\n      test: [\"CMD\", \"mysqladmin\", \"ping\", \"--silent\"]\n\n  dbproxy:\n    container_name: \"lab-maxscale\"\n    image: mariadb/maxscale:6.2\n    volumes:\n      - ./maxscale-config/maxscale.cnf:/etc/maxscale.cnf\n    ports:\n      - 4001:4001\n    depends_on:\n      - db\n```\n\n### consumer.go\n\nA local Go file will consume and log the modified items in the database:\n\n```go\npackage main\n\nimport (\n  \"fmt\"\n  \"log\"\n  \"os\"\n  \"os/signal\"\n  \"syscall\"\n\n  \"github.com/rafaeljusto/gomaxscale/v2\"\n)\n\nfunc main() {\n  consumer := gomaxscale.NewConsumer(\"127.0.0.1:4001\", \"example\", \"users\",\n    gomaxscale.WithAuth(\"maxuser\", \"maxpwd\"),\n  )\n  err := consumer.Start()\n  if err != nil {\n    log.Fatal(err)\n  }\n  defer consumer.Close()\n\n  fmt.Println(\"start consuming events\")\n\n  done := make(chan bool)\n  go func() {\n    consumer.Process(func(event gomaxscale.CDCEvent) {\n      switch e := event.(type) {\n      case gomaxscale.DDLEvent:\n        fmt.Printf(\"ddl event detected on database '%s' and table '%s'\\n\",\n          e.Database, e.Table)\n      case gomaxscale.DMLEvent:\n        fmt.Printf(\"dml '%s' event detected\\n\", e.Type)\n      }\n    })\n    done \u003c- true\n  }()\n\n  signalChanel := make(chan os.Signal, 1)\n  signal.Notify(signalChanel, syscall.SIGINT, syscall.SIGTERM)\n\n  select {\n  case \u003c-signalChanel:\n  case \u003c-done:\n  }\n\n  fmt.Println(\"terminating\")\n}\n```\n\n### Running\n\nFirst, start all services:\n```\n% docker-compose up -d\n```\n\nThen we can start consuming the items:\n```\n% go run consumer.go\n```\n\nTo see the magic happening you could do some database changes:\n```\n% docker-compose exec db mysql -u root -p abc123 -D example \\\n  -e \"INSERT INTO users (name, email) VALUES ('James Doe', 'james@doe.com')\"\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frafaeljusto%2Fgomaxscale","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frafaeljusto%2Fgomaxscale","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frafaeljusto%2Fgomaxscale/lists"}