https://github.com/larsnovikov/horgh-replicator
Golang binlog replication from MySQL to MySQL, PostgreSQL, Vertica, Clickhouse
https://github.com/larsnovikov/horgh-replicator
binlog clickhouse go golang master-slave master-slave-replication mysql postgresql replication tungsten vertica
Last synced: 5 months ago
JSON representation
Golang binlog replication from MySQL to MySQL, PostgreSQL, Vertica, Clickhouse
- Host: GitHub
- URL: https://github.com/larsnovikov/horgh-replicator
- Owner: larsnovikov
- Created: 2019-03-22T05:37:27.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2020-02-12T06:37:36.000Z (over 6 years ago)
- Last Synced: 2025-05-03T16:36:21.455Z (about 1 year ago)
- Topics: binlog, clickhouse, go, golang, master-slave, master-slave-replication, mysql, postgresql, replication, tungsten, vertica
- Language: Go
- Homepage:
- Size: 33.3 MB
- Stars: 55
- Watchers: 3
- Forks: 26
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-clickhouse - larsnovikov/horgh-replicator - This project is a Golang-based tool for binlog replication from MySQL to multiple databases, including PostgreSQL and ClickHouse. (Integrations / ETL and Data Processing)
README
# Mysql binlog replicator
##### Based on [JackShadow/go-binlog-example](https://github.com/JackShadow/go-binlog-example)
### Site
See full docs [here](https://larsnovikov.github.io/horgh)
### Master types
- MySQL
### Slave types
- MySQL
- PostgreSQL
- Yandex ClickHouse
- HP Vertica
### Quick start
See quick start tutorial [here](https://larsnovikov.github.io/horgh#quick_start)
### Testing
- Copy `src/.env.dist` to `src/.env` and set credentials.
- Configure your my MySQL master as `examples/master/mysql.conf`.
Don't forget to set `binlog_do_db=` and restart MySQL service.
- Create databases and tables as `examples/sql/`.
- Start Docker as `make start-dev`
- Run as `cd src` and `go run main.go listen` in docker container.
- Copy `examples/configs/user.json` and `examples/configs/post.json` to `src/system/configs`
- Execute `cd src` and `go run main.go load`
### Add tables to replicator
- Use `create-model ` to create json config for your table.
- Create table on slave.
- Set `` to `TABLE` param in `src/.env`
- Use `build-slave` to copy table data from master and set start position of log for table listener.
### Custom handlers for field value
- Create `plugins/user//handler.go` like `plugins/system/set_value/handler.go`
- Execute `go build -buildmode=plugin -o plugins/user//handler.so plugins/user//handler.go`
- Add to field description in your `src/system/configs/.json`
```
"beforeSave": {
"handler": "user/",
"params": [
"***"
]
}
```
##### System handlers
- If you want to set custom field value use `system/set_value` as `handler` param. Don't forget to set `params: [""]`
### Tools
- `set-position ` set start position of log for table listener
- `load` start loader for replication testing (for default tables user and post)
- `create-model ` create model json-file by master table structure
- `build-slave` create master table dump, restore this dump in slave, set start position of log for table listener
- `destroy-slave` truncate table, set empty position of log for table listener
### Modes
- Prod mode: build app and execute listener.
Use `make build-prod` and `make start-prod` to start and `make stop-prod` to stop.
- Dev mode: provides the opportunity for manual start and debug.
Use `make start-dev` to start and `make stop-dev` to stop.