https://github.com/sjmudd/mysql_data_transporter

Go code to help transport data between MySQL servers more efficiently
https://github.com/sjmudd/mysql_data_transporter

Last synced: 12 months ago
JSON representation

Go code to help transport data between MySQL servers more efficiently

• Host: GitHub
• URL: https://github.com/sjmudd/mysql_data_transporter
• Owner: sjmudd
• License: bsd-2-clause
• Created: 2017-07-03T19:39:56.000Z (almost 9 years ago)
• Default Branch: master
• Last Pushed: 2017-07-03T19:44:43.000Z (almost 9 years ago)
• Last Synced: 2025-03-17T11:21:36.453Z (about 1 year ago)
• Language: Go
• Size: 82 KB
• Stars: 1
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# mysql_data_transporter

Go code to help transport data between MySQL servers more efficiently

TASK: to successfully and efficiently copy data from one server to another.

* should be vendor nuetral if possible so should work between different vendors

  and major versions where possible.

* mysqldump doesn't work efficiently (too slow)

* mysqlpump works for 5.7 and later

* pt-archiver looked good but if you're not careful it'll delete your data (groan)

* I want to do this by pulling data from the source server in

  parallel (configurable) in chunks and push to the destination server in chunks.

* there are parcial tools for doing this but nothing that's complete

* Other options that might be of interest: do this by renaming

  tables and making blackhole tables and inserting into the bh tables

  generating binlogs (RBR format)

* data size is expected to be large. Say 500GB plus.

* do not store intermediate data on disk: we want to avoid this