https://github.com/samrocketman/example-mysql-live-migration

This repository serves as an example of performing a live MySQL database migration.
https://github.com/samrocketman/example-mysql-live-migration

migration mysql mysql-backup

Last synced: about 2 months ago
JSON representation

This repository serves as an example of performing a live MySQL database migration.

• Host: GitHub
• URL: https://github.com/samrocketman/example-mysql-live-migration
• Owner: samrocketman
• Created: 2017-02-10T04:20:02.000Z (over 9 years ago)
• Default Branch: main
• Last Pushed: 2020-10-18T07:11:50.000Z (over 5 years ago)
• Last Synced: 2025-02-24T08:13:35.851Z (over 1 year ago)
• Topics: migration, mysql, mysql-backup
• Language: Shell
• Homepage:
• Size: 4.88 KB
• Stars: 1
• Watchers: 3
• Forks: 3
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Example MySQL live migration

This project is to show an example of how to migrate a MySQL database from one

instance to another.  A common scenario is migrating a database from one remote

machine to another remote machine.

Prerequisites:

* At least 4 CPU cores.

* At least 16GB of system memory.

* Running on a computer which has `bash`, `mysql`, and `mysqldump` commands

  available (essentially bash and MySQL client only).

* [Vagrant][vagrant] for provisioning example database servers.  Vagrant depends

  on [VirtualBox][vbox].

This example uses dummy MySQL data from [`datacharmer/test_db`][test_db].

# Overview

The overview of a live MySQL migration process is:

1. On the source and destination database servers create a `datasync` database

   user which is capable of authenticating remotely with the database.  The user

   also needs all privileges granted on the database to be migrated.

2. Set [`mysql-migrate.sh`](mysql-migrate.sh) environment variables for both the

   source and destination database servers.  This is required to properly

   connect and authenticate with the source and destination database servers.

3. Execute `mysql-migrate.sh`.

# mysql-migrate.sh environment variables

`mysql-migrate.sh` script can have several environment variables set to

customize its behavior.  The default variables are to simplify running examples

in this repository.

Source database server vars:

* `SRC_DB_HOST` - hostname (default: `127.0.0.1`)

* `SRC_DB_PORT` - port of the MySQL service (default: `3333`)

* `SRC_DB_USER` - username of database user (default: `datasync`)

* `SRC_DB_PASSWORD` - password of database user (default: `syncpw`)

* `SRC_DB_DATABASE` - (default: `employees`)

Destination database server vars:

* `DST_DB_HOST` - hostname (default: `127.0.0.1`)

* `DST_DB_PORT` - port of the MySQL service (default: `3334`)

* `DST_DB_USER` - username of database user (default: `datasync`)

* `DST_DB_PASSWORD` - password of database user (default: `syncpw`)

> **WARNING:** `SRC_DB_PORT` and `DST_DB_PORT` are not default to 3306 which is

> the typical MySQL port.  Be aware of this.

# Run the example migration

Set up your database servers.  Execute:

    vagrant up

`vagrant up` will provision two database servers (`src_db` and `dst_db`) based

on the [`Vagrantfile`](Vagrantfile).  This will set up both database servers

with a MySQL service and creating the `datasync` database user on both servers.

It will also populate the `src_db` service with [dummy MySQL data][test_db] for

running the example migration.

Perform the migration.  Execute:

    bash mysql-migrate.sh

The above command will dump from the source database and import the dump to the

destination database.

> **Tip:** Before migrating, inspect both the source and destination database

> servers.

# Other helpful commands

Log into the source database server.

    vagrant ssh src_db

Log into the destination database server.

    vagrant ssh dst_db

You can log into the database directly from the database servers if you run

`mysql` as the `root` user.

    sudo mysql

Alternatively, connect to the destination database remotely using a local

`mysql` command.

    mysql -h 127.0.0.1 -P 3334 -u datasync -psyncpw

# Notes

* In the [`Vagrantfile`](Vagrantfile), the `datasync` user is allowed to

  authenticate from any remote machine.  This is not a recommended practice.

  Instead, limit the `datasync` user to authenticate only from known networks or

  computers in an active database environment.

* Don't use any example passwords in this repository to create accounts on

  active database servers.

* To improve performance of the destination database import, `/etc/my.cnf` can

  be edited to increase [`innodb_write_io_threads=64`][innodb_write_io] of the

  destination database server.

* If you encounter `mysql` timeouts because the data being exported is too large

  in a single statement, then add `--skip-extended-insert` to the `mysqldump`

  command.  Without the option migration time increases from 30 seconds to about

  20 minutes.  However, it is much more reliable.  I've encountered this need in

  databases for document management systems.

[test_db]: https://github.com/datacharmer/test_db

[vagrant]: https://www.vagrantup.com/

[vbox]: https://www.virtualbox.org/

[innodb_write_io]: https://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_write_io_threads