{"id":23172070,"url":"https://github.com/choonkeat/dbmigrate","last_synced_at":"2025-08-18T07:32:25.923Z","repository":{"id":57492516,"uuid":"162682709","full_name":"choonkeat/dbmigrate","owner":"choonkeat","description":"rails migrate inspired approach to database schema migrations but with plain sql files. and much faster.","archived":false,"fork":false,"pushed_at":"2023-12-03T03:39:18.000Z","size":81,"stargazers_count":10,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-08-14T01:52:28.699Z","etag":null,"topics":["database","golang","mariadb","migration-tool","mysql","postgresql","sql"],"latest_commit_sha":null,"homepage":"https://hub.docker.com/r/choonkeat/dbmigrate","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/choonkeat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-21T07:41:35.000Z","updated_at":"2024-04-18T05:44:02.000Z","dependencies_parsed_at":"2024-06-20T17:28:50.805Z","dependency_job_id":null,"html_url":"https://github.com/choonkeat/dbmigrate","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/choonkeat/dbmigrate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/choonkeat%2Fdbmigrate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/choonkeat%2Fdbmigrate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/choonkeat%2Fdbmigrate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/choonkeat%2Fdbmigrate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/choonkeat","download_url":"https://codeload.github.com/choonkeat/dbmigrate/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/choonkeat%2Fdbmigrate/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270961770,"owners_count":24675915,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-18T02:00:08.743Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","golang","mariadb","migration-tool","mysql","postgresql","sql"],"created_at":"2024-12-18T04:20:54.725Z","updated_at":"2025-08-18T07:32:25.649Z","avatar_url":"https://github.com/choonkeat.png","language":"Go","readme":"# dbmigrate [![Build Status](https://travis-ci.com/choonkeat/dbmigrate.svg?branch=master)](https://travis-ci.com/choonkeat/dbmigrate)\n\n`rails migrate` [inspired](https://blog.choonkeat.com/weblog/2019/05/database-schema-migration.html) approach to database schema migrations but with plain sql files. and much faster.\n\n## Getting started with docker-compose\n\n### Given a working app\n\nLet's say we have a simple docker-compose project setup with only a `docker-compose.yml` file\n\n```\n$ tree .\n.\n└── docker-compose.yml\n\n0 directories, 1 file\n```\n\nThat declares a postgres database (`mydb`) and an app (`myapp`) that uses the database somehow (in our contrived example, we're just listing the tables in our database with `\\dt`)\n\n\n``` yaml\nversion: '3'\nservices:\n    mydb:\n        image: \"postgres\"\n        environment:\n            - POSTGRES_DB=myapp_development\n            - POSTGRES_USER=myuser\n            - POSTGRES_PASSWORD=topsecret\n\n    myapp:\n        image: \"postgres\"\n        command: \u003e-\n            sh -c '\n                until pg_isready --host=mydb --port=5432 --timeout=30; do sleep 1; done;\n                psql postgres://myuser:topsecret@mydb:5432/myapp_development -c \\\\dt\n            '\n        depends_on:\n            - mydb\n```\n\nLet's see how our app runs\n\n```\n$ docker-compose up myapp\nCreating dbmigrate-example_mydb_1 ... done\nCreating dbmigrate-example_myapp_1 ... done\nAttaching to dbmigrate-example_myapp_1\nmyapp_1  | mydb:5432 - no response\nmyapp_1  | mydb:5432 - no response\nmyapp_1  | mydb:5432 - accepting connections\nmyapp_1  | Did not find any relations.\ndbmigrate-example_myapp_1 exited with code 0\n```\n\nThe output simply shows that\n- our `myapp` waited until `mydb:5432` was ready to accept connections\n- then it listed the tables inside the database and got `Did not find any relations` (which means no tables)\n\n### Add `dbmigrate`\n\nNow let's add `dbmigrate` to our docker-compose.yml (take special note that `myapp` has a new entry under `depends_on` also)\n\n``` yaml\nversion: '3'\nservices:\n    mydb:\n        image: \"postgres\"\n        environment:\n            - POSTGRES_DB=myapp_development\n            - POSTGRES_USER=myuser\n            - POSTGRES_PASSWORD=topsecret\n\n    myapp:\n        image: \"postgres\"\n        command: \u003e-\n            sh -c '\n                until pg_isready --host=mydb --port=5432 --timeout=30; do sleep 1; done;\n                psql postgres://myuser:topsecret@mydb:5432/myapp_development -c \\\\dt\n            '\n        depends_on:\n            - mydb\n            - dbmigrate\n\n    # by default apply migrations with `-up` flag\n    # we can run different commands by overwriting `DBMIGRATE_CMD` env\n    # try DBMIGRATE_CMD=\"-h\" to see what other flags dbmigrate can offer\n    dbmigrate:\n        image: \"choonkeat/dbmigrate\"\n        environment:\n            - DATABASE_URL=postgres://myuser:topsecret@mydb:5432/myapp_development?sslmode=disable\n        volumes:\n            - .:/app\n        working_dir: /app\n        command: ${DBMIGRATE_CMD:--up -server-ready 60s -create-db}\n        depends_on:\n            - mydb\n```\n\nAnd we can start creating a few migration scripts by running `docker-compose up dbmigrate` with a custom `DBMIGRATE_CMD` env variable\n\n```\n$ DBMIGRATE_CMD=\"-create users\" docker-compose up dbmigrate\ndbmigrate-example_mydb_1 is up-to-date\nCreating dbmigrate-example_dbmigrate_1 ... done\nAttaching to dbmigrate-example_dbmigrate_1\ndbmigrate_1  | 2019/06/01 08:58:05 writing db/migrations/20190601085805_users.up.sql\ndbmigrate_1  | 2019/06/01 08:58:05 writing db/migrations/20190601085805_users.down.sql\ndbmigrate-example_dbmigrate_1 exited with code 0\n```\n\n```\n$ DBMIGRATE_CMD=\"-create blogs\" docker-compose up dbmigrate\ndbmigrate-example_mydb_1 is up-to-date\nRecreating dbmigrate-example_dbmigrate_1 ... done\nAttaching to dbmigrate-example_dbmigrate_1\ndbmigrate_1  | 2019/06/01 08:58:14 writing db/migrations/20190601085814_blogs.up.sql\ndbmigrate_1  | 2019/06/01 08:58:14 writing db/migrations/20190601085814_blogs.down.sql\ndbmigrate-example_dbmigrate_1 exited with code 0\n```\n\nAfter running those 2 commands, we see that we've generated 2 pairs of _empty_ `*.up.sql` and `*.down.sql` files.\n\n```\n$ tree .\n.\n├── db\n│   └── migrations\n│       ├── 20190601085805_users.down.sql\n│       ├── 20190601085805_users.up.sql\n│       ├── 20190601085814_blogs.down.sql\n│       └── 20190601085814_blogs.up.sql\n└── docker-compose.yml\n\n2 directories, 5 files\n```\n\nWe add our SQLs into our respective files\n\n``` sql\n-- db/migrations/20190601085805_users.up.sql\nCREATE TABLE IF NOT EXISTS users (\n    \"id\" SERIAL primary key\n);\n```\n\n``` sql\n-- db/migrations/20190601085805_users.down.sql\nDROP TABLE IF EXISTS users;\n```\n\n``` sql\n-- db/migrations/20190601085814_blogs.up.sql\nCREATE TABLE IF NOT EXISTS blogs (\n    \"id\" SERIAL primary key\n);\n```\n\n``` sql\n-- db/migrations/20190601085814_blogs.down.sql\nDROP TABLE IF EXISTS blogs;\n```\n\n### Database schema migrations is now managed\n\nLet's see how our app runs after those changes\n\n```\n$ docker-compose up myapp\ndbmigrate-example_mydb_1 is up-to-date\nRecreating dbmigrate-example_dbmigrate_1 ... done\nRecreating dbmigrate-example_myapp_1     ... done\nAttaching to dbmigrate-example_myapp_1\nmyapp_1      | mydb:5432 - accepting connections\nmyapp_1      |               List of relations\nmyapp_1      |  Schema |        Name        | Type  | Owner  \nmyapp_1      | --------+--------------------+-------+--------\nmyapp_1      |  public | blogs              | table | myuser\nmyapp_1      |  public | dbmigrate_versions | table | myuser\nmyapp_1      |  public | users              | table | myuser\nmyapp_1      | (3 rows)\nmyapp_1      |\ndbmigrate-example_myapp_1 exited with code 0\n```\n\nHey, looks like we have 3 tables now:\n\n1. `blogs` created by our `db/migrations/20190601085814_blogs.up.sql`\n1. `users` created by our `db/migrations/20190601085805_users.up.sql`\n1. `dbmigrate_versions` created by `dbmigrate` for itself to track migration history.\n    - every time `dbmigrate` runs, it checks `dbmigrate_versions` table to know which files in `db/migrations` had been applied and skip them; which files have not been seen before and apply them\n\nWe can look at the logs of the `dbmigrate` container to see what had happened when `myapp` booted up just now\n\n```\n$ docker-compose logs dbmigrate\nAttaching to dbmigrate-example_dbmigrate_1\ndbmigrate_1  | 2019/06/01 08:59:32 [up] 20190601085805_users.up.sql\ndbmigrate_1  | 2019/06/01 08:59:32 [up] 20190601085814_blogs.up.sql\n```\n\n### That's it\n\nNow everytime `myapp` starts up, since it declares `depends_on: dbmigrate`, our `dbmigrate` container will be run automatically to apply any new migration files in `db/migrations`. To add new migration files there, just run `docker-compose up dbmigrate` with a custom `DBMIGRATE_CMD` env variable (see above)\n\n---\n\n## Basic operations\n\n### Create a new migration\n\n```\n$ dbmigrate -create describe your change\n2018/12/21 16:33:13 writing db/migrations/20181221083313_describe-your-change.up.sql\n2018/12/21 16:33:13 writing db/migrations/20181221083313_describe-your-change.down.sql\n```\n\ngenerate a pair of blank `.up.sql` and `.down.sql` files inside the directory `db/migrations`. configure the directory with `-dir` command line flag.\n\nthe numeric prefix of the filename is the `version`. i.e. the version of the file above is `20181221083313`\n\n### Migrate up\n\n```\n$ dbmigrate -up\n2018/12/21 16:37:40 [up] 20181221083313_describe-your-change.up.sql\n2018/12/21 16:37:40 [up] 20181221083727_more-changes.up.sql\n```\n\n1. Connect to database (defaults to the value of `DATABASE_URL` env; configure with `-url`)\n1. Start a db transaction\n1. Pick up each `.up.sql` file in `db/migrations` and iterate through them in ascending order\n    - if the file `version` is found in `dbmigrate_versions` table, skip it\n    - otherwise, execute the sql statements in the file\n    - if it succeeds, insert an entry into `dbmigrate_versions` table\n      ``` sql\n      CREATE TABLE dbmigrate_versions (\n        version text NOT NULL PRIMARY KEY\n      );\n      ```\n    - if fail, rollback the entire transaction and exit 1\n1. Commit db transaction and exit 0\n\n### Migrate down\n\n```\n$ dbmigrate -down 1\n2018/12/21 16:42:24 [down] 20181221083727_more-changes.down.sql\n```\n\n1. Connect to database (defaults to the value of `DATABASE_URL` env; configure with `-url`)\n1. Start a db transaction\n1. Pick up each `.down.sql` file in `db/migrations` and iterate through them in descending order\n    - if the file `version` is NOT found in `dbmigrate_versions` table, skip it\n    - otherwise, execute the sql statements in the file\n    - if succeeds, remove the entry `WHERE version = ?` from `dbmigrate_versions` table\n    - if fail, rollback the entire transaction and exit 1\n1. Commit db transaction and exit 0\n\nYou can migrate \"down\" more files by using a different number\n\n```\n$ dbmigrate -down 3\n2018/12/21 16:46:45 [down] 20181221083313_describe-your-change.down.sql\n2018/12/21 16:46:45 [down] 20181221055307_create-users.down.sql\n2018/12/21 16:46:45 [down] 20181221055304_create-projects.down.sql\n```\n\n### Show versions pending\n\nPrints a sorted list of versions found in `-dir` but does not have a record in `dbmigrate_versions` table.\n\n```\n$ dbmigrate -versions-pending\n20181222073750\n20181222073900\n20181222073901\n```\n\n### Configuring `DATABASE_URL`\n\n**PostgreSQL**\n\nWe're using [github.com/lib/pq](https://godoc.org/github.com/lib/pq) so environment variable look like this\n\n```\nDATABASE_URL=postgres://user:password@localhost:5432/dbmigrate_test?sslmode=disable\n```\n\nor\n\n```\nDATABASE_DRIVER=postgres\nDATABASE_URL='user=pqgotest dbname=pqgotest sslmode=verify-full'\n```\n\n\u003e NOTE: out of the box, this driver supports having multiple statements in one `.sql` file.\n\n**MySQL**\n\nWe're using [github.com/go-sql-driver/mysql](https://github.com/go-sql-driver/mysql#examples) so environment variables look like\n\n```\nDATABASE_DRIVER=mysql\nDATABASE_URL='user:password@tcp(127.0.0.1:3306)/dbmigrate_test'\n```\n\n\u003e NOTE: to have multiple SQL statements in each `.sql` file, you'd need to add `multiStatements=true` to the CGI query string of your `DATABASE_URL`. i.e.\n\u003e\n\u003e ```\n\u003e DATABASE_URL='user:password@tcp(127.0.0.1:3306)/dbmigrate_test?multiStatements=true'\n\u003e ```\n\u003e\n\u003e See the [driver documentation](https://github.com/go-sql-driver/mysql#multistatements) for details and other available options.\n\n## Handling failure\n\nWhen there's an error, we rollback the entire transaction. So you can edit your faulty `.sql` file and simply re-run\n\n```\n$ dbmigrate -up\n2018/12/21 16:55:41 20181221083313_describe-your-change.up.sql: pq: relation \"users\" already exists\nexit status 1\n$ vim db/migrations/20181221083313_describe-your-change.up.sql\n$ dbmigrate -up\n2018/12/21 16:56:05 [up] 20181221083313_describe-your-change.up.sql\n2018/12/21 16:56:05 [up] 20181221083727_more-changes.up.sql\n$ dbmigrate -up\n$\n```\n\n**PostgreSQL** supports rollback for most data definition language (DDL)\n\n\u003e one of the more advanced features of PostgreSQL is its ability to perform transactional DDL via its Write-Ahead Log design. This design supports backing out even large changes to DDL, such as table creation. You can't recover from an add/drop on a database or tablespace, but all other catalog operations are reversible.\n\u003e https://wiki.postgresql.org/wiki/Transactional_DDL_in_PostgreSQL:_A_Competitive_Analysis\n\n**MySQL** does not support rollback for DDL\n\n\u003e Some statements cannot be rolled back. In general, these include data definition language (DDL) statements, such as those that create or drop databases, those that create, drop, or alter tables or stored routines.\n\u003e https://dev.mysql.com/doc/refman/8.0/en/cannot-roll-back.html\n\nIf you're using MySQL, make sure to have DDL (e.g. `CREATE TABLE ...`) in their individual `*.sql` files.\n\n### Caveat: `-create-db` and database names\n\nThe SQL command `CREATE DATABASE \u003cdbname\u003e` does not work well (at least in postgres) if `\u003cdbname\u003e` contains dashes `-`. The proper way would've been to [quote](https://godoc.org/github.com/lib/pq#QuoteIdentifier) the value [when using it](https://github.com/choonkeat/dbmigrate/blob/5397b58246f8dfbfaf97897520eb8a9fdc5f129f/cmd/dbmigrate/main.go#L101) but alas there doesn't seem to be a driver agnostic way to quote that string [in Go](https://godoc.org/database/sql).\n\nThe workaround is, if `-create-db` is needed, use underscore `_` for your dbname instead of dashes `-`\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchoonkeat%2Fdbmigrate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchoonkeat%2Fdbmigrate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchoonkeat%2Fdbmigrate/lists"}