{"id":15623724,"url":"https://github.com/yogthos/migratus","last_synced_at":"2025-05-13T22:08:47.676Z","repository":{"id":1970639,"uuid":"2901792","full_name":"yogthos/migratus","owner":"yogthos","description":"MIGRATE ALL THE THINGS!","archived":false,"fork":false,"pushed_at":"2025-01-05T22:58:37.000Z","size":652,"stargazers_count":665,"open_issues_count":23,"forks_count":95,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-04-30T15:18:11.711Z","etag":null,"topics":["clojure-library","migrations"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yogthos.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"yogthos"}},"created_at":"2011-12-02T21:58:26.000Z","updated_at":"2025-04-17T05:37:20.000Z","dependencies_parsed_at":"2024-01-05T21:51:34.255Z","dependency_job_id":"370973a6-2c28-4816-990a-d6600a993720","html_url":"https://github.com/yogthos/migratus","commit_stats":{"total_commits":353,"total_committers":73,"mean_commits":4.835616438356165,"dds":0.4560906515580736,"last_synced_commit":"8a98f3db7016d7dfa269eb32abac4468ce60b6f1"},"previous_names":[],"tags_count":113,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yogthos%2Fmigratus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yogthos%2Fmigratus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yogthos%2Fmigratus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yogthos%2Fmigratus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yogthos","download_url":"https://codeload.github.com/yogthos/migratus/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254036829,"owners_count":22003654,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure-library","migrations"],"created_at":"2024-10-03T09:58:33.984Z","updated_at":"2025-05-13T22:08:42.638Z","avatar_url":"https://github.com/yogthos.png","language":"Clojure","funding_links":["https://github.com/sponsors/yogthos"],"categories":["Structural Migrations","Clojure"],"sub_categories":[],"readme":"# Migratus\n\n[![CircleCI](https://dl.circleci.com/status-badge/img/gh/yogthos/migratus/tree/master.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/yogthos/migratus/tree/master)\n\n![MIGRATE ALL THE THINGS!](https://cdn.rawgit.com/yogthos/migratus/master/migrate.png)\n\nA general migration framework, with implementations for migrations as SQL\nscripts or general Clojure code.\n\nDesigned to be compatible with a git based work flow where multiple topic\nbranches may exist simultaneously, and be merged into a master branch in\nunpredictable order.\n\nThis is accomplished two ways:\n\n1. Migration ids are not assumed to be incremented integers.  It is recommended that they be timestamps (e.g. '20111202091200').\n2. Migrations are considered for completion independently.\n\nUsing a 14 digit timestamp will accommodate migrations granularity to a second,\nreducing the chance of collisions for a distributed team.\n\nIn contrast, using a single global version for a store and incremented\nintegers for migration versions, it is possible for a higher numbered\nmigration to get merged to master and deployed before a lower numbered\nmigration, in which case the lower numbered migration would never get run,\nunless it is renumbered.\n\nMigratus does not use a single global version for a store. It considers each\nmigration independently, and runs all uncompleted migrations in sorted order.\n\n## Quick Start\n\n- add the Migratus dependency:\n\n[![Clojars Project](https://img.shields.io/clojars/v/migratus.svg)](https://clojars.org/migratus)\n[![Open Source Helpers](https://www.codetriage.com/yogthos/migratus/badges/users.svg)](https://www.codetriage.com/yogthos/migratus)\n\n- Add the following code to\n `resources/migrations/20111206154000-create-foo-table.up.sql`\n\n  `CREATE TABLE IF NOT EXISTS foo(id BIGINT);`\n\n- Add the following code to\n `resources/migrations/20111206154000-create-foo-table.down.sql`\n\n  `DROP TABLE IF EXISTS foo;`\n\n### Multiple Statements\n\nIf you would like to run multiple statements in your migration, then\nseparate them with `--;;`.  For example:\n\n```sql\nCREATE TABLE IF NOT EXISTS quux(id bigint, name varchar(255));\n--;;\nCREATE INDEX quux_name on quux(name);\n```\n\nThis is necessary because JDBC does not have a method that allows you to\nsend multiple SQL commands for execution.  Migratus will split your\ncommands, and attempt to execute them inside of a transaction.\n\nNote that some databases, such as MySQL, do not support transactional DDL\ncommands. If you're working with such a database then it will not be able\nto rollback all the DDL statements that were applied in case a statement\nfails.\n\n### Disabling transactions\n\nMigratus attempts to run migrations within a transaction by default.\nHowever, some databases do not support transactional DDL statements.\nTransactions can be disabled by adding the following line at the start\nof the migration file:\n\n```sql\n-- :disable-transaction\n```\n\n### Running Functions in Migrations\n\nFunctions inside migrations may need to be additionally wrapped, a PostgreSQL example would look as follows:\n\n```sql\nDO $func$\n BEGIN\n PERFORM schema_name.function_name('foo', 10);\nEND;$func$;\n```\n\n### Supporting `use` statements\n\nTo run migrations against several different databases (in MySQL, or \"schemas\" in Postgres, etc.), with embedded `use` statements in your migrations, specify the database in your migration-table-name in the connections, i.e. `database_name.table_name` not `table_name`.\n\n### Property substitution\n\nMigratus supports property substitution where migration files can contain placeholders with the format of `${property.name}`, these placeholders will be replaced with values found in the environment as a result of calling `(System/getenv)`.\n\nShell variables will be normalized into Java properties style by being lower cased and with `_` being transformed into `.`, e.g:\n\n```\nFOO_BAR - \u003e foo.bar\n```\n\nThis feature is enabled when the `:properties` flag is set in the configuration.\n\nMigratus will look for the following default properties:\n\n* `migratus.schema`\n* `migratus.user`\n* `migratus.database`\n* `migratus.timestamp` (defaults to the value of `(.getTime (java.util.Date.))`)\n\nAdditional property can be specified using the `:env` key or by providing a map of custom properties using the `:env` key:\n\n```clojure\n{:store :database\n :properties {:env [\"database.table\"]\n              :map {:database {:user \"bob\"}}}\n :db {:dbtype   \"h2\"\n      :dbname   \"site.db\"}}\n```\n\nFor example, given the following template:\n\n```sql\nGRANT SELECT,INSERT ON ${database.table} TO ${database.user};\n```\n\nThe environment variable associated with the `database.table` key will replace `${database.table}` tag in the template, while `{:database {:user \"bob\"}}` will replace `${database.user}` with `\"bob\"`.\n\n### Setup\n\n- Add Migratus as a dependency to your `project.clj`\n```clojure\n:dependencies [[migratus \u003cVERSION\u003e]]\n```\n\nThere are hidden dependencies on slf4j inside migratus, so to avoid errors or silent failures you'll need to also add\n\n```clojure\n[org.slf4j/slf4j-log4j12 \u003cVERSION\u003e]\n```\n\nor if you're using Timbre\n\n```clojure\n[com.fzakaria/slf4j-timbre \u003cVERSION\u003e]\n```\n\nNext, create a namespace to manage the migrations:\n\n```clojure\n(ns my-migrations\n (:require [migratus.core :as migratus]))\n\n(def config {:store                :database\n             :migration-dir        \"migrations/\"\n             :init-script          \"init.sql\" ;script should be located in the :migration-dir path\n             ;defaults to true, some databases do not support\n             ;schema initialization in a transaction\n             :init-in-transaction? false\n             :migration-table-name \"foo_bar\"\n             :db {:dbtype \"h2\"\n                  :dbname \"site.db\"}})\n\n;initialize the database using the 'init.sql' script\n(migratus/init config)\n\n;apply pending migrations\n(migratus/migrate config)\n\n;rollback the migration with the latest timestamp\n(migratus/rollback config)\n\n;bring up migrations matching the ids\n(migratus/up config 20111206154000)\n\n;bring down migrations matching the ids\n(migratus/down config 20111206154000)\n```\n\n#### Alternative setup\n\nIt is possible to pass a `java.sql.Connection` or `javax.sql.DataSource` in place of a db spec map, e.g:\n\n```clojure\n(ns my-migrations\n  (:require [next.jdbc :as jdbc]))\n\n(def connection (jdbc/get-connection\n                  {:dbtype \"h2\"\n                   :dbname \"site.db\"}))\n\n(def config {:db {:connection connection}})\n\n;; With next.jdbc \u003e= 1.2.790 you can use {:connection-uri ...} format (as well as raw {:datasource ...} without :user/:password).\n(def config {:db {:connection-uri ...}})\n\n;; Migratus will close the connection by default\n;; providing :managed-connection? hint allows managing the state of the connection externally\n;; in case you wish to reuse the connection for other purposes\n(def config {:connection conn :managed-connection? true})\n\n```\n\n```clojure\n(ns my-migrations\n  (:require [hikari-cp.core :as hk]))\n;; Hikari: https://github.com/tomekw/hikari-cp\n\n(def datasource-options {:adapter \"h2\"\n                         :url     \"jdbc:h2:./site.db\"})\n\n(def config {:db {:datasource (hk/make-datasource datasource-options)}})\n```\n\n#### Running as native image (Postgres only)\n\n[PGMig](https://github.com/leafclick/pgmig) is a standalone tool built with migratus that's compiled as a standalone GraalVM native image executable.\n\n### Generate migration files\n\nMigratus also provides a convenience function for creating migration files:\n\n```clojure\n(migratus/create config \"create-user\")\n;; minimal config needed to call create while specifying the destination path\n(migratus/create {:migration-dir \"migrations\"} \"create-user\")\n```\n\nThis will result with up/down migration files being created prefixed with the current timestamp, e.g:\n\n```\n20150701134958-create-user.up.sql\n20150701134958-create-user.down.sql\n```\n\n## Code-based Migrations\n\nApplication developers often encounter situations where migrations cannot be easily expressed as a SQL script. For instance:\n\n   - Executing programmatically-generated DDL statements\n     (e.g. updating the schema of a dynamically-sharded table).\n   - Transferring data between database servers.\n   - Backfilling existing records with information that must be\n     retrieved from an external system.\n\nA common approach in these scenarios is to write one-off scripts which an admin must manually apply for each instance of the application, but issues arise if a script is not run or run multiple times.\n\nMigratus addresses this problem by providing support for code-based migrations. You can write a migration as a Clojure function, and Migratus will ensure that it's run exactly once for each instance of the application.\n\n### Defining a code-based migration\n\nCreate a code-based migration by adding a `.edn` file to your migrations directory that contains the namespace and up/down functions to run, e.g. `resources/migrations/20170331141500-import-users.edn`:\n\n```clojure\n{:ns app.migrations.import-users\n :up-fn migrate-up\n :down-fn migrate-down\n :transaction? true}\n```\n\nMigrations will run within a transaction by default, set `transaction?` to `false` to disable transactions.\n\nThen, in `src/app/migrations/import_users.clj`:\n\n```clojure\n(ns app.migrations.import-users)\n\n(defn migrate-up [config]\n   ;; do stuff here\n   )\n\n(defn migrate-down [config]\n   ;; maybe undo stuff here\n   )\n```\n\n- The up and down migration functions should both accept a single\n  parameter, which is the config map passed to Migratus (so your\n  migrations can be configurable).\n- You can omit the up or down migration by setting `:up-fn` or\n  `down-fn` to `nil` in the EDN file.\n- The `:up-fn` and `:down-fn` entries can optionally be a vector containing the\n  migration function followed by additional args to be passed after\n  the config map, e.g. `{..., :up-fn [migrate-up \"arg1\" :arg2], ...}`.\n\n### Generate code-based migration files\n\nThe `migratus.core/create` function accepts an optional type parameter, which you can pass as `:edn` to create a new migration file.\n\n```clojure\n(migratus/create config \"import-users\" :edn)\n```\n\n### Mixing SQL and code-based migrations\n\nYou can include both SQL and code-based migrations in the same migrations directory, in which case they will be run intermixed in the order defined by their timestamps and their status stored in the same table in the migrations database. This way if there are dependencies between your SQL and code-based migrations, you can be assured that they'll run in the correct order.\n\n## Quick Start with Leiningen\n\nMigratus provides a Leiningen plugin:\n\n   - Add migratus-lein as a plugin in addition to the Migratus dependency:\n\n[![Clojars Project](https://img.shields.io/clojars/v/migratus-lein.svg)](https://clojars.org/migratus-lein)\n\n   - Add the following key and value to your project.clj:\n\n```clojure\n:migratus {:store :database\n           :migration-dir \"migrations\"\n           :db {:dbtype \"mysql\"\n                :dbname \"//localhost/migratus\"\n                :user \"root\"\n                :password \"\"}}\n```\n\nTo apply pending migrations:\n\n   - Run `lein migratus migrate`\n\nTo rollback the migration with the last creation timestamp:\n\n   - Run `lein migratus rollback`\n\nThen follow the rest of the above instructions.\n\n## Configuration\n\nMigratus is configured via a configuration map that you pass in as its first parameter. The `:store` key describes the type of store against which migrations should be run.  All other keys/values in the configuration map are store specific.\n\n### Databases\n\nTo run migrations against a database use a :store of :database, and specify the database connection configuration in the :db key of the configuration map.\n\n* `:migration-dir` - directory where migration files are found\n* `:exclude-scripts` - a collection of script name globs that will be excluded from migrations\n* `:db` - One of `java.sql.Connection` or `javax.sql.DataSource` instance or a `next.jdbc` database spec. See next.jdbc docs for the version you are using: https://cljdoc.org/d/com.github.seancorfield/next.jdbc/1.2.780/api/next.jdbc#get-datasource\n* `:command-separator` - the separator will be used to split the commands within each transaction when specified\n* `:expect-results?` - allows comparing migration query results using the `-- expect n` comment\n* `:tx-handles-ddl?` -  skips the automatic down that occurs on exception\n* `:init-script` -  string pointing to a script that should be run when the database is initialized\n* `:init-in-transaction?` - defaults to true, but some databases do not support schema initialization in a transaction\n* `:migration-table-name` - string specifying a custom name for the migration table, defaults to `schema_migrations`\n\n#### example configurations\n\n```clojure\n{:store :database\n :migration-dir \"migrations\"\n :exclude-scripts [\"*.clj\"]\n :db {:dbtype \"mysql\"\n      :dbname \"migratus\"\n      :user \"root\"\n      :password \"\"}}\n```\n\nThe `:migration-dir` key specifies the directory on the classpath in which to find SQL migration files. Each file should be named with the following pattern `[id]-[name].[direction].sql` where id is a unique integer `id` (ideally it should be a timestamp) for the migration, name is some human readable description of the migration, and direction is either `up` or `down`.\n\nWhen the `expect-results?` key is set in the config, an assertion can be added to the migrations to check that the expected number of rows was updated:\n\n```sql\n-- expect 17;;\nupdate foobar set thing = 'c' where thing = 'a';\n\n--;;\n\n-- expect 1;;\ndelete from foobar where thing = 'c';\n```\n\nIf Migratus is trying to run either the up or down migration and it does not exist, then an Exception will be thrown.\n\nSee test/migrations in this repository for an example of how database migrations work.\n\n### Modify sql fn\n\nIf you want to do some processing of the sql before it gets executed, you can provide a `:modify-sql-fn` in the config data structure to do so.\nIt expects a sql-string and can return either a modified sql-string or a sequence of sql-strings.\nThis is intended for use with http://2ndquadrant.com/en/resources/pglogical/ and similar systems, where DDL statements need to be executed via an extension-provided function.\n\n## Usage\n   Migratus can be used programmatically by calling one of the following functions:\n\n   | Function                                | Description              |\n   |-----------------------------------------|--------------------------|\n   | `migratus.core/init`                      | Runs a script to initialize the database, e.g: create a new schema.                                                                                |\n   | `migratus.core/create`                    | Create a new migration with the current date.                                                                                                      |\n   | `migratus.core/migrate`                   | Run `up` for any migrations that have not been run. Returns `nil` if successful, `:ignore` if the table is reserved. Supports thread cancellation. |\n   | `migratus.core/rollback`                  | Run `down` for the last migration that was run.                                                                                                    |\n   | `migratus.core/rollback-until-just-after` | Run `down` all migrations after `migration-id`. This only considers completed migrations, and will not migrate up.                                 |\n   | `migratus.core/up`                        | Run `up` for the specified migration ids. Will skip any migration that is already up.                                                              |\n   | `migratus.core/down`                      | Run `down` for the specified migration ids. Will skip any migration that is already down. \n   | `migratus.core/reset`                 | Reset the database by down-ing all migrations successfully applied, then up-ing all migrations.\n   | `migratus.core/pending-list`              | Returns a list of pending migrations.                                                                                                              |\n   | `migratus.core/migrate-until-just-before` | Run `up` for for any pending migrations which precede the given migration id (good for testing migrations).                                        |\n   | `migratus.core/squashing-list`            | Takes from-id and to-id as inputs (both inclusive) and returns the list of migrations to be squashed. \n   | `migratus.core/create-squash`             | Reads files within the specified id range and generates a new squashed migration file. It removes the original migration files and creates a new one with the last ID from the range and a user-provided name.\n   | `migratus.core/squash-between`           | Updates the migration table by marking the old IDs as squashed and replacing them with the new ID. No actual migration is applied, assuming they were previously applied.\n\nSee the docstrings of each function for more details.\n\nMigratus can also be used from leiningen if you add it as a plugin dependency.\n\n```clojure\n:plugins [[migratus-lein \u003cVERSION\u003e]]\n```\n\nAnd add a configuration :migratus key to your `project.clj`.\n\n```clojure\n:migratus {:store :database\n           :migration-dir \"migrations\"\n           :db {:dbtype \"mysql\"\n                :dbname \"migratus\"\n                :user \"root\"\n                :password \"\"}}\n```\n\n   You can then run the following tasks:\n\n   | Task                        | Description                                                                                |\n   |-----------------------------|--------------------------------------------------------------------------------------------|\n   | lein migratus create \u003cname\u003e | Create a new migration with the current date.                                              |\n   | lein migratus migrate       | Run 'up' for any migrations that have not been run.                                        |\n   | lein migratus rollback      | Run 'down' for the last migration that was run.                                            |\n   | lein migratus up \u0026 ids      | Run 'up' for the specified migration ids. Will skip any migration that is already up.      |\n   | lein migratus down \u0026 ids    | Run 'down' for the specified migration ids. Will skip any migration that is already down.  |\n   | lein migratus reset         | Run 'down' for all migrations that have been run, and 'up' for all migrations.             |\n   | lein migratus pending       | Run 'pending-list' to get all pending migrations.                                          |\n\n## Quickstart with native Clojure projects\n\nSee [clj-migratus](https://github.com/paulbutcher/clj-migratus) for more information.\n\n## Usage\n\nAdd the following to your `deps.edn`:\n\n```\n:aliases {:migrate {:extra-deps {com.github.paulbutcher/clj-migratus {:git/tag \"v1.0.0\"\n                                                                      :git/sha \"67d0fe5\"}}\n                     :main-opts [\"-m\" \"clj-migratus\"]}}\n```\n\nCreate a [Migratus configuration](https://github.com/yogthos/migratus#configuration) file. This can either be `migratus.edn`:\n\n```\n{:store :database\n :migration-dir \"migrations\"\n :db {:dbtype \"mysql\"\n      :dbname \"migratus\"\n      :user \"root\"\n      :password \"\"}}\n```\n\nOr (recommended) `migratus.clj`, allowing credentials to be taken from the environment:\n\n```\n{:store :database\n :db {:jdbcUrl (get (System/getenv) \"JDBC_DATABASE_URL\")}}\n```\n\nThen run, for example:\n\n```\n$ clj -M:migrate init\n$ clj -M:migrate migrate\n$ clj -M:migrate create create-user-table\n```\n\nSee [Migratus Usage](https://github.com/yogthos/migratus#usage) for documentation on each command.\n\n## Working on migratus itself\n\nYou can use either `lein` or `clj` for now as it has both project definitions.\n\nRun tests with kaocha:\n\n```\n   #  https://cljdoc.org/d/lambdaisland/kaocha/1.68.1059/doc/4-running-kaocha-cli\n\n   bin/kaocha --test-help\n\n   bin/kaocha --fail-fast\n\n   bin/kaocha --fail-fast --focus migratus.test.migration.sql/test-run-sql-migrations\n\n   # Run only integration tests - defined in tests.edn\n   bin/kaocha testcontainers\n```\n\n## License\n\nCopyright © 2016 Paul Stadig, Dmitri Sotnikov\n\nLicensed under the Apache License, Version 2.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyogthos%2Fmigratus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyogthos%2Fmigratus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyogthos%2Fmigratus/lists"}