{"id":14063376,"url":"https://github.com/michelp/metagration","last_synced_at":"2025-10-27T05:31:34.266Z","repository":{"id":145541017,"uuid":"263514807","full_name":"michelp/metagration","owner":"michelp","description":"Metagration: PostgreSQL Migrator in PostgreSQL","archived":false,"fork":false,"pushed_at":"2022-02-12T02:13:23.000Z","size":181,"stargazers_count":100,"open_issues_count":0,"forks_count":4,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-02-01T02:41:26.089Z","etag":null,"topics":["migration","plpgsql","postgresql"],"latest_commit_sha":null,"homepage":"","language":"PLpgSQL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michelp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-05-13T03:24:25.000Z","updated_at":"2024-10-27T22:36:43.000Z","dependencies_parsed_at":"2023-04-08T10:17:03.878Z","dependency_job_id":null,"html_url":"https://github.com/michelp/metagration","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michelp%2Fmetagration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michelp%2Fmetagration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michelp%2Fmetagration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michelp%2Fmetagration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michelp","download_url":"https://codeload.github.com/michelp/metagration/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238445843,"owners_count":19473822,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["migration","plpgsql","postgresql"],"created_at":"2024-08-13T07:03:18.560Z","updated_at":"2025-10-27T05:31:33.931Z","avatar_url":"https://github.com/michelp.png","language":"PLpgSQL","funding_links":[],"categories":["PLpgSQL"],"sub_categories":[],"readme":"[![Tests](https://github.com/michelp/metagration/actions/workflows/test.yml/badge.svg)](https://github.com/michelp/metagration/actions/workflows/test.yml)\n\n\u003cbr /\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"metagration.png\" /\u003e\n\u003c/p\u003e\n\n# Metagration: Logical PostgreSQL Migration\n\nMigrating logically replicated PostgreSQL databases is a delicate\ndance of applying the right script at the right time, and enduring\npossible downtime making sure replicas are correctly up to date.\nConsider the [warnings from the\ndocumentation](https://www.postgresql.org/docs/current/logical-replication-restrictions.html):\n\n  - The database schema and DDL commands are not replicated. The\n    initial schema can be copied by hand using pg_dump\n    --schema-only. Subsequent schema changes would need to be kept in\n    sync manually.\n    \nMetagration is a PostgreSQL migration tool written in PostgreSQL.\nMetagration \"up/down\" scripts are stored procedures and applied\nin-database by the database.  Creating and managing metagrations and\nactually running them are *completely decoupled*.\n\nMetagrations can be managed and replicated like any other data in your\ndatabase using whatever favorite tool you are already familar with.\nUsing tools like [pglogical](https://github.com/2ndQuadrant/pglogical)\nyou can then apply metagrations across logically replicated cluster at\nthe exact same point in time in the WAL stream.  Metagration keeps\ntrack of [restore\npoints](https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-ADMIN-BACKUP)\nbefore all changes so entire clusters can be\n[Point-In-Time-Recovered](https://www.postgresql.org/docs/14/continuous-archiving.html)\nto the same point in the transcation log, avoiding migration induced\nconflict errors.\n\n**Metagration has support for 100% of PostgreSQL's features, because it\n*is* PostgreSQL**:\n\n  - Up/Down scripts are stored procedures in any pl language.\n\n  - No external tools, any PostgreSQL client can manage metagrations.\n\n  - Cloud-friendly single ~470 line SQL file for any PostgreSQL \u003e= 11.\n\n  - One simple function for new SQL scripts.\n\n  - Procedures can be transactional, and transaction aware.\n\n  - Generates Point In Time Recovery restore points before migration.\n\n  - Metagrations can export/import to/from SQL files.\n\n  - Metagrations are just rows so `pg_dump/pg_restore` them.\n\n  - Can use pgTAP for powerful migration verification.\n\n  - Postgres docker container entrypoint friendly.\n  \n## Intro\n\nMetagrations are DDL change scripts wrapped in PostgreSQL stored\nprocedures run in a specific order either \"up\" or \"down\".  A\nmetagration is a script defined entirely within the database, there is\nno external migration tool or language.\n\nMetagration `script`s are what move the database from one revision to\nthe next.  Each script has a forward \"up\" procedure, and optionally a\nbackward \"down\" procedure to undo the \"up\" operation.  Script\nprocedures can be written in *any* supported stored procedure\nlanguage.  Metagration strictly enforces the revision order of the\nscripts applied.\n\nMetagration comes with a simple create function for writing fast up\nand down scripts in plpgsql, which often look exactly like their SQL\ncounterparts:\n\n    # SELECT metagration.new_script(\n          'CREATE TABLE public.foo (id bigserial);',\n          'DROP TABLE public.foo;'\n          );\n     new_script\n    --------\n          1\n\nThis creates a new script with revision `1`.  The function\n`metagration.new_script(up[, down])` expands the up and down code into\ndynamically created plpgsql functions.  Once the script is created, it\ncan then be run with `metagration.run()`\n\n    # CALL metagration.run();\n    # \\dt\n            List of relations\n     Schema | Name | Type  |  Owner\n    --------+------+-------+----------\n     public | foo  | table | postgres\n\nNow add another script with an unfortunate table name to be reverted:\n\n    # SELECT metagration.new_script(\n        'CREATE TABLE public.bad (id bigserial);',\n        'DROP TABLE public.bad;\n        );\n     new_script\n    --------\n          2\n    # CALL metagration.run();\n    # \\dt\n            List of relations\n     Schema | Name | Type  |  Owner\n    --------+------+-------+----------\n     public | foo  | table | postgres\n     public | bad  | table | postgres\n\nNow revision `2` can be reverted by calling `metagration.run()` with a\nspecific target revision, in this case back to 1, and the `bad` table\ngets dropped:\n\n    postgres=# CALL metagration.run(1);\n    # \\dt\n            List of relations\n     Schema | Name | Type  |  Owner\n    --------+------+-------+----------\n     public | foo  | table | postgres\n\nThe current, previous, and next revisions can be queried:\n\n    # SELECT metagration.previous_revision();\n     previous_revision\n    ------------------\n                     0\n    # SELECT metagration.current_revision();\n     current_revision\n    -----------------\n                    1\n    # SELECT metagration.next_revision();\n     next_revision\n    --------------\n                 2\n\nMetagrations can also be run with a relative revision parameter passed\nas a text string:\n\n    CALL metagration.run('1');  -- go forward one revision\n    CALL metagration.run('-3');  -- go back three revisions\n\nA log of all migrations, their start and end revisions, times, and\nrestore points are stored in the `metagration.log` table:\n\n    # SELECT * FROM metagration.log ORDER BY migration_start;\n     revision_start | revision_end |        migration_start        |         migration_end         | txid |           restore_point           | restore_point_lsn\n    ----------------+--------------+-------------------------------+-------------------------------+------+-----------------------------------+-------------------\n                  0 |            1 | 2020-05-13 23:13:02.830335+00 | 2020-05-13 23:13:02.831964+00 |  505 | 0|1|2020-05-13|23:13:02.830335+00 | 0/183F408\n                  1 |            3 | 2020-05-13 23:13:02.841926+00 | 2020-05-13 23:13:02.8432+00   |  505 | 1|3|2020-05-13|23:13:02.841926+00 | 0/1841A20\n                  3 |            4 | 2020-05-13 23:13:02.846628+00 | 2020-05-13 23:13:02.847429+00 |  505 | 3|4|2020-05-13|23:13:02.846628+00 | 0/1844730\n                  4 |            1 | 2020-05-13 23:13:02.848043+00 | 2020-05-13 23:13:02.850642+00 |  505 | 4|1|2020-05-13|23:13:02.848043+00 | 0/18459C0\n                  1 |            4 | 2020-05-13 23:13:02.852157+00 | 2020-05-13 23:13:02.858205+00 |  505 | 1|4|2020-05-13|23:13:02.852157+00 | 0/1846790\n\nBefore each metagration a recovery restore point is created with\n[`pg_create_restore_point`](https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-ADMIN-BACKUP)\nand can be used for Point In Time Recovery to the point just before\nthe migration and other recovery tasks.  The current transaction id is\nalso saved.\n\n## Dynamic Metagrations\n\nMetagration scripts are stored procedures, and can be fully dynamic in\nterms of the SQL they execute when run.  To facilitate this, the\n`run()` function accepts an optional `args jsonb` argument that is\npassed to each script when run.  This allows scripts to respond to\ndynamic variables at run time.\n\nFor plpgsql scripts built with `new_script`, optional local variable\ndeclarations can also be provided, in the following example, the index\nvariable `i` in the `FOR` loops are declared in the `up_declare` and\n`down_declare` parameters to `new_script()` shown here:\n\n    SELECT new_script(\n    $up$\n        FOR i IN (SELECT * FROM generate_series(1, (args-\u003e\u003e'target')::bigint, 1)) LOOP\n            EXECUTE format('CREATE TABLE %I (id serial)', 'foo_' || i);\n        END LOOP;\n    $up$,\n    $down$\n        FOR i IN (SELECT * FROM generate_series(1, (args-\u003e\u003e'target')::bigint, 1)) LOOP\n            EXECUTE format('DROP TABLE %I', 'foo_' || i);\n        END LOOP;\n    $down$,\n        up_declare:='i bigint',\n        down_declare:='i bigint'\n        );\n        \nTo run, pass an integer value for the `target` jsonb key in `args`:\n\n    # CALL metagration.run(args:=jsonb_build_object('target', 3));\n    # \\dt+\n                          List of relations\n     Schema |  Name | Type  |  Owner   |  Size   | Description \n    --------+-------+-------+----------+---------+-------------\n     public | foo_1 | table | postgres | 0 bytes | \n     public | foo_2 | table | postgres | 0 bytes | \n     public | foo_3 | table | postgres | 0 bytes | \n     \nIf your up script depends on `args`, it's likely your down scripts do\ntoo.  Pass them as well to revert or, get the args used in the up\nmigration from the `migration.log` table where they are saved.\n\n    # CALL metagration.run('-1', args:=jsonb_build_object('target', 3));\n    # \\dt+\n                          List of relations\n     Schema |  Name   | Type  |  Owner   |  Size   | Description \n    --------+---------+-------+----------+---------+-------------\n\n    # SELECT migration_args FROM metagration.log \n      WHERE revision_end = metagration.current_revision() \n      ORDER BY migration_end DESC LIMIT 1;\n      \n     migration_args \n    ----------------\n     {\"target\": 3}\n    (1 row)\n\n## Import and Exporting\n\nThe obvious question is, if metagrations are stored procedures that\nmakes DDL changes, who CREATEs the metagrations?  They can be created\nprogramatically as shown above with `new_script` or by inserting\ndirectly into the `metagraiton.script` table. They can be imported and\nexported using any PostgreSQL client or admin tool.  Because\nmetagrations are in-database, they are dumped and restored when the\ndatabase is backed up.\n\nYou can still check your metagrations into source control and stream\nthem into a new database when you initialize it, then call\n`metagrate.run()`.\n\nSince this process of creating metagrations is *decoupled* from the\nactual act of migration, it can be done using any of the many database\nmanagement tools for PostgreSQL. Because metagration scripts are\nstored procedures, they are stateless database objects that can be\nexported, imported, dropped and re-created as necessary.\n\nA helpful tool for doing this `metagration.export()`.  The `export()`\nfunction will generate SQL script file that `CREATE OR REPLACE`s the\nmigration scripts and optionally clear and run them. Simply capture\nthe output of this function, for example with:\n\n    psql -A -t -U postgres -c 'select metagration.export()' \u003e export_file.sql\n\nAnd then check it in to your source control.  The scripts can then be\nimported with\n[`psql`](https://www.postgresql.org/docs/current/app-psql.html) or any\nother PostgreSQL client:\n\n    psql \u003c export_file.sql\n\nThis will import all the migrations but not *run* them, for that you\nstill call `metagration.run()` or pass `run_migrations:=true` as shown\nbelow.\n\nIf `metagration.export(replace_scripts:=true)` is called the generated\nscript will truncate the `script` and `log` tables and re-insert all\nthe exported scripts.\n\nIf `metagration.export(transactional:=true)` the generated script will\nwrap itself in `BEGIN/COMMIT`.\n\nIf `metagration.export(run_migrations:=true)` the generated script\nwill run the migrations immediately after inserting them.\n\n## Docker Entrypoint\n\nMetagrations can be used easily from the standard postgres docker\ncontainer entry point directory.  The SQL code dumped from\n`export(run_migrations:=true)` (see above) can be dropped into the\n`/docker-entrypoint-initdb.d/` directory in the container and the\nmigrations will be inserted and automatically run when the new\ncontainer is initialized.\n\n## How does it work?\n\nMetagration scripts are stored procedures run *in revision order*.\nThis means that revision 2 is always run after 1, and before 3 when\nmigrating forward, and the opposite when going backwards.  It is not\npossible to insert \"in between\" two existing revisions, even if their\nrevisions are not consecutive.  A `BEFORE INSERT` trigger enforces\nthat new scripts must have a `revision \u003e max(revision)` for all\nexisting scripts.  While you can disable this trigger to bulk import\nrevisions you will be responsible for their revision order being\ncorrect.\n\nWhen a script is created with `metagration.new_script()` the up and down\ncode are substituted into the body dynamically generated plpgsql\nprocedure.  You don't have to use `new_script()`, a script can be written\nin any supported language that can write stored procedures, such as\npython and javascript.\n\nOne and only one script at a time can be `is_current = true`.  This is\nenforced with a `UNIQUE` partial index.  The procedure\n`metagration.run()` also does a 'LOCK ... SHARE MODE' on the script\ntable when it runs ensuring only one metagration script can run at a\ntime.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichelp%2Fmetagration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichelp%2Fmetagration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichelp%2Fmetagration/lists"}