{"id":44539274,"url":"https://github.com/toudi/gorpheus","last_synced_at":"2026-02-13T18:55:17.611Z","repository":{"id":57646761,"uuid":"150907305","full_name":"toudi/gorpheus","owner":"toudi","description":"database agnostic migration utility / library","archived":false,"fork":false,"pushed_at":"2025-07-27T18:07:56.000Z","size":152,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-27T20:17:21.905Z","etag":null,"topics":["agnostic","database","migrations","schema-migrations"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/toudi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-09-29T22:17:33.000Z","updated_at":"2024-11-22T14:27:58.000Z","dependencies_parsed_at":"2025-05-18T18:38:21.010Z","dependency_job_id":null,"html_url":"https://github.com/toudi/gorpheus","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/toudi/gorpheus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toudi%2Fgorpheus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toudi%2Fgorpheus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toudi%2Fgorpheus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toudi%2Fgorpheus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/toudi","download_url":"https://codeload.github.com/toudi/gorpheus/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toudi%2Fgorpheus/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29414285,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-13T06:24:03.484Z","status":"ssl_error","status_checked_at":"2026-02-13T06:23:12.830Z","response_time":78,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agnostic","database","migrations","schema-migrations"],"created_at":"2026-02-13T18:55:17.065Z","updated_at":"2026-02-13T18:55:17.604Z","avatar_url":"https://github.com/toudi.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# gorpheus\n\nThis code is a proof of concept migration manager. Please refer to examples directory for practical usage examples.\n\n## When it might and when it might not be helpful to me?\n\nI wrote this code because I was looking for a library (in go) that would implement migrations as django does them. This means:\n\n- the migrations are database-backend agnostic (i.e. you write migrations once and can execute them on any database you wish)\n- the migrations can be split across multiple directories\n- the migrations can depend on each other\n- the migrations can be applied within a single namespace (i.e. `./manage.py migrate my_namespace` in Django)\n- the migrations have human-readable names (i.e. 0001, 0002, and so on)\n- the migration can be a python function (so-called data migrations)\n- the forwards and backwards actions are specified within the same file / module\n\nThere are several libraries that implement migrations in go, however the one that I saw required that the migrations would be in a single directory, or would have versions that would\ndepend on timestamps or that forwards / backwards actions would be split into two files, or they did not support running go code as a migration at all.\n\nwhile using Django, I quickly discovered that relying on timestamps is not that helpful when it comes to numbering. Imagine for a second that 2 or more developers start working on their\nown branches of the code and each of the developers needs to modify the database. If they do that with incrementing numbers as a base then all that needs to be done during the merge process\nis renaming of the files. it is still possible to do that with timestamps of course, but it's not as user-friendly.\n\nOn the other hand, if you know exactly what would be the target database for your code then there's hardly any point in using my code - the other tools are definetely more suitable for that.\n\n## Basic concepts\n\nOk, if you made it here this means that you think the project might be beneficial to you :-) Here are the key concepts of the project.\n\n### Collection\n\ngorpheus organizes migrations in a collection. Each item of the collection must implement a `Migration` interface. The reason for this\nis simple - I wanted to support many possible sources of migration (file / embedded files / S3 storage, you name it.)\n\n### Migration\n\nEach element of the `Collection` array is a `Migration` interface. Each migration consists of the following things:\n\n#### Type (`uint8`)\n\n| Value | Description |\n| --- | --- |\n| `TypeSQL` | is straight forward - you need to provide the SQL that will be executed during forward / backwards migration. |\n| `TypeFizz` | will be translated to the target dialect with the help of wonderful gobuffalo/fizz library. |\n| `TypeGo` | is not an interpreted migration script, but rather a function that will perform the migration. |\n\n#### Version (`string`)\n\neach migration is represented by a version, which is a human-friendly string that contains the namespace. For instance:\n\n```\n       version\n/----------------------\\\n|                      |\nusers/0001_create_table.sql\n^^^^^ ^^^^ ^^^^^^^^^^^^\n  |\t    |       +---------- name\n  |     +------------------ version number\n  +------------------------ namespace\n```\n\n#### Depends (`[]string`)\n\ndependencies array. If you don't define any, gorpheus will assign the previous one from the same namespace (if it exists). The reason for that behavior\nis that gorpheus traverses the migrations collection and builds a graph of dependencies. Of course it would be tedious to manually enter dependencies\nin each of the migrations so that's why this process is optional.\n\n#### Reader (`io.ReadSeekCloser`)\n\nthis is how gorpheus will be able to read and parse sections of the migration script. This is also how it is possible to support other sources of storage\nthat I haven't concidered (i.e. S3, FTP, you name it)\n\n### Migration file and sections\n\nHere's an example migration file:\n\n```SQL\n-- gorph DEPENDS \nusers/0001_create_users \n-- end --\n-- gorph UP\nCREATE TABLE foo (\n    invoice_id INTEGER,\n    user_id INTEGER,\n    FOREIGN KEY(invoice_id) REFERENCES invoices(id),\n    FOREIGN KEY(user_id) REFERENCES users(id)\n);\n-- end --\n-- gorph DOWN\nDROP TABLE foo;\n-- end --\n```\n\nas you can see, it consists of 3 sections\n\n- dependencies section. This one is optional as I already mentioned - if you don't have any depedencies from other namespaces then gorpheus will automatically set the previous migration as a dependency\n- up / forwards section. this defines what will be done when forwards migration will be applied\n- down / backwards section. this defines what will be done when backwards migration will be applied\n\n### Go / data migrations\n\nWhat if you want to migrate your database but there's some logic that you want to apply which is hard (or even impossible) to write as an SQL query? This is where Go migrations come in place:\n\n```go\ntype MyDataMigration struct {\n\tmigration.GoMigration\n}\n\nfunc (dm *MyDataMigration) Up(tx *sqlx.Tx) error {\n\t_, err := tx.Exec(tx.Rebind(\"UPDATE users_inmemory SET email = ?;\"), \"foo@bar.baz\")\n\treturn err\n}\n\nfunc (dm *MyDataMigration) Down(tx *sqlx.Tx) error {\n\t_, err := tx.Exec(\"UPDATE users_inmemory SET email = null;\")\n\tfmt.Printf(\"Err=%v\", err)\n\treturn err\n}\n\nvar myDataMigration = \u0026MyDataMigration{\n\tGoMigration: migration.GoMigration{\n\t\tMigration: migration.Migration{\n\t\t\tVersion:   \"users/0004_inmem\",\n\t\t\tDepends:   []string{\"users/0003_inmem\"},\n\t\t},\n\t},\n}\n\ncollection.Register(myDataMigration)\n```\n\nAs you can see, the Go migration receives a database transaction (`tx`) and needs to return the error that will indicate whether a transaction will be commited or rolled back. You could do all sorts of things in here - for-loops, selects, etc, etc. This example is very trivial and could be as well defined as an regular sql migration, however I wanted to show the basics.\n\nFor obvious reasons, these migrations cannot live within the filesystem / embedFS but have to be compiled into the binary.\n\n## Tutorial\nOk, now that we have the basics out of the way let's get to some practical tutorial:\n\nFirst of all, you need to initialize a collection:\n\n```go\nc := gorpheus.Collection_init()\n```\n\nthen you can register the migrations. If you want to recursively go trough a\ndirectory, do the following:\n\n```go\nstorage.RegisterFSRecurse(c, \"some-dir\")\n```\n\nThis will walk recursively trough the directory \"some-dir\" and will register the migrations. The first level of directory\nwill get the namespace assigned as `default` and as the function recurses, it will change the namespace based on the subdirectory names.\n\nFor example:\n\n```\nmigrations/0001_something.sql         # will be recognized as default/0001_something.sql\nmigrations/0002_something.sql         # will be recognized as default/0002_something.sql\nmigrations/foobar/\nmigrations/foobar/0001_something.sql  # will be recognized as foobar/0001_something.sql\nmigrations/foobar/0002_something.sql  # will be recognized as foobar/0001_something.sql\n```\n\nIf you want to have more control over the namespaces then don't use the recurse function but specify everything by hand:\n\n```go\nstorage.RegisterFS(c, \"users\", \"users-ns\")\n```\n\nIf you have some files that you want to embed inside the final binary (go 1.16+ is required for this) you can do the following thing:\n\n```go\n//go:embed migrations/*\nvar embeddedMigrations embed.FS\n\nstorage.RegisterEmbedFS(c, embeddedMigrations, \"namespace\")\n```\n\nyou can also add some in-memory migrations too (I'm not really sure if this is beneficial, however I wrote this code way prior to go 1.16 when there was no embed functionality):\n\n```go\ntype NewMigration struct {\n\tmigration.Migration\n}\n\nvar newMigration = \u0026NewMigration{\n\tMigration: migration.Migration{\n\t\tVersion:   \"users/0003_inmem\",\n\t\tDepends:   []string{\"users/0002_something\"},\n\t},\n}\n\nfunc (n *NewMigration) UpScript() (string, uint8, error) {\n\treturn `create_table(\"users_inmemory\")`, migration.TypeFizz, nil\n}\n\nfunc (n *NewMigration) DownScript() (string, uint8, error) {\n\treturn `DROP TABLE users_inmemory`, migration.TypeSQL, nil\n}\n\n// don't forget to register it:\n\nc.Register(newMigration)\n```\n\n## The migration parameters\n\nThis is where you inform gorpheus how you wish to proceed - whether you want to migrate only a single namespace, or all of them, or to which revision.\n\n```go\n// in all of the examples you need to specify how gorpheus should connect to your database. I am ignoring this part in all examples just for brevity\n// - you can rely on a environment variable for database connection:\nvar myParams = \u0026gorpheus.MigrationParams{\n\tConnection: gorpheus.DbConnection{\n\t\tEnvKeyName: \"DATABASE_URL\",\n\t},\n\t// ...\n}\n\n// - or you can specify the connection URL manually:\nvar myParams = \u0026gorpheus.MigrationParams{\n\tConnection: gorpheus.DbConnection{\n\t\tConnectionURL: \"sqlite://example.sqlite\",\n\t},\n\t// ...\n}\n// - or you can specify existing *sql.DB instance, however you still need to pass the connection URL so that fizz can register that.\n//   from what I could understand, fizz doesn't actually do anything with the original DSN, however I'm not sure of that\nvar dsn = \"database-driver-params\"\ndb, err := sql.Open(\"driver\", dsn)\nvar myParams = \u0026gorpheus.MigrationParams{\n\tConnection: gorpheus.DbConnection{\n\t\tConn: db,\n\t\tConnectionURL: dsn,\n\t},\n\t// ...\n}\n\n// this is the default action - gorpheus will apply all outstanding forward migrations\nvar myParams = \u0026gorpheus.MigrationParams{}\n\n// this will migrate a single namespace to the latest version (including dependencies)\nvar myParams = \u0026gorpheus.MigrationParams{\n\tNamespace: \"users\",\n}\n\n// this will migrate a single namespace to a version specified by number\nvar myParams = \u0026gorpheus.MigrationParams{\n\tNamespace: \"users\",\n\tVersionNo: 123,\n}\n\nPlease note that if the database will be already migrated to, say, version number 125 within this namespace then gorpheus will roll back the namespace instead\n\n// this will instruct gorpheus to roll back all migrations from the namespace \"users\"\nvar myParams = \u0026gorpheus.MigrationParams{\n\tNamespace: \"users\",\n\tZero     : true,\n}\n\n// this will instruct gorpheus to roll back all migrations from all namespaces and destroy it's own state table\nvar myParams = \u0026gorpheus.MigrationParams{\n\tVaccuum: true,\n}\n\n// once you've decided on what you want to do, call Migrate function:\n\nerr := c.Migrate(\u0026params)\n\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoudi%2Fgorpheus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftoudi%2Fgorpheus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoudi%2Fgorpheus/lists"}