{"id":13514119,"url":"https://github.com/Shopify/ghostferry","last_synced_at":"2025-03-31T02:33:22.232Z","repository":{"id":31897859,"uuid":"129288584","full_name":"Shopify/ghostferry","owner":"Shopify","description":"The swiss army knife of live data migrations","archived":false,"fork":false,"pushed_at":"2025-03-26T11:43:06.000Z","size":20501,"stargazers_count":781,"open_issues_count":71,"forks_count":72,"subscribers_count":417,"default_branch":"main","last_synced_at":"2025-03-26T12:32:28.064Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://shopify.github.io/ghostferry","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Shopify.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-12T17:32:23.000Z","updated_at":"2025-03-12T19:17:18.000Z","dependencies_parsed_at":"2024-02-13T12:46:39.107Z","dependency_job_id":"cd61be78-6a0f-49ec-9c42-efb20eb6af06","html_url":"https://github.com/Shopify/ghostferry","commit_stats":{"total_commits":722,"total_committers":47,"mean_commits":"15.361702127659575","dds":0.5761772853185596,"last_synced_commit":"c4f3291088589171424e10cb5451fe4338a3d536"},"previous_names":[],"tags_count":44,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shopify%2Fghostferry","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shopify%2Fghostferry/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shopify%2Fghostferry/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shopify%2Fghostferry/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Shopify","download_url":"https://codeload.github.com/Shopify/ghostferry/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246407402,"owners_count":20772127,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T05:00:46.927Z","updated_at":"2025-03-31T02:33:22.225Z","avatar_url":"https://github.com/Shopify.png","language":"Go","readme":"Ghostferry\n==========\n\nGhostferry is a library that enables you to selectively copy data from one mysql instance to another with minimal amount of downtime.\n\nIt is inspired by Github's [gh-ost](https://github.com/github/gh-ost),\nalthough instead of copying data from and to the same database, Ghostferry\ncopies data from one database to another and has the ability to only\npartially copy data.\n\nThere is an example application called ghostferry-copydb included (under the\n`copydb` directory) that demonstrates this library by copying an entire\ndatabase from one machine to another.\n\nTalk to us on IRC at [irc.freenode.net #ghostferry](https://webchat.freenode.net/?channels=#ghostferry).\n\n- **Tutorial and General Documentations**: https://shopify.github.io/ghostferry\n- Code documentations: https://godoc.org/github.com/Shopify/ghostferry\n\nOverview of How it Works\n------------------------\n\nAn overview of Ghostferry's high-level design is expressed in the [TLA+\nspecification](https://en.wikipedia.org/wiki/TLA%2B), under the `tlaplus` directory. It may be good to consult with\nthat as it has a concise definition. However, the specification might not be\nentirely correct as proofs remain elusive.\n\nOn a high-level, Ghostferry is broken into several components, enabling it to\ncopy data. This is documented at\nhttps://shopify.github.io/ghostferry/main/technicaloverview.html\n\nDevelopment Setup\n-----------------\n\n### Installation\n\n#### For Internal Contributors\n\n`dev up`\n\n#### For External Contributors\n\n- Have Docker installed\n- Clone the repo\n- `docker-compose up -d`\n- `nix-shell`\n\nTesting\n---------------\n\n#### Run all tests\n\n- `make test`\n\n#### Run example copydb usage\n\n- `make copydb \u0026\u0026 ghostferry-copydb -verbose examples/copydb/conf.json`\n- For a more detailed tutorial, see the\n  [documentation](https://shopify.github.io/ghostferry).\n\n### Ruby Integration Tests\n\nKindly take note of following options:\n\n- `DEBUG=1`: To see more detailed debug output by `Ghostferry` live, as opposed\n  to only when the test fails. This is helpful for debugging hanging test.\n\nExamples:\n\nRun all tests\n\n`rake test`\n\nRun a single file\n\n`rake test TEST=test/integration/trivial_test.rb`\n\nor\n\n`ruby -Itest test/integration/trivial_test.rb`\n\nRun a specific test\n\n`DEBUG=1 ruby -Itest test/integration/trivial_test.rb -n \"TrivialIntegrationTest#test_logged_query_omits_columns\"`\n","funding_links":[],"categories":["Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FShopify%2Fghostferry","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FShopify%2Fghostferry","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FShopify%2Fghostferry/lists"}