{"id":21824344,"url":"https://github.com/shopify/ghostferry","last_synced_at":"2026-04-15T13:00:57.288Z","repository":{"id":31897859,"uuid":"129288584","full_name":"Shopify/ghostferry","owner":"Shopify","description":"The swiss army knife of live data migrations","archived":false,"fork":false,"pushed_at":"2026-04-08T13:35:12.000Z","size":21168,"stargazers_count":843,"open_issues_count":79,"forks_count":74,"subscribers_count":360,"default_branch":"main","last_synced_at":"2026-04-08T15:06:02.520Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://shopify.github.io/ghostferry","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Shopify.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2018-04-12T17:32:23.000Z","updated_at":"2026-03-23T23:56:02.000Z","dependencies_parsed_at":"2024-02-13T12:46:39.107Z","dependency_job_id":"0fee2ce6-e38b-44c5-8633-b53e7befe00d","html_url":"https://github.com/Shopify/ghostferry","commit_stats":{"total_commits":722,"total_committers":47,"mean_commits":"15.361702127659575","dds":0.5761772853185596,"last_synced_commit":"c4f3291088589171424e10cb5451fe4338a3d536"},"previous_names":[],"tags_count":68,"template":false,"template_full_name":null,"purl":"pkg:github/Shopify/ghostferry","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shopify%2Fghostferry","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shopify%2Fghostferry/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shopify%2Fghostferry/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shopify%2Fghostferry/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Shopify","download_url":"https://codeload.github.com/Shopify/ghostferry/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shopify%2Fghostferry/sbom","scorecard":{"id":129275,"data":{"date":"2025-07-21","repo":{"name":"github.com/Shopify/ghostferry","commit":"4fe847322b80bcd9b70108379ed8221a559a281c"},"scorecard":{"version":"v5.2.1-23-gc890b972","commit":"c890b9726ee1b17d989f84e79873b38836e8e6fa"},"score":5.4,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/c890b9726ee1b17d989f84e79873b38836e8e6fa/docs/checks.md#packaging"}},{"name":"Code-Review","score":9,"reason":"Found 9/10 approved changesets -- score normalized to 9","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/c890b9726ee1b17d989f84e79873b38836e8e6fa/docs/checks.md#code-review"}},{"name":"Maintained","score":10,"reason":"13 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/c890b9726ee1b17d989f84e79873b38836e8e6fa/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/c890b9726ee1b17d989f84e79873b38836e8e6fa/docs/checks.md#dangerous-workflow"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/c890b9726ee1b17d989f84e79873b38836e8e6fa/docs/checks.md#cii-best-practices"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/cla.yml:1","Warn: no topLevel permission defined: .github/workflows/pages.yml:1","Warn: no topLevel permission defined: .github/workflows/release.yml:1","Warn: no topLevel permission defined: .github/workflows/tests.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/c890b9726ee1b17d989f84e79873b38836e8e6fa/docs/checks.md#token-permissions"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/c890b9726ee1b17d989f84e79873b38836e8e6fa/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/c890b9726ee1b17d989f84e79873b38836e8e6fa/docs/checks.md#fuzzing"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/c890b9726ee1b17d989f84e79873b38836e8e6fa/docs/checks.md#binary-artifacts"}},{"name":"Signed-Releases","score":0,"reason":"Project has not signed or included provenance with any releases.","details":["Warn: release artifact release-4fe8473 not signed: https://api.github.com/repos/Shopify/ghostferry/releases/231552456","Warn: release artifact release-b3b50ec not signed: https://api.github.com/repos/Shopify/ghostferry/releases/230518101","Warn: release artifact release-119c388 not signed: https://api.github.com/repos/Shopify/ghostferry/releases/218329503","Warn: release artifact release-0b22b7b not signed: https://api.github.com/repos/Shopify/ghostferry/releases/210787916","Warn: release artifact release-4de0930 not signed: https://api.github.com/repos/Shopify/ghostferry/releases/208420471","Warn: release artifact release-4fe8473 does not have provenance: https://api.github.com/repos/Shopify/ghostferry/releases/231552456","Warn: release artifact release-b3b50ec does not have provenance: https://api.github.com/repos/Shopify/ghostferry/releases/230518101","Warn: release artifact release-119c388 does not have provenance: https://api.github.com/repos/Shopify/ghostferry/releases/218329503","Warn: release artifact release-0b22b7b does not have provenance: https://api.github.com/repos/Shopify/ghostferry/releases/210787916","Warn: release artifact release-4de0930 does not have provenance: https://api.github.com/repos/Shopify/ghostferry/releases/208420471"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/c890b9726ee1b17d989f84e79873b38836e8e6fa/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/c890b9726ee1b17d989f84e79873b38836e8e6fa/docs/checks.md#branch-protection"}},{"name":"Pinned-Dependencies","score":7,"reason":"dependency not pinned by hash detected -- score normalized to 7","details":["Warn: third-party GitHubAction not pinned by hash: .github/workflows/cla.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/Shopify/ghostferry/cla.yml/main?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/build-docs.sh:11","Info:  15 out of  15 GitHub-owned GitHubAction dependencies pinned","Info:   2 out of   3 third-party GitHubAction dependencies pinned","Info:   0 out of   1 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/c890b9726ee1b17d989f84e79873b38836e8e6fa/docs/checks.md#pinned-dependencies"}},{"name":"Security-Policy","score":9,"reason":"security policy file detected","details":["Info: security policy file detected: github.com/Shopify/.github/SECURITY.md:1","Info: Found linked content: github.com/Shopify/.github/SECURITY.md:1","Warn: One or no descriptive hints of disclosure, vulnerability, and/or timelines in security policy","Info: Found text in security policy: github.com/Shopify/.github/SECURITY.md:1"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/c890b9726ee1b17d989f84e79873b38836e8e6fa/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":7,"reason":"3 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-6f62-3596-g6w7","Warn: Project is vulnerable to: GHSA-r995-q44h-hr64","Warn: Project is vulnerable to: GO-2022-0493 / GHSA-p782-xgp4-8hr8"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/c890b9726ee1b17d989f84e79873b38836e8e6fa/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/c890b9726ee1b17d989f84e79873b38836e8e6fa/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-16T04:51:03.356Z","repository_id":31897859,"created_at":"2025-08-16T04:51:03.357Z","updated_at":"2025-08-16T04:51:03.357Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31569400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-08T14:31:17.711Z","status":"ssl_error","status_checked_at":"2026-04-08T14:31:17.202Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-27T17:57:41.397Z","updated_at":"2026-04-08T19:00:24.855Z","avatar_url":"https://github.com/Shopify.png","language":"Go","readme":"Ghostferry\n==========\n\nGhostferry is a library that enables you to selectively copy data from one mysql instance to another with minimal amount of downtime.\n\nIt is inspired by Github's [gh-ost](https://github.com/github/gh-ost),\nalthough instead of copying data from and to the same database, Ghostferry\ncopies data from one database to another and has the ability to only\npartially copy data.\n\nThere is an example application called ghostferry-copydb included (under the\n`copydb` directory) that demonstrates this library by copying an entire\ndatabase from one machine to another.\n\nTalk to us on IRC at [irc.freenode.net #ghostferry](https://webchat.freenode.net/?channels=#ghostferry).\n\n- **Tutorial and General Documentations**: https://shopify.github.io/ghostferry\n- Code documentations: https://godoc.org/github.com/Shopify/ghostferry\n\nOverview of How it Works\n------------------------\n\nAn overview of Ghostferry's high-level design is expressed in the [TLA+\nspecification](https://en.wikipedia.org/wiki/TLA%2B), under the `tlaplus` directory. It may be good to consult with\nthat as it has a concise definition. However, the specification might not be\nentirely correct as proofs remain elusive.\n\nOn a high-level, Ghostferry is broken into several components, enabling it to\ncopy data. This is documented at\nhttps://shopify.github.io/ghostferry/main/technicaloverview.html\n\nDevelopment Setup\n-----------------\n\n### Installation\n\n#### For Internal Contributors\n\n`dev up`\n\n#### For External Contributors\n\n- Have Docker installed\n- Clone the repo\n- `docker-compose up -d`\n- `nix-shell`\n\nTesting\n---------------\n\n#### Run all tests\n\n- `make test`\n\n#### Run example copydb usage\n\n- `make copydb \u0026\u0026 ghostferry-copydb -verbose examples/copydb/conf.json`\n- For a more detailed tutorial, see the\n  [documentation](https://shopify.github.io/ghostferry).\n\n### Ruby Integration Tests\n\nKindly take note of following options:\n\n- `DEBUG=1`: To see more detailed debug output by `Ghostferry` live, as opposed\n  to only when the test fails. This is helpful for debugging hanging test.\n\nExamples:\n\nRun all tests\n\n`rake test`\n\nRun a single file\n\n`rake test TEST=test/integration/trivial_test.rb`\n\nor\n\n`ruby -Itest test/integration/trivial_test.rb`\n\nRun a specific test\n\n`DEBUG=1 ruby -Itest test/integration/trivial_test.rb -n \"TrivialIntegrationTest#test_logged_query_omits_columns\"`\n\nReleasing new version\n---------------------\n\n### Canary\n\nTag your commit with `canary/*` and push, i.e.\n\n```bash\ngit tag --sign --message=\"Initial support for UUIDs as pagination keys\" canary/v1.1.2-uuid-pagination-keys-alpha-1\ngit push origin --tags\n```\n\nThis will create the release named by tag.\n\n### Production\n\nFinal releases are created automatically on merge to `main` branch, they will end up with `release-SHA` name.\n\nRemember to update version prior to bigger releases in `Makefile` along with updating the `CHANGELOG.md`.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshopify%2Fghostferry","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshopify%2Fghostferry","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshopify%2Fghostferry/lists"}