{"id":13469378,"url":"https://github.com/ankane/pgsync","last_synced_at":"2025-11-17T14:02:53.797Z","repository":{"id":38378281,"uuid":"47600993","full_name":"ankane/pgsync","owner":"ankane","description":"Sync data from one Postgres database to another","archived":false,"fork":false,"pushed_at":"2025-10-29T01:28:30.000Z","size":473,"stargazers_count":3396,"open_issues_count":14,"forks_count":215,"subscribers_count":34,"default_branch":"master","last_synced_at":"2025-11-14T15:13:43.007Z","etag":null,"topics":["postgresql"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ankane.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2015-12-08T05:37:40.000Z","updated_at":"2025-11-08T16:39:41.000Z","dependencies_parsed_at":"2023-02-19T02:30:59.860Z","dependency_job_id":"5a0969d0-b7fc-493c-a8df-827182ec99db","html_url":"https://github.com/ankane/pgsync","commit_stats":{"total_commits":651,"total_committers":18,"mean_commits":"36.166666666666664","dds":0.3855606758832565,"last_synced_commit":"de889467381892539c8272780091a5c1cbdadd8e"},"previous_names":[],"tags_count":43,"template":false,"template_full_name":null,"purl":"pkg:github/ankane/pgsync","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fpgsync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fpgsync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fpgsync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fpgsync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ankane","download_url":"https://codeload.github.com/ankane/pgsync/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fpgsync/sbom","scorecard":{"id":197156,"data":{"date":"2025-08-11","repo":{"name":"github.com/ankane/pgsync","commit":"14694efcf8236c7f8db17d10d265ab0757af029c"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4.3,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":9,"reason":"8 commit(s) and 3 issue activity found in the last 90 days -- score normalized to 9","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/build.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE.txt:0","Info: FSF or OSI recognized license: MIT License: LICENSE.txt:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build.yml:15: update your workflow using https://app.stepsecurity.io/secureworkflow/ankane/pgsync/build.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/build.yml:16: update your workflow using https://app.stepsecurity.io/secureworkflow/ankane/pgsync/build.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/build.yml:20: update your workflow using https://app.stepsecurity.io/secureworkflow/ankane/pgsync/build.yml/master?enable=pin","Warn: containerImage not pinned by hash: Dockerfile:1: pin your Docker image by updating ruby:3-alpine to ruby:3-alpine@sha256:e284f39a2103a564dca9771a81bfecb455b04cd3be4149b133ed7e508ef1b65f","Info:   0 out of   1 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   2 third-party GitHubAction dependencies pinned","Info:   0 out of   1 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-16T22:01:16.000Z","repository_id":38378281,"created_at":"2025-08-16T22:01:16.001Z","updated_at":"2025-08-16T22:01:16.001Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":284893575,"owners_count":27080531,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-17T02:00:06.431Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["postgresql"],"created_at":"2024-07-31T15:01:36.600Z","updated_at":"2025-11-17T14:02:53.792Z","avatar_url":"https://github.com/ankane.png","language":"Ruby","readme":"# pgsync\n\nSync data from one Postgres database to another (like `pg_dump`/`pg_restore`). Designed for:\n\n- **speed** - tables are transferred in parallel\n- **security** - built-in methods to prevent sensitive data from ever leaving the server\n- **flexibility** - gracefully handles schema differences, like missing columns and extra columns\n- **convenience** - sync partial tables, groups of tables, and related records\n\n:tangerine: Battle-tested at [Instacart](https://www.instacart.com/opensource)\n\n[![Build Status](https://github.com/ankane/pgsync/actions/workflows/build.yml/badge.svg)](https://github.com/ankane/pgsync/actions)\n\n## Installation\n\npgsync is a command line tool. To install, run:\n\n```sh\ngem install pgsync\n```\n\nThis will give you the `pgsync` command. If installation fails, you may need to install [dependencies](#dependencies).\n\nYou can also install it with Homebrew:\n\n```sh\nbrew install pgsync\n```\n\nOr [Docker](#docker).\n\n## Setup\n\nIn your project directory, run:\n\n```sh\npgsync --init\n```\n\nThis creates `.pgsync.yml` for you to customize. We recommend checking this into your version control (assuming it doesn’t contain sensitive information). `pgsync` commands can be run from this directory or any subdirectory.\n\n## How to Use\n\nFirst, make sure your schema is set up in both databases. We recommend using a schema migration tool for this, but pgsync also provides a few [convenience methods](#schema). Once that’s done, you’re ready to sync data.\n\nSync tables\n\n```sh\npgsync\n```\n\nSync specific tables\n\n```sh\npgsync table1,table2\n```\n\nWorks with wildcards as well\n\n```sh\npgsync \"table*\"\n```\n\nSync specific rows (existing rows are overwritten)\n\n```sh\npgsync products \"where store_id = 1\"\n```\n\nYou can also preserve existing rows\n\n```sh\npgsync products \"where store_id = 1\" --preserve\n```\n\nOr truncate them\n\n```sh\npgsync products \"where store_id = 1\" --truncate\n```\n\n## Tables\n\nExclude specific tables\n\n```sh\npgsync --exclude table1,table2\n```\n\nAdd to `.pgsync.yml` to exclude by default\n\n```yml\nexclude:\n  - table1\n  - table2\n```\n\nSync tables from all schemas or specific schemas (by default, only the search path is synced)\n\n```sh\npgsync --all-schemas\n# or\npgsync --schemas public,other\n# or\npgsync public.table1,other.table2\n```\n\n## Groups\n\nDefine groups in `.pgsync.yml`:\n\n```yml\ngroups:\n  group1:\n    - table1\n    - table2\n```\n\nAnd run:\n\n```sh\npgsync group1\n```\n\n## Variables\n\nYou can also use groups to sync a specific record and associated records in other tables.\n\nTo get product `123` with its reviews, last 10 coupons, and store, use:\n\n```yml\ngroups:\n  product:\n    products: \"where id = {1}\"\n    reviews: \"where product_id = {1}\"\n    coupons: \"where product_id = {1} order by created_at desc limit 10\"\n    stores: \"where id in (select store_id from products where id = {1})\"\n```\n\nAnd run:\n\n```sh\npgsync product:123\n```\n\n## Schema\n\nSync the schema before the data (this wipes out existing data)\n\n```sh\npgsync --schema-first\n```\n\nSpecify tables\n\n```sh\npgsync table1,table2 --schema-first\n```\n\nSync the schema without data (this wipes out existing data)\n\n```sh\npgsync --schema-only\n```\n\npgsync does not try to sync Postgres extensions.\n\n## Sensitive Data\n\nPrevent sensitive data like email addresses from leaving the remote server.\n\nDefine rules in `.pgsync.yml`:\n\n```yml\ndata_rules:\n  email: unique_email\n  last_name: random_letter\n  birthday: random_date\n  users.auth_token:\n    value: secret\n  visits_count:\n    statement: \"(RANDOM() * 10)::int\"\n  encrypted_*: null\n```\n\n`last_name` matches all columns named `last_name` and `users.last_name` matches only the users table. Wildcards are supported, and the first matching rule is applied.\n\nOptions for replacement are:\n\n- `unique_email`\n- `unique_phone`\n- `unique_secret`\n- `random_letter`\n- `random_int`\n- `random_date`\n- `random_time`\n- `random_ip`\n- `value`\n- `statement`\n- `null`\n- `untouched`\n\nRules starting with `unique_` require the table to have a single column primary key. `unique_phone` requires a numeric primary key.\n\n## Foreign Keys\n\nForeign keys can make it difficult to sync data. Three options are:\n\n1. Defer constraints (recommended)\n2. Manually specify the order of tables\n3. Disable foreign key triggers, which can silently break referential integrity (not recommended)\n\nTo defer constraints, use:\n\n```sh\npgsync --defer-constraints\n```\n\nTo manually specify the order of tables, use `--jobs 1` so tables are synced one-at-a-time.\n\n```sh\npgsync table1,table2,table3 --jobs 1\n```\n\nTo disable foreign key triggers and potentially break referential integrity, use:\n\n```sh\npgsync --disable-integrity\n```\n\nThis requires superuser privileges on the `to` database. If syncing to (not from) Amazon RDS, use the `rds_superuser` role. If syncing to (not from) Heroku, there doesn’t appear to be a way to disable integrity.\n\n## Triggers\n\nDisable user triggers with:\n\n```sh\npgsync --disable-user-triggers\n```\n\n## Sequences\n\nSkip syncing sequences with:\n\n```sh\npgsync --no-sequences\n```\n\n## Append-Only Tables\n\nFor extremely large, append-only tables, sync in batches.\n\n```sh\npgsync large_table --in-batches\n```\n\nNote: This requires the table to have a numeric, increasing primary key\n\nThe script will resume where it left off when run again, making it great for backfills.\n\n## Connection Security\n\nAlways make sure your [connection is secure](https://ankane.org/postgres-sslmode-explained) when connecting to a database over a network you don’t fully trust. Your best option is to connect over SSH or a VPN. Another option is to use `sslmode=verify-full`. If you don’t do this, your database credentials can be compromised.\n\n## Safety\n\nTo keep you from accidentally overwriting production, the destination is limited to `localhost` or `127.0.0.1` by default.\n\nTo use another host, add `to_safe: true` to your `.pgsync.yml`.\n\n## Multiple Databases\n\nTo use with multiple databases, run:\n\n```sh\npgsync --init db2\n```\n\nThis creates `.pgsync-db2.yml` for you to edit. Specify a database in commands with:\n\n```sh\npgsync --db db2\n```\n\n## Integrations\n\n- [Django](#django)\n- [Heroku](#heroku)\n- [Laravel](#laravel)\n- [Rails](#rails)\n\n### Django\n\nIf you run `pgsync --init` in a Django project, migrations will be excluded in `.pgsync.yml`.\n\n```yml\nexclude:\n  - django_migrations\n```\n\n### Heroku\n\nIf you run `pgsync --init` in a Heroku project, the `from` database will be set in `.pgsync.yml`.\n\n```yml\nfrom: $(heroku config:get DATABASE_URL)?sslmode=require\n```\n\n### Laravel\n\nIf you run `pgsync --init` in a Laravel project, migrations will be excluded in `.pgsync.yml`.\n\n```yml\nexclude:\n  - migrations\n```\n\n### Rails\n\nIf you run `pgsync --init` in a Rails project, Active Record metadata and schema migrations will be excluded in `.pgsync.yml`.\n\n```yml\nexclude:\n  - ar_internal_metadata\n  - schema_migrations\n```\n\n## Debugging\n\nTo view the SQL that’s run, use:\n\n```sh\npgsync --debug\n```\n\n## Other Commands\n\nHelp\n\n```sh\npgsync --help\n```\n\nVersion\n\n```sh\npgsync --version\n```\n\nList tables\n\n```sh\npgsync --list\n```\n\n## Scripts\n\nUse groups when possible to take advantage of parallelism.\n\nFor Ruby scripts, you may need to do:\n\n```rb\nBundler.with_unbundled_env do\n  system \"pgsync ...\"\nend\n```\n\n## Docker\n\nGet the [Docker image](https://hub.docker.com/r/ankane/pgsync) with:\n\n```sh\ndocker pull ankane/pgsync\nalias pgsync=\"docker run -ti --rm -v .:/conf -w /conf ankane/pgsync\"\n```\n\nThis will give you the `pgsync` command.\n\nFor databases on the host machine, use `host.docker.internal` as the hostname (on Linux, this requires `--add-host=host.docker.internal:host-gateway`).\n\n## Dependencies\n\nIf installation fails, your system may be missing Ruby or libpq.\n\nOn Mac, run:\n\n```sh\nbrew install libpq\n```\n\nOn Ubuntu, run:\n\n```sh\nsudo apt-get install ruby-dev libpq-dev build-essential\n```\n\n## Upgrading\n\nRun:\n\n```sh\ngem install pgsync\n```\n\nTo use master, run:\n\n```sh\ngem install specific_install\ngem specific_install https://github.com/ankane/pgsync.git\n```\n\nWith Homebrew, run:\n\n```sh\nbrew upgrade pgsync\n```\n\nWith Docker, run:\n\n```sh\ndocker pull ankane/pgsync\n```\n\n## Related Projects\n\nAlso check out:\n\n- [Dexter](https://github.com/ankane/dexter) - The automatic indexer for Postgres\n- [PgHero](https://github.com/ankane/pghero) - A performance dashboard for Postgres\n- [pgslice](https://github.com/ankane/pgslice) - Postgres partitioning as easy as pie\n\n## Thanks\n\nInspired by [heroku-pg-transfer](https://github.com/ddollar/heroku-pg-transfer).\n\n## History\n\nView the [changelog](https://github.com/ankane/pgsync/blob/master/CHANGELOG.md)\n\n## Contributing\n\nEveryone is encouraged to help improve this project. Here are a few ways you can help:\n\n- [Report bugs](https://github.com/ankane/pgsync/issues)\n- Fix bugs and [submit pull requests](https://github.com/ankane/pgsync/pulls)\n- Write, clarify, or fix documentation\n- Suggest or add new features\n\nTo get started with development:\n\n```sh\ngit clone https://github.com/ankane/pgsync.git\ncd pgsync\nbundle install\n\ncreatedb pgsync_test1\ncreatedb pgsync_test2\ncreatedb pgsync_test3\n\nbundle exec rake test\n```\n","funding_links":[],"categories":["Ruby","postgresql","Compiled list","Data","Awesome Ruby CLIs","Utilities","Gems"],"sub_categories":["plv8:","Replication","Database","Data Management"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fankane%2Fpgsync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fankane%2Fpgsync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fankane%2Fpgsync/lists"}