{"id":38728600,"url":"https://github.com/teamniteo/pg_subsetter","last_synced_at":"2026-01-17T11:22:27.862Z","repository":{"id":187640585,"uuid":"677293123","full_name":"teamniteo/pg_subsetter","owner":"teamniteo","description":"pg-subsetter is a powerful and efficient tool designed to synchronize a fraction of a PostgreSQL database to another PostgreSQL database on the fly.","archived":false,"fork":false,"pushed_at":"2025-07-14T08:49:28.000Z","size":61,"stargazers_count":4,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-14T11:37:17.568Z","etag":null,"topics":["golang","postgres","rdbms","subset"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/teamniteo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-08-11T07:56:00.000Z","updated_at":"2025-07-14T08:49:33.000Z","dependencies_parsed_at":"2025-01-08T11:27:58.306Z","dependency_job_id":"60270ecc-dd59-4e4a-a95c-fc589a9faccb","html_url":"https://github.com/teamniteo/pg_subsetter","commit_stats":null,"previous_names":["teamniteo/pg-subsetter","teamniteo/pg_subsetter"],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/teamniteo/pg_subsetter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teamniteo%2Fpg_subsetter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teamniteo%2Fpg_subsetter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teamniteo%2Fpg_subsetter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teamniteo%2Fpg_subsetter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/teamniteo","download_url":"https://codeload.github.com/teamniteo/pg_subsetter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teamniteo%2Fpg_subsetter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28506908,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T10:25:30.148Z","status":"ssl_error","status_checked_at":"2026-01-17T10:25:29.718Z","response_time":85,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","postgres","rdbms","subset"],"created_at":"2026-01-17T11:22:27.159Z","updated_at":"2026-01-17T11:22:27.848Z","avatar_url":"https://github.com/teamniteo.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pg_subsetter\n\n[![lint](https://github.com/teamniteo/pg_subsetter/actions/workflows/lint.yml/badge.svg)](https://github.com/teamniteo/pg_subsetter/actions/workflows/lint.yml) [![build](https://github.com/teamniteo/pg_subsetter/actions/workflows/go.yml/badge.svg)](https://github.com/teamniteo/pg_subsetter/actions/workflows/go.yml) [![vuln](https://github.com/teamniteo/pg_subsetter/actions/workflows/vuln.yml/badge.svg)](https://github.com/teamniteo/pg_subsetter/actions/workflows/vuln.yml) [![release](https://github.com/teamniteo/pg_subsetter/actions/workflows/release.yml/badge.svg)](https://github.com/teamniteo/pg_subsetter/actions/workflows/release.yml)\n\n\n`pg_subsetter` is a tool designed to synchronize a fraction of a PostgreSQL database to another PostgreSQL database on the fly, it does not copy the SCHEMA.\n\n\n### Database Fraction Synchronization\n`pg_subsetter` allows you to select and sync a specific subset of your database. Whether it's a fraction of a table or a particular dataset, you can have it replicated in another database without synchronizing the entire DB.\n\n### Integrity Preservation with Foreign Keys\nForeign keys play a vital role in maintaining the relationships between tables. `pg_subsetter` ensures that all foreign keys(one-to-one, one-to many, many-to-many) are handled correctly during the synchronization process, maintaining the integrity and relationships of the data.\n\n### Efficient COPY Method\nUtilizing the native PostgreSQL COPY command, `pg_subsetter` performs data transfer with high efficiency. This method significantly speeds up the synchronization process, minimizing downtime and resource consumption.\n\n### Stateless Operation\n`pg_subsetter` is built to be stateless, meaning it does not maintain any internal state between runs. This ensures that each synchronization process is independent, enhancing reliability and making it easier to manage and scale.\n\n### Sync required rows\n`pg_subsetter` can be instructed to copy certain rows in specific tables, the command can be used multiple times to sync more data.\n\n## Usage\n\n```\nUsage of subsetter:\n  -dst string\n    \tDestination database DSN\n  -exclude value\n    \tQuery to ignore tables 'users: all', can be used multiple times\n  -f float\n    \tFraction of rows to copy (default 0.05)\n  -include value\n    \tQuery to copy required rows 'users: id = 1', can be used multiple times\n  -src string\n    \tSource database DSN\n  -v\tRelease information\n  -verbose\n    \tShow more information during sync\n```\n\n\n### Example\n\n\nPrepare schema in target database:\n\n```bash\npg_dump --schema-only --no-owner --no-acl -n public -f schemadump.sql \"postgres://test_source@localhost:5432/test_source?sslmode=disable\"\npsql -f schemadump.sql \"postgres://test_target@localhost:5432/test_target?sslmode=disable\"\n```\n\nCopy a fraction of the database and force certain rows to be also copied over:\n\n```\npg_subsetter \\\n      -src \"postgres://test_source@localhost:5432/test_source?sslmode=disable\" \\\n      -dst \"postgres://test_target@localhost:5432/test_target?sslmode=disable\" \\\n      -f 0.5\n      -include \"user: id=1\"\n      -include \"group: all\"\n      -exclude \"domains: all\"\n\n```\n\n# Installing\n\n```bash\ncurl -Ls https://github.com/teamniteo/pg_subsetter/releases/latest/download/pg_subsetter_Linux_x86_64.tar.gz | tar -xz \u0026\u0026 mv pg_subsetter /usr/bin\n```\n\nFor other downloads see releases.\n\n## We're hiring!\n\nAt Niteo we regularly contribute back to the Open Source community. If you do too, we'd like to invite you to [join our team](https://niteo.co/careers)!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fteamniteo%2Fpg_subsetter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fteamniteo%2Fpg_subsetter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fteamniteo%2Fpg_subsetter/lists"}