{"id":31578997,"url":"https://github.com/koltyakov/pgcopy","last_synced_at":"2026-04-29T16:32:59.597Z","repository":{"id":307737858,"uuid":"1030527349","full_name":"koltyakov/pgcopy","owner":"koltyakov","description":"🐘 PostgreSQL data migration tool","archived":false,"fork":false,"pushed_at":"2025-12-31T16:46:59.000Z","size":1088,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-05T01:51:59.730Z","etag":null,"topics":["cli","data","database","golang","migration","postgresql","sync"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/koltyakov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-01T19:48:33.000Z","updated_at":"2025-12-31T16:42:05.000Z","dependencies_parsed_at":"2025-08-01T23:32:46.457Z","dependency_job_id":"1562d4cf-8a38-4725-9c19-3ba56add4a75","html_url":"https://github.com/koltyakov/pgcopy","commit_stats":null,"previous_names":["koltyakov/pgcopy"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/koltyakov/pgcopy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koltyakov%2Fpgcopy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koltyakov%2Fpgcopy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koltyakov%2Fpgcopy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koltyakov%2Fpgcopy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/koltyakov","download_url":"https://codeload.github.com/koltyakov/pgcopy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koltyakov%2Fpgcopy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32434828,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T13:34:34.882Z","status":"ssl_error","status_checked_at":"2026-04-29T13:34:29.830Z","response_time":110,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","data","database","golang","migration","postgresql","sync"],"created_at":"2025-10-05T20:18:10.028Z","updated_at":"2026-04-29T16:32:59.589Z","avatar_url":"https://github.com/koltyakov.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pgcopy\n\n**pgcopy** is a CLI tool for copying data between PostgreSQL databases with identical schemas. It's built for data-only migrations and supports parallel processing.\n\n![pgcopy in action](./assets/pgcopy.gif)\n\n\u003e Safety \u0026 Overwrite Semantics: pgcopy truncates destination tables and replaces their data. Review the short guide before running: see [Safety \u0026 Overwrite Semantics](docs/safety.md).\n\n## When to Use pgcopy\n\nDesigned for fast data-only syncs between PostgreSQL databases with matching schemas. Ideal when pg_restore is slow, you lack superuser privileges, or you need parallelism and robust foreign key handling. Not CDC/replication or a schema migration tool.\n\n\n## Quick Start\n\nSee the concise guide at docs/quick-start.md for install, safety, and the 4 commands you’ll use most.\n\n## Common Scenarios\n\n- Refresh staging from prod: `pgcopy copy --source \"postgres://user:pass@prod:5432/db\" --target \"postgres://user:pass@staging:5432/db\"`\n- Migrate hosts (data-only): `pgcopy copy --source \"postgres://user:pass@old:5432/db\" --target \"postgres://user:pass@new:5432/db\"`\n- Copy subset: `--include \"public.users,public.orders,public.products\"`\n- Speed up over WAN: `--parallel 8 --copy-pipe --compress`\n- Web dashboard: `--output web`\n\n## Features\n\n- High Performance: Parallel table copying with configurable workers\n- Batch Processing: Configurable batch sizes for optimal memory usage\n- Progress Tracking: Live progress monitoring with a visual progress bar enabled by default (stays fixed at top while logs scroll below)\n- Flexible Configuration: Support for connection strings, config files, and command-line options\n- Table Filtering: Include/exclude specific tables from the copy operation\n- Data Safety: Comprehensive validation and confirmation dialogs\n- Dry Run Mode: Preview what will be copied without actually copying data\n- Transaction Safety: Uses transactions to ensure data consistency\n- Advanced Foreign Key Handling: Automatically detects and manages foreign key constraints, including circular dependencies\n- Streaming COPY Pipeline: Direct source→destination streaming via PostgreSQL COPY for higher throughput (`--copy-pipe`)\n- In-flight Compression: Optional gzip compression of the COPY stream to reduce network I/O (`--compress`, requires `--copy-pipe`)\n\n## Installation\n\n### Download prebuilt binaries\n\nGrab the latest release for your OS from GitHub Releases:\n\n- https://github.com/koltyakov/pgcopy/releases\n\nAfter download:\n- macOS/Linux: `chmod +x pgcopy` \u0026\u0026 move to a directory on your PATH (e.g., `/usr/local/bin`)\n- Windows: use `pgcopy.exe` or add its folder to PATH\n\n### From Source\n\n```bash\ngit clone https://github.com/koltyakov/pgcopy.git\ncd pgcopy\nmake build\n```\n\nThe binary will be available in the `bin/` directory.\n\n### Install to System\n\n```bash\nmake install\n```\n\nThis will install `pgcopy` to `/usr/local/bin/`.\n\n## Usage\n\nStart with [docs/quick-start.md](docs/quick-start.md). For patterns and flags:\n- Safety \u0026 overwrite semantics: docs/safety.md\n- Wildcards and filtering: docs/wildcards.md\n- Output modes (plain, interactive, web): below\n- Streaming COPY and compression: see quick-start and performance docs\n\n## Commands\n\n### `copy`\n\nCopy data from source to target database.\n\n**Flags:**\n\n- `--source, -s`: Source database connection string\n- `--target, -t`: Target database connection string\n- `--parallel, -p`: Number of parallel workers (default: 4)\n- `--batch-size`: Batch size for data copying (default: 1000)\n- `--exclude`: Tables to exclude from copying (comma-separated, supports wildcards: `temp_*,*_logs`)\n- `--include`: Tables to include in copying (comma-separated, supports wildcards: `user_*,*_data`)\n- `--dry-run`: Show what would be copied without actually copying\n- `--skip-backup`: Skip confirmation dialog for data overwrite (use with caution)\n- `--output, -o`: Output mode: 'plain' (minimal output, default), 'progress' (progress bar), 'interactive' (live table progress), 'web' (web dashboard)\n- `--copy-pipe`: Use streaming COPY pipeline (source→target) instead of row fetch + insert\n- `--compress`: Gzip-compress streaming COPY pipeline (requires `--copy-pipe`)\n- `--exact-rows`: Compute exact source row counts with `COUNT(*)` during discovery to avoid bad estimates after TRUNCATE (slower on large tables)\n- `--no-timeouts`: Disable internal operation timeouts (use with caution; operations may wait indefinitely)\n\n### `list`\n\nList tables in a database with row counts and sizes.\n\n**Flags:**\n\n- `--source, -s`: Source database connection string\n- `--schema`: Specific schema to list (optional)\n\n### `version`\n\nDisplay version information.\n\n## Configuration\n\n### Connection Strings\n\nPostgreSQL connection strings follow the standard format:\n\n```\npostgres://username:password@hostname:port/database_name?sslmode=disable\n```\n\n## Foreign Key Management\n\npgcopy automatically handles complex foreign key constraints without superuser privileges. See [Foreign key handling](docs/foreign-keys.md) for strategies (replica vs drop/restore), idempotent restore, and recovery.\n\n## Performance Considerations\n\nSee [Performance](docs/performance.md) for comparisons and tuning guidance.\n\n#### Performance Characteristics\n\n- Parallel Processing: Multiple tables copied simultaneously\n- Batch Optimization: Configurable batch sizes for memory efficiency  \n- Connection Pooling: Optimized database connections\n- Progress Tracking: Real-time feedback without performance impact\n- Streaming Pipeline: COPY-based streaming eliminates per-row overhead and minimizes round-trips\n- In-flight Compression: Gzip can greatly reduce network transfer time on bandwidth-limited links\n\n### Output feature matrix\n\n| Feature | raw (plain) | interactive | web |\n|---------|:-----------:|:-----------:|:---:|\n| Minimal logs | ✓ | ✓ | ✓ |\n| Overall progress bar |  | ✓ | ✓ |\n| Per-table live status |  | ✓ | ✓ |\n| Real-time dashboard |  |  | ✓ |\n| Auto-open UI |  |  | ✓ |\n| Good for CI | ✓ |  |  |\n\n## Advanced docs\n\n- [Safety \u0026 Overwrite Semantics](docs/safety.md)\n- [Table filtering with wildcards](docs/wildcards.md)\n- [Foreign key handling](docs/foreign-keys.md)\n- [Internals overview](docs/internals.md)\n- [Architecture](docs/architecture.md)\n- [Performance](docs/performance.md)\n- [Build \u0026 Release](docs/build-release.md)\n- [Quality \u0026 Security](docs/quality.md)\n\n## Examples\n\nSee [Quick start](docs/quick-start.md) for compact, copy-pasteable examples covering dry-run, web UI, performance flags, and filtering.\n\nTip: If your environment requires very long-running operations, you can pass `--no-timeouts` to disable internal deadlines. This relies on server-side timeouts for protection and may wait indefinitely.\n\n## Building from Source\n\n### Prerequisites\n\n- Go 1.21 or later\n- PostgreSQL client libraries\n\n### Build\n\n```bash\n# Clone the repository\ngit clone https://github.com/koltyakov/pgcopy.git\ncd pgcopy\n\n# Install dependencies\nmake deps\n\n# Build\nmake build\n\n# Run tests\nmake test\n\n# Build for all platforms\nmake build-all\n```\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Add tests\n5. Run `make test` and `make lint`\n6. Submit a pull request\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Support\n\n- Create an issue on GitHub for bug reports or feature requests\n- Check existing issues before creating new ones\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoltyakov%2Fpgcopy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkoltyakov%2Fpgcopy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoltyakov%2Fpgcopy/lists"}