{"id":31103713,"url":"https://github.com/abc3/pg_transfer","last_synced_at":"2026-03-12T14:16:29.887Z","repository":{"id":313075098,"uuid":"1049891892","full_name":"abc3/pg_transfer","owner":"abc3","description":"Fast parallel Postgres table copying using CTID-based sharding in Zig","archived":false,"fork":false,"pushed_at":"2025-09-04T18:38:37.000Z","size":8,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-17T02:53:49.214Z","etag":null,"topics":["ctid","dump","postgres","zig"],"latest_commit_sha":null,"homepage":"","language":"Zig","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abc3.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-03T16:39:21.000Z","updated_at":"2025-09-04T18:38:40.000Z","dependencies_parsed_at":"2025-09-03T20:15:50.627Z","dependency_job_id":"68a3b033-568a-496b-aab6-5d3de068a507","html_url":"https://github.com/abc3/pg_transfer","commit_stats":null,"previous_names":["abc3/pg_transfer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/abc3/pg_transfer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abc3%2Fpg_transfer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abc3%2Fpg_transfer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abc3%2Fpg_transfer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abc3%2Fpg_transfer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abc3","download_url":"https://codeload.github.com/abc3/pg_transfer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abc3%2Fpg_transfer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30428009,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-12T14:00:25.264Z","status":"ssl_error","status_checked_at":"2026-03-12T13:59:52.690Z","response_time":114,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ctid","dump","postgres","zig"],"created_at":"2025-09-17T02:50:51.462Z","updated_at":"2026-03-12T14:16:29.868Z","avatar_url":"https://github.com/abc3.png","language":"Zig","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Copy Postgres tables between databases\n\n[![Zig Version](https://img.shields.io/badge/Zig-0.13.0+-blue.svg)](https://ziglang.org)\n[![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)\n[![Postgres](https://img.shields.io/badge/Postgres-14+-blue.svg)](https://www.postgresql.org)\n\n`pg_transfer` copies Postgres tables between databases quickly by splitting the work across multiple workers. Each worker handles a different part of the table using CTID ranges, streaming data directly.\n\n## Performance Comparison\n\nBenchmark results comparing pg_transfer with pg_dump+psql pipeline for transferring pgbench_accounts table:\n\n**pg_dump+psql command used for comparison:**\n```bash\npg_dump \"$SOURCE_DB\" -t pgbench_accounts --data-only | psql \"$DEST_DB\"\n```\n\n### 10M rows (1GB data) - pgbench with scaling factor 100\n| Tool | Real Time | User Time | Sys Time | Rows Transferred | Improvement |\n|------|-----------|-----------|----------|------------------|------------|\n| pg_transfer (4 workers) | 11.1s | 1.7s | 0.7s | 10,000,000 | - |\n| pg_dump+psql | 13.9s | 2.6s | 1.4s | 10,000,000 | +25% slower |\n\n### 20M rows (2GB data) - pgbench with scaling factor 200\n| Tool | Real Time | User Time | Sys Time | Rows Transferred | Improvement |\n|------|-----------|-----------|----------|------------------|------------|\n| pg_transfer (4 workers) | 22.8s | 3.2s | 1.4s | 20,000,000 | - |\n| pg_dump+psql | 30.1s | 4.8s | 2.8s | 20,000,000 | +32% slower |\n\n### 30M rows (3GB data) - pgbench with scaling factor 300\n| Tool | Real Time | User Time | Sys Time | Rows Transferred | Improvement |\n|------|-----------|-----------|----------|------------------|------------|\n| pg_transfer (4 workers) | 31.1s | 4.9s | 2.0s | 30,000,000 | - |\n| pg_dump+psql | 45.6s | 8.2s | 3.9s | 30,000,000 | +47% slower |\n\npg_transfer shows significantly faster real time and lower CPU usage compared to traditional pg_dump+psql pipeline. The performance advantage increases with larger tables due to better parallelization and reduced serialization overhead.\n\n## Installation\n\n### Quick Start\n```bash\n# Clone the repository\ngit clone https://github.com/abc3/pg_transfer.git\ncd pg_transfer\n\n# Build the tool\nmake build\n\n# Check possible arguments\n./zig-out/bin/pg_transfer -h\n```\n\n## Usage\n\n### Prerequisites\n\nBefore transferring data, you need to copy the table schema to the destination database:\n\n```bash\n# Copy table schema (structure) to destination database\npg_dump \"$SOURCE_DB\" -t pgbench_accounts --schema-only | psql \"$DEST_DB\"\n```\n\n**Note:** pg_transfer only transfers data, not table structure. The destination table must exist before running the transfer.\n\n### Basic Usage\n```bash\n# Simple table copy with default settings\n./zig-out/bin/pg_transfer \\\n  --source \"postgresql://user:pass@source:5432/db\" \\\n  --destination \"postgresql://user:pass@dest:5432/db\" \\\n  --table public.large_table\n\n# Copy specific table with custom workers\n./zig-out/bin/pg_transfer \\\n  --source \"postgresql://user:pass@source:5432/db\" \\\n  --destination \"postgresql://user:pass@dest:5432/db\" \\\n  --table public.massive_table \\\n  --workers 16 \\\n  --buffer 2\n```\n\n### Advanced Configuration\n```bash\n# Production-grade configuration for massive tables\n./zig-out/bin/pg_transfer \\\n  --source \"postgresql://prod:pass@prod-db:5432/production\" \\\n  --destination \"postgresql://staging:pass@staging-db:5432/staging\" \\\n  --table public.user_transactions \\\n  --workers 32 \\\n  --buffer 4\n```\n\n## Configuration Options\n\n| Flag | Type | Default | Description |\n|------|------|---------|-------------|\n| `--source` | string | Required | Source database connection string |\n| `--destination` | string | Required | Destination database connection string |\n| `--table` | string | Required | Table name to transfer |\n| `--workers` | u32 | CPU count | Number of parallel worker threads |\n| `--buffer` | u32 | 1 | Batch buffer size in MB |\n\n## Development\n\n### Setup Test Environment\n```bash\n# Setup 'public.pgbench_accounts' table on the source database\npgbench -i -s 100 postgresql://postgres:postgres@127.0.0.1:5432\n\n# Note: pg_transfer does not create schemas or tables\n# If the destination table doesn't exist, create it first:\npg_dump \"postgresql://postgres:postgres@127.0.0.1:5432/postgres\" -t pgbench_accounts --no-owner --no-privileges --data-only | psql \"postgresql://postgres:postgres@127.0.0.1:6432/postgres\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabc3%2Fpg_transfer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabc3%2Fpg_transfer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabc3%2Fpg_transfer/lists"}