{"id":51386035,"url":"https://github.com/kaskol10/cnpg-migrator","last_synced_at":"2026-07-03T20:33:38.045Z","repository":{"id":368215042,"uuid":"1280091528","full_name":"kaskol10/cnpg-migrator","owner":"kaskol10","description":"Web UI and API to migrate PostgreSQL (e.g. AWS RDS) to CloudNativePG on Kubernetes using dump \u0026 restore Jobs.","archived":false,"fork":false,"pushed_at":"2026-06-29T14:32:46.000Z","size":439,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-29T15:24:54.482Z","etag":null,"topics":["cloudnativepg","cnpg","helm","kubernetes","migration","postgresql","rds"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kaskol10.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-25T09:03:14.000Z","updated_at":"2026-06-29T14:35:06.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/kaskol10/cnpg-migrator","commit_stats":null,"previous_names":["kaskol10/cnpg-migrator"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/kaskol10/cnpg-migrator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaskol10%2Fcnpg-migrator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaskol10%2Fcnpg-migrator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaskol10%2Fcnpg-migrator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaskol10%2Fcnpg-migrator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kaskol10","download_url":"https://codeload.github.com/kaskol10/cnpg-migrator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaskol10%2Fcnpg-migrator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35101093,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-03T02:00:05.635Z","response_time":110,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudnativepg","cnpg","helm","kubernetes","migration","postgresql","rds"],"created_at":"2026-07-03T20:33:37.545Z","updated_at":"2026-07-03T20:33:38.037Z","avatar_url":"https://github.com/kaskol10.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CNPG Migrator\n\n[![CI](https://github.com/kaskol10/cnpg-migrator/actions/workflows/ci.yml/badge.svg)](https://github.com/kaskol10/cnpg-migrator/actions/workflows/ci.yml)\n[![Release](https://github.com/kaskol10/cnpg-migrator/actions/workflows/release.yml/badge.svg)](https://github.com/kaskol10/cnpg-migrator/actions/workflows/release.yml)\n\nA web-based tool to migrate PostgreSQL databases to **CloudNativePG (CNPG)** clusters running in Kubernetes.\n\nThe primary use case is moving off **AWS RDS** (or any external PostgreSQL) onto CNPG using a **dump \u0026 restore** workflow orchestrated by ephemeral Kubernetes Jobs.\n\n## Screenshots\n\n**New migration** — configure source (RDS) and target (CNPG) endpoints:\n\n![New migration form](screenshots/new-migration-form.png)\n\n**Options** — dump format, parallelism, PVC size, and restore mode:\n\n![Migration options](screenshots/migration-options.png)\n\n**Migration status** — progress through dump, restore, and completion:\n\n![Completed migration](screenshots/migration-completed.png)\n\n**Verification \u0026 logs** — per-database size comparison and job output:\n\n![Verification results and logs](screenshots/verification-and-logs.png)\n\n## Why migrate to CloudNativePG?\n\nRunning PostgreSQL on RDS works, but it sits **outside** your Kubernetes control plane:\n\n- **Cost** — RDS bills for the instance, storage, I/O, backups, and cross-AZ traffic. For many workloads, Postgres on the nodes you already run (via CNPG) is significantly cheaper at comparable reliability.\n- **Kubernetes-native operations** — CNPG is an operator: clusters, failover, backups, and restores are CRDs and Jobs you manage like any other app — not a separate AWS console workflow.\n- **Backup \u0026 restore** — CNPG integrates with object storage (S3, GCS, Azure Blob). Scheduled backups, PITR, and restore into a new cluster are first-class operator features.\n- **Portability** — Same pattern on any cloud or on-prem. You are not locked into one vendor's managed database pricing and networking model.\n\nRDS does **not** support bootstrapping CNPG via physical replication. The practical path is logical migration: `pg_dump` → `pg_restore`. This tool automates that inside Kubernetes so migrations are repeatable, observable, and aligned with how you already run CNPG.\n\n## Architecture\n\n```mermaid\nflowchart TB\n  subgraph users [Operators]\n    UI[React UI]\n  end\n\n  subgraph control [Control plane]\n    API[Go API server]\n    Store[(In-memory store)]\n  end\n\n  subgraph k8s [Kubernetes cluster]\n    API --\u003e|create PVC + Job| K8sAPI[K8s API]\n    K8sAPI --\u003e PVC[(PVC /dump)]\n    K8sAPI --\u003e Job[Migration Job]\n\n    subgraph job [Migration Job pod]\n      Dump[Init: pg_dump]\n      Restore[Main: pg_restore]\n      Dump --\u003e|writes dump| PVC\n      PVC --\u003e|reads dump| Restore\n    end\n\n    Restore --\u003e CNPG[(CNPG primary)]\n    Verify[Verify Job]\n    Restore -.-\u003e|on success| Verify\n  end\n\n  subgraph external [Source]\n    Source[(PostgreSQL\u003cbr/\u003eRDS or any host)]\n  end\n\n  UI --\u003e API\n  API --\u003e Store\n  Dump --\u003e|TCP 5432| Source\n  Restore --\u003e|in-cluster| CNPG\n  Verify --\u003e Source\n  Verify --\u003e CNPG\n```\n\n### Migration flow\n\n1. Operator submits source (PostgreSQL), target (CNPG), and options via the UI.\n2. API creates a **PVC** and a **Job** with two containers:\n   - **Init container** (`postgres:\u003csource_version\u003e`) runs `pg_dump`\n   - **Main container** (`postgres:\u003cmax(source,target)\u003e`) runs `pg_restore`\n3. Backend polls job status and pod logs; the UI shows progress and verification results.\n\n## Features\n\n- Web UI for migration configuration and status\n- Version-aware `pg_dump` / `pg_restore` client images\n- Single-database or all-databases migration\n- Post-migration verification (database presence and size comparison)\n- Optional ownership and grant preservation (with role migration from source)\n- Skip extension restore for CNPG-managed extensions (default on)\n- Helm chart for deployment\n\n## Limitations\n\n- **Logical migration only** — requires a maintenance window; not zero-downtime replication.\n- **Ownership** — by default objects are restored as the target user (`--no-owner --no-acl`). Enable **Preserve ownership and grants** in the UI to keep source owners and ACLs; use **Migrate roles from source** to create matching roles on CNPG first (RDS system roles are excluded). Role passwords are not copied (`--no-role-passwords`).\n- **Extensions** — CNPG often manages extensions via the Cluster `databases[].extensions` spec. **Skip extensions on restore** (default on) omits `CREATE EXTENSION` from `pg_restore` so the operator-provisioned extensions are used instead. The restore user is not a superuser on CNPG.\n- **Version downgrades** may fail (e.g. PG16 → PG15).\n- **No built-in authentication** on the UI — do not expose publicly without an auth layer (OAuth2 proxy, VPN, etc.).\n- **Credentials** are passed to Job pods as environment variables (Kubernetes Secrets integration is on the roadmap).\n\n## Prerequisites\n\n- Go 1.22+\n- Node.js 20+\n- Kubernetes cluster with the [CloudNativePG operator](https://cloudnative-pg.io/)\n- Helm 3 (for deployment)\n- Network path from migration pods to source PostgreSQL and CNPG\n- A `StorageClass` supporting `ReadWriteOnce` PVCs\n\n## Quick start (local development)\n\n```bash\nmake tidy\ncd frontend \u0026\u0026 npm install\n\n# Terminal 1 — API (uses ~/.kube/config)\nexport NAMESPACE=cnpg-migrator\nmake backend\n\n# Terminal 2 — UI dev server\nmake frontend\n```\n\nOpen http://localhost:5173\n\n## Deploy with Helm\n\nImages and charts are published to [GitHub Container Registry](https://github.com/kaskol10/cnpg-migrator/pkgs/container/cnpg-migrator) on every push to `main`. Tag a release with `v0.1.0` to publish semver versions.\n\n### From GHCR (recommended)\n\n```bash\nhelm upgrade --install cnpg-migrator oci://ghcr.io/kaskol10/charts/cnpg-migrator \\\n  --version 0.1.0 \\\n  --namespace cnpg-migrator \\\n  --create-namespace\n```\n\nThe chart defaults to `ghcr.io/kaskol10/cnpg-migrator` with `image.tag` matching the chart `appVersion`. For `main` builds, use the chart version shown in the [Release workflow](https://github.com/kaskol10/cnpg-migrator/actions/workflows/release.yml) summary (e.g. `0.1.0+abc1234`).\n\n### From source\n\n```bash\nhelm upgrade --install cnpg-migrator k8s/helm/cnpg-migrator \\\n  --namespace cnpg-migrator \\\n  --create-namespace \\\n  --set image.repository=ghcr.io/kaskol10/cnpg-migrator \\\n  --set image.tag=latest\n```\n\nPort-forward to access the UI:\n\n```bash\nkubectl port-forward -n cnpg-migrator svc/cnpg-migrator 8080:80\n```\n\nSee [k8s/README.md](k8s/README.md) for ingress, job scheduling, and values reference.\n\n### CNPG target host\n\nRestore defaults to `\u003ccluster\u003e-rw.\u003cnamespace\u003e.svc.cluster.local`. The **cluster name must match the CNPG Cluster resource name**.\n\nExample: if `kubectl get svc -n backstage` shows `backstage-postgresql-rw`:\n\n- **CNPG Cluster Name**: `backstage-postgresql`\n- **Namespace**: `backstage`\n\nOr set **Host** explicitly to `backstage-postgresql-rw.backstage.svc.cluster.local`.\n\n## Configuration\n\n| Environment Variable     | Default           | Description |\n|--------------------------|-------------------|-------------|\n| `ADDR`                   | `:8080`           | HTTP listen address |\n| `NAMESPACE`              | `cnpg-migrator`   | Namespace for migration Jobs and PVCs |\n| `IN_CLUSTER`             | `false`           | Use in-cluster kubeconfig |\n| `POSTGRES_VERSIONS`      | `13,14,15,16,17`  | Major versions in the UI |\n| `POSTGRES_IMAGE_PREFIX`  | `postgres:`       | Image prefix (`postgres:16`) |\n| `POSTGRES_IMAGES`        | —                 | Override: `16=postgres:16` |\n| `DEFAULT_SOURCE_VERSION` | latest in list    | Default source version |\n| `DEFAULT_TARGET_VERSION` | latest in list    | Default target version |\n| `DEFAULT_STORAGE_SIZE`   | `50Gi`            | Default PVC size |\n| `NODE_SELECTOR`          | —                 | Job pod node selector (`key=value,...`) |\n| `TOLERATIONS`            | —                 | Job tolerations (`key=value:NoSchedule;...`) |\n| `POLL_INTERVAL_SEC`      | `5`               | Job polling interval |\n\nThe `pg_restore` **client** uses the newer of source and target versions. Downgrading server versions may still fail if the dump uses newer features.\n\n## API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| GET | `/health` | Health check |\n| GET | `/api/v1/config` | Available PostgreSQL versions |\n| GET | `/api/v1/migrations` | List migrations |\n| POST | `/api/v1/migrations` | Create migration |\n| GET | `/api/v1/migrations/{id}` | Get status |\n| GET | `/api/v1/migrations/{id}/logs` | Get logs |\n| POST | `/api/v1/migrations/{id}/verify` | Run verification |\n| GET | `/api/v1/migrations/{id}/verification` | Verification results |\n| POST | `/api/v1/migrations/{id}/cancel` | Cancel migration |\n| DELETE | `/api/v1/migrations/{id}/resources` | Delete job and PVC |\n\n## Build\n\nThe Docker image is published for **linux/amd64** and **linux/arm64**:\n\n```bash\nmake docker\nmake docker-push IMAGE=ghcr.io/kaskol10/cnpg-migrator:0.1.0\n```\n\nOr cut a release tag — CI publishes automatically:\n\n```bash\ngit tag v0.1.0 \u0026\u0026 git push origin v0.1.0\n```\n\n## Project structure\n\n```\ncnpg-migrator/\n├── backend/                 # Go API server\n├── frontend/                # React UI\n├── k8s/\n│   ├── helm/cnpg-migrator/  # Helm chart\n│   ├── cnpg-cluster/        # Example CNPG target cluster\n│   └── README.md\n├── .github/workflows/\n│   ├── ci.yml               # PR checks\n│   └── release.yml          # Publish image + chart to GHCR\n├── Dockerfile\n└── Makefile\n```\n\n## Roadmap\n\n- [ ] Persist migrations (PostgreSQL or etcd)\n- [ ] Kubernetes Secrets for credentials\n- [ ] Pre-migration connectivity checks\n- [ ] WebSocket log streaming\n- [x] Post-migration verification\n- [ ] Row-level checksum verification\n- [ ] Generic PostgreSQL source presets (beyond RDS)\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaskol10%2Fcnpg-migrator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkaskol10%2Fcnpg-migrator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaskol10%2Fcnpg-migrator/lists"}