{"id":43407308,"url":"https://github.com/relaxe111/cdc-pipeline-generator","last_synced_at":"2026-05-10T09:09:20.609Z","repository":{"id":335744732,"uuid":"1146879948","full_name":"Relaxe111/cdc-pipeline-generator","owner":"Relaxe111","description":"Reusable library for generating Redpanda Connect CDC pipelines.","archived":false,"fork":false,"pushed_at":"2026-02-21T04:27:56.000Z","size":1125,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-02-21T11:44:36.136Z","etag":null,"topics":["cdc-pipeline-console-generator","py-script","redpanda","redpanda-connect"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Relaxe111.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-31T20:39:07.000Z","updated_at":"2026-02-21T04:27:59.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Relaxe111/cdc-pipeline-generator","commit_stats":null,"previous_names":["relaxe111/cdc-pipeline-generator"],"tags_count":74,"template":false,"template_full_name":null,"purl":"pkg:github/Relaxe111/cdc-pipeline-generator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Relaxe111%2Fcdc-pipeline-generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Relaxe111%2Fcdc-pipeline-generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Relaxe111%2Fcdc-pipeline-generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Relaxe111%2Fcdc-pipeline-generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Relaxe111","download_url":"https://codeload.github.com/Relaxe111/cdc-pipeline-generator/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Relaxe111%2Fcdc-pipeline-generator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29800909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-24T19:56:48.334Z","status":"ssl_error","status_checked_at":"2026-02-24T19:55:43.372Z","response_time":75,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdc-pipeline-console-generator","py-script","redpanda","redpanda-connect"],"created_at":"2026-02-02T16:09:35.836Z","updated_at":"2026-05-10T09:09:20.599Z","avatar_url":"https://github.com/Relaxe111.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CDC Pipeline Generator\n\n**Generate pipeline configurations for Change Data Capture (CDC) workflows.**\n\nA CLI-first tool that reads YAML service definitions and produces streaming pipeline configurations, SQL migrations, and deployment artifacts. Supports **db-per-tenant** and **db-shared** multi-tenancy patterns with configurable data transport backends.\n\n---\n\n## Architecture\n\nThe generator sits at the centre of a CDC pipeline — it reads source database schemas, produces sink table definitions and pipeline configurations, and renders the runtime artifacts consumed by the streaming layer.\n\n### Data Transport Options\n\nCDC data can be moved from source to sink through one of two paths:\n\n| Path | Transport | Typical Use |\n|------|-----------|-------------|\n| **Streaming** | Redpanda / Kafka | High-throughput, low-latency CDC with exactly-once semantics |\n| **FDW** | PostgreSQL Foreign Data Wrappers | Direct MSSQL→PG pull without an external message broker |\n\n#### Streaming (Redpanda / Kafka)\n\nSource change events are captured, streamed through a message broker, and consumed by sink processors that write to the target PostgreSQL database.\n\n```text\nMSSQL → CDC capture → Redpanda/Kafka → Bento sink → PostgreSQL\n```\n\n#### FDW (Foreign Data Wrapper)\n\nThe generator can produce configurations that use PostgreSQL Foreign Data Wrappers (`tds_fdw`) to pull data directly from MSSQL into staging tables, followed by merge procedures that apply changes to the target tables.\n\n```text\nMSSQL ← tds_fdw ← PostgreSQL (staging → merge → target)\n```\n\n#### Native PostgreSQL-to-PostgreSQL\n\nFor PostgreSQL source databases, native logical replication or polling-based CDC can be used without an external broker.\n\n```text\nPostgreSQL → native CDC polling → PostgreSQL target\n```\n\nAll three paths are configuration-driven — the generator produces the correct pipeline YAML, SQL migrations, and runtime helpers based on the chosen transport and source database type.\n\n---\n\n## Installation\n\n### Option A: Docker (zero host dependencies)\n\n```bash\ndocker pull asmacarma/cdc-pipeline-generator:latest\n```\n\n### Option B: Host install via pip\n\n```bash\n# Editable install for active development\npip install -e .\n\n# Or install directly from the repository\npip install .\n```\n\nAfter host install, the `cdc` command is available on your shell PATH.\n\n---\n\n## Quick Start\n\n### 1. Create a project and initialize\n\n```bash\nmkdir my-cdc-project \u0026\u0026 cd my-cdc-project\ncdc init\n```\n\nThis creates the project structure: `source-groups.yaml`, `services/`, `pipelines/`, directories.\n\n### 2. Scaffold a server group\n\n```bash\n# db-per-tenant (one database per customer)\ncdc scaffold my-group \\\n  --pattern db-per-tenant \\\n  --source-type mssql \\\n  --extraction-pattern \"^myapp_(?P\u003ccustomer\u003e[^_]+)$\"\n\n# db-shared (single database, multi-tenant)\ncdc scaffold my-group \\\n  --pattern db-shared \\\n  --source-type postgres \\\n  --extraction-pattern \"^myapp_(?P\u003cservice\u003e[^_]+)_(?P\u003cenv\u003e(dev|stage|prod))$\" \\\n  --environment-aware\n```\n\n### 3. Configure services and tables\n\n```bash\n# Create a service\ncdc manage-services config --create-service my-service\n\n# Add source tables\ncdc manage-services config --service my-service --add-source-table dbo.Users --primary-key id\ncdc manage-services config --service my-service --add-source-table dbo.Orders --primary-key order_id\n\n# Inspect and save source schemas\ncdc manage-services config --service my-service --inspect --all --save\n```\n\n### 4. Manage schemas and migrations\n\n```bash\n# Generate DDL migrations for the sink database\ncdc manage-migrations generate\n\n# Review changes\ncdc manage-migrations diff\n\n# Apply migrations\ncdc manage-migrations apply\n```\n\n### 5. Generate pipeline configurations\n\n```bash\n# Generate for a single service\ncdc generate --service my-service --environment dev\n\n# Generate for all services\ncdc generate --all --environment dev\n```\n\n---\n\n## Multi-Tenancy Patterns\n\n### db-per-tenant\n\nEach customer has a dedicated source database. The generator creates one source+sink pipeline per customer database.\n\n```\nExtraction pattern: ^myapp_(?P\u003ccustomer\u003e[^_]+)$\nMatches: myapp_customer_a, myapp_customer_b\n```\n\n### db-shared\n\nAll customers share a single database, differentiated by a column (e.g. `customer_id`) or schema. Requires `--environment-aware`.\n\n```\nExtraction pattern: ^myapp_(?P\u003cservice\u003e[^_]+)_(?P\u003cenv\u003e(dev|stage|prod))$\nMatches: myapp_users_dev, myapp_users_prod\n```\n\n---\n\n## Command Reference\n\n| Command | Description |\n| ------- | ----------- |\n| `cdc init` | Initialize a new CDC project |\n| `cdc scaffold \u003cname\u003e` | Scaffold a server group with database services |\n| `cdc manage-services config` | Create, list, inspect services and tables |\n| `cdc manage-services config --inspect-sink` | Inspect and save target sink schemas |\n| `cdc manage-migrations generate` | Generate PostgreSQL DDL migrations |\n| `cdc manage-migrations diff` | Show pending schema changes |\n| `cdc manage-migrations apply` | Apply migrations to target database |\n| `cdc generate` | Generate pipeline YAML configurations |\n| `cdc manage-source-groups` | Manage source database groups |\n| `cdc manage-sink-groups` | Manage sink/target groups |\n| `cdc validate` | Validate all configurations |\n\n---\n\n## Project Structure\n\n```text\ncdc-pipeline-generator/\n├── cdc_generator/           # Core library\n│   ├── cli/                # Click command groups\n│   ├── core/               # Pipeline generation, migration engine\n│   ├── helpers/            # Database, FDW, MSSQL utilities\n│   ├── service-schemas/    # YAML schema definitions and type adapters\n│   ├── templates/          # Jinja2 pipeline templates\n│   └── validators/         # Configuration and schema validation\n├── tests/                   # Test suite\n├── _docs/                   # Architecture, getting started, CLI reference\n├── examples/                # db-per-tenant and db-shared reference implementations\n├── setup.py / pyproject.toml  # Package metadata\n└── Dockerfile               # Docker runtime image\n```\n\n---\n\n## Development\n\nSee `_docs/getting-started/` for setup instructions, `_docs/architecture/` for design decisions, and `_docs/cli/` for the full CLI command reference.\n\nThe CDC CLI runs directly on the host. Install once and use `cdc` from any directory.\n\n- ✅ `cdc` command available everywhere on your host\n- ✅ Access to source and target databases\n- ✅ Fish shell with auto-completions (reload with `cdc reload-cdc-autocompletions`)\n- ✅ Git and SSH keys available\n\nOptionally, a dev container is available if you prefer an isolated environment:\n```bash\ndocker compose exec dev fish\n```\n\n---\n\n## 📁 Project Structure\n\n---\n\n## 📁 Project Structure\n\nAfter running `cdc scaffold`, your project will have:\n\n```\nmy-cdc-project/\n├── docker-compose.yml           # Optional infrastructure (databases, streaming)\n├── Dockerfile.dev               # Optional dev container image\n├── .env.example                 # Environment variables template\n├── .env                         # Your credentials (git-ignored)\n├── .gitignore                   # Git ignore rules\n├── source-groups.yaml           # Server group config (generated by cdc)\n├── README.md                    # Quick start guide\n├── services/                    # Service definitions (generated by cdc)\n│   └── my-service.yaml\n├── pipelines/                   # Pipeline templates + generated YAML\n│   ├── templates/               # source-pipeline.yaml, sink-pipeline.yaml\n│   └── generated/\n│       ├── sources/\n│       └── sinks/\n└── generated/                   # Generated non-pipeline output (git-ignored)\n  ├── schemas/                 # PostgreSQL schemas\n  └── pg-migrations/           # PostgreSQL migrations\n```\n\n---\n\n## 🔧 Advanced Usage\n\n### Using as Python Library\n\n```python\nfrom cdc_generator.core.pipeline_generator import generate_pipelines\n\n# Generate pipelines programmatically\ngenerate_pipelines(\n  service='my-service',\n  environment='dev',\n  output_dir='./pipelines/generated'\n)\n```\n\n### Custom Pipeline Templates\n\nPlace custom Jinja2 templates in `pipelines/templates/`:\n\n```yaml\n# pipelines/templates/source-pipeline.yaml\ninput:\n  mssql_cdc:\n    dsn: \"{{ dsn }}\"\n    tables: {{ tables | tojson }}\n    # Your custom configuration\n```\n\n### Environment-Specific Configuration\n\nUse environment variables in source-groups.yaml:\n\n```yaml\nserver:\n  host: ${MSSQL_HOST}        # Replaced at runtime\n  port: ${MSSQL_PORT}\n  user: ${MSSQL_USER}\n  password: ${MSSQL_PASSWORD}\n```\n\n### SQL-Based Source Custom Keys (Source + Sink)\n\nUse custom keys to compute per-database values during `--update` and write them\ninto each source environment entry (for example `customer_id`).\n\n```bash\n# Source groups: persist SQL custom key definition\ncdc manage-source-groups \\\n  --add-source-custom-key customer_id \\\n  --custom-key-value \"SELECT customer_id FROM dbo.settings\" \\\n  --custom-key-exec-type sql\n\n# Run update to execute the SQL per discovered database\ncdc manage-source-groups --update\n```\n\n```bash\n# Sink groups: same custom key model\ncdc manage-sink-groups \\\n  --sink-group sink_analytics \\\n  --add-source-custom-key customer_id \\\n  --custom-key-value \"SELECT customer_id FROM public.settings\" \\\n  --custom-key-exec-type sql\n\n# Run sink update to execute SQL per discovered sink database\ncdc manage-sink-groups --update --sink-group sink_analytics\n```\n\nGenerated shape (simplified):\n\n```yaml\nsources:\n  directory:\n    schemas: [public]\n    nonprod:\n      server: default\n      database: directory_db\n      table_count: 42\n      customer_id: cust-001\n```\n\nIf a key returns no value for a specific server/database, the update continues and\nprints a warning with that server/database context.\n\n---\n\n## 🤝 Contributing\n\n### For Library Contributors\n\nIf you want to contribute to the cdc-pipeline-generator library itself:\n\n```bash\n# Clone repository\ngit clone https://github.com/Relaxe111/cdc-pipeline-generator.git\ncd cdc-pipeline-generator\n\n# Install in editable mode with dev dependencies\npip install -e \".[dev]\"\n\n# Run tests\npytest\n\n# Format code\nblack .\nruff check .\n```\n\n### For Users\n\nIf you're using the library in your project, just install from PyPI as shown in [Installation](#-installation).\n\n---\n\n## 📚 Resources\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frelaxe111%2Fcdc-pipeline-generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frelaxe111%2Fcdc-pipeline-generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frelaxe111%2Fcdc-pipeline-generator/lists"}