{"id":34833035,"url":"https://github.com/tr1sm0s1n/cdc-cascade","last_synced_at":"2026-04-04T16:36:17.282Z","repository":{"id":319064194,"uuid":"1070560835","full_name":"tr1sm0s1n/cdc-cascade","owner":"tr1sm0s1n","description":"A production-ready reference implementation for Change Data Capture (CDC) using Debezium, showcasing real-time data streaming from PostgreSQL to consumers via Kafka.","archived":false,"fork":false,"pushed_at":"2025-12-27T17:15:33.000Z","size":60,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-29T15:30:37.184Z","etag":null,"topics":["cache-invalidation","cdc","concurrency","debezium","docker","fiber","golang","kafka","postgresql","queue","redis"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tr1sm0s1n.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-06T05:51:20.000Z","updated_at":"2025-12-27T17:15:37.000Z","dependencies_parsed_at":"2025-10-18T00:14:44.162Z","dependency_job_id":"2f97d2cc-2be7-49a2-a556-4c66d04900ee","html_url":"https://github.com/tr1sm0s1n/cdc-cascade","commit_stats":null,"previous_names":["tr1sm0s1n/cdc-cascade"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tr1sm0s1n/cdc-cascade","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tr1sm0s1n%2Fcdc-cascade","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tr1sm0s1n%2Fcdc-cascade/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tr1sm0s1n%2Fcdc-cascade/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tr1sm0s1n%2Fcdc-cascade/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tr1sm0s1n","download_url":"https://codeload.github.com/tr1sm0s1n/cdc-cascade/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tr1sm0s1n%2Fcdc-cascade/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31405705,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T10:20:44.708Z","status":"ssl_error","status_checked_at":"2026-04-04T10:20:06.846Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache-invalidation","cdc","concurrency","debezium","docker","fiber","golang","kafka","postgresql","queue","redis"],"created_at":"2025-12-25T15:57:32.098Z","updated_at":"2026-04-04T16:36:17.272Z","avatar_url":"https://github.com/tr1sm0s1n.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CDC Cascade\n\nA production-ready reference implementation for Change Data Capture (CDC) using Debezium, showcasing real-time data streaming from PostgreSQL to consumers via Kafka.\n\n## What's This?\n\nThis repository demonstrates a complete CDC pipeline with a practical architecture that you can use as a blueprint for your own projects. It captures database changes in real-time and streams them to downstream consumers—perfect for microservices synchronization, audit logging, cache invalidation, or building event-driven systems.\n\n**Use this when you need to:**\n\n- Sync data between microservices without tight coupling\n- Build real-time analytics pipelines\n- Maintain materialized views or caches\n- Implement audit logs or event sourcing\n- Learn CDC patterns hands-on\n\n## Architecture\n\nView the [complete sequence flow diagram](/sequence-flows.mmd) showing all interaction patterns: cache miss, cache hit, CDC invalidation, and cache rebuild.\n\n---\n\n### Components\n\n- **PostgreSQL**: Source of truth for all data\n- **Debezium**: CDC engine capturing PostgreSQL WAL changes\n- **Kafka**: Event streaming backbone for change events\n- **Go Application** (runs two goroutines):\n  - **Fiber REST API**: Handles client requests with cache-aside logic\n  - **CDC Consumer**: Listens to Kafka and invalidates Redis on changes\n- **Redis**: Cache layer with automatic CDC-driven invalidation\n\n## Project Structure\n\n```\n.\n├── config/                     # Database \u0026 cache configuration\n├── controllers/                # HTTP request handlers\n├── models/                     # Domain models \u0026 schemas\n├── queue/                      # CDC consumer \u0026 event processing\n│   ├── cdc.go                  # Event handlers \u0026 business logic\n│   └── runner.go               # Consumer lifecycle management\n├── scripts/                    # Setup \u0026 initialization\n│   ├── debezium-setup.sh       # Connector registration\n│   └── init.sql                # Database schema \u0026 seed data\n├── docker-compose.yml          # Service orchestration\n├── Dockerfile                  # Multi-stage build for API\n├── main.go                     # Entry point (API + CDC consumer)\n└── Makefile                    # Build, run, and management commands\n```\n\n## Configuration\n\n### Environment Variables\n\n```sh\n# Postgres\nDB_HOST=cdc-cascade-postgres\nDB_USER=admin\nDB_NAME=cdc-cascade-db\nDB_PASS=adminpw\nDB_PORT=5432\n\n# Redis\nREDIS_HOST=cdc-cascade-redis\nREDIS_PORT=6379\nREDIS_PASS=rootpw\n\n# API\nAPI_HOST=cdc-cascade-api\nAPI_PORT=8080\n\n# Kafka\nKAFKA_HOST=cdc-cascade-kafka\nKAFKA_CONSUMER_GROUP=cdc-cascade-kafka-consumers\nKAFKA_CDC_TOPIC=cdc-cascade-postgres.public.sinners\nKAFKA_BROKER_PORT=9092\nKAFKA_CONTROLLER_PORT=9093\n\n# Debezium\nDEBEZIUM_HOST=cdc-cascade-debezium\nDEBEZIUM_PORT=8083\n```\n\n## Quick Start\n\n### Prerequisites\n\n- Docker \u0026 Docker Compose\n- Make\n- Go 1.25+ (for local development)\n- 4GB RAM recommended\n\n### Run the Stack\n\n```bash\n# Clone the repository\ngit clone https://github.com/tr1sm0s1n/cdc-cascade.git\ncd cdc-cascade\n\n# Start all services\nmake up\n\n# Load .env if necessary\nsource .env\n\n# View logs\ndocker logs -f $API_HOST\n```\n\n### Debezium Connector\n\nTo register connector:\n\n```bash\nmake connect\n```\n\n### Verify It's Working\n\n**1. Enter database:**\n\n```bash\nmake enter\n```\n\n**1. Insert new data:**\n\n```sql\nINSERT INTO sinners (code, \"name\", class, libram, tendency)\nVALUES\n    (5, 'Augustus', 'S', 'War', 'Reticle');\n```\n\n**2. Check Kafka for CDC events:**\n\n```bash\ndocker exec -it $KAFKA_HOST /opt/kafka/bin/kafka-console-consumer.sh \\\n  --topic $DB_HOST.public.sinners \\\n  --bootstrap-server localhost:$KAFKA_BROKER_PORT \\\n  --from-beginning\n```\n\n**3. Query the API:**\n\n```bash\ncurl http://127.0.0.1:$API_PORT/api/v1/sinners/read/5\n```\n\n**4. Verify Redis cache:**\n\n```bash\ndocker exec -it $REDIS_HOST redis-cli GET 5\n```\n\n## Learning Path\n\n**For Beginners:**\n\n1. Start the stack and observe logs\n2. Make database changes, watch Kafka events\n3. Trace how events flow to Redis\n4. Experiment with the API endpoints\n\n**For Advanced Users:**\n\n1. Add custom CDC event handlers\n2. Implement transformation logic\n3. Add new tables to the CDC pipeline\n4. Integrate with your own services\n\n## Troubleshooting\n\n**Debezium not capturing changes?**\n\n- Verify PostgreSQL has `wal_level = logical`\n- Check table has PRIMARY KEY\n- Ensure table is in `table.include.list`\n\n**Kafka consumer lag?**\n\n- Check consumer group: `kafka-consumer-groups.sh --bootstrap-server localhost:$KAFKA_BROKER_PORT --describe --group $KAFKA_CONSUMER_GROUP`\n- Increase consumer parallelism if needed\n\n**Redis not updating?**\n\n- Verify CDC consumer is running in logs\n- Check Redis connectivity from API container\n\n## Resources\n\n- [Debezium Documentation](https://debezium.io/documentation/)\n- [Kafka Connect Deep Dive](https://kafka.apache.org/documentation/#connect)\n- [PostgreSQL Logical Replication](https://www.postgresql.org/docs/current/logical-replication.html)\n- [Fiber Framework](https://docs.gofiber.io/)\n- [Redis with Go](https://redis.io/docs/latest/integrate/go-redis/)\n\n---\n\n**Built with ❤️ for developers learning CDC patterns**\n\n_Questions? Open an issue or start a discussion!_\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftr1sm0s1n%2Fcdc-cascade","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftr1sm0s1n%2Fcdc-cascade","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftr1sm0s1n%2Fcdc-cascade/lists"}