https://github.com/nits302/change_data_capture_streaming
This project demonstrates the implementation of a Change Data Capture (CDC) system using Debezium, Kafka, Postgres, and Docker.
https://github.com/nits302/change_data_capture_streaming
cdc debezium docker-compose kafka zookeeper
Last synced: 6 months ago
JSON representation
This project demonstrates the implementation of a Change Data Capture (CDC) system using Debezium, Kafka, Postgres, and Docker.
- Host: GitHub
- URL: https://github.com/nits302/change_data_capture_streaming
- Owner: nits302
- Created: 2024-12-23T07:55:13.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2024-12-23T15:23:51.000Z (10 months ago)
- Last Synced: 2025-03-29T07:14:33.016Z (6 months ago)
- Topics: cdc, debezium, docker-compose, kafka, zookeeper
- Language: Python
- Homepage:
- Size: 12.3 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Change Data Capture Streaming Project with Debezium, Kafka, Postgres, Docker
A real-time Change Data Capture (CDC) streaming project that captures database changes from PostgreSQL and streams them to Apache Kafka using Debezium.
## Overview
This project demonstrates the implementation of a Change Data Capture (CDC) system using Debezium, Kafka, Postgres, and Docker. The system generates simulated financial transactions using a Python script and inserts them into a PostgreSQL database. It is particularly useful for setting up a test environment for CDC with Debezium. The script uses the faker library to create realistic yet fictitious transaction data.
## Architecture
The project uses the following components:
- Apache Kafka
- Apache ZooKeeper
- Debezium Connect
- Debezium UI
- PostgreSQL
- Confluent Control Center
## Prerequisites
- Docker and Docker Compose
- Python 3.x
- Required Python packages (see `requirements.txt`):
- Faker
- psycopg2-binary
- python-dateutil
- six
## Setup & Installation
1. Clone the repository:
```bash
git clone https://github.com/nits302/Change-Data-Capture-Streaming.git
cd cdc-streaming
```
2. Create and activate a virtual environment:
```bash
python -m venv venv
source venv/bin/activate # Linux/Mac
.\venv\Scripts\activate # Windows
```
3. Install dependencies:
```bash
pip install -r requirements.txt
```
4. Start the infrastructure using Docker Compose:
```bash
docker-compose up -d
```
Check containers in docker desktop:
5 - Access to `control center UI` monitor Topic health, Procuder and consumer performance, Offset, Cluster health: `localhost:9021`
6 - Debezium: `localhost:8080`
## Services
The following services will be available:
| Service | URL | Description |
| ------------------------ | --------------------- | ----------------------------- |
| Kafka Broker | localhost:9092 | Apache Kafka broker |
| Confluent Control Center | http://localhost:9021 | Kafka management UI |
| Debezium Connect | http://localhost:8093 | Debezium Connect REST API |
| Debezium UI | http://localhost:8080 | Debezium management interface |
| PostgreSQL | localhost:5435 | PostgreSQL database |
## Configuration
The project uses the following default configurations (from docker-compose.yml):
- PostgreSQL:
- Database: financial_db
- Username: postgres
- Password: postgres
- Port: 5435
- Kafka:
- Bootstrap servers: localhost:9092
- External access: localhost:29092
- Internal access: broker:29092
## 🚀 Video DEMO
[Demo Project Change Data Capture Streaming](https://youtu.be/pfwxqGCbFBY)
⚡️That's all for my project, thanks for watching. If you have any question, don't hesitate inbox me.⚡️