{"id":23096295,"url":"https://github.com/fa3001/realtime_change_data_capture_streaming","last_synced_at":"2025-04-03T19:42:40.859Z","repository":{"id":247554441,"uuid":"797567797","full_name":"FA3001/Realtime_Change_Data_Capture_Streaming","owner":"FA3001","description":null,"archived":false,"fork":false,"pushed_at":"2024-07-09T08:45:19.000Z","size":93,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-09T07:44:24.151Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FA3001.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-08T05:08:56.000Z","updated_at":"2024-07-09T08:45:23.000Z","dependencies_parsed_at":"2024-07-09T11:19:45.665Z","dependency_job_id":null,"html_url":"https://github.com/FA3001/Realtime_Change_Data_Capture_Streaming","commit_stats":null,"previous_names":["fa3001/realtime_change_data_capture_streaming"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FA3001%2FRealtime_Change_Data_Capture_Streaming","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FA3001%2FRealtime_Change_Data_Capture_Streaming/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FA3001%2FRealtime_Change_Data_Capture_Streaming/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FA3001%2FRealtime_Change_Data_Capture_Streaming/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FA3001","download_url":"https://codeload.github.com/FA3001/Realtime_Change_Data_Capture_Streaming/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247070455,"owners_count":20878579,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-16T22:33:32.216Z","updated_at":"2025-04-03T19:42:40.828Z","avatar_url":"https://github.com/FA3001.png","language":"Python","readme":"# CDC with Debezium, Kafka, Postgres, Docker \n\n## Overview\n\nThis Python script is designed to generate simulated financial transactions and insert them into a PostgreSQL database. It's particularly useful for setting up a test environment for Change Data Capture (CDC) with Debezium. The script uses the `faker` library to create realistic, yet fictitious, transaction data and inserts it into a PostgreSQL table.\n\n## System Architecture\n![arch](https://github.com/FA3001/Realtime_Change_Data_Capture_Streaming/assets/161286586/6d3071c5-5217-44f0-a54e-e5f1a7756e06)\n\n\n## Prerequisites\n\nBefore running this script, ensure you have the following installed:\n- Python 3.9+\n- `psycopg2` library for Python\n- `faker` library for Python\n- PostgreSQL server running locally or accessible remotely\n- Docker and Docker Compose installed on your machine.\n- Basic understanding of Docker, Kafka, and Postgres.\n\n## Installation\n\n1. **Install Required Python Libraries:**\n\n   You can install the required libraries using pip:\n\n   ```bash\n   pip install psycopg2-binary faker\n   ```\n\n## Services in the Compose File\n\n- **Zookeeper:** A centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.\n- **Kafka Broker:** A distributed streaming platform that is used here for handling real-time data feeds.\n- **Confluent Control Center:** A web-based tool for managing and monitoring Apache Kafka.\n- **Debezium:** An open-source distributed platform for change data capture.\n- **Debezium UI:** A user interface for managing and monitoring Debezium connectors.\n- **Postgres:** An open-source relational database.\n\n## Getting Started\n\n1. **Clone the Repository:**\n   Ensure you have this Docker Compose file in your local system. If it's part of a repository, clone the repository to your local machine.\n\n2. **Navigate to the Directory:**\n   Open a terminal and navigate to the directory containing the Docker Compose file.\n\n3. **Run Docker Compose:**\n   Execute the following command to start all services defined in the Docker Compose file:\n\n   ```bash\n   docker-compose up -d\n   ```\n\n   This command will download the necessary Docker images, create containers, and start the services in detached mode.\n\n4. **Verify the Services:**\n   Check if all the services are up and running:\n\n   ```bash\n   docker-compose ps\n   ```\n\n   You should see all services listed as 'running'.\n\n5. **Accessing the Services:**\n   - Kafka Control Center is accessible at `http://localhost:9021`.\n   - Debezium UI is accessible at `http://localhost:8080`.\n   - Postgres is accessible on the default port `5432`.\n\n6. **Shutting Down:**\n   To stop and remove the containers, networks, and volumes, run:\n\n   ```bash\n   docker-compose down\n   ```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffa3001%2Frealtime_change_data_capture_streaming","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffa3001%2Frealtime_change_data_capture_streaming","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffa3001%2Frealtime_change_data_capture_streaming/lists"}