https://github.com/jedrazb/cdc-kafka-setup
Docker setup for a local instance of cdc source that publishes updates to kafka
https://github.com/jedrazb/cdc-kafka-setup
cdc debezium-connector kafka mysql mysql-connector
Last synced: 8 months ago
JSON representation
Docker setup for a local instance of cdc source that publishes updates to kafka
- Host: GitHub
- URL: https://github.com/jedrazb/cdc-kafka-setup
- Owner: jedrazb
- Created: 2023-09-11T09:39:52.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2023-09-12T11:28:51.000Z (over 2 years ago)
- Last Synced: 2025-02-09T18:49:43.712Z (10 months ago)
- Topics: cdc, debezium-connector, kafka, mysql, mysql-connector
- Homepage:
- Size: 12.8 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# cdc-kafka-setup
Docker setup for a local instance of cdc source that publishes updates to kafka
### Setup
Start up containers
```bash
docker-compose up -d
```
Confirm containers are running
```bash
docker-compose ps
```
Wait for mysql to be ready, look for `/usr/sbin/mysqld: ready for connections. Version: '8.1.0'` in logs. Create mysql `users` table.
```bash
docker exec -i mysql mysql -uroot -proot mydb < create_table.sql
```
Start mysql CDC debezium connector
```bash
docker exec -i debezium curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d @/debezium-config.json
```
Ingest some data into mysql
```bash
docker exec -i mysql mysql -uroot -proot mydb < insert_data.sql
```
Verify that the data is present in mysql
```bash
docker exec -i mysql mysql -uroot -proot mydb < query_data.sql
```
Verify that the data is replicated into a kafka topic:
```bash
docker exec -it kafka kafka-topics --list --bootstrap-server kafka:9092
```
You should see `cdctest.mydb.users` present in the response.
Final verification step, list all data in the `cdctest.mydb.users` topic.
```bash
docker exec -it kafka kafka-console-consumer --topic cdctest.mydb.users --from-beginning --bootstrap-server localhost:9092
```
### How to connecto to Kafka
Kafka broker address to connect from host: `localhost:29092`
CDC topic: `cdctest.mydb.users`