https://github.com/thachlp/cdc-debezium-service
Change Data Capture (CDC) from MySQL using Debezium & Kafka
https://github.com/thachlp/cdc-debezium-service
changedatacapture debezium
Last synced: 3 months ago
JSON representation
Change Data Capture (CDC) from MySQL using Debezium & Kafka
- Host: GitHub
- URL: https://github.com/thachlp/cdc-debezium-service
- Owner: thachlp
- License: mit
- Created: 2024-07-12T03:01:50.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2025-01-04T04:47:15.000Z (5 months ago)
- Last Synced: 2025-01-04T05:25:01.680Z (5 months ago)
- Topics: changedatacapture, debezium
- Language: Java
- Homepage:
- Size: 560 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
### CDC Service for MySQL
### Overview
### Prerequisite
##### 1. Install
- [Docker](script/install_docker.sh)
- [Docker Compose](script/install_docker_compose.sh)
##### 2. MySQL Config
| Property | Expected Value | Verification SQL Statement |
|-------------------|----------------|---------------------------------------------|
| log_bin | ON | SHOW VARIABLES LIKE 'log_bin'; |
| binlog_format | ROW | SHOW VARIABLES LIKE 'binlog_format'; |
| binlog_row_image | FULL | SHOW VARIABLES LIKE 'binlog_row_image'; |
**User**:
- a user has `RELOAD` or `FLUSH_TABLES` privilege(s).
##### 3. Kafka Config
`auto.create.topics.enable` should be true
### How to run
##### 1. Start services
```shell
docker compose up -d
```
##### 2. Create source connector
```shell
curl -i -X POST \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
localhost:8083/connectors/ \
-d '{
"name": "source-connector",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"database.hostname": "mysql",
"database.port": "3306",
"database.user": "root",
"database.password": "123456",
"database.server.id": "184054",
"topic.prefix": "mysql",
"database.include.list": "coffee_shop",
"schema.history.internal.kafka.bootstrap.servers": "kafka:9092",
"schema.history.internal.kafka.topic": "schemahistory.coffee_shop"
}
}'
```
##### 3. Start consumer
```shell
cd cdc-debezium-consumer
./gradlew run -d
```
### Troubleshooting
##### 1. Check status of Kafka Connect service
```shell
curl -H "Accept:application/json" localhost:8083/
```
##### 2. Check available plugins
```shell
curl -s localhost:8083/connector-plugins
```
##### 3. Check list connectors registered with Kafka Connect
```shell
curl -H "Accept:application/json" localhost:8083/connectors/
```
#### Delete a connector
```shell
curl -X DELETE localhost:8083/connectors/source-connector
```
##### 4. Access the kafka container
```shell
docker exec -it cdc-debezium-consumer-kafka-1 /bin/bash
cd bin
```
##### 5. List kafka topic
```shell
kafka-topics.sh --list --bootstrap-server localhost:9092
```