https://github.com/ivpal/cdc-realtime-search
Project demonstrates realtime index updates via change data capture
https://github.com/ivpal/cdc-realtime-search
change-data-capture debezium elasticsearch kafka-connect redpanda spring-boot webflux
Last synced: 3 months ago
JSON representation
Project demonstrates realtime index updates via change data capture
- Host: GitHub
- URL: https://github.com/ivpal/cdc-realtime-search
- Owner: ivpal
- License: mit
- Created: 2023-02-25T16:53:03.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-02-25T19:02:55.000Z (over 2 years ago)
- Last Synced: 2025-03-12T21:19:34.145Z (7 months ago)
- Topics: change-data-capture, debezium, elasticsearch, kafka-connect, redpanda, spring-boot, webflux
- Language: Kotlin
- Homepage:
- Size: 117 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Change Data Capture Realtime Search
[](http://kotlinlang.org)
Users - service designed to store information about users. Changes in users database are captured via Debezium connector
and emit to Redpanda topic. Search service receives events from Redpanda and update index in Elasticsearch. Search service
also provides API for query user information from Elasticsearch.
## Setup
```shell
git clone https://github.com/ivpal/CDC-Realtime-Search.git
cd CDC-Realtime-Search/
```
For build Docker images you need Java 17 or higher:
```shell
./gradlew users:jibDockerBuild
./gradlew search:jibDockerBuild
```
# Run
```shell
docker-compose up -d
```
### Start PostgreSQL connector
```shell
curl --location --request POST 'localhost:8083/connectors/' \
--header 'Content-Type: application/json' \
--data-raw '{
"name": "users-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": "1",
"database.hostname": "users-db",
"database.port": "5432",
"database.user": "users",
"database.password": "users",
"database.dbname" : "postgres",
"database.server.id": "184054",
"topic.prefix": "users",
"database.history.kafka.bootstrap.servers": "redpanda:9092",
"database.history.kafka.topic": "schema-changes.users"
}
}'
```
## Usage
### Create user
```shell
curl --location --request POST 'localhost:8080/api/users' \
--header 'Content-Type: application/json' \
--data-raw '{
"username": "ivpal",
"firstname": "Pavel",
"lastname": "Ivanov"
}'
```
Search request:
```shell
curl --location --request GET 'localhost:8081/api/search?q=pa'
```
Response:
```json
[
{
"id": 1,
"username": "ivpal",
"firstname": "Pavel",
"lastname": "Ivanov"
}
]
```
### Update user
```shell
curl --location --request PUT 'localhost:8080/api/users/1' \
--header 'Content-Type: application/json' \
--data-raw '{
"username": "newusername",
"firstname": "Pavel",
"lastname": "Ivanov"
}'
```
Search request:
```shell
curl --location --request GET 'localhost:8081/api/search?q=pa'
```
Response:
```json
[
{
"id": 1,
"username": "newusername",
"firstname": "Pavel",
"lastname": "Ivanov"
}
]
```
### Delete user
```shell
curl --location --request DELETE 'localhost:8080/api/users/1'
```
Search request:
```shell
curl --location --request GET 'localhost:8081/api/search?q=pa'
```
Response:
```json
[]
```