https://github.com/yzevm/sync-postgresql-with-elasticsearch-example
🔄 Sync PostgreSQL to Elasticsearch via Debezium
https://github.com/yzevm/sync-postgresql-with-elasticsearch-example
debezium elasticsearch example master-slave postgresql real-time replication sync sync-postgresql
Last synced: about 2 months ago
JSON representation
🔄 Sync PostgreSQL to Elasticsearch via Debezium
- Host: GitHub
- URL: https://github.com/yzevm/sync-postgresql-with-elasticsearch-example
- Owner: yzevm
- Created: 2019-03-18T12:32:48.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2020-09-13T08:08:16.000Z (over 4 years ago)
- Last Synced: 2025-03-11T11:51:42.090Z (about 2 months ago)
- Topics: debezium, elasticsearch, example, master-slave, postgresql, real-time, replication, sync, sync-postgresql
- Language: Dockerfile
- Homepage:
- Size: 11.7 KB
- Stars: 98
- Watchers: 4
- Forks: 23
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Sync PostgreSQL with Elasticsearch via Debezium
### Schema
```
+-------------+
| |
| PostgreSQL |
| |
+------+------+
|
|
|
+---------------v------------------+
| |
| Kafka Connect |
| (Debezium, ES connectors) |
| |
+---------------+------------------+
|
|
|
|
+-------v--------+
| |
| Elasticsearch |
| |
+----------------+
```
We are using Docker Compose to deploy the following components:
* PostgreSQL
* Kafka
* ZooKeeper
* Kafka Broker
* Kafka Connect with [Debezium](http://debezium.io/) and [Elasticsearch](https://github.com/confluentinc/kafka-connect-elasticsearch) Connectors
* Elasticsearch
### Usage
```shell
docker-compose up --build
# wait until it's setup
./start.sh
```
### Testing
Check database's content
```shell
# Check contents of the PostgreSQL database:
docker-compose exec postgres bash -c 'psql -U $POSTGRES_USER $POSTGRES_DATABASE -c "SELECT * FROM users"'
# Check contents of the Elasticsearch database:
curl http://localhost:9200/users/_search?pretty
```
Create user
```shell
docker-compose exec postgres bash -c 'psql -U $POSTGRES_USER $POSTGRES_DATABASE'
test_db=# INSERT INTO users (email) VALUES ('[email protected]');
# Check contents of the Elasticsearch database:
curl http://localhost:9200/users/_search?q=id:6
```
```json
{
...
"hits": {
"total": 1,
"max_score": 1.0,
"hits": [
{
"_index": "users",
"_type": "_doc",
"_id": "6",
"_score": 1.0,
"_source": {
"id": 6,
"email": "[email protected]"
}
}
]
}
}
```
Update user
```shell
test_db=# UPDATE users SET email = '[email protected]' WHERE id = 6;
# Check contents of the Elasticsearch database:
curl http://localhost:9200/users/_search?q=id:6
```
```json
{
...
"hits": {
"total": 1,
"max_score": 1.0,
"hits": [
{
"_index": "users",
"_type": "_doc",
"_id": "6",
"_score": 1.0,
"_source": {
"id": 6,
"email": "[email protected]"
}
}
]
}
}
```
Delete user
```shell
test_db=# DELETE FROM users WHERE id = 6;
# Check contents of the Elasticsearch database:
curl http://localhost:9200/users/_search?q=id:6
```
```json
{
...
"hits": {
"total": 1,
"max_score": 1.0,
"hits": []
}
}
```