https://github.com/arenadata/kafka-postgres-connector
https://github.com/arenadata/kafka-postgres-connector
Last synced: 6 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/arenadata/kafka-postgres-connector
- Owner: arenadata
- License: apache-2.0
- Created: 2021-07-12T08:39:27.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2021-10-25T08:09:58.000Z (over 4 years ago)
- Last Synced: 2023-03-04T04:49:57.612Z (over 3 years ago)
- Language: Java
- Homepage:
- Size: 129 KB
- Stars: 2
- Watchers: 10
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# kafka-postgres-connector
A supporting service to the [Prostore main service](https://github.com/arenadata/prostore) `dtm-query-execution-core` that
communicates with the respective DBMS.
## Useful links
[Documentation (Rus)](https://arenadata.github.io/docs_prostore/getting_started/getting_started.html)
## Local deployment
### The cloning and building of kafka-postgres-connector
```shell script
#clone
git clone https://github.com/arenadata/kafka-postgres-connector ~/
# build without any tests
cd ~/kafka-postgres-connector
mvn clean install -DskipTests=true
```
### Connectors configuration
The connector configuration files `application.yml` are located in the respective folders
`kafka-postgres-writer/src/main/resources/` and `kafka-postgres-reader/src/main/resources/`.
To run connectors correctly one has to adjust respective configuration files for `kafka-postgres-writer` and `kafka-postgres-reader` to match the key values with the Prostore configuration, namely:
- `datasource: postgres: database ~ env: name`,
- `datasource: postgres: user ~ adp: datasource: user`,
- `datasource: postgres: password ~ adp: datasource: password`,
- `datasource: postgres: hosts ~ adp: datasource: host, adp: datasource: port`.
The connector services look for the configuration in the same subfolders (target) where `kafka-postgres-writer-.jar` and `kafka-postgres-reader-.jar` are executed.
So we create the respective symbolic links
```shell script
sudo ln -s ~/kafka-postgres-connector/kafka-postrges-writer/src/main/resources/application.yml ~/kafka-postgres-connector/kafka-postrges-writer/target/application.yml
sudo ln -s ~/kafka-postgres-connector/kafka-postrges-reader/src/main/resources/application.yml ~/kafka-postgres-connector/kafka-postrges-reader/target/application.yml
```
### Run services
#### Run the kafka-postgres-writer connector as a single jar
```shell script
cd ~/kafka-postgres-connector/kafka-postgres-writer/target
java -jar kafka-postgres-writer-.jar
```
#### Run the kafka-postgres-reader connector as a single jar
```shell script
cd ~/kafka-postgres-connector/kafka-postgres-reader/target
java -jar kafka-postgres-reader-.jar
```
### Available endpoints
#### kafka-postgres-writer
```shell script
http://kafka-postgres-writer-host:8096/newdata/start [POST]
http://kafka-postgres-writer-host:8096/newdata/stop [POST]
http://kafka-postgres-writer-host:8096/version [GET]
```
#### kafka-postgres-reader
```shell script
http://kafka-postgres-reader-host:8094/query [POST]
http://kafka-postgres-reader-host:8094/versions [GET]
```