https://github.com/baunz/struct-converters
kafka-connect converter for ingesting raw data into a JDBC sink
https://github.com/baunz/struct-converters
connector json json-records kafka-connect
Last synced: 5 months ago
JSON representation
kafka-connect converter for ingesting raw data into a JDBC sink
- Host: GitHub
- URL: https://github.com/baunz/struct-converters
- Owner: baunz
- License: mit
- Created: 2021-03-01T17:50:52.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2025-01-17T10:18:18.000Z (over 1 year ago)
- Last Synced: 2025-04-10T21:09:28.778Z (about 1 year ago)
- Topics: connector, json, json-records, kafka-connect
- Language: Java
- Homepage:
- Size: 28.3 KB
- Stars: 1
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# struct-converters
Convert schemaless, primitive kafka messages into ConnectData
## Why?
Most sink connectors in kafka-connect require a schema to insert the data. When dealing with topics that contain plain JSON records it's not possible to insert them them without an inline schema, as explained [here](https://www.confluent.de/blog/kafka-connect-deep-dive-converters-serialization-explained/#string-json-data)
Sometimes it is just not possible to change the JSON records to include the schema and we are okay with having the json "as-is" in the sink system. This still allows a "schema-on-read"-strategy, supported by RDBMS like MySql and Postgres with their JSON data types.
See [Further options](#further-options)
## Installation
Copy a release jar from this website into your [connect plugin path](https://docs.confluent.io/home/connect/userguide.html#installing-kconnect-plugins)
## Configuration
Given this example json
```
record-key: 456
record-value: {
"id" : "11eb50e4-e3b5-f40f-b709-36bc5ee27958",
"shopId" : 2001,
"origin" : "space",
"type" : "like",
"createDate" : "2021-01-12T13:34:16.653Z",
"payload" : {
"profileId" : "11eb50e4-e3b5-f40f-b709-36bc5ee27958",
}
}
```
and the following table created in the sink db
```
create table `super_event` (
`ID` VARCHAR(255) NOT NULL,
`VALUE` JSON,
PRIMARY KEY (ID)
) ENGINE = InnoDB
```
The following connector configuration
```
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
topics=super_event
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=com.github.baunz.kafka.connect.storage.StringAsStructConverter
connection.url=jdbc=mysql=//awesome-db/
insert.mode=UPSERT
pk.mode=record_key
pk.fields=id
```
creates a connector that fills the target table and allows the data to be queried like this (mysql example)
```
select
value->>'$.key',
value->>'$.payload.profileId'
from
super_event;
```
## Further Options
* Apply a schema to the raw JSON to via transforms (https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-json-schema/transformations/FromJson.html)
* Attach a schema with KSQL (https://rmoff.net/2020/01/22/kafka-connect-and-schemas/)