https://github.com/redhatinsights/expandjsonsmt
Kafka Connect SMT to expand JSON field
https://github.com/redhatinsights/expandjsonsmt
Last synced: about 2 months ago
JSON representation
Kafka Connect SMT to expand JSON field
- Host: GitHub
- URL: https://github.com/redhatinsights/expandjsonsmt
- Owner: RedHatInsights
- License: apache-2.0
- Created: 2019-08-29T13:31:57.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2023-11-30T12:27:52.000Z (over 1 year ago)
- Last Synced: 2025-03-27T18:52:37.003Z (2 months ago)
- Language: Java
- Size: 71.3 KB
- Stars: 17
- Watchers: 4
- Forks: 18
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Kafka connect SMT to expand JSON string
This java lib implements Kafka connect SMT (Single Message Transformation) to
extract JSON object from input string field.
## Config
Use it in connector config file like this:
~~~json
...
"transforms": "expand",
"transforms.expand.type": "com.redhat.insights.expandjsonsmt.ExpandJSON$Value",
"transforms.expand.sourceFields": "metadata"
...
~~~
Use dot notation for deeper fields (e. g. `level1.level2`).
## Install to Kafka Connect
After build copy file `target/kafka-connect-smt-expandjsonsmt-0.0.5-assemble-all.jar`
to Kafka Connect container `` copying to its docker image or so.
It can be done adding this line to Dockerfile:
~~~Dockerfile
COPY ./target/kafka-connect-smt-expandjsonsmt-0.0.5-assemble-all.jar $KAFKA_CONNECT_PLUGINS_DIR
~~~
Or download current release:
~~~Dockerfile
RUN curl -fSL -o /tmp/plugin.tar.gz \
https://github.com/RedHatInsights/expandjsonsmt/releases/download/0.0.5/kafka-connect-smt-expandjsonsmt-0.0.5.tar.gz && \
tar -xzf /tmp/plugin.tar.gz -C $KAFKA_CONNECT_PLUGINS_DIR && \
rm -f /tmp/plugin.tar.gz;
~~~
## Example
~~~bash
# build jar file and store to target directory
mvn package
# start example containers (kafka, postgres, elasticsearch, ...)
docker-compose up --build
# when containers started run in separate terminal:
cd dev
./connect.sh # init postgres and elasticsearch connectors
./show_topics.sh # check created topic 'dbserver1.public.hosts' in kafka
./show_es.sh # check transformed documents imported from postgres to elasticsearch
# ... stop containers
docker-compose down
~~~
## Build release file
- Remove `target` directory if it exists.
- Increment version in `pom.xml` (e.g. to `0.0.3`).
- Run build script: `./scripts/build_release.sh 0.0.3`.
- Take `*.tar.gz` file from `target` folder and publish it.