https://github.com/timeplus-io/kafka-connect-timeplus
The Timeplus sink connector allows moving data from Kafka to Timeplus Cloud
https://github.com/timeplus-io/kafka-connect-timeplus
Last synced: 2 months ago
JSON representation
The Timeplus sink connector allows moving data from Kafka to Timeplus Cloud
- Host: GitHub
- URL: https://github.com/timeplus-io/kafka-connect-timeplus
- Owner: timeplus-io
- License: apache-2.0
- Created: 2022-12-23T17:17:16.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2024-01-15T17:55:16.000Z (over 2 years ago)
- Last Synced: 2025-03-06T11:39:37.973Z (over 1 year ago)
- Language: Java
- Homepage: https://www.confluent.io/hub/timeplus/kafka-timeplus-connector-sink
- Size: 88.9 KB
- Stars: 2
- Watchers: 4
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Timeplus Sink Connector
A Kafka Connect plugin for Timeplus.
## Build
To build the project, run `mvn clean install`. If you want to include all dependency to simplify your deployment, run `mvn clean compile assembly:single`
## Configurations
The following configurations are supported
| key | value |
| -------------------------- | -------------------------------------------------- |
| name | ID for the plugin instance |
| connector.class | com.timeplus.kafkaconnect.TimeplusSinkConnector |
| timeplus.sink.address | Address to Timeplus Cloud or self-managed Timeplus Or Proton Address |
| timeplus.sink.workspace | Timeplus workspace ID, when set to empty, will use Proton as target |
| timeplus.sink.apikey | API Key for the workspace, 60 chars long |
| timeplus.sink.stream | destination stream name |
| timeplus.sink.createStream | whether to create the stream if not exist |
| timeplus.sink.dataFormat | stream data format. Either `raw` or `json`, when target Proton, only `raw` mode is supported |
Here is an example configuration of timeplus cloud as target:
```properties
name: TimeplusSink
connector.class: com.timeplus.kafkaconnect.TimeplusSinkConnector
tasks.max: 1
topics: my_topic
timeplus.sink.address: https://us.timeplus.cloud
timeplus.sink.workspace: abc123
timeplus.sink.apikey: 60_char_API_Key
timeplus.sink.stream: data_from_kafka
timeplus.sink.createStream: true
timeplus.sink.dataFormat: raw
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
```
Here is an example configuration of using Proton as target:
```properties
name: TimeplusSink
connector.class: com.timeplus.kafkaconnect.TimeplusSinkConnector
tasks.max: 1
topics: my_topic
timeplus.sink.address: http://localhost:3218
timeplus.sink.workspace:
timeplus.sink.apikey:
timeplus.sink.stream: data_from_kafka
timeplus.sink.createStream: true
timeplus.sink.dataFormat: raw
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
```
in case your want to externalize your API key, you can use configuration files refer to this [document](https://rmoff.net/2019/05/24/putting-kafka-connect-passwords-in-a-separate-file-/-externalising-secrets/)
Please check https://docs.timeplus.com/docs/kafka-connect for more details.
## Integration Test
to run integration test, you need have a timeplus cloud account and set following enironment , build your local docker image and then run the integration using mvn:
```sh
export TIMEPLUS_API_KEY=
export TIMEPLUS_ADDRESS=https://us.timeplus.cloud
export TIMEPLUS_WORKSPACE=
make docker
mvn test -Dtest=com.timeplus.kafkaconnect.integration.IntegrationTest
```
## Support
please email gang@timeplus.com if you have any question when using this plugin.