https://github.com/ClickHouse/clickhouse-kafka-connect

ClickHouse Kafka Connector
https://github.com/ClickHouse/clickhouse-kafka-connect

clickhouse kafka kafka-connect sink

Last synced: about 1 month ago
JSON representation

ClickHouse Kafka Connector

• Host: GitHub
• URL: https://github.com/ClickHouse/clickhouse-kafka-connect
• Owner: ClickHouse
• License: apache-2.0
• Created: 2022-06-21T07:55:37.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2024-08-27T02:11:40.000Z (4 months ago)
• Last Synced: 2024-08-27T03:49:24.364Z (4 months ago)
• Topics: clickhouse, kafka, kafka-connect, sink
• Language: Java
• Homepage:
• Size: 2.73 MB
• Stars: 144
• Watchers: 11
• Forks: 40
• Open Issues: 11
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE
  • Security: SECURITY.md
  • Support: Support Scripts.md

Awesome Lists containing this project

• awesome-clickhouse - ClickHouse/clickhouse-kafka-connect - The clickhouse-kafka-connect is the official Kafka Connect sink connector for ClickHouse that facilitates data delivery from Kafka topics to ClickHouse tables. (Integrations / Deployment and Management Tools)

README

        
# ClickHouse Kafka Connect Sink

## About

clickhouse-kafka-connect is the official Kafka Connect sink connector for [ClickHouse](https://clickhouse.com/).

The Kafka connector delivers data from a Kafka topic to a ClickHouse table.

## Documentation

See the [ClickHouse website](https://clickhouse.com/docs/en/integrations/kafka/clickhouse-kafka-connect-sink) for the full documentation entry.

## Design

For a full overview of the design and how exactly-once delivery semantics are achieved, see the [design document](./docs/DESIGN.md).

## Help

For additional help, please [file an issue in the repository](https://github.com/ClickHouse/clickhouse-kafka-connect/issues) or raise a question in [ClickHouse public Slack](https://clickhouse.com/slack).

## KeyToValue Transformation

We've created a transformation that allows you to convert a Kafka message key into a value.

This is useful when you want to store the key in a separate column in ClickHouse - by default, the column is `_key` and the type is String.

```sql

CREATE TABLE your_table_name

(

    `your_column_name` String,

    ...

    ...

    ...

    `_key` String

) ENGINE = MergeTree()

```

Simply add the transformation to your connector configuration:

    

```properties

transforms=keyToValue

transforms.keyToValue.type=com.clickhouse.kafka.connect.transforms.KeyToValue

transforms.keyToValue.field=_key

```