https://github.com/ably/kafka-connect-ably
Kafka Connector for publishing data from Kafka to Ably
https://github.com/ably/kafka-connect-ably
kafka kafka-connect kafka-connector sdk
Last synced: 11 months ago
JSON representation
Kafka Connector for publishing data from Kafka to Ably
- Host: GitHub
- URL: https://github.com/ably/kafka-connect-ably
- Owner: ably
- License: apache-2.0
- Created: 2021-02-10T02:36:56.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2024-07-19T14:35:49.000Z (almost 2 years ago)
- Last Synced: 2025-03-30T16:46:31.168Z (about 1 year ago)
- Topics: kafka, kafka-connect, kafka-connector, sdk
- Language: Java
- Homepage: https://ably.com/solutions/extend-kafka-to-the-edge
- Size: 800 KB
- Stars: 12
- Watchers: 23
- Forks: 7
- Open Issues: 29
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# Ably Kafka Connector
_[Ably](https://ably.com) is a realtime experiences infrastructure platform. Developers use our SDKs and APIs to build, extend and deliver realtime capabilities like Chat, Data Broadcast, Data Synchronization, Multiplayer Collaboration, and Notifications. Each month, Ably powers digital experiences in realtime for more than 350 million devices across 80 countries each. Organizations like HubSpot, Genius Sports, Verizon, Webflow, and Mentimeter depend on Ably’s platform to offload the growing complexity of business-critical realtime infrastructure at a global scale. For more information, see the [Ably documentation](https://ably.com/documentation)._
## Overview
The Ably Kafka Connector is a sink connector used to publish data from [Apache Kafka](http://kafka.apache.org/) into [Ably](https://ably.com) and is available on [Confluent Hub](https://www.confluent.io/hub/ably/kafka-connect-ably). It has also been tested with [AWS MSK](https://aws.amazon.com/msk/).
The connector will publish data from one or more [Kafka topics](https://docs.confluent.io/platform/current/kafka/introduction.html#main-concepts-and-terminology) into one or more [Ably channels](https://ably.com/documentation/core-features/channels).
The connector is built on top of [Apache Kafka Connect](http://docs.confluent.io/current/connect/) and can be run locally with Docker, installed into an instance of Confluent Platform or attached to an AWS MSK cluster through
[MSK Connect](https://aws.amazon.com/msk/features/msk-connect/).
## Install
Install the connector using the [Confluent Hub Client](#confluent-hub-installation) or [manually](#manual-installation) on Confluent Platform. Alternatively deploy it locally using [Docker](#docker). If installing in Amazon MSK see the AWS MSK section below.
### Confluent Hub installation
To install the connector on a local installation of Confluent using the Confluent Hub Client:
1. Ensure that the Confluent Hub Client is installed. See the [Confluent instructions](https://docs.confluent.io/home/connect/confluent-hub/client.html#installing-c-hub-client) for steps to complete this.
2. Run the following command to install the Ably Kafka Connector:
`confluent-hub install ably/kafka-connect-ably:`
Where `` is the latest version of the connector.
3. [Configure](#configuration) the connector.
### Manual installation
To manually install the connector on a local installation of Confluent:
1. Obtain the `.zip` of the connector from Confluent Hub or this repository:
**From Confluent Hub**:
Visit the [Ably Kafka Connector](https://www.confluent.io/hub/ably/kafka-connect-ably) page on Confluent Hub and click the **Download** button.
**From this repository**:
1. Clone the repository:
`git clone git@github.com:ably/kafka-connect-ably.git`
2. Build the connector using [Maven](https://maven.apache.org/):
`mvn clean package`
3. A `.zip` file will be produced in the `/target/components/packages/` folder after the process has run.
2. Extract the `.zip` into a directory specified in the `plugin.path` of your connect worker's configuration properties file. See the [Confluent instructions](https://docs.confluent.io/home/connect/install.html#install-connector-manually) for further information on this step.
3. [Configure](#configuration) the connector.
### Confluent Cloud Custom Connector
It is possible to use the connector as a plugin on Confluent Cloud as a [Custom Connector](https://docs.confluent.io/cloud/current/connectors/bring-your-connector/overview.html). These steps assume that you have created a Confluent Cloud account and configured your cluster.
> [!IMPORTANT]
> In order to run Ably connector, your Kafka cluster must reside in a [supported cloud provider and region](https://docs.confluent.io/cloud/current/connectors/bring-your-connector/custom-connector-fands.html#cc-byoc-regions).
1. Obtain the `.zip` of the connector as per [the manual installation guide](#manual-installation).
2. Inside the cluster on your Confluent Cloud account, add a new Connector
3. Instead of selecting Ably Kafka Connector from the Hub, instead click Add Plugin
4. Give the plugin a name, and set the class to `com.ably.kafka.connect.ChannelSinkConnector`
5. Upload the `.zip` file you obtained in step 1
6. In the plugin config, insert the following, replacing the placeholder with your Ably API key:
```json
{
"connector.class": "com.ably.kafka.connect.ChannelSinkConnector",
"tasks.max": "3",
"group.id": "ably-connect-cluster",
"topics": ",",
"client.id": "Ably-Kafka-Connector",
"channel": "#{topic}",
"message.name": "#{topic}_message",
"client.key": "",
"key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter.schemas.enable": "false"
}
```
7. When asked for an endpoint, enter `rest.ably.io:443:TCP`. If you are using Kafka Schema Registry, also add `:443:TCP`.
### AWS MSK
See the getting started instructions and example deployment in the [examples section](examples/msk_connect/README.md).
### Docker
There is a [`docker-compose.yml`](docker-compose.yml) file for standalone mode and an alternative [`docker-compose-distributed.yml`](docker-compose-distributed.yml) for distributed mode included in this repository that can be used to run the connector locally using [Docker Compose](https://docs.docker.com/compose/). The Docker Compose file is based on the [Confluent Platform Docker images](https://docs.confluent.io/platform/current/installation/docker/image-reference.html).
1. Create and configure a [configuration file](#configuration) ensuring you have set at least the basic properties.
**Note:** You must provide connector properties when starting connector in distributed mode.
An example cURL command to start the connector in distributed mode is:
```shell
curl -X POST -H "Content-Type: application/json" --data '{"name": "ably-channel-sink",
"config": {"connector.class":"com.ably.kafka.connect.ChannelSinkConnector", "tasks.max":"3",
"group.id":"ably-connect-cluster",
"topics":"topic1,topic2","client.id":"Ably-Kafka-Connector","channel":"#{topic}","message.name": "#{topic}_message",
"client.key":"" }}' http://localhost:8083/connectors
```
2. Start the cluster using:
`docker-compose up -d`
for standalone mode or
`docker-compose -f docker-compose-distributed.yml up -d` for distributed mode.
**Note**: You can view the logs using `docker-compose logs connector`
**Note 2**: You must start your connectors using [Connect REST interface](https://docs.confluent.io/platform/current/connect/references/restapi.html) when using distributed mode.
3. Once the containers have started, you can test the connector by subscribing to your Ably channel using [SSE](https://ably.com/documentation/sse) in a new terminal window. Replace `` with the channel set in your configuration file and `` with an API key with the capability to subscribe to the channel.
`curl -s -u "" "https://realtime.ably.io/sse?channel=&v=1.1"`
**Note**: SSE is only used as an example. An Ably SDK can also be used to subscribe to the channel.
4. Produce a set of test messages in Kafka using the Kafka CLI tool. Replace `` with one of the topics set in your configuration file.
```
docker-compose exec -T kafka kafka-console-producer --topic --broker-list kafka:9092 <,
then set the following configuration so that those messages are converted from Avro to JSON:
```
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=https://
```
If you're running the Ably Kafka Connector locally using Docker Compose as outlined above, then you can use the [`kafka-avro-console-producer` CLI](https://docs.confluent.io/platform/current/tutorials/examples/clients/docs/kafka-commands.html#produce-avro-records) to test producing Avro serialized messages by running the following:
```shell
docker-compose exec -T schema-registry kafka-avro-console-producer \
--topic topic1 \
--broker-list kafka:9092 \
--property key.schema='{"type":"string"}' \
--property parse.key=true \
--property key.separator=":" \
--property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"count","type":"int"}]}' \
--property schema.registry.url=http://schema-registry:8081 <