{"id":20770634,"url":"https://github.com/ably/kafka-connect-ably","last_synced_at":"2025-08-01T12:09:30.317Z","repository":{"id":39749310,"uuid":"337596665","full_name":"ably/kafka-connect-ably","owner":"ably","description":"Kafka Connector for publishing data from Kafka to Ably","archived":false,"fork":false,"pushed_at":"2024-07-19T14:35:49.000Z","size":819,"stargazers_count":12,"open_issues_count":29,"forks_count":7,"subscribers_count":23,"default_branch":"main","last_synced_at":"2025-03-30T16:46:31.168Z","etag":null,"topics":["kafka","kafka-connect","kafka-connector","sdk"],"latest_commit_sha":null,"homepage":"https://ably.com/solutions/extend-kafka-to-the-edge","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ably.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-10T02:36:56.000Z","updated_at":"2024-12-05T21:56:48.000Z","dependencies_parsed_at":"2024-11-18T00:04:08.723Z","dependency_job_id":null,"html_url":"https://github.com/ably/kafka-connect-ably","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ably%2Fkafka-connect-ably","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ably%2Fkafka-connect-ably/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ably%2Fkafka-connect-ably/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ably%2Fkafka-connect-ably/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ably","download_url":"https://codeload.github.com/ably/kafka-connect-ably/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251714965,"owners_count":21631812,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-connect","kafka-connector","sdk"],"created_at":"2024-11-17T12:11:01.577Z","updated_at":"2025-04-30T13:52:49.574Z","avatar_url":"https://github.com/ably.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ably Kafka Connector\n\n_[Ably](https://ably.com) is a realtime experiences infrastructure platform. Developers use our SDKs and APIs to build, extend and deliver realtime capabilities like Chat, Data Broadcast, Data Synchronization, Multiplayer Collaboration, and Notifications. Each month, Ably powers digital experiences in realtime for more than 350 million devices across 80 countries each. Organizations like HubSpot, Genius Sports, Verizon, Webflow, and Mentimeter depend on Ably’s platform to offload the growing complexity of business-critical realtime infrastructure at a global scale. For more information, see the [Ably documentation](https://ably.com/documentation)._\n\n## Overview\n\nThe Ably Kafka Connector is a sink connector used to publish data from [Apache Kafka](http://kafka.apache.org/) into [Ably](https://ably.com) and is available on [Confluent Hub](https://www.confluent.io/hub/ably/kafka-connect-ably). It has also been tested with [AWS MSK](https://aws.amazon.com/msk/).\n\nThe connector will publish data from one or more [Kafka topics](https://docs.confluent.io/platform/current/kafka/introduction.html#main-concepts-and-terminology) into one or more  [Ably channels](https://ably.com/documentation/core-features/channels).\n\nThe connector is built on top of [Apache Kafka Connect](http://docs.confluent.io/current/connect/) and can be run locally with Docker, installed into an instance of Confluent Platform or attached to an AWS MSK cluster through\n[MSK Connect](https://aws.amazon.com/msk/features/msk-connect/).\n\n## Install\n\nInstall the connector using the [Confluent Hub Client](#confluent-hub-installation) or [manually](#manual-installation) on Confluent Platform. Alternatively deploy it locally using [Docker](#docker). If installing in Amazon MSK see the AWS MSK section below.\n\n### Confluent Hub installation\n\nTo install the connector on a local installation of Confluent using the Confluent Hub Client:\n\n1. Ensure that the Confluent Hub Client is installed. See the [Confluent instructions](https://docs.confluent.io/home/connect/confluent-hub/client.html#installing-c-hub-client) for steps to complete this.\n\n2. Run the following command to install the Ably Kafka Connector:\n\n    `confluent-hub install ably/kafka-connect-ably:\u003cversion\u003e`\n\n    Where `\u003cversion\u003e` is the latest version of the connector.\n\n3. [Configure](#configuration) the connector.\n\n### Manual installation\n\nTo manually install the connector on a local installation of Confluent:\n\n1. Obtain the `.zip` of the connector from Confluent Hub or this repository:\n\n    **From Confluent Hub**:\n\n    Visit the [Ably Kafka Connector](https://www.confluent.io/hub/ably/kafka-connect-ably) page on Confluent Hub and click the **Download** button.\n\n    **From this repository**:\n\n    1. Clone the repository:\n\n        `git clone git@github.com:ably/kafka-connect-ably.git`\n\n    2. Build the connector using [Maven](https://maven.apache.org/):\n\n        `mvn clean package`\n\n    3. A `.zip` file will be produced in the `/target/components/packages/` folder after the process has run.\n\n2. Extract the `.zip` into a directory specified in the `plugin.path` of your connect worker's configuration properties file. See the [Confluent instructions](https://docs.confluent.io/home/connect/install.html#install-connector-manually) for further information on this step.\n\n3. [Configure](#configuration) the connector.\n\n\n### Confluent Cloud Custom Connector\n\nIt is possible to use the connector as a plugin on Confluent Cloud as a [Custom Connector](https://docs.confluent.io/cloud/current/connectors/bring-your-connector/overview.html). These steps assume that you have created a Confluent Cloud account and configured your cluster.\n\n\u003e [!IMPORTANT]  \n\u003e In order to run Ably connector, your Kafka cluster must reside in a [supported cloud provider and region](https://docs.confluent.io/cloud/current/connectors/bring-your-connector/custom-connector-fands.html#cc-byoc-regions).\n\n1. Obtain the `.zip` of the connector as per [the manual installation guide](#manual-installation).\n2. Inside the cluster on your Confluent Cloud account, add a new Connector\n3. Instead of selecting Ably Kafka Connector from the Hub, instead click Add Plugin\n4. Give the plugin a name, and set the class to `com.ably.kafka.connect.ChannelSinkConnector`\n5. Upload the `.zip` file you obtained in step 1\n6. In the plugin config, insert the following, replacing the placeholder with your Ably API key:\n\n```json\n{\n  \"connector.class\": \"com.ably.kafka.connect.ChannelSinkConnector\",\n  \"tasks.max\": \"3\",\n  \"group.id\": \"ably-connect-cluster\",\n  \"topics\": \"\u003ctopic1\u003e,\u003ctopic2\u003e\",\n  \"client.id\": \"Ably-Kafka-Connector\",\n  \"channel\": \"#{topic}\",\n  \"message.name\": \"#{topic}_message\",\n  \"client.key\": \"\u003cYOUR_ABLY_API_KEY\u003e\",\n  \"key.converter\": \"org.apache.kafka.connect.converters.ByteArrayConverter\",\n  \"value.converter\": \"org.apache.kafka.connect.converters.ByteArrayConverter\",\n  \"value.converter.schemas.enable\": \"false\"\n}\n```\n7. When asked for an endpoint, enter `rest.ably.io:443:TCP`. If you are using Kafka Schema Registry, also add `\u003cSCHEMA_REGISTRY_URL\u003e:443:TCP`.\n\n### AWS MSK\n\nSee the getting started instructions and example deployment in the [examples section](examples/msk_connect/README.md).\n\n### Docker\n\nThere is a [`docker-compose.yml`](docker-compose.yml) file for standalone mode and an alternative [`docker-compose-distributed.yml`](docker-compose-distributed.yml) for distributed mode included in this repository that can be used to run the connector locally using [Docker Compose](https://docs.docker.com/compose/). The Docker Compose file is based on the [Confluent Platform Docker images](https://docs.confluent.io/platform/current/installation/docker/image-reference.html).\n\n1. Create and configure a [configuration file](#configuration) ensuring you have set at least the basic properties.\n**Note:** You must provide connector properties when starting connector in distributed mode. \nAn example cURL command to start the connector in distributed mode is:\n\n\n```shell\n curl -X POST -H \"Content-Type: application/json\" --data '{\"name\": \"ably-channel-sink\",\n     \"config\": {\"connector.class\":\"com.ably.kafka.connect.ChannelSinkConnector\", \"tasks.max\":\"3\", \n     \"group.id\":\"ably-connect-cluster\",\n     \"topics\":\"topic1,topic2\",\"client.id\":\"Ably-Kafka-Connector\",\"channel\":\"#{topic}\",\"message.name\": \"#{topic}_message\",\n     \"client.key\":\"\u003cPut your API key here\u003e\" }}' http://localhost:8083/connectors\n```\n2. Start the cluster using:\n\n    `docker-compose up -d`\n for standalone mode or\n\n   `docker-compose -f docker-compose-distributed.yml up -d` for distributed mode.\n   \n**Note**: You can view the logs using `docker-compose logs connector`\n\n**Note 2**: You must start your connectors using [Connect REST interface](https://docs.confluent.io/platform/current/connect/references/restapi.html) when using distributed mode.\n\n\n3. Once the containers have started, you can test the connector by subscribing to your Ably channel using [SSE](https://ably.com/documentation/sse) in a new terminal window. Replace `\u003cchannel-name\u003e` with the channel set in your configuration file and `\u003cably-api-key\u003e` with an API key with the capability to subscribe to the channel.\n\n    `curl -s -u \"\u003cably-api-key\u003e\" \"https://realtime.ably.io/sse?channel=\u003cchannel-name\u003e\u0026v=1.1\"`\n\n    **Note**: SSE is only used as an example. An Ably SDK can also be used to subscribe to the channel.\n\n4. Produce a set of test messages in Kafka using the Kafka CLI tool. Replace `\u003ckafka-topic-name\u003e` with one of the topics set in your configuration file.\n\n    ```\n    docker-compose exec -T kafka kafka-console-producer --topic \u003ckafka-topic-name\u003e --broker-list kafka:9092 \u003c\u003cEOF\n    message 1\n    message 2\n    message 3\n    EOF\n    ```\n\n5. In the terminal window where you subscribed to the Ably channel, you will receive messages similar to the following:\n\n    ```\n    id: e026fVvywAz6Il@1623496744539-0\n    event: message\n    data: {\"id\":\"1543960661:0:0\",\"clientId\":\"kafka-connect-ably-example\",\"connectionId\":\"SuJTceISnT\",\"timestamp\":1623496744538,\"encoding\":\"base64\", \"channel\":\"kafka-connect-ably-example\",\"data\":\"bWVzc2FnZSAx\",\"name\":\"sink\"}\n\n    id: e026fVvywAz6Il@1623496744539-1\n    event: message\n    data: {\"id\":\"1543960661:0:1\",\"clientId\":\"kafka-connect-ably-example\",\"connectionId\":\"SuJTceISnT\",\"timestamp\":1623496744538,\"encoding\":\"base64\", \"channel\":\"kafka-connect-ably-example\",\"data\":\"bWVzc2FnZSAy\",\"name\":\"sink\"}\n\n    id: e026fVvywAz6Il@1623496744539-2\n    event: message\n    data: {\"id\":\"1543960661:0:2\",\"clientId\":\"kafka-connect-ably-example\",\"connectionId\":\"SuJTceISnT\",\"timestamp\":1623496744538,\"encoding\":\"base64\", \"channel\":\"kafka-connect-ably-example\",\"data\":\"bWVzc2FnZSAz\",\"name\":\"sink\"}\n    ```\n#### Publishing messages with a Push Notification\n\nMessages can be delivered to end user devices as Push Notifications by setting a Kafka message header named `com.ably.extras.push` with a notification payload, for example:\n\n```json\n{\n  \"notification\": {\n    \"title\": \"Notification title\",\n    \"body\": \"This is the body of notification\"\n  },\n  \"data\": {\n    \"foo\": \"foo\",\n    \"bar\": \"bar\"\n  }\n}\n```\n\nExtra Ably configuration is also required to enable push notifications, see the [Push Notification documentation](https://ably.com/docs/general/push).\n\n#### Publishing messages with schema\n\nThe Ably Kafka Connector supports messages which contain schema information by converting them to JSON before publishing \nthem to Ably. To check how to use schema registry and supported converters, see \n[Using Kafka Connect with Schema Registry](https://docs.confluent.io/platform/current/schema-registry/connect.html).\nFor example, if messages on the Kafka topic are serialized using Avro with schemas registered at https://\u003cyour-schema-registry-host\u003e, \nthen set the following configuration so that those messages are converted from Avro to JSON:\n```\nvalue.converter=io.confluent.connect.avro.AvroConverter\nvalue.converter.schema.registry.url=https://\u003cyour-schema-registry-host\u003e\n```\n\nIf you're running the Ably Kafka Connector locally using Docker Compose as outlined above, then you can use the [`kafka-avro-console-producer` CLI](https://docs.confluent.io/platform/current/tutorials/examples/clients/docs/kafka-commands.html#produce-avro-records) to test producing Avro serialized messages by running the following:\n```shell\n docker-compose exec -T schema-registry kafka-avro-console-producer \\\n   --topic topic1 \\\n   --broker-list kafka:9092 \\\n   --property key.schema='{\"type\":\"string\"}' \\\n   --property parse.key=true \\\n   --property key.separator=\":\" \\\n   --property value.schema='{\"type\":\"record\",\"name\":\"myrecord\",\"fields\":[{\"name\":\"count\",\"type\":\"int\"}]}' \\\n   --property schema.registry.url=http://schema-registry:8081 \u003c\u003cEOF\n\"key1\":{\"count\":1}\nEOF\n```\n\nYou should receive following Ably message where you subscribed. You will also receive an Avro-formatted key base64 encoded in the extras. \n\n```json\n{\n\t\"clientId\": \"Ably-Kafka-Connector\",\n\t\"connectionId\": \"VSuDXysgaz\",\n\t\"data\": {\n\t\t\"count\": 1\n\t},\n\t\"extras\": {\n    \"kafka\": {\n      \"key\": \"AAAAAKEIa2V5MQ==\"\n    }\n\t},\n\t\"id\": \"-868034334:0:351\",\n\t\"name\": \"topic1_message\",\n\t\"timestamp\": 1653923422360\n}\n```\n\nNote that configuring the schema registry and appropriate key or value converters also enables referencing of record\nfield data within [Dynamic Channel Configuration](#dynamic-channel-configuration)\n\n## Breaking API Changes in Version 2.0.0\n\nPlease see our [Upgrade / Migration Guide](UPDATING.md) for notes on changes you need to make to your configuration to update it with changes introduced by version 2.0.0 of the connector.\n\n## Configuration\n\nConfiguration is handled differently depending on how the connector is installed:\n\n| Installation | Configuration |\n| ------------ | ------------- | \n| Docker | Create a `docker-compose-connector.properties` file in the `/config` directory. An [example file](config/example-connector.properties) already exists. |\n| Single connect worker | Provide a configuration file as a [command line argument](https://docs.confluent.io/home/connect/userguide.html#standalone-mode). |\n| Distributed connect workers | Use the Confluent REST API [`/connectors` endpoint](https://docs.confluent.io/platform/current/connect/references/restapi.html#post--connectors) to pass the configuration as JSON. |\n\n### Configuration properties\n\nThe basic properties that must be configured for the connector are:\n\n| Property        | Description                                                                                                                                                                                                                                       | Type      | Default                                      |\n|-----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|----------------------------------------------|\n| channel         | The name of the [Ably channel](https://ably.com/documentation/realtime/channels) to publish to. See also: [Dynamic channel configuration](#Dynamic-Channel-Configuration)                                                                         | *String*  ||\n| client.key      | An API key from your Ably dashboard to use for authentication. This must have the [publish capability](https://ably.com/documentation/core-features/authentication#capabilities-explained) for the `channel` being published to by the connector. | *String*  ||\n| client.id       | The [Ably client ID](https://ably.com/documentation/realtime/authentication#identified-clients) to use for the connector.                                                                                                                         | *String*  | kafka-connect-ably-example                   |\n| name            | A globally unique name for the connector.                                                                                                                                                                                                         | *String*  | ably-channel-sink                            |\n| topics          | A comma separated list of Kafka topics to publish from.                                                                                                                                                                                           | *String*  |\n| tasks.max       | The maximum number of tasks to use for the connector.                                                                                                                                                                                             | *Integer* | 1                                            |\n| connector.class | The name of the class for the connector. This must be a subclass of `org.apache.kafka.connect.connector`.                                                                                                                                         | *String*  | `io.ably.kakfa.connect.ChannelSinkConnector` |\n\nThe advanced properties that can be configured for the connector are:\n\n| Property                          | Description                                                                                                                                            | Type      | Default |\n|-----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|---------|\n| message.name                      | Ably message name to publish. Can be a pattern, as per [Dynamic Channel Configuration](#dynamic-channel-configuration) below.                          | *String*  |         |\n| batchExecutionThreadPoolSize      | The maximum number of parallel outgoing REST API requests to publish content to Ably                                                                   | *Integer* | 10      |\n| batchExecutionMaxBufferSize       | The maximum number of records to publish in a single batch to Ably. The maximum size of these batches must be less than the [maximum batch publish size](https://ably.com/docs/api/rest-api#batch-publish). | *Integer* | 100     |\n| batchExecutionMaxBufferSizeMs     | The maxmium amount of time (in milliseconds) to wait for the batch publishing buffer to fill before publishing anyway                                  | *Integer* | 100     |\n| onFailedRecordMapping             | Action to take if a dyanmic channel mapping fails for a record. See [Dynamic Channel Configuration](#Dynamic-Channel-Configuration) for full details.  | *String*  | stop    |\n| client.async.http.threadpool.size | The size of the asyncHttp threadpool.                                                                                                                  | *Integer* | 64      |\n| client.fallback.hosts             | A list of custom fallback hosts. This will override the default fallback hosts.                                                                        | *List*    ||\n| client.http.max.retry.count       | The maximum number of fallback hosts to use when an HTTP request to the primary host is unreachable or indicates that it is unserviceable.             | *Integer* | 3       |\n| client.http.open.timeout          | The timeout period for opening an HTTP connection.                                                                                                     | *Integer* | 4000    |\n| client.http.request.timeout       | The timeout period for any single HTTP request and response.                                                                                           | *Integer* | 15000   |\n| client.proxy                      | Sets whether the configured proxy options are used.                                                                                                    | *Boolean* ||\n| client.proxy.host                 | The proxy host to use. Requires `client.proxy` to be set to `true`.                                                                                    | *String*  ||\n| client.proxy.non.proxy.hosts      | A list of hosts excluded from using the proxy. Requires `client.proxy` to be set to `true`.                                                            | *List*    ||\n| client.proxy.username             | The client proxy username. Requires `client.proxy` to be set to `true`.                                                                                | *String*  ||\n| client.proxy.password             | The client proxy password. Requires `client.proxy` to be set to `true`.                                                                                | *String*  ||\n| client.proxy.port                 | The client proxy port. Requires `client.proxy` to be set to `true`.                                                                                    | *Integer* ||\n| client.proxy.pref.auth.type       | The authentication type to use with the client proxy. Must be one of `BASIC`, `DIGEST` or `X_ABLY_TOKEN`. Requires `client.proxy` to be set to `true`. | *String*  | Basic   |\n| client.push.full.wait             | Sets whether Ably should wait for all the effects of push REST requests before responding.                                                             | *Boolean* ||\n| client.tls                        | Sets whether TLS is used for all connection types.                                                                                                     | *Boolean* | True    |\n| client.loglevel                   | Sets the verbosity of logging.                                                                                                                         | *Integer* | 0       |\n| client.environment                | Custom Ably environment (https://ably.com/docs/platform-customization)                                                                                 | *String*  ||\n\n## Buffering Records\n\nThe Ably Kafka connector buffers records locally so that larger batches can be sent to Ably in parallel for improved\nthroughput. This behaviour is configurable using these settings:\n\n* `batchExecutionMaxBufferSizeMs` - the maximum amount of time to wait (in milliseconds) for record data to accumulate \n  before submitting as many records as have been buffered so far to Ably.\n* `batchExecutionMaxBufferSize` - the maximum number of records to buffer before submitting to Ably\n* `batchExecutionThreadPoolSize` - the size of the thread pool used to submit buffered batches to Ably, and therefore\n  the maximum number of concurrent submissions to Ably per sink task.\n\nSome consideration should be given with respect to your workload and requirements:\n\n* `batchExecutionMaxBufferSizeMs` is effectively the minimum latency for outgoing records. Increasing this value gives\n  records more time to accumulate so that more can be sent in each batch to Ably for improved efficiency, but if your\n  workload requires that latency between Kafka and the end user is low, you may need to decrease this value.\n* `batchExecutionMaxBufferSize` can be used to set a maximum on the number of records being sent to Ably. You should\n  consider your typical outgoing Ably message sizes and ensure that this is not set so high that batch submissions to\n  Ably may exceed your account limits.\n\n### Message Ordering\n\nIf your workload requires that messages are sent to Ably channels in the order that they were published to Kafka Topic\npartitions, you will need to disable parallel submissions with sink tasks from the Ably connector as follows:\n\n* Set `batchExecutionThreadPoolSize=1` to prevent parallel submissions per task\n* Likely increase `max.tasks` to be the desired number of separate task instances required to achieve parallelism across\n  all topic partitions\n\n\n## Dynamic Channel Configuration\n\nAbly [Channels](https://ably.com/docs/channels) are very lightweight, and it's idiomatic to use very high numbers of distinct\nchannels to send messages between users. In many use cases, it makes sense to create an Ably channel per user or per session,\nmeaning there could be millions in total. Contrast with a typical Kafka deployment, where you're more likely to be putting\nrecords related to all users but of some common type through a single topic.\n\nTo enable \"fan-out\" to high numbers of channels from your Kafka topic, the Ably Kafka Connector supports Dynamic Channel\nConfiguration, whereby you configure a template string to substitute data from incoming Kafka records into the outgoing Ably\nChannel name. The same functionality is also supported for the [Message](https://ably.com/docs/channels/messages) `name`\nfield, if required.\n\nTo make use of this feature, simply set the `channel` and/or `message.name` settings in your Connector properties file\nto reference data from the incoming Kafka record key or value, or metadata from the Kafka Topic, using the `#{...}`\nplaceholder. Referencing data within a record key or value is only possible when making use of \n[Kafka Connector Schema Support](https://docs.confluent.io/platform/current/schema-registry/index.html), though Topic\nmetadata can always be referenced without a schema. References to unstructured `key` values is also supported, assuming\nthey can be converted to a string.\n\nFor example, the following configuration references a field within the record value and the topic name:\n\n```properties\nchannel = user:#{value.userId}\nmessage.name = #{topic.name}\n```\n\nThis assumes that a schema registry is available and contains a schema for record values, and that an appropriate\nconverter has been configured for values. If the value schema contains a field, `userId`, the configuration above\nwill substitute those values into the outgoing channel names as per the template. The Ably Message name will also\nbe set to the Kafka topic name, in this example.\n\nFields can be nested within other `Structs`, for example `value.someStruct.userId` would also be valid if `someStruct`\nhas `STRUCT` type and `userId` can be converted to a string. Given that both message and channel names are ultimately\nstrings, the referenced fields must be reasonably interpretable as a string. The supported conversions are:\n\n* String\n* Integer (any precision)\n* Boolean\n* Bytes (assumed to be UTF-8 encoded string data)\n\nThe table below summarises the substitutions that can be made within a `#{}` placeholder:\n\n| Placeholder            | Description                                                  |\n|------------------------|--------------------------------------------------------------|\n| `#{topic.name}`        | The Kafka Topic name                                         |\n| `#{topic.partition}`   | The Kafka Topic partition                                    |\n| `#{topic}`             | Alias for `topic.name`                                       |\n| `#{key}`               | The record key. Must be convertible to a string              |\n| `#{key.nestedField}`   | If key is a struct with a schema, uses `nestedField` value   |\n| `#{value.nestedField}` | If value is a struct with a schema, uses `nestedField` value |\n\n### Handling Failed Mappings\n\nDynamic channel mapping can fail at runtime, if:\n\n* The template references a field that a record is missing\n* The referenced field cannot be converted to a string\n\nIn these situations, it's possible to configure the desired error handling behaviour by setting the `onFailedRecordMapping`\nproperty to one of the following values:\n\n* `stop` (default) - Treat failed mappings as fatal and stop the Sink Task completely.\n* `skip` - Silently ignore records that cannot be mapped to the required template. Use this only if you're sure that\nthe Kafka topic contains irrelevant records you'd like to filter out this way.\n* `dlq` - Send records with failed mappings along with the error (as a header) to a configured \n[dead-letter queue](#dead-letter-queues).\n\n## Dead Letter Queues\n\nThe Ably Kafka Connector is able to forward records to a dead-letter queue topic using Kafka Connect dead-letter queue\nsupport. You can learn more about dead-letter queues [here](https://www.confluent.io/en-gb/blog/kafka-connect-deep-dive-error-handling-dead-letter-queues/).\n\nAs an example, adding the following configuration to your connector properties file will cause all failed records\nto be sent to a `dlq_ably_sink` topic with a replication factor of 1 and headers attached giving you full exception\ndetails for each record.\n\n```properties\nerrors.tolerance = all\nerrors.deadletterqueue.topic.name = dlq_ably_sink\nerrors.deadletterqueue.topic.replication.factor=1\nerrors.deadletterqueue.context.headers.enable=true\n```\n\nSituations in which the Ably connector will forward records to a dead-letter queue include:\n\n* Errors submitting to Ably, perhaps due to insufficient permissions to publish to a specific destination channel\n* Serious Ably service outages, after retrying with fallback endpoints\n* Failed dynamic channel mappings, if `onFailedRecordMapping = dlq`\n\n## Contributing\n\nFor guidance on how to contribute to this project, see [CONTRIBUTING.md](CONTRIBUTING.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fably%2Fkafka-connect-ably","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fably%2Fkafka-connect-ably","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fably%2Fkafka-connect-ably/lists"}