{"id":13697818,"url":"https://github.com/jcustenborder/kafka-connect-transform-common","last_synced_at":"2025-04-05T11:13:06.580Z","repository":{"id":37549935,"uuid":"104228450","full_name":"jcustenborder/kafka-connect-transform-common","owner":"jcustenborder","description":"Common Transforms for Kafka Connect.","archived":false,"fork":false,"pushed_at":"2024-07-31T08:01:56.000Z","size":173,"stargazers_count":153,"open_issues_count":39,"forks_count":57,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-10-13T05:29:37.446Z","etag":null,"topics":["kafka-connect","transformations"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jcustenborder.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-09-20T14:46:26.000Z","updated_at":"2024-10-12T14:47:08.000Z","dependencies_parsed_at":"2022-08-26T15:22:55.651Z","dependency_job_id":"b4e0ccda-1379-41d0-bf84-29d23d9cf547","html_url":"https://github.com/jcustenborder/kafka-connect-transform-common","commit_stats":null,"previous_names":[],"tags_count":41,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcustenborder%2Fkafka-connect-transform-common","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcustenborder%2Fkafka-connect-transform-common/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcustenborder%2Fkafka-connect-transform-common/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcustenborder%2Fkafka-connect-transform-common/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jcustenborder","download_url":"https://codeload.github.com/jcustenborder/kafka-connect-transform-common/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247325696,"owners_count":20920714,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka-connect","transformations"],"created_at":"2024-08-02T18:01:03.303Z","updated_at":"2025-04-05T11:13:06.547Z","avatar_url":"https://github.com/jcustenborder.png","language":"Java","funding_links":[],"categories":["Libraries","Data Formats \u0026 Transformations"],"sub_categories":["Kafka Connect","FreeSWITCH"],"readme":"# Introduction\n[Documentation](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common) | [Confluent Hub](https://www.confluent.io/hub/jcustenborder/kafka-connect-transform-common)\n\n\nThis project contains common transformations for every day use cases with Kafka Connect.\n\n# Installation\n\n## Confluent Hub\n\nThe following command can be used to install the plugin directly from the Confluent Hub using the\n[Confluent Hub Client](https://docs.confluent.io/current/connect/managing/confluent-hub/client.html).\n\n```bash\nconfluent-hub install jcustenborder/kafka-connect-transform-common:latest\n```\n\n## Manually\n\nThe zip file that is deployed to the [Confluent Hub](https://www.confluent.io/hub/jcustenborder/kafka-connect-transform-common) is available under\n`target/components/packages/`. You can manually extract this zip file which includes all dependencies. All the dependencies\nthat are required to deploy the plugin are under `target/kafka-connect-target` as well. Make sure that you include all the dependencies that are required\nto run the plugin.\n\n1. Create a directory under the `plugin.path` on your Connect worker.\n2. Copy all of the dependencies under the newly created subdirectory.\n3. Restart the Connect worker.\n\n\n\n\n# Transformations\n## [BytesToString](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/BytesToString.html)\n\n*Key*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.BytesToString$Key\n```\n*Value*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.BytesToString$Value\n```\n\n\n### Configuration\n\n#### General\n\n\n##### `charset`\n\nThe charset to use when creating the output string.\n\n*Importance:* HIGH\n\n*Type:* STRING\n\n*Default Value:* UTF-8\n\n\n\n##### `fields`\n\nThe fields to transform.\n\n*Importance:* HIGH\n\n*Type:* LIST\n\n\n\n\n## [ChangeCase](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/ChangeCase.html)\n\n*Key*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.ChangeCase$Key\n```\n*Value*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.ChangeCase$Value\n```\n\n\n### Configuration\n\n#### General\n\n\n##### `from`\n\nThe format to move from \n\n*Importance:* HIGH\n\n*Type:* STRING\n\n*Validator:* Matches: ``LOWER_HYPHEN``, ``LOWER_UNDERSCORE``, ``LOWER_CAMEL``, ``UPPER_CAMEL``, ``UPPER_UNDERSCORE``\n\n\n\n##### `to`\n\n\n\n*Importance:* HIGH\n\n*Type:* STRING\n\n*Validator:* Matches: ``LOWER_HYPHEN``, ``LOWER_UNDERSCORE``, ``LOWER_CAMEL``, ``UPPER_CAMEL``, ``UPPER_UNDERSCORE``\n\n\n\n\n## [ChangeTopicCase](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/ChangeTopicCase.html)\n\n```\ncom.github.jcustenborder.kafka.connect.transform.common.ChangeTopicCase\n```\n\nThis transformation is used to change the case of a topic.\n\n[✍️ Example](https://rmoff.net/2020/12/23/twelve-days-of-smt-day-12-community-transformations/#_change_the_topic_case) / [🎥 Video](https://www.youtube.com/watch?v=Z7k_6vGRrkc\u0026t=274s)\n\n### Tip\n\nThis transformation will convert a topic name like 'TOPIC_NAME' to `topicName`, or `topic_name`.\n### Configuration\n\n#### General\n\n\n##### `from`\n\nThe format of the incoming topic name. `LOWER_CAMEL` = Java variable naming convention, e.g., \"lowerCamel\". `LOWER_HYPHEN` = Hyphenated variable naming convention, e.g., \"lower-hyphen\". `LOWER_UNDERSCORE` = C++ variable naming convention, e.g., \"lower_underscore\". `UPPER_CAMEL` = Java and C++ class naming convention, e.g., \"UpperCamel\". `UPPER_UNDERSCORE` = Java and C++ constant naming convention, e.g., \"UPPER_UNDERSCORE\".\n\n*Importance:* HIGH\n\n*Type:* STRING\n\n*Validator:* Matches: ``LOWER_HYPHEN``, ``LOWER_UNDERSCORE``, ``LOWER_CAMEL``, ``UPPER_CAMEL``, ``UPPER_UNDERSCORE``\n\n\n\n##### `to`\n\nThe format of the outgoing topic name. `LOWER_CAMEL` = Java variable naming convention, e.g., \"lowerCamel\". `LOWER_HYPHEN` = Hyphenated variable naming convention, e.g., \"lower-hyphen\". `LOWER_UNDERSCORE` = C++ variable naming convention, e.g., \"lower_underscore\". `UPPER_CAMEL` = Java and C++ class naming convention, e.g., \"UpperCamel\". `UPPER_UNDERSCORE` = Java and C++ constant naming convention, e.g., \"UPPER_UNDERSCORE\".\n\n*Importance:* HIGH\n\n*Type:* STRING\n\n*Validator:* Matches: ``LOWER_HYPHEN``, ``LOWER_UNDERSCORE``, ``LOWER_CAMEL``, ``UPPER_CAMEL``, ``UPPER_UNDERSCORE``\n\n\n\n\n## [ExtractNestedField](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/ExtractNestedField.html)\n\n*Key*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.ExtractNestedField$Key\n```\n*Value*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.ExtractNestedField$Value\n```\n\n\n### Configuration\n\n#### General\n\n\n##### `input.inner.field.name`\n\nThe field on the child struct containing the field to be extracted. For example if you wanted the extract `address.state` you would use `state`.\n\n*Importance:* HIGH\n\n*Type:* STRING\n\n\n\n##### `input.outer.field.name`\n\nThe field on the parent struct containing the child struct. For example if you wanted the extract `address.state` you would use `address`.\n\n*Importance:* HIGH\n\n*Type:* STRING\n\n\n\n##### `output.field.name`\n\nThe field to place the extracted value into.\n\n*Importance:* HIGH\n\n*Type:* STRING\n\n\n\n\n## [ExtractTimestamp](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/ExtractTimestamp.html)\n\n*Key*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.ExtractTimestamp$Key\n```\n*Value*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.ExtractTimestamp$Value\n```\n\nThis transformation is used to use a field from the input data to override the timestamp for the record.\n\n[✍️ Example](https://rmoff.net/2020/12/23/twelve-days-of-smt-day-12-community-transformations/#_add_the_timestamp_of_a_field_to_the_topic_name) / [🎥 Video](https://www.youtube.com/watch?v=Z7k_6vGRrkc\u0026t=430s)\n\n\n### Configuration\n\n#### General\n\n\n##### `field.name`\n\nThe field to pull the timestamp from. This must be an int64 or a timestamp.\n\n*Importance:* HIGH\n\n*Type:* STRING\n\n\n\n\n## [HeaderToField](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/HeaderToField.html)\n\n*Key*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.HeaderToField$Key\n```\n*Value*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.HeaderToField$Value\n```\n\n\n### Configuration\n\n#### General\n\n\n##### `header.mappings`\n\nThe mapping of the header to the field in the message.\n\n*Importance:* HIGH\n\n*Type:* LIST\n\n\n\n\n## [NormalizeSchema](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/NormalizeSchema.html)\n\n*Key*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.NormalizeSchema$Key\n```\n*Value*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.NormalizeSchema$Value\n```\n\nThis transformation is used to convert older schema versions to the latest schema version. This works by keying all of the schemas that are coming into the transformation by their schema name and comparing the version() of the schema. The latest version of a schema will be used. Schemas are discovered as the flow through the transformation. The latest version of a schema is what is used.\n### Configuration\n\n\n\n## [PatternFilter](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/PatternFilter.html)\n\n*Key*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.PatternFilter$Key\n```\n*Value*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.PatternFilter$Value\n```\n\n\n### Configuration\n\n#### General\n\n\n##### `pattern`\n\nThe regex to test the message with. \n\n*Importance:* HIGH\n\n*Type:* STRING\n\n*Validator:* com.github.jcustenborder.kafka.connect.utils.config.validators.PatternValidator@4170ee0f\n\n\n\n##### `fields`\n\nThe fields to transform.\n\n*Importance:* HIGH\n\n*Type:* LIST\n\n\n\n\n## [PatternRename](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/PatternRename.html)\n\n*Key*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.PatternRename$Key\n```\n*Value*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.PatternRename$Value\n```\n\n\n### Configuration\n\n#### General\n\n\n##### `field.pattern`\n\n\n\n*Importance:* HIGH\n\n*Type:* STRING\n\n\n\n##### `field.replacement`\n\n\n\n*Importance:* HIGH\n\n*Type:* STRING\n\n\n\n##### `field.pattern.flags`\n\n\n\n*Importance:* LOW\n\n*Type:* LIST\n\n*Default Value:* [CASE_INSENSITIVE]\n\n*Validator:* [UNICODE_CHARACTER_CLASS, CANON_EQ, UNICODE_CASE, DOTALL, LITERAL, MULTILINE, COMMENTS, CASE_INSENSITIVE, UNIX_LINES]\n\n\n\n\n## [SchemaNameToTopic](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/SchemaNameToTopic.html)\n\n*Key*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.SchemaNameToTopic$Key\n```\n*Value*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.SchemaNameToTopic$Value\n```\n\nThis transformation is used to take the name from the schema for the key or value and replace the topic with this value.\n### Configuration\n\n\n\n## [SetMaximumPrecision](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/SetMaximumPrecision.html)\n\n*Key*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.SetMaximumPrecision$Key\n```\n*Value*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.SetMaximumPrecision$Value\n```\n\nThis transformation is used to ensure that all decimal fields in a struct are below the maximum precision specified.\n### Note\n\nThe Confluent AvroConverter uses a default precision of 64 which can be too large for some database systems.\n### Configuration\n\n#### General\n\n\n##### `precision.max`\n\nThe maximum precision allowed.\n\n*Importance:* HIGH\n\n*Type:* INT\n\n*Validator:* [1,...,64]\n\n\n\n\n## [SetNull](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/SetNull.html)\n\n*Key*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.SetNull$Key\n```\n*Value*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.SetNull$Value\n```\n\n\n### Configuration\n\n\n\n## [TimestampNow](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/TimestampNow.html)\n\n```\ncom.github.jcustenborder.kafka.connect.transform.common.TimestampNow\n```\n\nThis transformation is used to override the timestamp of the incoming record to the time the record is being processed.\n### Configuration\n\n\n\n## [TimestampNowField](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/TimestampNowField.html)\n\n*Key*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.TimestampNowField$Key\n```\n*Value*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.TimestampNowField$Value\n```\n\nThis transformation is used to set a field with the current timestamp of the system running the transformation.\n\n[✍️ Example](https://rmoff.net/2020/12/23/twelve-days-of-smt-day-12-community-transformations/#_add_the_current_timestamp_to_the_message_payload) / [🎥 Video](https://www.youtube.com/watch?v=Z7k_6vGRrkc\u0026t=679s)\n\n\n### Configuration\n\n#### General\n\n\n##### `fields`\n\nThe field(s) that will be inserted with the timestamp of the system.\n\n*Importance:* HIGH\n\n*Type:* LIST\n\n\n\n\n## [ToJSON](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/ToJSON.html)\n\n*Key*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.ToJSON$Key\n```\n*Value*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.ToJSON$Value\n```\n\n\n### Configuration\n\n#### General\n\n\n##### `output.schema.type`\n\nThe connect schema type to output the converted JSON as.\n\n*Importance:* MEDIUM\n\n*Type:* STRING\n\n*Default Value:* STRING\n\n*Validator:* [STRING, BYTES]\n\n\n\n##### `schemas.enable`\n\nFlag to determine if the JSON data should include the schema.\n\n*Importance:* MEDIUM\n\n*Type:* BOOLEAN\n\n\n\n\n## [ToLong](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/ToLong.html)\n\n*Key*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.ToLong$Key\n```\n*Value*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.ToLong$Value\n```\n\n\n### Configuration\n\n#### General\n\n\n##### `fields`\n\nThe fields to transform.\n\n*Importance:* HIGH\n\n*Type:* LIST\n\n\n\n\n## [TopicNameToField](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-transform-common/transformations/TopicNameToField.html)\n\n*Key*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.TopicNameToField$Key\n```\n*Value*\n```\ncom.github.jcustenborder.kafka.connect.transform.common.TopicNameToField$Value\n```\n\n\n### Configuration\n\n#### General\n\n\n##### `field`\n\nThe field to insert the topic name.\n\n*Importance:* HIGH\n\n*Type:* STRING\n\n\n\n\n\n# Development\n\n## Building the source\n\n```bash\nmvn clean package\n```\n\n## Contributions\n\nContributions are always welcomed! Before you start any development please create an issue and\nstart a discussion. Create a pull request against your newly created issue and we're happy to see\nif we can merge your pull request. First and foremost any time you're adding code to the code base\nyou need to include test coverage. Make sure that you run `mvn clean package` before submitting your\npull to ensure that all of the tests, checkstyle rules, and the package can be successfully built.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjcustenborder%2Fkafka-connect-transform-common","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjcustenborder%2Fkafka-connect-transform-common","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjcustenborder%2Fkafka-connect-transform-common/lists"}