{"id":27248826,"url":"https://github.com/bakdata/kafka-connect-plugins","last_synced_at":"2025-07-12T16:36:40.286Z","repository":{"id":221196238,"uuid":"600042371","full_name":"bakdata/kafka-connect-plugins","owner":"bakdata","description":"A collection Kafka Connect plugins","archived":false,"fork":false,"pushed_at":"2025-04-07T05:34:40.000Z","size":179,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-10T23:48:10.509Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bakdata.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-10T13:04:42.000Z","updated_at":"2025-04-07T05:34:43.000Z","dependencies_parsed_at":"2024-02-22T15:51:22.870Z","dependency_job_id":null,"html_url":"https://github.com/bakdata/kafka-connect-plugins","commit_stats":null,"previous_names":["bakdata/kafka-connect-plugins"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-connect-plugins","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-connect-plugins/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-connect-plugins/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-connect-plugins/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bakdata","download_url":"https://codeload.github.com/bakdata/kafka-connect-plugins/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248317727,"owners_count":21083528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-10T23:48:12.011Z","updated_at":"2025-04-10T23:48:12.504Z","avatar_url":"https://github.com/bakdata.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://dev.azure.com/bakdata/public/_apis/build/status/bakdata.kafka-connect-plugins?branchName=main)](https://dev.azure.com/bakdata/public/_build/latest?definitionId=35\u0026branchName=main)\n[![Sonarcloud status](https://sonarcloud.io/api/project_badges/measure?project=com.bakdata.kafka%3Akafka-connect-plugins\u0026metric=alert_status)](https://sonarcloud.io/project/overview?id=com.bakdata.kafka:kafka-connect-plugins)\n[![Code coverage](https://sonarcloud.io/api/project_badges/measure?project=com.bakdata.kafka%3Akafka-connect-plugins\u0026metric=coverage)](https://sonarcloud.io/project/overview?id=com.bakdata.kafka:kafka-connect-plugins)\n[![Maven](https://img.shields.io/maven-central/v/com.bakdata.kafka/kafka-connect-plugins.svg)](https://search.maven.org/search?q=g:com.bakdata.kafka%20AND%20a:kafka-connect-plugins\u0026core=gav)\n\n# Kafka Connect plugins\n\nA collection of Kafka Connect plugins.\n\n## Single Message Transforms (SMTs)\n\n### Convert\n\n#### Description\n\nConverts a byte record using the given converter class.\nThe [MirrorMaker](https://github.com/apache/kafka/blob/trunk/connect/mirror/README.md)\nconnector uses byte array records.\nTo apply other SMTs to these records,\nwe need to convert them to the appropriate format first.\n\nUse the concrete transformation type designed for the record key (`com.bakdata.kafka.Convert$Key`)\nor value (`com.bakdata.kafka.Convert$Value`).\n\n#### Example\n\nThis configuration snippet shows how to use `Convert`.\nIt converts the value to a string schema.\n\n```yaml\n\"transforms\": \"convert\",\n\"transforms.convert.type\": \"com.bakdata.kafka.Convert$Value\",\n\"transforms.convert.converter\": \"org.apache.kafka.connect.storage.StringConverter\"\n```\n\n#### Properties\n\n| Name        | Description                  | Type  | Default                    | Valid Values                                                                                                                                    | Importance |\n|-------------|------------------------------|-------|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------|------------|\n| `converter` | Converter to apply to input. | class | `ByteArrayConverter.class` | All classes that implement the [Kafka Converter interface](https://kafka.apache.org/34/javadoc/org/apache/kafka/connect/storage/Converter.html) | high       |\n\n### Drop field\n\n#### Description\n\nDrop any (nested) field for a given path.\n\nUse the concrete transformation type designed for the record key (`com.bakdata.kafka.DropField$Key`)\nor value (`com.bakdata.kafka.DropField$Value`).\n\n#### Example\n\nThis example shows how to configure and use `DropField`.\n\nImagine you have the following record value:\n\n```json\n{\n  \"collections\": [\n    {\n      \"complex_field\": {\n        \"dropped_field\": \"This field will be dropped.\",\n        \"kept_field\": 1234\n      },\n      \"boolean_field\": true\n    },\n    {\n      \"complex_field\": {\n        \"dropped_field\": \"This field will also be dropped.\",\n        \"kept_field\": 5678\n      },\n      \"boolean_field\": false\n    }\n  ],\n  \"primitive_field\": 9876\n}\n```\n\nThis configuration snippet shows how to use `DropField` to exclude the field `dropped_field`.\n\n```yaml\n\"transforms\": \"dropfield\",\n\"transforms.dropfield.type\": \"com.bakdata.kafka.DropField$Value\",\n\"transforms.dropfield.exclude\": \"collections.complex_field.dropped_field\"\n```\n\nThe value would transform into this:\n\n```json\n{\n  \"collections\": [\n    {\n      \"complex_field\": {\n        \"kept_field\": 1234\n      },\n      \"boolean_field\": true\n    },\n    {\n      \"complex_field\": {\n        \"kept_field\": 5678\n      },\n      \"boolean_field\": false\n    }\n  ],\n  \"primitive_field\": 9876\n}\n```\n\n#### Properties\n\n| Name      | Description                                         | Type     | Default | Valid Values                                              | Importance |\n|-----------|-----------------------------------------------------|----------|---------|-----------------------------------------------------------|------------|\n| `exclude` | Path to field to exclude from the resulting Struct. | `string` | -       | The path is separated by \".\" character. Example: `a.b.c`. | high       |\n\n## Predicates\n\n### NullPredicate\n\n#### Description\n\nPredicate testing that the value or a field of the value is null.\n\n#### Example\n\nThis configuration snippet shows how to use `NullPredicate`.\nIt drops all messages where `field1` is null:\n\n```yaml\n\"transforms\": \"filter\"\n\"transforms.filter.type\": \"org.apache.kafka.connect.transforms.Filter\"\n\"transforms.filter.predicate\": \"field1NonNull\"\n\"transforms.predicate.field1NonNull.type\": \"com.bakdata.kafka.NullPredicate\"\n\"transforms.predicate.field1NonNull.field\": \"field1\"\n```\n\n#### Properties\n\n| Name    | Description                                                                       | Type   | Default | Valid Values                      | Importance |\n|---------|-----------------------------------------------------------------------------------|--------|---------|-----------------------------------|------------|\n| `field` | Name of the field to check for null. If not provided, the whole value is checked. | String | `\"\"`    | All fields of the record's schema | medium     |\n\n## Installation\n\nIf you are using Docker to run Kafka Connect,\nyou can install the SMT by adding the JAR file to your Kafka Connect image.\nFor example:\n\n```dockerfile\nFROM confluentinc/cp-kafka-connect:latest\n\n# Install your source/sink connector(s)\n# ...\n\nENV CONNECT_PLUGIN_PATH=\"/connect-plugins,/usr/share/java\"\n\n# Clone the repo and build the project first. \n# Or download the JAR file from Sonatype.\nCOPY ./build/libs/*.jar /connect-plugins/kafka-connect-transformations/\n```\n\n## Development\n\nIf you want to contribute to this project, you can simply clone the repository and build it via Gradle.\nAll dependencies should be included in the Gradle files, there are no external prerequisites.\n\n```bash\n\u003e git clone git@github.com:bakdata/kafka-connect-plugins.git\n\u003e cd kafka-connect-plugins \u0026\u0026 ./gradlew build\n```\n\nPlease note, that we have [code styles](https://github.com/bakdata/bakdata-code-styles) for Java.\nThey are basically the Google style guide, with some small modifications.\n\n## Contributing\n\nWe are happy if you want to contribute to this project.\nIf you find any bugs or have suggestions for improvements, please open an issue.\nWe are also happy to accept your PRs.\nJust open an issue beforehand and let us know what you want to do and why.\n\n## License\n\nThis project is licensed under the MIT license.\nHave a look at the [LICENSE](https://github.com/bakdata/kafka-connect-plugins/blob/master/LICENSE) for more details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fkafka-connect-plugins","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbakdata%2Fkafka-connect-plugins","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fkafka-connect-plugins/lists"}