{"id":27248859,"url":"https://github.com/bakdata/kafka-large-message-serde","last_synced_at":"2025-04-10T23:48:19.266Z","repository":{"id":40469726,"uuid":"221921174","full_name":"bakdata/kafka-large-message-serde","owner":"bakdata","description":"A Kafka Serde that reads and writes records from and to Blob storage (S3, Azure, Google) transparently.","archived":false,"fork":false,"pushed_at":"2025-03-31T10:36:34.000Z","size":467,"stargazers_count":59,"open_issues_count":0,"forks_count":9,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-04-10T23:48:12.948Z","etag":null,"topics":["azure-blob-storage","deserialization","google-cloud-storage","kafka","kafka-streams","large-data","s3","serde","serialization","simple-storage-service"],"latest_commit_sha":null,"homepage":"https://medium.com/bakdata/processing-large-messages-with-kafka-streams-167a166ca38b","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bakdata.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-15T12:37:26.000Z","updated_at":"2025-03-31T10:36:37.000Z","dependencies_parsed_at":"2024-02-22T16:42:38.375Z","dependency_job_id":null,"html_url":"https://github.com/bakdata/kafka-large-message-serde","commit_stats":null,"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-large-message-serde","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-large-message-serde/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-large-message-serde/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-large-message-serde/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bakdata","download_url":"https://codeload.github.com/bakdata/kafka-large-message-serde/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248317725,"owners_count":21083528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure-blob-storage","deserialization","google-cloud-storage","kafka","kafka-streams","large-data","s3","serde","serialization","simple-storage-service"],"created_at":"2025-04-10T23:48:18.692Z","updated_at":"2025-04-10T23:48:19.255Z","avatar_url":"https://github.com/bakdata.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://dev.azure.com/bakdata/public/_apis/build/status/bakdata.kafka-large-message-serde?branchName=master)](https://dev.azure.com/bakdata/public/_build/latest?definitionId=20\u0026branchName=master)\n[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=com.bakdata.kafka%3Alarge-message\u0026metric=alert_status)](https://sonarcloud.io/dashboard?id=com.bakdata.kafka%3Alarge-message)\n[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=com.bakdata.kafka%3Alarge-message\u0026metric=coverage)](https://sonarcloud.io/dashboard?id=com.bakdata.kafka%3Alarge-message)\n[![Maven](https://img.shields.io/maven-central/v/com.bakdata.kafka/large-message-serde.svg)](https://search.maven.org/search?q=g:com.bakdata.kafka%20AND%20a:large-message-serde\u0026core=gav)\n\n# kafka-large-message-serde\nA Kafka Serde that reads and writes records from and to a blob storage, such as Amazon S3, Azure Blob Storage, and Google Cloud Storage, transparently.\nFormerly known as kafka-s3-backed-serde.\n\n## Getting Started\n\n### Serde\n\nYou can add kafka-large-message-serde via Maven Central.\n\n#### Gradle\n```gradle\nimplementation group: 'com.bakdata.kafka', name: 'large-message-serde', version: '2.0.0'\n```\n\n#### Maven\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.bakdata.kafka\u003c/groupId\u003e\n    \u003cartifactId\u003elarge-message-serde\u003c/artifactId\u003e\n    \u003cversion\u003e2.0.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nFor other build tools or versions, refer to the [latest version in MvnRepository](https://mvnrepository.com/artifact/com.bakdata.kafka/large-message-serde/latest).\n\n#### Usage\n\nYou can use it from your Kafka Streams application like any other Serde\n\n```java\nfinal Serde\u003cString\u003e serde = new LargeMessageSerde\u003c\u003e();\nserde.configure(Map.of(AbstractLargeMessageConfig.BASE_PATH_CONFIG, \"s3://my-bucket/\",\n        LargeMessageSerdeConfig.VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class), false);\n```\n\nThe following configuration options are available:\n\n``large.message.key.serde``\n  Key serde class to use. All serde configurations are also delegated to this serde.\n\n  * Type: class\n  * Default: `org.apache.kafka.common.serialization.Serdes$ByteArraySerde`\n  * Importance: high\n\n``large.message.value.serde``\n  Value serde class to use. All serde configurations are also delegated to this serde.\n\n  * Type: class\n  * Default: `org.apache.kafka.common.serialization.Serdes$ByteArraySerde`\n  * Importance: high\n\n``large.message.base.path``\n  Base path to store data. Must include bucket and any prefix that should be used, e.g., `s3://my-bucket/my/prefix/`. Available protocols: `s3`, `abs`.\n\n  * Type: string\n  * Default: \"\"\n  * Importance: high\n\n``large.message.max.byte.size``\n  Maximum serialized message size in bytes before messages are stored on blob storage.\n\n  * Type: int\n  * Default: 1000000\n  * Importance: medium\n\n``large.message.use.headers``\nEnable if Kafka message headers should be used to distinguish blob storage backed messages. This is disabled by default\nfor backwards compatibility but leads to increased memory usage. It is recommended to enable this option.\n\n  * Type: boolean\n  * Default: false\n  * Importance: medium\n\n``large.message.accept.no.headers``\n  Enable if messages read with no headers should be treated as non-backed messages. This allows enabling of large message\n  behavior for data that has been serialized using the wrapped serializer.\n\n  * Type: boolean\n  * Default: false\n  * Importance: medium\n\n``large.message.id.generator``\n  Class to use for generating unique object IDs. Available generators are: `com.bakdata.kafka.RandomUUIDGenerator`, `com.bakdata.kafka.Sha256HashIdGenerator`, `com.bakdata.kafka.MurmurHashIdGenerator`.\n\n  * Type: class\n  * Default: `com.bakdata.kafka.RandomUUIDGenerator`\n  * Importance: medium\n\n``large.message.s3.access.key``\n  AWS access key to use for connecting to S3. Leave empty if AWS credential provider chain or STS Assume Role provider should be used.\n\n  * Type: password\n  * Default: \"\"\n  * Importance: low\n\n``large.message.s3.secret.key``\n  AWS secret key to use for connecting to S3. Leave empty if AWS credential provider chain or STS Assume Role provider should be used.\n\n  * Type: password\n  * Default: \"\"\n  * Importance: low\n\n ``large.message.s3.sts.role.arn``\n   AWS STS role ARN to use for connecting to S3. Leave empty if AWS Basic provider or AWS credential provider chain should be used.\n\n   * Type: string\n   * Default: \"\"\n   * Importance: low\n\n\n ``large.message.s3.role.external.id``\n   AWS STS role external ID used when retrieving session credentials under an assumed role. Leave empty if AWS Basic provider or AWS credential provider chain should be used.\n\n   * Type: string\n   * Default: \"\"\n   * Importance: low\n\n ``large.message.s3.role.session.name``\n   AWS STS role session name to use when starting a session. Leave empty if AWS Basic provider or AWS credential provider chain should be used.\n\n   * Type: string\n   * Default: \"\"\n   * Importance: low\n\n``large.message.s3.jwt.path``\n   Path to an OIDC token file in JSON format (JWT) used to authenticate before AWS STS role authorisation, e.g. for EKS `/var/run/secrets/eks.amazonaws.com/serviceaccount/token`.\n\n   * Type: string\n   * Default: \"\"\n   * Importance: low\n\n``large.message.s3.region``\nS3 region to use. Leave empty if default S3 region should be used.\n\n  * Type: string\n  * Default: \"\"\n  * Importance: low\n\n``large.message.s3.endpoint``\nEndpoint to use for connection to Amazon S3. Leave empty if default S3 endpoint should be used.\n\n  * Type: string\n  * Default: \"\"\n  * Importance: low\n\n ``large.message.s3.path.style.access``\n  Enable path-style access for S3 client.\n\n  * Type: boolean\n  * Default: false\n  * Importance: low\n\n``large.message.abs.connection.string``\n  Azure connection string for connection to blob storage. Leave empty if Azure credential provider chain should be used.\n\n  * Type: password\n  * Default: \"\"\n  * Importance: low\n\n``large.message.gs.key.path``\n  Google service account key JSON path. Leave empty If the environment variable GOOGLE_APPLICATION_CREDENTIALS is set\n  or if you want to use the default service account provided by your computer engine. For more information about\n  authenticating as a service account please\n  refer to the [main documentation](https://cloud.google.com/docs/authentication/production).\n\n  * Type: string\n  * Default: \"\"\n  * Importance: low\n\n``large.message.compression.type``\n  The compression type for data stored in blob storage. The default is `none` (i.e. no compression). Valid values are `none`,\n  `gzip`, `snappy`, `lz4` and `zstd`.\n  Note: this option is only available when `large.message.use.headers` is enabled.\n\n  * Type: string\n  * Default: \"none\"\n  * Importance: low\n\n\n### Kafka Connect\n\nThis serde also comes with support for Kafka Connect.\nYou can add kafka-large-message-connect via Maven Central.\n\n#### Gradle\n```gradle\nimplementation group: 'com.bakdata.kafka', name: 'large-message-connect', version: '1.1.6'\n```\n\n#### Maven\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.bakdata.kafka\u003c/groupId\u003e\n    \u003cartifactId\u003elarge-message-connect\u003c/artifactId\u003e\n    \u003cversion\u003e2.0.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nFor other build tools or versions, refer to the [latest version in MvnRepository](https://mvnrepository.com/artifact/com.bakdata.kafka/large-message-connect/latest).\n\n#### Usage\n\nTo use it with your Kafka Connect jobs, just configure your converter as `com.bakdata.kafka.LargeMessageConverter`.\n\nIn addition to the configurations available for the serde (except `large.message.key.serde` and `large.message.value.serde`),\nyou can configure the following:\n\n``large.message.converter``\n  Converter to use. All converter configurations are also delegated to this converter.\n\n  * Type: class\n  * Default: `org.apache.kafka.connect.converters.ByteArrayConverter`\n  * Importance: high\n\nFor general guidance on how to configure Kafka Connect converters, please have a look at the [official documentation](https://docs.confluent.io/home/connect/configuring.html).\n\n### Cleaning up the bucket\n\nWe also provide a method for cleaning up all files on the blob storage associated with a topic:\n\n```java\nfinal Map\u003cString, Object\u003e properties = ...;\nfinal AbstractLargeMessageConfig config = new AbstractLargeMessageConfig(properties);\nfinal LargeMessageStoringClient storer = config.getStorer();\nstorer.deleteAllFiles(\"topic\");\n```\n\n## Development\n\nIf you want to contribute to this project, you can simply clone the repository and build it via Gradle.\nAll dependencies should be included in the Gradle files, there are no external prerequisites.\n\n```bash\n\u003e git clone git@github.com:bakdata/kafka-large-message-serde.git\n\u003e cd kafka-large-message-serde \u0026\u0026 ./gradlew build\n```\n\nPlease note, that we have [code styles](https://github.com/bakdata/bakdata-code-styles) for Java.\nThey are basically the Google style guide, with some small modifications.\n\n## Contributing\n\nWe are happy if you want to contribute to this project.\nIf you find any bugs or have suggestions for improvements, please open an issue.\nWe are also happy to accept your PRs.\nJust open an issue beforehand and let us know what you want to do and why.\n\n## License\nThis project is licensed under the MIT license.\nHave a look at the [LICENSE](https://github.com/bakdata/kafka-large-message-serde/blob/master/LICENSE) for more details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fkafka-large-message-serde","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbakdata%2Fkafka-large-message-serde","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fkafka-large-message-serde/lists"}