https://github.com/bakdata/kafka-large-message-serde

A Kafka Serde that reads and writes records from and to Blob storage (S3, Azure, Google) transparently.
https://github.com/bakdata/kafka-large-message-serde

azure-blob-storage deserialization google-cloud-storage kafka kafka-streams large-data s3 serde serialization simple-storage-service

Last synced: 2 months ago
JSON representation

A Kafka Serde that reads and writes records from and to Blob storage (S3, Azure, Google) transparently.

Host: GitHub
URL: https://github.com/bakdata/kafka-large-message-serde
Owner: bakdata
License: mit
Created: 2019-11-15T12:37:26.000Z (over 5 years ago)
Default Branch: master
Last Pushed: 2025-03-31T10:36:34.000Z (2 months ago)
Last Synced: 2025-04-10T23:48:12.948Z (2 months ago)
Topics: azure-blob-storage, deserialization, google-cloud-storage, kafka, kafka-streams, large-data, s3, serde, serialization, simple-storage-service
Language: Java
Homepage: https://medium.com/bakdata/processing-large-messages-with-kafka-streams-167a166ca38b
Size: 456 KB
Stars: 59
Watchers: 8
Forks: 9
Open Issues: 0
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE

Awesome Lists containing this project

README

[![Build Status](https://dev.azure.com/bakdata/public/_apis/build/status/bakdata.kafka-large-message-serde?branchName=master)](https://dev.azure.com/bakdata/public/_build/latest?definitionId=20&branchName=master)
[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=com.bakdata.kafka%3Alarge-message&metric=alert_status)](https://sonarcloud.io/dashboard?id=com.bakdata.kafka%3Alarge-message)
[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=com.bakdata.kafka%3Alarge-message&metric=coverage)](https://sonarcloud.io/dashboard?id=com.bakdata.kafka%3Alarge-message)
[![Maven](https://img.shields.io/maven-central/v/com.bakdata.kafka/large-message-serde.svg)](https://search.maven.org/search?q=g:com.bakdata.kafka%20AND%20a:large-message-serde&core=gav)

# kafka-large-message-serde
A Kafka Serde that reads and writes records from and to a blob storage, such as Amazon S3, Azure Blob Storage, and Google Cloud Storage, transparently.
Formerly known as kafka-s3-backed-serde.

## Getting Started

### Serde

You can add kafka-large-message-serde via Maven Central.

#### Gradle
```gradle
implementation group: 'com.bakdata.kafka', name: 'large-message-serde', version: '2.0.0'
```

#### Maven
```xml

com.bakdata.kafka
large-message-serde
2.0.0

```

For other build tools or versions, refer to the [latest version in MvnRepository](https://mvnrepository.com/artifact/com.bakdata.kafka/large-message-serde/latest).

#### Usage

You can use it from your Kafka Streams application like any other Serde

```java
final Serde serde = new LargeMessageSerde<>();
serde.configure(Map.of(AbstractLargeMessageConfig.BASE_PATH_CONFIG, "s3://my-bucket/",
LargeMessageSerdeConfig.VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class), false);
```

The following configuration options are available:

``large.message.key.serde``
Key serde class to use. All serde configurations are also delegated to this serde.

* Type: class
* Default: `org.apache.kafka.common.serialization.Serdes$ByteArraySerde`
* Importance: high

``large.message.value.serde``
Value serde class to use. All serde configurations are also delegated to this serde.

* Type: class
* Default: `org.apache.kafka.common.serialization.Serdes$ByteArraySerde`
* Importance: high

``large.message.base.path``
Base path to store data. Must include bucket and any prefix that should be used, e.g., `s3://my-bucket/my/prefix/`. Available protocols: `s3`, `abs`.

* Type: string
* Default: ""
* Importance: high

``large.message.max.byte.size``
Maximum serialized message size in bytes before messages are stored on blob storage.

* Type: int
* Default: 1000000
* Importance: medium

``large.message.use.headers``
Enable if Kafka message headers should be used to distinguish blob storage backed messages. This is disabled by default
for backwards compatibility but leads to increased memory usage. It is recommended to enable this option.

* Type: boolean
* Default: false
* Importance: medium

``large.message.accept.no.headers``
Enable if messages read with no headers should be treated as non-backed messages. This allows enabling of large message
behavior for data that has been serialized using the wrapped serializer.

* Type: boolean
* Default: false
* Importance: medium

``large.message.id.generator``
Class to use for generating unique object IDs. Available generators are: `com.bakdata.kafka.RandomUUIDGenerator`, `com.bakdata.kafka.Sha256HashIdGenerator`, `com.bakdata.kafka.MurmurHashIdGenerator`.

* Type: class
* Default: `com.bakdata.kafka.RandomUUIDGenerator`
* Importance: medium

``large.message.s3.access.key``
AWS access key to use for connecting to S3. Leave empty if AWS credential provider chain or STS Assume Role provider should be used.

* Type: password
* Default: ""
* Importance: low

``large.message.s3.secret.key``
AWS secret key to use for connecting to S3. Leave empty if AWS credential provider chain or STS Assume Role provider should be used.

* Type: password
* Default: ""
* Importance: low

``large.message.s3.sts.role.arn``
AWS STS role ARN to use for connecting to S3. Leave empty if AWS Basic provider or AWS credential provider chain should be used.