https://github.com/reugn/memento

Kafka messages non-blocking reprocessor and delayed producer service
https://github.com/reugn/memento

delayed-publish kafka kafka-producer kafka-producer-consumer kafka-proxy kafka-rest kafka-streams

Last synced: 21 days ago
JSON representation

Kafka messages non-blocking reprocessor and delayed producer service

• Host: GitHub
• URL: https://github.com/reugn/memento
• Owner: reugn
• License: apache-2.0
• Created: 2020-05-11T19:38:13.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2021-10-23T11:00:04.000Z (over 3 years ago)
• Last Synced: 2024-11-16T10:17:08.990Z (3 months ago)
• Topics: delayed-publish, kafka, kafka-producer, kafka-producer-consumer, kafka-proxy, kafka-rest, kafka-streams
• Language: Scala
• Homepage:
• Size: 30.3 KB
• Stars: 9
• Watchers: 5
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# memento

[![Build](https://github.com/reugn/memento/actions/workflows/build.yml/badge.svg)](https://github.com/reugn/memento/actions/workflows/build.yml)

Using [Apache Kafka](https://kafka.apache.org/), we can't keep on retrying a message without blocking the entire partition. But what if we would like

to delay/reprocess the current message and move on.

Kafka does not have a delayed message reception mechanism out of the box.

What if we could hack Kafka Streams and turn the flow into a delayed Kafka producer...

## Introduction

`memento` is a Kafka Streams application that could come in handy when you need to:

* Reprocess particular Kafka messages without blocking the partition

* Submit a delayed Kafka message

The message should contain the following headers:

* `origin` - a target topic name

* `ts` - a timestamp to emit the message

Delayed message submission over HTTP is also supported.

```

curl --location --request POST 'localhost:8080/store' \

--header 'Content-Type: application/json' \

--data-raw '{

"key": "key1",

"value": "abcd",

"origin": "test2",

"ts": 1588338000000

}'

```

The project utilizes the `KeyValueStore` as message storage. Inject your own implementation using the `Guice` module.  

More persistent KeyValueStores:

* [kafka-aerospike-state-store](https://github.com/reugn/kafka-aerospike-state-store)

## Getting started

`memento` is a Scala sbt application.  

Run locally:

```

sbt run -Dconfig.resource=application.conf

```

Build an application jar:

```

sbt assembly

```

Run from the jar:

```

java -jar assembly/memento-.jar -Dconfig.resource=conf/application.conf

```

## License

Licensed under the Apache 2.0 License.