https://github.com/blankscreen-exe/kafka-docker-deadletterqueue
This is a Pumpjack Data Works internship task where we have to setup a system which will keep the data stream from getting lost before going into the Kafka Broker and alert the producer to resend that specific data packet. All things networking included.
https://github.com/blankscreen-exe/kafka-docker-deadletterqueue
deadletter deadletter-queue kafka kafka-consumer kafka-producer python3
Last synced: 3 months ago
JSON representation
This is a Pumpjack Data Works internship task where we have to setup a system which will keep the data stream from getting lost before going into the Kafka Broker and alert the producer to resend that specific data packet. All things networking included.
- Host: GitHub
- URL: https://github.com/blankscreen-exe/kafka-docker-deadletterqueue
- Owner: Blankscreen-exe
- Created: 2022-09-18T17:00:38.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2022-09-25T13:46:27.000Z (over 2 years ago)
- Last Synced: 2025-01-08T20:45:50.155Z (4 months ago)
- Topics: deadletter, deadletter-queue, kafka, kafka-consumer, kafka-producer, python3
- Language: Python
- Homepage:
- Size: 12.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Kafka DeadLetter Exchange/Delay Mechanism
## Introduction
This mechanism is introduced as a solution to data being rejected or dropped before being produced to a `Kafka-Broker Topic`.
We also consider the possibilities that a `Kafka Broker` sends a data packet but that data is not recieved by a `consumer`.
This mechanism simply resolved that issue and established a `Dead Letter Queue` as a reliable queue which will not only host a sequence of dead-letters but also the corrupted data which should be allowed to enter into a `Kafka Queue`.
## References
1- Error Handling via Dead Letter Queue in Apache Kafka (source: [kai-waehner.de](https://www.kai-waehner.de/blog/2022/05/30/error-handling-via-dead-letter-queue-in-apache-kafka/))