https://github.com/trendyol/kafka-cronsumer

Cron based Kafka exception consumer with the power of auto retry & concurrency
https://github.com/trendyol/kafka-cronsumer

cron exception-handling go golang kafka

Last synced: 12 months ago
JSON representation

Cron based Kafka exception consumer with the power of auto retry & concurrency

Host: GitHub
URL: https://github.com/trendyol/kafka-cronsumer
Owner: Trendyol
License: mit
Created: 2022-07-17T13:19:24.000Z (almost 4 years ago)
Default Branch: main
Last Pushed: 2025-06-10T07:38:50.000Z (about 1 year ago)
Last Synced: 2025-06-10T08:34:26.995Z (about 1 year ago)
Topics: cron, exception-handling, go, golang, kafka
Language: Go
Homepage: https://medium.com/trendyol-tech/kafka-konsumer-two-years-journey-3e00b46c9ea3
Size: 767 KB
Stars: 86
Watchers: 5
Forks: 14
Open Issues: 4
Metadata Files:
- Readme: README.md
- Code of conduct: CODE-OF-CONDUCT.md
- Security: SECURITY.md

Awesome Lists containing this project

README

# Kafka C[r]onsumer [![Go Reference](https://pkg.go.dev/badge/github.com/Trendyol/kafka-cronsumer.svg)](https://pkg.go.dev/github.com/Trendyol/kafka-cronsumer) [![Go Report Card](https://goreportcard.com/badge/github.com/Trendyol/kafka-cronsumer)](https://goreportcard.com/report/github.com/Trendyol/kafka-cronsumer)

## Description

Kafka Cronsumer is mainly used for retry/exception strategy management.
It works based on cron expression and consumes messages in a timely manner
with the power of auto pause and concurrency.

[For details check our blog post](https://medium.com/trendyol-tech/kafka-exception-c-r-onsumer-37c459e4849d)

#### If you need a whole consumer lifecycle with exception management, check [Kafka Konsumer](https://github.com/Trendyol/kafka-konsumer)

## How Kafka Cronsumer Works

![How Kafka Cronsumer Works](.github/images/architecture.png)

## When to use it?

- Iteration-based back-off strategies are applicable
- Messages could be processed in an eventually consistent state
- Max retry exceeded messages could be ignored and send to dead letter topic
- To increase consumer resiliency
- To increase consumer performance with concurrency

## When to avoid?

- Messages should be processed in order
- Messages should be certainly processed (we discard messages if max retry is exceeded)
- Messages should be committed (we use auto-commit interval for increasing performance)
- Messages with TTL (Time to Live)

## Guide

### Installation

```sh
go get github.com/Trendyol/kafka-cronsumer@latest
```

### Examples

You can find a number of ready-to-run examples at [this directory](examples).

After running `docker-compose up` command, you can run any application you want.
Don't forget its cron based :)

#### Single Consumer

```go
func main() {
// ...
var consumeFn kafka.ConsumeFn = func (message kafka.Message) error {
fmt.Printf("consumer > Message received: %s\n", string(message.Value))
return nil
}

c := cronsumer.New(kafkaConfig, consumeFn)
c.Run()
}
```

#### Single Consumer With Dead Letter

```go
func main() {
// ...
var consumeFn kafka.ConsumeFn = func (message kafka.Message) error {
fmt.Printf("consumer > Message received: %s\n", string(message.Value))
return errors.New("error occurred")
}

c := cronsumer.New(kafkaConfig, consumeFn)
c.Run()
}
```

#### Multiple Consumers

```go
func main() {
// ...
var firstConsumerFn kafka.ConsumeFn = func (message kafka.Message) error {
fmt.Printf("First consumer > Message received: %s\n", string(message.Value))
return nil
}
first := cronsumer.New(firstCfg, firstConsumerFn)
first.Start()

var secondConsumerFn kafka.ConsumeFn = func (message kafka.Message) error {
fmt.Printf("Second consumer > Message received: %s\n", string(message.Value))
return nil
}
second := cronsumer.New(secondCfg, secondConsumerFn)
second.Start()
// ...
}
```

#### Single Consumer With Metric collector

```go
func main() {
// ...
var consumeFn kafka.ConsumeFn = func(message kafka.Message) error {
return errors.New("err occurred")
}

c := cronsumer.New(config, consumeFn)
StartAPI(*config, c.GetMetricCollectors()...)
c.Start()
// ...
}

func StartAPI(cfg kafka.Config, metricCollectors ...prometheus.Collector) {
// ...
f := fiber.New(
fiber.Config{},
)

metricMiddleware, err := NewMetricMiddleware(cfg, f, metricCollectors...)

f.Use(metricMiddleware)
// ...
}
```

## Configurations

| config
|--------------------
| `logLevel`
| `metricPrefix`
| `consumer.clientId`
| `consumer.cron`
| `consumer.backOffStrategy`
| `consumer.duration`
| `consumer.brokers`
| `consumer.topic`
| `consumer.groupId`
| `consumer.maxRetry`
| `consumer.concurrency`
| `consumer.minBytes`
| `consumer.maxBytes`
| `consumer.maxWait`
| `consumer.commitInterval`
| `consumer.heartbeatInterval`
| `consumer.sessionTimeout`
| `consumer.rebalanceTimeout`
| `consumer.startOffset`
| `consumer.retentionTime`
| `consumer.queueCapacity`
| `consumer.skipMessageByHeaderFn`
| `producer.clientId`
| `producer.brokers`
| `producer.batchSize`
| `producer.batchTimeout`
| `producer.balancer`
| `sasl.enabled`
| `sasl.authType`
| `sasl.username`
| `sasl.password`
| `sasl.rootCAPath`
| `sasl.intermediateCAPath`
| `sasl.rack` | description | default | example | --------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------|-------------------------------------------| | Describes log level, valid options are `debug`, `info`, `warn`, and `error` | info | | | MetricPrefix is used for prometheus fq name prefix. If not provided, default metric prefix value is `kafka_cronsumer`. Currently, there are two exposed prometheus metrics. `retried_messages_total_current` and `discarded_messages_total_current`. So, if default metric prefix used, metrics names are `kafka_cronsumer_retried_messages_total_current` and `kafka_cronsumer_discarded_messages_total_current` | kafka_cronsumer | | | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#Dialer) | | | | Cron expression when exception consumer starts to work at | | */1 * * * * | | Define consumer backoff strategy for retry topics | fixed | exponential, linear | | Work duration exception consumer actively consuming messages | NonStopWork (zero duration) | 20s, 15m, 1h, NonStopWork (zero duration) | | broker address | | | | Exception topic names | | exception-topic | | Exception consumer group id | | exception-consumer-group | | Maximum retry value for attempting to retry a message | 3 | | | Number of goroutines used at listeners | 1 | | | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.MinBytes) | 1 | | | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.MaxBytes) | 1 MB | | | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.MaxWait) | 10s | | | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.CommitInterval) | 1s | | | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.HeartbeatInterval) | 3s | | | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.SessionTimeout) | 30s | | | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.RebalanceTimeout) | 30s | | | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.StartOffset) | earliest | | | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.RetentionTime) | 24h | | | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.42#ReaderConfig.QueueCapacity) | 100 | | | Function to filter messages based on headers, return true if you want to skip the message | nil | | | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#Transport) | | | | Broker address if it is not given, uses consumer.Brokers addr | consumer.Brokers addr | | | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#Writer.BatchSize) | 100 | | | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#Writer.BatchTimeout) | 1s | | | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go#Balancer) | leastBytes | | | It enables sasl authentication mechanism | false | | | Currently we only support `SCRAM` | "" | | | SCRAM username | "" | | | SCRAM password | "" | | | [see doc](https://pkg.go.dev/crypto/tls#Config.RootCAs) | "" | | | | "" | | | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#RackAffinityGroupBalancer) | "" | |

### Exposed Metrics

| Metric Name | Description | Value Type |
|------------------------------------------|--------------------------------------|------------|
| kafka_cronsumer_retried_messages_total | Total number of retried messages. | Counter |
| kafka_cronsumer_discarded_messages_total | Total number of discarded messages. | Counter |

## Contribute

**Use issues for everything**

- For a small change, just send a PR.
- For bigger changes open an issue for discussion before sending a PR.
- PR should have:
- Test case
- Documentation
- Example (If it makes sense)
- You can also contribute by:
- Reporting issues
- Suggesting new features or enhancements
- Improve/fix documentation

Please adhere to this project's `code of conduct`.

## Maintainers

- [@Abdulsametileri](https://github.com/Abdulsametileri)
- [@emreodabas](https://github.com/emreodabas)

## Code of Conduct

[Contributor Code of Conduct](CODE-OF-CONDUCT.md). By participating in this project you agree to abide by its terms.

## Libraries Used For This Project

- [segmentio/kafka-go](https://github.com/segmentio/kafka-go)
- [robfig/cron](https://github.com/robfig/cron)
- [uber-go/zap](https://github.com/uber-go/zap)

## Additional References

- [Kcat](https://github.com/edenhill/kcat)
- [jq](https://stedolan.github.io/jq/)
- [golangci-lint](https://github.com/golangci/golangci-lint)
- [Kafka Console Producer](https://kafka.apache.org/quickstart)

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/trendyol/kafka-cronsumer

Awesome Lists containing this project

README