{"id":29179702,"url":"https://github.com/trendyol/kafka-cronsumer","last_synced_at":"2025-07-01T19:06:22.772Z","repository":{"id":62125714,"uuid":"514849016","full_name":"Trendyol/kafka-cronsumer","owner":"Trendyol","description":"Cron based Kafka exception consumer with the power of auto retry \u0026 concurrency","archived":false,"fork":false,"pushed_at":"2025-06-10T07:38:50.000Z","size":785,"stargazers_count":86,"open_issues_count":4,"forks_count":14,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-06-10T08:34:26.995Z","etag":null,"topics":["cron","exception-handling","go","golang","kafka"],"latest_commit_sha":null,"homepage":"https://medium.com/trendyol-tech/kafka-konsumer-two-years-journey-3e00b46c9ea3","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Trendyol.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":"CODE-OF-CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-07-17T13:19:24.000Z","updated_at":"2025-06-10T07:38:54.000Z","dependencies_parsed_at":"2023-11-08T11:32:35.041Z","dependency_job_id":"696dd750-392b-43c4-a062-b6fdade555cf","html_url":"https://github.com/Trendyol/kafka-cronsumer","commit_stats":{"total_commits":165,"total_committers":19,"mean_commits":8.68421052631579,"dds":0.303030303030303,"last_synced_commit":"d261a9a4ffd2fef5963688ea6fb85d71b0441147"},"previous_names":[],"tags_count":34,"template":false,"template_full_name":null,"purl":"pkg:github/Trendyol/kafka-cronsumer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Trendyol%2Fkafka-cronsumer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Trendyol%2Fkafka-cronsumer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Trendyol%2Fkafka-cronsumer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Trendyol%2Fkafka-cronsumer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Trendyol","download_url":"https://codeload.github.com/Trendyol/kafka-cronsumer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Trendyol%2Fkafka-cronsumer/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263021821,"owners_count":23401148,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cron","exception-handling","go","golang","kafka"],"created_at":"2025-07-01T19:06:08.503Z","updated_at":"2025-07-01T19:06:22.753Z","avatar_url":"https://github.com/Trendyol.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka C[r]onsumer [![Go Reference](https://pkg.go.dev/badge/github.com/Trendyol/kafka-cronsumer.svg)](https://pkg.go.dev/github.com/Trendyol/kafka-cronsumer) [![Go Report Card](https://goreportcard.com/badge/github.com/Trendyol/kafka-cronsumer)](https://goreportcard.com/report/github.com/Trendyol/kafka-cronsumer)\n\n\u003cdiv style=\"text-align:center\"\u003e\u003cimg src=\".github/images/cronsumer.png\"/\u003e\u003c/div\u003e\n\n## Description\n\nKafka Cronsumer is mainly used for retry/exception strategy management.\nIt works based on cron expression and consumes messages in a timely manner\nwith the power of auto pause and concurrency. \n\n[For details check our blog post](https://medium.com/trendyol-tech/kafka-exception-c-r-onsumer-37c459e4849d)\n\n#### If you need a whole consumer lifecycle with exception management, check [Kafka Konsumer](https://github.com/Trendyol/kafka-konsumer)\n\n## How Kafka Cronsumer Works\n\n![How Kafka Cronsumer Works](.github/images/architecture.png)\n\n## When to use it?\n\n- Iteration-based back-off strategies are applicable\n- Messages could be processed in an eventually consistent state\n- Max retry exceeded messages could be ignored and send to dead letter topic\n- To increase consumer resiliency\n- To increase consumer performance with concurrency\n\n## When to avoid?\n\n- Messages should be processed in order\n- Messages should be certainly processed (we discard messages if max retry is exceeded)\n- Messages should be committed (we use auto-commit interval for increasing performance)\n- Messages with TTL (Time to Live)\n\n## Guide\n\n### Installation\n\n```sh\ngo get github.com/Trendyol/kafka-cronsumer@latest\n```\n\n### Examples\n\nYou can find a number of ready-to-run examples at [this directory](examples).\n\nAfter running `docker-compose up` command, you can run any application you want. \nDon't forget its cron based :)\n\n#### Single Consumer\n\n```go\nfunc main() {\n // ...\n var consumeFn kafka.ConsumeFn = func (message kafka.Message) error {\n fmt.Printf(\"consumer \u003e Message received: %s\\n\", string(message.Value))\n return nil\n }\n\n c := cronsumer.New(kafkaConfig, consumeFn)\n c.Run()\n}\n```\n\n#### Single Consumer With Dead Letter\n\n```go\nfunc main() {\n // ...\n var consumeFn kafka.ConsumeFn = func (message kafka.Message) error {\n fmt.Printf(\"consumer \u003e Message received: %s\\n\", string(message.Value))\n return errors.New(\"error occurred\")\n }\n\n c := cronsumer.New(kafkaConfig, consumeFn)\n c.Run()\n}\n```\n\n#### Multiple Consumers\n\n```go\nfunc main() {\n // ...\n var firstConsumerFn kafka.ConsumeFn = func (message kafka.Message) error {\n fmt.Printf(\"First consumer \u003e Message received: %s\\n\", string(message.Value))\n return nil\n }\n first := cronsumer.New(firstCfg, firstConsumerFn)\n first.Start()\n\n var secondConsumerFn kafka.ConsumeFn = func (message kafka.Message) error {\n fmt.Printf(\"Second consumer \u003e Message received: %s\\n\", string(message.Value))\n return nil\n }\n second := cronsumer.New(secondCfg, secondConsumerFn)\n second.Start()\n // ... \n}\n```\n\n#### Single Consumer With Metric collector\n\n```go\nfunc main() {\n // ...\n var consumeFn kafka.ConsumeFn = func(message kafka.Message) error {\n return errors.New(\"err occurred\")\n }\n \n c := cronsumer.New(config, consumeFn)\n StartAPI(*config, c.GetMetricCollectors()...)\n c.Start()\n // ... \n}\n\nfunc StartAPI(cfg kafka.Config, metricCollectors ...prometheus.Collector) {\n // ...\n f := fiber.New(\n fiber.Config{},\n )\n \n metricMiddleware, err := NewMetricMiddleware(cfg, f, metricCollectors...)\n \n f.Use(metricMiddleware)\n // ...\n}\n```\n\n## Configurations\n\n| config | description | default | example |\n|----------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------|-------------------------------------------|\n| `logLevel` | Describes log level, valid options are `debug`, `info`, `warn`, and `error` | info | |\n| `metricPrefix` | MetricPrefix is used for prometheus fq name prefix. If not provided, default metric prefix value is `kafka_cronsumer`. Currently, there are two exposed prometheus metrics. `retried_messages_total_current` and `discarded_messages_total_current`. So, if default metric prefix used, metrics names are `kafka_cronsumer_retried_messages_total_current` and `kafka_cronsumer_discarded_messages_total_current` | kafka_cronsumer | |\n| `consumer.clientId` | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#Dialer) | | |\n| `consumer.cron` | Cron expression when exception consumer starts to work at | | */1 * * * * |\n| `consumer.backOffStrategy` | Define consumer backoff strategy for retry topics | fixed | exponential, linear |\n| `consumer.duration` | Work duration exception consumer actively consuming messages | NonStopWork (zero duration) | 20s, 15m, 1h, NonStopWork (zero duration) |\n| `consumer.brokers` | broker address | | |\n| `consumer.topic` | Exception topic names | | exception-topic |\n| `consumer.groupId` | Exception consumer group id | | exception-consumer-group |\n| `consumer.maxRetry` | Maximum retry value for attempting to retry a message | 3 | |\n| `consumer.concurrency` | Number of goroutines used at listeners | 1 | |\n| `consumer.minBytes` | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.MinBytes) | 1 | |\n| `consumer.maxBytes` | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.MaxBytes) | 1 MB | |\n| `consumer.maxWait` | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.MaxWait) | 10s | |\n| `consumer.commitInterval` | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.CommitInterval) | 1s | |\n| `consumer.heartbeatInterval` | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.HeartbeatInterval) | 3s | |\n| `consumer.sessionTimeout` | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.SessionTimeout) | 30s | |\n| `consumer.rebalanceTimeout` | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.RebalanceTimeout) | 30s | |\n| `consumer.startOffset` | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.StartOffset) | earliest | |\n| `consumer.retentionTime` | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#ReaderConfig.RetentionTime) | 24h | |\n| `consumer.queueCapacity` | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.42#ReaderConfig.QueueCapacity) | 100 | |\n| `consumer.skipMessageByHeaderFn` | Function to filter messages based on headers, return true if you want to skip the message | nil | |\n| `producer.clientId` | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#Transport) | | |\n| `producer.brokers` | Broker address if it is not given, uses consumer.Brokers addr | consumer.Brokers addr | |\n| `producer.batchSize` | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#Writer.BatchSize) | 100 | |\n| `producer.batchTimeout` | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#Writer.BatchTimeout) | 1s | |\n| `producer.balancer` | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go#Balancer) | leastBytes | |\n| `sasl.enabled` | It enables sasl authentication mechanism | false | |\n| `sasl.authType` | Currently we only support `SCRAM` | \"\" | |\n| `sasl.username` | SCRAM username | \"\" | |\n| `sasl.password` | SCRAM password | \"\" | |\n| `sasl.rootCAPath` | [see doc](https://pkg.go.dev/crypto/tls#Config.RootCAs) | \"\" | |\n| `sasl.intermediateCAPath` | | \"\" | |\n| `sasl.rack` | [see doc](https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.47#RackAffinityGroupBalancer) | \"\" | |\n\n### Exposed Metrics\n\n| Metric Name | Description | Value Type |\n|------------------------------------------|--------------------------------------|------------|\n| kafka_cronsumer_retried_messages_total | Total number of retried messages. | Counter |\n| kafka_cronsumer_discarded_messages_total | Total number of discarded messages. | Counter |\n\n\n## Contribute\n\n**Use issues for everything**\n\n- For a small change, just send a PR.\n- For bigger changes open an issue for discussion before sending a PR.\n- PR should have:\n - Test case\n - Documentation\n - Example (If it makes sense)\n- You can also contribute by:\n - Reporting issues\n - Suggesting new features or enhancements\n - Improve/fix documentation\n\nPlease adhere to this project's `code of conduct`.\n\n## Maintainers\n\n- [@Abdulsametileri](https://github.com/Abdulsametileri)\n- [@emreodabas](https://github.com/emreodabas)\n\n## Code of Conduct\n\n[Contributor Code of Conduct](CODE-OF-CONDUCT.md). By participating in this project you agree to abide by its terms.\n\n## Libraries Used For This Project\n\n- [segmentio/kafka-go](https://github.com/segmentio/kafka-go)\n- [robfig/cron](https://github.com/robfig/cron)\n- [uber-go/zap](https://github.com/uber-go/zap)\n\n## Additional References\n\n- [Kcat](https://github.com/edenhill/kcat)\n- [jq](https://stedolan.github.io/jq/)\n- [golangci-lint](https://github.com/golangci/golangci-lint)\n- [Kafka Console Producer](https://kafka.apache.org/quickstart)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrendyol%2Fkafka-cronsumer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrendyol%2Fkafka-cronsumer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrendyol%2Fkafka-cronsumer/lists"}