{"id":13562635,"url":"https://github.com/bsm/sarama-cluster","last_synced_at":"2025-04-03T18:34:08.463Z","repository":{"id":24740363,"uuid":"28152807","full_name":"bsm/sarama-cluster","owner":"bsm","description":"Cluster extensions for Sarama, the Go client library for Apache Kafka 0.9 [DEPRECATED]","archived":true,"fork":false,"pushed_at":"2020-01-08T11:57:22.000Z","size":339,"stargazers_count":1007,"open_issues_count":1,"forks_count":224,"subscribers_count":47,"default_branch":"master","last_synced_at":"2025-03-14T07:03:21.566Z","etag":null,"topics":["consumer","deprecated","go","kafka","sarama"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bsm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-12-17T19:50:39.000Z","updated_at":"2025-02-16T06:00:56.000Z","dependencies_parsed_at":"2022-08-07T11:01:18.853Z","dependency_job_id":null,"html_url":"https://github.com/bsm/sarama-cluster","commit_stats":null,"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bsm%2Fsarama-cluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bsm%2Fsarama-cluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bsm%2Fsarama-cluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bsm%2Fsarama-cluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bsm","download_url":"https://codeload.github.com/bsm/sarama-cluster/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247057191,"owners_count":20876532,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consumer","deprecated","go","kafka","sarama"],"created_at":"2024-08-01T13:01:10.586Z","updated_at":"2025-04-03T18:34:03.453Z","avatar_url":"https://github.com/bsm.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# Sarama Cluster\n\n[![GoDoc](https://godoc.org/github.com/bsm/sarama-cluster?status.svg)](https://godoc.org/github.com/bsm/sarama-cluster)\n[![Build Status](https://travis-ci.org/bsm/sarama-cluster.svg?branch=master)](https://travis-ci.org/bsm/sarama-cluster)\n[![Go Report Card](https://goreportcard.com/badge/github.com/bsm/sarama-cluster)](https://goreportcard.com/report/github.com/bsm/sarama-cluster)\n[![License](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)\n\nCluster extensions for [Sarama](https://github.com/Shopify/sarama), the Go client library for Apache Kafka 0.9 (and later).\n\n## DEPRECATION NOTICE\n\nPlease note that since https://github.com/Shopify/sarama/pull/1099 was merged and released (\u003e= v1.19.0) this library is officially deprecated. The native implementation supports a variety of use cases that are not available through this library.\n\n## Documentation\n\nDocumentation and example are available via godoc at http://godoc.org/github.com/bsm/sarama-cluster\n\n## Examples\n\nConsumers have two modes of operation. In the default multiplexed mode messages (and errors) of multiple\ntopics and partitions are all passed to the single channel:\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\t\"os/signal\"\n\n\tcluster \"github.com/bsm/sarama-cluster\"\n)\n\nfunc main() {\n\n\t// init (custom) config, enable errors and notifications\n\tconfig := cluster.NewConfig()\n\tconfig.Consumer.Return.Errors = true\n\tconfig.Group.Return.Notifications = true\n\n\t// init consumer\n\tbrokers := []string{\"127.0.0.1:9092\"}\n\ttopics := []string{\"my_topic\", \"other_topic\"}\n\tconsumer, err := cluster.NewConsumer(brokers, \"my-consumer-group\", topics, config)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tdefer consumer.Close()\n\n\t// trap SIGINT to trigger a shutdown.\n\tsignals := make(chan os.Signal, 1)\n\tsignal.Notify(signals, os.Interrupt)\n\n\t// consume errors\n\tgo func() {\n\t\tfor err := range consumer.Errors() {\n\t\t\tlog.Printf(\"Error: %s\\n\", err.Error())\n\t\t}\n\t}()\n\n\t// consume notifications\n\tgo func() {\n\t\tfor ntf := range consumer.Notifications() {\n\t\t\tlog.Printf(\"Rebalanced: %+v\\n\", ntf)\n\t\t}\n\t}()\n\n\t// consume messages, watch signals\n\tfor {\n\t\tselect {\n\t\tcase msg, ok := \u003c-consumer.Messages():\n\t\t\tif ok {\n\t\t\t\tfmt.Fprintf(os.Stdout, \"%s/%d/%d\\t%s\\t%s\\n\", msg.Topic, msg.Partition, msg.Offset, msg.Key, msg.Value)\n\t\t\t\tconsumer.MarkOffset(msg, \"\")\t// mark message as processed\n\t\t\t}\n\t\tcase \u003c-signals:\n\t\t\treturn\n\t\t}\n\t}\n}\n```\n\nUsers who require access to individual partitions can use the partitioned mode which exposes access to partition-level\nconsumers:\n\n```go\npackage main\n\nimport (\n  \"fmt\"\n  \"log\"\n  \"os\"\n  \"os/signal\"\n\n  cluster \"github.com/bsm/sarama-cluster\"\n)\n\nfunc main() {\n\n\t// init (custom) config, set mode to ConsumerModePartitions\n\tconfig := cluster.NewConfig()\n\tconfig.Group.Mode = cluster.ConsumerModePartitions\n\n\t// init consumer\n\tbrokers := []string{\"127.0.0.1:9092\"}\n\ttopics := []string{\"my_topic\", \"other_topic\"}\n\tconsumer, err := cluster.NewConsumer(brokers, \"my-consumer-group\", topics, config)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tdefer consumer.Close()\n\n\t// trap SIGINT to trigger a shutdown.\n\tsignals := make(chan os.Signal, 1)\n\tsignal.Notify(signals, os.Interrupt)\n\n\t// consume partitions\n\tfor {\n\t\tselect {\n\t\tcase part, ok := \u003c-consumer.Partitions():\n\t\t\tif !ok {\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\t// start a separate goroutine to consume messages\n\t\t\tgo func(pc cluster.PartitionConsumer) {\n\t\t\t\tfor msg := range pc.Messages() {\n\t\t\t\t\tfmt.Fprintf(os.Stdout, \"%s/%d/%d\\t%s\\t%s\\n\", msg.Topic, msg.Partition, msg.Offset, msg.Key, msg.Value)\n\t\t\t\t\tconsumer.MarkOffset(msg, \"\")\t// mark message as processed\n\t\t\t\t}\n\t\t\t}(part)\n\t\tcase \u003c-signals:\n\t\t\treturn\n\t\t}\n\t}\n}\n```\n\n## Running tests\n\nYou need to install Ginkgo \u0026 Gomega to run tests. Please see\nhttp://onsi.github.io/ginkgo for more details.\n\nTo run tests, call:\n\n\t$ make test\n\n## Troubleshooting\n\n### Consumer not receiving any messages?\n\nBy default, sarama's `Config.Consumer.Offsets.Initial` is set to `sarama.OffsetNewest`. This means that in the event that a brand new consumer is created, and it has never committed any offsets to kafka, it will only receive messages starting from the message after the current one that was written.\n\nIf you wish to receive all messages (from the start of all messages in the topic) in the event that a consumer does not have any offsets committed to kafka, you need to set `Config.Consumer.Offsets.Initial` to `sarama.OffsetOldest`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbsm%2Fsarama-cluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbsm%2Fsarama-cluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbsm%2Fsarama-cluster/lists"}