{"id":13416804,"url":"https://github.com/segmentio/kafka-go","last_synced_at":"2026-01-16T23:27:48.932Z","repository":{"id":37462614,"uuid":"90798757","full_name":"segmentio/kafka-go","owner":"segmentio","description":"Kafka library in Go","archived":false,"fork":false,"pushed_at":"2026-01-15T16:21:20.000Z","size":2049,"stargazers_count":8430,"open_issues_count":252,"forks_count":849,"subscribers_count":86,"default_branch":"main","last_synced_at":"2026-01-15T19:37:59.586Z","etag":null,"topics":["consumer","go","golang","golang-library","golang-native","kafka","kafka-client","producer","segment"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/segmentio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2017-05-09T22:50:27.000Z","updated_at":"2026-01-15T16:19:51.000Z","dependencies_parsed_at":"2023-11-15T21:27:31.208Z","dependency_job_id":"d83f2ad8-0ee1-4b05-918a-3fa5171ae273","html_url":"https://github.com/segmentio/kafka-go","commit_stats":{"total_commits":759,"total_committers":152,"mean_commits":4.993421052631579,"dds":0.6600790513833992,"last_synced_commit":"ebca72eaee918d303c532feb7ff29afdcd8c2efa"},"previous_names":[],"tags_count":72,"template":false,"template_full_name":null,"purl":"pkg:github/segmentio/kafka-go","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segmentio%2Fkafka-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segmentio%2Fkafka-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segmentio%2Fkafka-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segmentio%2Fkafka-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/segmentio","download_url":"https://codeload.github.com/segmentio/kafka-go/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segmentio%2Fkafka-go/sbom","scorecard":{"id":677177,"data":{"date":"2025-08-11","repo":{"name":"github.com/segmentio/kafka-go","commit":"af1725fb4fc0d856653afb2aeaeaa3cbfea18aec"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4.2,"checks":[{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":10,"reason":"all changesets reviewed","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: examples/consumer-logger/Dockerfile:4","Warn: containerImage not pinned by hash: examples/consumer-mongo-db/Dockerfile:4","Warn: containerImage not pinned by hash: examples/producer-api/Dockerfile:4","Warn: containerImage not pinned by hash: examples/producer-random/Dockerfile:4","Info:   0 out of   4 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":4,"reason":"branch protection is not maximal on development and all release branches","details":["Info: 'allow deletion' disabled on branch 'main'","Info: 'force pushes' disabled on branch 'main'","Warn: 'branch protection settings apply to administrators' is disabled on branch 'main'","Info: 'stale review dismissal' is required to merge on branch 'main'","Warn: required approving review count is 1 on branch 'main'","Warn: codeowners review is required - but no codeowners file found in repo","Warn: 'last push approval' is disabled on branch 'main'","Warn: 'up-to-date branches' is disabled on branch 'main'","Info: status check found to merge onto on branch 'main'","Info: PRs are required in order to make changes on branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":8,"reason":"2 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GO-2022-0635","Warn: Project is vulnerable to: GO-2022-0646"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-21T21:53:23.576Z","repository_id":37462614,"created_at":"2025-08-21T21:53:23.577Z","updated_at":"2025-08-21T21:53:23.577Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28487586,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T22:54:02.790Z","status":"ssl_error","status_checked_at":"2026-01-16T22:50:10.344Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consumer","go","golang","golang-library","golang-native","kafka","kafka-client","producer","segment"],"created_at":"2024-07-30T22:00:22.456Z","updated_at":"2026-01-16T23:27:48.909Z","avatar_url":"https://github.com/segmentio.png","language":"Go","funding_links":[],"categories":["Go","Popular","Misc","Middleware","Repositories","Clients"],"sub_categories":["Go"],"readme":"# kafka-go [![CircleCI](https://circleci.com/gh/segmentio/kafka-go.svg?style=shield)](https://circleci.com/gh/segmentio/kafka-go) [![Go Report Card](https://goreportcard.com/badge/github.com/segmentio/kafka-go)](https://goreportcard.com/report/github.com/segmentio/kafka-go) [![GoDoc](https://godoc.org/github.com/segmentio/kafka-go?status.svg)](https://godoc.org/github.com/segmentio/kafka-go)\n\n## Motivations\n\nWe rely on both Go and Kafka a lot at Segment. Unfortunately, the state of the Go\nclient libraries for Kafka at the time of this writing was not ideal. The available\noptions were:\n\n- [sarama](https://github.com/Shopify/sarama), which is by far the most popular\nbut is quite difficult to work with. It is poorly documented, the API exposes\nlow level concepts of the Kafka protocol, and it doesn't support recent Go features\nlike [contexts](https://golang.org/pkg/context/). It also passes all values as\npointers which causes large numbers of dynamic memory allocations, more frequent\ngarbage collections, and higher memory usage.\n\n- [confluent-kafka-go](https://github.com/confluentinc/confluent-kafka-go) is a\ncgo based wrapper around [librdkafka](https://github.com/edenhill/librdkafka),\nwhich means it introduces a dependency to a C library on all Go code that uses\nthe package. It has much better documentation than sarama but still lacks support\nfor Go contexts.\n\n- [goka](https://github.com/lovoo/goka) is a more recent Kafka client for Go\nwhich focuses on a specific usage pattern. It provides abstractions for using Kafka\nas a message passing bus between services rather than an ordered log of events, but\nthis is not the typical use case of Kafka for us at Segment. The package also\ndepends on sarama for all interactions with Kafka.\n\nThis is where `kafka-go` comes into play. It provides both low and high level\nAPIs for interacting with Kafka, mirroring concepts and implementing interfaces of\nthe Go standard library to make it easy to use and integrate with existing\nsoftware.\n\n#### Note:\n\nIn order to better align with our newly adopted Code of Conduct, the kafka-go\nproject has renamed our default branch to `main`. For the full details of our\nCode Of Conduct see [this](./CODE_OF_CONDUCT.md) document.\n\n## Kafka versions\n\n`kafka-go` is currently tested with Kafka versions 0.10.1.0 to 2.7.1.\nWhile it should also be compatible with later versions, newer features available\nin the Kafka API may not yet be implemented in the client.\n\n## Go versions\n\n`kafka-go` requires Go version 1.15 or later.\n\n## Connection [![GoDoc](https://godoc.org/github.com/segmentio/kafka-go?status.svg)](https://godoc.org/github.com/segmentio/kafka-go#Conn)\n\nThe `Conn` type is the core of the `kafka-go` package. It wraps around a raw\nnetwork connection to expose a low-level API to a Kafka server.\n\nHere are some examples showing typical use of a connection object:\n```go\n// to produce messages\ntopic := \"my-topic\"\npartition := 0\n\nconn, err := kafka.DialLeader(context.Background(), \"tcp\", \"localhost:9092\", topic, partition)\nif err != nil {\n    log.Fatal(\"failed to dial leader:\", err)\n}\n\nconn.SetWriteDeadline(time.Now().Add(10*time.Second))\n_, err = conn.WriteMessages(\n    kafka.Message{Value: []byte(\"one!\")},\n    kafka.Message{Value: []byte(\"two!\")},\n    kafka.Message{Value: []byte(\"three!\")},\n)\nif err != nil {\n    log.Fatal(\"failed to write messages:\", err)\n}\n\nif err := conn.Close(); err != nil {\n    log.Fatal(\"failed to close writer:\", err)\n}\n```\n```go\n// to consume messages\ntopic := \"my-topic\"\npartition := 0\n\nconn, err := kafka.DialLeader(context.Background(), \"tcp\", \"localhost:9092\", topic, partition)\nif err != nil {\n    log.Fatal(\"failed to dial leader:\", err)\n}\n\nconn.SetReadDeadline(time.Now().Add(10*time.Second))\nbatch := conn.ReadBatch(10e3, 1e6) // fetch 10KB min, 1MB max\n\nb := make([]byte, 10e3) // 10KB max per message\nfor {\n    n, err := batch.Read(b)\n    if err != nil {\n        break\n    }\n    fmt.Println(string(b[:n]))\n}\n\nif err := batch.Close(); err != nil {\n    log.Fatal(\"failed to close batch:\", err)\n}\n\nif err := conn.Close(); err != nil {\n    log.Fatal(\"failed to close connection:\", err)\n}\n```\n\n### To Create Topics\nBy default kafka has the `auto.create.topics.enable='true'` (`KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE='true'` in the bitnami/kafka kafka docker image). If this value is set to `'true'` then topics will be created as a side effect of `kafka.DialLeader` like so:\n```go\n// to create topics when auto.create.topics.enable='true'\nconn, err := kafka.DialLeader(context.Background(), \"tcp\", \"localhost:9092\", \"my-topic\", 0)\nif err != nil {\n    panic(err.Error())\n}\n```\n\nIf `auto.create.topics.enable='false'` then you will need to create topics explicitly like so:\n```go\n// to create topics when auto.create.topics.enable='false'\ntopic := \"my-topic\"\n\nconn, err := kafka.Dial(\"tcp\", \"localhost:9092\")\nif err != nil {\n    panic(err.Error())\n}\ndefer conn.Close()\n\ncontroller, err := conn.Controller()\nif err != nil {\n    panic(err.Error())\n}\nvar controllerConn *kafka.Conn\ncontrollerConn, err = kafka.Dial(\"tcp\", net.JoinHostPort(controller.Host, strconv.Itoa(controller.Port)))\nif err != nil {\n    panic(err.Error())\n}\ndefer controllerConn.Close()\n\n\ntopicConfigs := []kafka.TopicConfig{\n    {\n        Topic:             topic,\n        NumPartitions:     1,\n        ReplicationFactor: 1,\n    },\n}\n\nerr = controllerConn.CreateTopics(topicConfigs...)\nif err != nil {\n    panic(err.Error())\n}\n```\n\n### To Connect To Leader Via a Non-leader Connection\n```go\n// to connect to the kafka leader via an existing non-leader connection rather than using DialLeader\nconn, err := kafka.Dial(\"tcp\", \"localhost:9092\")\nif err != nil {\n    panic(err.Error())\n}\ndefer conn.Close()\ncontroller, err := conn.Controller()\nif err != nil {\n    panic(err.Error())\n}\nvar connLeader *kafka.Conn\nconnLeader, err = kafka.Dial(\"tcp\", net.JoinHostPort(controller.Host, strconv.Itoa(controller.Port)))\nif err != nil {\n    panic(err.Error())\n}\ndefer connLeader.Close()\n```\n\n### To list topics\n```go\nconn, err := kafka.Dial(\"tcp\", \"localhost:9092\")\nif err != nil {\n    panic(err.Error())\n}\ndefer conn.Close()\n\npartitions, err := conn.ReadPartitions()\nif err != nil {\n    panic(err.Error())\n}\n\nm := map[string]struct{}{}\n\nfor _, p := range partitions {\n    m[p.Topic] = struct{}{}\n}\nfor k := range m {\n    fmt.Println(k)\n}\n```\n\n\nBecause it is low level, the `Conn` type turns out to be a great building block\nfor higher level abstractions, like the `Reader` for example.\n\n## Reader [![GoDoc](https://godoc.org/github.com/segmentio/kafka-go?status.svg)](https://godoc.org/github.com/segmentio/kafka-go#Reader)\n\nA `Reader` is another concept exposed by the `kafka-go` package, which intends\nto make it simpler to implement the typical use case of consuming from a single\ntopic-partition pair.\nA `Reader` also automatically handles reconnections and offset management, and\nexposes an API that supports asynchronous cancellations and timeouts using Go\ncontexts.\n\nNote that it is important to call `Close()` on a `Reader` when a process exits.\nThe kafka server needs a graceful disconnect to stop it from continuing to\nattempt to send messages to the connected clients. The given example will not\ncall `Close()` if the process is terminated with SIGINT (ctrl-c at the shell) or\nSIGTERM (as docker stop or a kubernetes restart does). This can result in a\ndelay when a new reader on the same topic connects (e.g. new process started\nor new container running). Use a `signal.Notify` handler to close the reader on\nprocess shutdown.\n\n```go\n// make a new reader that consumes from topic-A, partition 0, at offset 42\nr := kafka.NewReader(kafka.ReaderConfig{\n    Brokers:   []string{\"localhost:9092\",\"localhost:9093\", \"localhost:9094\"},\n    Topic:     \"topic-A\",\n    Partition: 0,\n    MaxBytes:  10e6, // 10MB\n})\nr.SetOffset(42)\n\nfor {\n    m, err := r.ReadMessage(context.Background())\n    if err != nil {\n        break\n    }\n    fmt.Printf(\"message at offset %d: %s = %s\\n\", m.Offset, string(m.Key), string(m.Value))\n}\n\nif err := r.Close(); err != nil {\n    log.Fatal(\"failed to close reader:\", err)\n}\n```\n\n### Consumer Groups\n\n```kafka-go``` also supports Kafka consumer groups including broker managed offsets.\nTo enable consumer groups, simply specify the GroupID in the ReaderConfig.\n\nReadMessage automatically commits offsets when using consumer groups.\n\n```go\n// make a new reader that consumes from topic-A\nr := kafka.NewReader(kafka.ReaderConfig{\n    Brokers:   []string{\"localhost:9092\", \"localhost:9093\", \"localhost:9094\"},\n    GroupID:   \"consumer-group-id\",\n    Topic:     \"topic-A\",\n    MaxBytes:  10e6, // 10MB\n})\n\nfor {\n    m, err := r.ReadMessage(context.Background())\n    if err != nil {\n        break\n    }\n    fmt.Printf(\"message at topic/partition/offset %v/%v/%v: %s = %s\\n\", m.Topic, m.Partition, m.Offset, string(m.Key), string(m.Value))\n}\n\nif err := r.Close(); err != nil {\n    log.Fatal(\"failed to close reader:\", err)\n}\n```\n\nThere are a number of limitations when using consumer groups:\n\n* ```(*Reader).SetOffset``` will return an error when GroupID is set\n* ```(*Reader).Offset``` will always return ```-1``` when GroupID is set\n* ```(*Reader).Lag``` will always return ```-1``` when GroupID is set\n* ```(*Reader).ReadLag``` will return an error when GroupID is set\n* ```(*Reader).Stats``` will return a partition of ```-1``` when GroupID is set\n\n### Explicit Commits\n\n```kafka-go``` also supports explicit commits.  Instead of calling ```ReadMessage```,\ncall ```FetchMessage``` followed by ```CommitMessages```.\n\n```go\nctx := context.Background()\nfor {\n    m, err := r.FetchMessage(ctx)\n    if err != nil {\n        break\n    }\n    fmt.Printf(\"message at topic/partition/offset %v/%v/%v: %s = %s\\n\", m.Topic, m.Partition, m.Offset, string(m.Key), string(m.Value))\n    if err := r.CommitMessages(ctx, m); err != nil {\n        log.Fatal(\"failed to commit messages:\", err)\n    }\n}\n```\n\nWhen committing messages in consumer groups, the message with the highest offset\nfor a given topic/partition determines the value of the committed offset for\nthat partition. For example, if messages at offset 1, 2, and 3 of a single\npartition were retrieved by call to `FetchMessage`, calling `CommitMessages`\nwith message offset 3 will also result in committing the messages at offsets 1\nand 2 for that partition.\n\n### Managing Commits\n\nBy default, CommitMessages will synchronously commit offsets to Kafka.  For\nimproved performance, you can instead periodically commit offsets to Kafka\nby setting CommitInterval on the ReaderConfig.\n\n\n```go\n// make a new reader that consumes from topic-A\nr := kafka.NewReader(kafka.ReaderConfig{\n    Brokers:        []string{\"localhost:9092\", \"localhost:9093\", \"localhost:9094\"},\n    GroupID:        \"consumer-group-id\",\n    Topic:          \"topic-A\",\n    MaxBytes:       10e6, // 10MB\n    CommitInterval: time.Second, // flushes commits to Kafka every second\n})\n```\n\n## Writer [![GoDoc](https://godoc.org/github.com/segmentio/kafka-go?status.svg)](https://godoc.org/github.com/segmentio/kafka-go#Writer)\n\nTo produce messages to Kafka, a program may use the low-level `Conn` API, but\nthe package also provides a higher level `Writer` type which is more appropriate\nto use in most cases as it provides additional features:\n\n- Automatic retries and reconnections on errors.\n- Configurable distribution of messages across available partitions.\n- Synchronous or asynchronous writes of messages to Kafka.\n- Asynchronous cancellation using contexts.\n- Flushing of pending messages on close to support graceful shutdowns.\n- Creation of a missing topic before publishing a message. *Note!* it was the default behaviour up to the version `v0.4.30`.\n\n```go\n// make a writer that produces to topic-A, using the least-bytes distribution\nw := \u0026kafka.Writer{\n\tAddr:     kafka.TCP(\"localhost:9092\", \"localhost:9093\", \"localhost:9094\"),\n\tTopic:   \"topic-A\",\n\tBalancer: \u0026kafka.LeastBytes{},\n}\n\nerr := w.WriteMessages(context.Background(),\n\tkafka.Message{\n\t\tKey:   []byte(\"Key-A\"),\n\t\tValue: []byte(\"Hello World!\"),\n\t},\n\tkafka.Message{\n\t\tKey:   []byte(\"Key-B\"),\n\t\tValue: []byte(\"One!\"),\n\t},\n\tkafka.Message{\n\t\tKey:   []byte(\"Key-C\"),\n\t\tValue: []byte(\"Two!\"),\n\t},\n)\nif err != nil {\n    log.Fatal(\"failed to write messages:\", err)\n}\n\nif err := w.Close(); err != nil {\n    log.Fatal(\"failed to close writer:\", err)\n}\n```\n\n### Missing topic creation before publication\n\n```go\n// Make a writer that publishes messages to topic-A.\n// The topic will be created if it is missing.\nw := \u0026Writer{\n    Addr:                   kafka.TCP(\"localhost:9092\", \"localhost:9093\", \"localhost:9094\"),\n    Topic:                  \"topic-A\",\n    AllowAutoTopicCreation: true,\n}\n\nmessages := []kafka.Message{\n    {\n        Key:   []byte(\"Key-A\"),\n        Value: []byte(\"Hello World!\"),\n    },\n    {\n        Key:   []byte(\"Key-B\"),\n        Value: []byte(\"One!\"),\n    },\n    {\n        Key:   []byte(\"Key-C\"),\n        Value: []byte(\"Two!\"),\n    },\n}\n\nvar err error\nconst retries = 3\nfor i := 0; i \u003c retries; i++ {\n    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)\n    defer cancel()\n    \n    // attempt to create topic prior to publishing the message\n    err = w.WriteMessages(ctx, messages...)\n    if errors.Is(err, kafka.LeaderNotAvailable) || errors.Is(err, context.DeadlineExceeded) {\n        time.Sleep(time.Millisecond * 250)\n        continue\n    }\n\n    if err != nil {\n        log.Fatalf(\"unexpected error %v\", err)\n    }\n    break\n}\n\nif err := w.Close(); err != nil {\n    log.Fatal(\"failed to close writer:\", err)\n}\n```\n\n### Writing to multiple topics\n\nNormally, the `WriterConfig.Topic` is used to initialize a single-topic writer.\nBy excluding that particular configuration, you are given the ability to define\nthe topic on a per-message basis by setting `Message.Topic`.\n\n```go\nw := \u0026kafka.Writer{\n\tAddr:     kafka.TCP(\"localhost:9092\", \"localhost:9093\", \"localhost:9094\"),\n    // NOTE: When Topic is not defined here, each Message must define it instead.\n\tBalancer: \u0026kafka.LeastBytes{},\n}\n\nerr := w.WriteMessages(context.Background(),\n    // NOTE: Each Message has Topic defined, otherwise an error is returned.\n\tkafka.Message{\n        Topic: \"topic-A\",\n\t\tKey:   []byte(\"Key-A\"),\n\t\tValue: []byte(\"Hello World!\"),\n\t},\n\tkafka.Message{\n        Topic: \"topic-B\",\n\t\tKey:   []byte(\"Key-B\"),\n\t\tValue: []byte(\"One!\"),\n\t},\n\tkafka.Message{\n        Topic: \"topic-C\",\n\t\tKey:   []byte(\"Key-C\"),\n\t\tValue: []byte(\"Two!\"),\n\t},\n)\nif err != nil {\n    log.Fatal(\"failed to write messages:\", err)\n}\n\nif err := w.Close(); err != nil {\n    log.Fatal(\"failed to close writer:\", err)\n}\n```\n\n**NOTE:** These 2 patterns are mutually exclusive, if you set `Writer.Topic`,\nyou must not also explicitly define `Message.Topic` on the messages you are\nwriting. The opposite applies when you do not define a topic for the writer.\nThe `Writer` will return an error if it detects this ambiguity.\n\n### Compatibility with other clients\n\n#### Sarama\n\nIf you're switching from Sarama and need/want to use the same algorithm for message partitioning, you can either use \nthe `kafka.Hash` balancer or the `kafka.ReferenceHash` balancer:\n* `kafka.Hash` = `sarama.NewHashPartitioner`\n* `kafka.ReferenceHash` = `sarama.NewReferenceHashPartitioner`\n\nThe `kafka.Hash` and `kafka.ReferenceHash` balancers would route messages to the same partitions that the two \naforementioned Sarama partitioners would route them to.\n\n```go\nw := \u0026kafka.Writer{\n\tAddr:     kafka.TCP(\"localhost:9092\", \"localhost:9093\", \"localhost:9094\"),\n\tTopic:    \"topic-A\",\n\tBalancer: \u0026kafka.Hash{},\n}\n```\n\n#### librdkafka and confluent-kafka-go\n\nUse the ```kafka.CRC32Balancer``` balancer to get the same behaviour as librdkafka's\ndefault ```consistent_random``` partition strategy.\n\n```go\nw := \u0026kafka.Writer{\n\tAddr:     kafka.TCP(\"localhost:9092\", \"localhost:9093\", \"localhost:9094\"),\n\tTopic:    \"topic-A\",\n\tBalancer: kafka.CRC32Balancer{},\n}\n```\n\n#### Java\n\nUse the ```kafka.Murmur2Balancer``` balancer to get the same behaviour as the canonical\nJava client's default partitioner.  Note: the Java class allows you to directly specify\nthe partition which is not permitted.\n\n```go\nw := \u0026kafka.Writer{\n\tAddr:     kafka.TCP(\"localhost:9092\", \"localhost:9093\", \"localhost:9094\"),\n\tTopic:    \"topic-A\",\n\tBalancer: kafka.Murmur2Balancer{},\n}\n```\n\n### Compression\n\nCompression can be enabled on the `Writer` by setting the `Compression` field:\n\n```go\nw := \u0026kafka.Writer{\n\tAddr:        kafka.TCP(\"localhost:9092\", \"localhost:9093\", \"localhost:9094\"),\n\tTopic:       \"topic-A\",\n\tCompression: kafka.Snappy,\n}\n```\n\nThe `Reader` will by determine if the consumed messages are compressed by\nexamining the message attributes.  However, the package(s) for all expected\ncodecs must be imported so that they get loaded correctly.\n\n_Note: in versions prior to 0.4 programs had to import compression packages to\ninstall codecs and support reading compressed messages from kafka. This is no\nlonger the case and import of the compression packages are now no-ops._\n\n## TLS Support\n\nFor a bare bones Conn type or in the Reader/Writer configs you can specify a dialer option for TLS support. If the TLS field is nil, it will not connect with TLS.\n*Note:* Connecting to a Kafka cluster with TLS enabled without configuring TLS on the Conn/Reader/Writer can manifest in opaque io.ErrUnexpectedEOF errors.\n\n\n### Connection\n\n```go\ndialer := \u0026kafka.Dialer{\n    Timeout:   10 * time.Second,\n    DualStack: true,\n    TLS:       \u0026tls.Config{...tls config...},\n}\n\nconn, err := dialer.DialContext(ctx, \"tcp\", \"localhost:9093\")\n```\n\n### Reader\n\n```go\ndialer := \u0026kafka.Dialer{\n    Timeout:   10 * time.Second,\n    DualStack: true,\n    TLS:       \u0026tls.Config{...tls config...},\n}\n\nr := kafka.NewReader(kafka.ReaderConfig{\n    Brokers:        []string{\"localhost:9092\", \"localhost:9093\", \"localhost:9094\"},\n    GroupID:        \"consumer-group-id\",\n    Topic:          \"topic-A\",\n    Dialer:         dialer,\n})\n```\n\n### Writer\n\n\nDirect Writer creation\n\n```go\nw := kafka.Writer{\n    Addr: kafka.TCP(\"localhost:9092\", \"localhost:9093\", \"localhost:9094\"), \n    Topic:   \"topic-A\",\n    Balancer: \u0026kafka.Hash{},\n    Transport: \u0026kafka.Transport{\n        TLS: \u0026tls.Config{},\n      },\n    }\n```\n\nUsing `kafka.NewWriter`\n\n```go\ndialer := \u0026kafka.Dialer{\n    Timeout:   10 * time.Second,\n    DualStack: true,\n    TLS:       \u0026tls.Config{...tls config...},\n}\n\nw := kafka.NewWriter(kafka.WriterConfig{\n\tBrokers: []string{\"localhost:9092\", \"localhost:9093\", \"localhost:9094\"},\n\tTopic:   \"topic-A\",\n\tBalancer: \u0026kafka.Hash{},\n\tDialer:   dialer,\n})\n```\nNote that `kafka.NewWriter` and `kafka.WriterConfig` are deprecated and will be removed in a future release.\n\n## SASL Support\n\nYou can specify an option on the `Dialer` to use SASL authentication. The `Dialer` can be used directly to open a `Conn` or it can be passed to a `Reader` or `Writer` via their respective configs. If the `SASLMechanism` field is `nil`, it will not authenticate with SASL.\n\n### SASL Authentication Types\n\n#### [Plain](https://godoc.org/github.com/segmentio/kafka-go/sasl/plain#Mechanism)\n```go\nmechanism := plain.Mechanism{\n    Username: \"username\",\n    Password: \"password\",\n}\n```\n\n#### [SCRAM](https://godoc.org/github.com/segmentio/kafka-go/sasl/scram#Mechanism)\n```go\nmechanism, err := scram.Mechanism(scram.SHA512, \"username\", \"password\")\nif err != nil {\n    panic(err)\n}\n```\n\n### Connection\n\n```go\nmechanism, err := scram.Mechanism(scram.SHA512, \"username\", \"password\")\nif err != nil {\n    panic(err)\n}\n\ndialer := \u0026kafka.Dialer{\n    Timeout:       10 * time.Second,\n    DualStack:     true,\n    SASLMechanism: mechanism,\n}\n\nconn, err := dialer.DialContext(ctx, \"tcp\", \"localhost:9093\")\n```\n\n\n### Reader\n\n```go\nmechanism, err := scram.Mechanism(scram.SHA512, \"username\", \"password\")\nif err != nil {\n    panic(err)\n}\n\ndialer := \u0026kafka.Dialer{\n    Timeout:       10 * time.Second,\n    DualStack:     true,\n    SASLMechanism: mechanism,\n}\n\nr := kafka.NewReader(kafka.ReaderConfig{\n    Brokers:        []string{\"localhost:9092\",\"localhost:9093\", \"localhost:9094\"},\n    GroupID:        \"consumer-group-id\",\n    Topic:          \"topic-A\",\n    Dialer:         dialer,\n})\n```\n\n### Writer\n\n```go\nmechanism, err := scram.Mechanism(scram.SHA512, \"username\", \"password\")\nif err != nil {\n    panic(err)\n}\n\n// Transports are responsible for managing connection pools and other resources,\n// it's generally best to create a few of these and share them across your\n// application.\nsharedTransport := \u0026kafka.Transport{\n    SASL: mechanism,\n}\n\nw := kafka.Writer{\n\tAddr:      kafka.TCP(\"localhost:9092\", \"localhost:9093\", \"localhost:9094\"),\n\tTopic:     \"topic-A\",\n\tBalancer:  \u0026kafka.Hash{},\n\tTransport: sharedTransport,\n}\n```\n\n### Client\n\n```go\nmechanism, err := scram.Mechanism(scram.SHA512, \"username\", \"password\")\nif err != nil {\n    panic(err)\n}\n\n// Transports are responsible for managing connection pools and other resources,\n// it's generally best to create a few of these and share them across your\n// application.\nsharedTransport := \u0026kafka.Transport{\n    SASL: mechanism,\n}\n\nclient := \u0026kafka.Client{\n    Addr:      kafka.TCP(\"localhost:9092\", \"localhost:9093\", \"localhost:9094\"),\n    Timeout:   10 * time.Second,\n    Transport: sharedTransport,\n}\n```\n\n#### Reading all messages within a time range\n\n```go\nstartTime := time.Now().Add(-time.Hour)\nendTime := time.Now()\nbatchSize := int(10e6) // 10MB\n\nr := kafka.NewReader(kafka.ReaderConfig{\n    Brokers:   []string{\"localhost:9092\", \"localhost:9093\", \"localhost:9094\"},\n    Topic:     \"my-topic1\",\n    Partition: 0,\n    MaxBytes:  batchSize,\n})\n\nr.SetOffsetAt(context.Background(), startTime)\n\nfor {\n    m, err := r.ReadMessage(context.Background())\n\n    if err != nil {\n        break\n    }\n    if m.Time.After(endTime) {\n        break\n    }\n    // TODO: process message\n    fmt.Printf(\"message at offset %d: %s = %s\\n\", m.Offset, string(m.Key), string(m.Value))\n}\n\nif err := r.Close(); err != nil {\n    log.Fatal(\"failed to close reader:\", err)\n}\n```\n\n\n## Logging\n\nFor visiblity into the operations of the Reader/Writer types, configure a logger on creation.\n\n\n### Reader\n\n```go\nfunc logf(msg string, a ...interface{}) {\n\tfmt.Printf(msg, a...)\n\tfmt.Println()\n}\n\nr := kafka.NewReader(kafka.ReaderConfig{\n\tBrokers:     []string{\"localhost:9092\", \"localhost:9093\", \"localhost:9094\"},\n\tTopic:       \"my-topic1\",\n\tPartition:   0,\n\tLogger:      kafka.LoggerFunc(logf),\n\tErrorLogger: kafka.LoggerFunc(logf),\n})\n```\n\n### Writer\n\n```go\nfunc logf(msg string, a ...interface{}) {\n\tfmt.Printf(msg, a...)\n\tfmt.Println()\n}\n\nw := \u0026kafka.Writer{\n\tAddr:        kafka.TCP(\"localhost:9092\"),\n\tTopic:       \"topic\",\n\tLogger:      kafka.LoggerFunc(logf),\n\tErrorLogger: kafka.LoggerFunc(logf),\n}\n```\n\n\n\n## Testing\n\nSubtle behavior changes in later Kafka versions have caused some historical tests to break, if you are running against Kafka 2.3.1 or later, exporting the `KAFKA_SKIP_NETTEST=1` environment variables will skip those tests.\n\nRun Kafka locally in docker\n\n```bash\ndocker-compose up -d\n```\n\nRun tests\n\n```bash\nKAFKA_VERSION=2.3.1 \\\n  KAFKA_SKIP_NETTEST=1 \\\n  go test -race ./...\n```\n\n(or) to clean up the cached test results and run tests:\n```\ngo clean -cache \u0026\u0026 make test\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsegmentio%2Fkafka-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsegmentio%2Fkafka-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsegmentio%2Fkafka-go/lists"}