{"id":15204317,"url":"https://github.com/figma/confluent-kafka-go-dev","last_synced_at":"2025-10-02T22:31:45.345Z","repository":{"id":57521473,"uuid":"252532749","full_name":"figma/confluent-kafka-go-dev","owner":"figma","description":"[EXPERIMENTAL] Development / WIP / exploratory / test fork of confluent-kafka-go","archived":true,"fork":true,"pushed_at":"2020-04-02T19:27:39.000Z","size":10544,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-09-29T05:41:41.846Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"confluentinc/confluent-kafka-go-dev","license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/figma.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-02T18:16:59.000Z","updated_at":"2023-01-28T15:11:54.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/figma/confluent-kafka-go-dev","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/figma%2Fconfluent-kafka-go-dev","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/figma%2Fconfluent-kafka-go-dev/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/figma%2Fconfluent-kafka-go-dev/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/figma%2Fconfluent-kafka-go-dev/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/figma","download_url":"https://codeload.github.com/figma/confluent-kafka-go-dev/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235049009,"owners_count":18927715,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-28T05:41:36.017Z","updated_at":"2025-10-02T22:31:39.078Z","avatar_url":"https://github.com/figma.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"Confluent's Golang Client for Apache Kafka\u003csup\u003eTM\u003c/sup\u003e\n=====================================================\n\n**confluent-kafka-go** is Confluent's Golang client for [Apache Kafka](http://kafka.apache.org/) and the\n[Confluent Platform](https://www.confluent.io/product/compare/).\n\nFeatures:\n\n- **High performance** - confluent-kafka-go is a lightweight wrapper around\n[librdkafka](https://github.com/edenhill/librdkafka), a finely tuned C\nclient.\n\n- **Reliability** - There are a lot of details to get right when writing an Apache Kafka\nclient. We get them right in one place (librdkafka) and leverage this work\nacross all of our clients (also [confluent-kafka-python](https://github.com/confluentinc/confluent-kafka-python)\nand [confluent-kafka-dotnet](https://github.com/confluentinc/confluent-kafka-dotnet)).\n\n- **Supported** - Commercial support is offered by\n[Confluent](https://confluent.io/).\n\n- **Future proof** - Confluent, founded by the\ncreators of Kafka, is building a [streaming platform](https://www.confluent.io/product/compare/)\nwith Apache Kafka at its core. It's high priority for us that client features keep\npace with core Apache Kafka and components of the [Confluent Platform](https://www.confluent.io/product/compare/).\n\n\nThe Golang bindings provides a high-level Producer and Consumer with support\nfor the balanced consumer groups of Apache Kafka 0.9 and above.\n\nSee the [API documentation](http://docs.confluent.io/current/clients/confluent-kafka-go/index.html) for more information.\n\n**License**: [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0)\n\n\nExamples\n========\n\nHigh-level balanced consumer\n\n```golang\nimport (\n\t\"fmt\"\n\t\"gopkg.in/confluentinc/confluent-kafka-go.v1/kafka\"\n)\n\nfunc main() {\n\n\tc, err := kafka.NewConsumer(\u0026kafka.ConfigMap{\n\t\t\"bootstrap.servers\": \"localhost\",\n\t\t\"group.id\":          \"myGroup\",\n\t\t\"auto.offset.reset\": \"earliest\",\n\t})\n\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\tc.SubscribeTopics([]string{\"myTopic\", \"^aRegex.*[Tt]opic\"}, nil)\n\n\tfor {\n\t\tmsg, err := c.ReadMessage(-1)\n\t\tif err == nil {\n\t\t\tfmt.Printf(\"Message on %s: %s\\n\", msg.TopicPartition, string(msg.Value))\n\t\t} else {\n\t\t\t// The client will automatically try to recover from all errors.\n\t\t\tfmt.Printf(\"Consumer error: %v (%v)\\n\", err, msg)\n\t\t}\n\t}\n\n\tc.Close()\n}\n```\n\nProducer\n\n```golang\nimport (\n\t\"fmt\"\n\t\"gopkg.in/confluentinc/confluent-kafka-go.v1/kafka\"\n)\n\nfunc main() {\n\n\tp, err := kafka.NewProducer(\u0026kafka.ConfigMap{\"bootstrap.servers\": \"localhost\"})\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\tdefer p.Close()\n\n\t// Delivery report handler for produced messages\n\tgo func() {\n\t\tfor e := range p.Events() {\n\t\t\tswitch ev := e.(type) {\n\t\t\tcase *kafka.Message:\n\t\t\t\tif ev.TopicPartition.Error != nil {\n\t\t\t\t\tfmt.Printf(\"Delivery failed: %v\\n\", ev.TopicPartition)\n\t\t\t\t} else {\n\t\t\t\t\tfmt.Printf(\"Delivered message to %v\\n\", ev.TopicPartition)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}()\n\n\t// Produce messages to topic (asynchronously)\n\ttopic := \"myTopic\"\n\tfor _, word := range []string{\"Welcome\", \"to\", \"the\", \"Confluent\", \"Kafka\", \"Golang\", \"client\"} {\n\t\tp.Produce(\u0026kafka.Message{\n\t\t\tTopicPartition: kafka.TopicPartition{Topic: \u0026topic, Partition: kafka.PartitionAny},\n\t\t\tValue:          []byte(word),\n\t\t}, nil)\n\t}\n\n\t// Wait for message deliveries before shutting down\n\tp.Flush(15 * 1000)\n}\n```\n\nMore elaborate examples are available in the [examples](examples) directory,\nincluding [how to configure](examples/confluent_cloud_example) the Go client\nfor use with [Confluent Cloud](https://www.confluent.io/confluent-cloud/).\n\n\nGetting Started\n===============\n\nInstalling librdkafka\n---------------------\n\nThis client for Go depends on librdkafka v1.0.0 or later, so you either need to install librdkafka\nthrough your OS/distributions package manager, or download and build it from source.\n\n- For Debian and Ubuntu based distros, install `librdkafka-dev` from the standard\nrepositories or using [Confluent's Deb repository](http://docs.confluent.io/current/installation.html#installation-apt).\n- For Redhat based distros, install `librdkafka-devel` using [Confluent's YUM repository](http://docs.confluent.io/current/installation.html#rpm-packages-via-yum).\n- For MacOS X, install `librdkafka` from Homebrew. You may also need to brew install pkg-config if you don't already have it. `brew install librdkafka pkg-config`.\n- For Alpine: `apk add librdkafka-dev pkgconf`\n- confluent-kafka-go is not supported on Windows.\n\nBuild from source:\n\n    git clone https://github.com/edenhill/librdkafka.git\n    cd librdkafka\n    ./configure --prefix /usr\n    make\n    sudo make install\n\n\nInstall the client\n-------------------\n\nWe recommend that you version pin the confluent-kafka-go import to v1:\n\nManual install:\n```bash\ngo get -u gopkg.in/confluentinc/confluent-kafka-go.v1/kafka\n```\n\nGolang import:\n```golang\nimport \"gopkg.in/confluentinc/confluent-kafka-go.v1/kafka\"\n```\n\nlibrdkafka is included with the Go client and does not have to be installed\nseparately on the build, nor target, system.\n\n\n**Dynamic linking:** If you prefer to link librdkafka dynamically, which requires\nlibrdkafka to be installed on your system, you should build your application with `go build -tags dynamic`.\nTthe development of librdkafka and the Go client are kept in synch.\nIf you use the master branch of the Go client, then you need to use the master branch of librdkafka.\n\n\n\n\nSee the [examples](examples) for usage details.\n\n\n\nAPI Strands\n===========\n\nThere are two main API strands: function and channel based.\n\nFunction Based Consumer\n-----------------------\n\nMessages, errors and events are polled through the consumer.Poll() function.\n\nPros:\n\n * More direct mapping to underlying librdkafka functionality.\n\nCons:\n\n * Makes it harder to read from multiple channels, but a go-routine easily\n   solves that (see Cons in channel based consumer above about outdated events).\n * Slower than the channel consumer.\n\nSee [examples/consumer_example](examples/consumer_example)\n\n\nChannel Based Consumer (deprecated)\n-----------------------------------\n\n*Deprecated*: The channel based consumer is deprecated due to the channel issues\n              mentioned below. Use the function based consumer.\n\nMessages, errors and events are posted on the consumer.Events channel\nfor the application to read.\n\nPros:\n\n * Possibly more Golang:ish\n * Makes reading from multiple channels easy\n * Fast\n\nCons:\n\n * Outdated events and messages may be consumed due to the buffering nature\n   of channels. The extent is limited, but not remedied, by the Events channel\n   buffer size (`go.events.channel.size`).\n\nSee [examples/consumer_channel_example](examples/consumer_channel_example)\n\n\n\n\n\n\nChannel Based Producer\n----------------------\n\nApplication writes messages to the producer.ProducerChannel.\nDelivery reports are emitted on the producer.Events or specified private channel.\n\nPros:\n\n * Go:ish\n * Proper channel backpressure if librdkafka internal queue is full.\n\nCons:\n\n * Double queueing: messages are first queued in the channel (size is configurable)\n   and then inside librdkafka.\n\nSee [examples/producer_channel_example](examples/producer_channel_example)\n\n\nFunction Based Producer\n-----------------------\n\nApplication calls producer.Produce() to produce messages.\nDelivery reports are emitted on the producer.Events or specified private channel.\n\nPros:\n\n * Go:ish\n\nCons:\n\n * Produce() is a non-blocking call, if the internal librdkafka queue is full\n   the call will fail.\n * Somewhat slower than the channel producer.\n\nSee [examples/producer_example](examples/producer_example)\n\n\nTests\n=====\n\nSee [kafka/README](kafka/README.md)\n\nContributing\n------------\nContributions to the code, examples, documentation, et.al, are very much appreciated.\n\nMake your changes, run gofmt, tests, etc, push your branch, create a PR, and [sign the CLA](http://clabot.confluent.io/cla).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffigma%2Fconfluent-kafka-go-dev","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffigma%2Fconfluent-kafka-go-dev","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffigma%2Fconfluent-kafka-go-dev/lists"}