{"id":26246038,"url":"https://github.com/zillow/zkafka","last_synced_at":"2025-07-22T06:05:03.799Z","repository":{"id":249735568,"uuid":"832273788","full_name":"zillow/zkafka","owner":"zillow","description":"An efficient and scalable library for stateless Kafka message processing written in Go","archived":false,"fork":false,"pushed_at":"2025-06-26T20:53:12.000Z","size":394,"stargazers_count":63,"open_issues_count":1,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-06-26T22:06:35.693Z","etag":null,"topics":["confluent-kafka","go","golang","kafka","kafka-consumer","kafka-producer","librdkafka"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zillow.png","metadata":{"files":{"readme":"README.md","changelog":"changelog.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-07-22T17:15:54.000Z","updated_at":"2025-06-18T07:13:22.000Z","dependencies_parsed_at":"2024-09-06T20:12:43.458Z","dependency_job_id":"e243ec30-f724-4c4d-ae3e-eadf0eac6b6f","html_url":"https://github.com/zillow/zkafka","commit_stats":null,"previous_names":["zillow/zkafka"],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/zillow/zkafka","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zillow%2Fzkafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zillow%2Fzkafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zillow%2Fzkafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zillow%2Fzkafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zillow","download_url":"https://codeload.github.com/zillow/zkafka/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zillow%2Fzkafka/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266437369,"owners_count":23928235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["confluent-kafka","go","golang","kafka","kafka-consumer","kafka-producer","librdkafka"],"created_at":"2025-03-13T13:17:18.235Z","updated_at":"2025-07-22T06:05:03.774Z","avatar_url":"https://github.com/zillow.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# zkafka\n\n[![License](https://img.shields.io/github/license/zillow/zkafka)](https://github.com/zillow/zkafka/blob/main/LICENSE)\n[![GitHub Actions](https://github.com/zillow/zkafka/actions/workflows/go.yml/badge.svg)](https://github.com/zillow/zkafka/actions/workflows/go.yml)\n[![Codecov](https://codecov.io/gh/zillow/zkafka/branch/main/graph/badge.svg?token=STRT8T67YP)](https://codecov.io/gh/zillow/zkafka)\n[![Go Report Card](https://goreportcard.com/badge/github.com/zillow/zkafka)](https://goreportcard.com/report/github.com/zillow/zkafka)\n\n## Install\n\n`go get -u github.com/zillow/zkafka/v2`\n\n## About\n\n`zkafka` is built to simplify message processing in Kafka. This library aims to minimize boilerplate code, allowing the developer to focus on writing the business logic for each Kafka message. `zkafka` takes care of various responsibilities, including:\n\n1. Reading from the worker's configured topics\n2. Managing message offsets reliably - Kafka offset management can be complex, but `zkafka` handles it. Developers only need to write code to process a single message and indicate whether or not it encountered an error.\n3. Distributing messages to virtual partitions (details will be explained later)\n4. Implementing dead lettering for failed messages\n5. Providing inspectable and customizable behavior through lifecycle functions (Callbacks) - Developers can add metrics or logging at specific points in the message processing lifecycle.\n\n`zkafka` provides stateless message processing semantics ( sometimes, called lambda message processing).\nThis is a churched-up way of saying, \"You write code which executes on each message individually (without knowledge of other messages)\".\nIt is purpose-built with this type of usage in mind. Additionally, the worker implementation guarantees at least once processing (Details of how that's achieved are shown in the [Commit Strategy](#commit-strategy) section)\n\n**NOTE**: \n`zkafka` is  built on top of [confluent-kafka-go](https://github.com/confluentinc/confluent-kafka-go)\nwhich is a CGO module. Therefore, so is `zkafka`. When building with `zkafka`, make sure to set CGO_ENABLED=1.\n\n### Features\n\nThe following subsections detail some useful features. To make the following sections more accessible, there are runnable examples in `./examples` directory.\nThe best way to learn is to experiment with the examples. Dive in!\n\n#### Stateless Message Processing\n\n`zkafka` makes stateless message processing easy. All you have to do is write a concrete `processor` implementation and wire it up (shown below).\n\n```go \ntype processor interface {\n    Process(ctx context.Context, message *zkafka.Message) error\n}\n```\nIf you want to skip ahead and see a working processor check out the examples. Specifically `example/worker/main.go`.\n\nThe anatomy of that example is described here:\n\nA `zkafka.Client` needs to be created which can connect to the kafka broker. Typically, authentication\ninformation must also be specified at this point (today that would include username/password).\n\n```go \n    client := zkafka.NewClient(zkafka.Config{ BootstrapServers: []string{\"localhost:29092\"} })\n```\n\nNext, this client should be passed to create a `zkafka.WorkFactory` instance.\nThe factory design, used by this library, adds a little boilerplate but allows default policies to be injected and\nproliferated to all instantiated work instances. We find that useful at zillow for transparently injecting the nuts and bolts\nof components that are necessary for our solutions to cross-cutting concerns (typically those revolving around telemetry)\n\n```go \n    wf := zkafka.NewWorkFactory(client)\n```\n\nNext we create the work instance. This is finally where the dots are beginning to connect.\n`zkafka.Work` objects are responsible for continually polling topics (the set of whom is specified in the config object) they've been instructed\nto listen to, and execute specified code (defined in the user-controlled `processor` and `lifecycle` functions (not shown here))\n```go \n   topicConfig := zkafka.TopicConfig{Topic: \"my-topic\", GroupdID: \"mygroup\", ClientID: \"myclient\"}\n   // this implements the interface specified above and will be executed for each read message\n   processor := \u0026Processor{}\n   work := wf.Create(topicConfig, processor)\n```\n\nAll that's left now is to kick off the run loop (this will connect to the Kafka broker, create a Kafka consumer group, undergo consumer group assignments, and after the assignment begins polling for messages).\nThe run loop executes a single reader (Kafka consumer) which reads messages and then fans those messages out to N processors (sized by the virtual partition pool size. Described later).\nIt's a processing pipeline with a reader at the front, and processors at the back.\n\nThe run loop takes two arguments, both responsible for signaling that the run loop should exit.\n\n1. `context.Context` object. When this object is canceled, the internal\n   work loop will begin to abruptly shut down. This involves exiting the reader loop and processor loops immediately.\n\n2. signal channel. This channel should be `closed`, and tells zkafka to begin a graceful shutdown.\n   Graceful shutdown means the reader stops reading new messages, and the processors attempt to finish their in-flight work.\n\nAt Zillow, we deploy to a kubernetes cluster, and use a strategy that uses both\nmechanisms. When k8s indicates shutdown is imminent, we close the `shutdown` channel. Graceful\nshutdown is time-boxed, and if the deadline is reached, the outer `context` object\nis canceled signaling a more aggressive teardown. The below example passes in a nil shutdown signal (which is valid).\nThat's done for brevity in the readme, production use cases should take advantage (see examples).\n\n```go \n   err = w.Run(context.Background(), nil)\n```\n\n#### Hyper Scalability\n\n`zkafka.Work` supports a concept called `virtual partitions`. This extends\nthe Kafka `partition` concept. Message ordering is guaranteed within a Kafka partition,\nand the same holds true for a `virtual partition`. Every `zkafka.Work` object manages a pool\nof goroutines called processors (1 by default and controlled by the `zkafka.Speedup(n int)` option).\nEach processor reads from a goroutine channel called a `virtual partition`.\nWhen a message is read by the reader, it is assigned to one of the virtual partitions based on `hash(message.Key) % virtual partition count`.\nThis follows the same mechanism used by Kafka. With this strategy, a message with the same key will be assigned\nto the same virtual partition.\n\nThis allows for another layer of scalability. To increase throughput and maintain the same\nmessage ordering guarantees, there is no longer a need to increase the Kafka partition count (which can be operationally challenging).\nInstead, you can use `zkafka.Speedup()` to increase the virtual partition count.\n\n```shell\n// sets up Kafka broker locally\nmake setup;\n// terminal 1. Starts producing messages. To juice up the production rate, remove the time.Sleep() in the producer and turn acks off.\nmake example-producer\n// terminal 2. Starts a worker with speedup=5. \nmake example-worker\n```\n\n#### Configurable Dead Letter Topics\n\nA `zkafka.Work` instance can be configured to write to a Dead Letter Topic (DLT) when message processing fails.\nThis can be accomplished with the `zkafka.WithDeadLetterTopic()` option. Or, more conveniently, can be controlled by adding\na non nil value to the `zkafka.ConsumerTopicConfig` `DeadLetterTopic` field. Minimally, the topic name of the (dead letter topic)\nmust be specified (when specified via configuration, no clientID need be specified, as the encompassing consumer topic configs client id will be used).\n\n```go \n     zkafka.ConsumerTopicConfig{\n        ...\n       // When DeadLetterTopicConfig is specified a dead letter topic will be configured and written to\n       // when a processing error occurs.\n       DeadLetterTopicConfig: \u0026zkafka.ProducerTopicConfig{\n          Topic:    \"zkafka-example-deadletter-topic\",\n       },\n    }\n```\n\nThe above will be written to `zkafka-example-deadletter-topic` in the case of a processing error.\n\n\nThe above-returned error will skip writing to the DLT.\n\nTo execute a local example of the following pattern:\n\n```shell\n// sets up kafka broker locally\nmake setup;\n// terminal 1. Starts producing messages (1 per second)\nmake example-producer\n// terminal 2. Starts a worker which fails processing and writes to a DLT. Log statements show when messaages\n// are written to a DLT\nmake example-deadletter-worker\n```\n\nThe returned processor error determines whether a message is written to a dead letter topic. In some situations,\nyou might not want to route an error to a DLT. An example might be malformed data.\n\nYou have control over this behavior by way of the `zkafka.ProcessError`.\n\n```go \n    return zkafka.ProcessError{\n       Err:                 err,\n       DisableDLTWrite:     true,\n    }\n```\n#### Process Delay Workers\n\nProcess Delay Workers can be an important piece of an automated retry policy. A simple example of this would be\n2 workers daisy-chained together as follows:\n\n```go \n     workerConfig1 := zkafka.ConsumerTopicConfig{\n       ClientID: \"svc1\",\n       GroupID: \"grp1\",\n        Topic: \"topicA\",\n       // When DeadLetterTopicConfig is specified a dead letter topic will be configured and written to\n       // when a processing error occurs.\n       DeadLetterTopicConfig: \u0026zkafka.ProducerTopicConfig{\n          Topic:    \"topicB\",\n       },\n    }\n\n    workerConfig2 := zkafka.ConsumerTopicConfig{\n      ClientID: \"svc1\",\n      GroupID: \"grp1\",\n      Topic: \"topicB\",\n      // When DeadLetterTopicConfig is specified a dead letter topic will be configured and written to\n      // when a processing error occurs.\n      DeadLetterTopicConfig: \u0026zkafka.ProducerTopicConfig{\n         Topic:    \"topicC\",\n      },\n    }\n```\n\nMessages processed by the above worker configuration would:\n\n1.  Worker1 read from `topicA`\n2. If message processing fails, write to `topicB` via the DLT configuration\n3. Worker2 read from `topicB`\n4. If message processing fails, write to `topicC` via the DLT configuration\n\nThis creates a retry pipeline. The issue is that worker2, ideally would process on a delay (giving whatever transient error is occurring a chance to resolve).\nLuckily, `zkafka` supports such a pattern. By specifying `ProcessDelayMillis` in the config, a worker is created which will delay processing of\na read message until at least the delay duration has been waited.\n\n```go \n    topicConfig := zkafka.ConsumerTopicConfig{\n        ... \n       // This value instructs the kafka worker to inspect the message timestamp, and not call the processor call back until\n       // at least the process delay duration has passed\n       ProcessDelayMillis: \u0026processDelayMillis,\n    }\n```\n\nThe time awaited by the worker varies. If the message is very old (maybe the worker had been stopped previously),\nthen the worker will detect that the time passed since the message was written \u003e delay. In such a case, it won't delay any further.\n\nTo execute a local example of the following pattern:\n\n```shell\n// sets up kafka broker locally\nmake setup;\n// terminal 1. Starts producing messages (1 per second)\nmake example-producer\n// terminal 2. Starts delay processor. Prints out the duration since msg.Timestamp. \n// How long the delay is between when the message was written and when the process callback is executed.\nmake example-delay-worker\n```\n\n### Commit Strategy:\n\nA `zkafka.Work`er commit strategy allows for at least once message processing.\n\nThere are two quick definitions important to the understanding of the commit strategy:\n\n1. **Commit** - involves communicating with kafka broker and durably persisting offsets on a kafka broker.\n2. **Store** - is the action of updating a local store of message offsets which will be persisted during the commit\n   action\n\nThe `zkafka.Work` instance will store message offsets as message processing concludes. Because the worker manages\nstoring commits the library sets `enable.auto.offset.store`=false. Additionally, the library offloads actually committing messages\nto a background process managed by `librdkafka` (The frequency at which commits are communicated to the broker is controlled by `auto.commit.interval.ms`, default=5s).\nAdditionally, during rebalance events, explicit commits are executed.\n\nThis strategy is based off of [Kafka Docs - Offset Management](https://docs.confluent.io/platform/current/clients/consumer.html#offset-management)\nwhere a strategy of asynchronous/synchronous commits is suggested to reduce duplicate messages.\n\nThe above results in the following algorithm:\n\n\n1. Before message processing is started, an internal heap structure is used to track in-flight messages.\n2. After message processing concludes, a heap structure managed by `zkafka` marks the message as complete (regardless of whether processing errored or not).\n3. The inflight heap and the work completed heap are compared. Since offsets increase incrementally (by 1), it can be determined whether message processing\n   finished out of order. If the inflight heap's lowest offset is the same as the completed, then that message is safe to be **Stored**. This can be done repetitively\n   until the inflight heap is empty, or inflight messages haven't yet been marked as complete.\n\nThe remaining steps are implicitly handled by `librdkafka`\n1. *Commit* messages whose offsets have been stored at configurable intervals (`auto.commit.interval.ms`)\n2. *Commit* messages whose offsets have been stored when partitions are revoked\n   (this is implicitly handled by `librdkafka`. To see this add debug=cgrp in ConsumerTopicConfig, and there'll be COMMIT logs after a rebalance.\n   If doing this experience, set the `auto.commit.interval.ms` to a large value to avoid confusion between the rebalance commit)\n3. *Commit* messages whose offsets have been stored on close of reader\n   (this is implicitly handled by `librdkafka`. To see this add debug=cgrp in ConsumerTopicConfig, and there'll be COMMIT logs after the client is closed, but before the client is destroyed)\n\nErrors returned on processing are still stored. This avoids issues due to poison pill messages (messages that will\nnever be able to be processed without error)\nas well as transient errors blocking future message processing. Use dead lettering to sequester these failed messages or Use `WithOnDone()` option to register callback for\nspecial processing of these messages.\n\n\n### SchemaRegistry Support:\n\nzkafka supports schema registry. It extends `zfmt` to enable this adding three `zfmt.FormatterType`:\n```\n    AvroSchemaRegistry zfmt.FormatterType = \"avro_schema_registry\"\n\tProtoSchemaRegistry zfmt.FormatterType = \"proto_schema_registry\"\n\tJSONSchemaRegistry zfmt.FormatterType = \"json_schema_registry\"\n```\n\nThis can be used in ProducerTopicConfig/ConsumerTopicConfig just like the others. Examples have been added \n`example/producer_avro` and `example/worker_avro` which demonstrate the additional configuration (mostly there to enable the\nschema registry communication that's required)\n\n\n### Consumer/Producer Configuration\n\nSee for description of configuration options and their defaults:\n\n1. [Librdkafka Configuration](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)\n2. [Consumer Configuration](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html)\n3. [Producer Configurations](https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html)\n\nThese are primarily specified through the TopicConfig structs (`ProducerTopicConfig` and `ConsumerTopicConfig`).\nTopicConfigs includes strongly typed fields that translate\nto librdconfig values. To see translation see `config.go`. An escape hatch is provided for ad hoc config properties via\nthe AdditionalProperties map. Here config values that don't have a strongly typed version in TopicConfig may be\nspecified. Not all specified config values will work (for example `enable.auto.commit=false` would not work with this\nclient because that value is explicitly set to true after reading of the AdditionalProperties map).\n\n```go\ndeliveryTimeoutMS := 100\nenableIdempotence := false\nrequiredAcks := \"0\"\n\npcfg := ProducerTopicConfig{\n   ClientID:            \"myclientid\",\n   Topic: \"mytopic\",\n   DeliveryTimeoutMs:   \u0026deliveryTimeoutMS,\n   EnableIdempotence:   \u0026enableIdempotence,\n   RequestRequiredAcks: \u0026requiredAcks,\n   AdditionalProps: map[string]any{\n      \"linger.ms\":               float64(5),\n   },\n}\n\nccfg := ConsumerTopicConfig{\n   ClientID: \"myclientid2\",\n   GroupID:  \"mygroup\",\n   Topic: \"mytopic\",\n   AdditionalProps: map[string]any{\n      \"auto.commit.interval.ms\": float32(20),\n   },\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzillow%2Fzkafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzillow%2Fzkafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzillow%2Fzkafka/lists"}