{"id":13516292,"url":"https://github.com/lovoo/goka","last_synced_at":"2025-05-13T15:11:16.272Z","repository":{"id":39545993,"uuid":"86450135","full_name":"lovoo/goka","owner":"lovoo","description":"Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go.","archived":false,"fork":false,"pushed_at":"2025-05-02T04:22:09.000Z","size":6762,"stargazers_count":2427,"open_issues_count":36,"forks_count":175,"subscribers_count":79,"default_branch":"master","last_synced_at":"2025-05-02T05:24:47.538Z","etag":null,"topics":["goka","golang","kafka","microservices","processing","stream"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lovoo.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-03-28T11:06:58.000Z","updated_at":"2025-05-01T06:17:13.000Z","dependencies_parsed_at":"2023-09-23T17:52:42.598Z","dependency_job_id":"ecb90994-5da4-458f-8d69-f790877e0d8f","html_url":"https://github.com/lovoo/goka","commit_stats":{"total_commits":407,"total_committers":58,"mean_commits":7.017241379310345,"dds":0.683046683046683,"last_synced_commit":"e6a3579e783ed7a6120f3b176bb397186bcde062"},"previous_names":[],"tags_count":34,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lovoo%2Fgoka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lovoo%2Fgoka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lovoo%2Fgoka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lovoo%2Fgoka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lovoo","download_url":"https://codeload.github.com/lovoo/goka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253969259,"owners_count":21992263,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["goka","golang","kafka","microservices","processing","stream"],"created_at":"2024-08-01T05:01:21.091Z","updated_at":"2025-05-13T15:11:11.254Z","avatar_url":"https://github.com/lovoo.png","language":"Go","readme":"# Goka \n[![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause) [![Unit Tests/System Tests](https://github.com/lovoo/goka/actions/workflows/main.yml/badge.svg)](https://github.com/lovoo/goka/actions/workflows/main.yml)\n [![GoDoc](https://godoc.org/github.com/lovoo/goka?status.svg)](https://godoc.org/github.com/lovoo/goka) [![Go Report Card](https://goreportcard.com/badge/github.com/lovoo/goka)](https://goreportcard.com/report/github.com/lovoo/goka)\n\nGoka is a compact yet powerful distributed stream processing library for [Apache Kafka] written in Go. Goka aims to reduce the complexity of building highly scalable and highly available microservices.\n\nGoka extends the concept of Kafka consumer groups by binding a state table to them and persisting them in Kafka. Goka provides sane defaults and a pluggable architecture.\n\n## Features\n  * **Message Input and Output**\n\n    Goka handles all the message input and output for you. You only have to provide one or more callback functions that handle messages from any of the Kafka topics you are interested in. You only ever have to deal with deserialized messages.\n\n  * **Scaling**\n\n    Goka automatically distributes the processing and state across multiple instances of a service. This enables effortless scaling when the load increases.\n\n  * **Fault Tolerance**\n\n    In case of a failure, Goka will redistribute the failed instance's workload and state across the remaining healthy instances. All state is safely stored in Kafka and messages delivered with *at-least-once* semantics.\n\n  * **Built-in Monitoring and Introspection**\n\n    Goka provides a web interface for monitoring performance and querying values in the state.\n\n  * **Modularity**\n\n    Goka fosters a pluggable architecture which enables you to replace for example the storage layer or the Kafka communication layer.\n\n## Documentation\n\nThis README provides a brief, high level overview of the ideas behind Goka.\n\nPackage API documentation is available at [GoDoc] and the [Wiki](https://github.com/lovoo/goka/wiki/Tips#configuring-log-compaction-for-table-topics) provides several tips for configuring, extending, and deploying Goka applications.\n\n## Installation\n\nYou can install Goka by running the following command:\n\n``$ go get -u github.com/lovoo/goka``\n\n## Configuration\n\nGoka relies on [Sarama](https://github.com/IBM/sarama) to perform the actual communication with Kafka, which offers many configuration settings. The config is documented [here](https://godoc.org/github.com/IBM/sarama#Config).\n\nIn most cases, you need to modify the config, e.g. to set the Kafka Version.\n\n```go\ncfg := goka.DefaultConfig()\ncfg.Version = sarama.V2_4_0_0\ngoka.ReplaceGlobalConfig(cfg)\n```\n\nThis makes all goka components use the updated config.\n\nIf you do need specific configuration for different components, you need to pass customized builders to the \ncomponent's constructor, e.g.\n\n```go\ncfg := goka.DefaultConfig()\n// modify the config with component-specific settings\n\n\n// use the config by creating a builder which allows to override global config\ngoka.NewProcessor(// ...,\n\tgoka.WithConsumerGroupBuilder(\n\t\tgoka.ConsumerGroupBuilderWithConfig(cfg),\n\t),\n\t// ...\n)\n```\n\n## Concepts\n\nGoka relies on Kafka for message passing, fault-tolerant state storage and workload partitioning.\n\n* **Emitters** deliver key-value messages into Kafka. As an example, an emitter could be a database handler emitting the state changes into Kafka for other interested applications to consume.\n\n* **Processor** is a set of callback functions that consume and perform state transformations upon delivery of these emitted messages. *Processor groups* are formed of one or more instances of a processor. Goka distributes the partitions of the input topics across all processor instances in a processor group. This enables effortless scaling and fault-tolerance. If a processor instance fails, its partitions and state are reassigned to the remaining healthy members of the processor group. Processors can also emit further messages into Kafka.\n\n* **Group table** is the state of a processor group. It is a partitioned key-value table stored in Kafka that belongs to a single processor group. If a processor instance fails, the remaining instances will take over the group table partitions of the failed instance recovering them from Kafka.\n\n* **Views** are local caches of a complete group table. Views provide read-only access to the group tables and can be used to provide external services for example through a gRPC interface.\n\n* **Local storage** keeps a local copy of the group table partitions to speedup recovery and reduce memory utilization. By default, the local storage uses [LevelDB](https://github.com/syndtr/goleveldb), but in-memory map and [Redis-based storage](https://github.com/lovoo/goka/tree/master/storage/redis) are also available.\n\n\n## Get Started\n\nAn example Goka application could look like the following.\nAn emitter emits a single message with key \"some-key\" and value \"some-value\" into the \"example-stream\" topic.\nA processor processes the \"example-stream\" topic counting the number of messages delivered for \"some-key\".\nThe counter is persisted in the \"example-group-table\" topic.\nTo locally start a dockerized Zookeeper and Kafka instances, execute `make start` with the `Makefile` in the [examples] folder.\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"log\"\n\t\"os\"\n\t\"os/signal\"\n\t\"syscall\"\n\t\"time\"\n\n\t\"github.com/lovoo/goka\"\n\t\"github.com/lovoo/goka/codec\"\n)\n\nvar (\n\tbrokers             = []string{\"localhost:9092\"}\n\ttopic   goka.Stream = \"example-stream\"\n\tgroup   goka.Group  = \"example-group\"\n)\n\n// Emit messages forever every second\nfunc runEmitter() {\n\temitter, err := goka.NewEmitter(brokers, topic, new(codec.String))\n\tif err != nil {\n\t\tlog.Fatalf(\"error creating emitter: %v\", err)\n\t}\n\tdefer emitter.Finish()\n\tfor {\n\t\ttime.Sleep(1 * time.Second)\n\t\terr = emitter.EmitSync(\"some-key\", \"some-value\")\n\t\tif err != nil {\n\t\t\tlog.Fatalf(\"error emitting message: %v\", err)\n\t\t}\n\t}\n}\n\n// process messages until ctrl-c is pressed\nfunc runProcessor() {\n\t// process callback is invoked for each message delivered from\n\t// \"example-stream\" topic.\n\tcb := func(ctx goka.Context, msg interface{}) {\n\t\tvar counter int64\n\t\t// ctx.Value() gets from the group table the value that is stored for\n\t\t// the message's key.\n\t\tif val := ctx.Value(); val != nil {\n\t\t\tcounter = val.(int64)\n\t\t}\n\t\tcounter++\n\t\t// SetValue stores the incremented counter in the group table for in\n\t\t// the message's key.\n\t\tctx.SetValue(counter)\n\t\tlog.Printf(\"key = %s, counter = %v, msg = %v\", ctx.Key(), counter, msg)\n\t}\n\n\t// Define a new processor group. The group defines all inputs, outputs, and\n\t// serialization formats. The group-table topic is \"example-group-table\".\n\tg := goka.DefineGroup(group,\n\t\tgoka.Input(topic, new(codec.String), cb),\n\t\tgoka.Persist(new(codec.Int64)),\n\t)\n\n\tp, err := goka.NewProcessor(brokers, g)\n\tif err != nil {\n\t\tlog.Fatalf(\"error creating processor: %v\", err)\n\t}\n\tctx, cancel := context.WithCancel(context.Background())\n\tdone := make(chan bool)\n\tgo func() {\n\t\tdefer close(done)\n\t\tif err = p.Run(ctx); err != nil {\n\t\t\tlog.Fatalf(\"error running processor: %v\", err)\n\t\t} else {\n\t\t\tlog.Printf(\"Processor shutdown cleanly\")\n\t\t}\n\t}()\n\n\twait := make(chan os.Signal, 1)\n\tsignal.Notify(wait, syscall.SIGINT, syscall.SIGTERM)\n\t\u003c-wait   // wait for SIGINT/SIGTERM\n\tcancel() // gracefully stop processor\n\t\u003c-done\n}\n\nfunc main() {\n\tgo runEmitter() // emits one message every second forever\n\trunProcessor()  // press ctrl-c to stop\n}\n\n```\nA very similar example is also in *1-simplest*. Just run `go run examples/1-simplest/main.go`.\n\nNote that tables have to be configured in Kafka with log compaction.\nFor details check the [Wiki](https://github.com/lovoo/goka/wiki/Tips#configuring-log-compaction-for-table-topics).\n\n## How to contribute\n\nContributions are always welcome.\nPlease fork the repo, create a pull request against master, and be sure tests pass.\nSee the [GitHub Flow] for details.\n\n[Apache Kafka]: https://kafka.apache.org/\n[GoDoc]: https://godoc.org/github.com/lovoo/goka\n[examples]: https://github.com/lovoo/goka/tree/master/examples\n[GitHub Flow]: https://guides.github.com/introduction/flow\n","funding_links":[],"categories":["Go","Messaging","Development"],"sub_categories":["Search and Analytic Databases","Client libraries"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flovoo%2Fgoka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flovoo%2Fgoka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flovoo%2Fgoka/lists"}