{"id":46364442,"url":"https://github.com/runreveal/kawa","last_synced_at":"2026-03-05T02:01:18.473Z","repository":{"id":146404211,"uuid":"611337178","full_name":"runreveal/kawa","owner":"runreveal","description":"A blazingly fast event stream processing library powering the reveald event processing daemon.","archived":false,"fork":false,"pushed_at":"2025-11-24T16:12:03.000Z","size":1051,"stargazers_count":144,"open_issues_count":7,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-11-28T04:39:46.997Z","etag":null,"topics":["events","go","golang","logs","message-queue","stream-processing"],"latest_commit_sha":null,"homepage":"https://www.gokawa.dev/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/runreveal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-03-08T16:06:51.000Z","updated_at":"2025-11-24T16:07:58.000Z","dependencies_parsed_at":"2023-07-27T19:27:15.183Z","dependency_job_id":"947d51cc-813d-421c-a42d-408257808c1a","html_url":"https://github.com/runreveal/kawa","commit_stats":null,"previous_names":["runreveal/cheetah","runreveal/flow"],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/runreveal/kawa","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/runreveal%2Fkawa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/runreveal%2Fkawa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/runreveal%2Fkawa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/runreveal%2Fkawa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/runreveal","download_url":"https://codeload.github.com/runreveal/kawa/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/runreveal%2Fkawa/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30106124,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T01:39:18.192Z","status":"online","status_checked_at":"2026-03-05T02:00:06.710Z","response_time":93,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["events","go","golang","logs","message-queue","stream-processing"],"created_at":"2026-03-05T02:01:17.625Z","updated_at":"2026-03-05T02:01:18.464Z","avatar_url":"https://github.com/runreveal.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kawa\n\n\u003c!-- The space below the div closing tag is necessary --\u003e\n\u003cdiv align=center\u003e\n  \u003cdiv\u003e\n    \u003cimg src=\"docs/static/kawa.png\" width=\"300px\"\u003e\n    \u0026nbsp;\u0026nbsp;\n  \u003c/div\u003e\n  \u003cdiv\u003e\n    \u003ca href=\"https://pkg.go.dev/github.com/runreveal/kawa\"\u003e\u003cimg alt=\"Go Reference\" src=\"https://pkg.go.dev/badge/github.com/runreveal/kawa.svg\"/\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/runreveal/kawa/actions/workflows/ci.yml\"\u003e\u003cimg alt=\"GoFrame CI\" src=\"https://github.com/runreveal/kawa/actions/workflows/ci.yml/badge.svg\"/\u003e\u003c/a\u003e\n    \u003ca href=\"https://goreportcard.com/report/github.com/runreveal/kawa\"\u003e\u003cimg alt=\"Go Report Card\" src=\"https://goreportcard.com/badge/github.com/runreveal/kawa\"/\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/runreveal/kawa\"\u003e\u003cimg alt=\"License\" src=\"https://img.shields.io/github/license/runreveal/kawa.svg?style=flat\"/\u003e\u003c/a\u003e\n  \u003c/div\u003e\n\u003c/div\u003e\n\n---\n\nkawa (\"Kaa-Wah\") is an opinionated framework for scalable, reliable stream processing.\n\n# Installation\n\n## Kawa\n\nAdd the library to your project as you would any other Go library:\n\n```go\ngo get -u github.com/runreveal/kawa\n```\n\n# Design and Rationale\n\nSee https://blog.runreveal.com/kawa-the-event-processor-for-the-grug-brained-developer/\n\n# Roadmap\n\n- Ensure that consumers of kawa aren't subject to all the dependencies of the plugins.\n- Event Routing and/or Multiple Processors in kawa program\n- Dynamic Sources (e.g. Kafka Consumer Groups)\n\n# Disclaimer\n\nThis is nascent software, subject to breaking changes as we reach a good\nworking set of APIs, interfaces and data models.  Please try it out and help\nshape the direction of the project by giving us feedback!\n\n# Development \u0026 Extension\n\nThe source and destination interfaces are designed for simplicity of use and\nimplementation.  It should be easy to use the sources and destinations in an\nabstract manner without knowing the underlying implementation, and it should\nbe relatively easy to implement sources and destinations of various types.\n\nThe library provides multiple abstractions suitable for different purposes.\n\nThe easiest way to get started is by using the polling/batch implementations\nfor sources/destinations, respectively, since they require less overhead in\nterms of accounting for offset tracking to ensure at-least-once reliable\nprocessing.\n\nExtensions under the `x` package provide either generic or `[]byte` based\nsources, destinations and utility functions which aren't part of the core\nfunctionality of kawa.  They're provided for re-use in other applications.\n\nTo use them, import them into your program and apply the proper serialization\ntechniques relevant to your application.  See examples of this in practice in\nthe `cmd/kawad/internal` package, where we use it for system logs.\n\n## Configure and Run Design Pattern\n\nThe \"Configure and Run\" pattern is a pattern discovered while writing this\nframework that works nicely with other patterns and libraries in Go.\n\nThe general idea is as follows.  Each struct maintaining long running goroutines\ncan be made easy to reason about and operate by splitting it's configuration and\nruntime into two separate stages.\n\nThe first stage, \"Configure\", is simply the initialization of the struct.  Most\noften, this is the New function for the struct, with required arguments passed\nin first, and options passed in as a variadic functional options slice\nafterwards.  Occasionally, this may involve also implementing a translation\nlayer for serialization of the struct from JSON or some other serialization\nformat (see cmd/kawad/config.go for an example of this pattern).\n\nThe next stage, \"Run\", involves implementing the simple Runner interface:\n\n```golang\ntype Runner interface {\n\tRun(ctx context.Context) error\n}\n```\n\nImplementing this interface consistently across all instances of structs that\nhave long-running processes means that we can easily implement cancellation\nacross a broad number of distinct types via a common context variable.\n\nIt also means that any goroutine can trigger a shutdown by returning an error\nfrom the Run routine.\n\nWhile not absolutely required, following this pattern will enable the source or\ndestination to be seamlessly integrated into the daemon.\n\n## Implementing Sources\n\nSources are things that you read message-oriented data from.  At the most basic\nlevel, it's a collection of bytes that together represent some discrete event.\n\n### Polling Source\n\nWe recommend implementing polling sources when querying an API, or whenever\nit's easiest to implement a function to periodically get called.  The following\nis the interface which needs to be satisfied to implement a polling source.\n\n```golang\ntype Poller[T any] interface {\n\tPoll(context.Context, int) ([]kawa.Message[T], func(), error)\n}\n```\n\n### Streaming Source\n\nWe recommend implementing streaming sources when the source either implements\nit's own batching semantics (like Kafka), or when message latency is more\nimportant than message volume.\n\n## Implementing Destinations\n\nDestinations are things that you write message-oriented data to.\n\n### Batch Destination\n\nImplementing a batch destination is the easiest way to process messages as a\nbatch being written to some persistent storage.  It handles timeouts, batch size,\nand parallel writes at the configuration level so destinations only have to implement\na single method \"Flush\".\n\n```golang\ntype Flusher[T any] interface {\n\tFlush(context.Context, []kawa.Message[T]) error\n}\n```\n\n### Streaming Destination\n\nWe recommend implementing streaming destinations when the destination either\nimplements it's own batching semantics (like Kafka), or when message latency is\nmore important than message volume.\n\n# Supported sources\n\n - syslog\n - scanner\n - journald\n - mqtt\n\n# Supported destinations\n\n - s3 / r2\n - printer\n - runreveal\n - mqtt\n\n# Configuring the Daemon\n\n## Source Configuration\n\n### syslog\n\nWith the syslog config, and address and content type can be set.\n\n```\n{\n    \"type\":\"syslog\",\n    \"addr\":\"0.0.0.0:5514\",\n}\n```\n\n### journald\n\nJournald has no configuration, just set the type and kawa will read from journald.\n```\n{\n    \"type\":\"journald\"\n}\n```\n\n### scanner\n\nRead from stdin. Useful for testing or doing something you probably shouldn't.\n\n```\n{\n    \"type\":\"scanner\",\n}\n```\n\n### MQTT\nMQTT will listen on the supplied topic for new events.\n\nbroker and clientID are required to receive data.\nclientID must be unique from any other mqtt destinations or sources\nIf topic is not supplied, it will default to the wildcard `#`.\n\nDo not read events from the same topic that an MQTT destination is sending to otherwise kawa will create an infinite loop and eventually crash.\n\n```\n{\n  \"type\": \"mqtt\",\n  \"broker\": \"mqtt://broker.mqtt:1883\",\n  \"clientID\": \"kawa_src\",\n  \"userName\": \"\",\n  \"password\": \"\",\n  \"topic\": \"kawa/src\",\n\n  \"qos\": 1, // Optional defaults to 1 if not included\n  \"retained\": false, // Optional defaults to false if not included\n}\n```\n\n### Windows Event Logs\nListen for new windows event logs on the specified channel.\n\nWindows event log collection only works on Windows machines. Use the Windows build to run Kawad on a Windows machine. Kawad will need to be run as an administrator to have access to the event log stream.\n\nThe source config needs a required channel and an optional query. \nThe channel is the windows event log full name, e.g. to log the operational logs for the TaskScheduler the channel would be 'Microsoft-Windows-TaskScheduler/Operational'. \nThe query is a filter that can be used to limit the logs that are collected to specific events. View [Microsoft documentation](https://techcommunity.microsoft.com/t5/ask-the-directory-services-team/advanced-xml-filtering-in-the-windows-event-viewer/ba-p/399761) on how filtering works and how to create one to use.\n\nThe following example shows how to log every Security event on the machine.\n\n```\n{\n    \"type\": \"eventlog\",\n    \"channel\": \"Security\",\n    \"query\": \"*\"\n  }\n```\n\n\n## Destination Configuration\n\n### RunReveal\n\nWebhookURL is the only config argument and it is required.\n\n```\n{\n    \"type\":\"runreveal\",\n    \"webhookURL\": \"https://api.runreveal.com/.....\"\n}\n```\n\n### S3\n\nThe s3 destination is compatible with s3 and other s3 compatible interfaces. By default the s3 destination will pull credentials from the standard places the aws sdk looks, but they can optionally be set in the configuration.\n\ncustomEndpoint must be set for custom destinations, and in that case bucketRegion probably will not be set. bucketName is the only required argument.\n\nFor high volume or low volume, the batchSize can be tweaked but is set to 100 by default.\n\n```\n{\n    \"type\":\"s3\",\n    \"bucketName\":\"my-cool-log-bucket\",\n    \"bucketRegion\":\"us-east-2\",\n    \"batchSize\":1000,\n}\n```\n\n### Printer\n\nPrinter will print the results to stdout. Useful for testing and development.\n```\n{\n    \"type\":\"printer\",\n}\n```\n\n### MQTT\nMQTT will send events to the supplied topic.\n\nbroker and clientID are required to send data.\nclientID must be unique from any other mqtt destinations or sources\nIf topic is not supplied, it will default to the wildcard `#`.\n\n```\n{\n  \"type\": \"mqtt\",\n  \"broker\": \"mqtt://broker.mqtt:1883\",\n  \"clientID\": \"kawa_dst\",\n  \"userName\": \"\",\n  \"password\": \"\",\n  \"topic\": \"kawa/dest\",\n\n  \"qos\": 1, // Optional defaults to 1 if not included\n  \"retained\": false, // Optional defaults to false if not included\n}\n```\n\n## Source / Destination Wishlist\n\n - Kafka\n - redis\n - NATS\n - amqp\n - pubsub\n - Kinesis\n - memcache?\n - zmq?\n - NSQ?\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frunreveal%2Fkawa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frunreveal%2Fkawa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frunreveal%2Fkawa/lists"}