{"id":36846190,"url":"https://github.com/qntfy/frizzle","last_synced_at":"2026-01-12T14:33:00.983Z","repository":{"id":57554103,"uuid":"167253641","full_name":"qntfy/frizzle","owner":"qntfy","description":"The magic message bus","archived":false,"fork":false,"pushed_at":"2019-03-21T20:33:46.000Z","size":35,"stargazers_count":15,"open_issues_count":1,"forks_count":4,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-08-14T14:11:35.217Z","etag":null,"topics":["consumer","golang","golang-library","kafka","kinesis","message-bus","pipeline","producer","stream-processing","streaming-data"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/qntfy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-01-23T21:02:11.000Z","updated_at":"2023-12-23T18:22:06.000Z","dependencies_parsed_at":"2022-09-26T18:51:13.952Z","dependency_job_id":null,"html_url":"https://github.com/qntfy/frizzle","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/qntfy/frizzle","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qntfy%2Ffrizzle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qntfy%2Ffrizzle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qntfy%2Ffrizzle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qntfy%2Ffrizzle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/qntfy","download_url":"https://codeload.github.com/qntfy/frizzle/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qntfy%2Ffrizzle/sbom","scorecard":{"id":754249,"data":{"date":"2025-08-11","repo":{"name":"github.com/qntfy/frizzle","commit":"09018e64cd8e2cc2fa487cd3752170069a6e5e5f"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.9,"checks":[{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 0/7 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 9 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":9,"reason":"1 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GO-2022-0493 / GHSA-p782-xgp4-8hr8"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-22T21:09:20.499Z","repository_id":57554103,"created_at":"2025-08-22T21:09:20.499Z","updated_at":"2025-08-22T21:09:20.499Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28340400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T12:22:26.515Z","status":"ssl_error","status_checked_at":"2026-01-12T12:22:10.856Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consumer","golang","golang-library","kafka","kinesis","message-bus","pipeline","producer","stream-processing","streaming-data"],"created_at":"2026-01-12T14:33:00.108Z","updated_at":"2026-01-12T14:33:00.971Z","avatar_url":"https://github.com/qntfy.png","language":"Go","readme":"# Frizzle\n\n[![Travis Build Status](https://img.shields.io/travis/qntfy/frizzle.svg?branch=master)](https://travis-ci.org/qntfy/frizzle)\n[![Coverage Status](https://coveralls.io/repos/github/qntfy/frizzle/badge.svg?branch=master)](https://coveralls.io/github/qntfy/frizzle?branch=master)\n[![MIT licensed](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)\n[![GitHub release](https://img.shields.io/github/release/qntfy/frizzle.svg?maxAge=3600)](https://github.com/qntfy/frizzle/releases/latest)\n[![Go Report Card](https://goreportcard.com/badge/github.com/qntfy/frizzle#1)](https://goreportcard.com/report/github.com/qntfy/frizzle)\n[![GoDoc](https://godoc.org/github.com/qntfy/frizzle?status.svg)](http://godoc.org/github.com/qntfy/frizzle)\n\nFrizzle is a magic message (`Msg`) bus designed for parallel processing with many goroutines.\n  * `Receive()` messages from a configured `Source`\n  * Do your processing, possibly `Send()` each `Msg` on to one or more `Sink` destinations\n  * `Ack()` (or `Fail()`) the `Msg`  to notify the `Source` that processing completed\n\n## Getting Started\n\n**Start with the [example implementation](frizzle_integration_test.go)** which shows a simple canonical\nimplementation of a `Processor` on top of Frizzle and most of the core functions.\n\nhigh level interface\n```\n// Frizzle is a Msg bus for rapidly configuring and processing messages between multiple message services.\ntype Frizzle interface {\n\tReceive() \u003c-chan Msg\n\tSend(m Msg, dest string) error\n\tAck(Msg) error\n\tFail(Msg) error\n\tEvents() \u003c-chan Event\n\tAddOptions(...Option)\n\tFlushAndClose(timeout time.Duration) error\n\tClose() error\n}\n\nfunc Init(source Source, sink Sink, opts ...Option) Frizzle\n```\n\nThe core of the repo is a `Friz` struct (returned by `Init()`) which implements `Frizzle`. The intent is for\nseparate `Source` and `Sink` implementations (in separate repos) to be mixed and matched with the glue of\n`Frizzle`. A processing library can take a `Frizzle` input to allow easy re-use with multiple\nunderlying message technologies.\nFriz also implements `Source` and `Sink` to allow chaining if needed.\n\n### Source and Sink Implementations\n* **[Frinesis](https://github.com/qntfy/frinesis)** for AWS Kinesis\n* **[Frafka](https://github.com/qntfy/frafka)** for Apache Kafka\n* **[Basic](./basic)** for simple usage (part of Frizzle)\n\nIf you write a new implementation, we'd love to add it to our list!\n\n## Msg\nA basic interface which can be extended:\n```\n// Msg encapsulates an immutable message passed around by Frizzle\ntype Msg interface {\n\tID() string\n\tData() []byte\n\tTimestamp() time.Time\n}\n```\n\nA `frizzle.SimpleMsg` struct is provided for basic use cases.\n\n## Source and Sink\n\n```\n// Source defines a stream of incoming Msgs to be Received for processing,\n// and reporting whether or not processing was successful.\ntype Source interface {\n\tReceive() \u003c-chan Msg\n\tAck(m Msg) error\n\tFail(m Msg) error\n\tUnAcked() []Msg\n\tStop() error\n\tClose() error\n}\n\n// Sink defines a message service where Msgs can be sent as part of processing.\ntype Sink interface {\n\tSend(m Msg, dest string) error\n\tClose() error\n}\n```\n\n## Options\nFrizzle supports a variety of `Option` parameters for additional functionality to simplify your integration.\nThese can be included with `Init()` or added using a `friz.AddOptions()` call. Note that `AddOptions()`\nupdates the current friz and does not return anything.\n\nCurrently supported options:\n* `Logger(log *zap.Logger)` - Include a logger to report frizzle-internal logging.\n* `Stats(stats StatsIncrementer)` - Include a stats client for frizzle-internal metrics reporting.\n  See [Stats](#stats) for what metrics are supported.\n* `FailSink(s Sink, dest string)` - Provide a Sink and destination (kafka topic, kinesis stream etc)\n  where `Fail()`ed Msgs will be sent automatically.\n* `MonitorProcessingRate(pollPeriod time.Duration)` - Log the sum count of Acked and Failed Msgs every `pollPeriod`.\n* `ReportAsyncErrors()` - Launch a simple go routine to monitor the `Events()` channel. All events are logged at `Error` or `Warn` level;\n  any events that match `error` interface have a stat recorded. Logging and/or stats are disabled\n  if `Logger()`/`Stats()` have not been set, respectively.\n  * This is a most basic handling that does not account for any specific Event types from Source/Sink implementations;\n  developers should write an app specific monitoring routine to parse and handle specific Event cases\n  (for which this can be a helpful starting template).\n* `HandleShutdown(appShutdown func())` - Monitor for `SIGINT` and `SIGTERM`, call `FlushAndClose()` followed by\n  provided `appShutdown` when they are received.\n* `WithTransformer(ft FrizTransformer)` - Add a transformer to modify the Msg's before they are sent or received.\n  Currently only supports a \"Simple Separator\" Transformer which adds a specified record separator (such as newline)\n  before sending if it isn't already present, and removes the same separator on receive if it is present.\n\n## Events\nSince Source and Sink implementations often send and receive Msgs in batch fashion,\nThey often may find out about any errors (or other important events) asynchronously.\nTo support this, async events can be recovered via a channel returned by the `Friz.Events()` method.\nIf a Source/Sink does not implement the `Eventer` interface this functionality will be ignored.\n\n### Caveats for using `Events()`\n\n* Frizzle Events must provide a minimum `String()` interface; when consuming Events\na type assertion switch is highly recommended to receive other relevant information.\n  * **A `default:` trap for unhandled cases is also highly recommended!**\n  * For a reference implementation of the same interface see\n    **[here](https://github.com/confluentinc/confluent-kafka-go/blob/d87f439f4a3ac1a1f94f3071ce2ed2238e27fba4/examples/producer_channel_example/producer_channel_example.go#L48-L66)**\n* A Friz's `Events()` channel will be closed after all underlying Source/Sink `Events()` channels are closed.\n  * If a Friz is initialized without any Source/Sinks that implement `Events()`, the channel returned by\n    `Friz.Events()` will be closed immediately.\n\nIn addition to the `String()` method required by frizzle, currently only errors are\nreturned by frinesis (no other event types) so all Events recovered will also conform\nto `error` interface.\n\n## Transformers\nTransformers provide a mechanism to do simple updates to a `Msg` prior to a `Send()` or `Receive()`, which \ncan be added at initializiation but is otherwise transparent to the processor and Source/Sink.\nThis can be useful in a case where e.g. you need to apply a transform when running on one messaging platform\nbut not another, and don't want to expose the core processing code to information about which platform\nis in use.\n\nFrizzle supports adding Transformers with a `WithTransformer()` Option:\n```\n// WithTransformer returns an Option to add the provided FrizTransformer to a Frizzle\nfunc WithTransformer(ft FrizTransformer) Option\n\n// Transform is a type that modifies a Msg\ntype Transform func(Msg) Msg\n\n// FrizTransformer provides a Transform to apply when a Msg is sent or received\ntype FrizTransformer interface {\n\tSendTransform() Transform\n\tReceiveTransform() Transform\n}\n\n```\n\nAn example implementation to add and remove a separator suffix on each Msg is included in\n[transform.go](./transform.go). To reduce clutter we generally suggest implementing a new\nTransform in a separate repo, but we can consider adding high utility ones here.\n\n## Prereqs / Build instructions\n\n### Go mod\n\nAs of Go 1.11, frizzle uses [go mod](https://github.com/golang/go/wiki/Modules) for dependency management.\n\n### Install\n\n```\n$ go get github.com/qntfy/frizzle\n$ cd frizzle\n$ go build\n```\n\n## Running the tests\n\n`go test -v --cover ./...`\n\n## Configuration\nWe recommend building Sources and Sinks to initialize using [Viper](https://godoc.org/github.com/spf13/viper),\ntypically through environment variables (but client can do whatever it wants, just needs to provide the\nconfigured Viper object with relevant values). The application might use a prefix such as before the below values.\n\n### Basic\n| Variable | Required | Description | Default |\n|---------------------------|:--------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------:|\n| BUFFER_SIZE | source (optional) | size of `Input()` channel buffer | 500 |\n| MOCK | optional | mocks don't track sent or unacked Msgs, just return without error | false |\n\n## Stats\n`StatsIncrementer` is a simple interface with just `Increment(bucket string)`; based on `github.com/alexcesaro/statsd` \nbut potentially compatible with a variety of metrics engines. When `Stats()` is set, Frizzle records the following metrics. \nIf a `Logger()` has been set, each of the below also generates a Debug level log with the ID() of the Msg.\n\n| Bucket | Description |\n|---------------------------|--------------------------------------------------|\n| ctr.rcv | count of Msgs received from Source |\n| ctr.send | count of Msgs sent to Sink |\n| ctr.ack | count of Msgs Ack'ed by application |\n| ctr.fail | count of Msgs Fail'ed by application |\n| ctr.failsink | count of Msgs sent to FailSink |\n| ctr.error | count of `error`s from `Events()`* |\n\n\\* only recorded if ReportAsyncErrors is running\n\n## Contributing\nContributions welcome! Take a look at open issues. New Source/Sink implementations should be added in separate repos.\nIf you let us know (and link to test demonstrating it conforms to the interface) we are happy to link them here!\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqntfy%2Ffrizzle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqntfy%2Ffrizzle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqntfy%2Ffrizzle/lists"}