{"id":15102809,"url":"https://github.com/silverton-io/buz","last_synced_at":"2025-04-12T23:30:34.681Z","repository":{"id":37899505,"uuid":"449532046","full_name":"silverton-io/buz","owner":"silverton-io","description":"Serverless multi-protocol + multi-destination event collection system.","archived":false,"fork":false,"pushed_at":"2024-11-24T13:14:06.000Z","size":31598,"stargazers_count":202,"open_issues_count":33,"forks_count":26,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-04T02:09:53.182Z","etag":null,"topics":["analytics","analytics-tracking","cloudevents","cloudevents-schema","contracts","data","data-collection","data-platform","eventbridge","jsonschema","product-analytics","redpanda","redpanda-console","schema-registry","schema-validation","snowplow-analytics","streaming-analytics","streaming-data","webhook-receiver","webhook-server"],"latest_commit_sha":null,"homepage":"https://oss.buz.dev/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/silverton-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-19T03:22:10.000Z","updated_at":"2025-02-16T11:38:40.000Z","dependencies_parsed_at":"2024-04-26T19:26:40.609Z","dependency_job_id":"7d5408da-d6d2-4ff0-8c3b-39c5cdce645b","html_url":"https://github.com/silverton-io/buz","commit_stats":{"total_commits":364,"total_committers":14,"mean_commits":26.0,"dds":0.2939560439560439,"last_synced_commit":"6b8c5afbf1bad2ab71bd8526a2f1657bf9b23b67"},"previous_names":["silverton-io/honeypot"],"tags_count":115,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/silverton-io%2Fbuz","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/silverton-io%2Fbuz/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/silverton-io%2Fbuz/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/silverton-io%2Fbuz/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/silverton-io","download_url":"https://codeload.github.com/silverton-io/buz/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248647249,"owners_count":21139081,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","analytics-tracking","cloudevents","cloudevents-schema","contracts","data","data-collection","data-platform","eventbridge","jsonschema","product-analytics","redpanda","redpanda-console","schema-registry","schema-validation","snowplow-analytics","streaming-analytics","streaming-data","webhook-receiver","webhook-server"],"created_at":"2024-09-25T19:07:17.241Z","updated_at":"2025-04-12T23:30:34.656Z","avatar_url":"https://github.com/silverton-io.png","language":"Go","readme":"# Buz\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-brightgreen.svg)](https://opensource.org/licenses/Apache-2.0)\n![GitHub tag (latest SemVer)](https://img.shields.io/github/v/tag/silverton-io/buz)\n![test](https://github.com/silverton-io/buz/actions/workflows/test.yml/badge.svg)\n![lint](https://github.com/silverton-io/buz/actions/workflows/lint.yml/badge.svg)\n[![ref](https://pkg.go.dev/badge/github.com/silverton-io/buz/pkg.svg)](https://pkg.go.dev/github.com/silverton-io/buz/pkg)\n\n![buz](img/buzz.png)\n\n# What is Buz?\n\nBuz is a system for multi-protocol event collection, validation, annotation, and delivery.\n\nIt ships as a single lightweight binary for deployment flexibility.\n\nToss Buz into an [AWS lambda function](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=snowplow\u0026source=aws-lambda\u0026location=README\u0026rto=https://aws.amazon.com/lambda/), [GCP Cloud Run](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=gcp-cloud-run\u0026source=github\u0026location=README\u0026rto=https://cloud.google.com/run) service, or K8s pod and start collecting events in minutes.\n\n## Multiple Input Protocols\n\nBuz supports [multiple input protocols](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=multiple-event-protocols\u0026source=github\u0026location=README\u0026rto=https://buz.dev/inputs/overview) including:\n\n* [Snowplow Analytics](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=snowplow-analytics\u0026source=github\u0026location=README\u0026rto=https://buz.dev/inputs/saas/snowplow)\n* [Cloudevents](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=cloudevents\u0026source=github\u0026location=README\u0026rto=https://buz.dev/inputs/cloudNative/cloudevents)\n* [Self-describing JSON](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=self-describing-json\u0026source=github\u0026location=README\u0026rto=https://buz.dev/inputs/buz/self-describing)\n* [Webhooks](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=webhooks\u0026source=github\u0026location=README\u0026rto=https://buz.dev/inputs/buz/webhook)\n\nIt even hosts a [pixel](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=pixel\u0026source=github\u0026location=README\u0026rto=https://buz.dev/inputs/buz/pixel) for use in constrained tracking environments.\n\nSDK's are supported out of the box so you can point existing Snowplow Analytics or Cloudevents tracking directly to Buz and it will just work™.\n\nOr point your in-house SDK to it using self-describing JSON.\n\n## Multiple (Simultaneous) Destinations\n\nBuz supports [two dozen different destinations](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=destination-overview\u0026source=github\u0026location=README\u0026rto=https://buz.dev/outputs/overview) including:\n* [Redpanda](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=dest-redpanda\u0026source=github\u0026location=README\u0026rto=https://buz.dev/outputs/stream/redpanda)\n* [Postgres](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=dest-postgres\u0026source=github\u0026location=README\u0026rto=https://buz.dev/outputs/database/postgres)\n* [Kinesis Firehose](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=dest-firehose\u0026source=github\u0026location=README\u0026rto=https://buz.dev/outputs/stream/aws-kinesis-firehose)\n* [Google Pub/Sub](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=dest-pubsub\u0026source=github\u0026location=README\u0026rto=https://buz.dev/outputs/stream/google-pubsub)\n* [Splunk](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=dest-splunk\u0026source=github\u0026location=README\u0026rto=https://buz.dev/outputs/database/splunk)\n* [AWS EventBridge](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=dest-eventbridge\u0026source=github\u0026location=README\u0026rto=https://buz.dev/outputs/messageBus/aws-eventbridge)\n* [TimescaleDB](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=dest-timescale\u0026source=github\u0026location=README\u0026rto=https://buz.dev/outputs/timeseries/timescaledb)\n* [NATS](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=dest-nats\u0026source=github\u0026location=README\u0026rto=https://buz.dev/outputs/messageBus/nats)\n* ...and many more.\n\nYou can send events to **one or more** destinations, so fanning them out to where they need to Bee is simple. As is using Buz to migrate from one destination system to another.\n\n## Jsonschema-based validation\n\nEvery incoming payload is validated in microseconds using [JSON Schema](https://json-schema.org/).\n\nIf a payload doesn't conform to the associated schema, it is marked as such.\n\nIf a payload doesn't have an explicitly-associated schema (such as the case with arbitrary webhooks and pixels), payload contents are not validated. It is enveloped as `arbitrary` for downstream processing.\n\n## Onboard Schema Registry\n\nBuz ships with a lightweight schema registry that supports [multiple schema backends](https://buz.dev/schema-registry/overview) including:\n\n* [GCS](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=registry-gcs\u0026source=github\u0026location=README\u0026rto=https://buz.dev/schema-registry/backends/object/gcs)\n* [S3](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=registry-s3\u0026source=github\u0026location=README\u0026rto=https://buz.dev/schema-registry/backends/object/s3)\n* [Minio](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=registry-minio\u0026source=github\u0026location=README\u0026rto=https://buz.dev/schema-registry/backends/object/minio)\n* [Postgres](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=registry-postgres\u0026source=github\u0026location=README\u0026rto=https://buz.dev/schema-registry/backends/database/postgres)\n* [Mysql](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=registry-mysql\u0026source=github\u0026location=README\u0026rto=https://buz.dev/schema-registry/backends/database/mysql)\n* [Mongodb](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=registry-mongodb\u0026source=github\u0026location=README\u0026rto=https://buz.dev/schema-registry/backends/database/mongodb)\n* [Local filesystem](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=registry-filesystem\u0026source=github\u0026location=README\u0026rto=https://buz.dev/schema-registry/backends/buz/filesystem)\n* ..and more\n\nSchemas are [cached locally](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=registry-cache\u0026source=github\u0026location=README\u0026rto=https://buz.dev/schema-registry/overview#onboard-schema-registry-cache) once sourced from the configured backend. Cache ttl and maximum size are configurable bu have sane defaults.\n\nSchemas are available via HTTP at `/s/$PATH_TO_SCHEMA` or `/s/$SCHEMA_NAME`, depending on the backend.\n\nThe onboard schema cache can be purged via a `GET` or `POST` to the `/c/purge` route.\n\n## Payload Enveloping\n\nEach incoming payload is wrapped in a lightweight envelope.\n\nThis envelope appends a bit of metadata such as `isValid`, `buzTimestamp`, schema `vendor`, `namespace`, `version`, an event `uuid`, the associated `protocol`, etc. Metadata is then used to power payload **routing** and **sharding** as well as **namespace-level statistics**.\n\n**As an example of an `arbitrary pixel` event, wrapped in said envelope:**\n\n```\n{\n    \"uuid\": \"1f9a7a20-8fa7-4179-a0c2-35a80783854a\",\n    \"timestamp\": \"2023-05-03T02:50:59.464042Z\",\n    \"buzTimestamp\": \"2023-05-03T02:50:59.464042Z\",\n    \"buzVersion\": \"x.x.dev\",\n    \"buzName\": \"buz-bootstrap\",\n    \"buzEnv\": \"development\",\n    \"protocol\": \"pixel\",\n    \"schema\": \"io.silverton/buz/pixel/arbitrary/v1.0.json\",\n    \"vendor\": \"io.silverton\",\n    \"namespace\": \"buz.pixel.arbitrary\",\n    \"version\": \"1.0\",\n    \"isValid\": true,\n    \"payload\": {\n        \"msg\": \"hello\",\n        \"subject\": \"zander\"\n    }\n}\n```\n\n# Why Buz?\n\n### It's lightweight\n\n\nIt minimizes the typical infrastructure footprint of collecting events from many different sources and allows for low-infrastructure, highly-flexible rollouts.\n\n### It's flexible\n\n\nBuz doesn't care what your existing systems look like or what you want them to look like in the future.\n\nIt helps with the \"now\", and helps get your infrastructure to where you'd like it to be (without another migration).\n\n### It saves time and money\n\n\nBuz aims to **improve the lives of pipeline maintainers** and **drastically reduce long-term maintenance of event collection systems.**\n\n\nRoll it out fast, keep it going without much thought, and shut it off when it isn't doing anything.\n\n# Try it out\n\n(No, you don't need to talk to anyone. Though we're relatively friendly and there's a [Discord](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=discord\u0026source=github\u0026location=README\u0026rto=https://discord.com/invite/JFKVnVdF2m) if you want to...)\n\nYou'll need [go](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=golang\u0026source=github\u0026location=README\u0026rto=https://go.dev/doc/install) on your machine. But don't need to be a [gopher](https://go.dev/blog/gopher).\n\n\n## Bootstrapping Buz\n\n**Clone:**\n\n    $ git clone git@github.com:silverton-io/buz.git \u0026\u0026 cd buz\n\n\n**Bootstrap:**\n\n    $ make bootstrap\n\n\n## Sending sample events\n\nEvents will be sent to two sinks by default - colorized envelopes will be sent to `stdout` and sent to `buz_events.json` or `buz_invalid_events.json` files.\n\n### POST a cloudevent\n\n    curl -X POST localhost:8080/cloudevents -H 'Content-Type:application/cloudevents+json' -d '{\"dataschema\":\"io.silverton/buz/example/gettingStarted/v1.0.json\", \"data\": {\"userId\": 10, \"name\": \"you\", \"action\": \"didSomething\"}}'\n\n### POST an arbitrary webhook\n\n    curl -X POST \"localhost:8080/webhook\" -H 'Content-Type:application/json' -d '{\"arbitrary\": \"thing\"}'\n\n\n### POST a named (schematized) webhook\n\n    curl -X POST \"localhost:8080/webhook/io.silverton/buz/example/generic/sample/v1.0\" -H 'Content-Type:application/json' -d '{\"id\": \"10\"}'\n\n\n### POST self-describing JSON\n\n    curl -X POST \"localhost:8080/self-describing\" -H 'Content-Type:application/json' -d '{\"payload\": {\"schema\": \"io.silverton/buz/example/generic/sample/v1.0.json\", \"data\": {\"id\": \"10\"}}}'\n\n\n### GET an arbitrary pixel\n\n[localhost:8080/pixel?msg=hello\u0026subject=world](http://localhost:8080/pixel?msg=hello\u0026subject=world)\n\n\n### GET a named (schematized) pixel\n\n[localhost:8080/pixel/io.silverton/buz/example/generic/sample/v1.0?id=10](http://localhost:8080/pixel/io.silverton/buz/example/generic/sample/v1.0?id=10)\n\n\n# Buz plays nicely with others\n\nQuickstart documentation for setting up a lightweight streaming stack with Buz, a sample ui, nginx, Redpanda, and Kowl can [be found here](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=quickstart\u0026source=github\u0026location=README\u0026rto=https://buz.dev/examples/quickstart).\n\n\n# Deploying Buz\n\nBuz can be deployed in a [variety of ways](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=deployment-overview\u0026source=github\u0026location=README\u0026rto=https://buz.dev/deploying/overview). We've included end-to-end (terraformed) samples for AWS and GCP:\n\n* [Buz on AWS Lambda](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=deploy-aws\u0026source=github\u0026location=README\u0026rto=https://buz.dev/deploying/aws/lambda)\n* [Buz on GCP Cloud Run](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=deploy-gcp\u0026source=github\u0026location=README\u0026rto=https://buz.dev/deploying/gcp/cloud_run)\n\n# Documentation\n\nFull documentation can [be found here](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/linkClick/v1.0?link=full-documentation\u0026source=github\u0026location=README\u0026rto=https://buz.dev).\n\n\n🍻🐝\n![bee](https://docstream.buz.dev/pixel/io.silverton/buz/pixel/pageView/v1.0)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsilverton-io%2Fbuz","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsilverton-io%2Fbuz","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsilverton-io%2Fbuz/lists"}