{"id":13492364,"url":"https://github.com/digitalocean/firebolt","last_synced_at":"2025-06-11T01:09:18.422Z","repository":{"id":36103248,"uuid":"218868348","full_name":"digitalocean/firebolt","owner":"digitalocean","description":"Golang framework for streaming ETL, observability data pipeline, and event processing apps","archived":false,"fork":false,"pushed_at":"2024-06-14T07:33:51.000Z","size":393,"stargazers_count":709,"open_issues_count":10,"forks_count":51,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-03-28T10:48:45.983Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/digitalocean.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-31T21:58:02.000Z","updated_at":"2025-03-28T05:19:16.000Z","dependencies_parsed_at":"2024-01-14T04:45:57.482Z","dependency_job_id":"12d1c805-0f88-4369-abfb-82bd3834fe90","html_url":"https://github.com/digitalocean/firebolt","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digitalocean%2Ffirebolt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digitalocean%2Ffirebolt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digitalocean%2Ffirebolt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digitalocean%2Ffirebolt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/digitalocean","download_url":"https://codeload.github.com/digitalocean/firebolt/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digitalocean%2Ffirebolt/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259178543,"owners_count":22817388,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T19:01:05.344Z","updated_at":"2025-06-11T01:09:18.403Z","avatar_url":"https://github.com/digitalocean.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# firebolt ![Code Coverage Badge by Gopherbadger](coverage_badge.png)  ![Build Status](https://github.com/digitalocean/firebolt/actions/workflows/ci.yml/badge.svg) [![Go Report Card](https://goreportcard.com/badge/digitalocean/firebolt)](https://goreportcard.com/report/digitalocean/firebolt)\n![firebolt logo](docs/firebolt-xs.png)\n\nA golang framework for streaming event processing \u0026 data pipeline apps\n\n## Introduction\nFirebolt has a simple model intended to make it easier to write reliable pipeline applications that process a stream of data.\n\nIt can be used to build systems such as:\n* logging/observability pipelines\n* streaming ETL\n* event processing pipelines\n\nEvery application's pipeline starts with a single **source**, the component that receives events from some external system.  Sources \nmust implement the `node.Source` interface.\n\nWe provide one built-in source:\n\n * **kafkaconsumer** - Events come from a Kafka topic, and are passed to the root nodes as `[]byte`\n\nThe processing of your application is executed by its **nodes** which form a processing tree.  Data - events - flow down\nthis tree.   A parent **node** passes results down to it's child **nodes**.  Nodes may process events synchronously or\nasynchronously.  A synchronous node type `node.FanoutNode` returns a slice of results for fanout or 'demultiplexing' use \ncases.  Each node must implement the `node.SyncNode`, `node.FanoutNode`, or `node.AsyncNode` interfaces accordingly.\n\nWe provide two built-in node types:\n\n * **kafkaproducer** - Events are produced onto a kafka topic by an asynchronous producer.\n * **elasticsearch** - Events are bulk indexed into Elasticsearch.\n\nFirebolt has both run and compile-time dependencies on `librdkafka`, see [Developing](#developing)\n\n## Example: Logging Pipeline\nAt DigitalOcean, our first use of Firebolt was in our logging pipeline. This pipeline consumes logs from just about\nevery system we run.   The diagram below depicts the source and nodes in this application.\n\nThis system uses the built-in `kafkaconsumer` source (in yellow) and `kafkaproducer` and `elasticsearch` nodes (in green).\nThe blue nodes are custom to this application.\n\n   ![Logging Pipeline Node Diagram](docs/logging-nodes.png)   \n\n## What does Firebolt do for me?\n\nFirebolt is intended to address a number of concerns that are common to near-realtime data pipeline applications, making it easy\nto run a clustered application that scales predictably to handle large data volume.\n\nIt is not an analytics tool - it does not provide an easy way to support 'wide operations' like record grouping, windowing, \nor sorting that require shuffling data within the cluster.   Firebolt is for 'straight through' processing pipelines that are\nnot sensitive to the order in which events are processed.\n\nSome of the concerns Firebolt addresses include:\n * **kafka sources** Minimal configuration and no code required to consume from a Kafka topic, consumer lag metrics included\n * **kafka sinks** Same for producing to a Kafka topic\n * **loose coupling** Nodes in the pipeline are loosely coupled, making them easily testable and highly reusable\n * **simple stream filtering** Filter the stream by returning `nil` in your nodes\n * **convenient error handling** Send events that fail processing to a kafka topic for recovery or analysis with a few lines of config \n * **outage recovery: offset management** Configurable Kafka offset management during recovery lets you determine the maximum \"catch up\" to attempt after an outage, so you can quickly get back to realtime processing.\n * **outage recovery: parallel recovery** After an outage, process realtime data and \"fill-in\" the outage time window in parallel, with a rate limit on the recovery window.\n * **monitorability** Firebolt exposes Prometheus metrics to track the performance of your Source and all Nodes without writing code.  Your nodes can expose their own custom internal metrics as needed.\n * **leader election** Firebolt uses Zookeeper to conduct leader elections, facilitating any processing that may need to be conducted on one-and-only-one instance.\n\n\n## Documentation\n\n1. [Configuration ](docs/config.md) The configuration file format\n\n2. [Execution ](docs/executor.md) How Firebolt processes your data\n\n3. [Registry ](docs/registry.md) Adding node types to the registry\n\n4. [Sample Application Code ](docs/application.md)  Example code for running the Firebolt executor\n\n5. [Sources ](docs/sources.md) Implementing and using sources\n\n6. [Sync Nodes ](docs/sync-nodes.md) Implementing and using synchronous nodes\n\n7. [Fanout Nodes ](docs/fanout-nodes.md) Implementing and using fanout nodes\n\n8. [Async Nodes ](docs/async-nodes.md) Implementing and using asynchronous nodes\n\n9. [Leader Election ](docs/leader-election.md) Starting leader election and accessing election results\n\n10. [Messaging ](docs/messaging.md) How to send and receive messages between the components of your system\n\n11. [Metrics](docs/metrics.md) What metrics are exposed by default, and how to add custom metrics to your nodes\n\n\n## Built-In Types\n\n1. [Kafka Producer ](docs/node-kafkaproducer.md) Node for producing events onto a Kafka topic\n\n1. [Elasticsearch ](docs/node-elasticsearch.md) Node for indexing documents to an Elasticsearch cluster\n\n## Developing\n\nFirebolt depends on [librdkafka](https://github.com/edenhill/librdkafka) v1.3.0 or later.   To get started building a \nfirebolt app (or working on firebolt itself), install it following the \n[instructions here](https://github.com/edenhill/librdkafka#installation).\n\n An example for debian-based distros:\n```\nsudo wget -qO - https://packages.confluent.io/deb/5.4/archive.key | sudo apt-key add -\nsudo add-apt-repository \"deb [arch=amd64] https://packages.confluent.io/deb/5.4 stable main\"\nsudo apt-get update\nsudo apt-get install -y librdkafka1 librdkafka-dev\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdigitalocean%2Ffirebolt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdigitalocean%2Ffirebolt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdigitalocean%2Ffirebolt/lists"}