https://github.com/reservoird/reservoird

Opinionated lightweight pluggable stream processing framework
https://github.com/reservoird/reservoird

annotate filter framework go pluggable stream transform

Last synced: 3 months ago
JSON representation

Opinionated lightweight pluggable stream processing framework

• Host: GitHub
• URL: https://github.com/reservoird/reservoird
• Owner: reservoird
• License: apache-2.0
• Created: 2019-12-22T18:10:00.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2020-02-03T21:28:56.000Z (about 6 years ago)
• Last Synced: 2024-06-20T11:13:54.742Z (almost 2 years ago)
• Topics: annotate, filter, framework, go, pluggable, stream, transform
• Language: Go
• Homepage: http://reservoird.com
• Size: 119 KB
• Stars: 1
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.txt

Awesome Lists containing this project

README

          
[![Go Report Card](https://goreportcard.com/badge/github.com/reservoird/reservoird)](https://goreportcard.com/report/github.com/reservoird/reservoird)

# Reservoird

Reservoird is an opinionated light weight pluggable stream processing

framework written in golang

## Terminology

The following are the 4 types of plugins used within reservoird

- Ingester - how data is brought into reservoird

- Digester - how data is transformed within reservoird

- Expeller - how data is pushed out of reservoird

- Queue - how data is passed through reservoird

## Best Practices

1. Ingesters should only bring in data

2. Expellers should only push out data

3. Digesters should only transform/filter/annotate data

4. Queues can be either blocking or non-blocking. If non-blocking end name/repo

with nb.

5. All names should be unique. If your plugin is located at

github.com/reservoird/fwd then the recommended name is

com.github.reservoird.fwd

## Data Flow

How data flows through the system

[![](https://mermaid.ink/img/eyJjb2RlIjoiZ3JhcGggTFJcbiAgICBpbjAoaW5wdXQwKSAtLT4gaWcwW2luZ2VzdGVyMF1cbiAgICBpbjEoaW5wdXQxKSAtLT4gaWcxW2luZ2VzdGVyMV1cbiAgICBpbm4oaW5wdXROKSAtLT4gaWduW2luZ2VzdGVyTl1cbiAgICBzdWJncmFwaCByZXNlcnZvaXJkXG4gICAgaWcwIC0tPiBkaTBbZGlnZXN0ZXIwXVxuICAgIGlnMSAtLT4gZGkxW2RpZ2VzdGVyMV1cbiAgICBpZ24gLS0-IGRpbltkaWdlc3Rlck5dXG4gICAgZGkwIC0tPiBleFtleHBlbGxlcl1cbiAgICBkaTEgLS0-IGV4W2V4cGVsbGVyXVxuICAgIGRpbiAtLT4gZXhbZXhwZWxsZXJdXG4gICAgZW5kXG4gICAgZXggLS0-IG8ob3V0cHV0KSIsIm1lcm1haWQiOnsidGhlbWUiOiJkZWZhdWx0In19)](https://mermaid-js.github.io/mermaid-live-editor/#/edit/eyJjb2RlIjoiZ3JhcGggTFJcbiAgICBpbjAoaW5wdXQwKSAtLT4gaWcwW2luZ2VzdGVyMF1cbiAgICBpbjEoaW5wdXQxKSAtLT4gaWcxW2luZ2VzdGVyMV1cbiAgICBpbm4oaW5wdXROKSAtLT4gaWduW2luZ2VzdGVyTl1cbiAgICBzdWJncmFwaCByZXNlcnZvaXJkXG4gICAgaWcwIC0tPiBkaTBbZGlnZXN0ZXIwXVxuICAgIGlnMSAtLT4gZGkxW2RpZ2VzdGVyMV1cbiAgICBpZ24gLS0-IGRpbltkaWdlc3Rlck5dXG4gICAgZGkwIC0tPiBleFtleHBlbGxlcl1cbiAgICBkaTEgLS0-IGV4W2V4cGVsbGVyXVxuICAgIGRpbiAtLT4gZXhbZXhwZWxsZXJdXG4gICAgZW5kXG4gICAgZXggLS0-IG8ob3V0cHV0KSIsIm1lcm1haWQiOnsidGhlbWUiOiJkZWZhdWx0In19)

## Getting Started

1. Download the latest release