Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/reugn/go-streams

A lightweight stream processing library for Go
https://github.com/reugn/go-streams

aerospike data-pipeline data-stream etl kafka kafka-streams low-code nats-streaming pipeline pulsar redis stream-processing stream-processor streaming-api streaming-data streams throttling websocket windowing workflow

Last synced: 4 days ago
JSON representation

A lightweight stream processing library for Go

Host: GitHub
URL: https://github.com/reugn/go-streams
Owner: reugn
License: mit
Created: 2019-04-30T17:28:15.000Z (over 5 years ago)
Default Branch: master
Last Pushed: 2024-12-12T00:22:36.000Z (about 1 month ago)
Last Synced: 2025-01-01T18:05:47.080Z (11 days ago)
Topics: aerospike, data-pipeline, data-stream, etl, kafka, kafka-streams, low-code, nats-streaming, pipeline, pulsar, redis, stream-processing, stream-processor, streaming-api, streaming-data, streams, throttling, websocket, windowing, workflow
Language: Go
Homepage: https://pkg.go.dev/github.com/reugn/go-streams
Size: 527 KB
Stars: 1,955
Watchers: 29
Forks: 161
Open Issues: 6
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE

Awesome Lists containing this project

awesome-go - go-streams - Go stream processing library. (Stream Processing / HTTP Clients)
zero-alloc-awesome-go - go-streams - Go stream processing library. (Stream Processing / HTTP Clients)
go-awesome - go streams
awesome-go - go-streams - Go stream processing library. Stars:`2.0K`. (Stream Processing / HTTP Clients)
awesome-go-extra - go-streams - 04-30T17:28:15Z|2022-06-01T08:26:53Z| (Stream Processing / HTTP Clients)
awesome - go-streams - Go stream processing library (Go)
my-awesome - reugn/go-streams - pipeline,data-stream,etl,kafka,kafka-streams,low-code,nats-streaming,pipeline,pulsar,redis,stream-processing,stream-processor,streaming-api,streaming-data,streams,throttling,websocket,windowing,workflow pushed_at:2025-01 star:2.0k fork:0.2k A lightweight stream processing library for Go (Go)

README

        # go-streams

[![Build](https://github.com/reugn/go-streams/actions/workflows/build.yml/badge.svg)](https://github.com/reugn/go-streams/actions/workflows/build.yml)

[![PkgGoDev](https://pkg.go.dev/badge/github.com/reugn/go-streams)](https://pkg.go.dev/github.com/reugn/go-streams)

[![Go Report Card](https://goreportcard.com/badge/github.com/reugn/go-streams)](https://goreportcard.com/report/github.com/reugn/go-streams)

[![codecov](https://codecov.io/gh/reugn/go-streams/branch/master/graph/badge.svg)](https://codecov.io/gh/reugn/go-streams)

A lightweight stream processing library for Go.  

`go-streams` provides a simple and concise DSL to build data pipelines.

![pipeline-architecture-example](./docs/images/pipeline-architecture-example.png)

> [Wiki](https://en.wikipedia.org/wiki/Pipeline_(computing))  

> In computing, a pipeline, also known as a data pipeline, is a set of data processing elements connected in series, where the output of one element is the input of the next one. The elements of a pipeline are often executed in parallel or in time-sliced fashion. Some amount of buffer storage is often inserted between elements.

## Overview

Building blocks:

* Source - A Source is a set of stream processing steps that has one open output.

* Flow - A Flow is a set of stream processing steps that has one open input and one open output. 

* Sink - A Sink is a set of stream processing steps that has one open input. Can be used as a Subscriber.

Implemented Flows ([flow](https://github.com/reugn/go-streams/tree/master/flow) package):  

* Map

* FlatMap

* Filter

* Reduce

* PassThrough

* Split

* FanOut

* RoundRobin

* Merge

* Flatten

* Batch

* Throttler

* SlidingWindow

* TumblingWindow

* SessionWindow

* Keyed

Supported Connectors:

* Go channels

* File system

* Network (TCP, UDP)

* WebSocket

* [Aerospike](https://www.aerospike.com/)

* [Apache Kafka](https://kafka.apache.org/)

* [Apache Pulsar](https://pulsar.apache.org/)

* AWS ([S3](https://aws.amazon.com/s3/))

* Azure ([Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/))

* GCP ([Storage](https://cloud.google.com/storage/))

* [NATS](https://nats.io/)

* [Redis](https://redis.io/)

## Examples

Usage samples are available in the examples directory.

## License

Licensed under the MIT License.