{"id":13413837,"url":"https://github.com/reugn/go-streams","last_synced_at":"2025-05-14T01:02:53.007Z","repository":{"id":37276441,"uuid":"184304089","full_name":"reugn/go-streams","owner":"reugn","description":"A lightweight stream processing library for Go","archived":false,"fork":false,"pushed_at":"2025-03-22T16:21:17.000Z","size":563,"stargazers_count":2010,"open_issues_count":5,"forks_count":165,"subscribers_count":28,"default_branch":"master","last_synced_at":"2025-04-03T06:48:13.704Z","etag":null,"topics":["aerospike","data-pipeline","data-stream","etl","kafka","kafka-streams","low-code","nats-streaming","pipeline","pulsar","redis","stream-processing","stream-processor","streaming-api","streaming-data","streams","throttling","websocket","windowing","workflow"],"latest_commit_sha":null,"homepage":"https://pkg.go.dev/github.com/reugn/go-streams","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/reugn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-30T17:28:15.000Z","updated_at":"2025-04-02T09:11:53.000Z","dependencies_parsed_at":"2024-03-09T14:36:19.406Z","dependency_job_id":"8c36f1da-d06c-4a65-901f-488a700a98cf","html_url":"https://github.com/reugn/go-streams","commit_stats":{"total_commits":163,"total_committers":8,"mean_commits":20.375,"dds":"0.042944785276073594","last_synced_commit":"4557f02c7ca53cc5be82e8bffd076da2359011d9"},"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reugn%2Fgo-streams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reugn%2Fgo-streams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reugn%2Fgo-streams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reugn%2Fgo-streams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/reugn","download_url":"https://codeload.github.com/reugn/go-streams/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248208679,"owners_count":21065203,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aerospike","data-pipeline","data-stream","etl","kafka","kafka-streams","low-code","nats-streaming","pipeline","pulsar","redis","stream-processing","stream-processor","streaming-api","streaming-data","streams","throttling","websocket","windowing","workflow"],"created_at":"2024-07-30T20:01:50.822Z","updated_at":"2025-04-10T11:33:08.627Z","avatar_url":"https://github.com/reugn.png","language":"Go","readme":"# go-streams\n[![Build](https://github.com/reugn/go-streams/actions/workflows/build.yml/badge.svg)](https://github.com/reugn/go-streams/actions/workflows/build.yml)\n[![PkgGoDev](https://pkg.go.dev/badge/github.com/reugn/go-streams)](https://pkg.go.dev/github.com/reugn/go-streams)\n[![Go Report Card](https://goreportcard.com/badge/github.com/reugn/go-streams)](https://goreportcard.com/report/github.com/reugn/go-streams)\n[![codecov](https://codecov.io/gh/reugn/go-streams/branch/master/graph/badge.svg)](https://codecov.io/gh/reugn/go-streams)\n\n`go-streams` provides a lightweight and efficient stream processing framework for Go. Its concise DSL allows\nfor easy definition of declarative data pipelines using composable sources, flows, and sinks.\n\n![pipeline-architecture-example](./docs/images/pipeline-architecture-example.png)\n\n\u003e [Wiki](https://en.wikipedia.org/wiki/Pipeline_(computing))  \n\u003e In computing, a pipeline, also known as a data pipeline, is a set of data processing elements connected in series,\n\u003e where the output of one element is the input of the next one. The elements of a pipeline are often executed in\n\u003e parallel or in time-sliced fashion. Some amount of buffer storage is often inserted between elements.\n\n## Overview\nThe core module has no external dependencies and provides three key [components](./streams.go)\nfor constructing stream processing pipelines:\n\n- **Source:** The entry point of a pipeline, emitting data into the stream. (One open output)\n- **Flow:** A processing unit, transforming data as it moves through the pipeline. (One open input, one open output)\n- **Sink:** The termination point of a pipeline, consuming processed data and often acting as a subscriber. (One\n  open input)\n\n### Flows\nThe [flow](flow) package provides a collection of `Flow` implementations for common stream\nprocessing operations. These building blocks can be used to transform and manipulate data within pipelines.\n\n- **Map:** Transforms each element in the stream.\n- **FlatMap:** Transforms each element into a stream of slices of zero or more elements.\n- **Filter:** Selects elements from the stream based on a condition.\n- **Fold:** Combines elements of the stream with the last folded value and emits the new value.\n  Requires an initial value.\n- **Reduce:** Combines elements of the stream with the last reduced value and emits the new value.\n  Does not require an initial value.\n- **PassThrough:** Passes elements through unchanged.\n- **Split\u003csup\u003e1\u003c/sup\u003e:** Divides the stream into two streams based on a boolean predicate.\n- **FanOut\u003csup\u003e1\u003c/sup\u003e:** Duplicates the stream to multiple outputs for parallel processing.\n- **RoundRobin\u003csup\u003e1\u003c/sup\u003e:** Distributes elements evenly across multiple outputs.\n- **Merge\u003csup\u003e1\u003c/sup\u003e:** Combines multiple streams into a single stream.\n- **ZipWith\u003csup\u003e1\u003c/sup\u003e:** Combines elements from multiple streams using a function.\n- **Flatten\u003csup\u003e1\u003c/sup\u003e:** Flattens a stream of slices of elements into a stream of elements.\n- **Batch:** Breaks a stream of elements into batches based on size or timing.\n- **Throttler:** Limits the rate at which elements are processed.\n- **SlidingWindow:** Creates overlapping windows of elements.\n- **TumblingWindow:** Creates non-overlapping, fixed-size windows of elements.\n- **SessionWindow:** Creates windows based on periods of activity and inactivity.\n- **Keyed:** Groups elements by key for parallel processing of related data.\n\n**\u003csup\u003e1\u003c/sup\u003e** Utility Flows\n\n### Connectors\nStandard `Source` and `Sink` implementations are located in the [extension](extension) package.\n\n* Go channel inbound and outbound connector\n* File inbound and outbound connector\n* Standard Go `io.Reader` Source and `io.Writer` Sink connectors\n* `os.Stdout` and `Discard` Sink connectors (useful for development and debugging)\n\nThe following connectors are available as separate modules and have their own dependencies.\n* [Aerospike](https://www.aerospike.com/)\n* [Apache Kafka](https://kafka.apache.org/)\n* [Apache Pulsar](https://pulsar.apache.org/)\n* AWS ([S3](https://aws.amazon.com/s3/))\n* Azure ([Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/))\n* GCP ([Storage](https://cloud.google.com/storage/))\n* [NATS](https://nats.io/)\n* [Redis](https://redis.io/)\n* [WebSocket](https://en.wikipedia.org/wiki/WebSocket)\n\n## Usage Examples\nSee the [examples](examples) directory for practical code samples demonstrating how to build\ncomplete stream processing pipelines, covering various use cases and integration scenarios.\n\n## License\nLicensed under the MIT License.\n","funding_links":[],"categories":["开源类库","Stream Processing","流处理","Go","Open source library","Relational Databases","_Table of Contents_","服务端应用"],"sub_categories":["流处理","HTTP Clients","HTTP客户端","Stream Processing","On-Prem","Other Software","查询语"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freugn%2Fgo-streams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freugn%2Fgo-streams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freugn%2Fgo-streams/lists"}