https://github.com/nxdir-s/pipelines
pipelines contains generic functions for concurrent processing
https://github.com/nxdir-s/pipelines
concurrency go golang pipelines
Last synced: 5 months ago
JSON representation
pipelines contains generic functions for concurrent processing
- Host: GitHub
- URL: https://github.com/nxdir-s/pipelines
- Owner: nxdir-s
- License: mit
- Created: 2024-08-11T20:40:23.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-11-26T02:53:40.000Z (over 1 year ago)
- Last Synced: 2024-12-06T15:16:11.284Z (over 1 year ago)
- Topics: concurrency, go, golang, pipelines
- Language: Go
- Homepage:
- Size: 12.7 KB
- Stars: 12
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-go - pipelines - Generic pipeline functions for concurrent processing. (Data Structures and Algorithms / Pipes)
- awesome-go - pipelines - Generic pipeline functions for concurrent processing. (Data Structures and Algorithms / Pipes)
- awesome-go-with-stars - pipelines - 02-27 | (Data Integration Frameworks / Pipes)
- fucking-awesome-go - pipelines - Generic pipeline functions for concurrent processing. (Data Structures and Algorithms / Pipes)
- awesome-go-cn - pipelines - s/pipelines) (数据结构与算法 / 管道)
README
[](https://pkg.go.dev/github.com/nxdir-s/pipelines)
[](https://goreportcard.com/report/github.com/nxdir-s/pipelines)
[](https://raw.githack.com/wiki/nxdir-s/pipelines/coverage.html)
# pipelines
Pipelines contains generic functions that help with concurrent processing
## Usage
A pipeline can be created from a slice or map
```go
stream := pipelines.StreamSlice(ctx, data)
```
Or from a generator function
```go
func GenerateData(ctx context.Context) int { return rand.Intn(10) }
stream := pipelines.GenerateStream(ctx, GenerateData)
```
### FanOut
`FanOut` can be used to process data concurrently. Useful for I/O bound processes, but it can be used in any situation where you have a slice or map of data and want to introduce concurrency
```go
const MaxFan int = 3
fanOutChannels := pipelines.FanOut(ctx, stream, ProcessFunc, MaxFan)
```
### FanIn
`FanIn` can be used to merge data into one channel
```go
fanInData := pipelines.FanIn(ctx, fanOutChannels...)
```
## Example
```go
package main
import (
"context"
"fmt"
"math/rand"
"os"
"os/signal"
"strconv"
"time"
"github.com/nxdir-s/pipelines"
)
const (
MaxFan int = 3
)
func GenerateData(ctx context.Context) int {
return rand.Intn(5)
}
func Process(ctx context.Context, timeout int) string {
select {
case <-ctx.Done():
return "context cancelled"
case <-time.After(time.Second * time.Duration(timeout)):
return "slept for " + strconv.Itoa(timeout) + " seconds!"
}
}
func Read(ctx context.Context, messages <-chan string) {
for msg := range messages {
select {
case <-ctx.Done():
return
default:
fmt.Fprintf(os.Stdout, "%s\n", msg)
}
}
}
func main() {
ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
defer cancel()
stream := pipelines.GenerateStream(ctx, GenerateData)
fanOutChannels := pipelines.FanOut(ctx, stream, Process, MaxFan)
messages := pipelines.FanIn(ctx, fanOutChannels...)
go Read(ctx, messages)
select {
case <-ctx.Done():
fmt.Fprint(os.Stdout, "context canceled, exiting...\n")
os.Exit(0)
}
}
```