Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/nxdir-s/pipelines
Pipelines contains generic functions for concurrent data processing
https://github.com/nxdir-s/pipelines
concurrency generics go golang pipelines
Last synced: about 1 month ago
JSON representation
Pipelines contains generic functions for concurrent data processing
- Host: GitHub
- URL: https://github.com/nxdir-s/pipelines
- Owner: nxdir-s
- License: mit
- Created: 2024-08-11T20:40:23.000Z (5 months ago)
- Default Branch: main
- Last Pushed: 2024-11-22T18:00:31.000Z (about 2 months ago)
- Last Synced: 2024-11-22T18:37:37.812Z (about 2 months ago)
- Topics: concurrency, generics, go, golang, pipelines
- Language: Go
- Homepage:
- Size: 9.77 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Go Pipelines
Pipelines contains generic functions that help with concurrent processing
## Usage
A pipeline can be created from a slice or map
```go
stream := pipelines.StreamSlice(ctx, data)
```
Or from a generator function
```go
func GenerateData(ctx context.Context) int { return rand.Intn(10) }
stream := pipelines.GenerateStream(ctx, GenerateData)
```
### FanOut
`FanOut` can be used to process data concurrently. Useful for I/O bound processes, but it can be used in any situation where you have a slice or map of data and want to introduce concurrency
```go
const MaxFan int = 3
fanOutChannels := pipelines.FanOut(ctx, stream, ProcessFunc, MaxFan)
```
### FanIn
`FanIn` can be used to merge data into one channel
```go
fanInData := pipelines.FanIn(ctx, fanOutChannels...)
```
## Example
```go
package main
import (
"context"
"fmt"
"math/rand"
"os"
"os/signal"
"strconv"
"time"
"github.com/nxdir-s/pipelines"
)
const (
MaxFan int = 3
)
func GenerateData(ctx context.Context) int {
return rand.Intn(5)
}
func Process(ctx context.Context, timeout int) string {
select {
case <-ctx.Done():
return "context cancelled"
case <-time.After(time.Second * time.Duration(timeout)):
return "slept for " + strconv.Itoa(timeout) + " seconds!"
}
}
func Read(ctx context.Context, messages <-chan string) {
for msg := range messages {
select {
case <-ctx.Done():
return
default:
fmt.Fprintf(os.Stdout, "%s\n", msg)
}
}
}
func main() {
ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
defer cancel()
stream := pipelines.GenerateStream(ctx, GenerateData)
fanOutChannels := pipelines.FanOut(ctx, stream, Process, MaxFan)
messages := pipelines.FanIn(ctx, fanOutChannels...)
go Read(ctx, messages)
select {
case <-ctx.Done():
fmt.Fprint(os.Stdout, "context canceled, exiting...\n")
os.Exit(0)
}
}
```